## 3 Materials and Methods
### Prioritized Hindsight Experience Replay  
Typically, RL-based methods using a replay buffer are some of the most commonly used approaches in the learning process. These are supported by the overall learning trajectory and demonstrated by high effectiveness. The methods can store all state-action-reward-next state information in a buffer as a tuple. In the case of HER, it is designed to find the best path forward by exploring even the failed experience. This paper proposes Prioritized Hindsight Experience Replay(PHER), which combines the two previously discussed methods, PER and HER. PER can evaluate the weight based on the TD error value, with the error (\(\delta_{t}\)) defined in Equation (4). \[\delta_{t}\gets r+\gamma\max(Q_{target}(s_{i},a;\theta_{i}^{-})-Q(s_{i},a; \theta_{i}^{-}))^{2} \tag{4}\]  
Figure 4: RL-based learning process for device sensor control. Based on this equation, PHER aims to improve learning efficiency and achieve faster convergence in multi-goal environments. HER is fundamentally designed to enable the reuse of diverse experiences. To make this process more efficient, the selection of past experiences should be divided into two stages: by episode and by step within each episode. In this process, the priority \(p_{j}\) should be calculated in two steps. Initially, the probability distribution \(\sigma^{episode}P(j)\), representing the total sum of internal priority values within an episode, is determined using Equation (5). \[j\sim^{episode}P(j)=\frac{\sigma^{episode}p_{j}^{a}}{\sum_{i}^{q^{ episode}}p_{i}^{a}}\quad(\text{Choosing Episode}) \tag{5}\]  
Next, the stored steps within the selected episode are calculated using the probability distribution defined in Equation (6). This approach is effectively designed to leverage diverse experiences over a wider scope. \[j^{\prime}\sim^{step}P(j^{\prime})=\frac{\sigma^{step}p_{j}^{a}}{\sum_{i}^{ step}p_{i}^{a}}\quad(\text{Choosing Step}) \tag{6}\]  
The overall storage and learning process of PHER can be represented as shown in Figure 5. The RL model interacts with the environment, generating state space \(s_{T}\), the final goal \(g_{t}\), and the achieved goal \(ag_{t}\), which are then stored in the replay memory \(D\). The stored experiences are initially selected at the episode level, where an episode is chosen. Within the selected episode, a single experience is then selected for training and the RL-based model is subsequently trained using the HER framework for sampling. Proposed Integrated Method: Extended Maximum Actor-Critic and Prioritized Hindsight Experience Replay  
The proposed EMAC framework begins the policy learning phase using PHER. To select the action \(\pi_{\theta}\), the policy \(\pi_{\theta}\) can return the action of the actor to compare it with \(\pi_{\theta}^{max}\), and it can also obtain a maximum value during the learning process. The action is input as data by calculating the value of the state-action pair for comparison, and the higher action is selected among the input values. The selected action \(a_{t}\) needs to change the state value \(s_{t}\) to \(s_{t+1}\) with the state transition probability \(P(s_{t+1}|s_{t},a_{t})\), resulting in the update of the policy parameter \(\theta\). PHER, can also improve sample efficiency and facilitate reliable exploration of optimal paths, and a replay buffer via PHER \(D\) can store prioritized \(p_{t}\) and data. This approach enhances the original PER method by leveraging the properties of HER to overcome challenges and ensure consistent performance. Figure 5: Architecture of Prioritized Hindsight Experience Replay. In addition, the proposed learning method incorporates a combination of HER and PER for experience storage, which inherently involves saving both the desired and the achieved goals. PHER adjusts the experience reuse mechanism by combining HER with a modified version of PER, prioritizing experiences according to the sequence of episodes and steps. The subsequent learning process follows the HER framework to maintain the overall workflow.