## 5 Conclusions  
In this paper, we propose an Extended Maximum Actor-Critic (EMAC) using policy-gradient to improve the learning stability of the actor-critic method using the maximum revenue value of the actor. We also proposed Prioritized Hindsight Experience Replay (PHER), which modified the structure of the replay buffer. These methods are expected to improve the stability of control systems by using data from device sensors. Furthermore, to maximize the efficiency of the training data, we focus on collecting the data and applying it to various experimental settings to check the training performance and to show the optimal training process. Experimental results of EMAC demonstrated improved learning stability in continuous MuJoCo environments. Compared to TD3 in Gym's MuJoCo, EMAC achieved higher average return values, with increases of 14.75% in HalfCheetah-v4, 27.67% in Hopper-v4, 5.09% in Ant-v4, and 47.29% in Humanoid-v4, confirming its effective convergence. However, focusing experiments on a Mujoco environment may reduce reliability. To verify, additional experiments were performed in the DM-Control environment. The findings indicated that EMAC's performance was slightly lower compared to SAC, suggesting that further improvements are needed to improve both learning stability and performance. Furthermore, PHER emphasizes maximizing the efficiency of training data by collecting it and applying it across different experimental configurations to evaluate training performance and identify the most effective learning process. Therefore, it can  
Figure 14: Experimental results in FetchSlide-v2 for average success rate.  
Figure 13: Experimental results in FetchPush-v2 for average success rate.  
be observed that PHER generates a more stable model compared to other buffer methods. Moreover, based on the method-in-per-method approach, PHER effectively acquires and applies data, demonstrating its strong learning potential in multi-goal environments. The PHER method was evaluated experimentally on the basic multi-functional environment Fetchreach-V2, and achieved a high success rate in learning, confirming its effectiveness in reaching Multi-goal. However, we have found that additional methods are still necessary to improve performance in multi-goal, multi-objective design environments.  
Future work involves enhancing the buffer structure to achieve greater stability and performance, as well as further refining EMAC methods to improve reliability and efficiency. Additionally, the performance evaluation in this work was conducted in an online environment, which introduces certain limitations when transitioning to real-world applications. Therefore, in future work, we plan to validate the proposed method by improving it with a fast and robust learning approach designed to accommodate diverse offline, multi-target environments.  
Conceptualization, [PERSON]-H.K., [PERSON]-[PERSON] and [PERSON]-[PERSON]; methodology, [PERSON]-[PERSON], [PERSON]-[PERSON] and [PERSON]-[PERSON]; software, [PERSON]-H.K., [PERSON]-[PERSON] and [PERSON]-[PERSON]; validation, [PERSON]-H.K., [PERSON]-[PERSON] and [PERSON]-S.K.; formal analysis, [PERSON]-H.K., [PERSON] and [PERSON]-[PERSON]; resources, [PERSON]-[PERSON] and [PERSON]-[PERSON]; writing--original draft preparation, [PERSON]-[PERSON] and [PERSON]-[PERSON]; writing--review and editing, M.-S.K.; visualization, J.-H.K.; supervision, M.-S.K.; project administration, Y.-R.C. and M.-S.K. All authors have read and agreed to the published version of the manuscript. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) (No. RS-2022-NR074475 & RS-2022-00144000). Not applicable. Not applicable. Not applicable. Not applicable. Not applicable.  
The original contributions presented in the study are included in the article. Further inquiries can be directed to the corresponding author. The authors declare no conflicts of interest.