## 4 Experiment and Result
### Additional Assessment of Learning Outcomes via a Multi-Goal Strategy  
PHER is partially based on the configuration of HER, making it essential to evaluate its potential learning process in environments with sparse rewards and multi-goal settings. Upon evaluating PHER's performance, we observed that it trained better than the replay buffer and PER in Figure 9. This suggests that the capabilities of PHER, as proposed in this paper, are more robust compared to standard buffers and traditional PERs. We additionally tested and validated the environment in Gym-Robotics [38], as shown in Figure 10. The Reach environment used in this experiment is a simple setting designed to verify whether the learning process in a multi-goal environment enables the manipulator gripper to reach and maintain its position at the target location. The Pick-and-Place environment is designed to train the agent to pick up an object with the gripper and transport it to a designated target location. The Push environment focuses on the agent to push an object, \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Task** & **A2C** & **DDPG** & **PPO** & **SAC** & **TD3** & **EMAC** \\ \hline Finger-Spin & \(2.24\pm 0.44\) & \(106.56\pm 2.06\) & \(662.1\pm 3.49\) & \(976.72\pm 0.8\) & \(919.32\pm 1.81\) & \(986.56\pm 0.62\) \\ Finger-Turn\_easy & \(170\pm 37.75\) & \(157.61\pm 36.15\) & \(202.12\pm 24.03\) & \(952.83\pm 14.06\) & \(538.8\pm 48.96\) & \(941.45\pm 17.28\) \\ Fish-Swim & \(63.93\pm 5.44\) & \(71.66\pm 69.39\) & \(66.85\pm 4.0\) & \(83.11\pm 7.47\) & \(78.98\pm 9.63\) & \(193.1\pm 22.83\) \\ Ball\_in\_cup-Catch & \(127.54\pm 33.32\) & \(830.74\pm 11.98\) & \(944.12\pm 3.74\) & \(979.92\pm 1.17\) & \(975.41\pm 1.38\) & \(978.47\pm 1.21\) \\ Walker-Stand & \(280.56\pm 2.34\) & \(963.98\pm 2.61\) & \(234.60\pm 5.51\) & \(976.40\pm 2.43\) & \(971.64\pm 7.69\) & \(982.28\pm 1.28\) \\ Walker-Walk & \(287.75\pm 4.88\) & \(916.05\pm 4.54\) & \(343.33\pm 5.57\) & \(970.43\pm 1.21\) & \(937.14\pm 2.33\) & \(966.99\pm 1.13\) \\ Walker-Run & \(55.71\pm 0.58\) & \(679.75\pm 6.73\) & \(69.63\pm 1.48\) & \(588.05\pm 1.73\) & \(733.15\pm 1.58\) & \(684.13\pm 1.26\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results of average return and standard error in the DM-control environment. Figure 9: Buffer learning performance in the Humanoid-v4 environment. randomly positioned, to a designated target. In the slide environment, the agent learns to push and slide an object, generated at a random position, to reach a final destination. In this section, we evaluated the replay buffer to confirm the scalability of the original PER method for various environmental simulations, including multi-goal tasks. For experimental validation, the proposed method (PHER) was compared with the original HER, using TD3 in the previously described FetchReach-v2 environment. In this experimental environment, we set multiple goals to move the end-effector to a randomly provided coordinate within 50 steps by controlling a 7-DOF manipulator. The reward is represented by the negative Euclidean distance, which can be minimized down to zero. The training was performed for both methods over 100 iterations, with 800 episodes per iteration, and an overview of the success rate is available in Figure 11. The Reach-v2 environment, being the simplest, is designed to validate the effectiveness of the learning algorithms rather than assess their performance. Using these environments, we have ensured that all the methods being compared are training properly. The FetchPickAndPlace-v2 environment used in our experiments for accurate performance evaluation was designed to pick up an object from a random location and move it to a target location. As shown in Figure 12, PHER showed more consistent learning performance compared to HER in the environment. In a comparison of success rates over 100 iterations, HER recorded an average success rate of 57.72%, while PHER achieved a success rate of 73.69%, representing a performance improvement of about 16%.