## 4 Experiment and Result
### Evaluating Learning Performance: A Comparison with Alternative Reinforcement Learning Methods  
The MuJoCo environment from OpenAI Gym [36] includes jointed body structures and body frames. Table 1 presents an overview of environmental information, with complexity increasing proportionally to the number of joints and environmental constraints. as shown in Figure 6. The Hopper environment operates in a two-dimensional space, where the agent's structure consists of a torso, a thigh, a leg, and a single foot. The agent is trained to apply torque to its three joints to move in the positive direction along the \(x\)-axis to perform jumps. The Half-Cheetah environment operates under similar conditions as the Hopper environment. The agent's structure comprises two thighs, two shins, and two feet, and it learns to move forward along the positive \(x\)-axis by applying torque to six joints. The Ant environment features a 3D quadruped robot composed of a torso connected to four legs. The agent is designed to apply torque to eight joints to move forward. Finally, the Humanoid environment, similar to the Ant environment, is a 3D representation of a bipedal robot designed to simulate human motion. It consists of a pair of legs, arms, and tendons connecting the hips to the torso. The main goal for the agent in these environments is to learn to advance forward as fast and far as possible while maintaining balance and avoiding falls. Figure 6: Mujoco Environments of (**a**) HalfCheetah-v4, (**b**) Hopper-v4, (**c**) Ant-v4, and (**d**) Humanoid-v4. The Hopper-V4 and HalfCheetah-v4 in the MuJoCo are constrained to move only the \(x\)-axis. In contrast, Ant-v4 and Humanoid-v4 supported by MuJoCo can move in multiple directions, resulting in higher environmental complexity. The set value for the perturbation range of each joint has a fixed maximum angle, which prevents abnormal joint movements and defines the permissible range in robot control. Detailed values displayed by the environment are determined as shown in Table 1. Using RL, the aim is to acquire this angle value by ensuring stable control under certain circumstances. In this paper, our objective is to improve learning stability in complex environments and to develop an efficient behavior policy based on sensor state values. For the results, we designed the performance evaluation to standardize all experimental settings within a gym environment, ensuring fairness and objectivity. This approach minimizes potential biases caused by modifications to complex environment settings or parameters, while maintaining the comparability of experiments and ensuring objective performance metrics. Our proposed EMAC method was applied under the same experimental conditions, and its performance was evaluated in comparison to traditional actor-critic structures, considering the complexity of state-action pairs. To verify the experimental results, we integrated the newly designed replay buffer into the EMAC method built on the TD3 structure and performed comparative experiments with traditional RL-based methods such as TD3, DDPG, and PPO. Figure 7 shows the improvement in performance and the average return values over repeated training episodes in the HalfCheetah-v4 and Ant-v4 environments. HalfCheetah-v4 is a two-legged forward environment known for its stability compared to other environments. As shown in Figure 7a, the EMAC results demonstrate improved learning stability, leading to higher average returns compared to standard TD3. In Figure 7b, the learning process in the Ant-v4 environment appears unstable due to complex challenges, such as stumbling or falling backward during quadrupedal motion. However, as a result, we observe that applying the EMAC method to the simulation environments enhances the stability of the learning process. Figure 8 shows the improvement in performance and the average return values over training episodes in the Hopper-v4 and Humanoid-v4 environments. Hopper-v4 features an environment that enables agents to learn in a single-legged position.