## 4 Experiment and Result
### Evaluating Learning Performance: A Comparison with Alternative Reinforcement Learning Methods
The Hopper environment is designed as a setting where the agent operates with a single leg, to control three joints to facilitate movement. With the lowest number of controllable joints among the environments tested, the Hopper environment makes it more difficult for the agent to demonstrate stable behavior compared to models with two or more legs. While the previous works show relatively lower peak performance in some environments, our proposed EMAC method enhances the average rewards achieved in those settings as \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Information** & **HalfCheetah-v4** & **Hopper-v4** & **Ant-v4** & **Humanoid-v4** \\ \hline Action Num & 6 & 3 & 8 & 17 \\ \hline \multirow{3}{*}{Action Space} & Range: \(-1\)\(\sim\)1 & Range: \(-1.0\)\(\sim\)1.0 & Range: \(-1.0\)\(\sim\)1.0 & Range: \(-0.4\)\(\sim\)0.4 \\  & Size: (6,) & Size: (3,) & Size: (8,) & Size: (17,) \\  & Type: Float32 & Type: Float32 & Type: Float32 & Type: Float32 \\ \hline Observation Num & 17 & 11 & 27 & 45 \\ \hline \multirow{3}{*}{Observation Sapce} & Range: \(-\)inf\(\sim\)inf & Range: \(-\)inf\(\sim\)inf & Range: \(-\)inf\(\sim\)inf & Range: \(-\)inf\(\sim\)inf \\  & Size: (17,) & Size: (11,) & Size: (27,) & Size: (376,) \\ \cline{1-1}  & Type: Float64 & Type: Float64 & Type: Float64 & Type: Float64 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Information on environmental learning in MuJoCo. shown in Figure 7(a). The Humanoid-v4 environment, with its diverse control parameters and observation space, presents a challenging setting for learning. However, as shown in Figure 7(b), EMAC demonstrates faster convergence to stable and superior results compared to the baseline TD3 method. In addition, Table 2 presents numerical analyses of the performance of EMAC in different environments. The Experimental results show that while the proposed method does not always achieve the highest maximum value, its average return value (mean value) is significantly better compared to other existing methods. This suggests that the learning model proposed in this paper demonstrates improved learning stability compared to traditional models. It was observed that EMAC outperformed other Reinforcement Learning algorithms such as DDPG, PPO, and TD3 in terms of performance when evaluating the proposed EMAC in the Mujoco environment. However, it is necessary to consider the possibility that performance comparisons might be overly biased toward specific environments. Therefore, additional performance evaluations were conducted using the DM-Control Suite [37] environment. The additional experiments were performed in the F \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Algorithm** & **HalfCheetah-v4** & **Hopper-v4** & **Ant-v4** & **Humanoid-v4** \\ \hline DDPG & 7730.12 & 1294.84 & 884.41 & 223.63 \\ PPO & 1589.35 & 1527.66 & 997.45 & 542.27 \\ TD3 & 10,084.52 & 1777.23 & 5083.88 & 3629.36 \\ EMAC & 11,572.05 & 2268.97 & 5342.85 & 5345.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Result of learning average return value. Figure 8: Maximum vs. minimum average results after 5 experimental iterations of **(a)** Hopper-v4 and **(b)** Humanoid-v4. Figure 7: Maximum vs. minimum average results after 5 experimental iterations of **(a)** HalfCheetah-v4 and **(b)** Ant-v4. and Walker environments. These DM-Control environments are characterized by their ability to classify and adapt to various tasks, enabling task-specific learning processes. In this experiment, all observations were used as input data to evaluate the algorithm's performance in processing complex state information. First, the Finger environment was used for training with spin and turn-easy tasks. The Fish environment focused on the Swim task, while the Ball-in-Cup environment used the Catch objective. For an objective comparison, the learning performance of EMAC was evaluated against A2C, DDPG, PPO, SAC, and TD3.