## 4 Experimental Results
### OpenAI Robot Arm Scenarios  
The learning performances of the proposed method and other baselines in three tasks in the OpenAI Gym environment, Reacher-v1, Pusher-v0, and Thrower-v0 are analyzed in this subsection. The learning curves of all compared methods in these tasks are presented in Figure 3, where the shaded region represents the corresponding standard deviation over five independent trials with different random seeds. The final average returns obtained by each method over five trials are compared in Table 3. Both results demonstrate the significant superiority of our method in terms of not only learning efficiency and control performance. In the Reacher-v1 task, DDPG, TD3, SAC, and the proposed DDPG-DMP ultimately converged to similar average returns according the learning curves. Regarding the convergence speed, DDPG-DMP achieved an approximately \(-10\) average return in about 26,000 steps, while DDPG required around 33,000 steps, indicating an 11.7% improvement for our method. As a comparison, SAC required more interaction steps to converge in this  
Figure 6: Trajectories of angle position and angle velocity of joints wrist3, wrist4, and wrist5 in one complete rollout using trained DDPG and proposed DDPG-DMP agents. Figure 7: Trajectories of seven joints’ angle position on the simulated Panda robot arm in the PandaReachjoints-v2 task during one complete rollout using trained SAC, TD3, and proposed DDPG-DMP agents. task from a relatively low initial average return due to its inherent random exploration policy. Only TD3 achieved a learning efficiency comparable to DDPG-DMP. Please also note that DDPG-DMP only updates after executing a DMP trajectory segment every \(K=5\) steps while traditional RL methods like DDPG, TD3, and SAC update the agent at every step. This further showcases the significant advantage of our method in updating efficiency. Although the average return calculated over five-step samples resulted in a slight disadvantage for DDPG-DMP as shown in Table 3, this had no significant impact on actual control performance. In contrast, the NDP, which was also based on a DMP policy, struggled to leverage DMP's smooth control advantages in the this task. Its learning curve shows no significant improvements, resulting in our method outperforming the NDP by up to 52.7% in the final converged average return. In Pusher-v0, which has larger state and action spaces, the proposed DDPG-DMP outperformed the DDPG and NDP in the final converged average (with 43.7% and 26.7% higher average return, respectively) while achieving the best learning efficiency. Our method quickly converged to over \(-40\) average return within 30,000 steps. In the most challenging task Thrower-v0, it was clearly observed that the original DDPG failed to learn a proper policy to finish this task. Other traditional RL approaches like TD3 and SAC could not maintain their average return and finally converged to poor control performances with very high standard deviations. As a comparison, the NDP showcased the advantages of a DMP-based policy in control performance for the given task, which requires high coordination between trajectory smoothness and control actions across seven degrees of freedom. It consistently achieved a suboptimal average return, significantly outperforming DDPG, SAC, and TD3. However, NDP took about 150,000 steps to converge, which was relatively slow. In contrast, the proposed algorithm not only enhanced the advantages of the DMP-based policy in control performance but also maximized its learning efficiency. By generating smooth control action trajectories independently for all seven joints simultaneously, our method demonstrated improvements of 65.1%, 43.8%, 34.2%, and 16.2% compared to DDPG, TD3, SAC, and NDP, respectively. Remarkably, DDPG-DMP surpassed the NDP's peak average return using only one-third of the interactions, exhibiting superior learning efficiency. The NDP's poor performances in three torque-based tasks were reasonable.