## 4 Experimental Results  
### Simulation Settings  
In this section, we report the evaluation of the proposed method DDPG-DMP using five robot arm manipulation scenarios from two different benchmark environments. The first three tasks Thrower-v0, Pusher-v0, and Reacher-v1 were from OpenAI Gym ([[https://www.gymlibrary.dev/environments/mujoco/pusher/](https://www.gymlibrary.dev/environments/mujoco/pusher/)]([https://www.gymlibrary.dev/environments/mujoco/pusher/](https://www.gymlibrary.dev/environments/mujoco/pusher/)), accessed on 15 November 2024) and aimed to control a robot to throw, push the target object, or move the end-effector to the randomly generated target positions. We further selected the PandaReach-v2 task from the Panda robot arm simulation environment panda-gym ([[https://github.com/qgallouedec/panda-gym](https://github.com/qgallouedec/panda-gym)]([https://github.com/qgallouedec/panda-gym](https://github.com/qgallouedec/panda-gym)), accessed on 15 November 2024), where the agent directly controls the end-effector position of a Panda robot arm to reach the randomly generated target positions. To additionally demonstrate the advantages of our method in directly controlling robot joints, we extended the PandaReach-v2 task PandaReachjoints-v2 where the agent can directly control all joints of the Panda robot arm. The state-action spaces of the five tasks and the corresponding parameters for DDPG-DMP are shown in Table 2. Considering that all experimental environments supported the standard OpenAI Gym API, the proposed method can be directly applied to any real physical robot system compliant with OpenAI Gym API and ROS communication API, such as panda_robot package ([[https://github.com/rvl-lab-utoronto/panda_robot](https://github.com/rvl-lab-utoronto/panda_robot)]([https://github.com/rvl-lab-utoronto/panda_robot](https://github.com/rvl-lab-utoronto/panda_robot)), accessed on 15 November 2024), which is adapted for Panda robot arm hardware, and robo-gym package ([[https://github.com/jr-robotics/robo-gym](https://github.com/jr-robotics/robo-gym)]([https://github.com/jr-robotics/robo-gym](https://github.com/jr-robotics/robo-gym)), accessed on 15 November 2024), which supports the UR5 robot arm hardware. Most parameters, including the neural network structure, the learning rate of critic networks, and the target update rate, were set uniformly across tasks. We only adjusted the learning rate of actor networks and the steps for warm-up to meet the specific requirements of each task. In terms of the compared baselines, we selected several widely applied RLalgorithms: DDPG [32], Twin Delayed Deep Deterministic Policy Gradient (TD3) [34], and Soft Actor Critic (SAC) [33], as well as the NDP [30], which is based on a DMP policy. All compared methods were developed using PyTorch (DDPG, TD3, and SAC were implemented following open source code: [[https://github.com/Lizhi-sjtu/DRL-code-pytorch](https://github.com/Lizhi-sjtu/DRL-code-pytorch)]([https://github.com/Lizhi-sjtu/DRL-code-pytorch](https://github.com/Lizhi-sjtu/DRL-code-pytorch)) (accessed on 15 November 2024), and the NDP was implemented following its official code: [[https://github.com/shikharbahl/neural-dynamic-policies](https://github.com/shikharbahl/neural-dynamic-policies)]([https://github.com/shikharbahl/neural-dynamic-policies](https://github.com/shikharbahl/neural-dynamic-policies)), accessed on 15 November 2024). NDPs can be seen as the state-of-the-art RL approach using the DMP policy. DDPG, TD3, and SAC were selected as the traditional model-free RL approaches using the actor-critic framework. We did not consider other newer model-freeRL methods as baselines because our proposed DDPG-DMP can be directly applied to any method with an actor-critic framework. In this paper, we primarily focused on the improvement in learning performance when employing the DMP-based policy compared to the NDP and other similar approaches that do not use the DMP-based policy.