## 1 Introduction  
Describing the environment as a Markov Decision Process (MDP), reinforcement learning (RL) is a potential branch of machine learning that focuses on iteratively learning tasks through direct interactions without human prior knowledge [1; 2]. It has been widely implemented in both academia and industry, achieving significant success in areas such as games [3; 4; 5], unmanned systems [6; 7; 8], intelligent manufacturing [9; 10; 11], and robot control [12; 13; 14]. In contrast, implementing the unrestricted exploration behaviors of RL and neural network-based control policies in real-world engineering scenarios is extremely costly, particularly when dealing with noise and disturbances [15], which present substantial challenges for implementing learning algorithms to control sophisticated unmanned systems [16]. As a result, developing more sample-efficient methods remains a crucial challenge in the field of RL. The machine learning community has explored various approaches to enhance the sampling efficiency of RL. These included employing entropy and relative entropy to constraint the policy update to improve the agent's efficiency of exploration in high-dimensional states [17; 18], and generating pseudo samples from learned models to boost the training process [19; 20]. However, these methods still largely depend on traditional control policies that use neural networks to approximate and determine one-step actions based on the current state, lacking a mechanism for implementing efficient and smooth control trajectories. Overall, current state-of-the-art RL approaches have primarily enhanced learning capabilities by improving actor-critic network update mechanisms and building models for target tasks. However, research on improving learning efficiency and control performance from the perspective of the forms of control actions and their smoothness remains limited. Unlike the direct control policy based on neural networks, Dynamic Movement Primitives (DMPs) [21; 22] offer an appealing solution for designing robot manipulation trajectories while featuring high non-linearity and real-time performance. DMPs are widely used in robot control scenarios including pouring water without spillage [23; 24] and deformable object manipulation [25; 26; 27] owing to their robustness, guaranteed convergence, and easy spatiotemporal scaling. Inspired by DMPs' advantages in parameterizing smooth and efficient robot motion trajectories, the feasibility of adaptively learning DMP-based control policies in RL on the robot platform has been explored. As a pioneering work that integrated Path Integrals and DMPs within one RL framework, Policy Improvement with Path Integrals (PI2) first investigated the effectiveness of DMPs as the control policy in RL on a simulated quadruped robot with 12 degrees of freedom [28]. Based on this work, a DMP policy incorporating not only robot control trajectories but also contact force signal trajectories was proposed to enhance the performance of PI2, achieving superior performance on a KUKA robot arm in a real-world wood planing scenario [29]. Neural Dynamic Policies (NDPs) [30] attempted to embed DMPs into the actor networks under the modern RL's actor-critic network framework. They employed neural network-based policies to determine the parameters of the DMP controller, which generated smooth and efficient robot control trajectories. Evaluated by several control tasks built on a simulated Kinova Jaco robot arm, NDPs outperformed Proximal Policy Optimization (PPO) [31] in learning capability and control performance. Although NDPs can be considered a state-of-the-art RL approach using the DMP-based policy, the input states and output actions of their DMP-based policy must be consistent. Otherwise, a manually designed inverse controller--which is usually costly--is required. Therefore, it is difficult to support torque control tasks when the observed states are only related to joint angles. This limitation significantly restricts NDPs' potential applications in the field of robot control.