## 3 Approach  
### Neural Dynamic Policies  
An NDP [30] attempts to employ the DMP-based policy into the actor-critic structure of the traditional RL approach PPO. Its principle and workflow are illustrated in Figure 1. Receiving the input state \(\mathbf{s}_{t}\), the actor networks output the parameters of DMP rather than the action signal \(\mathbf{a}_{t}\):  
\[[g,\mathbf{w}]=\pi_{\mathbf{\phi}}(\mathbf{s}_{t}) \tag{13}\]  
where \(g\) and \(\mathbf{w}\) are the target position and the weights vector of a DMP-based policy, and \(\mathbf{\phi}\) is the parameter of the actor networks. Let \(\mathbf{y}_{0}\) represent the variables related to each action in the observed state \(\mathbf{s}_{t}\), and assuming that the information of previous \(\mathbf{y}\) is known (the acceleration and velocity at the initial moment could be initialized as zero and iterated), the NDP generates one trajectories with \(m=40\) steps based on the DMP dynamics in Equation (8):  
\[\dot{\mathbf{y}}_{j} =\alpha(\beta(g-\mathbf{y}_{j-1})-\dot{\mathbf{y}}_{j-1}+f(\mathbf{x}_{j}), \tag{14}\] \[\dot{\mathbf{y}}_{j} =\dot{\mathbf{y}}_{j-1}+\dot{\mathbf{y}}_{j-1}dj,\] \[\mathbf{y}_{j} =\mathbf{y}_{j-1}+\dot{\mathbf{y}}_{j-1}dj\]  
where \(dj=0.025\) is set following [30]. At this point, the NDP successfully generates a continuous control action trajectory of length \(m=40\) steps based on the input state \(\mathbf{s}_{t}\). The trajectory \(\left[\mathbf{y}_{j},\dot{\mathbf{y}}_{j},\ddot{\mathbf{y}}_{j}\right]_{j=1}^{m}\) includes position, velocity, and acceleration. An NDP employs an inverse controller to sample the position information from the generated trajectory \(\left[\mathbf{y}_{j},\dot{\mathbf{y}}_{j},\ddot{\mathbf{y}}_{j}\right]_{j=1}^{m}\) at equal intervals to reconstruct a five-step robot control action trajectory \(\mathbf{\mu}=\left[\mathbf{\mu}_{t},\mathbf{\mu}_{t+1},...,\mathbf{\mu}_{t+K-1}\right]\), where \(K=5\), which matches the time steps of state and action spaces for the target task. At the next step, the NDP samples five steps' exploration actions \(\mathbf{a}=\left[\mathbf{a}_{t},\mathbf{a}_{t+1},...,\mathbf{a}_{t+K-1}\right]\) based on \(\mathbf{\mu}\) and sequentially interacts with the external environment using these five-step actions. The five-step interactive samples \(\left\{\mathbf{s}_{t+k},\mathbf{a}_{t+k},\mathcal{R}_{s_{t+k}\hat{s}_{t+k}^{\prime}},\mathbf{s}_{t+k}^{\prime},\mathbf{v}(\mathbf{s}_{t+k})\right\}_{k=0}^{K-1}\) are sequentially collected and stored in the memory buffer \(\mathcal{D}\). The element \(v(\mathbf{s}_{t})\) is the output of critic networks (e.g., the estimated value function), which is required in the update of PPO. Please note that an NDP's inverse controller conducts a simplified mapping that discards velocity and acceleration information from the initial trajectory \(\left[\mathbf{y}_{j},\dot{\mathbf{y}}_{j},\dot{\mathbf{y}}_{j}\right]_{j=1}^{m}\). As a result, the output control trajectory \(\mathbf{a}\) only follows  
Figure 1: Principle and workflow of related work on NDP. the discrete positions of the DMP trajectory, compromising its quality and smoothness. This characteristic hinders an NDP's capability to generate DMP trajectories of physical variables absent from the observed states, such as generating torque trajectories from joint states. It is highlighted that such a conversion would require manually designed inverse controllers based on human prior knowledge [30], which significantly constrains an NDP's applicability in robotics. The update of actor and critic networks in NDPs is mainly similar to PPO, but instead of purely randomly sampling a mini-batch from the memory buffer \(\mathcal{D}\), it selects the entire \(K=5\) steps, continuous trajectory to ensure that the continuity of the generated trajectory is fully considered during training.