# Reinforcement Learning with Dynamic Movement Primitives for Obstacle Avoidance
## 4 Simulations and Experiments
### Simulations  
In the first simulation, we will test and compare the behaviors in Section 2 and, using our RL algorithm in Section 3, performing the same tasks. In addition, in this simulation, we set \(G=10\), \(\lambda_{0}=0.1\), \(\beta=1\), \(K=400\), \(\Sigma^{\mathbf{\theta}}=0.01\mathbf{I}\) and \(\Sigma^{\lambda}=0.001\). It should be noted that \(\mathbf{\theta}_{0}\) is learned from the initial desired trajectory. For the exploration noise \(\mathbf{\Sigma}^{\mathbf{\theta}}\) and \(\Sigma^{\lambda}\), higher exploration noise usually leads to quicker convergence, but it can also cause safety issues. The fixed noise is used in our work, so relatively small noise is chosen. In this way, the rate of convergence is slow; however, a better performance can be obtained. The learning results of potential field strength \(\lambda\) are presented in Figure 3. The cost does not converge to a perfectly good value. In the meantime, the potential strength \(\lambda\) converges to \(\lambda=0.0051\) in Figure 4. In order to evaluate the performance improvement of the method more intuitively, we calculate the sum of the distances(SOD) of all track points relative to the original ones in a DMP. We have calculated separately when SOD takes different values of \(\lambda\). As shown in Table 1, potential strength calculated by PI2 performances is 10.4% better than a constant value 0.01. \begin{table}
\begin{tabular}{c c c} \hline \(\lambda\) & **0.0051 (PI2)** & **0.01** \\ \hline SD (cm) & 73.55 & 82.08 \\ \hline \end{tabular}
\end{table} 
Table 1: SOD with different values of \(\lambda\). The PI2 algorithm used in this work is a random strategy improvement algorithm, but our optimization function focuses on the optimization of the overall trajectory, so it is difficult to achieve a particularly good overall effect under the condition of ensuring safety. To this end, we set a convergence threshold on the basis of selecting a suitable \(K\) value to improve the convergence speed. After all, convergence to the global optimum cannot be guaranteed [22]. Even so, the advantages of PI2 algorithm are still very obvious, and we can design different cost functions to achieve obstacle avoidance while completing different sub-tasks such as via-point task. In the via-point task, the goal is to pass through the pre-set point \(\mathbf{x}_{set}=\left[ \begin{array}{cc}0.15&0.2\end{array} \right]^{T}\) at \(t=200\) ms, so the cost function of this task could be described as  
\[J(\tau_{i})=\phi_{t_{N}}+\int_{t_{i}}^{t_{N}}\left(G\left\|\mathbf{x}_{t}^{200 ms }-\mathbf{x}_{set}^{200 ms}\right\|+\frac{1}{2}\theta_{t}^{T}\mathbf{R}\theta_{ t}\right)dt. \tag{23}\]  
In this cost function, we set \(G=10,000\), \(K=100\), \(\mathbf{\Sigma}^{\theta}=0.1\mathbf{I}\) and \(\lambda=0.0051\), the optimized potential strength; other parameters are the same as the first simulation. However, different from the first simulation, this simulation focuses on illustrating shape learning, and show the relationship between costs over motion and the corresponding trajectory shape. Therefore, the updating of potential field strength is not considered here, but the optimized results from the first simulation are used as the parameter set. The results are shown in Figure 5. We can see that the initial curve with the optimized potential is the same as the first simulation. After about 60 updates shown in Figure 6, the costs for  
Figure 4: Learning curve for obstacle avoidance. Left: cost profiles with roll-outs. Right: the potential strength \(\lambda\) profiles with roll-outs. Figure 3: Obstacle avoidance behavior with RI. algorithm: PI7  
via-point learning sessions have converged which terminates the training after that. For the via point learning with PI2, as the cost decreases from 19,996 to 20, the trajectory passes through the pre-set point (the blue point in Figure 5) without nearly any error. However, the performance for tracking the original desired trajectory is very poor while passing near the preset point.