# A Research on Manipulator-Path Tracking Based on Deep Reinforcement Learning
## 3 Simulation Results
### Simulation Results of Multi-Degree-of-Freedom Manipulator  
After exploring the application of reinforcement learning in path tracking, as well as its successful application to the two-link manipulator to achieve the tracking target, the application of the multi-degree-of-freedom manipulator UR5 to conduct path tracking under continuous control was also explored. The specific parameter settings for the experiment are detailed in Section 2.3.2. The experimental results are shown in the following figures. Figure 6(a),b show the path-tracking results without observation noise and with observation noise in the position-control mode, respectively. Figure 6(c),d show the speed-control results and the path-tracking results with and without observation noise in the mode, respectively. Different time intervals are set in each picture; the upper three curves of each picture are the results of not giving the terminal reward, and the lower three curves are the results of giving the terminal reward. Figure 6: Simulation results of path tracking of two-link manipulator based on DQN. Figure 7: _Cont._  
Figure 7: Path-tracking results of UR5 manipulator based on maximum-entropy reinforcement learning. (**a**) Tracking results without observation noise in position-control mode. (**b**) Tracking results with observation noise in position-control mode. (**c**) Tracking results without observation noise in velocity-control mode. (**d**) Tracking results with observation noise in velocity-control mode. In addition, this experiment also quantitatively analyzed the tracking results, calculated the average error between the obtained path and the target path under different experimental conditions and the average distance between the endpoint of the manipulator and the target point at the last moment, and set the average error to less than 0.07 and the error of the target point to less than 0.05 to achieve the tracking-accuracy requirements. The results are shown in Tables 1 and 2. The training-process curve is shown in Figure 8, where (a) is the training-process curve without observation noise in position-control mode, (b) is the training-process curve with the addition of observation noise in position-control mode, (c) is the training-process curve without observation noise in velocity-control mode, and (d) is the training-process curve with the addition of observation noise in velocity-control mode. The X-axis represents the number of training sessions and the Y-axis represents the reward value set in the reinforcement learning. There are six curves in each graph, which represent the training curves with different time intervals with or without terminal reward. The training curves that reach a reward value of 0 on each graph are those with the terminal reward added. In addition, since the system-dynamics model is not considered in the path-tracking experiment based on deep reinforcement learning, in order to verify the advantage of the method based on deep reinforcement learning without the need for the dynamic model, this paper further explores the influence of the change in dynamic characteristics in the experimental results. To this end, we changed the quality of the end effector, the trained model was tested, and the experimental results are shown in Tables 3 and 4. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{**w/o Observation-Noise**} & \multicolumn{3}{c}{**Observation-Noise**} \\
**Position Control** & & \multicolumn{3}{c}{**Interval**} & \multicolumn{3}{c}{**Interval**} \\ \cline{3-8}  & & **1** & **5** & **10** & **1** & **5** & **10** \\ \hline \multirow{3}{*}{Average error between tracks (m)} & w/o terminal reward & 0.0374 & 0.0330 & 0.0592 & 0.0394 & 0.0427 & 0.0784 \\  & terminal reward & 0.0335 & 0.0796 & 0.0502 & 0.0335 & 0.0475 & 0.0596 \\ \cline{1-1}  & w/o terminal reward & 0.0401 & 0.0633 & 0.0420 & 0.0443 & 0.0485 & 0.0292 \\ \cline{1-1}  & terminal reward & 0.0316 & 0.0223 & 0.0231 & 0.0111 & 0.0148 & 0.0139 \\ \hline \hline \end{tabular}
\end{table}