# A Research on Manipulator-Path Tracking Based on Deep Reinforcement Learning
## 3 Simulation Results
### Simulation Results of Multi-Degree-of-Freedom Manipulator

Table 1: Results of position-control-mode path tracking. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Position Control** & & **0.5 kg** & **1 kg** & **2 kg** & **3 kg** & **5 kg** \\ \hline \multirow{3}{*}{Average error between tracks (m)} & w/o observation noise & w/o terminal reward & 0.03742 & 0.03743 & 0.03744 & 0.03745 & 0.03746 \\  & & terminal reward & 0.03354 & 0.03359 & 0.03359 & 0.03355 & 0.03355 \\ \cline{1-1}  & observation noise & w/o terminal reward & 0.03943 & 0.03943 & 0.03943 & 0.03942 & 0.03941 \\ \cline{1-1}  & & terminal reward & 0.03346 & 0.03346 & 0.03346 & 0.03346 & 0.03345 \\ \cline{1-1}  & w/o terminal reward & 0.04047 & 0.04047 & 0.04048 & 0.04049 & 0.04050 \\ \cline{1-1} Distance between & w/o observation noise & terminal reward & 0.03165 & 0.03166 & 0.03157 & 0.03159 & 0.03161 \\ \cline{1-1} end points (m) & observation noise & w/o terminal reward & 0.04430 & 0.04441 & 0.04438 & 0.04430 & 0.04436 \\ \cline{1-1} \cline{2-7}  & terminal reward & 0.01110 & 0.01109 & 0.01109 & 0.01108 & 0.01105 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Analysis of dynamic characteristics of position control. The degree of smoothness [30; 31] in the trajectory of the manipulator's end effector has an impact on the overall working effect in both scientific trials and real-world production. A crucial reference point is the energy that the manipulator generates while it operates [32]. Therefore, the suggested algorithm's performance was compared to that of the conventional inverse-kinematics approach in this work in terms of trajectory smoothness and energy usage. In Tables 5 and 6, the experimental findings are displayed. The smoothness of the trajectory was determined by the angle between the tangent vectors of the neighboring points of the curve, and the smoothness of the manipulator motion was determined by calculating the average value of the turning angles across the entire trajectory, where the \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Velocity Control** & & **0.5 kg** & **1 kg** & **2 kg** & **3 kg** & **5 kg** \\ \hline \multirow{4}{*}{
\begin{tabular}{c} Average error \\ between tracks (m) \\ \end{tabular} } & w/o observation noise & w/o terminal reward & 0.03426 & 0.03427 & 0.03425 & 0.03426 & 0.03425 \\  & & terminal reward & 0.02826 & 0.02825 & 0.02866 & 0.02873 & 0.02882 \\  & observation noise & w/o terminal reward & 0.03478 & 0.03479 & 0.03483 & 0.03486 & 0.03497 \\  & & terminal reward & 0.03503 & 0.03503 & 0.03503 & 0.03502 & 0.03501 \\  & w/o observation noise & w/o terminal reward & 0.02326 & 0.02444 & 0.02436 & 0.02430 & 0.02422 \\  & & terminal reward & 0.00831 & 0.01201 & 0.01395 & 0.01463 & 0.01513 \\  & & w/o terminal reward & 0.04560 & 0.04562 & 0.04565 & 0.04569 & 0.04578 \\  & & terminal reward & 0.02748 & 0.02746 & 0.02743 & 0.02741 & 0.02733 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Analysis of Dynamic Characteristics of Velocity Control. Figure 8: Training-process curves. (**a**) Training-process curve without observed noise in position-control mode. (**b**) Training-process curve with observation noise in position-control mode. (**c**) Training-process curve without observed noise in speed-control mode. (**d**) Training-process curve with observation noise in speed-control mode. angle was expressed in the degree system. The energy consumption of the manipulator throughout the path-tracking process is calculated by Equation (16),  
\[E=\int_{t_{0}}^{t_{M}}P(t)dt\cong\sum_{k=0}^{M}P(k)\cdot dt \tag{16}\]  
\[P(k)=\sum_{i=1}^{n}P_{i}(k) \tag{17}\]  
\[P_{i}(k)=\tau_{i}(k)\cdot\dot{\theta_{i}}(k) \tag{18}\]  
where \(k\) is the \(k\)th path point in the entire path, \(i\) is the \(i\)th joint of the manipulator, \(\tau\), \(\dot{\theta}\) is the joint torque and joint speed, \(M\) is the number of path points, and \(dt\) is the distance between the path points.