# A Research on Manipulator-Path Tracking Based on Deep Reinforcement Learning
## 4 Discussion  
The data provided in Tables 1 and 2 were used to examine how path tracking is affected by the four additional factors indicated in Section 2.3.2. The experimental outcomes demonstrate that the algorithm performs satisfactorily for both position control and speed control, and the introduction of noise into system-wide observations during the simulation training contributed to an increase in the control strategy's robustness and noise opposition. It was found that when the value of n was excessively large (\(n=10\)), its convergence effect on the target point was better, but its tracking effect on the target path was worse; when \(n=1\), the situation was the opposite. Therefore, when choosing the value of n, it is necessary to accept a trade-off between the path-tracking effect and the final position of the end point. When the manipulator's end point exceeds the target point's permissible error range during the simulation-training procedure, increasing the reward can help the manipulator better converge to the target point, albeit at the cost of some route-tracking precision. In this experimental scheme, the target path generated by the RRT algorithm is clearly unsatisfactory, as shown in Figure 7. However, the tracking path generated by the SAC algorithm for reinforcement learning in accordance with this target path also becomes smoother under the condition of satisfying the tracking accuracy, which is advantageous for its actual execution. The SAC algorithm also has a better capacity for exploration, and it can accelerate training because it is based on the maximum-entropy feature. Figure 8 shows that the training process's curve reached convergence earlier than expected. Tables 3 and 4 show that even when the load changes, the model trained with an unchanged load quality still provides sufficient stability for deep-reinforcement-learning-based path tracking, demonstrating that kinetic-feature variations have no effect on the algorithm's performance. In light of the benefits of applying deep reinforcement learning for manipulator-path tracking, the approach in this study does not require dynamics modeling. The experimental findings demonstrate that the algorithm presented in this research can, for the most part, satisfy the tracking-accuracy requirements. The two assessment indices of trajectory smoothness and energy consumption, which are more significant in actual operations, were thus chosen for comparison when both methods matched the tracking-accuracy requirements when better than the conventional algorithm in the Jacobi-matrix approach. Tables 5 and 6 show that the proposed algorithm outperforms the  
Figure 10: Redundant-manipulator path-tracking training-process curve. conventional inverse-kinematics approach in terms of energy consumption and trajectory smoothness, demonstrating its efficacy and applicability. The current research on deep-reinforcement-learning algorithms applied to path tracking mainly includes the combination of the DQN algorithm and path tracking and the combination of the DDPG algorithm and path tracking [33]. The DQN method still depends on the robot system for the inverse kinematic solution of the position while performing action execution because it can only handle discrete state quantities and can only discretize the action space into 27 possible actions in Cartesian space. The DDPG algorithm resolves the discretization issue and permits continuous control during path tracking, but it lacks robustness, and even minor parameter changes can have a significant impact on the algorithm's performance, even causing it to stop working altogether. The combination of the SAC algorithm with the manipulator proposed in this paper allows continuous control in the action space during path tracking, and the input and output quantities are the angle and the angular acceleration, which directly control the robotic arm's performance without inverse kinematic effects on the action.