# A Research on Manipulator-Path Tracking Based on Deep Reinforcement Learning
## 1 Introduction  
A manipulator is a highly integrated mechanical system combined with electromechanical control. It is a typical time-varying and highly coupled multi-input, multi-output nonlinear system. Its regulation is extremely difficult because it is a complex system with several uncertainties. Researchers have studied manipulator-control techniques extensively over the past few decades. The existing control methods for robots include computational torque control, robust adaptive control [1], adaptive neural-network control [2], output-feedback control [3, 4], dead-zone nonlinear compensation control [5], virtual decomposition control [6], the sliding mode controller [7] and so on. The subject of path tracking [8, 9] is crucial to manipulator control. The challenge is to determine how to ensure that the manipulator's end follows the optimal path after the path-planning algorithm has successfully planned an optimal path. According to [PERSON] [10], each manipulator joint's acceleration and speed are broken out separately. Through position or speed feedback, they modify the target speed or desired acceleration of each joint and use the error as a control input. It is challenging to accurately determine the dynamic characteristics of each manipulator link because the end effector frequently damps various objects during actual manipulator control. The track and control of the manipulator are simultaneously made challenging by the presence of external interference and flaws in the dynamic modeling. In order to maintain the unknown components within a certain range and to account for the disparity between the predicted model and the real model of the manipulator, [PERSON] [11] introduced a robust term to the control input. In order to increase the resilience and stability of the tracking and compensate for the error created bythe external disturbance and the linearization of the dynamic model, [PERSON] [12] optimized the parameters of the neural-network controller using the Lyapunov stability criterion. To prevent collisions with objects in the workspace and irregular robot configurations, [PERSON] [13] presented nonlinear-model predictive control (NMPC), which allows the end effectors of the robot manipulator to follow preset geometric routes in Cartesian space. The robot's nonlinear dynamics, including the dynamics of the actuator, are also taken into account. [PERSON] [14] presented an intelligent control approach that combines sliding mode control (SMC) and fuzzy neural networks (FNN) to accomplish backstepping control for manipulator-path tracking problem. [PERSON] [15] introduced the concept of the virtual rate of change of torque and virtual voltage, which are linear relationships in the state and control variables. In order to improve the motion accuracy, he added kinetic constraints and transformed the non-convex minimum-time route-tracking problem into a convex optimization problem that can be solved quickly. The processes of exact dynamic modeling and inverse kinematics solutions are used in all of the methods mentioned above. However, as the manipulator's degree of freedom increases, the complexity of the dynamic modeling and the number of inverse kinematics solution calculations increase, and the inverse solution is not singular. All these approaches have restrictions in terms of how they can be used [16; 17]. Path-tracking issues can now be solved without the need for system-dynamics modeling thanks to a number of recent successful applications of reinforcement learning to decision-control problems. This opens up new perspectives on path-tracking issues. The manipulator-path-tracking problem can be modeled as a Markov decision process (MDP) using the reinforcement learning [18; 19] approach, which interacts with samples to optimize the control policy. [PERSON] [20] completed the path tracking of the UR5 manipulator using the depth-value-function approach (DQN). The action space is discretized, however, and it is challenging to finely control the manipulator.