# A Research on Manipulator-Path Tracking Based on Deep Reinforcement Learning
## 2 Method
### Deep-Reinforcement-Learning Algorithm Combined with Manipulator
#### 2.3.2 Application of Multi-Degree-of-Freedom Manipulator  
This paper applies the algorithm to a multi-degree-of-freedom manipulator, UR5, to achieve path tracking under continuous control. The UR5 simulation system, which is shown in Figure 3, uses a deep-reinforcement-learning algorithm to achieve path tracking in the presence of obstacles after generating a path by a conventional path-generation algorithm. The system's actions \(a:=[a_{1},~{}a_{2},~{}a_{3},~{}a_{4},~{}a_{5},~{}a_{6}]\in\mathbb{R}^{6}\), and states are set as \(s:=\left[\theta_{1},~{}\theta_{2},~{}\theta_{3},~{}\theta_{4},~{}\theta_{5},~{ }\theta_{6},~{}\dot{\theta}_{1},~{}\dot{\theta}_{2},~{}\dot{\theta}_{3},~{} \dot{\theta}_{4},~{}\dot{\theta}_{5},~{}\dot{\theta}_{6},~{}\Delta x,~{}\Delta y,~{}\Delta z\right]\in\mathbb{R}^{15}\), where \(\theta_{i},~{}\dot{\theta}_{i}\) is the angle and angular velocity of the first joint, and \(\Delta x,~{}\Delta y,~{}\Delta z\) are the distance between the endpoints \(p\) and the corresponding desired target points \(p^{*}\). The initial position of the endpoint is [\(-0.095,~{}-0.160,~{}0.892\)] and the initial position of the target point is [\(-0.386,~{}0.458,~{}0.495\)]. The desired path is generated by the traditional RRT [1; 29] path-generation algorithm with the stride set to 100. In addition, this experiment set up 4 additional variables to explore the impact of these factors on tracking performance:  
Figure 3: UR5 simulation system. Figure 2: Simulation system of two-link manipulator. 1. The upper control method of the manipulator adopts two control methods, position control and velocity control. The position control is the control of the joint angle, and the input action is the increment of the joint angle. The range of the increment at each moment is set as [\(-0.05\), \(0.05\)] rad. The velocity control is the control of joint angular velocity. The input action is the increment of joint angular velocity. The increment range of each moment was set to [\(-0.8\), \(0.8\)] rad/s in the experiment. In addition, the underlying control of the manipulator adopts the traditional PID torque-control algorithm. 2. Addition of noise to the observations. This paper set up two groups of control experiments, one of which added random noise to the observations; the noise was adopted from the standard normal distribution N(0,1), and the size was \(0.005\times\) N(0,1). 3. The setting of the time-interval distance \(n\). The target path points given by the manipulator at every n time are the target path points at the N*\(n\) time, where N = 1, 2, 3..., and are used to study the effect of different interval points on the tracking results. In the experiments, the interval distances were set as 1, 5, and 10, respectively. 4. Terminal reward. A control experiment was set up in which, during the training process, when the distance between the endpoint of the robotic arm and the target point was within 0.05 m (the termination condition is met), an additional +5 reward was given to study its impact on the tracking results. In order to improve sampling efficiency, the SAC algorithm combines the value-based and policy-based approaches, using the value-based approach to learn the Q-value function or state-value function V and the policy-based approach to learn the policy function, thus making it suitable for the continuous high-dimensional action space and, therefore, more appropriate for control over tracking of the manipulator. As a result, the SAC algorithm was chosen to control tracking of the multi-degree-of-freedom manipulator in this research. All network structures were as follows: each network contained two hidden layers, the number of nodes in each layer was 200, and the activation function of the hidden layer was set to ReLu.