# Data-Driven Control Algorithm for Snake Manipulator
## 2 Deep Deterministic Policy Gradient  
In the study of motion control of a two-link manipulator, [PERSON] compared 3 kinds of DRL algorithms: A3C, DPPO, and DDPG. He found that the DDPG algorithm has the best convergence effect on the control system. The convergence reward value of the algorithm is the most stable [25]. Since the DDPG algorithm has a better control effect on the manipulator [26] and the motion of the snake manipulator is mainly continuous, the application of the DDPG algorithm was studied in this work. DDPG adds a deterministic strategy network on the basis of the DQN to output action values; thus, compared with the DQN, it only needs to learn the Q network, while DDPG also needs to learn the strategy network. The network structure of DDPG is shown in Figure 1.  
From the figure, we can observe that DDPG has four networks: the current action network, the target action network, the current Q network, and the target Q network. The current action network is mainly used to update the policy network parameter \(\delta^{\mu}\). The network selects the current action A through the current state \(S\). By utilizing target action network sampling, one state \(S^{\prime}\) selects the optimal next action, and its network parameter \(\delta^{\mu^{\prime}}\) duplicates the parameter value of \(\delta^{\mu}\). The current Q network is mainly used to update the value network parameter \(\theta^{Q}\) and to calculate the current Q value. The target Q network is used to calculate the \(Q^{\prime}\) part of the Q value, and the network parameter \(\delta^{Q^{\prime}}\) copies the parameter \(\theta^{Q}\). The two processes of updating network parameters are different. The current network uses the SGD algorithm to update network parameters. The target network uses the soft update algorithm to update the network parameters, as shown in Equation (1). By using the soft update algorithm, the change in the target network parameters is small, and the training is easy to converge, but the learning speed is slow. \[\theta^{Q^{\prime}} \leftarrow\delta\theta^{Q}+(1-\tau)\theta^{Q^{\prime}} \tag{1}\] \[\theta^{\mu^{\prime}} \leftarrow\delta\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}\]  
In Equation (1), \(\theta^{Q^{\prime}}\) denotes the target Q network parameters. \(\theta^{Q}\) denotes the current Q network parameters. The current network parameters \(\theta^{\mu}\) and target network parameters \(\theta^{\mu^{\prime}}\) are constructed by policy \(\mu\). \(\delta\) is the update coefficient, and the value is often 0.1 or 0.01. Figure 1: DDPG network structure. For the definition of the network loss function, the current Q network loss function is similar to that in the DQN, mainly using mean square error, as shown in Formula (2). \[Loss=\sum_{i=1}^{m}(r_{i}-Q(s_{i},a_{i},\omega))^{2} \tag{2}\]  
In Formula (2), Loss denotes the loss function. \(r_{i}\) denotes the reward value of \(i\) moment. \(Q(s_{i},a_{i},\omega)\) denotes the current Q network Q value calculation function. Since the deterministic strategy is adopted in the current action network, its loss function is different from that of PG and A3C. \[\bigtriangledown_{J}(\theta)=\sum_{i}^{m}[\bigtriangledown_{a}Q(s_{i},a_{i}, \omega)\bigm{|}_{s=s_{i},a=\pi_{\theta}(s)}*\bigtriangledown_{\theta*\pi_{ \textit{lhats}}(s)}\bigm{|}_{s=s_{i}}] \tag{3}\]  
In Formula (3), \(\bigtriangledown\) denotes the gradient decreases. \(\pi_{\theta}(s)\) denotes the action selection strategy. For the loss function, the greater the Q value of the target action, the smaller the Loss. The smaller the Q value, the greater the Loss; thus, the status network returns to a negative Q value. \[J(\theta)=-\sum_{i}^{m}Q(s_{i},a_{i},\omega) \tag{4}\]  
In Formula (4), \(s_{i}\) denotes the state of the manipulator at \(i\) moment, including joint angle and angular velocity. \(a_{i}\) denotes the moment value of the manipulator at time \(i\), which is transformed from the state \(s_{i}\) to \(s_{i+1}\) through the dynamic model.