# Data-Driven Control Algorithm for Snake Manipulator
## 1 Introduction
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Classification Algorithm** & **Algorithm Name** & **Algorithm Improvement** \\ \hline Value-based reinforcement learning (DQN) & Nature DQN & Two identical Q network structures. A choice of action between decoupling target Q value and the calculation of target Q value. \\  & Prioritized replay DQN & The sample is prioritized. The value function of the Q network is divided into two parts: value function and advantage function. \\ \hline Strategy-based reinforcement learning method & Policy gradient & Value-based methods are replaced by policy-based methods. \\ \hline \hline \multirow{4}{*}{Hybrid algorithm actor-critic} & Actor-critic & The two methods, namely policy-based and value-based, are combined. \\  & Asynchronous training framework, network structure optimization, and evaluation point optimization. A general asynchronous concurrent reinforcement learning framework & Asynchronous training framework, network structure optimization, and evaluation point optimization. A general asynchronous concurrent reinforcement learning framework & \\ \cline{1-1}  & Deep deterministic policy gradient (DDPG) & Two actor networks and two critic networks, a total of four neural networks, are used to update the model parameters iteratively. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Classical Deep reinforcement learning algorithms. DRL has been proven to be effective in solving complex control problems of manipulators in OpenAI, such as operating [17], grasping [18; 19], and mobile tasks [20; 21]. [PERSON] applied the DRL strategy to the assembly task of a manipulator. The device completed a task that could not be realized using the traditional control strategy [22]. Through the priority division of the DRL network, [PERSON] realized high-precision millimeter-scale automatic assembly technology [23]. [PERSON] designed an obstacle avoidance algorithm based on DDPG, which solved the convergence problem of the obstacle avoidance motion of the manipulator and ensured its continuity and stability [24]. The contribution of this paper is to propose a data-driven snake manipulator control strategy to solve the control problem of snake manipulators in some complex environments. The main work is as follows:  
1. Based on the model control method, a control system of a two-link model snake manipulator based on DDPG was designed. First, according to the structure of the snake manipulator and the Lagrangian dynamic equation, the dynamic model of the two-link snake manipulator was established. Second, the Q network model and action network model of the DDPG agent were designed. Finally, the simulation results show that the control strategy based on DDPG has good convergence and strong anti-jamming ability. 2. In order to solve the problem of model uncertainty, a data-driven control method of the snake manipulator is proposed. When the number of connecting rods of the serpentine manipulator increases and the environment becomes complex, it is difficult to establish the model of an integral snake manipulator that renders the control effect of the model-based control strategy poor. In view of this, a data-driven control method based on DDPG is proposed. First, the data-driven control method requires a large number of input and output data, and the data set was established by using the results of the traditional control method as the training sample of this method. Second, the DDPG agent model was designed according to the input and output parameters. Finally, based on simulation and comparative results analysis, the feasibility and superiority of the control method was verified. The main sections of this paper are as follows. The Section 1 mainly introduces the control methods currently applied to the manipulator. It briefly combs through the control method of the manipulator, focusing on the DRL algorithm and its application in the manipulator.