# Nonlinear Nonsingular Fast Terminal Sliding Mode Control Using Deep Deterministic Policy Gradient
## 3 DDPG-NNFTSMC Control Design
### Design of DDPG Network
According to the actual situation, the correlation signal is multiplied by the adjustment coefficient and superimposed, and the results are provided for the reinforcement learning network as the reward function value. In actor-network, the role of actor current network is responsible for the iterative update of policy network parameter \(\theta\), according to the current state \(\Gamma_{\text{t}}\) selects the current action a and interacts with the environment to generate \(\Gamma_{\text{t+1}}\). The actor target network is responsible for selecting the optimal next action \(\text{a}_{\text{t}}\) according to the next state \(\Gamma_{\text{t+1}}\) sampled in the experience playback pool. At the same time, m samples are sampled in the experience set \(\times\) to calculate the Q value of the current state, and soft update is used to update the network weight periodically. \[\left\{ \begin{array}{l}\theta^{\text{Q}^{\prime}}\leftarrow\tau\theta^{ \text{Q}}+(1-\tau)\theta^{\text{Q}^{\prime}}\\ \theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}} \end{array} \right. \tag{26}\]  
When the neural network is needed to output the action, the current state \(\Gamma_{\text{t}}\) is inputted into the actor target network to get the output after selecting the action with the largest reward, \(\text{a}_{\text{t}}\) corresponds to a variable within a predefined range. The formula is as follows:  
\[\text{a}_{\text{t}}=\mu(\Gamma_{\text{t}}\,|\,\theta^{\,\mu})+\text{N}_{\text{ t}} \tag{27}\]  
To improve the foresight of action selection, the random noise \(\text{N}_{\text{t}}\) was added to ensure that the output action has certain randomness, which could be added or removed dynamically according to the need in practical application. The DDPG algorithm used in this paper has the following improvements compared with the traditional DDPG algorithm: 1. In the critic network, the output signal was changed from the single numerical to the combined control rate and error, so as to improve the performance requirements of control rate for the early stage of the model; 2. In the last layer of the performer network, the output was mapped from the discrete action output with the highest probability selected from the experience pool to the numerical output with the highest probability selected. The control diagram is as Figure 1:  
Aiming at the two joint control model used in this experiment, two lightweight DDPG networks are used to jointly control the target, which respectively undertakes the error compensation output and control adjustment of the two joints. In the experiment, using the traditional network architecture will make the training model too complex, the convergence speed is significantly reduced, and there will be an overfitting phenomenon. Therefore, the network uses two layers of network with 32 and 64 neurons as actor-network, two layers of critic-network with 32 neurons each are used to judge and correct the performer network. The control flow chart of DDPG-NNFTSMC is shown in Figure 2.  
Figure 1: DDPG network control structure chart.