# Nonlinear Nonsingular Fast Terminal Sliding Mode Control Using Deep Deterministic Policy Gradient
## 3 DDPG-NNFTSMC Control Design
### Design of DDPG Network  
According to the derivation process of Section 3.1, the uncertainty D in assumption 2 satisfies: \(|\text{D}|\leq\Lambda\) is an important condition of Lyapunov stability. The modified DDPG algorithm can improve the stability and anti-interference of the control system. The delay of control parameters' return and the update makes the action calculated according to the current state take effect in the next control phase, the weights of the neural network and the parameters of experience pool are updated synchronously. During the execution of the algorithm, the data onto the previous step combined with the error information is converted into a reward parameter, which is combined with a data area with N steps. At the beginning of training, the actor network \(\text{Q}(\Gamma,\text{a}|\theta^{\text{Q}})\) (using hyper-parameter \(\theta^{\text{Q}}\)) and critic network \(\mu(\Gamma|\theta^{\mu})\) (using hyperparameter theta \(\theta^{\mu}\)) are initialized, and the experience set \(\varkappa\) is initialized. For the critic algorithm, the current round monitoring data and state and the next action (combined as vectors: \(\Gamma\)) are used as the input parameters of the target network, and then scalar values are output to calculate target values. The formula for target value was:  
\[y_{i}=r_{i}+\gamma Q^{\prime}\Big{[}\Gamma_{i+1},\mu^{\prime}(\Gamma_{i+1}| \theta^{\mu^{\prime}})|\theta^{Q^{\prime}}\Big{]} \tag{23}\]  
The current network in the critic algorithm takes the latest state action history and current states \(\Gamma\) as input and outputs the new action to be taken. This paper uses a real-time update feedback training program with multiple training rounds. In each execution cycle, each batch (in random order) of training data is used to update the weights of the neural network using gradient descent. The critic network is updated by minimizing the mean square error between the output of Q and the original reward data. When taking action in the current state, according to the output of \(\mu\), the policy network is updated by minimizing the output of Q. For the action-value function of critic network output, one part is used to calculate the mean square error:  
\[\text{L}=\frac{1}{\text{N}}\sum_{i}\left[y_{i}-Q\Big{(}\Gamma_{i},\text{a}_{i }|\theta^{\text{Q}}\Big{)}^{2}\right] \tag{24}\]  
Network used to update actor part at the same time:  
\[\
abla_{\theta^{\mu}}\text{J}\approx\sum_{i}V_{\text{a}}Q\Big{(}\Gamma,\text{ a}|\theta^{\text{Q}}\Big{)}\bigg{|}_{\Gamma=\Gamma_{i},\text{a}=\mu(\Gamma_{i})} \
abla_{\theta^{\mu}}\mu(\Gamma|\theta^{\mu})\big{|}_{\Gamma_{i}} \tag{25}\]  
The key of the reward function is to make the network give correct feedback to the network according to the execution results after making decision actions. The merits of the reward function directly affect the training effect and convergence speeds. In reinforcement learning, there is no strict definition of the reward function. It only needs to be able to correctly evaluate the advantages and disadvantages of network output actions. There are two kinds of common reward functions. One is a sparse reward, which is often used in the game to score after completing the task. This kind of reward does not respond well to nondiscrete actions, so it is difficult to quantify the size of the reward. The other is a formal reward, which is only given in the target state, and not anywhere else. This formal reward is easier to promote neural network learning, even if the strategy does not find a solution to the problem, the reward function can also provide positive feedback. In this paper, the input value of the theoretical signal (including but not limited to position, velocity, acceleration, etc.) and the parameters of feedback adjustment (including but not limited to tracking position, tracking velocity, tracking error, control rate, etc.) are combined.