# Research on the Deep Deterministic Policy Algorithm Based on the First-Order Inverted Pendulum
## 2 Construction of a Control System Based on DDPG
### Actor-Critic Algorithm  
In DDPG, the Critic network mainly evaluates the quality of actions based on state value, while the Actor network determines the next action based on action policy. Firstly, an optimization policy based on state value is introduced. Its core is to form a \(Q\) table of states and actions in the environment. The data in the table include the value \(Q(s,a)\) of the action value function when taking an action from the current state. An action is selected based on the table, and it is executed. When selecting an action, the greedy algorithm is followed, as shown in Equation (5), where \(m\) is the total number of selectable actions. After operation,  
Figure 2: Markov decision process. a reward value will be obtained, updating \(Q(s,a)\) as shown in Equation (6), where a is the learning rate, and then the values in the Q-table are updated. \[\pi(a|s)=\left\{ \begin{array}{c}\epsilon/m+1-\epsilon\ if\ a^{*}=\text{ argmax}_{a\in A}Q(s,a)\\ \epsilon/m\ else\end{array} \right. \tag{5}\]  
\[Q(S,A)\leftarrow Q(S,A)+\alpha[R+\gamma\underset{a}{\text{max}}Q(S^{\prime}, A^{\prime})-Q(S,A)] \tag{6}\]  
In practical terms, there are often many state vectors and actions that can make tables very large and make them fall into dimensional disasters. So on the basis of Q-learning, a neural network was added to approximate the value function by neural network to form a deep Q-network. At the same time, the experience playback mechanism [23] and target network were added to improve the convergence of the network. It contains the current network and the target network, which have the same structure. The current network selects actions and updates the network parameter \(\theta\), as shown in Equation (7). The target network calculates the Q target value as \(Q(s^{\prime},a^{\prime};\theta^{\prime}_{i})\), and updates the network by minimizing the loss function shown in Equation (8), where \(\rho(. )\) is the probability distribution of sequence \(s\) and action \(a\). \[y_{i}=E_{s^{\prime}-E}[r+\gamma\text{max}_{a^{\prime}}Q(s^{\prime},a^{\prime}; \theta^{\prime}_{i})] \tag{7}\]  
\[L_{i}(\theta_{i})=E_{s\alpha\sim\rho(.)}[(y_{i}-Q(s,a;\theta_{i}))^{2}] \tag{8}\]  
Figure 3 shows the training process of DQN algorithm:  
From the above analysis, it can be seen that the DQN series of algorithms are suitable for dealing with discrete action problems, and the output policy is a deterministic policy. However, many problems in practice are continuous actions; they are not deterministic strategies but random strategies. Therefore, scholars have proposed reinforcement learning algorithms based on action strategies, which are algorithms used in Actor networks, i.e., policy gradient algorithms. By outputting the probability of action occurring, one or more optimal actions are selected. Assuming that there is an intelligent agent's exploration trajectory \(\tau,\ \pi(a|s,\theta)\) is a random policy. The policy gradient can be expressed as Equation (9). After this equation is solved, the parameters are updated in a gradient ascending manner, as shown in Equation (10). \[\
abla_{\theta}J(\theta)=\frac{1}{\text{N}}\sum_{i=1}^{N}\left[\sum_{t=0}^{T }\
abla_{\theta}\log\pi_{\theta}(a_{i,j}|s_{i,j})R(\tau^{(i)})\right] \tag{9}\]  
\[\theta\leftarrow\theta+\alpha\
abla_{\theta}J(\theta) \tag{10}\]  
By inputting the state into the policy network and outputting the probability of each action, the action can be selected within a continuous interval. However, its updates are based on round updates, which will decrease the efficiency of learning. Value-based algorithms and policy-based algorithms each have their own advantages and disadvantages. Figure 3: DQN algorithm flowchart. tages. So researchers combined the two to form the Actor-Critic algorithm. An Actor is a policy network responsible for interacting with the environment, inputting states and then outputting actions executed by the agent. Critic is a value network, inputting states and actions, outputting evaluations of the actions generated by the policy network.