# Research on the Deep Deterministic Policy Algorithm Based on the First-Order Inverted Pendulum
## 3 Improved DDPG Based on Local Optimization and Q-Value Overestimation  
### Recursive Small Experience Pool DDPG Algorithm  
In the Actor-Critic algorithm, the output actions of the Actor network depend on the guidance of the Critic network. However, due to the inherent problems of difficult convergence and easily falling into local optima, the combination of the two networks will further increase the difficulty of network convergence [25]. Therefore, in order to solve the problems in the Actor-Critic algorithm, some scholars have integrated the solutions in the DQN algorithm to form a deep deterministic policy gradient algorithm. The DDPG algorithm includes a Critic network based on value updates, an Actor network based on policy updates, and an experience pool for storing samples. The Critic network is divided into the current network \(C_{e}\) and the target network \(C_{t}\), while the Actor network is divided into the current network \(A_{e}\) and the target network \(A_{t}\). The samples in the experience pool are generated by \(A_{e}\), which generates an action from the current state \(S\), as shown in Equation (13):  
\[a_{t}=\mu(s_{t}|\theta^{\mu}) \tag{13}\]  
Among them, \(\mu\) represents the current strategy. \(\theta^{\mu}\) represents the parameters of the current network \(A_{e}\). After executing this action, the reward value \(r\) and the next state \(S_{t+1}\) are generated, and this set of values is stored in the experience pool. The current network \(C_{e}\) calculates the actual Q value \(Q(S_{i},a_{i}|\theta^{Q})\) and the target network \(C_{t}\) calculate the target Q value \(y_{i}\) as shown in Equation (14), where \(\theta^{Q^{\prime}}\) is the network parameter of the \(C_{t}\) network and \(\theta^{\mu^{\prime}}\) is the parameter of the \(A_{t}\) network. Then, the \(C_{e}\) network and the \(A_{e}\) network are updated by minimizing the value of loss function. The definition of the \(C_{e}\) network  
Figure 4: Actor–Critic algorithm flowchart.  
loss function is shown in Equation (15) and the loss function of \(A_{e}\) network is shown in Equation (16):  
\[y_{i}=r_{i}+\gamma Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1}|\theta^{\mu^{\prime}} )|\theta^{Q^{\prime}}) \tag{14}\]  
\[Loss=\frac{1}{N}{\sum_{i}{(y_{i}-Q(s_{i},a_{i}|\theta^{Q}))}^{2}} \tag{15}\]  
\[\
abla_{\theta^{\mu}}J\approx\frac{1}{N}{\sum\
abla_{a}Q(s,a|\theta^{Q})|_{s= s,a=\mu(s_{i})}\
abla_{\theta^{\mu}}\mu(s|\theta^{\mu})|_{s_{i}}} \tag{16}\]  
After \(\theta^{Q}\) and \(\theta^{\mu}\) are updated, \(\theta^{Q^{\prime}}\) and \(\theta^{\mu^{\prime}}\) are updated through soft updates. Unlike the DQN algorithm, which directly copies the parameters of the current network to the target network, this paper adds an updated coefficient \(\tau\) in the DDPG algorithm, which is determined by the size of the experience pool and the effective experience. In short, the experience pool is divided into n small experience pools, an evaluation function is introduced in advance to evaluate the small experience pool, the effective experience is stored in a new small experience pool, and all the effective experience is finally extracted for network updates. Although \(\tau\) is small, it can control each parameter to be updated relative to the previous one and effectively improve local optima and suboptimal, reduce training time, and improve control effectiveness. The updating method is shown in Equation (17):  
\[\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}} \tag{17}\]