## 3 Design of Deep Reinforcement Learning Controller Based on DDPG_DOB
### Parameters Updating of DDPG Algorithm  
The process of updating parameters in the DDPG algorithm is depicted in Figure 2.  
While updating network parameters, a small batch of samples is randomly selected from the experience pool to train the network. The target value is calculated using the target Critic network, as expressed in Equation (5):  
\[y_{i}=r_{i}+\gamma Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1}\Big{|}\theta^{\mu^{ \prime}})\Big{|}\theta^{Q^{\prime}}), \tag{5}\]  
In Equation (5), \(y_{i}\) represents the Q-value of the current action, \(r_{i}\) is the reward for each step, \(\gamma\) is the discount factor, \(\mu^{\prime}(.)\) is the target policy, \(Q^{\prime}(.)\) refers to the target value, and \(\theta^{\mu^{\prime}}\), \(\theta^{Q^{\prime}}\) denote the network parameters of the target Actor and target Critic networks, respectively.  
The current value of \(Q\) is calculated based on the current state value \(s_{i}\) and action value \(a_{i}\). Subsequently, the online Critic network is updated by minimizing the loss function, as shown in Equation (6):  
\[L(\theta^{Q})=\frac{1}{N}\sum_{i}\big{(}\underbrace{y_{i}-Q(s_{i},a_{i}\Big{|} \theta^{Q})}_{TD\_error}\big{)}^{2}, \tag{6}\]  
In Equation (6), \(\theta^{Q}\) represents the parameter of the online Critic network, and \(N\) signifies the number of samples.  
Parameter updates to the online Actor network are executed through policy gradient techniques, optimizing reward maximization.  
The objective function of the DDPG algorithm is defined as the expected value of the discounted cumulative reward \(J(\theta^{\mu})\):  
\[J(\theta^{\mu})=\mathbb{E}_{\mu}(r_{1}+\gamma r_{2}+\gamma^{2}r_{3}+\cdots+ \gamma^{n}r_{n+1}), \tag{7}\]  
To enhance the agent's reward acquisition, the parameter \(\theta^{\mu}\) should be updated to maximize this objective function. \(Q(s_{i},a_{i}|\theta^{Q})\), with the aim of maximizing the objective function. Consequently, the chain rule method is employed to derive the gradient of the objective function.  
\[\
abla_{\theta^{\mu}}\approx\frac{1}{N}\sum_{i}\left(\
abla_{a}Q(s,a\Big{|} \theta^{Q})\Big{|}_{s=s_{i},a=(s_{i})}\cdot\
abla_{\theta^{\mu}}\mu(s\Big{|} \theta^{\mu})\Big{|}_{s=s_{i}}\right), \tag{8}\]  
where \(\mu(s|\theta^{\mu})|_{s=s_{i}}\) denotes the deterministic policy; \(Q(s,a|\theta^{Q})\big{|}_{s=s_{i},a=\mu(s_{i})}\) signifies the generation of values \(Q\) by selecting actions based on the deterministic policy \(\mu\) in a given  
Figure 2: Parameter update process of the DDPG algorithm.  
state \(s_{i}\). The gradient ascent algorithm is employed to modify the parameters \(\theta^{\mu}\) of the objective function.  
Furthermore, the target network parameters undergo modification through the application of a soft update technique, incrementally adjusting the target network at each time step, as detailed in Equation (9):  
\[\left\{\begin{array}{l}\theta^{\Omega^{\prime}}\leftarrow\tau\theta^{\Omega}+( 1-\tau)\theta^{Q^{\prime}}\\ \theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu}\end{array} \right., \tag{9}\]  
where \(\tau\) symbolizes the soft update coefficient, \(\theta^{\mu}\) represents the parameters of the online Actor network, \(\theta^{Q}\) represents the parameters of the online Critic network, \(\theta^{\mu^{\prime}}\) denotes the attributes of the target Actor network, and \(\theta^{Q^{\prime}}\) signifies the characteristics of the target Critic network.  
Throughout the training process, the utilization of the soft update method serves to maintain gradual changes in the target network parameters, facilitating a consistent gradient computation for the online network and fostering facile convergence during the training procedure.