## 3 Methods  
### DDPG Introduction  
DDPG is a reinforcement learning algorithm that combines Q-learning and policy gradient methods. The network structure of DDPG is shown in Figure 1, including an online actor-network and an online critic network to learn optimal strategies. The online actor-network interacts with the external environment in real time, during which it takes actions under the online critic network's guidance. The online critic network supervises the online actor-network to conduct impending actions based on historical experience while continuously updating its own value function \(Q\). Notably, DDPG follows DQN's idea and introduces a replay buffer and two target networks to guarantee training stability. The replay buffer caches historical experiences in the form of \((s_{i},a_{i},r_{i},s_{i+1})\) for the critic network to reference. The target networks save strategies learned by two online networks in recent training epochs, with which they stabilize the critic's evaluation and actor's learning.  
During each training epoch, DDPG randomly samples a batch of historical experiences from the replay buffer to form a learning sample set \(N^{\bullet}\). For each sample \((s_{i},a_{i},r_{i},s_{i+1})\) in \(N^{\bullet}\), DDPG calculates the target value \(y_{i}\)[3],  
\[y_{i}=r_{i}+\gamma Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1}\Big{|}\theta^{\mu^{ \prime}})\Big{|}\theta^{Q^{\prime}}) \tag{1}\]  
where \(r_{i}\) represents the immediate reward, \(\gamma\) is the discount factor, the target actor-network \(\mu^{\prime}(s_{i+1}\,|\,\theta^{\mu^{\prime}})\) is estimated by its parameter \(\theta^{\mu^{\prime}}\), and \(Q^{\prime}\) is the evaluation reference provided by the target critic network and its parameter \(\theta^{Q^{\prime}}\) based on temporal difference method. Correspondingly, the online critic network calculates the loss \(L\) with target value \(y_{i}\)[3], such as,  
\[L=\frac{1}{N}\sum_{i}\left(y_{i}-Q(s_{i},a_{i}\Big{|}\theta^{Q})\right)^{2} \tag{2}\]  
where \(N\) is the number of experience samples, \(Q(s_{i},a_{i}\,|\,\theta^{Q})\) represents the evaluation made by the online critic network based on current state \(s_{i}\) and action \(a_{i}\). Meanwhile, the online  
Figure 1: DDPG framework.  
-critic network updates its parameters \(\theta^{Q}\) to minimize the loss \(L\). With this updated value function \(Q\), the online critic network guides the online actor to update its strategy \(\theta^{\mu}\)[3], such as,  
\[\
abla\theta^{\mu}J\approx\frac{1}{N}\sum_{i}\
abla_{A}Q(s,a\left|\theta^{Q} \right.)\left|s=si,a=\mu(si)\right.\
abla\theta^{\mu}\left.\mu(s\left|\theta^{ \mu}\right.)\right|s_{i} \tag{3}\]  
At the end of every training epoch, DDPG uses a'soft update' to update the weights of two target networks [3], which helps the target network track the online networks' progress and maintain learning stability, such as,  
\[\begin{array}{l}\theta^{\mu\prime}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta ^{\mu\prime}\\ \theta^{Q\prime}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q\prime}\end{array} \tag{4}\]  
Once the training is completed, the online actor determines the next action \(a\) based on its current state \(s\), strategy \(\theta^{\mu}\), and real-time interactions with the environment [3], such as,  
\[a=\mu(s|\theta^{\mu})+N \tag{5}\]  
where the random noise \(N\) is added by the actor-network to extend the exploration range.  
As shown in Equation (1), the reward function \(r\) directly influences the target value \(y_{i}\) during training, consequently altering the loss value \(L\) and the update of value function \(Q\). Thus, \(r\) reflects DDPG's preference in strategy learning, and a qualified \(r\), as the critic's feedback to the actor's learning, can help the critic network guide the actor to learn the expected strategy more effectively.