## 3 Methodology
### Reinforcement Learning Network
The construction of the actor network is first addressed, which is theoretically grounded in the Deterministic Policy Gradient (DPG) algorithm proposed by [PERSON] et al. in 2014 [29]. A stochastic policy is denoted as \(a\sim\pi_{\theta}(\cdot\mid s)\), whereas if the policy is deterministic, it can be denoted as \(a=\mu_{\theta}(s)\). The Deterministic Policy Gradient Theorem is:  
\[\
abla_{\theta\gamma}J=\mathbb{E}_{s_{t}\sim\rho^{\beta}}\Big{[}\
abla_{a}Q \Big{(}s,a\mid\theta^{Q}\Big{)}\Big{|}s=s_{t},a=\mu(s_{t})\cdot\
abla_{\theta _{\mu}}\mu(s\mid\theta^{\mu})\Big{|}s=s_{t}\Big{]} \tag{2}\]  
It has the same structure as the strategy gradient theorem. However, the deterministic gradient theorem is based on the fact that the value function for each state and action is already available. For a given state in the continuous action space, this research aims to construct an actor network that outputs the action that maximizes the value function through a Deterministic Policy Gradient:  
\[\mu(s)=\arg\max_{a}Q(s,a) \tag{3}\]  
The actor network is the strategy \(\mu\). Next, the research creates the critic network, which is the value function \(Q\), to satisfy the generalized actor-critic framework. The Bellman optimality equation is [27]  
\[Q^{*}(s,a)=\mathop{\mathrm{E}}_{s^{\prime}\sim p}\biggl{[}r(s,a)+\gamma\underset{a ^{\prime}}{max}\;Q^{*}\bigl{(}s^{\prime},a^{\prime}\bigr{)}\biggr{]} \tag{4}\]  
It states that the value of each state under the optimal policy must be equal to the expected return of the optimal action in this state. The Mean Square Bellman Error (MSBE) was used to measure the difference between the value of \(Q_{\phi}\) fitted by the neural network and the value calculated by [PERSON]'s equation:  
\[L(\phi,\mathcal{D})=\mathop{\mathrm{E}}_{(s,a,r,s^{\prime},d)\sim\mathcal{D}} \biggl{[}\biggl{(}Q_{\phi}(s,a)-\biggl{(}r+\gamma(1-d)\underset{a^{\prime}}{max} \;Q_{\phi}\bigl{(}s^{\prime},a^{\prime}\bigr{)}\biggr{)}\biggr{)}^{2}\biggr{]} \tag{5}\]  
When \(s^{\prime}\) is in a terminated state, \(d=1\); otherwise, \(d=0\). The loss function shown in Equation (5) is derived from the Bellman equation and implemented in Deep Q-Learning, as introduced by [PERSON] et al. in their work on Atari learning [6; 30]. The actor network takes the environmental state \(s\) as input and outputs the action \(a\) that maximizes the value function \(Q\) for that state, thereby forming the deterministic policy \(\mu\). This network performs gradient ascent directly on the value function \(Q\), aiming to find the action \(a\) that yields the highest \(Q\)-value. The \(Q\)-value used here is obtained from the critic network's output in the previous iteration. The critic network takes both the environmental state \(s\) and the action \(a\) as inputs, which are generated by the actor network, and outputs the estimated \(Q\)-value. This network is trained using target values computed through the Bellman optimality equation. To stabilize training, DDPG introduces the target network that mirrors the structure of the main networks, which include actor and critic networks. This target network updates more slowly than the main network to prevent training instability caused by frequent updates. The target network parameters are initially copied from the main network. This research employs soft updates, where target network parameters gradually move toward the main network parameters with a small step size, allowing for incremental updates during each episode. The target actor network generates the next actions, while the target critic network estimates target \(Q\)-values used to update the main network. In reinforcement learning, the replay buffer is used to store the experience data (states, actions, rewards, etc.) of the agent's interaction with the environment. It is another important component of the DDPG algorithm. Training stability and efficiency are improved by randomly sampling mini batches from this buffer, breaking temporal correlations in the data.