# Power Battery Scheduling Optimization Based on Double DQN Algorithm with Constraints
## 4 Methodology  
### Deep Reinforcement Learning Based on Double DQN  
Reinforcement learning is a branch within machine learning that emphasizes exploring action learning based on the environment [34]. As shown in Figure 6, the Markov decision process is a classical mathematical framework for describing reinforcement learning problems. Firstly, the agent perceives the state \(\mathrm{S_{t}}\) at the current moment \(\mathrm{t}\), according to strategy \(\pi\), chooses an action from the action space, and executes it. The environment will feedback the action selected by the agent and judge the correctness of the action choice of time \(\mathrm{t}\) with the reward function. The agent enters a new state \(\mathrm{S_{t+1}}\) and continues to select actions. In order to weaken the vision limitations of agent, considering the impact of future rewards on existing choices, the reward at time \(t\) can be written as:  
\[R_{t}=r_{t+1}+\gamma r_{t+2}+\cdots=\sum_{k=1}^{\infty}\gamma^{k}r_{t+k+1} \tag{13}\]  
The goal of reinforcement learning is that the agent finds the optimal strategy \(\pi_{*}\) that brings the maximum reward through continuous learning. The optimal strategy consists of the actions chosen at each step. We assign a value to each action, writing them as \(Q(s,a)\), which represents the Q value obtained by the agent when selecting action a in state \(\mathrm{s}\). The larger the Q value, the greater the probability that the action will be selected by the agent, and the greater the probability that the agent will bring a greater reward in the future after selecting this action. \[Q^{*}(s_{t},a_{t})=argmax_{\pi}Q^{\pi}(s_{t},a_{t}) \tag{14}\]  
The reward function in reinforcement learning is essentially a long\(-\)term feedback. The Q value is easier to calculate compared to the reward function in each step. Initially, the Q\(-\)learning algorithm used the exhaustive method to fit the Q value, but it could only represent limited state action space. There is a risk of falling into the \"curse of dimensionality\". In 2015, the DeepMind team combined deep neural networks with reinforcement learning to form the DQN algorithm [35]. Reinforcement learning has entered a new world. The overall idea of the double DQN algorithm adopted in this paper is that, starting from the initial state, the agent always selects the action with the highest Q value until the target state. The model outputs the reward value and, finally, the optimal strategy. To calculate the Q value, the double DQN algorithm uses the neural network as function approximator to fit the Q value, allowing for continuous state spaces. The input is the current state of the agent, and the output is the Q value of each action. The parameters are trained until convergence. \[\tilde{Q}(s,a|\theta)\approx Q(s,a) \tag{15}\]  
where \(\theta\) is the neural network parameter. Neural network training should have sufficient samples, so there is an experience pool in the double DQN algorithm. During initialization, the agent randomly selects the action and stores results in the form of quads (\(\mathrm{s}\), \(\mathrm{a}\), \(\mathrm{r}\), \(\mathrm{s}^{\prime}\)) in the experience pool. When training the neural network, randomly sampling a batch of data from the empirical pool, which is called experience replay, alleviates the correlation  
Figure 6: Markov decision process. between the data. In addition, a target network with stable training effect is added to prevent the original network overfitting. Therefore, there are two networks with identical structures and different parameters in the double DQN algorithm, which are called policy network and target network, respectively. A fixed goal can ensure the stability of the network obtained by learning. Set the policy network and target network parameters to \(\theta\) and \(\theta^{\prime}\), respectively. Randomly take a quadruple \((s_{t},a_{t},r_{t},s_{t+1})\) from the experience pool; the policy network selects an action that maximizes the output of the neural network according to Equations (16) and (17).