## 3 Methodology
### Deep Q-Network Algorithm  
When the state space and action space are small, the Q-Learning algorithm can effectively obtain the optimal strategy. However, in practice, for complex system models, these spaces are often large, which is impractical both for maintaining Q-tables and finding state-action values. In this case, the Q-Learning algorithm may not be able to find the optimal strategy. The deep Q-network introduces a deep learning algorithm to overcome this shortcoming. With the excellent performance of deep learning in complex feature extraction, DQN gives up maintaining a Q-table and uses a neural network to fit a function to output the results of different actions in different states. The Q-value solves the problem of the low learning efficiency of the Q-Learning algorithm in a high-dimensional state space and action space. Compared with maintaining Q-tables, using functions to output Q-values only requires storing the network structure and parameters of the neural network and has stronger generalization capabilities. When training the DQN algorithm, the agent will choose to perform the action with the highest value based on the greedy strategy or randomly select an action in the action space. Then, it will receive a reward, \(r_{n}\), given by the environment and move to the next state, \(s_{n+1}\). The obtained \(<s_{n},a_{n},r_{s}^{a},s_{n+1}>\) are stored in the experience replay buffer. When the number of samples stored in the experience replay buffer reaches the maximum capacity, a small batch of samples are randomly selected from them to train the estimated Q-networkand the target Q-network. Then, the estimated Q-network parameters can be updated using the gradient descent method and periodically update the target Q-network parameters. Figure 3 shows the structure of the DQN algorithm. The goal is to maximize the discounted accumulated reward, which is denoted as  
\[r_{t}=\sum_{n=0}^{\infty}\gamma^{n}r_{t+n} \tag{12}\]  
The RL algorithm uses the state-action value function, \(Q_{\pi}(s,a)=E[r_{t}|s_{t}=s,a_{t}=a]\), to describe the correctness of selecting action \(a\) in state \(s\), and the optimal function is denoted as  
\[Q^{*}(s,a)=\text{max}Q^{\pi}(s,a) \tag{13}\]  
The Bellman equation of the optimal Q-function can be denoted as  
\[Q^{*}(s,a)=E[r(s,a,s^{\prime})+\gamma\underset{a^{\prime}}{\text{max}}Q^{\pi}( s^{\prime},a^{\prime})] \tag{14}\]  
where \(\pi\) is the policy function, \(E\) is the mathematical expectation, and \(s^{\prime}\) and \(a^{\prime}\) are the state and action of the next moment, respectively. \(r(s,a,s^{\prime})\) is the instant reward, ad \(\gamma\underset{\pi}{\text{max}}Q^{\pi}(s^{\prime},a^{\prime})\) is the long-term reward. We take the DNN as the function approximator to fit the Q-function. \[Q(s,a;\theta)\approx Q^{*}(s,a) \tag{15}\]  
where \(\theta\) is the weight parameter of the Q-network. The DQN is a value-based approach generally used in the context of RL. To alleviate the instability of the non-linear network representation function, DQN introduces an experience replay buffer and the target Q-network. The experience replay buffer with a fixed capacity has been introduced. The replay buffer stores the status, actions, rewards, and next moment status generated by the interaction between the agent and the environment during the Markov decision process. DQN stores samples in the experience replay buffer and then uniformly samples mini-batch transmission sequences from the experience replay buffer. Through the experience replay buffer, all sample data can be used multiple times, which helps achieve higher data utilization efficiency. The method based on the experience replay buffer can extract arbitrary data, solving the problem of excessive correlation between samples, which reduces the training effect. Furthermore, it can improve the stability of the algorithm. Online learning strategies may affect the performance of the algorithm and lead to local optimization problems. Figure 3: The structure of DQN algorithm.