## 2 Related Works
### Q-Learning Network
#### 2.2.3 Dueling Double Deep Q-Network  
In reinforcement learning, estimating state values is essential, though evaluating every action in each state is not always required. The dueling double deep Q-network (D3 QN) [12] extends the DDQN framework by integrating [PERSON]'s [10] dueling architecture into both the primary and target networks. This approach decouples the Q-value estimation into two components: the state value function, which quantifies the expected return under policy \(\pi\) for a given state, and the action advantage function, which measures the relative benefit of selecting a specific action. By separating these elements, the D3 QN enhances Q-value precision and mitigates local optima risks inherent in single-network dueling structures. The state-action value function [18], representing the anticipated reward when executing action a under policy \(\pi\) in state s, contrasts with the state value function that captures the policy's overall performance in s. Their difference reflects the incremental advantage of choosing a in s, as formulated in Equation (12).  
\[A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s) \tag{12}\]  
In the dueling network architecture, two distinct data streams are utilized: one stream generates the state value \(V(s;\theta,\beta)\), whereas the other stream outputs the action advantage value \(A(s,a\,;\theta,\alpha)\)[18]. Here, \(\theta\) denotes the parameters responsible for processing features in the input layer, while \(\alpha\) and \(\beta\) correspond to the parameters for the state value and advantage streams, respectively. The output of DDQN utilizing the dueling network structure is given by Equation (13):  
\[Q(s,a\,;\theta,\alpha,\beta)=V(s;\theta,\beta)+A(s,a\,;\theta,\alpha) \tag{13}\]  
Given that the network directly produces the Q-value, the state value V and action A remain unidentified. To improve identifiability, central processing of action A is employed, which enhances optimization stability while preserving performance. Here, \(a^{\prime}\) represents all possible actions, and \(avgA(s,a^{\prime};\theta,\alpha)\) represents the average advantage function value across all actions. The corrected Q-value is given in Equation (14):  
\[Q(s,a\,;\theta,\alpha,\beta)=V(s;\theta,\beta)+\big{(}A(s,a\,;\theta,\alpha)- avgA\big{(}s,a^{\prime};\theta,\alpha\big{)}\big{)} \tag{14}\]The objective function is formulated as presented in Equation (15), and the loss function utilized to adjust the network parameters is given by Equation (16):  
\[y^{D3 QN}=E_{(s,\alpha,r,s^{\prime})}[r+\gamma\cdot Q_{T}(s^{\prime},\text{argmax} Q_{M}(s^{\prime},a^{\prime};\theta,\alpha,\beta);\theta^{\prime},\alpha,\beta)] \tag{15}\]  
\[L(\theta,\alpha,\beta)=E_{(s,\alpha,r,s^{\prime})}[y^{D3 QN}-Q_{M}(s^{\prime},a^ {\prime};\theta,\alpha,\beta)^{2}] \tag{16}\]  
where \(\theta\) and \(\theta^{\prime}\) correspond to the parameters of the primary and target networks, respectively. Additionally, \(\alpha\) and \(\beta\) denote the parameters of the action advantage function A and the state value function V within the dueling network framework. \(Q_{T}\) is the action value function of the target network, while \(Q_{M}\) is the action value function of the main network. Finally, \(\text{argmax}Q_{M}(s^{\prime},a^{\prime};\theta,\alpha,\beta)\) represents the action \(a^{\prime}\) that optimizes the Q-value of the primary network in state \(s^{\prime}\).