## 4 Proposed Algorithm  
### Foundation of Dueling Double Deep Q-Network  
Beforehand, we briefly introduce a deep Q network (DQN). It has great advantages in solving high-dimensional computing problems and decision-making problems. It is worth noting that DQN has an evaluation Q-network and target Q-network. The target Q value of the target Q-network is  
\[TargetQ_{d}=r+\gamma\mathrm{max}_{a^{\prime}}Q_{d}\big{(}s^{\prime},a^{\prime} ;\theta\big{)} \tag{11}\]  
Furthermore, the updated formula of Q value in DQN can be denoted by  
\[Q_{d}(s,a,\theta)=Q(s,a,\theta)+\varepsilon(TargetQ_{d}-Q(s,a,\theta)) \tag{12}\]  
where \(\gamma\) is the discount factor,\(\varepsilon\) is the value of the greedy strategy;\(\theta\) is the network parameters of the public part of the network. However, DQN also has some shortcomings, such as overestimation problems when learning the Q value and it can only learn the Q value of taking a certain action in a certain state. Therefore, we introduce D3 QN composed of Double DQN and Dueling DQN.  
The main difference from DQN is that Double DQN divides into two steps when calculating TD target. First, the action corresponding to the maximum Q value is obtained by evaluating the network. It can be given as  
\[a_{\mathrm{max}}\big{(}s^{\prime},\theta\big{)}=\mathrm{argmax}_{a^{\prime}}Q \big{(}s^{\prime},a;\theta\big{)}; \tag{13}\]  
afterward, the action obtained from the above equation is inputted into the target Q network, and the Q value is calculated as follows  
\[TargetQ_{o}=r+\gamma\mathrm{max}_{a^{\prime}}Q_{target}\big{(}s^{\prime},a_{ \mathrm{max}}\big{(}s^{\prime},\theta\big{)};\theta^{*}\big{)} \tag{14}\]  
Therefore, the updated formula for the Q value can be reformulated as  
\[Q_{o}(s,a,\theta)=Q(s,a,\theta)+\varepsilon(TargetQ_{o}-Q(s,a,\theta)) \tag{15}\]  
The advantage of Dueling DQN over DQN is that it can learn the value function of each state, without considering what action to take in that state. The network output of the original DQN algorithm is divided into the following two parts: the value function and the advantage function, which are mathematically expressed as  
\[Q_{u}(s,a;\theta,\xi,\kappa) =\mathrm{V}(s;\theta,\kappa)+A(s,a;\theta,\xi) \tag{16}\]where \(\xi\) denotes the advantage function parameter, and \(\kappa\) denotes the action value function parameter. However, the V value and the \(A\) obtained from Q in the above formula are not unique. In order to solve this problem, the advantage function is decentralized. Therefore, the Q value can be denoted by  
\[Q_{u}(s,a;\theta,\xi,\kappa)=V(s;\theta,\kappa)+\bigg{(}A(s,a;\theta,\xi)- \frac{1}{|A|}\sum_{a^{\prime}}A\big{(}s,a^{\prime};\theta,\xi\big{)}\bigg{)} \tag{17}\]  
Moreover, the target Q value of D3 QN combining Double DQN and Dueling DQN can be given by  
\[TargetQ=r+\gamma\text{max}_{a^{\prime}}Q_{u}\big{(}s^{\prime},a_{\text{max}} \big{(}s^{\prime},\theta\big{)};\theta^{*}\big{)} \tag{18}\]  
Finally, we can formulate the update of the Q value in D3 QN as  
\[Q(s,a;\theta,\xi,\kappa)=Q_{u}(s,a;\theta,\xi,\kappa)+\varepsilon(\text{TargetQ }-Q_{u}(s,a;\theta,\xi,\kappa)). \tag{19}\]  
On the other hand, LSTM has been widely used in reinforcement learning in recent years. Due to the fact that LSTM solves the gradient disappearance and gradient explosion problems during the training of long sequences, it can have better performance in long sequences. User requirements in our research scenario are sometimes variable. Moreover, the selection of the resource allocation action of the current time slot and the opening of the dual connection are related to the resource allocation of the previous time slot and the demand of the current time slot. The memory function of LSTM can effectively capture changes in service demand, so we introduce LSTM into D3 QN to solve the proposed resource allocation problem in this paper.