## 3 DRL-Based Solution
### The DQN-RS Algorithm
We randomly select \(\mathbf{A}_{k}\) from the action space with probability \(\varepsilon\), otherwise the action is selected by:  
\[\mathbf{A}_{k}=\left.\underset{\mathbf{A}_{k}}{\operatorname{argmax}}Q(s,a| \;\omega,\zeta,\vartheta)\right|_{s=\mathbf{S}_{t},a=\mathbf{A}_{k}} \tag{12}\]  
The dueling DQN-based algorithm also uses a dual network structure with an online network and a target network like the conventional DQN. Meanwhile, we also introduce the experience replay buffer with capacity \(U\). After executing the selected action \(\mathbf{A}_{k}\) at arbitrary \(k\)-th time slot, the agent receives the immediate reward \(R_{k}\) defined in Section 3.2 and transfers it to the next state \(\mathbf{S}_{k+1}\). We continue to track the experience in a replay memory \(\mathcal{D}\) with the state transition sample data \((\mathbf{S}_{k},\mathbf{A}_{k},R_{k},\mathbf{S}_{k+1})\)[20]. When training the network, we randomly take \(X\) sets of experience data from \(\mathcal{D}\) as a minibatch of the samples. An arbitrary set of experience data in the minibatch is denoted as \((\mathbf{S}_{t},\mathbf{A}_{t},R_{t},\mathbf{S}_{t+1})\). Then, the loss function can be expressed as:  
\[\mathcal{L}(\omega,\zeta,\vartheta)\approx\frac{1}{X}{\sum_{l}} \big{(}r_{l}+\gamma\text{max}_{\mathbf{A}_{l+1}}Q^{\prime}\big{(}\mathbf{S}_{l +1},\mathbf{A}_{l+1}|\;\omega^{\prime},\zeta^{\prime},\vartheta^{\prime}\big{)} \;-Q(\mathbf{S}_{t},\mathbf{A}_{l}|\;\omega,\zeta,\vartheta)\big{)}^{2} \tag{13}\]  
where \(\omega^{\prime}\), \(\zeta^{\prime}\) and \(\vartheta^{\prime}\) represent the weights of the target network, which correspond to \(\omega\), \(\zeta\) and \(\vartheta\), respectively. Defining the learning rate as \(\zeta\), the online network can be updated by the gradient descent method, shown in Equation (14):  
\[ \begin{array}{l}\omega\leftarrow\omega-\varsigma\cdot\
abla_{\omega} \mathcal{L}\\ \zeta\leftarrow\zeta-\varsigma\cdot\
abla_{\zeta}\mathcal{L}\\ \vartheta\leftarrow\vartheta-\varsigma\cdot\
abla_{\vartheta}\mathcal{L} \end{array}  \tag{14}\]  
where \(\
abla_{\omega}\), \(\
abla_{\zeta}\) and \(\
abla_{\vartheta}\) represent the gradient vectors of \(\mathcal{L}(\omega,\zeta,\vartheta)\) with respect to \(\omega\), \(\zeta\) and \(\vartheta\), respectively. The weights of the target network are then updated through the slow tracking of the learned online networks, with the learning rate to be \(\lambda\):  
\[ \begin{array}{l}\omega^{\prime}\leftarrow\lambda\omega+(1-\lambda)\omega^{ \prime}\\ \zeta^{\prime}\leftarrow\lambda\zeta+(1-\lambda)\zeta^{\prime}\\ \vartheta^{\prime}\leftarrow\lambda\vartheta+(1-\lambda)\vartheta^{\prime} \end{array}  \tag{15}\]  
Our proposed DQN-RS algorithm is summarized in Algorithm 1. The total algorithm complexity is defined as the product of the total time steps and the algorithm complexity of each time step. We know that the algorithm complexity of an arbitrary time step is determined by the calculation of the parameter updates in the neural network. According to [21], the computational complexity for updating all the neural network parameters is \(\mathcal{O}(p\times q)\) per time step, where \(p\) and \(q\) denote the action dimension and the number of parameters, respectively. In our proposed DQN-RS algorithm, the action dimension is \(N\). The number of parameters in the proposed algorithm can be expressed as \(N\cdot\left(|\omega|+|\theta|+|\zeta|\right).\) Therefore, the approximate complexity at each time step of the proposed DQN-RS algorithm is \(\mathcal{O}(N\cdot\left(|\omega|+|\theta|+|\zeta|\right)\), which is far less than that of the exhaustive method \(\mathcal{O}(N^{M})\).