## 4 Dueling DQN-Based Routing Protocol
### Routing-Decision-Learning Based on Dueling DQN
The objective function of the Q-network was to minimize the loss function \(L(\mathbf{w}_{t},\mathbf{\alpha}_{t},\mathbf{\beta}_{t})\). The expression of the loss function was the following:  
\[L(\mathbf{w}_{t},\mathbf{\alpha}_{t},\mathbf{\beta}_{t})=\mathbb{E}\Big{[}\big{(}Q_{\rm target }-Q(s_{t},a_{t};\mathbf{w}_{t},\mathbf{\alpha}_{t},\mathbf{\beta}_{t})\big{)}^{2}\Big{]} \tag{16}\]  
where \(Q_{\rm target}=\mathbb{E}\big{[}R_{t+1}+\gamma Q^{\prime}\big{(}s_{t+1},a_{t+ 1};\mathbf{w}^{\prime};\mathbf{\alpha}^{\prime},\mathbf{\beta}^{\prime}\big{)}\big{]}\), and the discount factor \(\gamma\) is a predetermined hyperparameter. We set it as 0.9, based on previous studies. Next, we used the adaptive moment estimation (Adam) to optimize the gradient of the loss function. The gradient of the loss function was expressed by the following:  
\[\
abla_{\mathbf{w},\mathbf{\alpha}_{t},\mathbf{\beta}_{t}}L(\theta_{t})=\mathbb{E}\Big{[} \big{(}Q_{\rm target}-Q(s,a;\mathbf{w}_{t},\mathbf{\alpha}_{t},\mathbf{\beta}_{t})\
abla_ {\mathbf{w},\mathbf{\alpha}_{t},\mathbf{\beta}_{t}}Q(s_{t},a_{t};\mathbf{w}_{t},\mathbf{\alpha}_{t },\mathbf{\beta}_{t})\Big{]} \tag{17}\]  
We utilized a soft update instead of a hard copy for updating the parameters of the target Q-network in order to conserve memory space and stabilize the learning process. The soft update was calculated by the following:  
\[\left\{ \begin{array}{l}\mathbf{\alpha}^{\prime}=(1-\tau)\mathbf{\alpha}^{\prime}+ \tau\mathbf{\alpha}\\ \mathbf{\omega}^{\prime}=(1-\tau)\mathbf{\alpha}^{\prime}+\tau\mathbf{\omega}\\ \mathbf{\beta}^{\prime}=(1-\tau)\mathbf{\beta}^{\prime}+\tau\mathbf{\beta}\end{array} \right. \tag{18}\]  
where \(\tau\in[0,1]\) denotes the compromise between the weights of the target Q-network and current Q-network. Although soft updates cost more time, it increased the learning stability. In the training phase, for the sake of balancing exploration and exploitation, we proposed an improved upper-confidence-bound (IUCB) strategy to select the action, which was expressed as follows:  
\[a_{i}=\left\{\arg\max_{a\in\mathcal{A}_{i}}\left\{Q_{i}(s_{i},a_{i})+\sqrt{ \frac{\ln N_{s_{i}}}{N_{s_{i},a}}}\right\}\right. \tag{19}\]  
where \(N_{s_{i}}\) denotes the number of states \(s_{i}\) that were visited within a fixed period of time, and \(N_{s_{i},a}\) denotes the number of times to select the \(k\)th largest Q-value of candidate forwarding nodes in state \(s_{i}\) within this fixed period of time. The meaning of the parameters \(N_{s_{i}}\) and \(N_{s_{i},a}\) were different from the original UCB policy. They were designed according to our scenario, as compared to [19; 20], in which they adopted a \(\varepsilon\)-greedy strategy for the action. Although the \(\varepsilon\)-greedy strategy was easy to achieve, the IUCB performed better than the \(\varepsilon\)-greedy strategy. The IUCB not only considered the Q-value but also the number of times to select the \(k\)th largest Q-value of the candidate forwarding nodes. Therefore, it could fully explore the action that had been left unexplored at the beginning, and then it tended to exploit that action. After constant iterations, the algorithm achieved convergence. In the testing phase, we used a softmax strategy to select the next-hop forwarding node, that is, we selected the next-hop node based on the probability instead of selecting the next-hop node with the maximum Q-value. The benefit of softmax was that it avoided localized optimums and split the traffic, which was expressed by the following:  
\[P(a)=\frac{\exp(Q(s_{i},a)/\tau)}{\sum_{a_{i}\in A_{i}}\exp(Q(s_{i},a_{i})/ \tau)} \tag{20}\]  
where \(A_{i}\) is the set of actions of node \(i\), and \(\tau\) is the temperature parameter; according to our experience, we set \(\tau=0.05\). If \(\tau\) was large, the probability distribution of different candidate forwarding nodes had few differences, resulting in a larger loss.