## 4 Dueling DQN-Based Routing Protocol
### Routing-Decision-Learning Based on Dueling DQN  
After modeling the packet-forwarding process as an MDP, we used a dueling DQN to learn routing decisions. The purpose of a reinforcement learning algorithm is to ensure agents learn an optimal policy \(\pi^{*}(a\mid s)\). There are two methods employed to learn the optimal policy in RL algorithms. One is by learning the policy function directly, and the other is by learning the state function \(V^{*}=\max V^{\pi}(s)\) or action function \(Q^{*}=\max Q^{\pi}(s,a)\), indirectly. The state function \(V^{\pi}_{k}(s)\) is the expectation of the sum of all rewards from current state \(s\) when forwarding the \(k\)th packet to final state \(d\) with the policy \(\pi\), which is expressed by the following:  
\[V^{\pi}_{k}(s)=\mathbb{E}^{\pi}\Bigg{\{}\sum_{t=0}^{\infty}\gamma^{t}r_{k}(t+1 )\Bigg{\}} \tag{12}\]  
where \(\gamma\in[0,1]\) is the discount factor, which is used to determine the compromise between immediate and future rewards. According to the Bellman equation, the action function \(Q^{\pi}_{k}(s,a)\) can be expressed by the following:  
\[Q^{\pi}_{k}(s,a)=r_{k}(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim p}\big{[}V^{\pi}_ {k}(s^{\prime})\big{]} \tag{13}\]  
As opposed to [19; 20], we adopted a DQN to learn routing decisions. The Q-value was output by the neural network to evaluate the quality of the action. However, it could be inaccurate, because the \(Q(s,a)\) were related to both state and action, respectively. The degree of correlation between them was not the same. Therefore, in our paper, we utilized a deep dueling Q-network (dueling DQN) algorithm [45] to learn the routing decisions by optimizing the structure of the neural network. Instead of using the original paper's convolutional neural network (CNN) network structure, we used a four-layer neural network to simplify the algorithm. Whether a CNN was used did not affect the dueling DQN algorithm in our scenario. The simple four-layer neural network had an input layer, two hidden layers, and an output layer, which is depicted in Figure 2.  
For dueling DQN, the Q-value was divided into two parts. The first part was only related to state \(s\), which had no relationship with specific action \(a\). We referred to this as the state function \(V(s;\mathbf{w},\mathbf{\alpha})\), which was scalar. The second part was related to both state \(s\) and action \(a\), which we referred to as the advantage function \(A(s,a;\mathbf{w},\mathbf{\beta})\), which was a  
Figure 2: The neural network structure based on a dueling DQN. vector. Each value of the vector corresponded to one action. Then, the Q-value could be represented as follows:  
\[\mathbf{Q}(s,a;\mathbf{w},\mathbf{\alpha},\mathbf{\beta})=V(s;\mathbf{w},\mathbf{\alpha})+A(s,a;\mathbf{w}, \mathbf{\beta}) \tag{14}\]  
where \(w\) is the network parameters vector of the common part, \(\alpha\) is the network parameter vector of the state function, and \(\beta\) is the network parameter vector of the advantage function. However, there was an unidentifiable problem, i.e., given a Q-value, it was impossible to obtain a unique state function and advantage function. To overcome this problem, the advantage function had to be processed by mean centering. Then, the Q-value could be rewritten, as follows:  
\[Q(s,a;\mathbf{w},\mathbf{\alpha},\mathbf{\beta})=V(s;\mathbf{w},\mathbf{\alpha})+\left(A(s,a;\mathbf{w },\mathbf{\alpha})-\frac{1}{\mathcal{A}}\sum_{a^{\prime}\in A}A\big{(}s,a^{\prime };\mathbf{w},\mathbf{\beta}\big{)}\right) \tag{15}\]  
There were two Q-networks in the designed dueling DQN algorithm structure: one was the current Q-network \(Q(s,a;\mathbf{w},\mathbf{\alpha},\mathbf{\beta})\), and the other was the target Q-network \(Q^{\prime}(s,a,\mathbf{w}^{\prime};\mathbf{\alpha}^{\prime},\mathbf{\beta}^{\prime})\), where \(\mathbf{w},\mathbf{\alpha},\mathbf{\beta}\) are the parameter vectors of the current Q-network, and \(\mathbf{w}^{\prime},\mathbf{\alpha}^{\prime},\mathbf{\beta}^{\prime}\) are the parameter vectors of the target Q-network.