## 4 Routing Policy for DQN  
In order to adapt to dynamic networks with frequent topological changes, GNN-DRL algorithm employs a distributed routing decision mechanism to assign routing decisions to each node instead of pre-constructing routing paths. DQN is executed and distributed at each node, while the centralized training of DQN is implemented to improve stability. In this paper, the \(Q\) value of each neighboring node (i.e., an action) is computed in order. The state representation of the link associated with the node is taken as the input to the DQN, and the \(Q\) value of the neighbor node is taken as the output of the DQN. After obtaining the \(Q\) values of all the neighbor nodes, the neighbor node with the largest \(Q\) value is selected as the next-hop node to forward the packet. In each time slot \(\tau\), the hidden representation representing the network output (i.e., the GGraphSage model, whose output data are the network-side state \(h_{i}^{\tau}\)) will be connected with the user-side state \(s_{d}^{\tau}\). Then the network state vector \(h^{\tau}=\left[h_{i}^{\tau},s_{d}^{\tau}\right]\) will be used as the input to the DQN model. The structure of the neural network model for the DQN agent in the end-to-end transmission routing problem is shown in Figure 3. The model takes the network link quality matrix as the initial state and feeds it into the neural network. The hidden layer of the neural network consists of a two-layer fully connected network and an activation function. Finally, the model outputs \(Q\) values corresponding to multiple candidate paths for end-to-end transmission. According to the \(Q\) values, the candidate paths can be selected as end-to-end transmission paths by a greedy strategy. Figure 4 shows the model of the next node selection framework based on DQN which consists of three components: environment, agent and reward. The DRL agent is operated by interacting with the network simulation environment and the learning process is implemented by DQN. During execution progress, the node that saves the packet fixes its state and selects an action to determine the next hop node using DQN. After performing an action, the node receives a reward from its environment. During training, the centralized trainer draws a small random sample from the experience pool and updates its parameters by minimizing the loss of the DQN. After the parameters are updated, the centralized trainer sends the updated parameters to each node. Upon receiving the updated parameters, each node updates the parameters of its DQN. Then, the old experiences are deleted, and the new policy parameters are used to collect new experiences. This progress is repeated until convergence or a predefined criterion is reached. At the beginning, the DQN agent goes through all traffic demands and tries to change its current routing configuration. For each traffic demand, the DQN agent evaluates and assigns the demand for each possible node on this traffic routing path with the GNN. The DQN outputs a Q value for each action and selects the action to be performed by the agent from the Q values of all actions. Finally, the network performs the selected action, which achieves a new state, a reward and a flag when the agent has completed iterating over all traffic demands. The reward is set to the minimum end-to-end delay between two steps. Figure 4: DQN-based next node selection framework. Figure 3: Architecture of the DQN agent. Since the deviation of \(Q^{\pi}(s,a)\) in the MDP solution of Equation (4) is usually very high, the dominance function \(A^{\pi}(s,a)\) is proposed to replace the value function \(Q^{\pi}(s,a)\), which is expressed as follows:  
\[A^{\pi}(s_{\tau},a_{\tau})=Q^{\pi}(s_{\tau},a_{\tau})-V^{\pi}(s_{\tau})=R_{\tau} -V^{\pi}(s_{\tau}) \tag{9}\]  
where \(V^{\pi}(s)\) is a function of the state value expected to be returned in state \(s\). The superiority of \(A^{\pi}(s,a)\) in selecting an action \(a\) from the set of actions in a particular state \(s\) can be seen.