## 2 System Model
### Markov Decision Model  
In this subsection, the state space, the action space and the reward function are defined, respectively. (1) State space  
The network state is defined by link characteristics such as link delay, link capacity and current link utilization. These characteristics are stored in a fixed size vector filled with zero padding. At the beginning of a DRL event, a hidden feature vector representation of all nodes is generated by a graph neural network. When the DRL assigns a traffic demand to a specific \(src-dst\) path (that is, a sequence of links in the network), the network state changes. And the user state (including the source node \(src\), the destination node \(dst\), and the traffic demand \(bw\)) is added in the current network state. The ultimate goal is to reduce the average end-to-end delay and increase the network throughput at the end of the event (i.e., when the DRL agent iterates over all traffic demands). (2) Action space  
The number of possible routing paths corresponding to each traffic demand (that is, \(src-dst\) node pair) leads to a high-dimensional action space, even in small networks. It makes the routing problem for DRL agents complicated, as it requires estimating which of the executable actions will result in the smallest end-to-end delay in a long period of time. In other words, the minimum end-to-end delay can be achieved by finding the optimal routing path for each traffic demand. Furthermore, actions need to be defined in a way that is invariant to the alignment of edges and nodes to exploit the ability of GNNs to generalize over graphs (i.e., only link-level features rather than specific identifiers or labels). In order to adapt to dynamic networks with frequent topological changes, GNN-DRL algorithm employs a distributed routing decision mechanism that assigns a routing decision to each node instead of pre-constructing routing paths. We assume that a node \(u_{i}\) with \(m\) neighboring nodes will make a routing decision in the time period \(t\in\{1,2,\cdots,T\}\). In the process of selecting the routing path, the node \(u_{i}\) with the packet needs to select a neighboring node as the next hop to forward the packet. In time period \(t\), we define an action \(a_{t}\in A\), where \(A=\{b_{1},b_{2},\cdots,b_{m}\}\) is the set of neighboring nodes of the node \(u_{i}\). The action \(a_{t}\) represents the selected neighbor node to forward the packet for the node \(u_{i}\), i.e., the node \(u_{i}\) selects the neighboring node \(b_{j}\) to forward the packet. (3) Reward function  
The reward function represents the different immediate rewards from different routing selections. In this chapter, the reward function \(R\) of the DQN agent is related to the QoS parameters which are as follows: delay (\(L\)) and rate (\(W\)). The goal of each agent is to reduce the network delay and improve the overall throughput of the network in a long period of time. The reward function \(R\) is determined as follows:  
\[R=\alpha\cdot W-\beta\cdot L. \tag{2}\]  
where \(\alpha,\beta\in[0,1]\) is the adjustable weight determined by the routing policy. In general, the strategy function \(\pi(a|s):S\times A\rightarrow[0,1]\) is the conditional probability distribution of the agent when it chooses action \(a\) in state \(s\). The goal of the policy at moment \(\tau\) is to maximize the reward \(R_{\tau}\) accumulated in the historical experience as follows:  
\[R_{\tau}=\sum\limits_{k=0}^{\infty}\gamma^{k}R_{\tau+k}. \tag{3}\]  
where \(\gamma\in(0,1)\) is a discount factor that weighs the historical and current reward data for the agent.