## 5 Algorithm
### E2E Operation and DGA Architecture
Once calculated, the M-SDNC verifies the feasibility of E2 EP considering individual flows' QoS requirements and the limitations of the network infrastructure:

\[\begin{split} b_{q_{i}}^{\text{GFRR}}+b_{ij}^{\psi,\text{GFRR}}& \geq c_{ij}^{\psi};\,\forall(i,j)\in a_{t}^{\psi}\\ &\sum_{(i,j)\in a_{t}^{\psi}}\!\!\!



stored in the respective replay buffers \(\mathcal{H}\) (i.e, TED), and used for training of the involved DGAs (trainAgent).

To update the current and target Q-networks, the standard DDQN training procedure is followed--i.e., update of the current Q-Network model every \(|t|\) steps using the extended Bellman equation (Equation (20)), computing the temporal difference error (calculated using the selected loss function) and its back-propagation to update the model weights.

\[Q^{*}(s_{t},a_{t})=r_{t}+\gamma Q(s_{t+1},\operatorname*{argmax}_{a}Q(s_{t+1}, a;\theta_{t}^{-});\theta_{t}) \tag{20}\]

Also, a soft update of the Target Q-Network model is performed every \(N\) steps [47] with a discount factor \(\tau\): \(\theta^{-}\leftarrow\tau*\theta+(1-\tau)*\theta^{-}\). The fundamental part of the 3 DQR lies in the way of calculating paths, i.e., the GetPath function that is implemented by M-QR/DQR and a key decision component--DGA. The DQR architecture of both domain- and overlay-level variants is shown in Figure 4.

Every IDP/E2 EP request issued by SDNC/M-SDNC towards DQR/M-QR includes key flow parameters: source node \(src\), destination node \(dst\), and QoS class. First, DQR/M-QR gets the current network state \(s_{t}\) (provided by M-SDNC/SDNC; getEnvironmentState) and extends it with additional features including, i.a., node/edge expiry information (cf. Section 5.2). DQR/M-QR also includes the information on persistent NTN graphs, which is delivered by GIMMF (aggregation every 5 s, 10 s, etc.). Once state enrichment is achieved, DQR/M-QR calculates \(k\) candidate paths for flow source, destination pairs (\(f_{src}.f_{dst}\)) using the Dijkstra shortest path algorithm, and delay as the metric (GetShortestPaths). For each candidate path, virtual path allocation is performed (virtualPathAlloc), which involves updating links parameters' (GFBR, MFBR, utilisation, cf. Equation (12)) as if the path allocation succeeded. The obtained state for path \(k\), \(s_{t}^{k}\), is later fed to the DGA current policy model \(Q_{\theta}^{k}\). For each state, the policy outputs the single Q-value \(qval=Q(s_{t}^{k},a_{t}^{k};\theta)\) of the allocation. The tuples of action \(a_{t}^{k}\), state \(s_{t}^{k}\), and \(qval_{t}^{k}\) are stored in the buffer \(B_{\sigma}^{\theta}\) to be compared with other virtual path allocations. The final path \(a_{t}\) is selected based on the highest Q-value among the candidate paths and sent back to SDNC/M-SDNC. If no path is available, DQR/M-QR stores the flow metadata to exclude the respective samples from the DGA training process. After the allocation is performed, the respective states in TED are updated with features added by DQR/M-QR (expiry, etc.). Figure 4: Architecture and interactions of DQR/M-QR and DGA (current and target Q-networks and loss function). Domain and overlay identifiers \(\emptyset,\psi\) are omitted for simplification. The adopted router architecture design aims to maintain modularity and customisability. The clear separation of concerns between components (modules responsible for state enrichment, virtual path allocation, path selection, and DRL-specific ones--Q-networks and loss function) enables further extensions or tuning based on domain specificity and requirements (e.g., adding additional node/edge features, adopting multi-path routing scheme, using another DRL method or architecture, etc.). Moreover, for easier implementation and integration with SDN frameworks, DQR/M-QR are loosely coupled SDN applications that offer routing services based on provided network data (from SDNC, TED, GIMMF, etc.). Both current and target Q-networks comprise the same neural network architecture.