## 5 Algorithm
### E2E Operation and DGA Architecture
To this end, two metrics are considered: flow rejection rate (fraction of rejected flow allocation requests) and the standard deviation of link utilisation \(std(u_{ij})\) expressing the overall imbalance of traffic across all links in a time step. For each test case, the agents were trained in low-traffic environments for 200 episodes lasting 0.5 h each (500 flow requests per episode--\(N_{flows}=500\)) and the models were chosen that accumulated the highest aggregate episodic reward. The respective training curves for the 3 DQR model are shown in Figure 6.

For both scenarios, the model tends to reach satisfactory reward values after about 100 training episodes. For the s1-c scenario, sudden performance degradation can be observed that might have been caused by over-training (due to the smaller state space in the NTN domain). Nonetheless, the model is able to recover after \(\approx\)40 episodes to a decent performance. The disproportions across domains and scenarios are caused by different network topologies, the number of flow rerouting operations, and rerouted flow types (inter-/intra-domain, QoS class). Based on the aggregate reward, the models that performed the best in their respective environments were selected. For further evaluation, the models were tested against the \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**5 QI** & **PDB [ms]** & **PER** & **GFBR** & **MFBR** & **QF** & **Example Service** \\ \hline
1 & 100 & \(10^{-2}\) & 75 & 150 & 0.3 & Conversational voice \\
2 & 150 & \(10^{-3}\) & 2000 & 5000 & 0.5 & Conversational video \\
4 & 300 & \(10^{-6}\) & 1000 & 2000 & 0.8 & Non-conversational video (buffered streaming) \\
75 & 50 & \(10^{-2}\) & 500 & 1000 & 0.9 &
\begin{tabular}{c} A2X messages, \\ aircraft telemetry \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 2: QoS profiles used for evaluation (based on [48]). Figure 6: Episodic reward obtained by 3 DQR model in low-traffic environment: for domain agents TN, NTN and overlay \(\psi\) (**left**); total agents’ reward (**right**). increasing network load, up to \(N_{flows}=3000\), which corresponds to a highly congested network. The results for the best models are shown in Figure 7. First, 3 DQR yields a substantial decrease in flow rejection rate compared to H-SP routing for a medium network load and higher (\(N_{flows}\geq 1000\)). Starting with \(\approx\)2% (s0-c, \(N_{flows}=1000\)), the reduction reaches up to \(\approx\)13.5% (s0-c, \(N_{flows}=3000\)). This is due to the ability of consecutive intra-domain and inter-domain optimisation, which allows for offloading feeder links and better exploiting both ISLs and TN infrastructure. This allows 3 DQR to allocate the flows evenly across the available links and minimise the risk of congestion. A performance drop can be observed for lower network loads. This is caused by the number of candidate paths each router processed during the flow allocation. Increasing the number of candidate paths increases the potential length of individual network paths (both domain and overlay level), at the same time impacting the overall delay and packet error rate (leading to rejections caused by exceeding the maximum thresholds, i.e., PDB or PER). Nonetheless, the observed degradation is rather small (lower than 2%) and can be potentially mitigated by reducing the number of evaluated candidate paths. Figure 7 also outlines the performance of a combined DNN-based DDQN with H-SP routing and 3 DQR-U, where both fail to provide satisfying results. Using the foremost increases the rejection rate up to \(\approx\)10%, which indicates overall poor reasoning capabilities and difficulties in converging in the changing network conditions. Regarding 3 DQR-U, it yields comparable results to H-SP routing, but at the same time, it improves the load distribution in the network (see further). Nonetheless, 3 DQR provides much better results with a minimal increase in complexity, thus emphasising the importance of overlay level TE in the concept.