# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 3 Results
### Simulation  
The first simulation was conducted in topology 1, as shown in Figure 8. Topology 1 generates packets according to Table 6. Topology 1 is an extension of the example topology, and the packet generated from source 0 must be transmitted first to minimize the maximum E2E latency. The minimum value of the maximum E2E latency of topology 1 obtained in the same way as in the example topology is 5T. To compare the performance of the RL-based scheduler, we scheduled using FIFO, RR, NAT-HA, and MED-HA. In addition to topology 1, topology 2, topology 3, and topology 4, which will be described later, we compared the performance of these four schedulers. The maximum E2E latency and latency fairness index were used as performance indicators. Figure 9 is a graph of the learning results of the RL-based scheduler in topology 1. It can be seen that both SARL and MARL achieve minimization of the maximum E2E latency. During the first 500 episodes, the RL model does not train, so the sum of rewards does not increase or the maximum E2E latency decreases. After 500 episodes, as the RL model starts learning, the sum of rewards increases and reaches the maximum value, and the maximum E2E latency is minimized. Looking at the MARL graph, it can be seen that even if the moving average graph of the sum of rewards changes, the moving average graph of the maximum E2E latency does not change. This means that the maximum E2E latency can be minimized even if the RL model does not reach the global optimum. However, when the RL model reaches the global optimum, the maximum E2E latency is always minimized. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Time** & **Source 0** & **Source 1** & **Source 2** & **Source 3** & **Source 4** \\ \hline
0 & 1 & 1 & & \\ \hline T & & 1 & & \\ \hline
2T & & & 1 & \\ \hline
3T & & & & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Packet generation scenario in topology 1. Figure 8: Topology 1. The second simulation was performed in topology 2 as shown in Figure 10. Topology 2 generates packets according to Table 7. Topology 2 is a tree-shaped topology and an environment where E2E latency can diverge due to increased queuing delay caused by bottleneck links. In this environment, the RL model can learn from more diverse queue situations than topology 1 due to the presence of bottleneck links. Figure 11 is a graph of the training results of the RL-based scheduler in topology 2. Both SARL and MARL achieved minimization of the maximum E2E latency. In the case of SARL, it can be confirmed that the local optimum was reached and maintained during the first 1000 episodes of training. This phenomenon is caused by the stability of RL algorithms and deep learning networks. This is because the state space size and the action space size in the SARL of topology 2 are large compared to the deep learning network structure. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Time** & **Source 0** & **Source 1** & **Source 2** & **Source 3** & **Source 4** & **Source 5** \\ \hline
0 & 1 & 1 & 1 & 1 & \\ \hline T & & & & & 1 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Packet generation scenario in topology 2. Figure 10: Topology 2. Figure 9: Training result of RL model in topology 1. (a): SARL, (b): MARL. To see this clearly, a simulation of sending more packets in topology 2 was performed. In the packet generation scenario, 10 packets were randomly generated, but the maximum length of the scenario was 20T. Figure 12 shows the performance comparison between the DQN algorithm and the DDQN algorithm in this environment and the performance difference according to the deep learning network structure. Unlike previous simulations, more packets are randomly generated, resulting in more varied states. In such an environment, the DQN algorithm converges to a lower optimum than DDQN regardless of the deep learning network structure. This is because the stability of DQN is lower than that of DDQN due to the overestimation problem, which is a limitation of DQN.