# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 3 Results
### Simulation
In terms of maximum latency fairness, RR can have a large  
In order to examine the performance of SARL in an environment where random flows are generated, additional simulations were performed in topology 2. Since this simulation is also an environment that generates a random flow, the number of training episodes was increased to 10,000, and the hyperparameters were modified accordingly. Because the state is diversified due to the random flow, we changed some of the deep learning network structure and parameters so that the RL-based scheduler could learn well. SARL changed the deep learning network structure to (state space size \(\times\) 256 \(\times\) 256 \(\times\) 256 \(\times\) action space size) while maintaining the layer type. The MARL environment maintains the existing deep learning network structure because the state does not vary much due to random flow. Figure 19 shows the moving average of the maximum E2E latency of each scheduler in a network environment where random flows are generated in Topology 2. The proposed RL-based scheduler outperformed FIFO and RR and showed similar performance to MED-HA and NAT-HA. In topology 2, even in the fixed scenario, the RL-based scheduler showed the same performance as MED-HA and NAT-HA, so it can be said that SARL and MARL reached the global optimum. In topology 2, unlike topology 4, RR performed worse than FIFO. The reason is that, contrary to topology 4, the remaining hop count of packets is always the same when they want to go out at the same time. In such an environment, it is not good to guarantee the service of queues like RR, as the latency experienced by the network determines the maximum E2E latency. Figure 20 and Table 12 show the latency fairness index of each scheduler in % units. The latency fairness index was obtained for 10,000 episodes for each episode. In terms of average latency fairness, NAT-HA was the best, followed by SARL, MED-HA, MARL, FIFO, and RR in order of performance. As with the maximum E2E latency results, MED-HA and RL-based schedulers had similar latency fairness indices. However, NAT-HA produced better results than MED-HA and the RL-based scheduler because topology 2 is an environment in which E2E latency diverges and packets accumulate in queues. In topology 2, only packets with the same number of remaining hops content, so there is no difference in the maximum E2E latency of NAT-HA or other proposed schedulers. If each queue has a packet with the maximum estimated delay, MED-HA or an RL-based scheduler serves random queues. However, at this time, the estimated delay of HoQ may be different. NAT-HA is advantageous in terms of latency fairness because it determines the order of packet service considering only HoQ. Compared to the proposed schedulers, FIFO and RR have an average latency fairness index of 0.5% and 2% smaller, respectively. The reason why the difference with FIFO is small is that there is no large performance difference from the scheduler proposed by the node connected to the bottleneck link for the same reason as NAT-HA. However, the average latency fairness index is smaller than that of the proposed scheduler due to performance differences in other nodes. Figure 19: Moving average of maximum E2E latency per scheduler when random flows are generated in topology 2. Figure 21 and Table 13 are the results of calculating the maximum latency fairness index for each episode for 10,000 episodes. Looking at the maximum latency fairness index in Figure 21, unlike the latency fairness index, there is no difference in average maximum latency fairness between the proposed algorithms. This is the same reason why there is no difference in the maximum E2E latency described above. FIFO, like latency fairness, has an average maximum latency fairness index that is about 0.5% lower. RR is about 4.5% lower than the average maximum latency fairness index. This is because the bottleneck link in topology 2 exists right before the destination node, so RR increases the maximum E2E latency.