# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 3 Results
### Simulation
This simulation implements only MARL among RL-based schedulers. Topology 4 has 9 nodes, 2 input ports, and 2 output ports, respectively, and the action space size in SARL is 2\({}^{(2\times 18)}\), which is too large to be implemented. On the other hand, MARL can be implemented because the action space size is 4 and the state space size is 16. Figure 16 shows the moving average of the maximum E2E latency of each scheduler in a network environment where random flows are generated in topology 4. Both MED-HA and MARL showed the best performance, followed by NAT-HA, RR, and FIFO. In topology 4, RR performs better than FIFO because the number of remaining hops for packets that want to go out at the same time may be different. In a fixed scenario, there is no performance difference between RR and FIFO because these cases cannot be continuously experienced. However, in an environment that creates a random flow, it is better to guarantee the service of the queue in this case in terms of maximum E2E latency. \begin{table}
\begin{tabular}{c c} \hline \hline
**Flow Number** & **Rout** \\ \hline
0 & 1-2-5-6-9 \\
1 & 4-1-2 \\
2 & 4-7-8 \\
3 & 7-8-5-6-3 \\
4 & 3-2-5-4-7 \\
5 & 6-3-2 \\
6 & 6-9-8 \\
7 & 9-8-5-4-1 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: The routes of the flow in topology 4. Figure 15: Topology 4 [34]. In topology 4, Figure 17 and Table 10 show the latency fairness index of packets per scheduler. The fairness indexes obtained (5) were represented with %. The latency fairness index was calculated for each episode, and the episode repeated a total of 10,000 times. The average latency fairness performance of each scheduler was, in order, MED-HA, MARL, NAT-HA, FIFO, and RR. The RL-based scheduler and MED-HA had about 4% higher latency fairness than other schedulers, and were good in terms of upper and lower bounds. In the case of NAT-HA, maximum E2E latency decreased, but the latency fairness index did not increase significantly compared to FIFO and RR. NAT-HA shows an average latency fairness index similar to that of FIFO and RR. NAT-HA behaves similarly to RR in topology 4. Since only two flows compete with each other for each output port in each node, if the service is based on the arrival time, there is a high probability that the service will be provided like RR. However, unlike RR, one queue can be continuously serviced, and the ingress node of a flow like node 1 often prefers to service a specific flow, so the average latency fairness index is higher than that of RR. For the same reason, FIFO also behaves like RR, and the average latency fairness index of RR and NAT-HA is similar, but FIFO obtains the advantage that NAT-HA obtains from an ingress node probabilistically, so the average latency fairness index is lower than that of NAT-HA. Figure 16: Moving average of maximum end-to-end latency per scheduler when random flows are generated in topology 4. Figure 17: Box plot of latency fairness index per scheduler from 10,000 episodes in topology 4 with the random flow. Figure 18 and Table 11 show the results of calculating the maximum latency fairness index for each episode for 10,000 episodes. As shown in Figure 18, maximum latency fairness results show good performance in the same order as the latency fairness results. MED-HA and RL-based schedulers had 2-3% higher average maximum latency fairness than other schedulers and have a higher lower bound. Through this, it was shown that the proposed RL-based scheduler and MED-HA reduced the maximum E2E latency while increasing the fairness of all latencies and the fairness of the maximum latency between flows. RR had the worst performance in terms of latency fairness index and maximum fairness index. In terms of latency fairness, the latency fairness index increases when the difference between latencies is small. However, since RR is transmitted as specified regardless of the order in which it entered the node or network, the deviation between latencies inevitably increases.