# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 4 Conclusions  
In this paper, we proposed an RL-based scheduler that minimizes the maximum E2E latency and evaluated its performance with other schedulers through simulations. The RL model was implemented based on DDQN with PER structure using TensorFlow. In addition, for performance comparison, two heuristic algorithms that can minimize E2E latency, NAT-HA and MED-HA, were proposed. In the network topology, the node assumes that the output port module has as many queues as the number of input ports, and that the link always takes the transmission delay of T for every packet. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Maximum Latency Fairness Index [\%]**} \\ \cline{2-7}  & **FIFO** & **RR** & **NAT-HA** & **MED-HA** & **SARL** & **MARL** \\ \hline Mean & 99.22 & 94.99 & 99.68 & 99.7 & 99.77 & 99.74 \\ Min & 96.38 & 91.33 & 96.49 & 96.5 & 97.51 & 96.76 \\ Max & 99.93 & 98.54 & 99.98 & 99.99 & 99.97 & 99.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Maximum latency fairness in topology 2 with the random flow. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Topology**} & \multicolumn{6}{c}{**Average Maximum E2E Latency**} \\ \cline{2-7}  & **FIFO** & **RR** & **NAT-HA** & **MED-HA** & **SARL** & **MARL** \\ \hline
1 & 5.9375T & 5.9375T & 5.5T & 5T & 5T & 5T \\
2 & 9T & 9T & 8T & 8T & 8T & 8T \\
3 & 9T & 9T & 9T & 9T & 8T & 8T \\
2 with & 43.711T & 44.13T & 43.281T & 43.27T & 43.279T & 43.274T \\ random flow & 11.134T & 10.481T & 10.173T & 9.433T & & 9.423T \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Comparison of average maximum E2E latency by the scheduler. Figure 21: Box plot of maximum latency fairness index per scheduler from 10,000 episodes in topology 2 with the random flow. The RL agent is implemented in two ways: SARL and MARL. In SARL, the controller receives information and issues commands through message exchange with nodes as agents. The simulation did not take the delay into account due to these message exchanges. In MARL, the agent for each node makes an independent decision based on the information in the node, so there is no delay due to message exchange. In an RL environment, the states use information from the queues. MARL uses the queue information of all the output ports of a single node, whereas SARL uses the queue information of all the output ports of all the nodes as the state. In the RL environment, the agent selects a queue to serve each output port as an action. SARL selects a queue to be serviced by the output port for each node observed by the controller, which is the agent. In the RL environment, the reward was obtained through the estimated delay of transmitted and remaining packets. Estimated latency refers to the minimum E2E latency that a packet can currently achieve. Simulations were performed in four topologies, and in topologies 2 and 4, random flows were generated and simulations were performed. FIFO, RR, NAT-HA, and MED-HA simulations were additionally performed for performance comparison with the RL-based scheduler. Simulation results showed that FIFO and RR did not achieve maximum E2E latency minimization in all topologies. NAT-HA achieved the minimization of the maximum E2E latency only in topology 2, and MED-HA achieved the minimization of the maximum E2E latency in topologies 1 and 3. The RL-based scheduler achieved the minimization of maximum E2E latency in both SARL and MARL in all topologies, which are fixed scenarios. In scenarios with random flow generation, the RL-based scheduler showed the lowest average maximum E2E latency for all topologies, the same for MED-HA. Depending on the topology, the average maximum E2E latency of NAT-HA was equal to or higher than that of the RL-based scheduler. In terms of fairness, the RL-based scheduler showed a 4% and 0.5% higher average latency fairness index than FIFO and 4% and 2% higher than RR, respectively, by topology.