# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 3 Results
### Simulation
Additionally, the performance of each algorithm depends on the deep learning network structure. This is because the state space size and the action space size in the SARL of topology 2 are large compared to the deep learning network structure, and the states that occur due to the random flow are diverse. In SARL, the state space size and action space size are determined by the number of input ports, the number of output ports, and the number of nodes. In particular, the action space size increases in proportion to the square of the number of output ports and the number of nodes. Therefore, in a topology with a large number of nodes, it is necessary to increase the size of the deep learning network structure to learn more stably. If the state is not varied, it falls into the local optimum for a short time as shown in Figure 11. However, when the state is diversified, the stability of the deep learning network structure is directly related to the performance. In Figure 12, when the size of the deep learning network structure increases from 64 \(\times\) 64 to 256 \(\times\) 256, both DQN and DDQN algorithms find a better optimum. The third simulation was performed in topology 3 as shown in Figure 13. Topology 3 generates packets according to Table 8. Unlike the topologies discussed above, topology 3 assumes that a delay has already been experienced when generating a packet. It is assumed that packets generated by sources 0, 1, and 2 have the experienced delay of T, 7T, and 3T, respectively. Topology 3 can achieve the minimization of the maximum E2E latency by emptying the queue to immediately send out the packet, which determines the maximum E2E latency. Since topology 3 is a topology consisting of one node, the implementation of SARL and MARL is not different. Figure 11: Training result of RL model in topology 2. (**a**): SARL, (**b**): MARL. Figure 14 is a graph of the learning results of the RL-based scheduler in topology 3. RL achieves the minimization of maximum E2E latency. In Figure 14, it can be seen that the sum of rewards and the maximum E2E latency change rapidly around episode 1000. This is because topology 3 is a small topology consisting of one node, and the time slot required for simulation is also small, so the state is not varied. In such a topology, the cumulative reward and maximum E2E latency can vary significantly with one action, as shown in Figure 14.  
Figure 14: Training result of RL model in topology 3. Figure 12: Training results according to SARL’s RL algorithm and deep learning network structure when random flows are generated in topology 2. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Time** & **Source 0** & **Source 1** & **Source 2** \\ \hline
0 & 1 & & 1 \\ \hline T & & 1 & \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Packet generation scenario in topology 3. Figure 13: Topology 3. According to the previous simulation, it was confirmed that the proposed RL method works well in the fixed scenario. Unlike previous simulations, this simulation examines the performance of the RL scheduler in an environment that generates random flows. A random flow generates 10 packets in a packet generation scenario of up to 20T in length. In an environment that generates a random flow, the number of packets passing through the network increases and the simulation takes longer, so the training episodes were increased from 3000 to 10,000. The hyperparameters were also modified accordingly. Unlike the existing topologies, the topology examines the performance of the RL scheduler in a topology where a symmetric cycle exists. The topology used for the simulation is shown in Figure 15. Table 9 shows the flow path along Topology 4. Topology 4 is a topology with cycles, where the scheduler's choice can affect itself in the future. In this topology, we wanted to determine if scheduling is possible with the RL-based scheduler considering the possible future impact. Topology 4 generates random flows because it is not possible to construct a scenario such that a particular flow overlaps all other flows.