# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 3 Results  
### Simulation Setup  
The simulation environment has the same characteristics as the environment described in the example topology and operates in units of T. The simulation is designed to create a scenario where packets want to go out from the node to the same port at the same time, using 2 \(\times\) 2 nodes as in the example topology. In the 2 \(\times\) 2 node, there is one queue for each input port at the output port. Schedulers other than FIFO determine the queue to receive service from among the two queues for each T. To exclude the possibility of packet loss in the simulation, the capacity of the queue is assumed to be infinite. Since the maximum delay time increases only when packets undergo maximum queuing delay in the queue, packet loss is not considered in the simulation. The RL model was implemented using TensorFlow. TensorFlow is an E2E open source platform for building machine learning models [33]. The structure of the deep neural  
Figure 6: In the example topology, when packets want to go out simultaneously from node 0 at time 0, the service of each heuristic algorithm is displayed. Figure 7: In the example topology, when packets want to go out simultaneously from node 1 at time T, the service of each heuristic algorithm is displayed. network used in the simulation is shown in Table 4. The Dense layer is a layer that connects all inputs and outputs. Rectified linear unit (ReLu) is a type of activation function and is a nonlinear function, as shown in (7). \[\text{f}(x)=\text{max}(0,x). \tag{7}\]  
The size of the Q-network was set to the smallest size that was found to be optimal through simulations. Although there would be no problem in learning even with larger hidden layers, the size of the Q-network had to be considered within the limited resources of the nodes in the MARL environment, as it must be implemented on them. The size of the Q-network can vary depending on the environment, and for the random packet generation scenario, the size was increased to 256 \(\times\) 256 to accommodate the increased complexity of the environment. The RL buffer uses PER and has a size of 50,000. Table 5 shows the hyperparameters of the RL model. The learning rate of the RL model decreases according to the cosine decay method, and the cosine decay follows (8). The d and \(\alpha_{0}\) are hyperparameters that determine the period during which the learning rate decays. \[\text{Learning rate}=\alpha_{0}\times 0.5\times\bigg{(}1+\text{cos} \bigg{(}\pi\times\frac{\text{min}(\text{number of episodes},\ \ d)}{d}\bigg{)}\bigg{)}, \tag{8}\]  
The reason for reducing the learning rate is to keep the global optimum from deviating due to the high learning rate. The episode starts at time 0 and ends when all packets have reached their destination or time 100T. The reason for ending an episode at a specific point in time is that the proposed reinforcement learning scheduler may select an empty queue, preventing packets from being sent out. The end time of 100T was chosen, taking into consideration the topology, number of generated packets, and their timing. A total of 3000 episodes were repeated per simulation. During the first 500 episodes of the simulation, the RL model does not train and stores samples in a buffer. After that, the RL model is trained for the remaining 2500 episodes. Among the hyperparameters, epsilon decay and learning rate decay period were determined considering the number of learning episodes. Other hyperparameters were determined empirically while conducting simulations. The simulation was repeated 15 times for each topology by dividing it into SARL and MARL, and the results were obtained. \begin{table}
\begin{tabular}{c c} \hline \hline
**Type** & **Size** \\ \hline Input & The state size \\ Dense & 64 \\ ReLu & 64 \\ Dense & 64 \\ ReLu & 64 \\ Dense(Output) & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Deep learning network architecture.