# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 2 Materials and Methods
### Problem Statement
As in the example topology, in the fixed packet generation scenario, since the maximum E2E latency for each scheduler is within a fixed range, performance can be compared only with the maximum E2E latency. However, in random packet generation scenarios, different performance evaluation indicators are needed because the maximum observable E2E latency is different for each scenario. For quantitative performance evaluation, latency fairness and maximum latency fairness are evaluated using [PERSON]'s fairness index [32]. The fairness index is obtained using (5). \[\text{Fairness index}=\frac{\left(\sum_{i=1}^{n}x_{i}\right)^{2}}{n\sum_{i=1}^{ n}x_{i}^{2}}, \tag{5}\]  
In (5), \(x_{i}\) is the E2E latency of packet \(i\) and n is the number of packets. For maximum latency fairness, \(x_{i}\) is the maximum E2E latency of packets within flow \(i\) and n is the number of flows. The value of the fairness index is between 0 and 1. If the maximum E2E latency over all the packets is minimized, then the E2E latencies of other packets inevitably increase, so latency fairness increases. Therefore, the quantitative evaluation of performance is possible through the latency fairness index. When using SARL in the example topology, the states that the agent can observe are the states of node 0 and node 1. In SARL, there is a virtual controller connected to these nodes, receiving state information from the nodes, and based on this, the controller determines the operation of the node's scheduler. The controller notifies each node of the decided action, and the node operates the scheduler accordingly. In the simulation, the delay due to the exchange of information between the node and the controller is not taken into account. When using MARL in the example topology, the agent implements an agent on node 0 and node 1. The MARL method uses the IQL. IQL generally has poor performance, but in the assumed network topology, all agents are identical and operate sequentially, so IQL can show sufficient performance because there is little influence on each other. Each agent can only observe the state of its node. In MARL, the node and the agent are the same, so the scheduler can be operated through the RL model without exchanging information between the agent and the node. In MARL, agents cannot directly observe E2E latency. Therefore, the agent minimizes the maximum value of the estimated delay that can be calculated by the node without minimizing the maximum E2E latency. Let the \(i\)th packet in the queue be called \(p_{i}\), \(i\geq 1\). If the number of remaining hops of \(p_{i}\) is \(p_{i}^{h}\), the generated time of \(p_{i}\) is \(p_{i}^{g}\), and the length of \(p_{i}\) is \(p_{i}^{l}\), the estimated delay of \(p_{i}\) can be obtained using (6). \[p_{i}^{\prime}\text{s estimated delay}=\left(p_{i}^{h}-1\right)\times\left( \frac{p_{i}^{l}}{C}\right)+\text{current time}-p_{i}^{g}+\sum_{k=1}^{i}p_{k}^{l}/C, \tag{6}\]  
In order not to calculate the transmission delay redundantly, the remaining number of hops is used by subtracting 1. The estimated delay is the expected minimum E2E latency of \(p_{i}\), where there is only one queue. To calculate (6) in the node, the packet must contain information about the generated time and the number of remaining hops as metadata. Both metadata have been used in previous studies [8; 16; 22], but either one of them was used or it was calculated and used at the node without being recorded in the packet. As shown in Equation (6), the estimated delay can be reflected by matching the units of the two metadata,so it is not governed by specific metadata. Since the estimated delay is consequently the same as the E2E latency, if the maximum value of the estimated delay is minimized, the maximum E2E latency is minimized. SARL can achieve E2E latency by adding information exchange between the controller and the destination node so that the agent can obtain the state information of the destination node.