# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 1 Introduction
[PERSON]'s study [16] showed that the proposed deep RL-based scheduling is superior to other schedulers in terms of fairness and average latency. Although not in a network environment, [PERSON]'s study [17] showed excellent fairness and efficiency in various problems through hierarchical RL and decentralized RL in a multi-agent environment. In addition to these, many studies are underway that apply RL and deep learning in a network environment [18; 19]. Studies on packet scheduling with RL in the field of networks have also been proposed. In [PERSON]'s study [20], Q-learning was applied to design a scheduler that is robust to changes in data characteristics and satisfies the data latency requirements. [PERSON]'s study [21] used the deep Q-network for resource allocation for packet scheduling and satisfied the quality of service (QoS) of more packets than round-robin and priority-based schedulers. Among studies that increase fairness, some studies increase latency fairness [22; 23; 24]. These studies are intended to solve the problem of low deadline achievement rate due to physical distance by considering the number of remaining hops of the packet. For example, the hop base multi-queue (HBMQ) proposed in [22] is a scheduler that puts packets into different queues for each number of remaining hops and provides a round-robin service. As a result, the service of packets passing over a long distance is guaranteed and the deadline achievement rate is increased. However, the objective of past studies is mainly focused on the increase in latency fairness incidentally to equalize the deadline achievement rate, rather than on the increase in the fairness of E2E latency itself. Even if the deadline is met, users may experience different qualities of service between flows. In the current network, it is common to experience service loss as the distance between the user and the server increases. However, in terms of QoE, this cannot be considered fair. Therefore, latency fairness must be considered even after the flow meets the deadline. In services such as metaverse, which provide constant latency regardless of the user's physical location, the fairness of latency itself must be considered beyond the level where latency fairness satisfies the deadline. To provide maximally fair latency to all users, latency fairness should be increased based on users experiencing maximum latency. Therefore, we propose a scheduler that increases latency fairness by minimizing the maximum E2E latency. In this paper, we use an RL approach to minimize the maximum E2E latency for fairness in terms of latency experienced by packets from different paths. We note that the flow path is fixed, unlike the setting in which flow routing is taken into account described in [PERSON]'s study [8]. In addition, packet scheduling is performed by considering not only the remaining hop count but also the experienced delay up to the present. In [PERSON]'s study [8], the packet scheduler is a probability-based method calculated with weights by changing the weight calculation method, but the proposed method uses an RL approach, so a more flexible response is plausible. We proposed a method for estimating the E2E latency of a packet at a node by using two metadata: the number of remaining hops and the generation time of the packet. This allows the node to calculate the estimated delay of the packet. It was demonstrated that this estimated delay can be used as the state and reward in the RL-based scheduler to increase fairness and reduce the maximum E2E latency. By implementing the RL-based scheduler not only in a single-agent environment but also in a multi-agent environment with a node as an agent, it was shown that fairness can be increased even in a multi-hop environment using the RL-based scheduler. When the packet generation scenario is fixed, the RL-based scheduler always achieves the minimization of the maximum E2E latency. In addition, it was shown through simulations that the RL-based scheduler achieved the minimization of maximum E2E latency in various topologies.