# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 1 Introduction  
Fairness is an important ingredient in the consideration of many network problems, such as packet scheduling, congestion control, and bandwidth allocation. In terms of quality of service (QoS), fairness has been considered along with QoS in order to satisfy QoS for as many flows as possible with limited resources. Since it is highly correlated with QoS in these circumstances, the maximization of fairness has been extensively studied [1, 2]. One approach recently proposed to enhance fairness employed the concept of metadata that describes the information from other data. It is used for scheduling and routing by adding metadata to packets, particularly in segment routing (SR) [3], deterministic networks (DetNet) [4], and so on. SR-based load balancers achieved better fairness while showing lower mean response time compared to random load balancers [5]. In [PERSON]'s study [6], a jitter upper bound guarantee technique using timestamp metadata for DetNet service was proposed, thereby increasing the quality of experience (QoE) fairness. It used real-time transport protocol (RTP) packets to use timestamp fields to guarantee jitter upper boundthrough timestamps and buffers. In the DetNet data plane [7], it is specified that metadata such as timestamps can be exchanged between applications, so it is possible to guarantee the upper limit of the jitter in the DetNet environment even if RTP packets are not used. Some studies have used the remaining hop counts as metadata to improve fairness and performance. [PERSON]'s study [8] conducted routing and packet scheduling through weights calculated as remaining hop counts in a Network-on-Chip (NoC) environment. In [PERSON]'s study [8], packets were divided into packets with small remaining hop counts (PSR) and packets with large remaining hop counts (PLR). Accordingly, preferentially transferring packets with small remaining hop counts (PPSR) and preferentially transferring packets with large remaining hop counts (PPLR) were proposed as routing and packet scheduling methods. PPSR is good in terms of average end-to-end (E2E) latency to transmit PSR first, but fairness is low. Conversely, PPLR has good fairness but low performance. Additionally, an adaptive remaining hop count (ARHC) flow control method for switching between PPSR and PPLR was proposed. As a result of the simulation, ARHC showed better performance than PPLR, and fairness better than PPSR and round-robin, and better performance when the network is over-saturated. Another example of using metadata is programmable packet scheduling. In the case of programmable packet scheduling, it is sometimes assumed that the packet has metadata [9; 10; 11; 12]. The packet's metadata can be in the payload or in the header. When adding metadata to the packet header, even if no fields are added to the header, metadata can be added while maintaining the existing protocol header structure by using information that can be inferred or unused fields [13; 14]. Recently, studies have been proposed using reinforcement learning (RL) for such problems. Existing optimization or control algorithms in the current network environment, where a large volume and variety of traffic is serviced, have a disadvantage in that they are difficult to implement and apply in practice due to their high computational complexity. On the other hand, reinforcement learning can be applied to various environments because it can learn and infer the network environment or parameters by itself using observed raw data. Furthermore, since it is easy to incorporate the behavior of the agent and resource observation into a reinforcement learning problem in a network environment, many studies have attempted to solve network problems using reinforcement learning. [PERSON]'s study [15] used the actor-critic RL to optimize scheduling and video QoE in a wireless environment, and showed better performance in fairness than traditional algorithms and other learning algorithms.