# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 2 Materials and Methods
### Scheduler Using Metadata
As mentioned earlier, the estimated delay is the minimum E2E latency that the packet can achieve. The actual E2E latency varies depending on the situation of other queues or nodes that will pass in the future, but in a situation where a packet that determines the maximum E2E latency is observed, the estimated delay is equal to the maximum E2E latency. Additionally, the estimated delay can better estimate the E2E latency at the node just before the destination. Therefore, MED-HA can perform better than FIFO or RR in terms of maximum E2E latency. Additionally, unlike [8], MED-HA does not use only the remaining hop count but also uses the packet-generated time, so it is effective to reduce the maximum E2E latency because packets have different weights even in the same remaining hop count situation. In addition, [8] used probability-based scheduling to prevent starvation, but MED-HA prevented starvation by increasing the estimated delay when packets were not sent. By the same logic, unlike NAT-HA, MED-HA considers the number of remaining hops, so that when the network arrival times of packets in a single network are the same, the MED-HA can service the packet more efficiently. MED-HA has better performance than NAT-HA, but it uses a larger amount of metadata and requires a process of calculating the estimated delay of all packets. ```
C = link capacity M = number of the input port Queue[1..M] max_estimated_delay = 0  idx = 0 bit_length = 0 while true for i in 1..M: if Queue[i] is not empty for j in 1..Queue[i].length estimated_delay = Queue[i][j].remaining_hop_count \(\times\) (Queue[i][j].length/C) + time.now - Queue[i][j].generated_time + (bit_length/C) bit_length += Queue[i][j].length if max_estimated_delay < estimated_delay idx = i max_estimated_delay = estimated_delay send(Queue[idx].head) Queue[idx].dequeue
```  
**Algorithm 3** MED-HA in the output port module. We demonstrate how the two heuristic algorithms work in the example topology. Figure 6 represents the output port module of node 0 at time 0. Since source 0 and source 1 send packets to node 0 at the same time, the network arrival time and generated time of these packets are the same. NAT-HA provides a uniformly random service among these queues because the network arrival time of each HoQ is the same. MED-HA first calculates the estimated delay of all packets. Since the current time is 0, both packets have experienced a delay of 0, and since both packets are HoQ, their order in the queue is 0. After calculating the estimated delay, the estimated delay of the packet in queue 0 is 3T, and the estimated delay of the packet in queue 1 is 2T. Since packets with the maximum estimated delay are in queue 0, MED-HA serves queue 0. Figure 7 expresses the output port module when the packet of source 0 and the packet of source 2 want to go out from node 1 at the time T. Since the network arrival time of HoQ of queue 0 is smaller than the network arrival time of HoQ of queue 1, NAT-HA serves queue 0. MED-HA calculates the estimated delay of the packets. Although the remaining hop counts of the two packets are the same, the experienced delay up to the present is different because the packet generation time is different. As a result of calculating the estimated delay, the estimated delay of the packet in queue 0 is 3T, and the estimated delay of the packet in queue 1 is 2T. Since packets with maximum estimated delay are in queue 0, MED-HA serves queue 0. In the example topology, NAT-HA can minimize the maximum E2E latency with a probability of 50% depending on which packet goes first at time 0. In contrast, MED-HA always minimizes the maximum E2E latency.