# Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler
## 2 Materials and Methods
### SARL and MARL  
RL models perceive the state information from the environment and select the action based on that. Through actions, agents obtain rewards and change states. By repeating this, the RL model learned an optimal policy, a series of actions that maximizes the expected amount of reward. To solve the problem of minimizing the maximum E2E latency of the network using RL, the state, action, and reward are defined as follows. * State  
The state uses information that can be obtained from the queue for each output port. For each queue, the remaining hop count of the first packet, the delay to the present of the first packet, the queue length, and the maximum estimated delay of the packets in the queue are obtained. The state is information from which the maximum E2E latency can be inferred. The number of remaining hops and the delay to the presence of the first packet are information for calculating the estimated delay of the first packet. The first packet is important because the estimated delay and the actual E2E latency are almost similar. The queue length can infer the degree to which it affects other queues. The maximum estimated delay of packets in the queue is the closest value to the target maximum E2E latency. In MARL, the state space size is 4 \(\times\) M \(\times\) N. In SARL, the state space size is 4 \(\times\) M \(\times\) N \(\times\) H. In the example topology, node 1 has 2 input ports and 2 output ports, so the state space size in MARL is 16. However, if only one of the output ports wants to send packets out at the same time, the other output ports may use FIFO. Therefore, the state space size can be reduced to 8 by using only the information of the queue of one output port. SARL can likewise reduce the state space size from 32 to 16. In a fixed topology and scenario, the state space size can be reduced in this way to reduce the computational cost of the RL model, but in a non-stationary environment, all information must be used, so the state space size cannot be reduced. * Action  
The action is to select a queue to service per output port. The action is performed when both transmission and reception are completed. Since the number of queues per output port is equal to the number of input ports, the action space size in MARL is N \(\times\) M. In SARL, the action space size is N \(\times\) M \(\times\) H. The action space size can be reduced just like the state space size. In the example topology, with MARL, the action space size of node 1 is 4, but the valid number of the output port is 1, so the action space size can be reduced to 2. Similarly, in SARL, the action space size can be reduced from 8 to 4. * Reward  
The reward is computed using Algorithm 1. The agent receives the reward when all transmissions and receptions of the action are completed once the action is performed for each output port. When a packet is transmitted, it is compensated using the estimated delay of the transmitted packet and the maximum estimated delay of the packet in the queue after all transmission and reception are completed. The key to Algorithm 1 is that the larger the estimated delay of the transmitted packet is and the smaller the maximum estimated delay of the remaining packets is in the queue, the greater the compensation is. If a packet is not sent even though there are packets in the queue to send, a penalty is given with the amount of the square of the maximum expected latency of the packets in the queue. In SARL, all rewards of nodes are summed up and used as the actual reward, so the penalty obtained by not sending from one node should be able to affect the entire network. Assuming that the estimated delay of one packet is n, the maximum reward that can be given from this packet in SARL is n\({}^{2}\). For this reason, it is beneficial to operate by work conserving in all nodes by receiving a penalty of n\({}^{2}\) when the transmission is not performed, so the penalty is calculated as a square. The rewards obtained for each output port in MARL are added up and used as the actual reward.