## 3 Related Studies and Motivation
### Reinforcement-Learning-Based Network Routing  
Reinforcement learning is a method of learning how to behave in each environment, and proceeds by receiving feedback as positive or negative rewards for the actions conducted by the agent under various situations [31]. In [32], the authors proposed a QAR algorithm for achieving time-efficient and adaptive QoS provisioning packet delivery through reinforcement learning and QoS-aware compensation capabilities. Furthermore, the simulation results demonstrate that QAR outperforms existing learning solutions and provides fast convergence with QoS provisioning to facilitate practical implementation in large software-service-defined networks. In [33], the authors proposed RL4 Net, which is a type of deep deterministic policy gradient-based Q-learning. The state is encapsulated into the amount of traffic flowing between each pair of routers (total size), which corresponds to a weight update determining how the router selects the interface to forward, and the reward is calculated based on the delay. In addition, in [34], the authors proposed applying Q-learning directly to packet routing, and proposed an efficient routing policy in dynamically changing networks without needing to know the network topology or traffic patterns in advance. In [35], the authors proposed CQ-routing, through which the learned routing policy under a high load shows more than twice the performance of Q-routing in terms of the average packet delivery time. It has also been indicated that CQ-routing can sustain higher load levels than both Q-routing and shortest path routing. Moreover, in [36], the authors proposed NNQ-routing using a neural network approximation to improve the scalability of Q-routing. The Q function was replaced by a three-layer-conceptron model, and the successful packet delivery of Q-learning adopted positive compensation, delay,and loss packets as negative compensation. In [37], the authors proposed two adaptive reinforcement-learning-based spectral recognition routing protocols, with Q-learning and dual-reinforcement learning, respectively. Spectrum-aware DRQ routing learns the optimal routing policies 1.5-times faster than spectrum-aware Q-routing at low and middle network loads. Under high network loads, the routing policies learned are seven times better than those of spectrum-aware Q-routing. Moreover, in [38], the authors proposed critical flow rerouting-reinforcement learning (CFR-RL), a reinforcement learning-based scheme that learns policies to automatically select critical flows for traffic matrices. The state uses a traffic matrix for time t, and the action generates multiple actions for time step t at each state. A reward is used to reroute the critical flows to balance the link utilization and set up the flows to reflect the network performance. In [39], the authors also proposed positive and negative compensation values using network throughput and delay to optimize the reward for network throughput. After proper learning, the agent proposes dueling double-deep Q-learning-based RL-routing, which predicts the future behavior of the underlying network and suggests better routing paths between switches. Moreover, in [40], the researchers improved the problem of a poor response ability of neural networks when generating multicast routing trees by always learning the DQN for the same environment. In other words, we propose a multicast routing tree methodology that learns and responds to a new environment each time it learns. Related studies on reinforcement-learning-based network routing methodologies yield routing paths by learning in the direction in which the routing nodes become agents and reduce the latency based on Q-learning without specific models. In particular, deep reinforcement learning is efficient because agents yield optimal routing paths through the learning of neural network models based on data collected from the network.