# Deep Reinforcement Learning for Resource Management on Network Slicing: A Survey
## 6 Resource Scheduling
### Scheduling Based on RL and DRL  
The following paragraphs review latter investigations using RL and DRL to perform resource scheduling in NSL or related technologies. Ref. [89] presented an intelligent resource scheduling approach based on Asynchronous Advantage Actor-Critic (A3C) [172] (i.e., one of the most recent and powerful DRL algorithms) for improving resource utilization while guaranteeing isolation between slices in 5G-RAN. The approach operated with slices created on a substrate mobile network based on SDN and NFV. The A3C agent operated with a space of states represented as the set of users per slice and the assigned spectrum resources. The approach used a Gaussian probability distribution function to derive a stochastic policy for selecting actions modeled as resources programmed to each slice. The A3C was simulated in TensorFlow and trained with synthetic data traffic generated from Gaussian distribution. Refs. [90,91] presented a scheduling approach based on Q-learning to program the execution of VNFs composing a service function chain while minimizing delay. The approach considered delay as the difference between the end execution time of the first VNF and the end execution time of the last VNF belonging to the chain. The Q-learning agent employed a space of states modeled from the state of network  
Figure 6: Resource scheduling architecture using RL/DRL. function virtual infrastructure. The agent's actions defined the VNF chosen for execution in a time \(t\). The evaluation of the proposed approach considered a system composed of four NFV nodes and five network services delay-sensitive with different setting parameters. The packets of service arrived at the NFV nodes following a Poisson distribution. Ref. [92] introduced a packet scheduler framework based on various RL algorithms and intended to minimize packet delay and packet drop rates in RAN slices sharing radio resources at each transmission time interval. The scheduler used QV-learning, QV2-learning, QVMAX-learning, and QVMAX2-learning [173; 174] to achieve the optimal action-value function. The algorithms considered a space of states based on the quality indicator of channels, active users at each transmission time interval, arrival rates in data queues, and performance demands of network services. The framework modeled the state of actions as the number of resource blocks to allocate per transmission interval. The authors implemented the proposed framework in the LTE-Sim simulator with the RRM-Schedules C/C++ tool. For the evaluation, they used constant bit rate and variable bit rate to model the traffic of specific applications such as video, VoIP, FTP, and Web browsing. Furthermore, they generated constant traffic in random periods and variable traffic from a Pareto distribution. Ref. [94] proposed a Q-learning-based resource scheduling approach to achieve VNF chaining that is adaptive and cost-effective in 5G optical networks. The approach modeled the space of states regarding vCPUs used by each server node. The space of actions was represented as the physical server node selected for deploying VNFs in a time \(t\). Ref. [93] introduced an end-to-end NSL resource scheduling scheme based on DQN and intended to minimize the SLA violations of slices (guarantee performance and service reliability) and maximize resource utilization. The DQN agent learned to dynamically manage the resources of 5G network slices depending on the perceived demands by using a convolutional neural network (CNN) [175; 176] composed of four convolutional layers; the output layer was activated by ReLU and Softmax functions. The scheme modeled the space of states regarding the number of allocated resources and the percentage of usage of all available resources scheduled for the slices. The space of actions was defined as the percentage in which the resources of each slice must be increased or decreased in a time \(t\). Ref.