# Deep Reinforcement Learning for Resource Management on Network Slicing: A Survey
## 5 Resource Orchestration
### Orchestration Based on DRL  
The following paragraphs review recent DRL investigations to perform resource orchestration in NSL or related technologies. Ref. [80] proposed a framework based on DQN, policy gradient, and actor-critic to orchestrate service function chains dynamically. The framework determines the placement of VNFs (cloud or edge) that form the service chains and the paths to connect them while guaranteeing real-time traffic requests. The DQN agent considered the space of states as the traffic flow rate of services and the status of VNFs. The framework defined the space of actions regarding the number of VNFs to activate and traffic flow to schedule. The performance evaluation included heterogeneous NFV/MEC-enabled IoT network scenarios emulated with the networkX tool. The authors used the scenarios to create a synthetic dataset helpful to train the framework's algorithms in the TensorFlow framework. Ref. [81] introduced a DQN-based cross-slice resource orchestration approach to improve performance and maximize the expected long-term revenue in RAN slicing where multiple tenants compete for channel resources. The DQN agent used two DNNs with 2 hidden layers of 16 neurons; the Tanh function [163] was employed to activate the output layer. The approach defined the space of states as a tuple comprising data about mobile users (status and location) and arriving packets during a scheduling slot. The space of actions was modeled as the wireless radio resources to allocate to each tenant. The approach proposed was emulated using TensorFlow and trained with 5000 episodes. The environment comprised a physical RAN with 4 base stations covering 400 locations each. The requests raised by the mobile users followed a Poisson distribution. Ref. [82] introduced a framework that scales and places service chains (composed of VNFs) seeking to achieve lower system costs in 5G networks. The framework used Online DRL extended with an actor-critic method that employed two LSTMs activated by ReLU. The online DRL agent operated with a space of states modeled considering data about the VNFs deployed in previous and current times and upcoming flows from the traffic model. The DRL agent defined the space of actions as the placement of service chains to serve all flows in a time \(t\). The authors evaluated their framework using real-world Web traffic obtained from Huawei Inc., Hong Kong, China. [83]. Eight Google locations comprised the data center network used in the actual evaluation. [PERSON]. [84] proposed a multiobjective approach based on TD3 (i.e., is a model-free, online, off-policy DRL method [164]) to orchestrate the computing resources in small cells connected to a central unit while minimizing  
Figure 5: Resource orchestration architecture using RL/DRL. the latency, energy consumption, and VNF instantiation costs (maximize profit). The TD3 agent used two DNNs activated by ReLu and modeled the space of states as the number of requested services per slice, the computing resources allocated to each VNF in the service chains, and the number of instantiated VNFs. According to traffic fluctuation, the approach defined the actions to increase or decrease the computing resources assigned to each VNF. The approach proposed used six deep neural networks implemented in PyTorch. The performance evaluation was realized in openAI gym considering two-tenant scenarios with different latency and CPU constraints requirements. The UE packet arrival followed a Poisson distribution. [PERSON]. [85] introduced a resources orchestration approach based on a constraint-aware online DRL algorithm devised to optimize resource utilization while meeting SLAs in end-to-end network slices. The DRL agent used two DNNs activated by Leaky ReLU [165]. The approach modeled the space of states considering the average traffic of slices, the number of users waiting in the queue, and the slice performance in the last time slot.