## 3 ML Application in O-RAN
### Resource Management  
In [44], the authors compare the performance of on-policy and off-policy DRL methods. The former is based on Proximal Policy Optimization (PPO) and the latter on a Sample Efficient Actor-Critic with Experience Replay (ACER). This process was conducted under an O-RAN setup. The O-RAN architecture is a suitable technology for DRL implementation, since it includes mechanisms that enable AI for more efficient network management and orchestration. In particular, both Near-RT and Non-RT RICs are designed for hosting AI workflows, namely DRL models. Since the objective is to optimize the resource allocation of a real-time surveillance video application, the types of services were classified as latency-sensitive and latency-tolerant. In this direction, two slices are established and managed by an O-RAN cross-slice resource orchestrator hosted by the SMO. One slice serves video surveillance cameras to transmit real-time videos via O-RAN to the 5G Core Network (CN) and then to a Control Center (CC) for real-time monitoring, and a slice serving the latency-tolerant users. The performance of the DRL on-policy mechanism was shown to provide better overall results namely in terms of implementation simplicity, performance stability, good trade-off between users latency and energy consumption, and faster convergence. In [45], the authors consider the problem in which the eMBB and URLLC services compete for limited and insufficient computing resources, and the operator must balance the allocation of these resources to users of both services in multiple O-RUs/shared O-Cloud while maximizing fairness. The problem is initially modeled as an Integer Linear Programming (ILP) problem. However, given the high complexity of solving the NP-Hard ILP problem, a policy gradient-based RL algorithm aiming to solve a Markov Decision Process (MDP) is used instead. The latter is expected to perform similarly to the ILP solver, and both approaches are compared. Simulation results showed that the RL agent performed close to the optimal results of the ILP solver not deviating from the ILP by more than 6%, while being fairer at the same time. The authors of [46] proposed an RL-based framework to manage traffic flows while taking advantage of the O-RAN ecosystem. The framework receives periodic reports from the O-RAN DU about the network status and dynamically adapts the per-flow resource allocation for which each traffic flow can compete, and identifies the corresponding modulation and coding scheme (MCS) that best fits the traffic flow KPIs and the channel quality. The RL-based dynamic resource controller solution that leverages a policy differential semi-gradient State-Action-Reward-State-Action (SARSA) targets the minimization of the maximum difference between desired and actual throughput, across all active traffic flows. The framework was integrated into an O-RAN platform, and is deployed as an xApp in the Near-RT RIC. The deployed framework is very flexible, and can adapt its architecture based on the number of traffic flows. Additionally, it is possible to create multiple policy instances, each independently and sequentially serving a subset of users, improving the framework's scalability. The research conducted in [47] employs RL for adaptive resource allocation, demonstrating its utility in the context of Non-RT RIC. The ML agent deployed in the Non-RT RIC acquires knowledge and learns a radio resource allocation policy capable of adapting to dynamic environments, while simultaneously meeting diverse energy-driven criteria. Within the learning context, key information such as the mean and variance of the channel quality indicator (CQI) from the previous period and the bit number of new data are aggregated at the onset of each period. Subsequently, three transmit parameters, including transmit power, the highest MCS, and the maximum transmission airtime, are selected and transmitted to the Near-RT RIC so that the transmit parameters can be applied to BSs.