# On the Design of a Network Digital Twin for the Radio Access Network in 5G and Beyond
## 3 NDT for RL Training Process  
NDT allows performing the training of RL solutions for the RAN. The training process of a RL model consists of learning the optimal policy that selects the best decision (i.e., action) for each possible situation (i.e., state), which is the one that obtains the maximum reward. To achieve this, during the training process a RL agent (i.e., the learner) iteratively interacts with an environment, where at each iteration the RL agent obtains a state, selects an action, and then, as a result of the last action, receives a reward that evaluates how good or bad the last action was for the last state, along with the new state [17]. Actions are selected according to both exploitation, where actions are selected following the RL agent's policy, and exploration, where actions are selected randomly. From this interaction, the RL agent updates the policy. This is repeated until a convergence condition is reached, when the training process is terminated. RL solutions can be applied to problems of the RAN involving some sort of decision making, mainly associated with RRM, RNM or SON functions. RL solutions for RRM functions dynamically manage the provisioned resources operating at different time scales from milliseconds to a few seconds and cover solutions for the different layers of the protocol stack. Examples of RL applicability for RRM functions are channel coding, power control and dynamic spectrum access at the physical layer, scheduling at the MAC layer, admission control for Radio Resource Control (RRC) and dual connectivity at the Packet Data Convergence Protocol (PDCP) layer. Moreover, RL can be applied to RNM and SON functions to support the network planning, deployment and operation of the RAN in terms of configuration, optimization and fault management, operating at longer-term time scales. For instance, RL applicability examples related to SON functions include capacity and coverage optimization, mobility robustness optimization, mobility load balancing or cross-slice capacity optimization, also referred to as capacity sharing, for assigning available resources to the different slices. Despite the wide applicability of RL for the RAN, the training of these solutions represents a relevant challenge from the implementation perspective. This is because the trial-and-error behaviour involved during the training of RL solutions can lead to unacceptable degradations in network performance if the training is performed directly on the real RAN. To address this problem, the training of RL can be safely performed on a RAN NDT. This can be conducted by considering the RL solution as an _App_ in Figure 1 that interacts with a RAN NDT instance during the training. Figure 2 includes the proposed functional model for the training of RL models for the RAN by using a RAN NDT. To facilitate its practical feasibility, the model is aligned with the main components and functionalities involved in the training of ML models in the O-RAN Alliance ML workflow [16]. These functionalities are coloured blue in Figure 2. As  
proposed components in this paper, the functional model includes a RAN NDT instance, which is coloured green, and different components to support the training of the RL-based models on it, coloured orange. The training of RL models is performed at the ML training host, which includes different components, namely, _model training & testing, model selection, model optimization_ and _model refine_, all of them proposed in the O-RAN Alliance ML workflow [16]. Among these ML training host components, the RL model training is carried out by the RL agent under the _model training & testing_, where the RL policy iteration process takes place through the iterative interaction of the RL agent with the RAN NDT instance that constitutes the training environment, making use of exploration and exploitation. Hence, along the training process the RL agent iteratively obtains the states and rewards from the RAN NDT instance and sends the actions to it.