# On the Design of a Network Digital Twin for the Radio Access Network in 5G and Beyond
## 3 NDT for RL Training Process
Then, according to the states, actions and rewards, the policy is updated. The specific algorithm for updating the policy depends on the considered RL technique (e.g., for Q-learning and DQN techniques the corresponding algorithms can be found in [17; 18], respectively). Note that the states and rewards will result from the sent action, but also from the emulation dynamics and components' configurations in the RAN NDT instance. During the training, different KPIs are monitored to assess the model convergence and determine whether the training process is completed. Afterwards, the model testing takes place, consisting of validating the behaviour of the trained model. This is performed by selecting the actions according to the trained policies under exploitation mode and by triggering them on the RAN NDT instance. During this process, the performance of the trained policies can be assessed through different RAN-related KPIs (e.g., throughput, error rates, etc.). Once the training and validation are completed, the RL model (i.e., the trained policy) can be packaged up, e.g., as a container, and be deployed on the real RAN environment, where the RL model actions are executed during a stage known as _inference_. Within the ML training host, the _model selection_ function selects the configurations for the inference and the training of the RL model based on the requirements of the RL solution e.g., training accuracy, training time, hardware resources in training and inference, inference speed, etc. To control and configure the training environment, the _environment specification_ function is proposed to be included in the _model selection_ functionality. Specifically, this function configures the RAN NDT instance by specifying the models of the RAN NDT instance that are required for the training of the RL model. To this end, the _environment specification_ interacts with the _NDT management_ module in Figure 1 to configure the required models in the _model activation & configuration_ functionality. For example, this might include the activation and configuration of the gNB and scenario topology models in Figure 1 of the RAN NDT instance according to the gNBs configurations and topology in the real RAN, respectively. Moreover, to ensure high performance of the RL model training, the RL agent needs to experience, during the training, those relevant situations (i.e., states) that will be experienced later on during inference. The _environment specification_ function  
Figure 2: Functional model of the RL model training process on NDT. The training is performed on the ML training host that includes different functionalities identified in the O-RAN ML workflow. is also responsible for the configuration of the RAN NDT instance parameters to emulate these situations in the _emulating running control_ functionality of the _NDT management_. For instance, different traffic levels or load distributions among the different gNBs can be set up for emulation. As a RAN NDT instance can be fed by data from the real RAN environment to be used by the _service mapping models_ shown in Figure 1, another responsibility of the _environment specification_ function is to specify the data needed by the RAN NDT instance. Based on this configuration, data are gathered from the real RAN environment and stored in the _data repository_ of the RAN NDT instance. Regarding the _model optimization_ component in the ML training host, it allows the optimization of the hyperparameters of the RL model (e.g., number of layers of a neural network in case of deep RL, number of neurons per layer, etc.) based on certain hardware or performance metrics requirements such as model accuracy, model size, inference speed, memory used, etc. Then, the RAN NDT instance can be used to test the performance of different policies associated with different hyperparameter configurations. Finally, the _model refine_ component allows, if required, the upgrade of the model through re-training after it has already been deployed in the real RAN.