# On the Design of a Network Digital Twin for the Radio Access Network in 5G and Beyond
###### Abstract
Network optimization processes can benefit from NDTs for tuning different parameter configurations and for exploring new policies to be used by certain optimization functions, such as load balancing, capacity sharing for network slicing, etc. Another application of NDTs for the RAN is the training of Machine Learning (ML) solutions [13], which can be used for dealing with multiple problems in different areas of the RAN such as physical layer processing, Medium Access Control (MAC), Radio Resource Management (RRM), Radio Network Management (RNM) or Self-Organizing Networks (SON). Among the ML techniques, Reinforcement Learning (RL) solutions are of special interest in the RAN because they are conceived for optimally solving decision-making problems. As noted by [14], they have been used in the literature for problems such as cell selection, channel selection, resource allocation (scheduling), power allocation, small cell activation/deactivation for energy efficiency, adaptive modulation and coding, etc. The use of NDT for training RL algorithms can be the key to the practical adoption of RL solutions in real RANs. This is because RL learning is achieved from the interactions with the environment based on exploitation (i.e., selecting the best actions according to the current learnt policy) and exploration (i.e., selecting some actions randomly to enhance the current learnt policy with the outcomes of new actions). This exploration behaviour becomes one of the key challenges to applying RL algorithms in real RANs, since the fundamentally online trial-and-error learning behaviour may lead to unacceptable degradations in network performance during the learning. In this context, NDT brings the opportunity to train RL algorithms for the RAN on a virtualized, updated and safe version of the real RAN. The use of NDTs for training RL solutions for the RAN has been proposed in [15], where the training of a Distributed Deep Q-Network (DDQN) agent that learns the optimal network slicing policy is performed on an NDT. Moreover, the work in [13] proposes a framework for training a RL agent online using an NDT, which is applied for a specific RL-based solution for massive MIMO configuration. While the works in [13] and [15] exemplify the use of NDT for the training of RL solutions, none of the existing works has discussed the capabilities, requirements, procedures and management tasks of an NDT for the training of RL solutions for the RAN. Furthermore, given the standardization efforts of some bodies, such as the O-RAN Alliance, on frameworks for training ML and RL solutions for the RAN, the way to integrate NDTs into these frameworks is a relevant aspect that remains a gap in the literature. This paper covers these two research gaps by proposing the integration of NDTs on current frameworks of O-RAN alliance for training RL solutions and by introducing different functionalities required for the configuration and operation of the NDT to conduct the training. Therefore, the main novelties and contributions of this paper are twofold:  
First, motivated by the need to further specify the design of an NDT for the RAN subsystem, a novel RAN NDT architecture is proposed. The proposed architecture, which considers the terminology and structure of the NDT proposed by IETF in [10], includes the different components required for building and representing the different RAN entities and the functionalities for gathering real data, managing and operating the NDT. Detailed insights on the component's requirements and operation of an NDT for the RAN subsystem are provided, as a difference from the prior works in [5; 6; 7; 8; 9], which only propose a high-level design of NDTs embracing several subsystems, or the work in [3], which basically identifies the components to be included in a RAN NDT but without providing many details on the component's requirements.