## 4 Enhanced DQN Based on Digital Twin  
### Digital Twin Enhancement Mechanism  
Digital twin is a virtual representation of the physical world, which continuously interacts with the physical world and predicts and supplements various behaviors of the physical world [24]. The application of digital twins is often due to the limitations of observations in the physical world, and thus additional virtual modeling is required to supplement predictions. Through digital twin, additional information is obtained to improve the performance of physical systems. Deep learning is an important driver of digital twin, because deep learning itself is a data-driven model, which can obtain expected patterns based on real-world data feedback. As mentioned in [25], 5G networks and related applications often utilize rich data in different formats, based on which, deep learning can be used to create models of 5G traffic/network behavior with the knowledge of historical data as well as real-time Internet of Things (IoT) data from multiple sources to help detect anomalies and predict any potential bottlenecks in traffic performance. Deep learning-based digital twins can also significantly improve network performance in multiple domains and layers, including optimization of physical layer operations, power control, modulation and coding schemes, waveform selection, and mobility management based on signal-to-interference-plus-noise ratio prediction, caching in mobile edge computing, network slicing, etc. [26]. This paper also utilizes an Artificial Intelligence (AI) method to create a copy of the real world to help predict real-world handover events. In general, the real world is only able to provide limited feedback, which in this paper's context, is the real-time rewards for the selected handover parameters. However, some data for other handover parameters, such as RSRP, could also be collected at the same time. Digital twin is proposed to utilize these data to predict rewards. Specifically, after a communication system selects a set of A3 handover parameters, the handover behavior must satisfy the TTT window length. In other words, when a handover is triggered, the A3 buffer has retained the RSRP value of TTT consecutive measurement periods, which also includes potential A3 events that can be triggered with a lower TTT value, as shown in Figure 2. Furthermore, even if the A3 event corresponding to the current TTT value is not triggered, the A3 events corresponding to a lower TTT value may also exist. However, since the actual system can only perform handover according to one set of parameters, only the reward value corresponding to one set of parameters can be obtained, which is a waste of potential A3 event triggers. This proposed digital twin is also considered to be deployed on the terminal side. This premise ensures that the approach in this paper is UE-specific, that is, data collection and processing, virtual event prediction as well as handover parameter decision are completed at the terminal side. Through the digital twin based on the LSTM network, the potential A3 events could also be taken advantage of. Specifically, after the potential A3 event is triggered, the RSRPs of each base station at several historical-measurement periods are fed to the digital twin to obtain the handover failure rate and the ping-pong rate prediction for the corresponding handover parameter. Then, its reward value is calculated and fed back to the Q-network. The digital twin is able to take advantage of the RSRP values that would have been difficult to utilize and enrich rewards. The specific construction method of the digital twin is explained in Section 4.2. The algorithm of Digital twin enhanced DQN (DTe-DQN) is as follows.