## 4 Enhanced DQN Based on Digital Twin
### Digital Twin Based on LSTM  
The LSTM-based digital twin proposed in this paper includes the following stages: real-world data collection, LSTM-network prediction, and unbiased estimation of handover failure rate and ping-pong rate, as shown in Figure 3.  
In the first stage, the digital twin reads data from the A3 buffer. When some candidate base stations in the A3 buffer satisfy the Formula (1) for a certain length \(\mathrm{TTT}_{\mathrm{DT},\ j}\), the RSRP of several measurement periods before this moment is extracted as the input of the LSTM network. Although the current actual handover TTT length is larger, these A3 triggers with smaller TTT are regarded as virtual handovers in digital twin. It should be noted that all potential A3 events triggered by this module are based on the same Hys value as the physical system, which can ensure better backward compatibility of the method, or otherwise additional A3 buffers might be needed. In the second stage, the prediction task of the LSTM network is to predict whether the current handover is successful and whether ping-pong handover occurs according to the RSRP series. The RSRP series is expected to reflect the user's relative signal strength trend with each base station, so machine learning can be used to extract possible future events. LSTM has the ability to process temporal series data [27]; the authors in [8] utilized LSTM to find the optimal handover timing by analyzing historical RSRP data and predicting future RSRP data. In [28; 29], LSTM was used for wireless network traffic prediction. LSTM is a special kind of recurrent neural network (RNN), mainly to alleviate the problem of gradient disappearance and gradient explosion during long sequence training [30]. Only one state exists within a single loop of an ordinary RNN, while four states exist within a single loop (also called a cell) of an LSTM. The cell state stores the state information of the current LSTM cell and passes it to the LSTM cell at the next moment. The current cell receives the cell state from the previous moment and works together with  
Figure 3: Stages of the DTe-DQN. the signal input received by the current cell to generate the cell state of the current cell. In each cell, specially designed \"gates\" are used to introduce or remove information in the cell state. As shown in Figure 4, The LSTM cell is composed of an input gate, a forget gate, an output gate and a cell state: the input gate determines how much of the input data of the network at the current moment needs to be saved to the cell state; the forget gate determines how much of the unit state at the previous moment needs to be retained to the current moment; and the output gate controls how much of the current cell state needs to be left to the current output value [28]. The LSTM network in this paper is constructed as shown in Figure 5. The network includes an input layer, an LSTM layer, a fully-connected layer (FC layer) and an output layer. At each moment, the current RSRPs from the base stations considered are fed to the network. The output layer is a softmax layer for classification task of two classes, represented as \"0/1\". The LSTM layer at h\({}_{t}\) needs to receive information from the LSTM layer at h\({}_{t-1}\) and deliver information to the LSTM layer at h\({}_{t+1}\). The LSTM network is now in place, using historical RSRPs to predict whether handover failure or ping-pong occurs. In the third stage, after establishing an LSTM network that predicts a single handover failure or handover ping-pong event, this paper proposes an unbiased estimation method for the failure rate and ping-pong rate in the digital twin. Taking handover failure as an example, if LSTM prediction is not deployed, the number of handover failures observed in actual system  
Figure 4: LSTM cell structure. Figure 5: LSTM network.