## 2 System Model and the Optimization Problem
In this paper, after the action \(a_{t}\) has been made, the reward value at moment \(t+1\) is calculated as:  
\[r_{t+1}=1-\alpha_{1}p_{f}-\alpha_{2}p_{pp}-\alpha_{3}, \tag{9}\]  
where \(\alpha_{1}\), \(\alpha_{2}\) and \(\alpha_{3}\) are scaling coefficients. After getting the reward value, the weights of the Q-network could be updated according to the back propagation principle, where the loss value can be calculated by the following formula:  
\[L=(\mathrm{Q}(s_{t},~{}d_{t})-r_{t+1})^{2}. \tag{10}\]It should be noted that the DQN in this paper can be deployed on the terminal side in order to ensure that the difference of signal conditions of each UE is taken into account. This paper now exhibits the basic DQN algorithm. ```
Initialize Q-network;  Initialize \(\varepsilon=1\); for(\(i=1\); \(i<n_{\text{period}}\); \(i++\))do  Update standard deviation of signal strength \(sd\); if(\(i\) mod \(n_{\varepsilon}==0\))do \(\varepsilon\leftarrow\beta\varepsilon\); endif if(U(\(0,1)<\varepsilon\))do  Select an action randomly; elsedo  Predict an action with Q-network forward propagation; endif  Set A3 parameters by the selected/predicted action;  Collect HO failure rate \(p_{\textit{failure}}\) and HO pingpong rate \(p_{\textit{pingpong}}\);  Calculate reward \(r\) as in (9);  Update Q-network with loss calculated as in (10); endfor
```  
**Algorithm 1**DQN Algorithm  
Where \(n_{\text{period}}\) is the total number of training cycles, \(sd\) is the standard deviation of signal fading, which can be obtained by measuring resource blocks in the time-frequency domain, and \(\varepsilon\) is a random exploration factor. In each training cycle, based on the fading standard deviation real-time measurement, the best action is predicted by the Q-network. After corresponding this action to the A3 parameter, the handover failure rate and pingpong rate of the system are collected, and the reward value is calculated. Then the reward value is utilized to update the Q-network. In each iteration, there is a probability \(\varepsilon\) to choose a random action instead of using the Q-network prediction. The purpose of this mechanism is to ensure the initial random exploration and reduce the chance of falling into a worse local optimum. At the same time, \(\varepsilon\) gradually decays with the iterative process, which becomes \(\beta\) times that of the previous one for each \(n_{\varepsilon}\) iterations. This mechanism ensures the random decay in the initial phase, as well as stabilize the final convergence results of the Q-network. The advantages of algorithm 1 in this paper are as follows. The signal fading condition is considered, specifically, when the signal fading is in a low range, the Q-network tends  
Figure 1: The DQN algorithm. to adopt a more aggressive handover parameter strategy, or otherwise a conservative handover parameter strategy; this DQN module does not participate in the whole handover process, but just fetches the handover event statistics to give parameter suggestions, in other words, is independent from traditional mobile communication systems, which does not change the existing A3 handover process, resulting in strong backward compatibility. Algorithm 1 also has the disadvantages of most DQN methods, that is, when the action space to be predicted is large, the convergence effect would degrade; the DQN in this paper utilizes continuous value of signal fading standard deviation as input, which puts forward higher requirements to the generalization ability of the Q-network. This may take a long time to train in order to achieve better prediction performance. In the following, an LSTM-based digital twin enhancement strategy is proposed to improve the convergence efficiency and performance of the DQN.