## 4 Enhanced DQN Based on Digital Twin
### Digital Twin Enhancement Mechanism
```
Initialize Q-network; Initialize \(\varepsilon=1\); for(\(i=1\); \(i\leq n_{\text{peiot}}\); \(i++\)) do  Update standard deviation of signal strength \(sd\); if(\(i\) mod \(n_{\varepsilon}==0\)) do \(\varepsilon\leftarrow\beta\varepsilon\); endif if(U(\(0,1\)) \(<\)) do  Select an action randomly; else do  Predict an action with the Q-network forward propagation; endif  Set A3 parameters \((\text{Hys}_{\text{DT},j},\text{TTT}_{i})\) by the selected/predicted action;  Update digital twin A3 parameter set \(\Omega_{\text{DT}}=\{(\text{Hys},\text{TTT})|\text{Hys}=\text{Hys}_{\text{p}}, \text{TTT}<\text{TTT}_{i},\text{TTT}\in\text{B}\}\)  Collect HO failure rate \(p_{\mathit{failure}}\) and HO pingpong rate \(p_{\mathit{pingpong}}\);  Calculate reward \(r\) as in (9);  Update Q-network as in (10); for(\(j=1\); \(j\leq\text{card}(\Omega_{\text{DT}});j++\)) do  Collect RSRP sequential values triggered by A3 parameter \(\left(\text{Hys}_{\text{DT},j},\text{TTT}_{\text{DT},j}\right)\);  Predict HO failure rate \(p_{\mathit{failure}}\) and HO pingpong rate \(p_{\mathit{pingpong}}\) via LSTM-based digital twin;  Calculate reward \(r\) as in (9);  Update Q-network with loss calculated as in (10); endfor endfor
```  
**Algorithm 2****DTe-DQN Algorithm**  
Where \(\Omega_{\text{DT}}\) is a set containing all potential trigger parameters, which includes all parameters with Hys equal to the physical system and smaller TTT values, and \(\left(\text{Hys}_{\text{DT},j},\text{TTT}_{\text{DT},j}\right)\) is the \(j\)-th element in \(\Omega_{\text{DT}}\). By predicting the virtual handover reward value from the digital twin, algorithm 2 is able to obtain more diverse reward information within a limited data collection period. The diversity of rewards is reflected that under the same signal fading condition, reward values for different handover parameters can be obtained. Diverse rewards can greatly improve  
Figure 2: Digital twin enhanced DQN. the search efficiency in the DQN training process, thereby alleviating the convergence difficulties caused by large-scale action space and continuous value input. Note that if the real-world reward of a feedback cycle were repeatedly used to update the Q-network, the performance would not improve. This is because this does not introduce more effective exploration, but repeats old experience. Instead, the repeated training might lead to a worse local optimum. It should also be noted that both the ordinary DQN algorithm and DTe-DQN algorithm have low requirements on the computational resources or efficiency of the actual system. In each iteration, statistics of several handover events are collected for the iterative process and the act of gathering itself might take a long time. Although this process could be accelerated by sharing handover information between multiple UEs, it is not possible to take less than the time it takes that most UEs perform a handover, i.e., in the order of seconds. Therefore, the efficiency of the algorithms is mainly limited by the real-time feedback rather than computational resources or power where the algorithms are deployed.