## 2 System Model and the Optimization Problem
So the effective handover rate is defined as:  
\[p_{valid}=1-p_{f}-p_{pp}, \tag{4}\]  
and the optimization problem in this paper is maximizing the effective handover rate through searching handover parameters, expressed as:  
\[\begin{array}{c}\underset{w\in\Omega_{\text{HO}}}{\text{max}}p_{\mathit{eff },\ w}\\ s.t.\quad\quad\text{Hys}>0\\ \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad  
likely to occur. At this time, lower Hys and TTT parameters, that is, more aggressive handover parameters, can be used to ensure a higher success rate. When the signal fading condition is severe, the signal strength fluctuates greatly, and thus, ping-pong handover is more likely to occur. It is necessary to adopt a more conservative strategy (larger Hys, TTT parameters) to reduce the ping-pong rate. Although a higher effective handover rate may be obtained, the handover failure rate might also increase [21]. Therefore, it is necessary to select different handover parameters for different fading conditions to ensure the robustness of the system [22].  
Assuming that when the fading condition changes, which is described as fluctuation in both time and space domain in [20], the handover failure rate and ping-pong rate of the system also change accordingly, and the optimization problem presented in formula (5) is also to be transformed into the following optimization problem in order to get the optimal decision strategy \(\hat{V}\):  
\[\hat{V}=\underset{V}{\mathrm{argmax}}\int_{sd_{1}}^{sd_{2}}p_{valid,~{}V(sd)} \mathrm{d}sd, \tag{6}\]  
where \(sd\in[sd_{1},~{}sd_{2}]\) is the fading standard deviation, \(V(sd)\) denotes the handover parameter decision function based on \(sd\):  
\[V:[sd_{1},~{}sd_{2}]\rightarrow\Omega_{\mathrm{HO}}, \tag{7}\]  
Since the fading standard deviation is a continuous value, it is difficult for training samples to traverse all situations, so it is necessary to find a suitable decision modeling method. Compared with methods such as interpolation and polynomial fitting, the Q-learning method based on deep neural network is expected to have better results [14]. Although there is some research on the DQN method for handover parameter optimization, it has the following disadvantages: either the parameter selection is not optimized for signal fading conditions, or there are too many protocol procedures that need to be modified, resulting in poor backward compatibility. The purpose of this paper is to establish a DQN while considering signal fading characteristics with good backward compatibility. The DQN realizes the best action selection given by the Markov decision process (MDP) [14]. It models the surrounding environment as several states and actions, and agents use unsupervised learning to learn the surrounding environment. The agent makes actions that receives rewards to update current decision strategy. The purpose of the DQN is to find the optimal decision strategy, which utilizes the Q-network to perform the best action \(d_{t}\) based on current state \(s_{t}\). In this paper, the signal fading condition is regarded as state \(s\in\Omega_{s}\), where \(\Omega_{s}=\{s|s\in[sd_{1},~{}sd_{2}]\}\) is the state space. Furthermore, the handover parameter configuration is regarded as action \(a\in\Omega_{a}\), where \(\Omega_{a}\) represents the action space, which corresponds to all configurable handover parameters in this paper, in other words, \(\mathrm{card}(\Omega_{a})=\mathrm{card}(\Omega_{\mathrm{HO}})\). As shown in Figure 1, the Q-network is established to obtain the value \(\mathrm{Q}(s_{t},~{}a_{t})\) of each action \(a_{t}\) in the state \(s_{t}\), and the action decision is determined by the following formula [23]:  
\[\hat{d_{t}}=\underset{a_{t}\in\Omega_{a}}{\mathrm{argmax}}\mathrm{Q}(s_{t},~{ }a_{t}), \tag{8}\]  
In order to obtain a suitable Q-network, it is necessary to interact with the real environment and obtain the corresponding reward value of an action.