## 4 Enhanced DQN Based on Digital Twin
### Digital Twin Based on LSTM
under the \(n_{1}\) measurements is assumed to follow the Bernoulli distribution \(X\)\(\sim\)\(\mathcal{B}\)\(\left(n_{1},p_{f}\right)\), where \(p_{f}\) is the handover failure probability, let \(\mathrm{S_{x}}=\left\{x_{1},\ x_{2}\mathrm{L},x_{n_{1}}\right\}\), \(x_{i}\in\left\{0,\ 1\right\}\) represents the sample of the \(i\)-th measurement. \(x_{i}=1\) is for event occurring, i.e., the handover failing, and \(x_{i}=0\) is for event not occurring. Then the maximum likelihood unbiased estimation of \(p_{f}\) can be expressed as:  
\[p_{f}=\frac{1}{n_{1}}\sum\
olimits_{i=1}^{n_{1}}x_{i}. \tag{11}\]  
This is an estimation method when statistics from an actual system can be collected. However, when using LSTM to predict a single event, uncertainty would be introduced because its true positive probability \(p_{p}\) and true negative probability \(p_{n}\) are not 1. Here \(p_{p}\) represents the probability that the LSTM predicts the event to occur when it does occur, and \(p_{n}\) represents that the LSTM predicts the event not to occur when it does not occur. Therefore, the estimator (11) could not be applied for scenarios with LSTM prediction. After obtaining a series of event samples \(\mathrm{S_{y}}=\left\{y_{1},\ y_{2},\ldots,y_{n_{2}}\right\}\), \(y_{i}\in\left\{0,\ 1\right\}\) predicted by LSTM, this paper proposes the following proposition. Proposition 1:  
With LSTM-predicting events, the unbiased estimation of \(p_{f}\) is expressed as:  
\[p_{f}=\frac{p_{n}-1+\frac{1}{n_{2}}\sum\
olimits_{i=1}^{n_{2}}y_{i}}{p_{n}+p_ {p}-1}. \tag{12}\]  
Proof. If the handover failure were predicted to occur, it could be a correct prediction of the event occurring or a false prediction of the event not occurring. So the probability of LSTM predicted handover failure is calculated as:  
\[ \begin{array}{ll}\mathrm{p}(y_{i}=1)&=\mathrm{p}(x_{i}=1)\mathrm{p}(y_{i}=1 |x_{i}=1)+\mathrm{p}(x_{i}=0)\mathrm{p}(y_{i}=1|x_{i}=0)\\ &=p_{f}p_{p}+(1-p_{n})\Big{(}1-p_{f}\Big{)}\end{array} . \tag{13}\]  
The number of LSTM predicted failures obeys the Bernoulli distribution \(Y\sim\mathcal{B}(n_{2},\ p_{Y})\), \(p_{Y}=\mathrm{p}(y_{i}=1)=p_{f}p_{p}+(1-p_{n})\Big{(}1-p_{f}\Big{)}\), where \(p_{Y}\) is derived by Equation (13). Then, the unbiased estimate of the Bernoulli distribution parameter \(p_{Y}\) can be obtained by the maximum likelihood criterion:  
\[p_{Y}=\frac{1}{n_{2}}\sum\
olimits_{i=1}^{n_{2}}y_{i}. \tag{14}\]  
Since estimator (14) is the unbiased estimator of \(p_{Y}\), the expectation of estimator (14) is equal to \(p_{Y}\), expressed as:  
\[\mathrm{E}(\frac{1}{n_{2}}\sum\
olimits_{i=1}^{n_{2}}y_{i})=p_{f}p_{p}+(1-p_{ n})\Big{(}1-p_{f}\Big{)}. \tag{15}\]  
Calculate the expectation of Formula (12) substituting Formula (15) and the following equation can be derived:  
\[\mathrm{E}\Big{(}p_{f}\Big{)}=p_{f}. \tag{16}\]  
Since the expectation of the estimator (12) for the parameter \(p_{f}\) is equal to \(p_{f}\), the estimator (12) is the unbiased estimation of \(p_{f}\). Proposition 1 is proved. Corollary 1:  
The variance of unbiased estimation of (12) is expressed as:  
\[\mathrm{D}\Big{(}p_{f}\Big{|}\mathrm{S_{Y}}\Big{)}=\frac{1}{n_{2}}\cdot\left( \frac{1}{p_{n}+p_{p}-1}\right)^{2}p_{Y}(1-p_{Y}). \tag{17}\]  
Proof:Due to Bernoulli distribution \(Y\sim\mathcal{B}(n_{2},\ p_{Y})\), the variance of unbiased estimation of \(p_{Y}\) is expressed as:  
\[\mathrm{D}(\hat{p_{Y}}) =\Big{(}\frac{1}{n_{2}}\Big{)}^{2}\cdot n_{2}p_{Y}(1-p_{Y}) \tag{18}\] \[=\frac{1}{n_{2}}p_{Y}(1-p_{Y})\]  
Get the variance of Formula (12) and substitute (18) to derive the following equation:  
\[\mathrm{D}\Big{(}\hat{p_{f}}\Big{|}\mathrm{S}_{Y}\Big{)} =\Big{(}\frac{1}{p_{n}+p_{p}-1}\Big{)}^{2}\cdot\mathrm{D}\Big{(} \frac{1}{n_{2}}\sum_{i=1}^{n_{2}}y_{i}\Big{)} \tag{19}\] \[=\frac{1}{n_{2}}\cdot\Big{(}\frac{1}{p_{n}+p_{p}-1}\Big{)}^{2}p_{ Y}(1-p_{Y})\]  
Corollary 1 is proved.