## 4 Strategy Creation for UAV Path Planning Based on Q-Learning
### The Optimal Strategy for UAV Path Planning Based on Q-Learning
If we suppose that the observation noise of the sensors obeys zero-mean Gauss distribution, then the measurement error distribution of the system at time \(k\) is  
\[w_{k}\,\sim(0,Q_{k}(m_{k},s_{k})). \tag{17}\]  
Accordingly, the belief level of the system observation at present is  
\[b_{k}(m_{k})=(m_{k}-\mu_{k},\hat{\sigma}_{k}), \tag{18}\]  
where  
\[\hat{\sigma}_{k+1}=\frac{1}{\frac{1}{\hat{\sigma}_{k+1|k}}+s_{k+1}}. \tag{19}\]  
The observation model of the standard path according to the measurements from the onboard sensors of UAV is  
\[z_{k}^{m}=\mathrm{I}_{k}m_{k}^{p}+\chi_{k}\,,\,\chi_{k}\,\sim(0,\hat{\varsigma }_{k}), \tag{20}\]  
where \(\mathrm{I}_{k}\) is an identity matrix; since the standard path is determined in advance, the dynamic of the standard path can be seen as the direct measurement according to the onboard sensors relative to the UAV, that is to say, Equation (20) can be regarded as a dynamic model of the standard path superimposing the onboard sensors' measurement error. Then, according to matrix theory and stochastic process theory, we have  
\[\hat{\sigma}_{k+1|k}=\mathrm{I}_{k}\hat{\sigma}_{k}\mathrm{I}_{k}^{T}+\hat{ \varsigma}_{k}=\hat{\sigma}_{k}+\hat{\varsigma}_{k} \tag{21}\]  
\[b_{k}=\Big{(}b_{k}^{s},\;\;b_{k}^{m},\;\;b_{k}^{u},\;\;b_{k}^{s}\Big{)}, \tag{13}\]0  
Combining with Equation (2), it can be seen in Equation (19) that the covariance matrix of the system at time \(k+1\) depends on the measurements at that time. Since these observations cannot be measured exactly, the position of the UAV at time \(k+1\) can only be approximated by the measurements of the standard path at the exact time, and it needs to be adjusted according to the line of sight between the sensors and the standard path. Consequently, the minimum mean square covariance of the current planning path and the standard path, that is to say, the cost function of the path planning system, can be written as below:  
\[b_{k}=\Big{(}b_{k}^{s},\;\;b_{k}^{m},\;\;b_{k}^{u},\;\;b_{k}^{s}\Big{)}, \tag{13}\]1  
Accordingly,  
\[b_{k}=\Big{(}b_{k}^{s},\;\;b_{k}^{m},\;\;b_{k}^{u},\;\;b_{k}^{s}\Big{)}, \tag{13}\]2  
Consequently, the corresponding optimal control strategy of the UAV path planning based on Q-learning can be obtained with the minimum value of Equation (24) winning. To handle the hidden state uncertainty of the UAV path planning problem, we suppose there is \(a\%\) distance uncertainty and \(b\%\) angle uncertainty in the observation covariance distribution \(w_{k}\,\sim(0,Q_{k}(m_{k},s_{k}))\) of the system, and accordingly, if the position coordinates of the standard path \(m_{k}\) at time \(k\) are \((\rho_{k},\theta_{k},\varphi_{k})\), then the standard deviation between the planning path and the standard path at time \(k\) is denoted below:  
\[b_{k}=\Big{(}b_{k}^{s},\;\;b_{k}^{m},\;\;b_{k}^{u},\;\;b_{k}^{s}\Big{)}, \tag{13}\]3If the intersection angle among the standard path point to the centroid of the UAV and the axes of the sensor is \(\gamma\), then the process noise covariance matrix \(Q_{k}\) can be calculated as below:  
\[Q_{k}=U_{k}\left[ \begin{array}{ccc}\rho_{k}^{2}&0&0\\ 0&\theta_{k}^{2}&0\\ 0&0&\varphi_{k}^{2}\end{array} \right]U_{k}^{T}, \tag{26}\]  
where  
\[U_{k}=\left[ \begin{array}{ccc}\cos\gamma_{k}&-\sin\gamma_{k}&0\\ \sin\gamma_{k}&\cos\gamma_{k}&0\\ 0&0&1\end{array} \right]. \tag{27}\]  
Obviously, the eigenvalues of the matrix \(Q_{k}\) are \(\left\{\rho_{k}^{2},\theta_{k}^{2},\varphi_{k}^{2}\right\}\).