## 4 Strategy Creation for UAV Path Planning Based on Q-Learning
### The Optimal Strategy for UAV Path Planning Based on Q-Learning  
Generally speaking, the target of UAV path planning problem is a minimum mean square cumulative error between the current solving path and the standard path over a given time sequence \(k=[0,1,2,\cdots,H-1]\), and the expected cumulative error on the time sequence can be described as  
\[J_{H}=E\Big{[}{\sum}_{k=0}^{H-1}c\big{(} \begin{array}{cc}b_{k},&u_{k}\end{array}  \big{)}\big{]}. \tag{10}\]  
The action selection of the UAV at time \(k\) depends on the observation measurements of all the former \(k-1\) time. If there is an optimal action that would lead the UAV to the expected path at time \(k\), then there is an optimal action sequence relying on the feedback of the belief state. Therefore, the cost function can be given as below:  
\[J_{H}=E\Big{[}{\sum}_{k=0}^{H-1}c\big{(} \begin{array}{cc}b_{k},&u_{k}\end{array}  \big{)}\big{|}b_{0}\Big{]}, \tag{11}\]  
where,  
\[c(b_{k},\;\;u_{k})=\int C(x_{k},\;\;u_{k})b_{k}(x)dx. \tag{12}\]  
Belief state \(b_{k}\) is the posterior probability distribution of hidden states of the UAV path planning system at time \(k\), that is,  
\[b_{k}=\Big{(}b_{k}^{s},\;\;b_{k}^{m},\;\;b_{k}^{u},\;\;b_{k}^{s}\Big{)}, \tag{13}\]  
where \(b_{k}^{s}=\delta(s-s_{k})\), \(b_{k}^{u}=\delta(\mu-\mu_{k})\), \(b_{k}^{e}=\delta(\sigma-\sigma_{k})\), and \(b_{k}^{m}\) is the posterior probability distribution of the standard path information measured by the onboard sensors of the UAV. According to [PERSON]'s optimal principle [28], the optimal cost function value \(J_{H}^{*}\) under the current given belief state \(b_{0}\) can be written as  
\[J_{H}^{*}(b_{0})=\min_{u}\big{\{}c(b_{0},u)+E\big{[}J_{H-1}^{*}(b_{1})|b_{0},u \big{]}\big{\}}, \tag{14}\]  
where \(b_{1}\) is the random value of the belief state at the next time step, \(J_{H-1}^{*}\) is the optimal cumulative cost over the time sequence \([0,1,\cdots,H-1]\), and \(E\big{[}J_{H-1}^{*}(b_{1})|b_{0},u\big{]}\) denotes the conditional expectation under the given current belief state \(b_{0}\) at time \(k=0\) and after the UAV taking the action \(u\). Accordingly, the optimal strategy at time \(k=0\) can be given as below:  
\[\pi_{0}^{*}(b_{0})=\arg J_{H}^{*}(b_{0})=\arg\big{\{}\min_{u}\big{\{}c(b_{0},u )+E\big{[}J_{H-1}^{*}(b_{1})|b_{0},u\big{]}\big{\}}\big{\}}. \tag{15}\]  
Consequently, the optimal control strategy of arriving at the target path at time \(k\) is  
\[\pi_{k}^{*}(b_{k})=\arg J_{H-k}^{*}(b_{k})=\arg\big{\{}\min_{u}\big{\{}c(b_{k},u)+E\big{[}J_{H-k-1}^{*}(b_{k+1})|b_{k},u\big{]}\big{\}}\big{\}}. \tag{16}\]  
However, in practice the second term \(E\big{[}J_{H-k-1}^{*}(b_{k+1})|b_{k},u\big{]}\) in the optimal control strategy \(\pi_{k}^{*}(b_{k})\) (the above formula) is hard to obtain exactly; therefore, this problem needs to be transformed as follows in order to find the optimal answer. Suppose the UAV always takes the optimal flight control strategy at every flying path point starting from time \(k=0\); if the optimal action sequence corresponds to the optimal cumulative cost function \(J_{H}^{*}\) over the time sequence \([0,1,\cdots,H-1]\) is \(\big{(}u_{0}^{*},u_{1}^{*},\cdots,u_{H-1}^{*}\big{)}\), the covariance matrix sequence among the planned optimal path points is \(\big{(}\sigma_{1}^{*},\sigma_{2}^{*},\cdots,\sigma_{H}^{*}\big{)}\). Because the measurement of the system is related to the standard path information and the movement state of the UAV, the measurement error comes from two aspects, the estimation of the standard path and the estimation of the movement state of the UAV itself. According to the observation model, as Equation (1) shows, if the standard path information is fully observed, there is a partially observable hidden state in the current movement of the UAV.