## 4 Proposed Trajectory Optimization Approach
### Proposed Model  
We consider that the UAV can return to the control station for charging when the battery is about to run out, and then continue its data collection task. This paper uses the SMDP to model the UAV-assisted data collection process, and regards the UAV responsible for data collection as an agent, which is designed as follows.  
* State space model \(\mathcal{S}\): The system state space includes the state of the UAV and the AoI of the sensor equipment. At each time \(t\), the system state is expressed as \(s_{t}=\{p_{u}(t),e(t),\triangle(t)\}\). Among them, \(p_{u}(t)\), \(e(t)\) are the own state of the UAV, representing the current position and remaining energy of the UAV, respectively, and \(\triangle(t)=\{\triangle_{1}(t),\dots,\triangle_{n}(t)\}\) represents the current AoI of each sensing device.
* Work space model \(\mathcal{A}\): In this paper, the no-fly areas due to terrain and other reasons are not considered, and the UAV can reach the target area by flying in a straight line. Therefore, the action space includes three categories: \(a_{f},a_{c}\), and \(a_{r}\), where \(a_{f}=\{a_{f,0},a_{f,1},...,a_{f,n},...,a_{f,N}\},n\in\mathcal{N}\) represents the flight action of the UAV, and \(a_{f,n}\) is the action of the UAV flying straight from the current position to the \(n\)-th sensing device, in which \(a_{f,0}\) represents the UAV flying in a straight line above the control station. \(a_{c}\) indicates that the UAV performs a data collection action, and \(a_{r}\) indicates that the UAV performs a charging action.
* Option set \(\mathcal{O}\): The set of options to be executed by the UAV agent in this chapter includes \(o_{c},o_{r}\), where \(o_{c}=\{o_{c,1},\dots,o_{c,n},\dots,o_{c,N}\},n\in\mathcal{N}\), and \(o_{c,n}\) represents the data of the \(n\)-th sensor device for the collection tasks. The UAV first flies in a straight line from the current position to the top of the \(n\)-th sensing device, executing the action \(a_{f,n}\), and then executes the data collection action \(a_{c}\). The completion of data collection is the suspension state. \(o_{r}\) represents the option of returning home for charging. The strategy of this option is that the UAV first flies from the current position to the control station executing the action \(a_{f,0}\), and then executes the charging action \(a_{r}\). The suspension state means that the power of the UAV reaches \(E_{max}\). In this SMDP, the selectable option set of the agent in each state is the entire option set \(I_{0}=\mathcal{S}\).
* Reward function \(\mathcal{R}\): In the SMDP constructed in this paper, the agent can only get rewards after executing an option, and all options use the same reward function. The instantaneous reward function of the \(i\)-th decision can be expressed as \[r_{o,i}=\left\{\begin{array}{cl}\frac{1}{T_{i}}\sum_{t=-j-1}^{j-1}T_{j}\\ \frac{1}{T_{i}}\sum_{t=-j-1}^{j-1}T_{j}\end{array}Q(t)+\beta_{e}\frac{e(\sum_{ j=1}^{i}T_{j})}{E_{max}}&\text{for}&e(\sum_{j=1}^{i}T_{j})\geq e_{min},\\ p&\text{for}&e(\sum_{j=1}^{i}T_{j})<e_{min}.\end{array}\right.\] (12) It can be seen from the above formula (12) that there are two main situations for instantaneous rewards. \(e_{min}\) represents the minimum energy consumption that can guarantee the return of the UAV. After an option is executed, it will bring a penalty value \(p\) if the energy consumption of the UAV is less than the threshold. Otherwise, the instantaneous reward is mainly composed of two parts: the first item represents the average QoS during the execution of the option, and the second item represents the reward brought by the current remaining energy consumption. \(\beta_{e}\) represents the weight of energy consumption rewards, which is used to balance the proportion of the energy consumption rewards with the overall rewards.