## 4 Proposed Trajectory Optimization Approach
### Proposed H-DQN Algorithm  
We use the SMDP to model the data collection and flight process of the UAV. In fact, the entire decision-making process is layered. The upper-level strategy \(u\) is used to select an option from the option set, and the strategy \(\pi_{o}\) in the option set is used to perform a series of bottom-level actions to complete the goal of the option. In the more typical deep reinforcement learning methods used to solve the SMDP problems, such as the hierarchical DQN (H-DQN) algorithm, both \(u\) and \(\pi_{o}\) need to be trained. For example, in a scene where there is a no-fly zone, the high-level strategy \(u\) is used to learn which option should be executed in the current state; the low-level strategy \(\pi_{o}\) learns the flight rules in the target area, bypassing the no-fly zone to reach the target point in the option. However, in the modeling of the SMDP in the above section, in order to simplify the scene, it is assumed that there is no no-fly zone, so the low-level strategy \(\pi_{o}\) can reach the target point in a straight flight. This is a pre-defined strategy that does not need to be learned. This chapter uses the H-DQN architecture to solve the SMDP problem. The H-DQN algorithm needs to first initialize the \(Q\) network parameters, experience the playback pool, and initialize the environment state and decision index in each training round. For each decision cycle, the agent selects an option \(o_{i}\) based on the current system state and a \(Q\) network based on the greedy strategy \(\varepsilon\). It randomly selects an option from the option space with the probability of \(\varepsilon\), and selects an option \(o_{i}=\underset{o}{\arg\max}\,Q(s_{i},o\mid\theta)\) according to the \(Q\) network with the probability of \(1-\varepsilon\). Then, it executes the strategy of the option. After the execution is complete, the system reaches the new state \(s_{i+1}\) and gets the reward \(r_{o,i}\), saving the four-tuple \(\langle s_{i},o_{i},s_{i+1},r_{o,i}\rangle\) into the experience replay pool. This algorithm does not use a fixed greedy coefficient, but allows itself to be continuously updated during the training process. The update formula can be given by  
\[\varepsilon=e^{\omega_{d}t_{epi}}, \tag{13}\]  
where \(n_{epi}\) represents the current episode round number, and \(\omega_{d}\) is the decay weight. \(\varepsilon\) will continue to decrease during the training process, together with the probability of exploration. When \(\omega_{d}\) is smaller, \(\varepsilon\) decreases faster. After updating the greedy coefficient, the time stamp is updated according to (3) and (4). After that, \(K\) quadruple samples are randomly sampled from the empirical playback pool and the target value is calculated. The calculation formula for the target value can be expressed as  
\[y_{k}=\left\{ \begin{array}{cc}r_{o,k}&\text{when}&s_{k}^{\prime}\text{ is the terminal state,}\\ r_{o,k}+\gamma\,\underset{a^{\prime}}{\max}Q(s_{k}^{\prime},o^{\prime}\mid\theta^{ \prime})&\text{when}&s_{k}^{\prime}\text{ is not the terminal state.}\end{array} \right. \tag{14}\]  
Then, \(\sum_{k=1}^{K}(y_{k}-Q(s_{k},o_{k}\mid\theta))\) is used as the objective loss function to update the parameters of the \(Q\) network. The H-DQN algorithm flow is shown as Algorithm 1. Every \(C\) updates cycles, and the parameters of the \(Q\) network are assigned to the target network for updating. When the remaining energy of the UAV is less than 0 or the mission execution time is reached, the current training round ends and the next round of training is entered.