## 4 The TRAJ-Q-GPSR Algorithm
### Traj-Q-GPSR Algorithm
#### 4.3.2 Design of the Reward Function  
The reward function forms the cornerstone of Q-learning, directly shaping the optimization direction of routing decisions. It is designed to balance multiple objectives--transmission efficiency, energy distribution, network load, and mobility prediction--while improving stability by mitigating routing voids. The composite reward function is expressed as  
\[R(s,a)=R_{\text{two-hop}}(s,a)+w_{v}R_{v}(s,a)+w_{h}P(s,a) \tag{24}\]  
where \(R_{\text{two-hop}}(s,a)\) denotes the two-hop reward, \(R_{v}(s,a)\) is the velocity projection reward, and \(P(s,a)\) is the routing void penalty term. The coefficients \(w_{v}\) and \(w_{h}\) represent the weights of the velocity reward and the void penalty, respectively.  
* Two-hop reward The two-hop reward The two-hop reward is composed of single-hop rewards and their extension. The single-hop reward \(R(s\to n)\) evaluates the immediate benefit of selecting a neighbor \(U_{n}\) as the next hop when UAV node \(U_{i}\) is in state \(s\). It considers three key factors: distance, residual energy, and queue load, defined as \[R(s\to n)=w_{d}R_{d}(n)+w_{e}R_{e}(n)+w_{q}R_{q}(n)\] (25) 1. Distance Reward \(R_{d}(n)\) Promotes selection of nodes closer to the destination, calculated as the proportion of distance reduced: \[R_{d}(n)=\frac{d_{i}-d_{n}}{d_{i}}\] (26) Here, \(d_{i}=\|p_{i}(t)-p_{D}\|\) is the Euclidean distance from the current node to the destination \(U_{D}\), and \(d_{n}=\|p_{n}(t)-p_{D}\|\) is the distance from the neighbor to the destination. If \(d_{n}<d_{i}\), then \(R_{d}(n)>0\), yielding a positive reward; otherwise, it is negative. 2. Energy Reward \(R_{e}(n)\) This prioritizes nodes with higher remaining energy to extend network lifetime: \[R_{e}(n)=\frac{E_{n}}{E_{\max}}\] (27) where \(E_{n}\) is the remaining energy of \(U_{n}\), and \(E_{\max}\)is the maximum energy capacity. 3. Queue Length Reward \(R_{q}(n)\) This prefers nodes with lower loads to reduce transmission delay: \[R_{q}(n)=1-\frac{Q_{n}}{Q_{\max}}\] (28) Expanding the single-hop reward to consider the influence of the next-hop neighbors, the two-hop reward is defined as \[R_{\text{two-hop}}(s,a)=R(s\to n)+\sum_{m\in\mathcal{N}(n)}\frac{1}{D_{m}}R(n \to m)\] (29) where \(R(n\to m)\) is the single-hop reward from \(U_{n}\) to its neighbor \(U_{m}\), and \(D_{m}\) is the degree of \(U_{m}\) (number of neighbors). The normalization factor \(\frac{1}{D_{m}}\) prevents reward inflation in highly connected regions.
* Velocity Projection Reward In UAV networks, node mobility affects link stability. The velocity projection reward \(R_{v}(s,a)\) leverages trajectory information to evaluate whether the selected next-hop UAV is moving toward the destination, improving adaptation to network dynamics. Given the velocity vector \(v_{n}(t)\) of UAV \(U_{n}\) and the directional vector \(d=p_{d}-p_{n}(t)\) from \(U_{n}\) to the destination \(U_{d}\), the reward is defined as \[R_{v}(s,a)=\cos(\theta)=\frac{v_{n}(t)\cdot d}{|v_{n}(t)||d|}\] (30) A positive \(\cos(\theta)\) indicates movement toward the destination, yielding a positive reward; a negative value indicates movement away, resulting in a penalty. This encourages selecting UAVs whose motion trends enhance link availability.
* Routing Void PenaltyTo prevent routing voids--where no neighbor is closer to the destination than the current node--the penalty term refines void severity assessment. Void severity \(H(s)\) is defined as  
\[H(s)=1-\frac{N_{\text{closer}}(s)}{N_{\text{total}}(s)} \tag{31}\]  
where \(N_{\text{closer}}(s)\) is the number of neighbors closer to the destination than \(U_{i}\), and \(N_{\text{total}}(s)\) is the total number of neighbors. The penalty is  
\[P(s,a)=-H(s) \tag{32}\]  
If \(H(s)=1\), indicating a full routing void, the penalty is maximized.