## 4 The TRAJ-Q-GPSR Algorithm
### Traj-Q-GPSR Algorithm
#### 4.3.3 Adaptive Q-Learning Parameters  
The learning rate \(\alpha\) and discount factor \(\gamma\) are critical parameters in Q-learning, directly influencing the convergence speed and decision-making performance of the algorithm. Given the high dynamics and task diversity in UAV networks, these parameters require adaptive adjustment. In rapidly changing environments, they should facilitate quick learning, while in more stable conditions, they enable fine-tuned decision-making. This approach ensures a balanced trade-off between exploration and exploitation, thereby enhancing adaptability to topology variations.  
The learning rate \(\alpha\) determines the extent to which new information influences the existing Q-values. A fixed \(\alpha\) approach can struggle in dynamic environments: it may converge too slowly when the environment changes drastically or over-adjust and deviate from the optimal solution when the environment is stable. To address this, an adaptive adjustment mechanism based on Q-value fluctuations is proposed. When Q-values fluctuate significantly, indicating unstable learning, \(\alpha\) should increase to speed up updates. When Q-values stabilize, \(\alpha\) should decrease to refine the strategy. The formula for dynamically adjusting \(\alpha\) is  
\[\alpha=\alpha_{\text{min}}+(\alpha_{\text{max}}-\alpha_{\text{min}})\cdot\frac {1}{1+|Q(s,a)-Q_{\text{prev}}(s,a)|} \tag{33}\]  
where \(\alpha_{\text{max}}\) and \(\alpha_{\text{min}}\) are the maximum and minimum allowable values for the learning rate, \(Q(s,a)\) and \(Q_{\text{prev}}(s,a)\) represent the current and previous Q-values, respectively. This strategy enables UAVs to rapidly adjust their policies in dynamic environments while maintaining stability during convergence.  
The discount factor \(\gamma\) determines the balance between immediate rewards and long-term returns. In UAV networks, changes in the neighbor set reflect the network's topological instability. When the neighbor set changes significantly, future uncertainty rises, and \(\gamma\) should decrease to reduce reliance on future rewards. When changes are minimal, \(\gamma\) can increase to prioritize long-term planning. The neighbor change rate is defined as  
\[\Delta N=|N(t_{k})-N(t_{k-1})| \tag{34}\]  
Based on this, the adaptive adjustment formula for \(\gamma\) is  
\[\gamma=\gamma_{\text{min}}+(\gamma_{\text{max}}-\gamma_{\text{min}})\cdot e^{- \beta\Delta N} \tag{35}\]  
where \(\gamma_{\text{max}}\) and \(\gamma_{\text{min}}\) denote the upper and lower bounds of the discount factor, and \(\beta\) is a tuning parameter. When \(\Delta N\) is large (indicating a highly dynamic environment), \(\gamma\) decreases, focusing on immediate decisions over uncertain future rewards. When \(\Delta N\) is small (stable environment), \(\gamma\) nears \(\gamma_{\text{max}}\), emphasizing long-term gains. This approach enhances adaptability in dynamic conditions while maintaining optimization in stable ones.  
By dynamically adjusting the learning rate \(\alpha\) based on Q-value fluctuations and the discount factor \(\gamma\) based on neighbor change rates, this strategy enables Q-learning to quickly adapt to environmental shifts in UAV networks while ensuring long-term convergence stability. This mechanism significantly boosts the efficiency and robustness of routing decisions, offering an effective solution for learning and optimization in UAV network applications.