## 4 Reinforcement Learning-Based Joint Optimization Strategy for Flight Path and Resource Allocation
### Device Association and Power Control Algorithm  
In this paper, when GPFs make action selections, the moving strategy of APs is not considered. Only the current position of APs at the current time slot is taken as the environmental state. Therefore, the decision-making process for device association and power control of GPFs can be modeled as a time-varying K-armed bandit model with multiple intelligent agent states. The action selection of the intelligent agent is considered not to affect changes in the environmental state. This model regards GPFs as intelligent agents, and the system can be represented by a quadruple \(\Phi=\{\mathcal{M},\mathcal{S},\omega,\mathcal{R}\}\). The definition of the agent's state space, action space, and reward function is as follows:  
1. State Space: To simplify the algorithm complexity, in the proposed algorithm, the distances between APs and intelligent agents are discretized into \(L\) state distributions based on the provided channel gain. The distances between each AP and intelligent agent constitute a state space with \(L^{M}\) elements. 2. Action Space: Defined as the device association and power control schemes that agents can choose. Agents determine their action choices based on the current state matrix and the policy \(\pi\) specified by the algorithm at the current time. 3. Reward Function: The reward function for the agent \(n\) at time slot \(t\) is defined as the instantaneous throughput and instantaneous energy consumption as follows: \[R_{n}(s(t),a(t))=R_{n}(t)\cdot\Delta t,\] (16) \[P_{n}(s(t),a(t))=E_{n}(t),\] (17) where \(R_{n}(t)\) represents the transmission rate of the agent \(n\) in time slot \(t\) and \(E_{n}(t)\) represents the actual communication energy consumption generated by the agent \(n\) in time slot \(t\). Let \(Q_{n}^{k}(s,a)\) represent the estimated value of the action chosen by intelligent the agent \(n\) in state \(s\) after \(k-1\) selections, which is expressed as follows: \[Q_{n}^{k}(s,a)=\frac{R_{n}^{1}+R_{n}^{2}+\cdots+R_{n}^{k-1}}{k-1},\] (18) \[Q_{n}^{k+1}(s,a)=Q_{n}^{k}(s,a)+\frac{1}{k}\Big{(}R_{n}^{k}-Q_{n}^{k}(s,a) \Big{)}\] (19)  
Let \(P_{n}^{k}(s,a)\) represent the estimated cost of the agent \(n\) choosing action \(a\) in state \(s\) after \(k-1\) selections, which is expressed as follows:  
\[P_{n}^{k}(s,a)=\frac{P_{n}^{1}+P_{n}^{2}+\cdots+P_{n}^{k-1}}{k-1}, \tag{20}\]  
\[P_{n}^{k+1}(s,a)=P_{n}^{k}(s,a)+\frac{1}{k}\Big{(}P_{n}^{k}-P_{n}^{k}(s,a) \Big{)}, \tag{21}\]  
then, the efficiency estimation function for the agent \(n\) choosing action \(a\) in state \(s\) for the \(k\)-th time can be expressed as  
\[\eta_{n}^{k}(s,a)=\frac{Q_{n}^{k}(s,a)}{P_{n}^{k}(s,a)}=\frac{R_{n}^{1}+R_{n}^ {2}+\cdots+R_{n}^{k-1}}{P_{n}^{1}+P_{n}^{2}+\cdots+P_{n}^{k-1}}. \tag{22}\]  
The policy for the action selection of the agent \(n\) is represented as  
\[A_{n}^{t}=\underset{a}{\text{argmax}}\Big{[}\eta_{n}^{k}(s,a)\Big{]}. \tag{23}\]  
This signifies that the agents always choose the strategy that maximizes the current estimated energy efficiency. However, to balance exploration and exploitation, there is a probability \(\varepsilon\in(0,1)\) for exploration to discover better actions. The specific policy is expressed as follows:  
\[\pi_{n}^{\varepsilon}\big{(}A_{n}^{t}|s\big{)}= \begin{cases}1-\varepsilon,\;A_ {n}^{t}=\underset{a}{\text{argmax}}\Big{[}\eta_{n}^{k}(s,a)\Big{]}\\ \varepsilon,\text{otherwise}\end{cases} , \tag{24}\]  
and to enhance the algorithm's overall performance and expedite the convergence speed of iterations, \(\varepsilon\) can be set as a linear function related to the iteration episodes \(N_{\text{epi}}\), as shown below:  
\[\varepsilon=0.5-\rho_{\text{epi}}\cdot N_{\text{epi}}, \tag{25}\]  
where \(\varepsilon\) represents the exploration rate and \(\rho_{\text{epi}}\) denotes the rate at which the exploration rate decreases with \(N_{\text{epi}}\). When \(\varepsilon=0\), the training process will conclude, yielding the corresponding training results.