## 4 Reinforcement Learning-Based Joint Optimization Strategy for Flight Path and Resource Allocation
### Device Association and Power Control Algorithm
Based on the aforementioned model, we propose a device association and power control algorithm based on the multi-agent K-armed bandit. The specific algorithm procedure is illustrated in Algorithm 2. Firstly, we initialize the maximum training episodes \(N_{\text{epi}}\), exploration parameter \(\varepsilon\), state-action value function \(Q_{n}^{1}(s,a)\), and energy consumption estimation function \(P_{n}^{1}(s,a)\), along with the positions and initial workload \(\mu_{n}=(\mu_{n,\text{immediate}},\mu_{n,\text{non-immediate}})\) for each GPF (Step 1). In each time slot within an episode loop (Step 2), it is determined whether GPFs have immediate data tasks based on \(\mu_{n}\). If immediate tasks exist, execute policy \(\pi_{n,\text{immediate}}\) (Step 2a), which means that GPFs will upload real-time data to the nearest AP at maximum power. If the optional APs have reached the maximum number of connections, they will be directly connected to the LEO satellite. If only non-immediate data tasks remain, select action \(a_{n}(t)\) based on policy \(\pi_{n}^{\varepsilon}\) (Step 2b). Then, calculate the instantaneous reward \(R_{n}(s(t),a(t))\) and instantaneous cost \(P_{n}(s(t),a(t))\) based on action \(a_{n}(t)\), update the remaining workload, and use the remaining workload as a state change for the iteration of Algorithm 1 in the next step (Step 2c). Update the state-action value function \(Q_{n}^{k}(s,a)\) and energy cost estimation function \(P_{n}^{k}(s,a)\) using \(R_{n}(s(t),a(t))\) and \(P_{n}(s(t),a(t))\), and compute the energy efficiency estimation function \(\eta_{n}^{k}(s,a)\) for the current time \(t\) (Step 2d). After completing one training episode, initialize the main algorithm parameters and execute the next training episode (Step 3). Repeat the above process until all training episodes are completed (Step 4). ```
(1) Initialization: Maximum training episodes \(N_{\text{epi}}\), exploration parameter \(\varepsilon\), state-action value function \(Q_{n}^{1}(s,a)=0\), and energy cost estimation function \(P_{n}^{1}(s,a)=0\). (2) For GPF \(n\), given its position and initial workload \(\mu_{n}=\big{(}\mu_{n,\text{immediate}},\mu_{n,\text{non-immediate}}\big{)}\). (3) For each iteration in episode \(N_{\text{epi}}\), \(t=1,2,\ldots,T\), prioritize GPFs based on the type and quantity of the remaining workload. Then, instruct each GPF to independently execute the following steps: (a) Check if immediate data tasks still exist; if yes, execute policy \(\pi_{n,\text{immediate}}\); (b) If only non-immediate data tasks remain, select action \(a_{n}(t)\), based on policy \(\pi_{n}^{\varepsilon}\); (c) Based on the executed action \(a_{n}(t)\), obtain instantaneous reward \(R_{n}(s(t),a(t))\), and instantaneous cost \(P_{n}(s(t),a(t))\), update the remaining workload, and use the remaining workload as a state change for the iteration of Algorithm 1 in the next step; (d) Update the state-action value function \(Q_{n}^{k}(s,a)\) and energy cost estimation function \(P_{n}^{k}(s,a)\) based on the instantaneous reward and cost, and update the energy efficiency estimation function \(\eta_{n}^{k}(s,a)\) for the current time. (4) Set \(N_{\text{epi}}=N_{\text{epi}}-1\). (5) Repeat steps (2) to (4) until \(N_{\text{epi}}=0\). ```  
**Algorithm 2** Algorithm for Device Association and Power Control Based on Multi-Agent