## 4 Reinforcement Learning-Based Joint Optimization Strategy for Flight Path and Resource Allocation
### Aerial Platform Flight Path Optimization Algorithm  
In this paper, each AP makes independent decisions in each time slot based on the value of each state provided by environmental feedback. The environment changes based on the decisions made and the rewards obtained in each state, eventually forming a finite MDP within an episode. Therefore, the MDP can be employed to model this subproblem. In this model, each AP is considered as an intelligent agent. The system can be described using a quadruple \(\Phi=\{\mathcal{M},\mathcal{S},\alpha,\mathcal{R}\}\), where \(\mathcal{M}\) represents the number of agents, \(\mathcal{S}\) represents the states in which each agent is located, \(\alpha\) represents the actions chosen by the agents in those states, and \(\mathcal{R}\) represents the rewards obtained from the actions of the agents in those states. Based on the four elements mentioned above, the definition of the agent's state space, action space, and reward function is as follows:  
1. State Space: The entire two-dimensional coordinates within the communication task range of all APs are discretely represented. The obtained set of two-dimensional coordinates constitutes the state space of the agents. The position \(q_{m}(t)=[x_{m}(t),y_{m}(t)]^{T}\) of each agent in each time slot represents the agent's state \(S_{m}\). 2. Action Space: we simplify the Computational complexity in this algorithm by considering the agent's maximum speed constraint \((v_{max}\Delta t)\) and selecting a set of equidistant fixed displacement values \(\lambda_{k},\;k=1,\cdots,K\). By applying these fixed displacement values to both the \(x\) and \(y\) coordinates of the two-dimensional state positions, a combination of \(K^{2}\) attainable target positions in one time slot is generated. The selection of all fixed displacement values for the \(x\) and \(y\) axes constitutes the action space of the agent. 3. Reward Function: The reward function \(R_{m}(a_{m}|s_{m})\) for AP \(m\) is defined as the weighted sum of the channel gains of various GPFs at the target position based on their communication workload. This sum is then added to a function of the distances between the AP and other APs and multiplied by a certain proportionality factor \(\rho\). The expression is as follows: \[R_{m}(A_{m}|S_{m})=\sum_{i=1}^{N}\gamma_{i}h_{i,m}(t)+\rho\cdot\log_{2}\Bigl{(} 1+\sum_{j=1,j\
eq m}^{M-1}d_{j,m}\Bigr{)}\] (15)where \(\gamma_{i}\) serves as the weighting factor for the workload of GPFs. \(h_{i,m}(t)\) represents the channel gain between GPF \(i\) and AP \(m\). \(d_{j,m}\) denotes the distance between AP \(m\) and other APs. The purpose of the second term is to prevent prolonged path overlap between different APs. Based on the proposed model, we present a comprehensive sub-iteration multi-agent iterative AP path optimization algorithm that considers both the positions and workload of GPFs. The specific algorithm procedure is illustrated in Algorithm 1. Initially, the maximum training episodes \(N_{\text{epi}}\) are initialized, and the initial state \(s_{m}(0)\) is given for each AP (Step 1). In each time slot within an episode loop (Step 2), the algorithm calculates the reward function for each reachable state of each AP (Step 2a). According to a greedy strategy, the APs move and update their states \(s_{m}(t)\) (Step 2b). In the overall algorithm flow, the weights of the reward function are updated based on the remaining communication workload for GPFs derived from Algorithm 2 (Step 2c). After completing one training episode, the primary algorithm parameters are initialized, and the next training episode is executed (Step 3). This process is repeated until all training episodes are completed (Step 4). ```
(1) Initialization: Maximum training episodes \(N_{\text{epi}}\), for any AP \(m\), given its initial position state \(s_{m}(0)\).