## 3 Federated Learning for Space-Air-Ground Integrated Networks
### Device Selection Optimization Strategy Based on Hierarchical Reinforcement Learning
#### 3.4.1 Hierarchical Reinforcement Learning Strategy  
When addressing device selection problems in space-air-ground integrated networks, traditional single reinforcement learning methods prove inefficient due to varied device types, network conditions, and computational capabilities. We introduce a hierarchical reinforcement learning strategy dividing the selection process into global (high-level) and local (low-level) layers, effectively managing device heterogeneity and dynamic network topology changes. The hierarchical reinforcement learning approach separates device selection into Meta-Policy (selecting device categories) and Device-Level Policy (selecting specific devices), enabling targeted optimization at each layer. We introduce the Markov Decision Process (MDP) [33], using MDP to obtain a device selection policy to help edge nodes make optimal decisions in dynamic network environments. In MDP, the device selection and aggregation process is decomposed into several steps, optimized at both the global layer (device category selection) and local layer (specific device selection). The Markov Decision Process defines five elements of the decision-making process: state space (\(S\)), action space (\(A\)), reward function (\(R\)), state transition probability (\(T\)), and discount factor (\(\gamma\)). The objective of the global layer is to select device categories, i.e., which types of devices (satellites, UAVs, or ground devices) participate in federated learning tasks. The global layer's state space includes parameters such as bandwidth, coverage area, and latency of device categories. The reward function measures the contribution of device category selection to the global model. Global layer state space \(S_{\text{high}}\) is represented as follows:  
\[S_{\text{high}}=\{bandwidth,coverage\_area,latency\} \tag{6}\]  
where _bandwidth_ represents the bandwidth of the device category, _coverage_area_ represents the coverage area of the device category, and _latency_ represents the latency of the device category. Global layer action space \(A_{\text{high}}\) is represented as follows:  
\[A_{\text{high}}=\{satellite,\text{UAV},ground\} \tag{7}\]  
where _satellite_ represents selecting space-based layer devices such as satellites, UAV represents selecting aerial layer devices such as unmanned aerial vehicles, and _ground_ represents selecting ground devices. Global layer reward function \(R_{\text{high}}\) is represented as follows:  
\[R_{\text{high}}=\lambda_{h1}\cdot coverage\_area+\lambda_{h2}\cdot bandwidth+ \lambda_{h3}\cdot(1/latency) \tag{8}\]  
where \(\lambda_{h1}\), \(\lambda_{h2}\), and \(\lambda_{h3}\) are weight coefficients used to adjust the influence of coverage area, bandwidth, and latency of device categories on the reward function. The state transition for high-level decisions is as follows:  
\[T(S_{\text{high}},A_{\text{high}})=P(S^{\prime}_{\text{high}}\Big{|}S_{\text{ high}},A_{\text{high}}) \tag{9}\]  
where \(T(S_{\text{high}},A_{\text{high}})\) represents the state transition probability after the device category selects action \(A_{\text{high}}\) from state \(S_{\text{high}}\).