## 3 Federated Learning for Space-Air-Ground Integrated Networks
### Device Selection Optimization Strategy Based on Hierarchical Reinforcement Learning
#### 3.4.3 Meta-Learning-Based Strategy Optimization  
In privacy-protected federated learning within space-air-ground integrated networks, traditional reinforcement learning (RL) for device selection optimization exhibits limitations such as slow convergence speed. Therefore, this paper proposes the use of meta-learning [34] to accelerate policy adjustment and improve both the adaptability and convergence speed of reinforcement learning across different network environments. Compared to general meta-learning methods, Model-Agnostic Meta-Learning (MAML) [35] offersadvantages including rapid adaptation to new tasks, improved sample efficiency, enhanced generalization capabilities, and reduced communication overhead. Let the parameter of the reinforcement learning policy function be \(\theta\). We aim to learn a good initialization \(\theta_{\text{meta}}\) through MAML so that when encountering a new network environment, the policy can quickly converge. In each device selection task \(T_{i}\), the device selection policy \(\pi_{\theta}\) is used to execute device selection. \(\pi_{\theta}\) is a mapping from state space to action space, defining which action \(A_{i}\) to choose given state \(S_{i}\), with the goal of maximizing returns by selecting the best action \(A_{i}\). The state information \(S_{i}\) of device \(i\) is represented as follows:  
\[S_{i}=(C_{i},B_{i},n_{i},connection\_status_{i},last\_update\_time_{i}) \tag{19}\]  
where \(connection\_status_{i}\) indicates the connection status of device \(i\) with edge nodes or satellites, indicating whether the connection is stable, and \(last\_update\_time_{i}\) represents the time when device \(i\) last uploaded a model update, reflecting the device's online time and connection stability. \(\pi_{\theta}\) is specifically represented as follows:  
\[\pi_{\theta}(S_{i})=\text{argmax}_{A_{i}}Q(S_{i},A_{i}) \tag{20}\]  
where \(\pi_{\theta}(S_{i})\) represents the probability or deterministic choice of policy \(\pi_{\theta}\) selecting action \(A_{i}\) in state \(S_{i}\), \(S_{i}\) represents the state information of the device, \(A_{i}\) represents the device selection action, and \(Q(S_{i},A_{i})\) is the Q-value function, representing the expected return after selecting action \(A_{i}\) in state \(S_{i}\). The task loss function \(L_{\text{task}_{i}}(\theta)\) represents the performance of policy \(\pi_{\theta}\) on task \(T_{i}\), defined as follows:  
\[L_{\text{task}_{i}}(\theta)=\mathbb{E}_{S_{i}\sim\mathcal{S}}[R(S_{i},A_{i})] \tag{21}\]  
where \(\mathbb{E}\) represents expectation, indicating the average return of device selection actions under the given policy \(\pi_{\theta}\), \(S_{i}\) represents the state (device category, bandwidth, computational capability, etc.) in task \(T_{i}\), \(A_{i}\) represents the action (device selection) in task \(T_{i}\), and \(R(S_{i},A_{i})\) is the reward function for task \(T_{i}\) (return of device selection). The goal of MAML is to minimize the loss across all tasks by updating \(\theta_{\text{meta}}\). The meta-loss function is defined as follows:  
\[L_{\text{meta}}(\theta_{\text{meta}})=\frac{1}{N}\sum_{i=1}^{N}L_{\text{task }_{i}}(\theta_{\text{meta}}) \tag{22}\]  
where \(L_{\text{task}_{i}}(\theta_{\text{meta}})\) is the loss for task \(T_{i}\), with device selection performed using the reinforcement learning policy \(\pi_{\theta}\), and \(\theta_{\text{meta}}\) is the initial policy parameter optimized by meta-learning. For each task \(T_{i}\), the device selection strategy parameters are updated through gradient descent:  
\[\theta_{\text{task}_{i}}=\theta_{\text{meta}}-\alpha\
abla_{\theta}L_{\text{ task}_{i}}(\theta) \tag{23}\]  
where \(\alpha\) is the step size (learning rate). The meta-parameter \(\theta_{\text{meta}}\) is updated as follows:  
\[\theta_{\text{meta}}=\theta_{\text{meta}}-\beta\
abla_{\theta_{\text{meta}} }L_{\text{meta}}(\theta_{\text{meta}}) \tag{24}\]  
where \(\beta\) is the meta-learning learning rate, adjusting the influence of task loss on the meta-learning process.