## 3 Federated Learning for Space-Air-Ground Integrated Networks
### Device Selection Optimization Strategy Based on Hierarchical Reinforcement Learning
#### 3.4.1 Hierarchical Reinforcement Learning Strategy
Based on the current resource state and requirements of device categories, the device category selection policy is updated using Q-learning as follows:  
\[Q_{\text{high}}(S_{\text{high}},A_{\text{high}})\gets Q_{\text{high}}(S_{ \text{high}},A_{\text{high}})+\alpha\Big{[}R_{\text{high}}+\gamma\text{max}_{A ^{\prime}_{\text{high}}}Q_{\text{high}}(S^{\prime}_{\text{high}},A^{\prime}_{ \text{high}})-Q_{\text{high}}(S_{\text{high}},A_{\text{high}})\Big{]} \tag{10}\]  
where \(Q_{\text{high}}(S_{\text{high}},A_{\text{high}})\) represents the Q-value of device category selection, i.e., the quality of selecting device category \(A_{i}\) in state \(S_{i}\), \(\alpha\) is the learning rate, used to control the speed of information updates during the learning process, \(\gamma\) is the discount factor, used to balance the relationship between immediate rewards and future rewards, and \(R_{\text{high}}\) is the high-level decision reward function, representing the priority of device categories. Local layer decisions are responsible for selecting specific devices within the chosen device category based on factors such as computational capability, bandwidth, data volume, and network status for participation in aggregation. The local layer's state space includes the computational capability, bandwidth, and local training accuracy of devices. Local layer state space \(S_{\text{low}}\) is represented as follows:  
\[S_{\text{low}}=\{C_{i},B_{i},n_{i},connection\_status_{i}\} \tag{11}\]  
where \(C_{i}\) represents the computational capability of device \(i\), \(B_{i}\) represents the bandwidth of device \(i\), \(n_{i}\) represents the data volume on device \(i\), and \(connection\_status\) represents the connection status of the device with edge nodes or satellites. Local layer action space \(A_{\text{low}}\) is represented as follows:  
\[A_{\text{low}}=\{device_{1},device_{2},\dots,device_{n}\} \tag{12}\]  
where \(A_{\text{low}}\) represents the set of specific devices selected within the chosen device category, and \(device_{i}\) represents a specific selected device. Local layer reward function \(R_{\text{low}}\) is represented as follows:  
\[R_{\text{low}}=\lambda_{l1}\cdot\text{accuracy}(\theta_{i})+\lambda_{l2}\cdot \frac{B_{i}}{C_{i}}+\lambda_{l3}\cdot privacy\_loss_{i} \tag{13}\]  
where accuracy\((\theta_{i})\) represents the accuracy of the locally trained model of device \(i\), \(\frac{B_{i}}{C_{i}}\) represents the ratio of bandwidth to computational capability of device \(i\), and \(privacy\_loss_{i}\) represents the privacy leakage risk of device \(i\). The state transition for low-level decisions is represented as follows:  
\[T(S_{\text{low}},A_{\text{low}})=P(S^{\prime}_{\text{low}}\big{|}S_{\text{low} },A_{\text{low}}) \tag{14}\]  
where \(T(S_{\text{low}},A_{\text{low}})\) represents the state transition probability after the device selects action \(A_{\text{low}}\) from state \(S_{\text{low}}\). \(S^{\prime}_{\text{low}}\) represents the new state to which the system transitions from the current low-level state \(S_{\text{low}}\) after executing action \(A_{\text{low}}\). Similarly to high-level decisions, the device selection policy for low-level decisions is updated using Q-learning as follows:  
\[Q_{\text{low}}(S_{\text{low}},A_{\text{low}})\gets Q_{\text{low}}(S_{ \text{low}},A_{\text{low}})+\alpha\Big{[}R_{\text{low}}+\gamma\text{max}_{A^{ \prime}_{\text{low}}}Q_{\text{low}}(S^{\prime}_{\text{low}},A^{\prime}_{\text{ low}})-Q_{\text{low}}(S_{\text{low}},A_{\text{low}})\Big{]} \tag{15}\]  
where \(Q_{\text{low}}(S_{\text{low}},A_{\text{low}})\) is the \(Q\)-value for low-level decisions, i.e., the current value estimate for the state-action pair. By combining high-level and low-level decisions, edge nodes can select device categories at the global layer and make detailed selections based on device performance and task requirements at the local layer, thereby optimizing the device selection strategy and adapting to different devices and network conditions.