## 4 Intelligent Weather-Conscious AMC Scheme for Global Satellite-to-Ground Communications
### DRL-Based Decision Model  
As MIMO antennas are often used in satellite-to-ground channel communication, the conventional look-up table method cannot cope with it; moreover, in order to be compatible with the gap between different devices and to adapt to the dynamically changing characteristics of the satellite-to-ground channel, we adopt a DRL-based decision model for selecting the optimal coding scheme for each moment. For simplicity of presentation, we use \(s_{t},\ a_{t},\ r_{t}\) to represent state, action, reward, respectively, at moment \(t\). The state of the decision model is the satellite-to-ground channel state estimated by the estimation model, action is defined as \(a_{t}=\{DR_{t}\}\), and reward is represented as throughput \(r_{t}=\{DR_{t}[1-PER(DR_{t},SNR_{t})]\}\). We suppose a trajectory exists in the MDP problem and that the trajectory describes the interaction process between the environment and the DRL agent. Therefore, we can obtain the rewards of each time in the trajectory, and the real cumulative reward at state \(s_{t}\) is  
\[V^{\prime\prime\pi}(s_{t})=\sum_{\tau=t}^{T(\pi)}\gamma^{\tau-t}r_{\tau} \tag{12}\]  
where discount factor \(\gamma\in[0,1]\) is a hyperparameter that balances short-term and long-term returns. There is less accurate data available for learning in satellite-to-ground channel communication, so we want to use historical data fully. Therefore, the Proximal policy optimization (PPO) algorithm [40], based on the actor-critic algorithm [41], is used as the gradient update algorithm. Actor and critic are the two neural networks in the agent. The actor-network is responsible for making decisions and selecting the best action \(a_{t}\), while the critic network is responsible for scoring and evaluating the choice of the actor. We use this sampled value as the expected cumulative reward to train the critic network. The loss function is defined as  
\[\mathcal{L}_{c}(\phi)=||V^{\pi}_{\phi}(s_{t})-V^{\prime\pi}(s_{t})||_{2} \tag{13}\]  
where \(\phi\) is a parameter of the critic network. The environments of satellite-to-ground channels are similar, under similar environments and similar SNRs, and the choices are likely to be the same. To fully use the information from other trajectories, we introduced importance sampling into gradient propagation. \[\underset{\theta}{\text{maximize}}\quad\hat{\mathbb{E}}\bigg{[}\frac{\pi_{ \theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}\left(a_{t}|s_{t}\right)}\hat{ A}\bigg{]} \tag{14}\]  
where \(\pi_{\theta}(a_{t}|s_{t})\) is the current policy and \(\pi_{\theta_{old}}(a_{t}|s_{t})\) is the old policy for collecting trajectory, \(\hat{A}_{t}\) is the estimation of advantage function, which measures how much a specific action \(a_{t}\) is better than the average actions at state \(s_{t}\). To reduce the bias of advantage function, we employ an exponentially weighted method to obtain the Generalized Advantage Estimation (GAE) [42]:  
\[\hat{A}_{t}=\sum_{\tau=t}^{T(\pi)}(\gamma\lambda)^{\tau-t}\Big{(}r_{t}+\gamma V ^{\pi}_{\phi}(s_{t+1})-V^{\prime\pi}(s_{t})\Big{)} \tag{15}\]  
where \(\lambda\in[0,1]\) is a hyperparameter. If \(t+1>T(\pi)\), we have \(V^{\pi}_{\phi}(s_{t+1})=0\). Referenced by the gradient descent, we obtain the first-order derivative solution, which is closer to the second-order derivative solution, by adding soft constraints. Due to excessive deviations in the trajectory, we adopted the method in [40] to avoid large gradient deviations.