## 4 Intelligent Weather-Conscious AMC Scheme for Global Satellite-to-Ground Communications
### DRL-Based Decision Model
\[\mathcal{L}^{\mathcal{CLIP}}(\theta) =\mathbb{E}\bigg{[}\sum_{t=0}^{T}\left[\min\big{(}r(\theta)\hat{A }^{\pi_{\xi}},\text{clip}(r(\theta),1-\epsilon,1+\epsilon)\hat{A}^{\pi_{\xi}} \big{)}\right] \tag{16}\]  
where \(r(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\xi}}(a_{t}|s_{t})}\) is the ratio between the new policy and the old policy, \(\epsilon\) is a hyperparameter that denotes the tolerance for the deviation level, and \(\text{clip}(r(\theta),1-\epsilon,1+\epsilon)\) modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving \(r(\cdot)\) outside of the interval \([1-\epsilon,1+\epsilon]\). Therefore, we can formulate the objective function of the actor network as  
\[\mathcal{L}_{a}(\theta)=\mathcal{L}^{CLIP}(\theta)+\zeta\mathbb{E}_{t}[ \mathcal{H}(\pi_{\theta_{a}}(a_{t}|s_{t}))] \tag{17}\]  
where \(\mathcal{H}(\pi_{\theta_{a}}(a_{t}|s_{t}))\) is an entropy bonus that encourages exploration and \(\zeta\) is a balancing hyperparameter. We summarize the training procedure of purposed intelligent weather-conscious AMC in Algorithm 1. Each expectation term is evaluated by the averaged results of a batch of samples. ```
1 Randomly initialize parameters of the estimation model as \(\chi\);
2 Randomly initialize parameters of the critic and the actors as \(\phi\) and \(\theta\);
3 Initialize estimation model learning rate \(\alpha_{E}\), batch size \(b_{E}\), max epoch \(E_{max}\);
4 Initialize decision model learning rate \(\alpha_{D}\), batch size \(b_{D}\), sample reuse time \(N\), and initial state \(s_{0}\);
5 Initialize trajectory buffer \(\mathbf{M}\) with size \(||\mathbf{M}||\); // Training of estimation model
6 forepoch \(E:=1\) to \(E_{max}\)do
7 Collect a batch \(b_{E}\) of weather and location information \((lat_{t},lont_{t},x_{t},y_{t},z_{t},w_{t})\);
8 Estimate satellite-to-ground channel state as \(\hat{s_{t}}\);
9 Update estimation model: \(\chi\leftarrow\chi-\alpha_{E}\
abla_{\chi}\mathcal{L}(\chi)\); // Training of decision model
10 Current state \(s_{t}\gets s_{0}\);
11 forstep \(S:=1\) to \(S_{max}\)do // Collecting trajectory
12 while\(\mathbf{M}\) is not filleddo
13 Collect weather and location information \((lat_{t},lont_{t},x_{t},y_{t},z_{t},w_{t})\);
14 Estimate satellite-to-ground channel state as \(\hat{s_{t}}\);
15 Sample action \(a_{t}\sim\pi_{\theta}(a_{t}|\hat{s_{t}})\);
16 Execute \(a_{t}\) and observe reward \(r_{t}\), the next state \(s_{t+1}\);
17 Append \((s_{t},a_{t},r_{t},s_{t+1})\) into \(\mathbf{M}\);
18 if\(s_{t+1}\) is the terminate statethen
19\(s_{t}\leftarrow\) reinitialized state \(s_{0}\);
20
21 else
22\(s_{t}\gets s_{t+1}\);
23\(S\gets S+1\); // Updating Network
24 Compute state value for states in \(\mathbf{M}\) with (12);
25 Compute advantage for states in \(\mathbf{M}\) with (15);
26 forepoch \(e:=1\) to \([N\times(||\mathbf{M}||/b_{D})]\)do
27 Sample \(B\) samples from \(\mathbf{M}\);
28 Compute \(\mathcal{L}_{c}(\phi)\) and \(\mathcal{L}_{a}(\theta)\) with these samples;
29 Update critic: \(\phi\leftarrow\phi-\alpha_{D}\
abla_{\phi}\mathcal{L}_{c}(\phi)\);
30 Update actor: \(\theta\leftarrow\theta-\alpha_{D}\
abla_{\theta}\mathcal{L}_{a}(\theta)\);
31 Clear memories in \(\mathbf{M}\). ```  
**Algorithm 1****Training of the Intelligent Weather-Conscious AMC. **