## 5 Simulation Results
### Results and Discussion  
We evaluate the average reward of constrained SAC with different users, which is depicted in Figure 4. Clearly, it converges in all four scenarios around 200 to 300 episodes, and its average reward increases as users grow in the ISTN. Since we initialize the Lagrangian multiplier as 1 and the rate constraint as 4. The start points of four lines are \(-8\), \(-12\), \(-16\) and \(-20\), respectively. Two things need to be noted. The first is that the fluctuation becomes greater when users grow. The reason for this is that when more users served in the network, the action space becomes larger, making it more difficult to design the precoders \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Value** & **Note** \\ \hline \(b\) & 0.158 & Average power of multi-path component \\ \hline \(\Omega\) & 1.29 & Average power of LoS component \\ \hline \(m\) & 19.4 & Nakagami-\(m\) parameter \\ \hline \(f\) & 12 GHz & Satellite frequency \\ \hline \(d\) & 1000 km & Distance between satellite and users \\ \hline \(K\) & 2, 3, 4, 5 & Number of users \\ \hline \(P_{t}\) & 40 W & Transmission power of satellite \\ \hline \(P_{c}\) & 5 W & Circuit power \\ \hline \(\Delta\) & 0.1 & Difference between last state and current state \\ \hline \(R^{\text{th}}\) & 4 bps/Hz & Rate constraint \\ \hline \(\rho\) & 0.98 & Discount factor \\ \hline \(\lambda\) & 1 & Lagrangian multiplier \\ \hline \(\mu_{\lambda}\) & \(10^{-3}\) & Learning rate of \(\lambda\) \\ \hline \(\mu_{\pi}\) & 0.2 & Temperature parameter for entropy \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. for each user. Another thing is that the gap between two lines shrinks as users increase. Given the transmission power of the satellite, the achievable rate of each user will slightly decrease when more users join the network, resulting in slowing the growth speed of the average reward. The energy efficiency performance of constrained SAC with different users is shown in Figure 5. We can see that it shares some similarities with Figure 4, such as the greater fluctuation and smaller gap with more users. The reasons are the same--more users enlarge the action space. Since energy efficiency is the reward without constraint, the start points of these four lines are all 0. Next, we verify whether the rate constraint is satisfied with the proposed algorithm. In Figure 6, we can see that with different users, constrained SAC meets all the demands. For clarity, we add four straight lines to indicate the rate constraints in four scenarios. The capacity generated by the proposed algorithm all exceed the constraints. With these three  
Figure 4: Average reward of constrained SAC with different users. Figure 5: Energy efficiency of constrained SAC with different users. results, the constrained SAC can maximize the energy efficiency under the rate constraint. It provides an alternative solution for constrained optimization problem. To evaluate the performance of constrained SAC (denoted as CSAC in the figures) with other benchmark algorithms, we choose the original SAC with no constraint and DDPG incorporating constraint into its reward function (denoted as DDPG in the figures) [44]. The existing model-based SCA optimization (denoted as SCA in the figures) is compared with these model-free DRL methods as well [15; 16]. \[r^{(t)}_{\text{DDPG}}=EE\Big{(}1-\eta^{(t)}\Big{)}, \tag{32}\]  
where \(\eta(t)\) is the penalty defined as  
\[\eta^{(t)}=\frac{1}{K}\sum_{i=1}^{K}\mathbb{I}\Big{(}R^{(t)}_{k}-R^{\text{th}} \Big{)}, \tag{33}\]  
in which \(\mathbb{I}(\cdot)\) is a indicator function given by  
\[\mathbb{I}(x)= \begin{cases}1&x<0\\ 0&x\geq 0\end{cases} . \tag{34}\]  
Whenever the rate is lower than the threshold, the reward of DDPG will be penalized. Therefore, to achieve the maximum reward, the rate will meet the requirement. In the three users scenario, since the original SAC is unconstrained, this means its reward is the energy efficiency.