## 4 Simulation Results
Figure 1: Normalized PDFs of experimental and synthesized data. (**a**) 10 September 2003 20:10–20:30. (**b**) 10 September 2003 00:30–00:50. (**c**) 12 September 2003 00:30–00:50. (**d**) 13 September 2003 23:30–23:50. (**e**) 16 September 2003 20:10–20:30. Finally, the stochastic policy that will determine the agents' actions was investigated. In Figure 3, three different probability distributions, i.e., truncated Normal \((\mu,\sigma,0,P_{s})\), truncated Weibull \((k,\lambda,0,P_{s})\), and truncated Exponential \((\lambda,0,P_{s})\) were tested and evaluated. Normal and Weibull are two-parameter distributions while Exponential is one-parameter. It is observed that the two-parameter distributions are the best choice and that truncated Normal performs slightly better. The finalized policy structure consists of a four-layer DNN, as reported in Table 3. In addition, the truncated Normal (TN) distribution is selected as a stochastic power policy for the five optical channels. The output nodes are used as parameters of the five TN distributions--means and standard deviations. The optimization algorithm involved in the training is RMSProp with step learning rate scheduling. Specifically, 5000 episodes with a duration of \(T=100\) time-steps  
Figure 3: Performance results for five optical satellite downlinks using three different policy distributions: truncated Normal \((\mu,\sigma,0,P_{s})\), truncated Weibull \((k,\lambda,0,P_{s})\), and truncated Exponential \((\lambda,0,P_{s})\). \(P_{s}=1.0\,W,\ P_{av}=2.5\,W\). Figure 2: The loss function and predicted average capacity in terms of the (**a**) number of hidden layers (200 nodes/layer) and (**b**) number of neurons (2 hidden layers). \(P_{s}=1.0\,W,\ P_{av}=3.0\,W\). \begin{table}
\begin{tabular}{c c c} \hline \hline
**Channel** & **\# Nodes** & **Activation** \\ \hline Input & 5 & Linear \\ Hidden \#1 & 200 & ReLu \\ Hidden \#2 & 100 & ReLu \\ Output & 10 & Softplus \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Policy DNN Architecture. are considered to have frequent weight updates. The initial learning rate is set to 0.001 and the discount factor to 0.5 because fading negatively affects the impact of an agent's actions on his future expected returns [20]. Also, given the total power constraint, the agents' behaviors affect others' rewards due to the unpredictability of their neighbors' actions. Higher \(\gamma\) is also undesirable because it decelerates the response to channel fluctuations [20]. The proposed algorithm was implemented in PyTorch. In Figure 4, the predicted average capacity (a) and the constraint function (b) \(\sum_{t=0}^{L-1}\mathrm{E}[P_{t}(h_{t})]-P_{av}\) are depicted for \(P_{av}=2.0\) W. In Figure 5, the average channel powers are illustrated for the proposed, the supervised, the DQN, the Equal Power, and the Random Power algorithms for \(P_{av}=2.0\) W. Likewise, in Figure 6, the predicted average capacity (a) and the constraint function (b) are depicted for \(P_{av}=3.0\) W, and in Figure 7, the average channel powers are illustrated for \(P_{av}=3.0\) W. Finally, in Figure 8, the predicted average capacity (a) and the constraint function (b) are depicted for \(P_{av}=4.0\) W, and in Figure 9, the average channel powers are illustrated for \(P_{av}=4.0\) W. The DQN is a model-free, off-policy method that tries to predict the Q-values (expected future rewards) for every state-action pair using the \(\epsilon\)-greedy policy that alternates between exploration and exploitation [20; 21; 22]. DQN employs discretized actions: \(\{0,\overline{P}\,/\,A-1,2\overline{P}\,/\,A-1,\ldots,\overline{P}\}\) where A is the number of actions [20; 21; 22]. Here, A = 21 was chosen. Figure 4: Learning episodes of five different PA algorithms for five optical channels: (**a**) the average system capacity and (**b**) the constraint function. \(P_{s}=1.0\) W, \(P_{av}=2.0\) W.  
Figure 5: The average channel powers of five different PA algorithms for five optical channels during 5000 episodes.