## 4 Simulation Results  
In this section, the numerical results of our proposed DQN-RS algorithm in cooperative DF-FSO systems are illustrated. The greedy relay selection is also considered as a comparison, where the relay with the best CSI is selected. The simulation parameters are presented in Table 1. The simulation results were obtained on the deep learning framework in TensorFlow 1.2.1 of Python 3.6. As mentioned above, the number of neurons in the input layer was equal to \(2N\), which was the dimension of the state space. Similarly, the number of neurons in the output layer was equal to the number of relay nodes \(N\), which coincided with the dimension of the action space. In addition, there were 200 neurons in the common feature layer, whose active function was the Relu function. There were also 200 neurons in the value layer and \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Number of Relay Nodes \(N\) & 4 \\ Number of Time Slots \(M\) & 10,100 \\ Switching Loss \(\tau\) & 0.2 \\ Photodetector Responsivity \(\eta\) & 0.9 \\ Fading parameters \((\alpha,\beta,A_{0},\rho)\) & 5.9776, 4.3980, 0.0032, 6.2552 \\ Normalized Power \(P/\sigma_{r},P/\sigma_{s}\) & 5 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation Parameters. the action advantage layer, whose active function was the linear function in our proposed DQN-RS algorithm. We trained the deep neural networks with a minibatch size of \(X=200\). The replay memory capacity \(U\) was equal to one-third of the total training steps. The discount factor \(\gamma\) was set to 0.9. First, the convergence performance of the mentioned DQN-RS algorithm is illustrated. We set both the learning rates to 0.001. The channel capacity versus episodes and the loss via training steps are presented in Figure 4a,b, respectively. As shown in Figure 4a, it could be divided into three parts as the episodes increased. In the first part, the agent was busy filling the replay buffer. The training process did not start until the buffer was full around the 1333 rd episode, where the second part began. During the second part, the agent learned to achieve a higher \(Q\)-value by selecting different relay nodes. In the last part, the parameter \(\varepsilon\) was set to 0, which means the agent had the ability of dealing with the relay selection issue. According to Figure 4b, the DQN-RS algorithm converged after 2000 steps. In addition, it can be seen in Figure 4b that the curve of the loss function presents periodic fluctuations, which resulted from the fact that a bad experience may be chosen from the replay memory during the training process. Figure 5 shows the difference of the proposed DQN-RS algorithm and the greedy algorithm; it depicts an example of relay selection in 10 TSs. As shown in Figure 5, the greedy algorithm (the green line) always choses the link with the largest channel condition, i.e., \(\mathrm{argmax}\Big{\{}\mathrm{min}\Big{\{}h_{k}^{N_{k}},g_{k}^{N_{k}}\Big{\}} \Big{\}}\). That is why the 2 nd relay node was selected in 2 nd TS, rather than the 4 th relay node. The greedy algorithm did not consider the switching loss. Moreover, frequent handovers would increase the time the system was unavailable, thereby reducing the channel capacity of the system, i.e., \(\mathbb{C}_{\mathrm{greedy}}=3.1439\mathrm{bps}/\mathrm{Hz}<\mathbb{C}_{ \mathrm{DQN-RS}}=3.5337\mathrm{bps}/\mathrm{Hz}\), even though there were other relay nodes with better CSI. Figure 4: Convergence performance of the proposed DQN-RS algorithm. (**a**) Channel capacity versus episodes, (**b**) loss function via training steps. Figures 6 and 7 show the average channel capacity versus different switching losses and number of relay nodes, respectively. On the basis of Figure 6, the average channel capacity decreased with a larger switching loss \(\tau\). This is evident because unavailable time will increase with the growth of the switching loss, when the system switches between the relay nodes.