## 3 DRL-Based Solution
### The DQN-RS Algorithm  
The simplest RL tasks, where the state and action spaces are small enough for the state action values, i.e., the \(Q\)-values to be represented as tables, can be solved by a regular \(Q\)-learning method. In this paper, we try to solve a relay selection problem in a cooperative DF-FSO system with continuous state and discrete action space. Under this circumstance, it is impractical to maintain all \(Q\)-values. Hence, we use an approximate solution method that can be applied effectively to complex problems. The DQN algorithm is a typical algorithm based on the value-based approximation method, which uses some neural networks to approximate the action value function and can learn finite discrete values from a high-dimensional state space and output action with the highest action value. The dueling DQN improves the algorithm performance by optimizing the neural network structure [23]. In this way, the \(Q\)-values are more robust and can be estimated with lower variance. The architecture of our proposed dueling DQN-based DQN-RS algorithm is shown in Figure 3.  
As can be seen in Figure 3, our DQN-RS framework consists of two neural networks. One is the online network, the other is the target network, and both share the same structure. Different from the conventional DQN which directly estimates the \(\underline{Q}\)-values by one stream, the dueling architecture consists of two streams of fully connected layers that represent state value \(V(s)\) and action advantage \(A(s,a)\), respectively, while sharing common feature layers. Then, the two streams are combined via a special aggregating layer to produce the estimated \(Q\)-values. After estimating these \(Q\)-values, the agent can choose the action with the highest \(Q\) value, i.e., select the current relay. As the action is completed, the environment changes in the next time slot, which corresponds to the change of state in the new slot. The state can be observed by the agent as the input to the DQN-RS networks. Our proposed algorithm proceeds in an iterative manner according to the process described above. As shown in Figure 3, a replay buffer storing experience data is used to train the two networks. The motivation for using this dueling DQN structure is to solve the observability problem in conventional DQN. For some states, such as the channel information in our system model, their next states will not be affected by the taken actions at the current time step. Hence, it is desirable to estimate \(V(s)\) independently of \(A(s,a)\). We define \(V(s)\) and \(A(s,a)\) in Equations (8) and (9)  
\[V(s) =V(s|\;\omega,\theta) \tag{8}\] \[A(s,a) =A(s,a|\;\omega,\zeta) \tag{9}\]  
where \(\omega\) denotes the parameters of the convolutional layers, \(\theta\) and \(\zeta\) are the parameters of the two streams of fully connected layers. Then, state action value function or \(Q\)-value function in the dueling DQN algorithm can be expressed as:  
\[ \begin{array}{ll}Q(s,a)&=Q(s,a|\;\omega,\zeta,\theta)\\ &=V(s|\;\omega,\theta)+A(s,a|\;\omega,\zeta)\end{array}  \tag{10}\]  
In our system, in order to improve the performance of our algorithm, we calculate the \(Q\)-value by:  
\[ \begin{array}{ll}Q(s,a)&=Q(s,a|\;\omega,\zeta,\theta)\\ &=V(s|\;\omega,\theta)+\left(A(s,a|\;\omega,\zeta)-\frac{1}{|A|}\sum\limits_{a^ {\prime}}A(s,a^{\prime}|\;\omega,\zeta)\right)\end{array}  \tag{11}\]  
where the number of elements is indicated by \(|\bullet|\). Figure 3: Proposed DQN-RS algorithm framework. In our proposed DQN-RS algorithm, we take the state \(\mathbf{S}_{k}=\left[\mathbf{I}_{k-1},\mathbf{h}_{k},\mathbf{g}_{k}\right]\) introduced in Section 3.2 as the input. We denote the output action in the DQN-RS algorithm as \(\mathbf{A}_{k}=\mathbf{I}_{k}\). These actions are selected by the \(\varepsilon\)-ready method.