## 3 DRL-Based Solution
### Markov Decision Process  
In this subsection, we investigate an RL framework to solve our optimization problem described in Equation (6), where the agent learns to select the optimal relay to obtain the largest cumulative reward. The problem of RL is formulized as the incompletely known MDP. On the basis of the Markov property, the next state of a system is only related to the current state and the current selected action, and it is independent of earlier states.  
An MDP always consists of a four-tuple, namely, \((\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})\), where \(\mathcal{S}\) denotes the state space, \(\mathcal{A}\) represents the action space, \(\mathcal{R}\) is the immediate reward function, and \(\mathcal{P}\) is the transition probability. It has to be considered that statistical knowledge over the model-free environment is not available. In our problem, the state transition probability \(\mathcal{P}\) was unknown. Thus, the detailed physical meaning of \((\mathcal{S},\mathcal{A},\mathcal{R})\) is illustrated.  
* **State Space**\(\mathcal{S}\): The current state space of the whole system at the \(k\)-th TS is defined as \(\mathbf{S}_{k}=\left[\mathbf{I}_{k-1},\mathbf{h}_{k},\mathbf{g}_{k}\right]\), which consists both the previous selected relay node and the current CSI. \(\mathbf{I}_{k}\) represents the selected relay in the \(k\)-th TS by one-hot code.
* **Action Space**\(\mathcal{A}\): In our system model, the agent, as the decision maker, makes an optimal decision on the relay selection. The current action space of the whole system at \(k\)-th TS is defined as \(\mathbf{A}_{k}=\mathbf{I}_{k}\).
* **Immediate reward function**\(\mathcal{R}\): After taking action \(\mathbf{A}_{k}\) in state \(\mathbf{S}_{k}\), the system will transfer to the next state \(\mathbf{S}_{k+1}\); meanwhile, the agent will obtain an immediate reward \(R_{k}\). In our optimization problem, the immediate reward was defined as the current capacity defined in Equation (3), i.e., \(R_{k}=\mathbb{C}_{k}\).  
In RL tasks, an agent learns an optimal strategy to maximize the expected discounted future cumulative reward in Equation (5). In order to measure the quality of any arbitrary strategy \(\pi\), the value function \(Q^{\pi}(s,a)\) is introduced, which is defined to be the expected discounted cumulative reward in the future as the long-term reward by executing an action \(a\) in an environmental state \(s\), i.e., \(Q^{\pi}(s,a)=\mathbb{E}_{\pi}[G_{k}|s=\mathbf{S}_{k},a=\mathbf{A}_{k}]\). The value function can be expressed in a recursive form, as shown in Equation (7), which is always called the Bellman equation.  
\[\begin{array}{ll}Q^{\pi}(s,a)&=\mathbb{E}_{\pi}[r_{k}+\gamma G_{k+1}|\;s= \mathbf{S}_{k},a=\mathbf{A}_{k}]\\ &=\mathbb{E}_{\pi}[r_{k}+\gamma Q^{\pi}(\mathbf{S}_{k+1},\mathbf{A}_{k+1})|\;s= \mathbf{S}_{k},a=\mathbf{A}_{k}]\end{array} \tag{7}\]  
So far, finding an optimal policy in RL tasks means to solve the above Bellman equation. To achieve this goal, we just transferred our optimization objective in Equation (7). Our DRL-based algorithms to solve this equation is presented in the following subsection.  
It needs to be mentioned that the markers \((\mathcal{S},\mathcal{A},\mathcal{R})\), only used to explain MDP, indicate the whole sets of state space, action space and reward space, while the marker \(s\) (or \(a\); \(r\)) denotes an arbitrary element belonging to the corresponding set. The markers \((\mathbf{S}_{k},\mathbf{A}_{k},R_{k})\) stand for the instantaneous element in the \(k\)-th time slot in the proposed DQN-RS algorithm.