# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 3 RL Based Relay Selection Method for UWA Cooperative Networks
### RL Based Relay Selection for UWA Cooperative Communication  
The objective of the proposed RL based relay selection for UWA cooperative communication is to select the potential relay node with high mutual information and low access delay for the improvement of long-term network throughput. A framework of the RL based UWA cooperative communication is shown in Figure 4, where the environment is the UWA channel, the agent is the source node. The specific design of the states, actions and rewards are as follows.  
**States:** The state for reinforcement learning is combination of received feedback SNR and system mutual information:  
\[\mathbf{D}^{(t)}=[\gamma_{\text{SD}}^{(t-1)},\gamma_{r_{i}D}^{(t-1)},I^{(t-1)}] \tag{13}\]  
where \(\gamma_{\text{SD}}^{(t-1)}\) and \(\gamma_{r_{i}D}^{(t-1)}\) represent the channel states of \(S\)-\(D\) and \(r_{i}\)-\(D\) link in time round \(t-1\), respectively. \(\gamma_{\text{SD}}^{(t)}\) and \(\gamma_{r_{i}D}^{(t)}\) are defined in Equations (2) and (3), and the channel gain can be calculated as Section 2. \(I^{(t-1)}\) represents the whole system state at times round \(t-1\) in mutual information. It should be noted that when the agent performs an action at time \(t\) according to the state \(s^{(t)}\), the SNR and mutual information are actually the feedback at time \(t-1\). UWA system has to select the relay with the available outdated CSI.  
**Actions:** The action is to select one of the \(N\) nodes as a relay  
\[\mathbf{A}=\{a^{(t)}\} \tag{14}\]  
where \(\{a^{(t)}\}\in\mathbf{A}=\{1,2,\ldots,i,\ldots,N\}\). \(a^{(k)}=i\) indicates that node \(i\) is selected as a relay for cooperation in time round \(t\).  
**Rewards:** According to the analysis in Sections 1.2 and 2.3, We define the reward as a function of system mutual information and total access delay when selecting a relay  
\[r^{(t)}=\beta_{1}\times\ln^{(I^{(t)})}-\beta_{2}\times T_{Dr_{i}D}-\rho\times u \tag{15}\]  
where \(\beta_{1}\) is the proportional coefficients of mutual information, and concavity view of logarithmic function can well capture the system utility for data analysis. \(\beta_{2}\) is the proportional coefficients of access delay, and \(\rho\) is the scale coefficient of energy consumption factor \(u\).  
Figure 4: Framework of RL based UWA cooperative communication.  
In the UWA cooperative network, when the selected relay node has the higher mutual information, the system will achieve higher rewards and throughput. The cumulative reward of the system with RL can be expressed as  
\[r^{(0)}+\zeta r^{(1)}+\zeta r^{(2)}+\cdots=\sum_{t=0}^{M}\zeta r^{(t)} \tag{16}\]  
where \(r^{(t)}\) is the immediate reward of \(t\)th round, \(\zeta\) is the discount factor and \(M\) is the total number of iterations of RL. The smaller the \(\zeta\) value is, the more immediate reward is considered. The greater the \(\zeta\) value is, the more future reward is considered. The objective of RL based dynamic cooperation is to select the optimal relay node to reach the maximum amount of data during the long-term network deployment.  
**State-action Q function:** The Q-table is used in the RL algorithm to store the maximum future expectation reward that can be obtained by taking different actions in each state. The agent will use the \(\varepsilon\)-greedy algorithm to select the relay node with the maximum Q-value for cooperative communicate at the current moment. The Q-value update function for iteration is  
\[Q\Big{(}s^{(t)},a^{(t)}\Big{)}\leftarrow(1-\sigma)Q\Big{(}s^{(t)},a^{(t)} \Big{)}+\sigma[r^{(t)}+\zeta V(s^{(t+1)})] \tag{17}\]  
\[V(s^{(t)})=\max_{a\in\{1,2,\cdots,N\}}Q(s^{(t)},a^{(t)}) \tag{18}\]  
where \(\sigma\in(0,1]\) is the learning rate, which represents the speed of updating Q-value. V-table stores the maximum Q-value in each state.