# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 2 System Model  
### UWA Cooperative Communication System Model  
In this paper, we consider an UWA sensors network as shown in Figure 3. The sensor nodes are deployed underwater for sensing and transmitting important data to the sea-surface destination node. The sensor nodes are equipped with UWA modem for data transmission. As the sensor nodes are equipped with power limited battery, the transmission power can not be too high for the consideration of network lifetime. At the same time, the attenuation of UWA channel is severe and the loss of connectivity usually happens. On this account, the relay nodes which are randomly distributed between source and destination can help to forward the collected data to improve transmission quality of service of whole system. In this paper, we consider an application scenario where a source node \(S\) sends data to a destination node \(D\) and selects an optimal relay node \(r_{i}\) for cooperation, as shown in Figure 3.  
In this paper, time division multiple access (TDMA) protocol is employed to avoid interference among the nodes. When the relay node forwards the information, it consumes time or frequency resources, which will downgrade the spectrum efficiency. For example, when the system selects \(M\) relay nodes for cooperation, only one efficient data packet has been transmitted in \(M+1\) time slots. The idea of \"opportunistic relay\" [41] has proved that the \"best relay\" cooperative strategy can also achieve full set gain compared to the strategy of selecting multiple relays. The best relay strategy greatly reduces the system cooperative overhead and still can realize relatively high throughput. Therefore in this paper, the data is sent in time round, and one best relay is selected in one round. In the initialization round, the source node broadcasts the data and one random relay node forwards the data. In the normal round of data transmission, node \(S\) sends the data in the first time slot, and the selected best relay forwards the data in the second time slot. In this way, the destination node only needs to select and notify an optimal underwater relay node without calculating and feeding back channel information of all relay nodes, which greatly scale down the complexity of the system. The relay modes for cooperative communication are usually divided into amplify-and-forward (AF) and decode-and-forward (DF) according to how the relay nodes process the overheard received signals. The AF mode amplifies and forwards the received signal directly to the destination. While the DF mode decodes the received signal, re-encodes the signal and then forwards it to destination [42; 43]. The AF is simple to implement, but the noise will also been amplified. The DF has no such drawback due to the decoding and encoding operations at the relay node. Especially in UWA channel, the sources of noise are complex, such as the wind waves noise, turbulence noise, shipping activity noise, thermal noise, etc. The noise level is high, which is usually several tens of dB. When AF  
Figure 3: An UWA cooperative network system. is employed, it will amplify the noise when amplify the whole received signal. Previous studies in [17; 44] have shown how DF can efficiently work in UWA communication and network with experimental evaluation. So different from the previous UWA QL based cooperation [30; 31; 32], the research focus of this paper is on the scenario that the relay nodes employ the DF protocol for cooperation. For DF system, the mutual information of the entire system when selecting relay \(i\) at time round \(t\) can be expressed as  
\[I^{(t)}=\frac{1}{2}\text{log}_{2}\Big{(}1+\gamma_{SD}{}^{(t)}+\gamma_{r_{i}D}{ }^{(t)}\Big{)} \tag{1}\]  
where \(\gamma_{SD}\) and \(\gamma_{r_{i}D}\) are the received SNRs to measure the channel quality.