# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 1 Introduction
### Motivations and Contributions  
Principally, the signal attenuation from the source node to the destination node is distance-dependent in both terrestrial electromagnetic and underwater acoustic networks. Correspondingly, the SNR criteria based optimal relay usually has a shorter distance to the source or destination node than other potential relays. However, the SNR based criteria may not be effective without taking specific characteristics of UWA channel into consideration. Low UWA speed leads to long propagation delay of UWA signals [38; 39]. Considering the long propagation delay together with the distance independent characteristics of UWA channel, we will discuss how these phenomena affect the SNR based cooperative criteria. * **Shadow zones:** In radio communications, the path of the radio wave is modeled as straight line. In large scale underwater communication networks, the speed of sound in seawater is not a constant, so the actual UWA propagation path is a curve where the sound ray bends to the lower velocity layer of the water and leads to nonlinear sound propagation [40]. As shown in Figure 1, the nonlinear UWA propagation will form communication shadow zones and convergence zones. The sound rays will converge to convergence zones, and the node in shadow zones can only receive few sound rays. Assuming that potential relay node \(r_{1}\) and \(r_{2}\) are located among the source \(S\) and destination \(D\), and \(r_{1}\) is located in the convergence zone and \(r_{2}\) is located in the shadow zone. By SNR based cooperative criteria, \(r_{1}\) will be chosen as optimal relay. However, due to the nonlinear propagation of sound, the sound ray of \(r_{1}\) might transmit a longer way which consumes more time for one-trip data transmission. The channel capacity obtained from higher SNR may not compensate the longer propagation delay, and finally leading to lower throughput. * **Underwater obstacles:** Underwater random obstacles will interfere the signals between potential relay nodes and source/destination. For example, as shown in Figure 2, potential relay node \(r_{2}\) is close to the destination node but there are fish schools as obstacle, which results in worse channel condition than \(r_{1}\). While relay node \(r_{1}\) is far away but there are no obstacles. By SNR only cooperative criteria, \(r_{1}\) might be chosen as optimal relay, while it is far away from destination. In this case, there is no guarantee which relay node is better in the view of network throughput. So we should take both channel quality and propagation delay into consideration. * **Frequency selective fading:** UWA channel is a typically frequency selective fading channel with severe multi-path effect, where multi-path propagation can result from signal reflections from surfaces, bottoms and water objects or bending along the axis of the lowest sound speed. The signals from the source node propagate through different paths to the destination node may result in the signal elimination on specific frequency. The signal attenuation due to frequency-selective fading is independent of distance, and it only depends on the signal frequency and the multi-path delay difference between the multiple paths. For example, as shown in Figure 2, suppose that the channels between \(S\) to \(r_{3}\) and \(r_{2}\) to \(D\) are in frequency selective fading, and \(r_{1}\) will be chosen as relay by SNR based criteria. Nevertheless, with longer propagation delay, it is uncertain that \(r_{1}\) will have higher throughput than \(r_{3}\). Different from radio network solutions, if above typical phenomena of the UWA channel are not considered, the best relay selection may not be effective in practical underwater scenario. In this paper, we propose a RL based relay selection scheme for UWA networks that takes both channel quality and transmission delay into consideration, which is more  
Figure 1: Test results of nonlinear sound ray propagation in the Arctic region, (Urick [40]).