# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 1 Introduction
### Related Work
Some research studies focus on UWA relay selection by exploiting machine learning algorithms. For example, [PERSON] et al. [21] modeledrelay selection as a contextual bandit problem, where the relay is selected based on one bit of contextual communication environment information rather than a large amounts of instantaneous or statistical CSI information. The proposed scheme has a stable performance in complex UWA environment and simplifies the relay selection process for efficient cooperative transmission. [PERSON] et al. [22] proposed a hierarchical adversarial multi-armed bandit learning framework to improve adaptive relay decision. The proposed framework exploits heuristic interactions between hierarchical frameworks to integrate reward estimation, information prediction, adaptive exploration and decision making within a holistic algorithm to maximize learning efficiency. It enables cooperative communication networks with greater stability and lower communication costs in complex and varying underwater environments. The above two studies have shown the potential of artificial intelligence learning for underwater systems. Different from bandits learning problem in [21; 22], reinforcement learning (RL) provides a general and powerful computational framework for sequential decision making problem. RL can satisfy the optimization in actual sequential time varying UWA channels, where the channel usually cannot be modeled into a linear pattern. Reinforcement learning (RL) has been used to address the problem of maximizing rewards or achieving specific goals by learning strategies during the interaction of an agent with its environment. In UWA communication systems, some researchers have employed RL to solve adaptive problems, such as adaptive data and image transmission [23; 24], and adaptive routing [25; 26; 27]. In terms of relay selection, [PERSON] et al. [28] firstly proposed QL-based relay selection algorithm (QL-RSA) in wireless sensor networks. QL-RSA defines the relay nodes as the set of states and whether to select a new relay as the set of actions, which requires less feedback information and provides each source a self-learning capability. QL-RSA receives the feedback reward by learning AF cooperative environment in time-varying Rayleigh fading channels to maximize the network throughput. In wireless sensor networks, [PERSON] et al. [29] modeled the process of relay selection for cooperative communications as a Markov decision process (MDP) and proposed a deep reinforcement learning based relay selection scheme, where the channel state and relay nodes are defined as the state and actions of RL, respectively. The best relay is selected from multiple relay nodes for better performance of outage probability and system capacity. The above researches model channel theoretically and assume the feedback is perfect for wireless cooperative sensor networks [28; 29]. For implementation RL based relay selection in UWA sensor networks, [PERSON] et al. extended the scheme [29] to internet of underwater things (IoUT) network [30] with additional power adjust strategy to maximize the end-to-end signal to noise ratio (SNR) of the system. [PERSON] et al. [31] investigated reinforcement learning based joint relay selection and power allocation in energy harvesting UWA cooperative networks. In the proposed model, a joint state expression is presented to better reveal the relationship between learning and environment, and a reward function that consists of channel capacity and energy consumption is proposed for adjusting power strategy to balance instantaneous capacity and long-term quality of service (QoS). [PERSON] et al. [32] investigated the power allocation problem in energy harvesting full duplex UWA cooperative relay network by reinforcement learning. The relay adjusts transmission power by the battery level, harvested energy and CSI to maximize the cumulative rate of the UWA network.