# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 1 Introduction
### Related Work  
Basic UWA communication networks establish two-way acoustic links between underwater devices such as various sensors and autonomous underwater vehicles. The acoustic network is then connected to a surface station which can be further connected to offshore data center. The network design principles for each network layer have been discussed in [9; 10]. The authors explore potential applications and research directions for underwater sensor networking [11], such as monitoring, navigation and localization [12]. In the early-stage research, the design of UWA network is commonly performed in a layered structure, typically including the physical layer, the data link layer and the network layer [9; 10; 11]. For higher layer protocol design and network performance evaluation, the physical layer parameters are usually assumed to be ideal, even though the errors and special features of UWA channel will affect network performance. In recent years, the imperfect feature in physical layer has drawn more and more attention for practical applications. For example, in the localization scenario, how the error sources influence the estimated localization result has been analyzed in [13], where the error sources include the length of baseline, the error in ranging, etc. In UWA adaptive communication, how the imperfect channel state information affects the performance of multi-carrier adaptive modulation has been analyzed by experimental data [14]. In this paper, the proposed cooperative relay selection scheme will model the imperfect outdated channel state information to combine with the reinforcement learning process, and will be evaluated by both simulation and experiment. The UWA cooperative communication has the advantage that it is easily to be integrated with the existing system including both static [15] and mobile networks [16]. For integration with data collection network, [PERSON] et al. [17] proposed selective relay cooperation protocol and dynamic node cooperative protocol for a practical underwater data collection network, where UWA sensor nodes can cooperate with each other without adding additional relay nodes. Several lake experiments with full protocol implementation have demonstrated that the proposed protocols can significantly improve the network performance. For dynamic UWA network with joined new nodes, [PERSON] et al. [18] proposed a network access mechanism, which allows nodes that have joined the network to help new nodes access the network through multi-hop relay forwarding. In the relay path determination phase, they presented a relay path selection algorithm based on lifetime and energy efficiency. This algorithm can significantly extend the network lifetime and improve energy efficiency. For integration with physical layer modulation scheme, [PERSON] et al. [19] investigated both optimal relay selection and power loading issues for UWA orthogonal frequency division multiplexing (OFDM) cooperative transmission. They considered amplify-and-forward (AF) relaying with perfect channel state information (CSI) in the frequency selective Rician fading channel. The optimal relay selection and power allocation problems are solved in two phase to minimize the bit error rate (BER) and maximize the system capacity. For relay cooperation in linear UWA networks, [PERSON] et al. [20] researched the relay deployment problem in two aspects: which conditions a relay should be deployed and where to deploy it for the optimal performance. A closed-form expression for the open distance and optimal placing position have been derived, and simulation results demonstrate that properly introducing a relay can reduce the network energy consumption almost without increasing the end-to-end delay. All the above UWA cooperation protocols assume theoretical channel model or perfect channel state information for analysis. The intelligent algorithms have the potential to learn optimum strategy for performance improvement in complex UWA channels.