# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 5 Discussion  
The proposed reinforcement learning UWA cooperative scheme comprehensively considers the physical layer channel characteristics to design the expression of RL for improvement of network throughput. At the same time, the integration of simulated annealing algorithm with reinforcement learning in the relay selection strategy and pre-training of RL can help to accelerate the learning process and improve system performance. The results and findings of this paper and the implications are discussed as follows. For the design of state, action and reward for RL based relay selection, the long propagation delay characteristic is considered in this paper. The joint reward expression is established with system mutual information, access delay and energy consumption. In research area of reinforcement learning based UWA relay selection, reference [30] firstly considered mutual information as reward of RL. Furthermore, with the consideration of energy consumption, reference [31] set channel capacity and energy consumption as reward, and reference [32] set channel gain and battery level as state, and channel capacity as reward. This paper additionally considers the long access delay characteristic of UWA channel for RL system design. The transmission rate has been improved compared to reference [30] as shown in Figures 11 and 14. As our proposed scheme jointly considers channel quality and access delay, the node with higher channel quality and lower access delay will be selected as relay, which will finally lead to optimal effective throughput. The consideration of long propagation delay also can be employed in related UWA network applications. With the consideration of dynamic exploration of RL system, simulated annealing algorithm is integrated to the relay selection strategy to better balance exploration and exploitation of RL. In the early stage, the SA-RL relay selection will fully explore the random actions. As shown in Figure 7, the reward is fluctuating and has small values during the first 800 rounds. With the number of learning iteration increasing, the temperate of SA is getting lower, so that the RL will explore random action with lower probability. Specifically, when the minimum \(\varepsilon\) is set to 0, the exploration rate will approach to 0 after the training turns to convergence. The convergence results will approach to ideal optimum as shown in the figures in Sections 4.1.3 and 4.2. In comparison with constant \(\varepsilon\)-greedy strategy in [30; 31; 32], their curves fluctuate more severely after convergence than our proposed dynamic exploration strategy. When constant \(\varepsilon\) is a small value, the constant \(\varepsilon\)-greedy strategy is easily trapped into local optimum in complex environments. As SA-RL can explore with higher temperature, it has the advantage of avoiding to be trapped in local optimum. The integration of simulated annealing algorithm with reinforcement learning in  
Figure 14: Data transmission rate, experimental data. the relay selection strategy can be adopted in various applications related to the underwater acoustic communication and network applications. As UWA environment is complex and there is no standard channel model for algorithm evaluation, experiment is a persuasive evaluation method. Reinforcement learning has the ability to learn from the interaction of environment and agent without any prior knowledge, so the Q-table can be initialized with all zeros, such as in [24; 25; 26; 27; 28; 29; 30; 31; 32]. Considering practical implementation, this paper has shown how the pre-training helps to improve the system performance. From the simulation results in Figures 9 and 10, it can be observed that the pre-training can accelerate the process of convergence, finally improve the long-term system reward. The experimental results in Figures 12 and 13 also show the same results. Then we will discuss how much per-trained computational burden is held for performance improvement. The pre-training data set and test data set are 1:1 in Section 4.