# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 6 Conclusions  
This paper has proposed a reinforcement learning based relay selection scheme for UWA cooperative communication, where the unique characteristics of UWA channel have been considered for the design of state and reward. Moreover, the action exploration strategy is improved by integrating the simulated annealing algorithm with dynamic greedy factor to speed up the learning process and convergence, and the fast reinforcement learning scheme has been proposed for practical implementation. Both simulation and experimental data have been used for system evaluation. Numerical results have revealed that the proposed scheme can select the best cooperative relay with good channel quality and low access delay for optimum data rate in comparison to the scheme without the consideration of access delay. The SA exploration strategy can effectively improve the convergence performance. The SA-RL and SA-FRL can improve the system reward and reduce access delay in comparison with basic RL scheme. With pre-trained Q-table, SA-FRL scheme has the fastest learning speed and the best learning result compared with other schemes. The experimental results have shown the effectiveness of proposed scheme in the actual UWA network.  
Conceptualization, Y.Z. and Y.L.; data curation, Y.Z.; funding acquisition, Y.Z., A.W., B.W., Y.L. and W.B.; investigation, Y.Z. and Y.S.; methodology, Y.Z. and X.S.; resources, B.W. and W.B.; software, Y.S.; supervision, Y.Z. and X.S.; validation, Y.S.; writing--original draft preparation, Y.S.; writing--review and editing, Y.Z. and A.W. All authors have read and agreed to the published version of the manuscript.  
This research was funded by National Natural Science Foundation of China under grant No. 61801372, U19B2015, 62001360, 61801371.  
**Institutional Review Board Statement**: Not applicable.  
**Informed Consent Statement**: Not applicable.  
**Data Availability Statement**: Publicly available datasets were analyzed in this study. This data can be found here: [[http://millitsa.coe.neu.edu/projects.html](http://millitsa.coe.neu.edu/projects.html)]([http://millitsa.coe.neu.edu/projects.html](http://millitsa.coe.neu.edu/projects.html)), accessed on 30 November 2021.  
**Conflicts of Interest**: The authors declare no conflict of interest.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Number of Training** **Rounds**} & **Normalized Access** & **Improvement (\%)** & **Improvement (\%)** \\  & **Delay of SA-FRL** & **Compared to SA-RL** & **Compared to** \\  & **Relay Selection** & **Relay Selection** & **Random Relay Selection** \\ \hline
200 & 1.00038 & 0.0032\% & 0.027\% \\
400 & 1.000031 & 0.0039\% & 0.027\% \\
600 & 1.000029 & 0.0042\% & 0.028\% \\
800 & 1.000025 & 0.0046\% & 0.028\% \\
1000 & 1.000024 & 0.0047\% & 0.028\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Normalized access delay of SA-FRL with different amount of pre-training data, experimental channel.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Number of Training** **Rounds**} & **Reward of SA-FRL** & **Improvement (\%)** & **Improvement (\%)** \\  & **Relay Selection** & **Compared to SA-RL** & **Compared to** \\  & **Relay Selection** & **Relay Selection** & **Random Relay Selection** \\ \hline
200 & \(-\)1.019 & 9.4\% & 46.5\% \\
400 & \(-\)0.996 & 11.6\% & 47.7\% \\
600 & \(-\)0.987 & 12.2\% & 48.1\% \\
800 & \(-\)0.977 & 13.2\% & 48.7\% \\
1000 & \(-\)0.972 & 13.7\% & 49\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Reward of SA-FRL with different amount of pre-training data, experimental channel.