# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 4 Numerical Results  
### Simulation Results  
#### 4.1.1 Simulation Setup  
Simulations have been conducted to evaluate the performance of the proposed relay selection scheme in UWA data collection networks. The simulated UWA network includesone source node, one destination node and four relay nodes, which are located in an area with 3 km diameter. To generate more realistic CSI for reinforcement learning, the time-varying UWA channels are simulated as Section 2. The channels can be simulated by setting the geography and environmental parameters, such as communication distance and depth, communication frequency band, speed of sound, the variation of distance and height of transmitter/receiver, sea surface/bottom, et al. This simulation method fully considers the actual signal transmission characteristics of physical layer, including large-scale path variation and small-scale scattering [45]. According to this method, the time-varying channel impulsive response will be generated, and the channel gain of the corresponding frequency band can be obtained accordingly. The SNRs can be calculated by Equations (2) and (3), where the sound source level (SL) can be set to a certain value, and the noise power level (NL) can be calculated by Wenz model. To incorporate these parameters with the time-varying channel gain together, the received SNR sequence can be calculated finally. To mitigate the effect of frequency selective fading, the multi-carrier modulation has been used for physical layer communication scheme. Specifically, OFDM is employed where the number of subcarriers is set to 1024 and the frequency is 14-20 kHz. The settings are matched with AquaSeNT OFDM modem, which are accordance with the experimental setup which we will further used for evaluation. The parameters of UWA environment for simulation are listed as Table 1. It can be seen that parameters including the sound speed, the sea surface and bottom, the location of transmitter/receiver, the scattering intra-paths are all time varying within defined variation range. The topology in Figure 3 is used as an example for simulation in this paper. The distance between the nodes can be generated randomly. Figure 6 has shown examples of the CIR with certain communication distance from sensor nodes to destination node \(D\) for illustration. It can be seen that the multi-path delay and amplitude keep varying during the simulation. From the sub-figures of Figure 6, it can be seen that the longer communication distance leads to lower amplitude of CIR. The multi-path structures of the five nodes are different. With the generated CSI, the corresponding channel gain can be calculated. According to the generated time-varying CIRs, the range of channel gains of \(S\)-\(D\) and \(r_{i}\)-\(D\) are [-30, -28.5], [-28.8, -27.4], [-28.3, -26.7], [-27, -25.4], [-26, -24.8], respectively. \begin{table}
\begin{tabular}{c c} \hline
**UWA Environment Parameters** & **Value** \\ \hline Sound speed & 1200–1500 m/s \\ Spreading factor & 1.7 \\ Surface height & 100 m \\ Variation of surface height & \(\pm\)1 m \\ Transmitter/receiver height & \(\geq\)5 m \\ Variation of transmitter/receiver height & \(\pm\)1 m \\ Channel distance & [3, 2.5, 2.25, 1.75, 1.45] km \\ Variation of channel distance & \(\pm\)2 m \\ Number of intra-paths & \(\pm\)1 m \\ Mean of intra-path amplitudes & 0.025 \\ Variance of intra-path amplitudes & 0.000001 \\ \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters of UWA environment. Figure 6: Channel impulse responses from sensors nodes to destination node. (**a**) Received CIR by \(D\) sent from \(S\), \(d=3\) km. (**b**) Received CIR by \(D\) sent from \(r_{1}\), \(d=2.5\) km. (**c**) Received CIR by \(D\) sent from \(r_{2}\), \(d=2.25\) km. (**d**) Received CIR by \(D\) sent from \(r_{3}\), \(d=1.75\) km. (**e**) Received CIR by \(D\) sent from \(r_{4}\), \(d=1.45\) km. For reinforcement learning, the CSI is quantized according to the CSI sequence to construct a Q-table.