# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 4 Numerical Results
### Simulation Results
#### 4.1.3 Performance of SA-FRL Relay Selection  
The following cases are considered for comparative analysis. * **Ideal optimum:** as an upper bound, the ideal optimum supposed CSIs are perfectly known, and the relay with maximum reward will be selected accordingly. * **SA-FRL, \(1\to 0\):** simulated annealing optimized fast reinforcement learning relay selection with dynamic \(\varepsilon\) from 1 to 0. * **SA-RL, \(1\to 0\):** simulated annealing optimized reinforcement learning relay selection with dynamic \(\varepsilon\) from 1 to 0. * **SA-RL, \(1\to 0.05\):** simulated annealing optimized reinforcement learning relay selection with dynamic \(\varepsilon\) from 1 to 0.05. * **Random:** the relay is selected randomly from the available relay set in each time round. As shown in Figure 9, the rewards of proposed scheme and other schemes have been compared. It can be obviously seen that the proposed SA-FRL and SA-RL schemes have much better reward than random relay selection scheme. After convergence, the rewards of proposed schemes with different parameters are all over 80% better than random relay selection. Among the SA based learning schemes, SA-FRL with \(\varepsilon_{\min}=0\) has best convergence performance. For convergence speed comparison, SA-FRL with \(\varepsilon_{\min}=0\) reaches convergence after 500 transmissions rounds, while the SA-RL with \(\varepsilon_{\min}=0\) and SA-RL with \(\varepsilon_{\min}=0.05\) schemes begin to converge to a stable strategy after 800 time rounds. By using the fast learning strategy, the convergence speed has increased by 38% in comparison with  
Figure 8: Comparison of access delay of RL with/without SA strategy. Figure 7: Comparison of rewards of RL with/without SA strategy. the strategy without pre-training. For convergence value comparison, With \(\varepsilon\to 0\), both SA-FRL and SA-RL will achieve the rewards that are near to ideal optimum solution, and SF-FRL is still a little better than SA-RL. The simulation results have shown that the proposed SA-FRL scheme has the fastest convergence speed and the best learning reward. Even with outdated feedback, our proposed scheme can achieve optimum reward, owing to the dynamic exploration strategy. Figure 10 shows the convergence of normalized access delay with the iteration rounds of relay selection. Three SA and RL based relay selection schemes can achieve the stable convergence state with the increase of the number of signal transmissions. The normalized access delay of three SA and RL based relay selection schemes are much less than random strategy, and the access delay has been reduced about 15%. The SA-FRL scheme with \(\varepsilon_{\text{min}}=0\) can achieve the optimal stable state at the fastest speed. With fast learning strategy, SA-FRL scheme with \(\varepsilon_{\text{min}}=0\) begin to converge after about 500 transmissions round, while the SA-RL with \(\varepsilon_{\text{min}}=0\) and SA-RL schemes with \(\varepsilon_{\text{min}}=0.05\) begin to converge to a stable strategy after 800 time rounds. The proposed SA-FRL scheme is pre-trained with similar scenarios, and it selects relay nodes with higher reward and lower access delay in the initial learning process, so it has the fastest learning speed and learning results. When \(\varepsilon\to 0\), the minimum access delay can be achieved. Figure 10: Performance convergence of access delay. Figure 9: Performance convergence of reward. Figure 11 compares the data rates of three schemes, including SA-FRL considering access delay, SA-RL considering access delay, SA-RL without considering access delay (SNR based criteria in [30]), where \(\varepsilon_{\min}=0\). It can be seen from Figure 11 that regardless of the access delay, the data transmission rate is about 2.9 bps/Hz with SNR based criteria. When considering the access delay, the data transmission rate is about 3.45 bps/Hz after convergence, which has 17% improvement. SA-FRL also has the best data rate and convergence speed.