# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 4 Numerical Results
### Simulation Results
#### 4.1.2 Performance of SA-RL Relay Selection Scheme  
The performance of the SA-RL UWA relay selection scheme with dynamic \(\varepsilon\) is simulated and compared with RL scheme with invariant \(\varepsilon\) strategy. For SA-RL relay selection strategy, the initial value of annealing temperature is set to \(\varepsilon_{\max}=1\), and then approaches to minimum value. Two parameter settings, \(\varepsilon_{\min}=0.05\) and \(\varepsilon_{\min}=0\) have been simulated for evaluation. For invariant \(\varepsilon\) comparison, the settings are \(\varepsilon=0.2\) and \(\varepsilon=0.1\). The random relay selection is compared as a lower bound.  
As shown in Figure 7, the proposed SA-RL relay selection can achieve optimal reward after convergence, as the greedy factor is dynamically adapted. The fluctuation of reward of SA-RL tends to be stabler than invariant \(\varepsilon\) strategy after convergence, because the greedy factor is very small after convergence, which means the agent explores random action with low probability.  
In the first 800 rounds, the reward of SA-RL is lower than invariant \(\varepsilon\). As in the early stage of SA-RL, the numerical value of \(\varepsilon\) is high, which indicates that the agent will explore the relay more randomly with higher probability \(\varepsilon\) as Equation (19).  
With the iteration rounds increases, \(\varepsilon\) gets lower, and the probability of randomly selection decreases. The reward keeps increasing and finally converges to the optimal value. After convergence, the reward of SA-RL scheme with \(\varepsilon_{\min}=0\) is about 43% higher than invariant \(\varepsilon=0.1\), 60% higher than invariant \(\varepsilon=0.2\), and 88% higher than random selection. The reward of SA-RL with \(\varepsilon_{\min}=0\) is still increased by 21% in comparison with \(\varepsilon_{\min}=0.05\).  
Figure 8 has shown the access delay performance of the proposed SA-RL algorithm. The normalized access delay is defined as \(T_{S\!r_{i}D}/\min\{T_{S\!r_{i}D}\}\). It can be seen that normalized access delay of SA-RL scheme is close to the minimum and is most stable after convergence. Compared with other invariant \(\varepsilon\) value strategies, the SA-RL dynamic \(\varepsilon\) strategy reduces the access delay of the system by 2%, 4%, respectively. SA-RL reduces the access delay by 15% in comparison with random relay selection. From Figures 7 and 8, it is approved that the proposed SA-RL has the highest reward and lowest access delay compared to classic instant \(\varepsilon\) strategy. It should be noticed that, for classical RL with invariant \(\varepsilon\), if \(\varepsilon\) is very small at the beginning of training, it is easily to be trapped in local maximum.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{2}{c}{**Parameters of SA-RL**} \\ \hline Max\_episode & Max episode of learning per & 10 \\ \(M\) & Rounds of learning per episode & 2000 \\ \(\sigma\) & Learning rate & 0.9 \\ \(\xi\) & Discount factor & 0.9 \\ \(\lambda\) & Cooling rate & 0.996 \\ \(K\) & Constant & 1 \\ \(\varepsilon_{\max}\) & Initial maximum greedy factor & 1 \\ \(\varepsilon_{\min}\) & minimum greedy factor & [0.05, 0] \\ \hline \hline \end{tabular}
\end{table}
Table 2: Simulation parameters of SA-RL.