# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 4 Numerical Results
### Experimental Results  
To further prove the performance of proposed SA-RL and SA-FRL, our experimental data measured in Mansfield Hollow Lake, Connecticut, USA in October 2014 is used for evaluation [14]. There are four sensor nodes, including one node that works as destination data center, one node works as source node and two nodes work as relay nodes. The source node is 198 m away from the destination node. The relay node 1 and the relay node 2 are 78 m and 77 m away from the destination node, respectively. Relay nodes can overhear the source node, and employ DF mode for cooperative communication. The depth of the lake is about 3-5 m. The AquaSeNT OFDMM model is hanged about 1 m from the water surface. The lake has a muddy bottom full of plants, and a lot of plankton. The sailing of motor boats have brought additional interference during the lake test. It is a complex time-varying UWA multi-path channel, and the CSIs from relay nodes to the destination node are time variant.  
With the experimental channel state information, three relay selection schemes have been compared, including of SA-FRL, SA-RL and random relay selection, where \(\varepsilon_{\min}=0\).  
Figure 12 has shown the rewards of the three schemes. It can be seen that both SA-FRL and SA-RL schemes can achieve obviously better rewards than random relay selection. The reward of SA-FRL is closer to the ideal optimum than RL after convergence, due to the benefit of per-training. After convergence, the average reward of SA-FRL have improved by 52% and the average reward of SA-RL have improved by 50% in comparion with random relay selection. At the same time, SA-FRL scheme takes fewer rounds to converge to a relatively stable status. Figure 12 has shown that the SA-FRL scheme has faster learning speed and better learning result than the SA-RL and random relay selection scheme.  
Figure 13 has shown the performance of the access delay for these three schemes. The SA-FRL scheme will turn to convergence with the least iteration rounds, and is closest to the ideal optimum access delay. Later, the SA-RL scheme begins to converge to be stable  
Figure 11: Data transmission rate.  
with the continually increase of the number of signal transmissions, and the convergence value is a little bit smaller than SA-FRL.  
Figure 14 has compared the data rate of three schemes including, SA-FRL considering access delay, SA-RL considering access delay and traditional RL without considering access delay (SNR based criteria). In the lake experiment, as the distances between nodes are relatively close, the distinction of access delay is not obvious. Herein, the data rate of SA-FRL and SA-RL have increased about 1%. Although the improvement is not great, the performance curves already can converge to be close to ideal optimal value. This observation is consistent with that from simulation results. In conclusion, the effectiveness of the proposed scheme has been approved by lake experimental data.  
Figure 12: Performance convergence of reward, experimental data.  
Figure 13: Performance convergence of access delay, experimental data.