# Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks
## 2 System Model
### UWA Channel Model  
UWA systems have to be designed to operate in the complex underwater environment, where the system geometry and channel conditions are varying during the deployment. Especially for resources allocation system, it is crucial to have a relatively accurate channel model for evaluation of the algorithms and protocols before experiment. Beam tracing tools which use ray theory can provide deterministic channel impulsive response (CIR), but they do not consider random channel variations, such as BELLHOP [45; 46]. Some studies have been conducted to statistically model the UWA channel. The relay selection schemes in [30; 31] have utilized the path loss function to model channel gain, Wenz noise model to calculate the noise power level (NL), and fading coefficient that follows specific fading distribution to simulate small scale fading. This type of method can simulate the received SNR statistically. In this paper, the time-varying UWA channel has been simulated in the view of physical layer with the consideration of both acoustic propagation loss and inevitable random channel variations. The random UWA channel variations include location uncertainty of transmitter/receiver, the motion of sea surface, and small-scale random scattering, etc. The variations will affect not only the locally averaged received power, but also the instantaneous fast varying channel response. The channel transfer function will be modeled with acoustic propagation loss and channel variations, and then the gain contained in the channel can be calculated accordingly [45]. The propagation path loss of UWA signal is related to carrier frequency \(f\) and transmission distance \(d\). The UWA propagation loss \(A(d,f)\) can be calculated by the empirical formula in reference [4] as  
\[A(d,f)=d^{k}\alpha(f)^{d} \tag{4}\]where \(d^{k}\) stands for spreading loss and \(\alpha(f)^{d}\) stands for absorption loss in water. \(k\) is the spreading factor. \(\alpha(f)\) is absorption coefficient which can be calculated by the [PERSON]'s empirical formula in dB per kilometer as  
\[10\lg[\alpha(f)]=0.11\frac{f^{2}}{1+f^{2}}+44\frac{f^{2}}{4100+f^{2}}+2.72\times 1 0^{-4}f^{2}+0.003 \tag{5}\]  
where \(f\) is in KHz. Due to the displacement of the transmitter/receiver in underwater environment, as well as the changes in the sea surface height or the shape of seafloor, the system geometry is varying. These factors cause the uncertain change of multi-path. Suppose that the \(p\)th path has an average communication distance \(\bar{d}_{p}\), and \(d_{p}\) can be modeled as a variable that follows Gaussian distribution. The channel transfer function is proportional to reflection and inversely proportional to path loss. For \(p\)th path, the corresponding average channel transfer function \(\bar{H}_{p}\) of \(p\)th path with average propagation distance \(\bar{d}_{p}\) can be expressed as  
\[\bar{H}_{p}(f)=\frac{\Gamma_{p}}{\sqrt{A(d_{p},f)}}=\frac{\Gamma_{p}}{\sqrt{ \left(\frac{d_{p}}{\bar{d}_{0}}\right)^{k}\alpha(f)^{\bar{d}_{p}-\bar{d}_{0}} }}\bar{H}_{0}(f)=\bar{h}_{p}\bar{H}_{0}(f) \tag{6}\]  
where \(\Gamma_{p}\) is the cumulative reflection coefficient of the seabed and the sea surface. For normalized expression to add all multi-paths together, normalized multi-path propagation distance \(\bar{d}_{0}\) is introduced. \(\bar{H}_{0}(f)\) is the normalized reference channel transfer function referred to \(\bar{d}_{0}\). \(\bar{h}_{p}\) is the average gain of the normalized path. Modeling the \(p\)th multi-path as a path with gain \(h_{p}\) and delay \(\tau_{p}\), then the total channel transfer function can be represented as  
\[H(f)=\bar{H}_{0}(f)\sum_{p}h_{p}e^{-j2\pi f\tau_{p}} \tag{7}\]  
where \(\tau_{p}\) can be expressed as a function of propagation speed \(c\) as \(\tau_{p}=d_{p}/c-t_{0}\), and \(t_{0}\) is the normalized reference delay.