## 5 Adaptive Data Transmission Algorithm Based on Reinforcement Learning  
### Reinforcement Learning Model  
Reinforcement learning is a kind of learning that aims to obtain the best mapping strategy from environmental state to action. The best strategy can maximize the reward, which is the evaluation of the movement quality. Due to the limited information provided by the external environment, reinforcement learning can improve the ability to adapt to the dynamic environment by continuously learning and acquiring feedback knowledge in the process of action evaluation. Reinforcement learning algorithms are often described by Markov decision processes (MDPs). The process of reinforcement learning adopts the five tuples of \(\left(\text{S},\text{ A},\text{P},\text{R},\gamma\right)\), where S stands for environment status, A stands for action, P stands for transition probability, R stands for reward, and \(\gamma\) represents the discount rate. Agents, according to the environment status S, cumulative reward R, and discount rate \(\gamma\), determine the transition probability P, and generate action A, then switch to the new state S! [48]. In underwater wireless sensor networks, the process of data transmission can also be regarded as Markov decision processes (MDPs). Reinforcement learning is used to select the best relay node and meet the required data transmission accuracy, as shown in Figure 4.  
Assuming that the nodes of the underwater wireless sensor network can be expressed as follows:  
\[N=\{n_{1},n_{2},n_{3},\cdots,n_{m}\}, \tag{8}\]  
where \(n\) represents the node and \(m\) is the number of nodes, each node in the network can obtain its own location information. Therefore, a virtual pipeline can be built on the vector between the data source node and the target sink node, as shown in Figure 5. The virtual pipe radius of the node i is set to:  
\[R^{i}_{\text{pipe}}=-\frac{R-R^{i}_{\text{ini}}}{E_{\text{ini}}}\times\overline {E}^{i}+R\, \tag{9}\]  
where \(E_{\text{ini}}\) is the initial energy of the node, \(\overline{E}^{i}\) is the average residual energy of the neighbor nodes, \(R\) is the signal transmission distance of the node, and \(R^{i}_{\text{ini}}\) is the initial radius of the pipeline. When the neighbor nodes have more residual energy, they can maintain a small pipeline radius, allowing the data to acquire the shortest path. When the neighbor nodes have less residual energy, they can enlarge the pipeline radius of transmission, allowing more neighbor nodes to participate in packet forwarding. The candidate relay node set \(N_{relay}(i)\) of the current node \(n_{i}\), can be expressed as follows:  
\[N_{\text{relay}}(i)=\left\{n_{j}\in N_{i}(t)\subseteq\left\{N\big{|}\text{ dep}(n_{j})-\text{dep}(n_{i})\leq 0\right\}\cap\text{ neighbors}(n_{i})\cap\text{ pipe}(n_{i})\right\}\, \tag{10}\]  
At time \(t\), the node set of candidate relay nodes for \(n_{i}\) is \(N_{i}(t)\). The node set of neighbors overridden by the one-hop transmission of the node \(n^{\prime}_{i}s\) signal is \(\text{neighbors}(n_{i})\). The node set in the \(n^{\prime}_{i}s\) virtual pipe is \(\text{pipe}(n_{i})\). The node set with a shallower depth than the current node \(n_{i}\) is \(\left\{N\big{|}\text{dep}(n_{j})-\text{dep}(n_{i})\leq 0\right\}\). The relay node selection needs to be within the range defined by Formula (10). **Definition 1**. : _At time \(t\), if the packet to be sent is at node \(n_{i}\), then the current environment state \(S\) can be defined as follows:_  
\[S=\underset{\text{acc}}{\cup}\Big{(}\{n_{i}\}\cup\text{ N}_{\text{relay}}(i,\text{acc})\Big{)}, \tag{11}\]  
_where 'acc' denotes that the monitoring data are located in the variable accuracy range of the node, which is determined by \(\left\{\text{accuracy}_{\text{high}},\cdots,\text{accuracy}_{\text{low}}\right\}\) or \(\{n_{\text{min}},\cdots,\text{n}_{\text{max}}\}\).