## 5 Adaptive Data Transmission Algorithm Based on Reinforcement Learning
### Reinforcement Learning Model
The environment includes not only all possible relay nodes, but also all possible data accuracy on different nodes._  
Figure 4: Framework of adaptive data transmission model based on reinforcement learning. **Definition 2**. : _At time \(t\), action \(A\) can be defined as follows:_  
\[\mathrm{A}=\left\{\mathrm{a}_{\mathrm{j,acc}}\big{|}(\mathrm{n}_{\mathrm{j}}\in \mathrm{S},\;\mathrm{acc}\in\{\mathrm{n}_{\mathrm{min}},\cdots,\;\mathrm{n}_{ \mathrm{max}}\})\right\}\,, \tag{12}\]  
_If the relay node completes data forwarding, it will be rewarded. If the relay node is closer to the water surface and has more residual energy, and the delay of transmission channel is smaller, it will be more conducive to the efficient transmission of data to the sink node, and it will be given additional rewards. If it adopts higher transmission accuracy and transmits more correction data, it will also receive additional rewards._  
**Definition 3**. : _At time \(t\), if the packet is at node \(\mathrm{n}_{\mathrm{i}}\), andn\({}_{\mathrm{i}}\)selects \(\mathrm{n}_{\mathrm{j}}\)as a relay node, and sends the packet with the accuracy 'acc', the reward function is as follows:_  
\[\mathrm{R}_{\mathrm{n}_{\mathrm{i}}\mathrm{n}_{\mathrm{j,acc}}}^{\mathrm{a}_{ \mathrm{i,acc}}}=-\mathrm{R}_{0}-\left[\varphi_{\mathrm{e}}\times\mathrm{co}( \mathrm{e})+\varphi_{\mathrm{t}}\times\mathrm{co}(\mathrm{t})+\varphi_{ \mathrm{1}}\times\mathrm{co}(\mathrm{l})\right]\,, \tag{13}\]  
_Part I: the fixed cost \(\mathrm{R}_{0}\) will be accumulated with each data forwarding. The existence of \(\mathrm{R}_{0}\) can help the agent select a route with fewer hops. However, the shortest route is not always the best route. It also needs to comprehensively consider the factors such as the nodes, channels, and data transmission accuracy._  
_Part II: \(\mathrm{co}(\mathrm{e})\) is the cost for node energy. The existence of \(\mathrm{co}(\mathrm{e})\) can help the agent select the node having larger residual energy as the relay node, which is more conducive to the continuation of the network lifecycle. \(\mathrm{co}(\mathrm{e})\) can be expressed as follows:_  
\[\mathrm{co}(\mathrm{e})=1-\frac{\mathrm{E}_{\mathrm{res}}^{\mathrm{i}}}{\sum_{ k\in\mathrm{N}_{\mathrm{delay}}(\mathrm{i})}\mathrm{E}_{\mathrm{res}}^{\mathrm{k}}}\,, \tag{14}\]  
\[N=\{n_{1},n_{2},n_{3},\cdots,n_{m}\}, \tag{8}\]0 Schematic diagram of virtual pipeline. The candidate relay nodes in the virtual pipeline are represented in red, and the color of the sensor nodes represents the type of collected data. _where \(\rm E_{res}^{j}\) represents the residual energy of the next hop node, \(\sum_{k\in N_{\rm relay}(i)}E_{res}^{k}\) represents the total remaining energy of the candidate node set, and \(\varphi_{e}\) is the sensitivity coefficient of residual energy cost of neighbor nodes. Part III: \(\rm co(t)\) is the cost for the delay of the transmission channel. The existence of \(\rm co(t)\) can help the agent select the node having smaller transmission delay as the relay node. This means that the channel between the current node and the relay node is stable and reliable, has a low bit error rate and has less congestion. \(\rm co(t)\) can be expressed as follows:_  
\[N=\{n_{1},n_{2},n_{3},\cdots,n_{m}\}, \tag{8}\]1  
_where \(\overline{t_{total}^{i\to j}}\) is the average of total delay from node \(\rm n_{i}\) to node \(\rm n_{j}\). The total delay refers to the time from \(\rm n_{i}\) sending the data until \(\rm n_{i}\) hears \(\rm n_{j}\) forwarding the same data, which includes the sender delay in \(\rm n_{i}\), the propagation delay from \(\rm n_{i}\) to \(\rm n_{j}\), the processing and waiting delay in \(\rm n_{j}\), and the propagation delay from \(\rm n_{j}\) to \(\rm n_{i}\). \(\rm t_{timeout}\) is the maximum value of the total delay node \(\rm n_{i}\) to node \(\rm n_{j}\), marking data transmission failure. It is generally defined according to the transmission conditions and business requirements.