## 3 Design of SROA
### SROA Protocol Description
When packet transmission is successful, it is defined as:

\[c(en)=(1-\frac{E_{res}^{n_{j}}-E_{r}}{E_{ini}})+(1-\frac{E_{res}^{i}-E_{s}}{E_{ ini}}) \tag{16}\]

In Equation (16), \(E_{r}\) and \(E_{s}\) represent the energy consumption to receive and transmit packets. The sensor nodes with higher residual energy have lower energy-related costs, thereby balancing energy distribution and increasing network lifespan in UASNs. _c(delay)_ is a reflection of the congestion in the underwater sensor network. The nodes with many packets in their buffers will have long network latency. It is defined as:

\[c(delay)=1-\frac{1}{p_{b}^{n_{j}}+1} \tag{17}\]where \(P_{b}^{n_{j}}\) is the number of buffered packets of the neighboring node. With more packets in a neighbor node's buffer, the waiting time for the packet to be successfully forwarded from that node will be longer, causing data packets to wait in the queue for a longer period of time. As a result, _c(delay)_ is comparatively greater. The packet delivery-related cost, denoted by _c(pdr)_, represents the transmission quality in UASNs. The SROA calculates the PDR using the acoustic signal attenuation model, which is defined in Section 2.2 and indicated as \(p(d_{j},m)\):

\[c(pdr)=p(d_{j},m) \tag{18}\]

The packet delivery ratio is a crucial metric for assessing transmission reliability. The node with the highest delivery ratio is thought to be more trustworthy in packet advancement, hence it is more likely to be chosen as the forwarder. Since the SROA mainly aims to improve transmission reliability and energy efficiency, _c(en)_, _c(delay)_, and _c(pdr)_ are in the range of (0, 1) by definition, which is enabled to balance \(\alpha_{1}\) and \(\alpha_{2}\) in Equation (15) by further tuning the weights for various demands. They are, by definition, in the range (0, 1), allowing us to balance \(\alpha_{1}\) and \(\alpha_{2}\) in Equation (15) by fine-tuning the weights for various demands. However, the failure of the transmission also occurs in the real environment. If the packet retransmission approaches the limit and the receiver still does not receive the packet, significant energy and time will be consumed. Retransmission of data packets results in extra delay and energy consumption, raising the cost of unsuccessful transmission. The failure reward function is described as:

\[R_{s_{i}s_{j}}^{a_{j}}=-R_{0}-\alpha_{1}*[c(en)+\alpha_{2}*c(delay)] \tag{19}\]

where

\[ \begin{array}{l}\alpha_{1},\alpha_{2}\in(0,1)\\ c(en)=c_{i}(\text{en})=1-\frac{E_{in}^{i}-E_{s}*N_{\text{max}}}{E_{ini}}\\ c(delay)=1-\frac{1}{P_{b}^{n_{j}}+1}\end{array} \]

According to the definition of the reward function, the direct reward for successful and failed transmissions is defined as follows:

\[r_{i}(a_{j})=P_{s_{i}s_{j}}^{a_{j}}R_{s_{i}s_{j}}^{a_{j}}+P_{s_{i}s_{i}}^{a_{j }}R_{s_{i}s_{i}}^{a_{j}} \tag{20}\]

In order to estimate the acoustic channel state and state transition probability, each node records recent packet transmissions locally. The lost packets are indicated as \(\lambda\) and \(\text{n}\) is the total number of packet transfers. Therefore, the loss rate \(P_{s_{i}s_{i}}^{a_{j}}\) and the successful transmission rate \(P_{s_{i}s_{j}}^{a_{j}}\) are stated as follows:

\[ \begin{array}{l}P_{s_{i}s_{i}}^{a_{j}}=\frac{\lambda}{n}\\ P_{s_{i}s_{j}}^{a_{j}}=1-\frac{\lambda}{n}\end{array}  \tag{21}\]

Therefore, substituting \(P_{s_{i}s_{i}}^{a_{j}}\) and \(P_{s_{i}s_{j}}^{a_{j}}\) into the reward function, the reward function can be updated:

\[Q(s_{i},a_{j})=r_{i}(a_{j})+\gamma((1-\frac{\lambda}{n})Q^{*}(s_{j})+(\frac{ \lambda}{n})*Q^{*}(s_{i})) \tag{22}\]

The Q-value is related to the actions taken in the underwater environment and information exchange of the network. Initially, the Q-value of each node is set to zero, except the sink node. When a node delivers a packet, it updates its Q-value based on the information from the forwarder.