## 4 Proposed CLCQ Protocol
### Selection of Next Hop  
The design of this protocol incorporates considerations of the channel quality of the actual underwater environment, thereby improving the PDR. As a cross-layer routing protocol based on Q-learning, the CLCQ can select the next hop forwarding node by calculating function values through the current buffer state, the residual energy, and the BER of point-to-point communication. Note that competitive MAC protocols can potentially lead to collisions, which may result in packet loss. In order to mitigate this risk, the CLCQ employs Slotted FAMA [32] as the MAC protocol and makes further improvements. To prevent packet collisions, each packet must be transmitted within the corresponding time slot. The length of a time slot is defined as follows:  
\[slot\ =\ T_{\text{prop}}+T_{\text{trans}}+T_{\text{guard}} \tag{16}\]  
\[T_{\text{prop}}\ =\ \frac{range}{V_{\text{sound}}} \tag{17}\]  
\[T_{\text{trans}}\ =\ \frac{packetSize}{Rb} \tag{18}\]  
where \(T_{\text{prop}}\) is the maximum propagation delay, \(T_{\text{trans}}\) is the transmission delay, and \(T_{\text{guard}}\) is the guard interval, set to 0.001 s.  
The goal of Q-learning is to learn a policy that can achieve the maximum cumulative reward by continuously updating the Q-value. The design of the reward function is crucial for the effectiveness of Q-learning as it determines the behavioral goals and learning motivations of the agent. Assuming that node \(n_{i}\) holds a packet under state \(s_{i}\), the action of node \(n_{i}\) sending the packet to node \(n_{j}\) in its forwarding candidate set is denoted as \(a_{ij}\), with the obtained reward denoted as \(R_{ij}\), and the Q-function defined as \(Q(s_{i},a_{ij})\). Some symbols are shown in Table 1.  
If the packet is successfully transmitted from node \(n_{i}\) to node \(n_{j}\), the reward function can be obtained by the following:  
\[ \begin{array}{rl}R_{sij}^{a_{ij}}=&-g_{0}-\varphi_{q}\times c(q)-\varphi_{t} \times c(t)-\varphi_{c1}\times\big{[}b(n_{i})+b\big{(}n_{j}\big{)}\big{]}\\ &\hskip 14.226378 pt+\varphi_{c2}\times\big{[}d(n_{i})+d\big{(}n_{j}\big{)} \big{]}\end{array}  \tag{19}\]  
The reward function mainly consists of four parts: a fixed reward, a channel-quality-related reward, a delay-and-congestion-related reward, and an energy-consumption-related reward. Among them, \(g_{0}\) is the fixed reward with a weight set to 1, while the other weights (\(\varphi_{q}\), \(\varphi_{t}\), \(\varphi_{c1}\), and \(\varphi_{c2}\)) are all less than or equal to 1. **Energy-consumption-related reward**: \(b(n_{i})\) is the reward related to the remaining energy, defined as follows [23]:  
\[b(n_{i})\ =\ 1-\frac{E_{res}(n_{i})}{E_{init}(n_{i})} \tag{20}\]  
wherein \(E_{res}(n_{i})\) and \(E_{init}(n_{i})\), respectively, represent the remaining energy and initial energy of the node. When all nodes have the same initial energy, the lower the remaining energy of node \(n_{i}\), the higher the reward \(b(n_{i})\), and the more difficult it is for the two nodes to communicate. \(d(n_{i})\) is a function measuring the energy distribution of the node, defined as follows:  
\[d(n_{i})\ =\ \frac{2}{\pi}\arctan\big{(}E_{res}(n_{i})-\overline{E}(n_{i}) \big{)} \tag{21}\]  
where \(\overline{E}(n_{i})\) is the average remaining energy of all neighbor nodes of node \(n_{i}\). The greater the discrepancy between a node's residual energy and the mean residual energy of its neighboring nodes, the more likely it is that the node will be selected as the next hop node. **Delay-and-congestion-related reward**: Congestion occurs when there are a large number of packets in the buffer of a node. The handshake-based Slotted FAMA protocol transmits a limited number of packets per handshake. For nodes with a high load, a certain number of handshakes is required to release the load. Consequently, the congestion issue in the network can be viewed as a delay, and the reward can be expressed as follows [15]:  
\[c(t)\ =\ 1-\frac{1}{t_{total}+1} \tag{22}\]