## 4 Applications of ML-Based Routing Algorithms in WSNs
### Underwater Sensing  
Underwater WSNs (UWSNs) can be widely used in marine resource surveying, pollution monitoring, aided navigation, and tactical surveillance. The most important feature that distinguishes the underwater environment from other environments is the difference in the propagation, which also leads to the difference in its communication methods. UWSNs are mainly divided into two categories: underwater acoustic WSNs (UAWSNs) and underwater optical WSNs (UOWSNs). Although there are many similarities between UWSNs and traditional WSNs, some characteristics of UWSNs mean that traditional routing concepts cannot be directly applied to the design of routing algorithms in UWSNs. The acoustic routing of UAWSNs faces challenges in terms of dynamic structure, high energy consumption, long delay, and narrow bandwidth. Traditional routing protocols are difficult to adapt to the underwater environment. A routing model based on RL and game theory was proposed in [76], which provides an effective packet forwarding mechanism to solve the acoustic transmission routing problem in UAWSNs. This scheme uses a routing based on Q-learning, which works by successively improving its evaluations of the quality of particular actions at particular states. During the operation of UAWSNs, sensor nodes make routing decisions by considering end-to-end delay, hop count, packet error rate, and energy efficiency. Under dynamic acoustic conditions, the sensor nodes attempt to maximize their own profit in a distributed online manner based on the Q-learning approach. This solution enables each sensor node to understand the situation and determine the best routing path through a step-by-step interactive feedback process. During the routing operation, each node periodically updates the routing information and reevaluates the current strategy and selects one of its neighbor nodes for packet forwarding. This method effectively achieves the maximization of the expected benefit and improves the energy efficiency. Different from UAWSNs, UOWSNs have attracted wide attention for their advantages such as high transmission rate, ultra-wideband, and low latency. However, UOWSNs have limited communication distance, and nodes must transmit data in a multi-hop manner, which leads to the problem of high probability of instability in the connection and makes them have higher requirements for link quality. To realize the adaptive ability in the dynamic environment, a distributed multiagent RL routing (DMARL) scheme was designed [77]. Due to the information interaction between agents, RL algorithms based on multiple agents are beneficial to help agents learn environmental knowledge from a global perspective. A distributed network model is established to support multi-hop data transmissions, which considers the energy, link stability, and data transmission quality to modify the reward function of the routing algorithm. In order to prolong the lifetime of the network, a distributed value function (DVF) based on the Q-learning iterative formula is used to update the Q value of each node. The DVF \(Q_{i}^{t+1}\big{(}s_{i^{\prime}}^{t},a_{i}^{t}\big{)}\) is defined as:  
\[Q_{i}^{t+1}\big{(}s_{i^{\prime}}^{t},a_{i}^{t}\big{)} =(1-\alpha)Q_{i}^{t}(s_{i^{\prime}}^{t},a_{i}^{t})+\alpha\{r^{t+1 }(  
\[af(j)\ =\frac{E_{res}^{j}}{\sum_{i^{\prime}\in I}E_{res}^{i^{\prime}}} \tag{15}\]  
where \(V_{j}^{t}(s_{j}^{t+1})\) and \(V_{r}^{t}(s_{r}^{t})\) represent the state value functions to estimate the next state \(s_{j}^{t+1}\) and other neighbors' state \(s_{r}^{t}\), \(r^{t+1}(s_{j}^{t+1})\) represents a direct reward that node \(i\) (with the current state) will receive at time \(t+1\), \(\alpha\) is the type of learning rate that depends on the link stability, \(I\) is the set of \(i\)'s neighbors, \(af(j)\) represents the received global reward (GR), which indicates the transmission direction of the data packet (the quality of the action performed).