## 4 Applications of ML-Based Routing Algorithms in WSNs
### Intelligent Healthcare  
In recent years, ML-based schemes have been widely used to collect various physiological and vital signals of patients in intelligent medical systems to assist users in health management. There are several advantages of healthcare-aware WSNs (HWSN): high QoS, low cost, low power consumption, easy integration, and low complexity. Different from other application fields, since smart medical care is closely related to the life and health of patients, the loss or delay of any important information may lead to irreparable losses. Therefore, there are higher requirements for QoS and delay in HWSN.  
In healthcare applications, the occurrence of medical emergencies may lead to problems such as sudden traffic, network congestion, premature death of overloaded nodes, and dynamic changes in routing paths. Therefore, doctors may not be able to obtain the latest information from patients in time. To address these issues, a distributed congestion control and routing algorithm was proposed in HWSN [74]. In the proposed scheme, deployed sensor nodes are divided into different levels to transmit their sensing information through the congestion-free path, which minimizes the total energy consumption and improves QoS. Vital signals related to sensitive information (such as the heart rate, respiratory status, and blood sugar) in healthcare applications are set at a high priority to ensure the reliability and low latency of critical information delivery.  
To reduce the energy consumption of collecting patient physiological data in medical scenarios and extend the lifetime of wireless body sensor networks (WBSN), a cluster-based routing protocol was proposed [75], which combines the Q-Learning method and the cluster method to optimize the route between the node and the telemedicine station and reduce the delay of data packet transmission. The scope of cluster head nodes is planned, and the distance between two cluster heads is controlled to ensure better cluster area coverage. To obtain an optimized route to forward the data packet from the source node to the destination, the Q-learning algorithm is used. The node selects a route according to a strategy that attempts to maximize the cumulative reward that a packet may have in subsequent transitions from the current node to the next node, resulting in the shortest path from the beginning to the end. Let \(r\left(X_{j}z_{i},a_{j}\right)\) be the immediate reward that the packet acquires by executing an action \(a_{j}\) at node location \(X_{j}z_{i}\), the Q-value \(Q\left(X_{j}z_{i},a_{j}\right)\) at node \(X_{j}z_{i}\) with the route \(a_{j}\) is given by:  
\[Q\left(X_{j}z_{i},a_{j}\right)\ =r\left(X_{j}z_{i},a_{j}\right)\ +\ \gamma MaxQ\left(\delta\left(X_{j}z_{i},a_{j}\right),a\right) \tag{11}\]  
where \(\delta\left(X_{j}z_{i},a_{j}\right)\) represents the next node location due to the selection of route \(a_{j}\) at the node \(X_{j}z_{i}\). If the next state node is \(X_{j}z_{i}\), then \(Q\left(\delta\left(X_{j}z_{i},a_{j}\right),a^{\prime}\right)\ =Q\left(X_{j}z_{j},a^{ \prime}\right)\). \(\gamma\) is a constant that ensures the convergence of the sum, and its value ranges from 0 to 1. If the value is close to 0, the immediate reward is considered, while for the value close to 1, the future reward with greater weight is considered.  
It is very time-consuming to obtain Q values of all possible routing nodes. By identifying the path closer to the destination a\({}^{\prime}\), the Q-learning algorithm is transformed into an enhanced Q-Learning (QL-CLUSTER) algorithm, which improves the computing speed by reducing paths and minimizes memory requirements. This method effectively reduces computing cost of finding the best route.