## 3 ML-Based Routing Algorithms in WSNs
### Reinforcement-Learning-Based Routing Methods  
Reinforcement learning enables the system to learn and improve from cases and depends on agents who are constantly acting in the environment and receiving rewards and punishments based on the results of their actions. Q-learning is the most typical and widely used reinforcement learning algorithm, in which the maximum action value is considered. Q-learning reaches convergence in continuous iterations to obtain the optimal value matrix \(Q(s_{t},a_{t})\), making its computational complexity relatively low:  
\[Q(s_{t},a_{t})=Q(s_{t},a_{t})+\alpha\left[r(s_{t},a_{t})\ +\ \gamma\max_{a_{t+1} \in A}Q(s^{\prime},a_{t+1})-Q(s_{t},a_{t})\right], \tag{9}\]  
where \(Q(s_{t},a_{t})\) is value matrix of action selection, \(s_{t}\) is the state observed at time \(t\), \(a_{t}\) is the action selected at time \(t\), \(\alpha\) is the learning rate, \(\gamma\) is the discount factor, and \(r(s_{t},a_{t})\) is the reward for choosing action \(a_{t}\) in state \(s_{t}\). In [69], a Q-learning reliable routing with a weighting agent approach (QLRR-WA) was proposed. In the construction of the routing graph, QLRR-WA adopts a weighted cost equation to select nodes and neighbors. The states of agents represent the set of weights, and the action is continuously taken in the environment to learn the set of weights that can optimize network performance. Based on average network latency and expected network lifetime, the agent is given a reward or punishment to change the agent's next action policy. Meanwhile, QLRR-WA tends to improve the network reliability by constructing an uplink graph in which nodes have at least two neighbors to forward data to the gateway. In [70], a Q-probabilistic routing (Q-PR) scheme is proposed based on reinforcement learning and a Bayesian decision model, which learns from previous routing decisions and local interaction among neighbor nodes to make routing schemes adjust to future conditions. Q-PR calculates the optimal routes depending on the power constraints, the importance of the messages, and the expected delivery rate. The Bayesian approach makes decisions by considering the data importance, node profiles, expected transmission, and received energy to send the group to the set of candidate neighboring nodes. However, the RL-based routing algorithm faces the issue of a limited cognition of knowledge. Therefore, this algorithm is not suitable for highly dynamic environments because it takes a long time to learn history information. To address the routing challenges caused by intermittent connections between devices and lack of fixed paths for message transmission in opportunistic Internet of Things networks (OppiToTs), a reinforcement-learning-based routing protocol RLProph was proposed [71]. The algorithm optimizes the routing process by the Markov decision process (MDP) and reduces energy consumption with the optimal routing. Compared with other context-independent routing, RLProph automatically determines the decision to maximize the reward based on the context and achieves better performance in information transmission. To maximize message delivery and minimize network overhead, a MDP is formulated as \((S,A,P_{a}(s,s^{\prime}),R_{a},\gamma)\), where \(S\) is a finite set of states describing the environment, \(A\) is the finite set of actions, \(P_{a}(s,s^{\prime})=P(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)\) is the probability  
RLProph uses a planning-based reinforcement learning approach for routing. The routing process uses optimal strategies to make intelligent and effective routing decisions, thereby increasing the probability of message delivery and reducing overhead. Considering that IoT applications have stricter requirements, such as more flexible and efficient routing, the reinforcement-learning-based routing decision is favored for its high flexibility and accuracy. Based on reinforcement learning, two centralized and distributed routing schemes were designed and implemented in WSNs [72].