## 4 Reinforcement Learning for IoT Networking Management
### RL Model for RPL Management
\begin{array}{cccc}&y_{1}&\ldots&y_{j}\\ \par Rank(x_{1})&\left[\begin{array}{c}Q_{(Rankx_{1},y_{1})}\\ p_{(s_{1},a_{1})}^{\pi}\end{array}\right]&\ldots&\left[\begin{array}{c}Q_{ (Rankx_{1},y_{j})}\\ p_{(s_{1},a_{j})}^{\pi}\end{array}\right]\\ \par Rank(x_{2})&\left[\begin{array}{c}Q_{(Rankx_{2},y_{1})}\\ p_{(s_{2},a_{1})}^{\pi}\end{array}\right]&\ldots&\left[\begin{array}{c}Q_{ (Rankx_{2},y_{j})}\\ p_{(s_{2},a_{j})}^{\pi}\end{array}\right]\\ \vdots&\vdots&\ddots&\vdots\\ \par Rank(x_{i})&\left[\begin{array}{c}Q_{(Rankx_{i},y_{1})}\\ p_{(s_{i},a_{1})}^{\pi}\end{array}\right]&\ldots&\left[\begin{array}{c}Q_{ (Rankx_{i},y_{j})}\\ p_{(s_{i},a_{j})}^{\pi}\end{array}\right]\end{array} \right) \tag{6}\]  
The reward is utilized to update the Q-value matrix \(Q(s_{t},a_{t})\) using Equation (2). This current Q-value matrix affects the subsequent action selection. For action selection, we can adopt the \(\epsilon\) greedy mechanism. In the learning process, the learning agent balances the short-term and long-term gains with exploration and exploitation. The \(\epsilon\) greedy mechanism performs the exploration using \(\epsilon\) as a probability parameter. For action selection, a random number _rand_ between 0 and 1 is generated, which is then compared with the probability parameter \(\epsilon\). \[a_{t}= \begin{cases}argmin_{y_{x}\in Y_{x}}\{Rank(y_{x})\},&rand<\epsilon,\\ argmin_{a_{i}}Q(s_{t},a_{j}),&rand\geq\epsilon,\end{cases}  \tag{7}\]  
Exploration is performed if \(rand<\epsilon\). During the exploration phase, node \(x\) selects its best alternative parent \(y_{x}\) with a minimum rank. If \(rand\geq\epsilon\), the action with the largest return reward will be selected. Let \(i\) be the index of action from a set of actions \(A\) and \(a_{i}\) represent the \(i\)-th action in action set \(A\). (3) Reward Function:The core of Equation (4) is the reward function \(r(s_{t},a_{t})\). The learning objectives are achieved using appropriate reward functions. With each action \(a_{t}\), state \(s_{t}\) changes to \(s_{t+1}\) and reward function \(r(s_{t},a_{t})\) updates to \(r(s_{t+1},a_{t+1})\). The reward function \(r\) of the RPL-based network layer algorithm can be defined with regard to link quality assessment between the child and parent node. The number of link layer retransmission attempts reflects the throughput of an individual link. The ETX measures MAC layer frame transmissions and retransmissions. ETX estimation involves measuring the probability of the frame loss ratio at the link \(l\) to each neighboring node in both the forward (\(d_{f}\)) and reverse (\(d_{r}\)) directions. The probability of unsuccessful frame transmission from node \(x\) to node \(y\) is calculated as follows [88]:  
\[p=1-\Big{(}1-p_{f}\Big{)}\times(1-p_{r}) \tag{8}\]where \(p_{f}\) is the probability of a transmission failure and \(p_{r}\) is the probability of a reception failure. The ETX for a successful delivery of a frame within a single hop after \(k\) attempts is measured as:  
\[ETX_{l}=\sum_{k=1}^{\infty}k\times p^{k}\times(1-p)^{k-1}=\frac{1}{(1-p)} \tag{9}\]  
The measurement of ETX with respect to the forward delivery ratio \(d_{f}\), i.e., \((1-p_{f})\), and reverse delivery ratio \(d_{r}\), i.e., \((1-p_{r})\), is calculated as:  
\[ETX_{l}=\frac{1}{\left(d_{f}\times d_{r}\right)} \tag{10}\]  
Alternatively, the ETX of the link is the inverse of the probability of successful packet delivery or link reliability, represented as:  
\[ETX_{l}=\frac{1}{\text{reliability }(l)} \tag{11}\]  
The reward associated with transmitting from a child node to a parent node is defined as follows:  
\[R=\text{sgn}(ETX_{\text{new}}-ETX_{\text{old}}) \tag{12}\]  
where \(ETX_{\text{new}}\) represents the new value of ETX, \(ETX_{\text{old}}\) represents the previous value of ETX, and the sign function sgn returns -1 if \(ETX\) increases and +1 if the value of \(ETX\) decreases.