## 4 Reinforcement Learning for IoT Networking Management
### RL Model for RPL Management  
We can observe from the previous section that devices such as IoT nodes can learn certain policies to improve the network performance. This section describes how the RPL-based network layer described in Section 2.2 can be mapped to the RL mechanism. RL-based RPL routing is illustrated in Figure 3. The RPL forms a DAG according to a specific OF. The nodes construct and maintain these DODAGs using DIO messages. DIO messages significantly contribute to the total network control overheads as they are periodically broadcasted to maintain and update the network topology. To conserve scarce resources, the RPL must generate minimum control overheads while maintaining network quality. (1) State Space: State \(s_{t}\) (\(s_{t}\in S\)) can be defined as the state observed by the DODAG child at time \(t\), which refers to the rank status of the child node in the DAG. Each child node has a set of \(i\) states, \(S=(0,1,2,\ldots,i)\). The selection of the forwarding path in the RPL mechanism can be based on node rank values. The MRHOF-based RPL mechanism uses ETX for rank calculation. The rank is measured as,  
\[Rank(x)=h(x)+Rank(y)+ETX(x,y) \tag{4}\]  
where \(h(x)\) is the hop-count of child node \(x\) towards the root node, \(Rank(y)\) is the selected parent \(y\) rank, and \(ETX(x,y)\) is the ETX value between child \(x\) and parent \(y\). The ETX value acts as a reward function for Equation (4). The reward function is described in Equation (12). Based on the reward function, each node updates its rank value during each state transition process. The rank of the root node is \(Rank(root)=1\). Each node broadcasts a DIO message containing its rank value. After receiving DIO messages, the nodes can generate a list of potential parents as,  
\[Y_{x}=\left\{n_{x}\in N\mid h(n_{x})<h(x),ETX(x,n_{x})<\delta\right\} \tag{5}\]  
where \(Y_{x}\) represents the list of potential parents of child node \(x\), \(n_{x}\) represents the set of all one-hop neighboring nodes of child \(x\), and \(\delta\) is a threshold to remove neighbors with unreliable links. To avoid routing loops, the child selects a forwarding path only if its number of hops from sink nodes is less than its own hop counts \(h(n_{x})<h(x)\). Figure 3: Illustration of RL-based RPL routing. (2) Action Space:A node can select a parent from the list of potential parents. The action \(a_{t}\) (\(a_{t}\in A\)) is defined as the selection of forwarding parent \(y\) from the list of potential forwarding parents. Environment \(E\) is defined as a wireless medium. When the IoT node performs the action \(a_{t}\) at the time \(t\), the state changes from \(s_{t}\) to \(s_{t+1}\). The IoT node receives a reward from the environment \(E\). A node \(x\) selects a parent \(y_{x}\) from a list of potential parents \(Y_{x}\). In a given time period \(t\), a node \(x\) has some rank value obtained from the last iteration using (5). At a given rank status, the node can select the next parent from the list of a finite number of potential parents \((Y_{x})\). Thus, at each state \(s\), there is a set of a finite number of permissible actions, which are elements of set \(A\). The state-action space is \(S\times A\), where \(S=(s_{1},s_{2},\ldots,s_{i})\) and \(A_{(s)}=\big{(}a_{1},a_{2},\ldots,a_{j}\big{)}\). The Q-values are stored in a look-up table for each state-action pair as,  
\[\left(