## 7 Classification of the Surveyed AI Techniques
### Machine Learning Techniques
#### 7.1.3 Reinforcement Learning  
RL deals with how an agent learns while interacting with an environment via the use of a reward system. The agent receives a delayed reward in the next time step based on which it evaluates its previous action. There are two variants of RL. The model-based and model-free RL. In the model-based approach, a transition probability maps a current state with an action and a resulting next state [141]. Thus, an agent's task is to learn an optimal policy that maximizes its reward or reduces its cost as it navigates through the environment [56]. Examples of such algorithms are Dyna Q and Monte Carlo methods. On the other hand, an agent in model-free RL relies on trial-and-error actions to update its knowledge about the environment, e.g., temporal difference learning and Q-learning. An RL problem is modeled using the Markov decision process (MDP) framework. An MDP is a 5-tuple [S, A, P, R, \(\mathbf{S_{0}}\)] where S is the set of possible states, A is the set of corresponding actions, P(St+1 St, At) represents the dynamics and R(St, At, St+1) is the reward, R(s, a, \(\mathbf{S_{0}}\)) represents the reward given to the agent at state s, after performing an action a and terminating in state \(\mathbf{S_{0}}\)[142]. The objective of MDPs is to find an optimal control policy that can maximize a given average reward per unit time or, a policy that minimizes the average cost per unit time. The value of a policy \(\pi\left(\mathbf{V^{\pi}}\right)\), (i.e., the expected discounted reward if starting in some state and following a policy \(\pi\)) can be expressed using the Bellman equation given as  
\[\mathbf{V^{\pi}}(\mathbf{s})=\mathbf{R}(\mathbf{s})+\boldsymbol{\gamma}\sum_ {\mathbf{s}\in\mathbf{S}}\mathbf{P}(\mathbf{s^{\prime}}|\mathbf{s},\mathbf{a })\mathbf{V^{\pi}}(\mathbf{s^{\prime}}) \tag{13}\]  
whereas the optimal value function (value function of an optimal policy \(\boldsymbol{\pi^{\star}}\) a policy with the highest value) can be obtained using  
\[\mathbf{V^{*}}(\mathbf{s})=\mathbf{R}(\mathbf{s})+\boldsymbol{\gamma}\underset {\mathbf{a}\in\mathbf{A}}{\mathbf{max}}\sum_{\mathbf{s^{\prime}}\in\mathbf{S}} \mathbf{P}(\mathbf{s^{\prime}}|\mathbf{s},\mathbf{a})\mathbf{V^{*}}(\mathbf{s ^{\prime}}) \tag{14}\]  
In CS tasks, RL is used to solve planning, control, optimization, and learning-related problems e.g., retransmission scheduling in 802.15.4e LLDN [143], intrusion detection system [144], self-learning power control [145], power consumption scheduling in an EH IoT node [146; 147], sampling rate configuration of EH sensors [148]. A major challenge when using RL techniques for cognitive sensing tasks is the difficulty in training active agents whose drop in performance could adversely affect the overall system. Another difficulty encountered is the memory-intensive nature of some RL algorithms as well as the need to limit exploratory moves during learning where the agent's safety is paramount. The large and continuous state and action spaces of some sensing tasks is also a challenge that needs to be addressed efficiently [149]. In ref. [150], authors develop three RL-based methods that address the user access control and battery prediction problems in a multiuser EH-based IoT system. The LSTM-DQN-based scheduling algorithm uses causal information about the channel and node battery states to find an optimal policy that maximizes the long-term discounted uplink sum rate. The battery state prediction algorithm uses deep LSTM to minimize prediction loss. The efficiency of the algorithms was tested under different conditions with results showing they were efficient in mitigating the addressed problems. In ref. [151], the authors formulate the resource allocation problem of IoT fog nodes using a Markov decision process (MDP) framework. For each request from an IoT user, the node decides whether to serve it locally at the edge using its resources or to refer it to the cloud to conserve its valuable resources.