## 2 Literature Review  
### Fundamental Concepts  
RL is a type of machine learning that makes an agent learn the dynamic behavior of its environment by taking actions based on its current state, which improves learning with time (maximizing the cumulative reward concept) using trial-and-error interaction with the environment [10]. For example, a sink interacts with all the sensor nodes in the network to make routing decisions for the sensor nodes. In this case, the agent is the sink, the environment is the sink's neighborhood, the state is the current unicast tree the nodes are using to send data packets, and the action is the selection of the next unicast tree to be used by all the nodes to send data packets. An RL problem is solved by modeling the problem as a Markov Decision Process (MDP), as shown in Figure 1. The MDP is made up of four tuples \((S,A,P,R)\), where \(S\) denotes the states set, \(A\) is the actions set, \(P\) is the state transition probability matrix, and \(R\) is the reward function. \(P\) and \(R\) combined together are used to describe the model of the environment. The probability of being in a given state \(S_{t+1}=\hat{s}\) from a current state \(S_{t}=s\) by taking action \(A_{t}=a\), and getting a reward \(R_{t+1}=r\) is known as the transitional probability, given in Equation (1) [16]. The reward function enables the environment to provide feedback as a form of reward to the agent. The reward is the measurement of the effect of the recently taken action by the agent from its current state. The two methods of RL problems are model-free and model-based methods. In the model-free method, the agent enhances its policy without inferential knowledge of the model of the environment. That is, the matrix of the state transition probabilities is not needed. Whereas, in the model-based method, the agent learns the model of the environment by computing the matrix of the state transition probabilities and then enhances its policy to approach optimality. The model-based methods learn faster than the model-free methods because the information stored in their internal model is reused. However, the model-based methods are not generally used in practice due to their dependency on the initial environment model's accuracy and the larger size of storage costs and computations. Consequently, in both methods, the role of the agent is to maximize a discounted global reward received over time while finding a policy that maps states to actions. A policy \(\pi_{t}\) determines the learning behavior of an agent at a given instance of time \(t\). In RL, what is bad or good in an immediate sense is depicted by a reward function, whereas what is bad or good in the long run is depicted by a value function. There are two types of Q-value functions, which are the action-value function and the state-value function. The action-value function computes how good it is for an agent in a given state to take a given action. At the same time, the state-value function computes how good it is for an agent to be in a given state. The majority of the work carried out on RL to solve unicast routing problems in networks used model-free methods [17]. This is because they do not need the network's environment models, which are difficult to get as a result of their dynamically changing properties like the residual battery capacities, lifetime of nodes, etc., in WSNs. In RL, the quality of taking an action, \(A_{t}=a\) from a state, \(S_{t}=s\), is given using a Q-value, \(Q(s,a)\), which is known as the state-action-value function. The aim of finding a solution to an MDP is to learn an optimal policy that maps states to actions in order to maximize the cumulative reward. The aim of finding a solution to an MDP is the same as finding the Q-fixed points given by the Bellman equation as shown in Equation (2). \[Q^{\pi}(s,a)=r(s,a)+\gamma\sum_{\hat{s}}p(\hat{s}|s,a)\underset{\hat{a}}{ \max}\{Q^{\pi}(\hat{s},\hat{a})\} \tag{2}\]  
Figure 1: RL Model as an MDP [10].