# Self-Learning Power Control in Wireless Sensor Networks
## 2 Background  
RL is one of the three big groups in Machine Learning together with supervised learning and unsupervised learning. In supervised learning, the system learns from labelled data, providing a map between input and known output in order to make associations for future data. In unsupervised learning, the data is unlabelled and the system learns how to classify it. Each class is defined by features that help the system to distinguish coming data. Instead, RL is used for self-learning systems in an unknown environment. The training is done either on a batch of stored data [37] or online, using real-time data [38]. In both manners, the data is obtained through observations, taking certain actions. Our protocol lies in the group of RL. Initially the devices do not have information about the environment, thus they adapt to any kind of scenario, learning through real-time data. RL is a solution to an MDP, whenever the environment respects the Markovian properties. The MDP is a framework of sequential tasks in instants of time, which are required to make decisions. Solutions of MDPs are obtained through different methods that depend on the time horizon of the system. If the system is operative only for a fixed period, the time horizon is finite and the MDPs can be solved by the Bellman optimality equation using either the dynamic programming approach or the value iteration approach. Otherwise, if the system is operative on an infinite time horizon, the solution methods are: value iteration, policy iteration, linear programming, approximation method and online learning [32]. The latter includes RL. RL involves two main entities: an agent and the environment (Figure 1). The agent is the learner and decision-maker, while the environment is an unknown entity that influences the agent's performance. The system strives to achieve a specific target; thus, the agent iteratively learns the actions to pursue such target, adapting to various circumstances. Particular attention is required for the definition of the agent and the environment, based on the target. For instance, the environment can be represented as anything that the agent cannot control. The latter observes the environment on an episodic basis, where the episodes are defined by the developer. At a given \(k\)-th episode, the status of the environment is represented by the value \(s_{k}\in S\subseteq\mathbb{N}\), where \(S\) is a set of states. The agent interacts with the environment by taking decisions, called actions \(a_{k}\in A\subseteq\mathbb{N}\), where \(A\) is a set of actions. The succession of state-action-state is formulated by transition probabilities. The system decides to select an action under a certain state following a policy, \(\pi_{k}(a|s)\). Actions can have either a positive or negative impact on the system in order to accomplish the target. Thus, the system provides a feedback or reward \(r_{k}\in R\subseteq\mathbb{R}\) to evaluate the effect of the actions, where \(R\) is a set of rewards. The reward is both a quality value and the goal of the agent. Given the fundamental parameters of the system, the Markovian property is defined as follows:  
\[Pr\{R_{k+1}=r,S_{k+1}=s|S_{0},A_{0},R_{1},...,S_{k-1},A_{k-1},R_{k}, S_{k},A_{k}\}=\] \[Pr\{R_{k+1}=r,S_{k+1}=s|S_{k},A_{k}\} \tag{1}\]  
Only the last state and action are necessary to know the probability of getting certain states and rewards in the next episode. Hence, it is not necessary to memorize all the past values but only the ones that happened in the last event. The total reward in the long term, or return of the system, knowing all the values obtained per episode, is calculated by a sum of rewards. To avoid the divergence of the series, the return is multiplied with the discount factor, \(0\leq\gamma\leq 1\), obtaining the discounted return \(D\):  
\[D_{k}=\sum_{n=0}^{\infty}\gamma^{n}\cdot R_{k+n+1} \tag{2}\]  
The discounted return can be calculated only after getting the rewards.