# Self-Learning Power Control in Wireless Sensor Networks
## 3 Related Work  
As explained in the previous section, RL is one of the possible solutions to MDP. Therefore, we firstly provide a deep investigation to all the works that have addressed MDP for power control with offline solutions, and then we focus only on those that have considered RL. [PERSON] et al. categorize works in the literature per MDP extension and application in WSNs [32]. Among the applications considered in their survey, the one that is related to our work is resource and power optimization. Within the group of works under such application class, the methods use the classic MDP, except for few that are based either on SG or POMDP [39; 40]. Furthermore, only one approach based on online learning (i.e., RL) [33] is reported, where the difference against the other solutions is that the system does not know the environment. For instance, [PERSON] et al. apply SG to CDMA ALOHA networks for the selection of the Signal Noise Ratio (SNR) threshold, in order to decide whether to transmit or wait in a determined time slot [39]. The nodes know in advance the channel status before transmitting. The results show the variation of the threshold and throughput with respect to the number of nodes in the network. The proposed method is compared with two cases where there is no control and no channel information respectively. Instead, [PERSON] et al. apply POMDP in their system, which is formed by one transmitter node (i.e., the agent) that interacts with a static environment [40]. The wireless channel is without a fading model, and the interferers have only one power level available for transmission. Since the agent observes partially the environment, it creates belief states as an estimation of the environment status (i.e., the interferers are either idle or transmitting). The actions that the agent can take are either a transmission with low power or high power, wait idle or listen to the channel. The policy to take one action under one belief state is set by fixed transition probabilities. The agent gets either a positive reward, if a packet is received, or a negative one otherwise. The authors show how the computational time varies by increasing the number of states and actions. Other works presented in the survey [32] use MDP [34; 35; 36; 41]. An industrial application is studied by [PERSON] et al., where the state information of one plant is transmitted by a sensor node to a controller [34]. The goal is to minimize the transmission power and the state estimation error at the controller side. The optimal power is obtained following an MDP formulation and solving the Bellman equation. However, given the onerous calculations that are needed to solve the Bellman equation, only the expression of the optimal power is provided. Then, the approximate dynamic programming is used to define the suboptimal power policy, which is evaluated through numerical simulations. The results show the transmission decision (transmit or not) with respect to two plant states and the channel gain. TPC is also used for neighbour discovery, minimizing the energy consumption [35; 36]. Unlike the work of [PERSON] et al. [35], the one of [PERSON] [36] considers the energy also in listening mode for the energy consumption calculation. Both approaches model the problem as MDP and they solve it offline via linear and dynamic programming. In the results, the authors analyse through numerical simulations the average energy consumption to discover the neighbours related to the density of nodes in the network. [PERSON]'s results are compared against two schemas: an ideal schema of perfect knowledge and a simple one, where the power is doubled [35]. Instead, [PERSON]'s results are compared against the optimal and suboptimal cases [36]. A centralized approach for energy savings is introduced in the literature, preventing the use of certain transmission power levels while a device is in a specific battery level range [41].