# Self-Learning Power Control in Wireless Sensor Networks
## 4 Method
### Q-Learning for Multiple-Agent Systems  
When the system is composed by multiple agents, we adopt a Dec-POMDP model. Each agent is independent and interacts with a common environment, as depicted in Figure 2. Having independent learners, the complexity of the algorithm is proportional to the number of states and actions. Otherwise, learning in a fully joint state-action space, the agents need the knowledge of other agents' decisions (i.e., actions or actions and rewards per transmitted packet) and keep track of action values for each one. If the shared information involves both actions and rewards, the complexity increases as \(c\sim n_{s}\cdot n_{a}^{n_{ag}}\), where \(n_{s}\) is the number of states, \(n_{a}\) the number of actions and \(n_{ag}\) the number of agents [52].  
The system follows the principles of game theory, such as the agents are players of a game (i.e., a mathematical model) that obtain a payoff by taking simultaneous actions [53]. The payoff is dependent to the target of the players. If they aim at improving only their own performance, the game is called competitive. Otherwise, if they gain a positive reward by helping each other, the game is cooperative. Moreover, if the players share the same target, they play a common interest game. In our model, the sensor nodes play a cooperative, common interest game, since the target includes transmission power reduction for interference mitigation and energy savings. At the same time, the nodes address the maximization of the PRR, over a window of \(N\) packets. At the end of a game, the system reaches an equilibrium, such as a combination of actions per player. The equilibrium is reached after an exploration phase, where the agents try different actions and get a reward from the environment. Therefore, the payoff of one agent is influenced by the decision of every component of the system. The optimal equilibrium is achievable if one agent knows exactly the actions and rewards of the other ones, while taking an action itself. This is not practicable in WSNs, due to an excessive increase in the complexity and memory requirements of the system, which would require continuous, instantaneous transfer of information via wireless communications. In Dec-POMPD, the grade of complexity to find the optimal solution is NEXP-complete, which means that the solution is provably not achievable [32].  
Figure 2 illustrates the interaction of N agents with a common environment. Each agent \(i\) has a separate counter \(k_{i}\) per episode, with \(1\leq i\leq N\), as well as its own 3-tuple parameters: state \(s^{i}_{k_{i}}\), action \(a^{i}_{k_{i}}\) and reward \(r^{i}_{k_{i}}\). The action \(a^{i}_{k_{i}}\) influences the environment in conjunction with the simultaneous actions \(a^{j}_{k_{j}}\) of the other agents \(j\), with \(1\leq j\leq i-1\) and \(i+1\leq j\leq N\). The union of these actions forms the joint action \(\overline{a_{k}}\). Among the actions \(a^{j}_{k_{j}}\), one agent can be inactive and does not trigger its own cycle for Q-value updates. At the end of the episode \(k_{i}\), the environment reveals the new state \(s^{i}_{k_{i}+1}\) and reward \(r^{i}_{k_{i}+1}\) to the agent \(i\).  
Figure 2: Multi-agent reinforcement learning schema.