# Self-Learning Power Control in Wireless Sensor Networks
## 5 Simulations
### Results
#### 5.2.2 Multi-Agent WSN  
The multi-agent WSN is composed of four node pairs distributed in a Manhattan grid topology (Figure 5b). The transmitters of each couple represent the agents of the QL-TPC protocol. Each agent observes a dynamic environment that is furthermore modified by the action of other agents. One action is a transmission that enables communication between two nodes but also creates interference with  
Figure 9: Performance in WSN with two nodes during the testing period (4 m node distance): average PRR (**a**); average latency (**b**); and energy consumption per bit (**c**). other nodes. In Figure 10, the outcome of the Q-learning process is illustrated. The distance \(d\) for each couple is equal to 2 m. Figure 10b presents the probability that the environment is in a certain state during a simulation, when the inter-arrival time \(\mu\) is equal to 25 ms. The most likely state is Q-state 4 that is further evaluated in Figure 10a for TX1, as illustrated in Figure 5b. The Q-value is showed, marking the separation of the trend among the phases of learning, convergence and testing, as detailed in Table 3. During the learning phase, the Q-value is in a transient period that starts with a rise and then continues with an oscillation. The agent explores the effect of every action on the environment using a high learning factor. Therefore, the choice of each action provides different variance of the oscillations. The oscillations are reduced in the next phase of convergence, since the learning factor is lowered. In addition, the exploitation of the best action is more frequent than the explorations. Indeed, the predominant contribution is marked and the wider oscillations are due to the small margin of exploration. In this phase, the curve converges to the maximum Q-value obtained by one action. The selected transmission power is equal to \(-32\) dBm and is then used during the whole period of the testing phase. In the latter phase, the network performance is evaluated. Similarly, other transmission power levels are selected for other states, as shown in Figure 10b. Figures 11 and 12 depict the results when \(d\) is equal to 2 m and 4 m, respectively. As expected, the performance generally deteriorates compared to the single agent system. This happens especially when \(\mu\) is equal to 25 ms. Indeed, considering the homogeneous case in Figure 11a, the PRR for every power level drops to around 98%. However, in Figure 11b, the latency is higher, starting with 11.2 ms using the lowest transmission power level and stabilizing to around 10.3 ms. Raising the distance between the nodes, the performance worsens, since the signal strength in SINR decreases and the bit error rate increases. Figure 12 reveals the decay of the performance in the homogeneous network at the lowest power levels. The combination of the attenuation and interference brings the PRR to drop and the latency to boost, while using low transmission power, between \(-35\) and \(-25\) dBm. For instance, the PRR drops by 31% and 40% in the case of the transmission power set to \(-35\) dBm and the lowest inter-arrival time, comparing Figures 9a and 11a, respectively, with Figure 12a. For the same case, in Figure 12b, the latency has a pick of 16 s. This is due to the higher packet generation rate with respect to the packet transmission rate, such as the average retransmissions and CCA attempts are increased to 1.5 packets and the average queue occupancy to 631 packets. The aforementioned scenarios present different environmental conditions than in the previous section. Figures 11 and 12 show how QL-TPC performs by adapting the agents' decision to different environment. The marks per inter-arrival time are four and correspond to the average value of each transmitter per performance parameter, at the selected average transmission power. The values of PRR, latency and energy per bit obtained with QL-TPC are similar to the homogeneous network ones per selected transmission power.