# Self-Learning Power Control in Wireless Sensor Networks
## 5 Simulations
### Setup
It decides whether to switch on the radio for either transmission or reception, or to switch off the radio and keep it idle. Depending on the status of the radio, the device supplies different currents, and thus consumes dissimilar energies. The energy is calculated as \(E=V\cdot i(rs)\cdot T\), where \(i(rs)\) is the current used during a period \(T\) in a certain radio status \(rs\), and V is the supply voltage. The possible statuses are: TX_ON and RX_ON, if either the transmitter or the receiver are turned on; SWITCH, as a transient mode, when the radio is switching between TX_ON and RX_ON. The radio is constantly in the receiving mode, RX_ON, and switches in the transmitting mode, TX_ON, only when there are pending messages to be sent. The finite state machine of the energy model is shown in Figure 6. The specifications for the supply voltage and current at each radio status are referred to the AT86 RF233 low power  
Figure 5: WSNs with Manhattan grid topology: (**a**) single agent—one wireless communication without interference; and (**b**) multi-agent—four interfering pairs. transceiver [58] and listed in Table 2. The current in the transmission mode, \(i_{tx}(p_{tx})\), follows a linear current model taking the transmission power as an argument:  
\[i_{tx}(p_{tx})=\frac{p_{tx}}{V\cdot\eta}, \tag{12}\]  
where \(\eta\) is the efficiency of the amplifier that is adapted to obtain the same referred value for the power equal to 0 dBm, as in [58]. Therefore, \(\eta\) is equal to 2.8%. The supply voltage is equal to 3 V.  
The WSN is analysed in two different cases: HG and QL-TPC. In both cases, the transmission power for the ACKs is selected at random, uniformly distributed among the available power levels, at the beginning of one simulation and kept constant. In the HG case, all nodes use the same amount of transmission power, among the 20 available levels. The lowest and the highest transmission powers are equal to \(-35\) dBm and 10 dBm. For each power level, the performance parameters are averaged by \(R=10\) runs of one simulation. Each run uses different sub-stream of pseudo-random numbers, within the whole stream generated by one seed in NS3. The random variables are: power level selection for ACKs, packet generation time, fading per link and packet drop. In the QL-TPC case, all the nodes run the Q-learning algorithm. Each node obtains one value per parameter related to one power level. Such power level is obtained for a specific node, during the testing phase, by a weighted average of the transmission power used for all the \(R\) simulation runs. Therefore, the average transmission power in dBm by one node is calculated as follows:  
\[\vec{F_{tx}}=\frac{\sum_{r=1}^{R}\sum_{l=1}^{n_{pts}}p_{tx,l}^{r}\cdot n_{l}^{ r}}{\sum_{r=1}^{R}\sum_{l=1}^{n_{pts}}n_{l}^{r}}, \tag{13}\]  
where \(p_{tx,l}^{r}\) is the transmission power in dBm, corresponding to the \(l\)-th power level and \(r\)-th simulation repetition, and \(n_{l}^{r}\) is the number of occurrences of \(p_{tx,l}^{r}\) usage. In this case, there are two more random variables to consider: \(\epsilon\)-greedy strategy and power level in the exploration phase. The performance is analysed through three parameters: PRR, latency and energy per bit. PRR is a ratio between the acknowledged and transmitted packets over a window of \(N=10\) packets; the latency is the time \begin{table}
\begin{tabular}{c c c} \hline \hline
**Radio status** & **Parameter** & **Value** \\ \hline TX\_ON & \(i_{tx}\) & 11.8 mA @ 0 dBm \\ RX\_ON & \(i_{rx}\) & 11.8 mA \\ SWITCH & \(i_{switch}\) & 6 mA \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Current specification of the AT86 RF233 transceiver per radio status. Figure 6: Finite state machine of the energy model. difference between the generation and the acknowledgement of a packet; the last term is the energy consumed per transmitted bits. The simulations are divided into seven periods of 600 s and one of 1800 s, as shown in Table 3. The simulation time is divided in three phases: learning, convergence and testing.