# Self-Learning Power Control in Wireless Sensor Networks
## 6 Experiments  
In this section, we provide the experimental setup and results of QL-TPC for a point-to-point communication. The system is the same as in Section 4.1, but the setup is different than the one used for the simulations. We use TelosB sensor nodes with Contiki OS. TelosB has 48 kB of program flash, 10 kB of data RAM and 1 MB of external flash. The nodes are connected to a cabled test-bed, which is formed by BeagleBone Black (BBB) devices linked via Ethernet to a server, through a Virtual Private Network (VPN). Each node is connected via USB to one BBB. Using the test-bed, we are able to store and analyse reliably the traffic that is generated, transmitted, received and lost through the ether. The TelosB nodes run the Contiki program that implements the QL-TPC protocol. In Contiki, we use the unicast routing protocol in the network layer, and Contiki-MAC in the MAC layer. The latter is an asynchronous radio duty cycling protocol, which has been created following the characteristic of the IEEE 802.15.4 standard. In Contiki-MAC, the radio is in sleeping mode by default (i.e., the radio is off) and it turns on every 0.5 ms for 0.192 ms. The transmitter sends copies of the same packet every 0.4 ms. Using this procedure, the nodes save energy and, at the same time, are able to communicate [59]. The program allocates memory in the device as follows: around 28 kB for the executable code, 118 bytes for the initialized data and 6.8 kB reserved for uninitialized data. The former is written in the ROM, the second in the EEPROM and the latter in the RAM. The BBBs use a Python program that reads the data from the TelosB and forwards it to a central server via a RabbitMQ messaging system. The format of the data from the TelosB to the server is JSON. The server runs a Python application that parses the JSON data and makes a CSV file, which is used for data analysis in MATLAB. The experiments are performed in the lab. The nodes are distant 8 m between each other. The choice of \(\epsilon\) in time follows an exponential function \(\epsilon=\lambda^{N}\) with \(\lambda=0.998\). The value for \(\alpha\) and \(\gamma\) is constant and set to 0.8. The number of actions are 8, which correspond to the transmit power levels of the TelosB. The state is determined based on the number of retransmissions, number of CCA attempts and the latency as follows:  
\[s_{k}=L^{2}\cdot\|\overline{cca_{k}^{q}}\|+L\cdot\|\overline{ret_{k}^{q}}\|+ \overline{lat_{k}^{q}}, \tag{14}\]  
with \(L=4\) the number of quantization levels, \(\overline{cca_{k}^{q}}\), \(\overline{ret_{k}^{q}}\) and \(\overline{lat_{k}^{q}}\) the average quantized number of CCA attempts, retransmissions and latency of the \(k\)-th episode, respectively. The quantization association is shown in Table 4. The latency is classified in low, medium and high, based on the values obtained during the experiments. They are associated to the range equal to the cumulative difference between the maximum and the minimum, divided by 3. When a packet is lost, the latency is NA and is ignored for the average in the window \(W\). Finally, the reward is calculated as the PRR over \(W\), in its continuous form. The experiment is 750 min long. The inter-arrival time of the packet generation is equal to 3 s. The power level used to send ACKs is constant and set to 7. The size of the Q-matrix is equal to 2.048 kB, given that the matrix contains an amount of float numbers equivalent to the multiplication of \(n_{s}\cdot n_{ptx}\). The values of the parameters involved in the experiments are listed in Table 5.  
In Figure 16, PRR and Q-values rise over time, showing that the system learns and improves its performance. When the system is mostly exploring (i.e., \(\epsilon\gg 0\)), PRR is unstable, but as \(\epsilon\) drops, PRR stabilizes between 90% and 98%. PRR never reaches 100%, because the system keeps exploring, since \(\epsilon\)