# Self-Learning Power Control in Wireless Sensor Networks
## 4 Method
### Q-Learning Transmission Power Control (QL-TPC)
#### 4.3.2 Software Architecture and Flow Chart  
The TPC protocol, depicted in the software architecture in Figure 3 and flow chart in Figure 4, is implemented in 3 blocks: the IEEE 802.15.4 PHY and MAC layers, Database Manager (DBM) and QL-TPC. The former and the latter communicate with the DBM for gathering, saving and sharing data. The MAC layer from the standard is customized to interact with the two new modules, DBM and QL-TPC. Looking at Figure 3, the 802.15.4 MAC module indicates to DBM the data to convert into the state and reward for the algorithm through the Transmission Power Management Entity (TPME), using the TPME_WRITE.indiaction message. However, it requires from DBM the transmitpower to set for the next transmission via TPME_READ.request. DBM provides the power level in the TPME_READ.confirm message, that is then forwarded to the 802.15.4 PHY module by a PD_DATA.request message.  
The Q-learning algorithm and \(\epsilon\)-greedy strategy are implemented in QL-TPC, therefore the updated Q-values and the action strategy to pursue (i.e., random or maximum) are indicated through the Q-value Management Entity (QME), via the QME_WRITE.indication message to DBM. QL-TPC requests from DBM new state-action-reward triplet, when available, in order to update the Q-values, by sending QME_READ.request. Then, the reply is contained in a QME_READ.confirm message. DBM handles a database (DB) that saves the information obtained by the MAC and QL-TPC modules (i.e., an array of values containing the old and new Q-states, action taken and reward obtained; the Q-matrix; and the flag to indicate the action strategy) [54].  
The IEEE 802.15.4 MAC and PHY models and QL-TPC are two blocks of processes that run independently. Their cycle of operations are detailed in the flowchart of Figure 4. The MAC and PHY block starts when the transmitter has a packet in the queue ready for sending. In the case of the first packet to be sent, a random action is selected and the state at the initial iteration is the lowest (\(s_{0}=0\)). When a node has buffered a packet for transmission, it accesses the medium through Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) up to a standard maximum \(n_{CCA}\). If the CCA  
Figure 4: Flowchart.  
Figure 3: Software architecture.  
is successful, the packet is transmitted and an acknowledgement (ACK) is expected. If the ACK is not received, a copy of the packet is retransmitted, repeating for a maximum of \(n_{ret}\), times. If the \(n_{CCA}\) or \(n_{retr}\) are exceeded, the packet is dropped. The same transmission power is used for \(N\) consecutive packets, whereas the CCA and retransmissions are recursively averaged and rounded to the closest integer. After the transmission of the \(N\)-th packet, the reward \(r_{k+1}\) is calculated, as well as the CCA and retransmission values are associated to a state \(s_{k+1}\) in DBM. \(s_{k+1}\), \(r_{k+1}\) and \(s_{k}\) are inserted in one entry of the Packet Reception History (PRH) table in DBM, which is required by QL-TPC to update the Q-values. At this point, DBM checks a flag that indicates whether to select a random action or to exploit the action with the maximum Q-value. Such action is used for the transmission of the following \(N\) packets, setting the same power. The MAC layer communicates the chosen power level to the PHY level that is set for future transmissions. The QL-TPC process starts with the initialization of the \(Q_{0}(s,a)\) values to 0. Afterwards, the process waits for indications by DBM whether the PRH table is not empty. In that case, QL-TPC reads the values in the first entry of the table, updates the variables \(\alpha(k)\) and \(\epsilon(k)\) to referred values, and calculates the new value for \(Q_{k}(s,a)\). Subsequently, the \(\epsilon\)-greedy strategy is used to decide the action for the next round of transmissions. Both processes are repeated for the entire simulation time.