## 5 ACTOR Design
### The Building Blocks  
ACTOR differentiates from the previous works [10; 21; 24] as follows: Primarily, ACTOR aims to improve the throughput by mitigating the density of the networks and the hidden terminal problem. Consequently, the energy consumption is also reduced due to lower transmission power settings and fewer retransmissions of packets. Second, the selection of the transmission power settings is based on the UCB algorithm. Also, the non-stationary extension (discounted UCB) allows ACTOR to adapt to the dynamism in the links. ACTOR uses lightweight integer arithmetic to simplify the complexities of handling floating point operations. Third, ACTOR updates the ETX as a routing metric to avoid excessive parent changes while the algorithm figures out the best transmission power. Fourth, ACTOR integrates mechanisms to take care of parent-child links while adapting the transmission power so that parents do not neglect the link to children and selfishly reduce their transmission power. When tuning transmission power based on the reception ratio of the packets, nodes encounter the exploration-exploitation trade-off. That is because, the more often a transmission power is used, the more confident one can get about the reliability of that specific transmission power. The drawback of this exploration is that trying sub-optimal transmis  
Figure 3: Measured PDR for samples of packets being sent with different transmission power settings at specific distances. sion power settings only to evaluate them may cause some packets to be lost. However, a certain level of exploration is inevitable since the agent (node) learns about the environment based on the rewards (acknowledgments) that correspond to the actions (transmission power settings). The ultimate goal of the agent is to find the optimal action with the minimum exploration. Before sending actual packets, the reliability of the links can only be determined by measuring the signal strength and the noise ratio. Despite a small memory footprint, RPL networks are expected to scale to thousands of nodes. Hence, the protocol limits the number of neighbors in its routing table. In non-storing mode, only the root node keeps track of the downward routes. Both the storing and non-storing modes of RPL favor collection-based traffic. We exploit these design choices in the transmission power control to limit the exploration to a reasonable level. This leads to faster exploration and reduces the size of the routing table, and fewer data packets are sacrificed to evaluate the transmission power settings. Algorithm 1 summarizes the algorithm used by ACTOR to select the transmission power. Nodes initialize their neighbor table by setting their transmission power to the maximum and adding the neighbor to the table if required. After sending each packet, nodes update their action values. In this case, action values are the ratio of acknowledged packets for each transmission power and parent. Then, they select the action (transmission power and parent) that maximizes the UCB (as defined in Section 5.3). This selection only considers the upward link, but each node must also meet the requirements of its children. In ACTOR, children declare their demands to their preferred parents using a bit in the data packet's header. Each node uses a transmission power that (i) satisfies its children and (ii) satisfies the UCB requirements.