## 5 Algorithm
### Opportunistic Route Exploration  
An essential part of the behavior of ML systems is _exploration_; _rl4 dnt_ explores the space of solutions by two mechanisms. The first is the standard \(\epsilon-greedy\) policy, where, at the action-selection time, there is a probability \(\epsilon\) of selecting a random action instead of the known best (the one with minimum associated Q-value). The second exploration mechanism is to generate multiple message copies, where each one is routed independently. This replication is managed using _Binary Spray and Focus_ (_BSF_) [22]. This is similar to the _Binary Spray and Wait_ technique mentioned in Section 2, differing only in its behavior when the copies count reaches one. In _BSW_, this single copy is held locally and delivered only to the destination device if it is met directly. In _BSF_, this copy can be handed over to another device and then remain hopping from device to device until meeting the destination. Notice that if the emitting device sets the _number of copies_ parameter to one, the algorithm behaves as a pure forwarding algorithm. As usual in Q-learning, the exploration parameters can be reduced during the system's lifetime, favoring exploration in the early stages and exploitation in later stages to improve performance and convergence. The pseudocode for _rl4 dnt_ is shown in Algorithm 2. The entry point is on line 16, which is called on every encounter after updating the ONM model when a forwarding decision must be made (we assume we are handling a single message _msg_). For this purpose, a list of candidate destination states is built on lines 17-24. One of the candidates is the _survival_ action (line 18). As mentioned before, this can be directly obtained from the _rl4 dnt_ model if already available from previous evaluations. If not, it can be a placeholder, as this action is known to exist. Besides the survival action, all copy actions are added to the list of candidates (line 19). If one of the neighboring agents is the target for the message, the message is delivered, and no further processing is made. Once a list of candidates is built, line 25 selects a copy or _survival_ target using the \(\epsilon-greedy\) selection function (lines 3-7). After this, line 26 updates the Q value. That is done on lines 8-18, applying Equation (7). In our case, line 12 computes immediate hop count costs from Table 2.  
If the selected action was a forwarding, _Binary spray and focus_ is used on lines 27-33 to either forward half message copies if there are more than one (line 28) or migrate the single available copy to the new device (line 31). As a baseline, we implemented a straightforward Q-learn algorithm not based on ONM. Unlike _rl4 dnt_, only actions associated with forwarding are maintained (no _survival_ actions), and forwarding is performed if the best candidate has a better Q-evaluation than its own. In Algorithm 2, this implies the removal of line 18, the condition on line 27, and \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Time Edge** & **Copy Edge** \\ \hline Latency & \(T_{dest}-T_{source}\) & 0 \\ \hline Hop count & 0 & 1 \\ \hline Power consumption & 0 & \(Pwr(source,dest)\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Immediate edge costs \(R\) for rl4 dnt for various distance definitionsthe substitution of the action selection function on line 7. The message-copies handling is identical to \(rl4 dtn\). This allows us to evaluate the impact of using ONM to learn explicit actions for keeping messages. Any improvement from Q-learn to \(rl4 dtn\) can be assigned to the use of the ONM model. Please, notice that the computational complexity of both algorithms is roughly equal. Both are evaluated in the same situations (at forwarding decisions), the only difference being that \(rl4 dtn\) has one more transition to consider: the explicit storing action.