## 5 Algorithm
### States and Actions  
In our approach, each network device is an agent and maintains the Q-table for all the associated system states and possible actions from its point of view. Because the optimum action to choose depends on the time at which the agent is, we divide the scenario's duration into time slots (see Figure 3a) and then use each slot from every device as a potential state for the RL process. Thus, a state is defined by a pair \((\mathit{networkdevice},\mathit{timeslot})\), and actions are the possible state transitions (see Figure 3b). Notice that actions are not the edges from the ONM; ONM edges connect ONM nodes, which are defined by a device and an encounter time stamp. RL actions connect states \begin{table}
\begin{tabular}{c c} \hline \hline Agent & Mobile device. \\ \hline \multirow{2}{*}{State} & A (device, timeslot) pair, where the time is divided into set \\  & time slots. \\ \hline \multirow{2}{*}{Action} & Connect two states, either on the same device (store data) \\  & or another (forward data). \\ \hline \multirow{2}{*}{Evaluation time} & Encounter between devices (i.e., ONM node). \\ \hline \multirow{2}{*}{Q-Metric and task} & Minimization of a distance metric. \\ \hline \multirow{2}{*}{Determinism} & States and actions are non-deterministic. \\ \end{tabular}
\end{table} 
Table 1: Reinforcement learning structure for rlddtn algorithm. Figure 3: rlddtn learning model. (**a**) The ONM model with the time slots and the learning agents marked in red; (**b**) Q-learning states ad actions. Each state is associated to a device and time slot; (**c**) The Q-tables for all devices. Figure 2: Models for routing over a graph using Q-learning. The learning agents are marked in red. Navigation problem. (**a**) The agent is the message, actions and states are graph nodes; (**b**) Routing Problem. Each node is an agent, the actions are next-hop nodes. defined by a device and a discretized time slot. The resulting Q-tables are shown in Figure 3c. Paralleling the ONM edges, there are two possible classes of actions. In one, the destination of the state transition is a state associated with the same origin device but at a later time slot; this represents storing data between different time slots. In the other scenario, the destination is a state with a different device but in the same time slot; this represents a data-forwarding action. Notice that the state space can be seen as the time discretization of the ONM graph; multiple graph nodes from a device that occur in the same time slot map to a single state. For example, in Figure 3a, there are two encounters by node \(B\) in time slot \(T_{2}\). Both encounters are represented by the same state \((B,2)\) in Figure 3b. Furthermore, an action can capture the effect of multiple ONM edges. For example, all encounters between a pair of devices that happened in the same time slot, each modeled by a separate ONM _copy edge_, are represented as the same action: forwarding between the involved devices in the given time slot. Actions that connect subsequent states within the same device are called _survival actions_, the idea being that when the link exists, data have survived between time slots. Notice that a _survival action_ is available on all states except for the last state associated with a device. The Q-learning is performed each time a forwarding decision is made, this is on each ONM graph node. If there are multiple forwarding decision evaluations within a time slot, they all contribute to the computation of the time slot's Q-value. The time slot size is a configuration parameter for r1 dtn. In one extreme, when there is a single time slot that spans the whole mobility scenario, the r1 dtn model behaves as a single, static connectivity graph. In this case, there is a single _forwarding action_ computed for connecting each pair of nodes, and there are no _survival actions_. The Q-value for the actions is computed from all the forwardings made through the scenario, and no temporal distribution of encounters is captured.