## 6 Simulation  
We test the algorithms using the _RioBuses_ dataset [23], which collects the GPS trajectories of buses in Rio de Janeiro, Brazil. It contains over 12,000 units, moving over 700 bus lines through November 2014. In this work, we down-sample the dataset to 1000 buses, which is still a large use-case for an ON [24]. We generate ten distinct down-sampled setsto produce statistics. Nevertheless, the ONM representation allows simulating the entire dataset, even using desktop-grade hardware [8]. We run the algorithm in a scenario where all units emit two messages, one at 00:00 and another at 12:00, directed to a single fixed collection point located in one of the city bus terminals. Successful delivery is achieved when a message arrives within the same day. Messages underway are removed every night at midnight, but the learned protocol parameters are kept. To calibrate the algorithm parameters, we simulate the first seven days of the trace. The results are shown in Figure 4. The delivery rate is computed at the end of the corresponding day. The reduction in delivery rate on days 4 and 5 corresponds to the weekend, with reduced bus services (1 October 2014 was a Wednesday). Figure 4a shows the impact of the \(\alpha\) parameter on the learning process (see Equation (7)). The algorithm was configured with eight copies per message. The best learning is achieved with a very aggressive \(\alpha=1.0\). As described in Section 5, the Q-parameters are learned per time slot to allow for learning different patterns throughout the day. On the other hand, more slots mean fewer encounters per slot, which slows down learning. The impact of the number of time slots is shown in Figure 4b. It can be seen as the single-slot scenario learns fastest, as it considerably outperforms the alternatives at the end of the first day. Nevertheless, 6 or 12 slots finally catch up and outperform after day 3. Having 24 slots degrades performance, possibly because the number of encounters per slot is too small to sustain effective reinforcement learning. In _rl4 dtn_, reinforcement learning is performed by observing the propagation of data messages through the network. Thus, having multiple replicas of a message improves the learning process. Additionally, it improves the delivery rate, as it is enough for a single message copy to arrive for a successful delivery. This effect can be seen in Figure 4c. On the other hand, producing multiple copies of messages increases the data transmitted in the network. This effect will be discussed in Section 6. Figure 5 compares _rl4 dtn_ delivery performance with _BSW_, _PRoPHET_ and the reference Q-learning protocol. The average over ten simulations with different buses samples is shown, with the corresponding interquartile range. The number of time slots used for _rl4 dtn_ and Q-learning is 6, the number of message copies is 8, and the \(\epsilon\) parameter driving the Q-learning exploration is set at 1.0 at the beginning and then reduces linearly to 0.5 on the last day. _BSW_ underperforms considerably. _PRoPHET_ suffers the most from the reduction in mobility through the weekend. It can be seen that _rl4 dtn_ offers a better delivery rate. Furthermore, the performance difference between Q-learn and _ld4 dtn_ shows the impact of using the ONM as a base for the learning process. Figure 4: Learning model parameters’ impact on delivery rate. (a) Learning-rate parameter \(\alpha\); (b) Number of time slots; and (c) Number of message copies. Figure 6 displays a 1:50 sample of the Q-values as they evolve through the system's learning. This shows the convergence behavior of the Q-learning process. Notice that the Q-values estimate the cost of an opportunistic path; in our scenario, this cost is the hop count. It can be seen that the system starts with Q values up to 6, and from day three, it settles on many paths of one bus hop, considerably less but roughly similar paths of length 2 and 3, and very few of length 4. The gap with almost no deliveries in the early morning can be seen.