# Solving Task Scheduling Problems in Dew Computing via Deep Reinforcement Learning
## 7 Conclusions and Future Work  
This work proposed to distribute jobs in Dew environments using RL. By following a trial-and-error strategy, the RL agent learned how to assign jobs to devices effectively--outperforming existing methods by up to 77% in GIPS performance. Perhaps our most  
Figure 6: The plot shows the test performance (over 1000 testing instances) when the agent is trained using sequences of jobs sampled from the same distribution as the testing instances. Since some of the baselines’ lines overlap, we also included their performance in the legend. Note that the RL agent quickly learned a policy that outperforms existing methods in the unseen testing instances. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & **Train Set** & \multicolumn{4}{c}{**Upward Generalization**} \\ \hline Jobs: & 300–500 j & 300–500 j & 1000 j & 1500 j & 2000 j & 2500 j \\ \hline BPA & — & 0.31 & 0.34 & 0.38 & 0.40 & 0.42 \\ E-SEAS & — & 0.31 & 0.30 & 0.33 & 0.35 & 0.37 \\ RR & — & 0.30 & 0.30 & 0.31 & 0.33 & 0.35 \\ W-Rand & — & 0.30 & 0.30 & 0.32 & 0.34 & 0.37 \\ W-RR & — & 0.30 & 0.29 & 0.31 & 0.33 & 0.34 \\ \hline PPO & 0.54 & **0.55** & **0.52** & **0.47** & **0.44** & **0.43** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Upward generalization of policies that were trained using randomly generated job sequences in a Dew environment with 25 devices measured in GIPS. notable finding was that RL agents tend to learn policies that generalize well to unseen instances. This includes out-of-distribution examples, such as longer sequences of jobs than those seen during the training. These are promising results. Beyond Dew computing, it is interesting to see a real-world application where Deep RL generalizes well from only one training instance. Investigating whether this is a common feature of large-scale scheduling problems (or only a peculiarity of Dew computing) is one of the many research directions that this work opens. Another direction for future work is to study the performance of RL agents that are more _sample efficient_ than PPO. As we said, PPO is one of the preferred methods for solving problems that require generalization. However, PPO is also known for being data-hungry--which is why PPO needed millions of interactions with the environment to become a state-of-the-art scheduler in our experiments. Now that we know that PPO works well in practice, it makes sense to try other RL methods that are more sample efficient, such as off-policy methods [60] and model-based methods [61]. In this work, we focused on scheduling jobs in a fixed Dew environment (i.e., an environment where the set of available devices is fixed). However, in some Dew environments, it is not uncommon that devices enter and leave the network over time [62]. This further complicates the process of assigning jobs to devices. We note that extending our method to handle such cases is relatively straightforward. The key is to show the RL agent situations in which a device leaves or enters the local network during the episode. That said, training the agent in such a setting will likely require more computational power. Finally, it is worth mentioning that there is an important open question that we did not entirely address here. It is unclear how to evaluate the effectiveness of a scheduler in a Dew environment. There is no standard metric in the literature. Some metrics consider the GIPS [7], the number of jobs completed [31], the network latency [63], or the battery levels [64]. However, they all have advantages and disadvantages. In this work, we used GIPS because it pushes the agent to get the most out of the available resources. In order to maximize the GIPS, an optimal scheduler would keep all the available devices busy (if possible). Otherwise, the scheduler might improve its GIPS performance by assigning jobs to the IDLE devices. However, the disadvantage is that by doing so, the agent might decide to exploit the battery-dependent devices until their battery levels are close to zero.