# Solving Task Scheduling Problems in Dew Computing via Deep Reinforcement Learning
###### Abstract  
Due to mobile and IoT devices' ubiquity and their ever-growing processing potential, Dew computing environments have been emerging topics for researchers. These environments allow resource-constrained devices to contribute computing power to others in a local network. One major challenge in these environments is task scheduling: that is, how to distribute jobs across devices available in the network. In this paper, we propose to distribute jobs in Dew environments using artificial intelligence (AI). Specifically, we show that an AI agent, known as Proximal Policy Optimization (PPO), can learn to distribute jobs in a simulated Dew environment better than existing methods--even when tested over job sequences that are five times longer than the sequences used during the training. We found that using our technique, we can gain up to 77% in performance compared with using human-designed heuristics. D 2022 2022 202211 11 July 20221 15 July 20222 202211 July 2022211 July 2022211 July 2022211 July 202221 July 2022211 July  
To accomplish this, the agent improves its current policy (a mapping from observations to actions) by learning from its past experiences. Powered by deep learning, RL agents have been used to solve complex decision-making problems across different research areas, from robotics [10] to conversational agents [11] and drug discovery [12]. Here, we propose to let an RL agent learn how to effectively distribute jobs in a Dew environment. Previous works have explored the use of RL for distributing jobs in Edge and Cloud computing [13; 14; 15; 16]. However, RL has not been tried in Dew computing. Dew computing has its own particular challenges that do not normally arise in Cloud computing or Edge computing. For instance, in Dew computing, some of the devices might run out of battery or users might start interacting with them. In addition, existing works have not studied whether RL agents can learn policies that generalize well to new situations. Our work shows that deep RL agents can learn to offload tasks better than state-of-the-art heuristic methods, even when tested in previously unseen situations. The main contributions of our work are then as follows. First, we propose to use Deep RL to distribute jobs in a Dew environment. By Deep RL, we refer to a family of RL agents that use deep neural networks to encode policies. In contrast to tabular RL (such as Q-learning [17] or SARSA [18]), Deep RL algorithms can learn policies that generalize to unseen situations and solve problems with continuous state spaces--as is the case in Dew computing. Specifically, we used a Deep RL agent, called _Proximal Policy Optimization (PPO)_[19], to offload tasks in a _fixed_ Dew environment (i.e., a Dew environment with a fixed configuration of devices). By following a trial-and-error strategy, the agent learns about the features of each device and how to assign jobs so that they are completed as soon as possible. Our second contribution is to develop an interface that connects OpenAI Gym [20] with a state-of-the-art Dew computing simulator [21]. This interface allows us to quickly test different RL methods for task offloading in Dew environments. In fact, our interface allowed us to test another deep RL method, called A3C [22], but its performance was considerably worse than PPO. Finally, we empirically demonstrate that by constantly showing new situations to the agent, the agent learns to generalize: that is, to appropriately distribute sequences of jobs that arrive in patterns and sizes that the agent has not seen during training. As a brief summary of our empirical findings, we discovered that RL agents can learn to effectively distribute jobs in fixed Dew environments--largely outperforming state-of-the-art heuristics with respect to the number of instructions per second that are executed in the environment. To do so, however, the agent must be trained for a long time. As such, there is a trade-off.