# Solving Task Scheduling Problems in Dew Computing via Deep Reinforcement Learning
## 6 Experimentation
### Discussion on Generalization  
The previous results are encouraging. They show that RL agents can find ways to distribute jobs in Dew environments that are considerably better than distributing jobs using a heuristic method. However, it is still unclear whether the policy that the agent learned generalizes well: that is, whether that policy will work equally well on sequences of jobs that are different from the one used for training the agent. Indeed, it is possible that the agent is _overfitting_ the sequence of jobs that was used for the training. The agent may have even memorized a sequence of actions that leads to high rewards in our specific training instance--which is a common behavior in deep RL [54, 58]. Figure 5: Training performance on the fixed job experiments. In both cases, the RL agent learned to distribute jobs better than existing state-of-the-art methods. We note that most _machine learning_ models, including neural networks, assume that the training and test data are drawn from the same probability distribution in order to provide generalization guarantees [59]. In addition, the larger the training set is, the more likely it is that the model will generalize well. This behavior has also been observed in deep RL [50; 54; 55]. In short, the main conclusions from this body of work are two: (i) RL agents generalize better if they are presented with a diverse set of training instances, and (ii) the farther the testing instance is from the training distribution, the less likely it is that the agent will perform well (without been further trained). To investigate this topic, we generated a set of 1000 job sequences (of different sizes) to test the generalization capabilities of the policy learned by the agent. Specifically, we generated six test sets. Each test set contains 1000 job sequences of a certain size, which were generated using the job generation procedure from EdgeDewSim [21] (as described in Section 6.2.1). The size of the sequence depends on the test set. We created test sets with sequences in the following sizes: 1500, 2500, 3500, 4500, 5500, and 6500 jobs. Note that the agent's performance on the test sets with 1500 and 2500 jobs will tell us whether the RL agent is generalizing to unseen instances that were sampled from the same distributions as the training instance. In contrast, the test sets with more than 2500 jobs allow us to study out-of-distribution generalization, where the longer the job sequences are, the harder it will be for the agent to generalize well. Table 2 shows the performance of the RL agent (and the baselines) in each of the test sets. We tested two RL agents: PPO-1500 and PPO-2500. They are the same agent, but they were trained using the job sequence of 1500 jobs and 2500 jobs, respectively. Recall that we trained 16 agents per environment, so we are reporting the generalization performance of the agent that achieved the highest training performance among those 16 runs. The test performance was computed as the average GIPS across the 1000 testing instances. In addition, note that we are not further training PPO in any of the test environments. We are just running the learned policy of PPO-1500 and PPO-2500 and reporting their performance on each of the test sets. Overall, Table 2 shows two expected and two unexpected results. The first expected result is that the performance of PPO decreases from the training set to the test set. For instance, the performance of PPO-1500 decreased from 1.73 to 1.35 in the test set with 1500 jobs. This is a symptom of overfitting, which is expected since we trained PPO-1500 using only one training instance. The second expected result is that the performance of PPO tends to decrease as we test on longer job sequences. In fact, BPA starts to dominate when the job sequences have 5500 or more jobs.