# Deep Reinforcement Learning for Selection of Dispatch Rules for Scheduling of Production Systems
## 5 Method Implementation and Evaluation
### Evaluation and Discussion  
The DRL agent was trained by exploring the use case's production environment. The trained model was then able to perform under different circumstances and obtain scheduling solutions. To evaluate the proposed method, the DRL scheduling algorithm was compared with random decision-making and dispatch rules. For testing, production scenarios were randomly generated with the following characteristics. * Workload: Random workloads were generated with the limitations described in Section 4.2. * Resource availability: In each episode, one line may be unavailable. This is a random event happening in the simulation. The implementation of the testing phase was realized with the use of Python as well. For random workloads, each algorithm should propose an action and be evaluated. An iteration included five steps (one for each algorithm). In each iteration, a production scenario was generated. The initial status of the production system, as well as the workload for each run, was randomly generated based on the production system requirements as addressed in Section 4.2. In the beginning, one or no machines was unavailable, while all tasks were pending and ready to be dispatched to resources. After this, each algorithm proposed its action, and the simulation run was initialized. The result of each run, which was the makespan returned by the DES model, was stored. Thus, for each scenario, one could identify the performance of each algorithm. In Figures 6 and 7, the moving average of the rewards from the DRL agent's actions is presented. The first figure shows the progress of the rewards' moving average for 30 measurements and the second one for 100 measurements. Each value is the average value of 30 or 100 production scenarios. As the figures show, the average reward tends to be larger in the last few steps of the training, starting at 0.8-0.85 and finishing at 1-1.05. This is a 19-20% reward improvement observed in the training procedure. From the training procedure, there are two main conclusions. The first one is that the DRL agent was able to train well and learn better actions over time. Starting from large makespans, on average, it was able to reduce the average scheduling makespan by 20%. In the same manner, a reward improvement is observed, which is inversely related to the makespan. The second observation is that the DRL agent does not follow the typical reward progress seen in other reinforcement learning problems. In other words, the reward does not start from zero and reach higher values at the end of the training. Initially, the training rewards are high, and this is because there is not a terminal state other than the production's completion. This means that the agent's reward cannot be negative or zero. It can only be proportional to the makespan achieved, as defined in the methodology. In this implementation, the agent does not learn how to navigate between states following actions in order to reach a terminal state and achieve its goal. From the beginning, the agent can  
Figure 7: Rewards during training (moving average of 100 values). perform any action and reach a terminal state. However, the agent learns how to navigate through these states in order to maximize the reward. Thus, despite the fact that the rewards during the first phase of the training are already large, the agent is able to find even better solutions to the problem. Lastly, the agent explores a vast environment, affecting its performance by necessitating the learning of problem-solving in a production system and the ability to generalize to achieve optimal outcomes across a variety of circumstances. One hundred testing scenarios were randomly generated in order to identify average behavior, since the randomness introduced by the production scenarios, as well as the scale of the problem, had to be adequately addressed in the results. In Table 3, the makespans achieved for each algorithm for the testing scenarios are presented.