## 5 Discussion  
As learning progresses, the rewards in Figure 13(a) and cumulative rewards (score) in Figure 13(b) tended to increase for most methods, and the total machining completion time in Figure 13(c) gradually decreased. However, when the learning steps exceeded a certain value, the total machining completion time increased, which caused performance degradation. In particular, Figure 13(c) shows the following results. The performance of the PPO algorithm in terms of the total machining completion time was better than that of the DQN algorithm. The suggested use of a subspace outperformed the use of the entire space. It showed that scheduling by a negative reward was impossible as the total processing completion time continuously increased. The performance of the scheduling system and the learning effect were higher when predicted rewards were used instead of actual occurrence rewards.  
The experimental results for the performance comparison are shown in Tables 4 and 5 in Section 4. Table 4 summarizes the results of the performance comparison of the experiments in the first environment. Among the three existing heuristic methods, the proximity-first scheduling method exhibited the best performance. The results of the reinforcement learning algorithm revealed that it performed very poorly in the early stages of learning. However, its performance gradually improved as the learning progressed. In agreement to the previous learning results, the performance degraded on excessive learning. When the PPO algorithm was applied to the entire environment, the peak performance was achieved after 40 million time steps. Subsequently, the performance did not improve further. The partial environment experiment showed improved completion times overall. The performance was achieved at 25 h, 36 min, 11 s at 30 million time steps. Comparing the reward forms, a negative reward could not minimize the total processing completion time because scheduling was impossible. Therefore, the inverse reward method established in this study was confirmed to achieve a better scheduling performance than the other methods. Finally, when applying the reward prediction method, i.e., the final method developed in this study, the best performance time was 24 h, 13 min, 38 s. This was approximately 59.90% better than that of the existing proximity-priority method. This result was achieved with a considerably shorter learning time compared to those used in the other methods.  
Table 5 lists the results of the performance comparison experiments in the second environment. The FIFO method showed the best performance for heuristics in this environment. The second experiment also showed the best performance when using the subspace and inverse/prediction reward method developed in this study. With 28 h, 5 min, and 46 s performance at 20 million time steps, this was an 83.90% improvement over 51 h, 40 min, and 10 s for FIFO, the best of the traditional heuristics. We confirmed the performance by comparing it with the experimental results of existing studies compared to heuristic methods. The results of our experiment showed a higher improvement than the improvements of existing studies. The performance improvements compared to the heuristic methods in existing reinforcement learning studies were 14.1-15.9% [32], 2.6-26.8% [33], and 12.6-22.3% [34]. Therefore, we conclude that the method established in this study is effective in solving real-world scheduling problems.  
Our work did not address exceptional situations such as machine failures or delays during work. For future research, the following contents can be considered. If the processing time is prolonged by a delay during the processing, follow-up measures are required to compensate for the previously paid reward. For a large difference between the reward calculated in advance and the actual processing time, it will be possible to calculate the actual reward by excluding the additional time from the reward. The broader consideration of the exception situations will be the focus of future investigations.