## 3 Proposed Method
### Common Experimental Setup  
The server specifications and library version used in the experiments are listed in Table 1. The hyperparameter sets for experiments using the DQN and PPO algorithms are reported in Tables 2 and 3, respectively.  
The simulation experiments replicated an actual production line. Generally, these experiments were conducted assuming that a total of 20 machines were arranged in a line mixed with three processes, _C1_, _C2_, and _C3_. In this study, basically, the experiment was conducted in a large environment with a problem size (number of machines \(\times\) number of jobs) of 20 \(\times\) 600 or more. To confirm their effectiveness in different environments, the experiments were conducted in two environments. For the first experiment, the numbers of _C1_, _C2_, and _C3_ machines were 8, 8, and 4, respectively. The processing times for the three processes were set as 800, 800, and 400 s, respectively. The input/return operation time of the material was set as 17 s for all processes. Assuming the trays to be exhausted are of 10 layers and that 9 additional materials are supplied, the objective was to process the material from a total of 100 layers. The size of the raw material buffer was limited to 6, and the size of the _C1_, _C2_, and _C3_ process completion buffers was limited to 4, 4, and 6, respectively. The RTU speed was set as 1.2 m/s. The aim of this study was to process 600 raw material units (100 layers \(\times\) 6 pieces) into 600 process units after executing the _C1_, _C2_, and _C3_ processes.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Resource** & **Version** \\ \hline OS & Ubuntu 18.04.3 \\ CPU & Intel® Xeon® Silver 4210R CPU @ 2.40 GHz \\ GPU & NVIDIA GeForce RTX 3090 \\ PyTorch & 1.8.1 \\ CUDA & 10.2 \\ CuDNN & 8.11 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Experimental environment.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline learning rate & 1 \(\times\) 10\({}^{-4}\) \\ discount factor & 9.9 \(\times\) 10\({}^{-1}\) \\ buffer size & 1 \(\times\) 10\({}^{6}\) \\ batch size & 32 \\ \hline learning time step & 1 \(\times\) 10\({}^{8}\) \\ epsilon greedy probability & 1.0\(\rightarrow\) 5 \(\times\) 10\({}^{-2}\) \\ target network update cycle & 1 \(\times\) 10\({}^{4}\) \\ Q value network layer/number of nodes & 2/64 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameter set in experiments with DQN.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline learning rate & 3 \(\times\) 10\({}^{-4}\) \\ discount factor & 9.9 \(\times\) 10\({}^{-1}\) \\ buffer size & 64 \\ learning time step & 5 \(\times\) 10\({}^{7}\) \\ clip range & 1 \(\times\) 10\({}^{-1}\) \\ target network update cycle & 1 \(\times\) 10\({}^{4}\) \\ policy network layer/number of nodes & 2/64 \\ value network layer/number of nodes & 2/64 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Hyperparameter set in experiments with PPO.  
For the second experiment, the numbers of \(C1\), \(C2\), and \(C3\) machines were 8, 6, and 6, respectively. The processing times for the three processes were set as 800, 600, and 600 s, respectively. The input/return operation time of the material was set as 20 s for all processes. Assuming the trays to be exhausted were of 10 layers and that 9 additional materials were supplied, the aim was to process material from a total of 100 layers, similar to the first experiment. The size of the raw material buffer was limited to 8, and the size of the \(C1\), \(C2\), and \(C3\) process completion buffers was limited to 6, 6, and 8, respectively. The RTU speed was set as 1.2 m/s. The aim of this study was to process 800 raw material units (100 layers \(\times\) 8 pieces) into 800 process units after performing the \(C1\), \(C2\), and \(C3\) processes.