# Manufacturing Scheduling Using Colored Petri Nets and Reinforcement Learning
## 5 Simulation and State Space Results
### Case Study Performance Evaluation  
Simulation results were obtained for five batches of 28 orders each, with mean order size (i) twoline item per order; (ii) three line items per order; and (iii) four line items per order. The Q-learning was employed for a training period of 1000 cycles. Each simulation experiment was run for 100 replications. Replication results were averaged for each simulation experiment. Simulation results for the scheduling of the first batch of orders for each considered case are shown in Table 5 and Figure 8. Simulation results in Table 5 show the scheduling rule followed by the agent, the number of agent Q-learning visits to each state, and the total robot travel time, \(t_{\text{tot}}\), of the obtained scheduling. Results in Table 5 show that the agent selects the first policy action, i.e., the \(X\)-coordinate based heuristic in the considered cases. The \(X\)-coordinate based heuristic is expected to yield optimal solutions when the length of each zone is much longer than its width. Thus, the agent was trained to select the action that yields an optimal solution. Figure 8 shows the number of agent visits to each state. Approximate analytical model travel time expressions for a robotic P2B system with randomized storage were presented in Khachatryan and McGinnis [62]. The expected item travel time is bounded by the height in time of the pick zone in the analytical model. For the simulated system this limit is equal to 6 s. The simulated mean item travel time results in Table 5 are close to the 6 s limit. Results in Table 5 indicate that the scheduling agent visits the terminal dummy state, as well as states 1 to 4. It rarely visits higher number states; however, these states were kept in the policy table (Table 1) for different problem cases. The Q-learning converges to an optimal solution in all cases. Results also show that as the mean order size increases the number  
Late order arrival has been considered in order to evaluate the performance of the scheduling method in emergence. Four late orders of one line item each were added to the normal workload of five batches of 28 orders each with mean order size (i) three line items per order; and (ii) four line items per order. The scheduling policy in case of late order arrivals, shown in Table 6, consists of 10 states, and two dummy states. The immediate reward values, shown in Table 7, take into account the total travel time, \(\text{t}_{\text{tot}}\) as well as a balanced workload among robots. The symbol \(\wedge\) represents the logic AND operation in Tables 6 and 7, whereas \(\text{t}_{\text{tot,toboti}}\), represents the total robot travel time of robot \(i\), \(i\) = 1, 2, 3, 4. In each state the agent could select between two actions: (1) to pick all four emergency orders from pick zone 1; or (2) to pick two emergency orders from pick zone 1 and two emergency orders from pick zone 2. The agent followed the \(X\)-coordinate based heuristic to implement the order sequencing after the action selection. The Q-learning algorithm was employed by the agent for a training period of 1000 cycles. Each simulation experiment was run for 100 replications. Replication results were averaged for each simulation experiment. Table 8 shows the order picking workload distribution among pick zones, under the different policy actions, for one batch of orders with mean order size 3 line items per order and 4 emergency orders. Simulation results were obtained for different implementations of the Q-learning. In one implementation the scheduling agent fully exploited the Q-learning method, i.e., employed the Q-learning with \(\varepsilon\) = 0, whereas in a different experiment it followed an \(\varepsilon\)-greedy policy, with \(\varepsilon\) = 0.2. Simulation results for the scheduling of the first batch of orders for each considered case are shown in Table 9 and Figures 9-14.