# Manufacturing Scheduling Using Colored Petri Nets and Reinforcement Learning
## 4 The Case Study
### The Implemented CTPN System Model  
The hierarchical CTPN system model has been implemented using CPN Tools [64]. The top page of the implemented CTPN model shown in Figure 2 includes the Scheduling Agent and the Simulation Environment submodels as substitution transitions. Basic color set declarations of the CTPN model are shown in Table 3.  
A substitution transition Order Arrivals in the simulation environment submodel generates the orders. A total of 140 orders are generated in discrete time steps. The order inter-arrival times are exponentially distributed with a mean inter-arrival time of 100 time units. The order color token is a record with fields: orderno (the order number), itemno (the number of line items), and AT (arrival time). The order size follows the binomial distribution function, with a mean of one to four line items. The list of orders, a token of color set ORDER_STREAM, is placed in place Order Stream Queue (place P1 in the top model page). The Scheduling Agent submodel is shown in Figure 3. When the Scheduling Heuristics substitution transition receives the list of orders in place P6, it implements the order batching heuristic. The Scheduling Heuristics substitution transition includes the Create Batch Workload substitution transition and the Batch Routing substitution transition in order to implement the order batching heuristic. The Create Batch Workload substitution transition generates a list of five batches, a token of color set BATCH_LIST. BATCH_LIST, is a list of color set BATCH_INFO, a record with fields: batch_ord (the number of the batch of orders), batch_wkld (the list of orders included in the current batch), and  
Figure 2: The top page of the implemented hierarchical CTPN model of the scheduling system. batch_itm_cnt (the total number of line items included in the current batch). The Batch Routing substitution transition generates the routing for the batches of orders. The Scheduling Heuristics output tokens are placed in places P7, Batch Workload and P8, Order Stream Schedule (places P2 and P3 in the top model page, respectively). Place Batch Workload holds the workload per pick zone for all five batches. Place Order Stream Schedule holds the schedule for all five batches. Each customer order in the Order Stream Schedule output token is represented by a record with fields orderid (the order entry number, prondo (SKU type), hitmno (number of line items), and pickzono (pick zone number). The Scheduling Heuristics output token for a batch of orders is placed in place P9. The pick zone knowledge base substitution transition is then enabled. Its output tokens, held in places P10 and P11, are the possible pick locations (tote number and compartment number) for the line items to be picked from each zone. Pick locations are generated using the discrete uniform distribution. discrete(1,30) generates the available pick zone tote locations and discrete(1,40) generates the available compartment locations for each line item. Line items can be retrieved from two possible locations in each zone. The Order Sequencing substitution transition is then enabled. The scheduling agent starts the scheduling cycles (a total of 1000) for the current batch of orders. Place P12 generates the Scheduling cycle index, a token of color set INT, ranging from 1 to 1000. The Order Sequencing substitution transition is shown in Figure 4. The Policy action 1 and Policy action 2 substitution transitions output pick locations to places P21 and P22 respectively. Pick locations are chosen for each line item from the corresponding input lists, held in places P17 and P18, representing available pick zone tote and compartment locations. Either Policy action 1 or Policy action 2 substitution transition is enabled, depending on the value of the policy action index token (places P19 and P20). The policy action index takes the values 1 or 2, depending on the action that the scheduling agent decides to take in each state based on the policy table (Table 1).