# Manufacturing Scheduling Using Colored Petri Nets and Reinforcement Learning
## 4 The Case Study
### The Order-Picking Scheduling
#### 4.2.2 Order Sequencing  
Order-picking sequencing involves the generation of the order-picking sequence for each robot by the agent. In this scheduling step, the agent employs the Q-learning in order to make the optimal decision. The agent's goal that guides it to the optimal scheduling decision is to achieve the minimal total robot travel time for each batch of orders. Picker travel time is an important parameter in the design of order picking systems. For the system under consideration robot travel time impacts the learning capability developed by the agent, i.e., the optimal order sequencing. It is a parameter affecting the environment's state and immediate reward calculation, thus the Q-values. Robot movement is characterized by the Chebyshev metric. A gantry robot is characterized by linear movement, thus its position is specified using the Cartesian coordinate system. Independent motors permit simultaneous movement in both horizontal and vertical planes. If the robot moves between two points in the pick area (\(xy\) plane), with coordinates (\(x_{1}\), \(y_{1}\)) and (\(x_{2}\), \(y_{2}\)), according to the Chebyshev metric it travels a distance equal to \(\max(\,|\,x_{2}-x_{1}\,|\,\), \(\,|\,y_{2}-y_{1}\,|\,)\). Thus, the travel time between two points depends on the maximum of the horizontal and vertical movement times [63]. The robot pick travel time for the P2B system consists of the time it takes to move from a pick location to the closest empty buffer (loaded travel movement) and the time it takes to move from the buffer to the next pick location (empty travel movement). The parameters used in the system simulation for robot travel time calculation are: horizontal movement speed = 1.5 m/s, vertical movement speed = 0.4 m/s, length of a zone = 45 m, height of a zone = 2.4 m.  
The Q-learning is employed by the agent for 1000 cycles in order to do the order sequencing for the current batch of orders. In each cycle the agent produces an order sequencing solution implemented by the CTPN system model. In each cycle, since each line item is located in two pick locations, the agent chooses randomly between the two possible locations and makes a list of the pick locations for each robot. Each agent policy action is the generation of the ordered list/sequence of the pick locations for each robot in the current batch by the agent. The agent can take one of the following two available policy actions in order to produce the order sequencing in each state:  
1. use the \(X\)-coordinate based heuristic [63]. The sequencing of orders in each zone is done based on the increasing order of the \(X\)-coordinates of the SKU type locations, assuming that the length of each zone is much longer than its width. 2. Choose randomly the pick locations in each zone. The state of the system is the criterion taken into account by the agent in order to decide its next action. For the order-picking scheduling case study, the total travel time of all robots for the current batch of orders, \(\text{t}_{\text{tot}}\), is the state determination criterion. It is calculated from the CTPN modelsimulation in each scheduling cycle. The scheduling policy includes 10 states, and two dummy states, shown in Table 1. The Q-values for all states were initialized to a value of zero, since no prior knowledge is known for any state. Robot travel times are calculated using the pick location information and the Chebyshev metric. The average total travel time used by the agent in order to perceive the current state is calculated from the average total travel time value, \(\text{tot}_{\text{t,aver}}\), calculated from 1000 replications of total travel time values of random lists of pick locations (pick tote and pick compartment locations) corresponding to the total line item number of the current batch of orders. The order sequencing for the calculation of average total travel time is done using the \(X\)-coordinate based heuristic.