# Manufacturing Scheduling Using Colored Petri Nets and Reinforcement Learning
## 4 The Case Study
### The Order-Picking Scheduling
#### 4.2.2 Order Sequencing
More specifically, it is assumed that the pick locations are not close to each other, thus since the length of the zone is much greater than the width, the horizontal movement times are longer than the vertical movement times. Therefore order sequencing is done based on the increasing order of the \(X\)-coordinates of the SKU type locations. The dummy terminal state corresponds to an optimal or near optimal solution. Each time the agent takes an action, it receives an immediate reward (or a penalty), r, as a result of its action, and the Q-values are updated according to Equation (1). The immediate rewards should reflect the scheduling optimality criterion, i.e., the long term goal of the agent when it decides its next action. The immediate reward values, shown in Table 2, are related to the total travel time of all robots, \(\text{t}_{\text{tot}}\), for the current batch of orders, range from \(-4.0\) to \(4.5\), and were chosen by trial and error. The learning rate, \(\alpha\), and the discount factor, \(\gamma\), are taken equal to \(0.1\) and \(0.9\) accordingly [6; 53]. The agent can follow an \(\varepsilon\)-greedy method, with \(\varepsilon\) = \(0.2\), i.e., \(20\%\) of the time it randomly selects an action. After the completion of the 1000 cycle period the agent selects the scheduling that minimizes the total travel time of all robots for the current batch of orders. In particular, in each scheduling cycle, the agent stores in a knowledge base a list the cycle index and the corresponding order sequencing. A separate list holds the cycle index and the corresponding total travel time of all robots. After the end \begin{table}
\begin{tabular}{c c c c} \hline \hline
**State** & **State Definition** & \begin{tabular}{c} \(X\)**-Coordinate Based** \\ **Heuristic-Q(s,a) Pair** \\ \end{tabular} & \begin{tabular}{c} **Random Pick Location** \\ **Selection-Q(s,a) Pair** \\ \end{tabular} \\ \hline \
\begin{tabular}{c} Dummy \\ \end{tabular} &
\begin{tabular}{c} Initial state: total robot travel time (\(\text{t}_{\text{tot}}\)) = 0 \\ Terminal state:0.98 \(\times\)\(\text{t}_{\text{tot}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.02\)\(\times\)\(\text{t}_{\text{tot,aver}}\) \\ \end{tabular} & 0 & 0 \\
1 & \(0.95\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.05\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(1,1) & Q(2,1) \\
2 & \(0.9\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.1\times\)\(\text{t}_{\text{tot,aver}}\) & Q(2,1) & Q(2,2) \\
3 & \(0.85\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.15\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(3,1) & Q(3,2) \\
4 & \(0.8\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.25\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(4,1) & Q(4,2) \\
5 & \(0.75\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.25\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(5,1) & Q(5,2) \\
6 & \(0.7\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.3\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(6,1) & Q(6,2) \\
7 & \(0.65\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.35\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(7,1) & Q(7,2) \\
8 & \(0.6\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.45\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(8,1) & Q(8,2) \\
9 & \(0.55\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\leq\)\(\text{t}_{\text{tot}}\)\(\leq\)\(1.45\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(9,1) & Q(9,2) \\
10 & \(0.55\)\(\times\)\(\text{t}_{\text{tot,aver}}\)\(\geq\)\(\text{t}_{\text{tot}}\)\(\cdot\)\(\text{t}_{\text{tot}}\)\(\cdot\)\(1.45\)\(\times\)\(\text{t}_{\text{tot,aver}}\) & Q(10,1) & Q(10,2) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Policy table for the scheduling agent.