PDQN - A Deep Reinforcement Learning Method for Planning with Long Delays: Optimization of Manufacturing Dispatching
Authors: David C Jenkins
René Arendt Sørensen
Vikramank Singh
Philip Kaminsky
Anil Aswani
Ramakrishna Akella
Venue: ICLR 2022 Submitted (2022)
Keywords: This paper proposes the PDQN
a novel method based on integrating Deep Reinforcement Learning and abstract planning for developing dynamic scheduling for policies in  systems with very delayed rewards
e.g. manufacturing.

Abstract: Scheduling is an important component in Semiconductor Manufacturing systems, where decisions must be made as to how to prioritize the use of finite machine resources to complete operations on parts in a timely manner. Traditionally, Operations Research methods have been used for simple, less complex systems. However, due to the complexity of this scheduling problem, simple dispatching rules such as Critical Ratio, and First-In-First-Out, are often used in practice in the industry for these more complex factories. This paper proposes a novel method based on Deep Reinforcement Learning for developing dynamic scheduling policies through interaction with simulated stochastic manufacturing systems. We experiment with simulated systems based on a complex Western Digital semiconductor plant. Our method builds upon DeepMind’s Deep Q-network, and predictron methods to create a novel algorithm, Predictron Deep Q-network, which utilizes a predictron model as a trained planning model to create training targets for a Deep Q-Network based policy. In recent years, Deep Reinforcement Learning methods have shown state of the art performance on sequential decision-making processes in complex games such as Go. Semiconductor manufacturing systems, however, provide significant additional challenges due to complex dynamics, stochastic transitions, and long time horizons with the associated delayed rewards. In addition, dynamic decision policies need to account for uncertainties such as machine downtimes. Experimental results demonstrate that, in our simulated environments, the Predictron Deep Q-network outperforms the Deep Q-network, Critical Ratio, and First-In-First-Out dispatching policies on the task of minimizing lateness of parts.

Under review as a conference paper at ICLR 2022
PDQN A DEEP REINFORCEMENT LEARNING METHOD FOR PLANNING WITH LONG DELAYS: OPTIMIZATION OF MANUFACTURING DISPATCHING
Anonymous authors
Paper under double-bling brind review
ABSTRACT
Scheduling is an important component Semiconductor Manufacturing systems, where decisions must be made as to how to prioritize the use of finite machine resources to complete operations on parts in a timely manner. Traditionally, Oper- ations Research methods have been used for simple, less complex systems. How- ever, due to the complexity of this scheduling problem, simple dispatching rules such Critical Ratio, and Iness-In-Firt--Our
often used in practice in the in- dustry for these more complex factories. This paper proposes a novel method based on Deep Reinforcement Learning for developing dynamic scheduling poli- cies through interaction with simulated stochastic manufacturing systems. We experiment with simulated systems based on a complex Western Digital semicon- ductor plant. Our method builds upon DeepMind's Deep Q-network, and pre- dictron methods to create a novel algorithm, Predictron Deep Q-network, which utilizes predictron model as a trained planning model to create training tar- gets for a Deep Q-Network based policy. recent years, Deep Reinforcement Learning methods have shown state of the art performance sequential decision- making processes in complex games such as Go. Semiconductor manufacturing systems, however, provide significant additional challenges due to complex dy- namics, stochastic transitions, and long time horizons with the associated delayed rewards. In addition, dynamic decision policies need to account for uncertain- ties such as machine downtimes. Experimental results demonstrate that, in our simulated environments, the Predictron Deep Q-network outperforms the Deep Q-network, Critical Ratio, and of minimizing lateness of parts.
dispatching policies on the task
INTRODUCTION
This paper proposes a method based on Deep Reinforcement Learning for automated production scheduling in semiconductor manufacturing systems. In such systems, scheduling decisions must be made for processing operations. These systems involve machines that each perform operations to process a variety of different semiconductor devices. Each type of device requires specific set of operations to be performed which varies depending the type of device. These systems also include re-entrant connections, and the machines may be subject to machine failures, in which ma- chines break down during processing and need be repaired before resuming operations. In the industry, static dispatching policies such as Critical Ratio (CR) or First-In-First-On (FIFO) are of- ten used, together with manual adjustments failing machines. Obtaining efficient scheduling and dispatch policies at every machine, especially in a system with machine failures, is a challenging and complex task. There are multiple key productivity indicators that are considered manufacturing systems. Included in these are throughput, cycle time, and lateness with respect to due dates. Op- timizing weighted combinations of these productivity indicators is a challenging task. this paper we introduce novel Reinforcement Learning algorithm for planning in such environments. Due to the complexity of the issue, this paper we focus specifically on the task of reducing lateness with respect due dates. It should be noted that this objective function, in itself, and in isolation, a measure that manufacturing system researchers have studied for decades, with less than desired success. In each factory system modeled in this paper, there is of machines that are used to

---

Under review as a conference paper at ICLR 2022
process boxes of semiconduetor devices on sheets of silicon wafers. These boxes referred as parts. At each point in time that a machine becomes available, a dynamic scheduling decision (dis- patching decision) must be made to what that machine should do next. In this paper, this decision is modeled as a choice of which part to process next from the queue at that machine group. Here, a Markov Decision Process (MDP). An MDP includes a state, action,
this process is modeled and reward where the state is the information describing the system, the action is a choice of which decision make, and the reward quality of that action. In the factory systems being modeled, the semiconductor devices produced are hard drive head chips which are used to read from and write hard drives. There a range of different head types produced in the facility. In our experiments, we simulate the first 20 processing steps for each head type to model the front of line the facility.
signal returned after an action is taken which indicates the
In recent years Deep Reinforcement Learning (RL) techniques have been demonstrated to show remarkable performance in a number of previously challenging domains such as complex games (Mnih et al. 2015 Silver et al. 201 6a b Berner et al. 2019 Vinyals et al. 2019 Schrittwieser et al.) 2020). In addition, some work has been done in applying such techniques to manufacturing systems, e.g. Waschneck al. (2018). One of the main difficulties in this domain is delayed rewards with particularly long delays; other domains with this feature include medical interventions in healthcare. Consequently, our focus in this paper is develop a planning method that can account for extra long delays in rewards, together with highly stochastic dynamics.
So, this paper presents the Predictron Deep Q-Network (PDQN), novel Deep RL technique that combines the Deep Q-Network (DQN) (Mnih et al. (0155) and predictron (Silver et al. (017)) meth- ods to learn policy for dispatching of parts in simulated system modeled after a semiconductor manufacturing facility. The DQN is a model-free RL optimization algorithm that trains through ex- perience to estimate Q-values which can be used form policy. The predictron is model-based policy evaluation algorithm that can be rolled forward multiple "imagined" planning steps to predict future rewards and values. The PDQN uses the predictron as a trained background planning model to generate value estimates for use in fine-tuning a pre-trained DQN. By doing this, it is possible to incorporate background planning as part of the training process. This combination helps the algo- rithm account for the highly delayed rewards encountered in these factory systems. Inspiration for this setup comes in part from Dyna (Sutton 1990) in which a model is used to train a policy, the main difference is that the PDQN uses an abstract model to perform its planning. The PDQN is also closely related to methods such as Value Prediction Network (VPN) (Oh et al., (017)) and MuZero (Schritwieser al. 2020), but is different from these papers and approaches by using background planning instead of decision time planning and by using an arbitrary number of steps in between each abstract state representation. Background planning has recently been shown to be the largest contributor to policy improvement when using planning in model- RL (Hamrick et al. (021). We compare the PDQN with both DQN and two standard factory dispatching policies, CR and FIFO.
2 RELATED WORK
2.1 DEEP REINFORCEMENT LEARNING IN PRODUCTION SCHEDULING
In recent years, work has been done in applying deep reinforcement learning to production schedul- ing tasks (Cadavid et al. 2019). Stricker et al. (2018) presents a Q-learning with artificial neural network function approximation dispatching method. This method was demonstrated to outperform a First-In- First-In-First-OO Out on the task of maximizing utilization and minimizing lead time on a small sim- ulated semiconductor manufacturing system.
In Zhang et al. (2200) method is similarly proposed to automatically generate priority dispatch rules using deep reinforcement learning agent. Here, Graph Neural Network-based scheme is used to embed the states. This work, however, only applies to job shop problems in which there are a fixed number of jobs to be completed. Therefore this approach would be inapplicable to the more realistic production scheduling problem encountered in semi-conductor manufacturing where new jobs are repeatedly being added into the system and production can continue indefinitely. In addition, the work in Zhang et al. (2020) only considers deterministic systems which don't 't account for uncertainties such machine failures.
2

---

Under review as a conference paper at ICLR 2022
In Katsikopoulos & Engelbrecht (2003) an approach to addressing Markov decision processes with delays and asynchronous cost collection is considered. However that work assumes fixed or deter- ministic delays stochastic delays which are independent of the state. In the systems considered in this paper the delays are dependent upon the policy so this approach may not be applicable. In Derman et al. (202T) they consider MDPs in which there are action delays such that actions executed some number of steps after they are chosen. This is a separate issue than the one we are addressing with PDQN. In our case actions are executed immediately, the challenge in our case is that actions affect the return over many time steps. In Campbell et al. (2016) they consider the prob- lem of applying Q-learning with stochastic time delays in the reward signal. This is again different from the issue we are addressing received immediately it is that the effects of each action impact the return over ong horizons which makes learning and credit assignment difficult. Delayed feedback is also considered in Walsh et al. 22088. However it is limited to fixed, constant delays, which is not applicable for our case.
in our problem. It is not that specific reward values are not
2.2
DEEP Q-NETWORK
DQN Mnih et al. (2015) is Q-learning method, which works by estimating the expected discounted future return for given state and action pair. For DQNs this is accomplished by using a Neural Network function approximator DQN uses mini-batch stochastic gradient descent to update the weights of the neural network based on the gradient of a loss function to minimize the expected value of the loss. For DQNs the Mean Squared Error (MSE) loss is used as seen in Equation |
[(()= = [[(r + Y max Q(s', al: 0) - Q(s, a))
a'
where 0 represents the weight parameters of the neural network. 0 represents earlier copy of 0, which is used to form a target network. The purpose of the target network is to estimate the expected value of the next state. The training is stabilized by only updating @ after a number of training iterations, thereby giving the online network a stable target. An alternative to this is to use soft update technique where the target is gradually updated towards the online network.
Q-learning is considered an off-policy method, as it can train on data collected by a different policy. To get the best result, however, exploration and exploitation should be tuned according the prob- lem. A common solution is to use the c-greedy policy on the Q-function, where c is slowly decayed over time. Experienced data is stored as set of {state, action, reward, next state] in an experience replay buffer and sampled according to some distribution, originally uniformly.
2.3 PREDICTRON
The predictron (Silver et al. 2017) is an architecture for model-based value estimation and pol- icy evaluation. It consists of a fully abstract model which works by "imagining" a sequence of waypoints, each simultancously describing an arbitrary number of steps into the future and esti- mation of the value from the abstract waypoint state. It is strongly related to methods such as n-step TD-learning (Watkins 1989) and eligibility traces (Sutton & Barto 2018). The predictron lears a representation function which outputs the first abstract state. Furthermore, it learns K sets of functions, where each set includes value function v². next abstract state function sh: reward function for the transition to the next abstract state r* a discount value function y* and an eligibility trace function *
The predictron has two outputs describing the predicted returns (preturns), g O:K and g². Here g the set of K pretums, with one k-preturn g for each abstract step k. as seen in Equation 2 The X-preturn g*, seen in Equation 3 is the weighted average of the k-pretums, where the X-weights are determined using the learned eligibility trace parameters.
g² = -=r++7((f 7r222 + 7 + +g (r +
(2)
K
g"
*
(3)
k==
3

---

Under review as a conference paper at ICLR 2022
where
(1 - X 11 X
K
w
(4)
K-1 1
X
otherwise
The predictron is trained by minimizing the MSE loss for both g and g", as defined in Equations aadd 6
LUu
Em [g" [s]
(5)
2K
k=U
LA
E [g]s]]
E. [s>]""1>)]'
where Ep is the sampled sum of discounted rewards gained in the episode, and Em is the predicted value from the model.
Thirdly, an optional loss is minimized, the consistency loss, improving the consistency between the k-preturns and the x-pretum, as seen in Equation 7
L
Em g [s]
Em lg |s]]
A=
The predictron is the inspiration for both the VPN (Oh et al. 2017) and MuZero (Schrittwieser et al., 2020), in the sense that they all use abstract state-space representations. The main difference that VPN and MuZero also train a policy for control, whereas the predictron is purely policy evaluation. VPN and MuZero both use tree searches conduct their planning, whereas the predictron uses an estimation of eligibility traces to different depths of its abstract version of the expected future.
3
METHODS
This chapter describes the general setup of the methods used in this paper. Section 3.1 describes how the simulated factory is modeled as an MDP. Section 33.2 describes the CR and FIFO dispatching policies. Section 3.3 describes the proposed new PDQN method while the Neural Network architec- ture and hyperparameter setup for both the DQN and PDQN are described in the Appendix section
A.1
3.1
MDP MODELLING
3.1.1
STATE SPACE REPRESENTATION
The state-space representation consists of the set of variables S = ShaUShh USm representing the number of parts of each type at each sequence step. the number of parts due for each head type, the number of parts that are past due, and the machine that the dispatch decision corresponds to.
First, information about the work in process (WIP) is included in the state space. Let Nut. be the number of parts in the factory of head type ht at step j of production. Included in the state space are values of Not. for all combinations head type ht H, where H is the set of all head types and sequence step
Jut. where Jn: is the set of sequence steps of head type ht, Jnt {1, 2, ... Mar), where MA. is the number of sequence steps for head type ht. Let this set state variables be S then Sh. [ H) A (j J.e)
Second, information about the number of parts that are due for each head type is included in the state space. Let Dre be the number of parts due for head type ht. Let the set of due part state variables be Sha, then Snd = DDae ltt E H}
Third, information about the parts which are past due included well. As such the values Pat representing the number of parts past due for each head type ht are included as well. Let this set of state variables be Shp- then Sep (Prel ht H}
4

---

Under review as a conference paper at ICLR 2022
Lastly, the state space includes the machine variables where Sn is one-hot vector indicating the machine that the dispatching decision for that time step corresponded to.
3.1.2 ACTION SPACE
action is made every time machine ready to process new part and a part is in its queue. The action determines which part to process next, and consists of selecting head type and a sequence step. Let Hm be the head types present in the queue for the affected machine m, then A (ht E Hm. Jnc)
3.1.3 REWARD SIGNAL
The reward signal is designed to penalize the agent for being late on parts with respect to their due dates. To accomplish this the reward at each time step is made to be negative and proportional the amount of time that has elapsed during the time step and the number of parts that are past due. This can be represented as an integral over the time between two time steps. Let 4 the time which time step occurs in the MDP representing the factory. Let R2 be the reward for time step i.
R== f"" Na(t) dt 44-1
(8)
where Na(() is the number of parts that are past due at time This way the agent will be repeatedly penalized each time step at a rate which is proportional to the number of parts that are past due. This serves heuristic that will encourage the agent to complete parts before their due dates.
3.2 CRITICAL RATIO AND FIRST-IN-FIRST- DISPATCHING POLICIES
CR dispatching is part priority rule based on the ratio between remaining time until due and the remaining time needed to complete processing of a part. The CR dispatching policy proceeds by selecting the part from the queue with the smallest critical ratio value. This will prioritize the parts which are most late and should therefore reduce the lateness of parts.
FIFO dispatching is done by selecting the part to process which considering a single queue, FIFO may reduce max waiting time at that queue by always processing the part that has been waiting for the longest. However, this policy may not be optimal for the whole system as it does not take into account the state other queues.
added to the queue first. When
3.3 PDQN
Presented here is the PDQN algorithm. PDQN addresses the need for learning from highly delayed rewards and dynamic uncertainty due to machine downtimes by including abstract planning for large number of steps. PDQN consists of two parts, an optimal decision policy determining compo- nent based DQN, and an abstract planning trajectory- -bsed value estimation component based on the predictron; this latter value estimate function the predictron is fed to the DQN during training. In traditional DQN, the model trains by minimizing the difference between its Q value estimate and a target value, as seen in Equation [ This target formed by sampling the reward for one step and then estimating the discounted return from the subsequent state using the same DQN architecture but with older weights, referred to as the target model. In the PDQN, we instead train the predictron part estimate the value of states under the DON policy, and this value estimate as the target for the DQN, effectively substituting max, Q(s', a'; with E,, [s'],, as seen in Equation 9
L()) = (r + Em [g'1 - Qss, a; 0
where Em, [s'] is the predictron estimate for the discounted return from the subsequent state s' and is the reward given by the environment.
By using the predictron as target, the policy is trained using background planning. This is because the predictron part is trained to estimate the actual return from running the policy. By incorporating background planning, the policy can be trained towards better targets, which can take much more delayed rewards into account, and by using learned eligibility trace weights, it can learn to better
5

---

Under review as a conference paper at ICLR 2022
assign the right weight to late rewards. As the policy converges towards a better policy, the predictron will have to be updated to fit the policy again. Where, in traditional DQN, the target model is updated by simply copying the weights of the online model, we here update the target from the predictron by training it on new samples collected by the policy. This is done by fixing the policy for a number of steps and collecting samples of states along with the following h rewards and the value of the hd state as estimated by the policy. It is important to note, that the predictron is trained in a supervised manner, where the data is collected using the policy. With i as the time step, the target for the predictron Ep ig ss is defined as
A-1
ss = Σ (n" Ratn) + +- *
(10)
==
where R is the actual return from the following steps and mar( a: 0.)) is the expected value of the ht state, estimated by the policy. Therefore the target for the predictron is biased by the estimate of the policy. When using long horizon the policy will have a small effect on the total discounted return target, while the actual return from within the horizon will increase the variance of the target. Shorter horizon lengths, however, would be more biased by the policy and have lower variance. Consequently, there a trade-off between these that may lead to different horizon lengths being optimal for different scenarios. The tested PDQN here uses a horizon higher than the estimated number of steps needed to complete on average more than two batches of parts at any given time of the environment. The loss for the predictron is then calculated using Equations and 12 and the consistency update loss from Equation 7
LOKK
Σ
Em [g |s]]
(11)
2K
k=
LA [s. - E, |s]l
(12)
2
As a policy evaluation method, the expectation is that the predictron model can provide better es- timates of long-term returns than the DQN based Q-value estimator itself. Better return estimates will then create stronger targets for learning when used to train the policy. We expect the target to be better as the predictron architecture can create an abstract planning model, including abstract states which are rolled forward to predict future returns weighed by a learned eligibility trace. The desirable features of the algorithm derive from the predictron and Eligibility Traces properties in incorporating: 1. The true delayed rewards, rather than inaccurate surrogates and 2. The most effective bias- viai-variance trade-off, with the associated dimensional reduction. We hypothesize that these characterizes would enable it to firstly perform very well. Secondly, it would likely dominate stand-alone Q-learning approaches such as DQN and variants.
The choice of using the predictron for planning instead of using decision time planning methods, such tree search, when actions are taken and the completion of the parts those actions relate to, actions generally have very little effect on the expected return over the next few states. Therefore if a tree search should be used, it would need to be traversed to a high depth before seeing the outcome of the immediate action taken. By using the predictron with an external control policy, effectively using background planning, the policy can get an estimate of the long-term state value directly on the state. It is therefore the target for the policy that trained using background training, and through this, the policy is indirectly trained through background training as well. The PDQN training algorithm is summarized in Algorithm allog with a detailed hyperparameter setup in the appendix section The predictron architecture used is shown in Figure I It is alternation of the original architecture from Silver et al.) (2017). The main change that all convolutional layers are replaced with fully connected layers. This alternation is made as the state representation in our environment is not spatially related in the same manner as the environments used in the original paper, making local convolutional filters less meaningful.
based on the nature of the environment. Due to the large delays between
4 EXPERIMENT
To evaluate the PDQN method, two different factory systems are used: a balanced factory system, which was a balanced version of real factory system, and a randomly generated factory system,

---

Under review as a conference paper at ICLR 2022
FC
FC
PC
FC
FC
PC
FC
FC
FC
FC
FC
FC
FC
FC
FC
FC
FC
FC
2.
7
vx
Figure 1: The fully connected version of the original predictron implemented with 16 depth layers with individual weights. Each fully connected layer has 128 neurons, except for the output layers which have 1. is the input state while g0: are the abstract states. V are the estimated values for each abstract state. ,0 are the expected abstract rewards received for transitioning from one abstract state the next. 7 -0: is the expected abstract discount factors to apply for each abstract step. 10:K-1 is the expected eligibility assigned to each abstract step. As the abstract steps arbitrarily long, 7 and vary as well.
which was made publicly available. We compared the mean lateness and the sum of lateness, which was the return received by the agent during an episode. We compared the performance against the CR. FIFO, and DQN policies For the evaluation, the DQN and PDQN policies were fixed, meaning that they were not allowed to train on the data from the test set.
4.1 ENVIRONMENT SETUP
Two environments are considered, a balanced 20 sequence steps (B20) factory system and gener- ated 20 sequence steps (G20) factory system.
The B20 factory setup was based on a subset of real semiconductor manufacturing facility owned by Western Digital Corporation.
The G20 factory setup was based on additional data from Western Digital Corporation. However these sets were randomly generated in order to match the distributions of the data without including proprietary information. Details about factory settings are included in the appendix in Section A.2 and all the factory files needed run the simulations for the generated systems are included the supplemental materials.
4.2 TRAINING AND MODEL SELECTION
Initial experiments showed that when comparing the performance of the trained models of the same type, large variations were seen between the models, but a low variation was seen on the individual models. This was expected come from the random initialization of the policies, well as from the randomness used for exploration. To account for the variation seen between models, multiple models were trained for both DQN and PDQN. For each factory system, 10 different DQNs were trained. Each of these DQN models was then used to initialize PDQN model training session. The PDQN models were trained for iterations, each resulting terms of lateness, were selected through validation simulations, Further training details, architecture, and hyperparameter setup are explained detail in Section A.
separate model. The best models, in
4.3 MAIN TEST RESULTS
The best performing DQN and PDQN model from each environment was tested on 50 simulations with different sample paths. Each sample path was initialized with seed ensure reproducible
'Iitiial experiments did not show an improved performance the DQN by applying the extensions pro- posed in Hessel et al. (2018). Furthermore, MuZero was considered, as according to Hamrick et al. (2021) this method incorporates planning. However, the results from the implementation did not converge. This method was prohibitively computationally heavy which incorporates MCTS may not be appropriate for such complex stochastic environments.
on this complex environment. This type of online planning

---

Under review as a conference paper at ICLR 2022
behavior from the environment when used compare the performance of different policies. For comparison, the mean sum of lateness for the completed parts is used.
FIFO
DQN
PDON
le6
1e7
H 
S Lateness
2.0
1.5
1.0
0.5
0.0
B20 MTTF:10,000
B20 LT0F:100,000
G20 1TTF:10,000
G20 MTTF:100,000
Figure 2: Here, the results from the objective function used to train the RL based policies are compared. The results are based on test runs for each setup of each tested environment.
Figure 2 shows the results from the 50 runs on all the tested environments. It is seen that for both factory systems, DQN and PDQN outperform CR and FIFO in terms of the mean sum of the lateness, which is the objective function used. Furthermore, PDQN outperform DQN on the same metric in all systems except the B20 system with MTTF of 10,000. The results seem to indicate that either DQN and PDQN perform similarly well, with very small difference in performance in the one case that DQN is superior, or the PDQN typically significantly outperforms DQN; we observe this B20 with MTTF of 100,000 and G20 with MTTF of 10,000.
Overall, the results indicate that PDQN succeeds in improving the performance over the DQN policy, which again improves upon the standard factory dispatching policies, on the given lateness objective function in the tested environment. From smart factory perspective, choosing PDQN with the correct parameters from the validation step appears to provide robust close-to-optimgs performance for the given objective function.
This performance improvement validates our initial hypothesis that incorporating the long delay reward data from the planning simulation, and the bias-variance features of the predictron, should together provide a powerful approach to addressing very long delays in highly dynamic, stochastic, and large-dimensional systems.
4.4 ADDITIONAL OBSERVATIONS TOWARDS FUTURE RESEARCH
In this sub-section, provide additional results motivate second objective function, in- tegrating lateness and throughput. To be clear, we do not consider this objective function this paper. An interesting observation is that for both DQN and PDQN, the sum of lateness often in- creases when the MTTF is increased for the two systems. By measuring the mean lateness and the number of completed parts from the 50 test runs, seen in Figure some additional insights this increase, are made. The results can be seen in Tables 3 and 4 in the appendix section A.I
PDQN outperforms the DQN in terms of mean lateness except the B20 system with MITF of 10,000. However, it is interesting to see how the number of completed parts, in general, is lower for the RL methods compared the CR and FIFO policies. This result indicates that the used objective function of reducing the lateness of parts might result in unwanted behavior in terms of throughput, which is not part of the current objective function. Future work might need to consider an objective combining lateness and throughput. However, we do note that in the G20 system with an MTTF of 10,000, the PDQN throughput outperformed CR.
The G20 system has also been tested with WIP level of 15. The results from this test is shown in Table the appendix. The PDQN and DQN performance on those systems indicated that
2The initial 2x the WTP x the relcase batchsize number of parts are discarded from the results to only consider the stcady-state system after hurn period.
8

---

Under review as a conference paper at ICLR 2022
the learned policies favors higher WIP levels. This is possibly because higher WIP levels allow the policy to have greater flexibility, as more parts will line up at the machine queues, effectively increasing the number of allowed actions. It is interesting, however unexpected, that FIFO is better than CR on almost all parameters in the G20 setup, and also achieves lower mean lateness in the B20 setup. We suspect this result to be caused by the of the CONWIP release policy and the high rate of machine failures. For example, CONWIP applied only to bottleneck machines would be more efficient. Future work should consider experimenting with a wide range of release policies.
CR
TU31
DQN
MDON
Lateness
1500
150
1000
100
15000
35000
14500
32500
Completed
14000
30000
13500
27500
B20
G20 MTTF:10,000
G20
10T0:10,000
10TF:100,000
MTTF:100,000
Figure 3: Additional observations regarding mean lateness (Top) and the number of completed parts (Bottom) are shown here. The data is shown with the inner quartiles (colored boxes), the median (black lines), the mean (white dots), and the min and max values (whiskers). Both the mean lateness and the number of completed parts reported for all parts completed after the initial two full system runs, which are removed to align the observation with the objective function.
We observe that the validation step is fairly effective in the choice of hyperparameters including model parameters and iterations. Future research should experiment with hyperparameter optimiza- tion of the PDQN method to avoid this excess amount of training.
CONCLUSIONS
In this paper, we approach the problem of dispatching in simulated Semiconductor Manufacturing systems by using Deep RL techniques. We present the PDQN, novel Deep RL approach combining DQN with value estimates from the predictron. We evaluate the use of both DQN and PDQN on two factory systems. The Deep RI. methods are compared against CR and FIFO dispatching policies. The results show that PDQN outperforms CR. FIFO, and DQN in terms of lateness of part in the systems and that both Deep RL methods outperform CR and FIFO on this task.
From these results, we see that our hypothesis holds true, in that using the predictron architecture better predict target values with very large delays, and provide powerful bias-variance trade-off's. can indeed increase the performance of a DQN based policy. That is, by incorporating this trained abstract planning model, the policy seems better learn from delayed rewards in systems such as the dynamic manufacturing systems described here. Currently. however, there is large variance in the performance of models after training, so carefully choosing trained model by validating its performance is recommended.
Scheduling in semiconductor manufacturing facilities a complex task that has a significant effect on the efficiency of production. Estimation of long-term values in this domain is especially difficult due to the nature of the factory systems. By including predictron methods, which have improved were able to train models that outperform the DQN, CR. and FIFO our
predictive ability, chosen objective of reducing lateness of parts. The methods presented in this paper progress the application of Deep RL to scheduling algorithms in this domain.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All the code and the factory setting files included in the supplementary material. To generate new factory systems use the fact_ fact_file_pp www.fffeeeee...e.... seng.en.pp. py file, to reproduce the G20 system use seed=0. The PDQN algorithm is described in Algorithm|
REFERENCES
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ily Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. ArXiv, abs/ 1912.0668,, 2019.
Juan Pablo Usuga Cadavid, Samir Lamouri, Bernard Grabot, and Arnaud Fortin. Machine learning in production planning and control: A review of empirical literature. IFAC-PapersOOnLine, 52 (13::385-300, January 2019.
Jeffrey Camphell, Sidney N Givigi, and Howard M Schwartz. Multiple model Q-Learning for stochastic asynchronous rewards. J. Intell. Rob. Syst., 81(3):407-422, March 2016.
Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with non-stationary
markov policies. In International Conference on Learning Representations, 2021. URL https:
//openzeview. net wwr.mmiiiiiioomommmi.m... RMMKeP2gR
Jessica Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Wither- spoon, Thomas Anthony, Lars Holger Buesing, Petar Veličković, and Theophane Weber. On the role of planning in model-based deep reinforcement learning. In International Confer- ence on Learning Representations, 2021. URL httpt::t///// bettps://operreviee net mnet/formm? fwrumorrmooooooooo TTM64DGB21
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, April 2018.
Konstantinos Katsikopoulos and S.F. Engelbrecht. Engelbrecht, s.e.: Markov decision processes with delays and asynchronous cost collection. ieee trans. autom. control 48(4), 568-574. Auto- matic Control, IEEE Transactions on, 48:568 574, 05 2003. doi: 10. 00.100//AC.2200.807777 809799.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland. Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx http:///txddd...g/11/.../ www.:o..gg..oooo 1038 httee//ntuuteeeee
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6118 6128. Curran Associates, Inc., 2017. URL http://0--waaeeperrrrrooooooeaaa
http:://appen
pdd
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan. Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 (7839):604 609, December 2020.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game go with deep neural networks and tree search. Nature, 529(7587):484 489, January 60.
10

---

Under review as a conference paper at ICLR 2022
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587) 489, January 2016b. ISSN 0028-0836. doi: 10.1038/nature 16961
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3191-3199, International Convention Centre, Sydney, Australia, 06-11 mlr. prwps / http///00///vvveev html
Aug 2017. PMLR. TRRLhtpp:: httl:tttppp/prccedinng
Nicole Stricker, Andreas Kuhnle, Roland Sturm, and Simon Friess. Reinforcement learning for adaptive order dispatching the semiconductor industry. CIRP Annals, 67(1)5511-514, 2018. ISSN 0007-8506. doi: http:////........../////////////
2018.04.0 URL https: wwww.
sciencedirect com/
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based approx- imating dynamic programming. In Proceedings of the Seventh International Conference on Machine Learming, pp. 224. Morgan Kaufmann, 1990.
Richard S. Sutton and Andrew G. Barto, Reinforccemett Learning: An Introduction. The MIT Press, second edition, 2018. URL http://inccmpleeteeeeessssss net//oo//hetoo book -2nd.
html
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre, Ziyu Wang. Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in stareraft II using multi-agent reinforcement learning. Nature, 575(7782): 350-354, November 2019.
Thomas Walsh, Lihong Li, and Michael Littman. Learning and planning environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18:83-105, 02 2008. doi: 10. 0007//10458-008 008-996-7.
Bernd Waschneck, André Reichstaller, Lenz Belzner, Thomas Altenmüller, Thomas Bauernhansl, Alexander Knapp. and Andreas Kyek. Optimization of global production scheduling with deep reinforcement learning. Procedia CIRP, 72:126441122 2018. ISSN 2212-8271. doi: https://doi. org/10. 1100:000006//pppnnr200000..00 212 URL.ttpps:: htap:p::::/www///// /ww..wwwk.... sci encedi rect wwt.cooooossooooo com/sct ence/ 51st CIRP Conference on Manufacturing Systems.
atticlee/s11/2221221118888
C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
Cong Zhang, Wen Song. Zhiguang Cao, J Zhang, Puay Siew Tan, and Chi Xu. Learning to dispatch for job shop scheduling via deep reinforcement learning. NeurIPS, 2020.
A APPENDIX
A.1 ARCHITECTURE AND HYPERPARAMETER SETUP
Here present the environment- dependent DQN and PDQN hyperparameter settings for the periments. We used the same hyperparameter settings for all experiments. The hyperparameters tuned based experiments the B20 system.

---

Under review as a conference paper at ICLR 2022
The architecture of the DQN was implemented as 4 layer fully connected neural network, where the first three layers had 400, 250, and 125 neurons respectively, and used the ReLU activation function. The last layer was the output layer, which had N, number of neurons, where Na was the number of actions in the environment. The DQN was trained using the MSE loss with the Adam optimizer. The DQN was trained with batch size of 32 and a learning rate of 0.005 using discount factor of 0.99. The target network was softly updated with T = 0.125. Actions were chosen using the e-greedy policy on the Q-function with e starting at 1.0 and decaying with 0.999 each step. A summary of the DQN hyperparameters is shown in Table [
The architecture of the predictron part, seen in Figure I is fully connected version of the original predictron architecture (Silver et al., (017), i.e., all convolution layers have been replaced with fully connected layers. All layers in the architecture, except for the output layers, have 128 neurons and use ReL.U activations. The number of neurons for each abstract state-space representation serves as an encoding mechanism, where fewer neurons will result in a more compressed representation. Initial results show that using 128 neurons had a small positive effect on the performance when comparing to using 8, 16, 32, 64, and 256 neurons.
The depth of the predictron was set to 16, where each depth layer used its own weights. This configuration was chosen, as it was the best performing configuration in Silver et al. (2017). L2 regularization used to counter overfitting. The training conducted using the MSE loss with the Adam optimizer. The batch size of the predictron was set to 128, and the number of training batches per iteration was 128. The PDQN had the same discount rate the DQN, 7 0.99, and a horizon of 500, meaning that the target for the predictron was the sum of the actual discounted return of the following 500 steps and the discounted DQN value estimate for state 544-00- With this setting we weighed the actual discounted return by 0.993, while weighing the remaining estimated value by 0.007.
Id dx
Ju 0.99 dx 0.993
iinf dr Jo 0.99
The horizon of 500 was found as 2.5 times the number of steps required to complete one batch on an empty system.
The policy part of the PDQN was initialized to be a copy of pre-trained DQN. It was trained in the same manner as the DQN, with the only difference being that the target was estimated by the predictron part and that it was trained for 50,000 steps per iteration. A summary of the PDQN hyperparameters is shown in Table 2 The environment is set to run for an initial burn-in period using the greedy policy on the Q-function to skip the initial part of the environment before initiating the training.
First the predictron is trained for an initial number of steps, to allow it to learn a good estimate for the value of the policy. Then the policy is updated with the predictron value estimate as the target for an initial number of steps. Then the number of steps for training the predictron and the policy is set to a fixed value, and the alternation between training the two parts is continued for number of iterations. During training. the Q function is updated after every step using the Adam optimizer on a small batch of data. The actions for the policy are sampled using epsilon greedy with fixed epsilon. When training the predictron, the actions are sampled using the greedy policy on the Q function. The predictron updated after all samples in that sequence have been collected in a supervised learning manner.
A.2 SETTINGS AND GENERATION OF FACTORY SYSTEMS
The balanced 20 sequence steps (B20) factory system was based on a subset of real semiconductor manufacturing facility owned by Western Digital Corporation. In this system, the first 20 processing steps for each head type were simulated. The machine groups, process routings, and processing times in the simulated system were set to match the real system. Other factory parameters were set based on simulation results under the CR dispatch policy. Some machines were added to the simulated system to reduce bottle-necking at stations with high utilization. The system was tested with two machine failure rates to experiment with the robustness to changes in this parameter. The Mean Time To Failure (MTTF) was therefore set to either 100,000 minutes and 10,000 minutes, while the Mean Time To Repair (MTTR) was set to 120 minutes, and the WIP level was set to 30 release batches where each batch contained one part of each head type. The due date lead times and
12

---

Under review as a conference paper at ICLR 2022
Table Hyperparameter setup for DQN
Hyperparameter
Value
Learning rate Batch size
0.005
32
Discount factor 7
0.99
Exploration rate
1.0 - 0.02
decay rate
0.999
Soft update coefficient r
0.125
Training steps
500, 000
Replay buffer size
10, 000
Table 2: Hyperparameter setup for PDQN
Predictron part
Policy part
Learning rate
0.01
Learning rate
0.005
L2 weight
0.01
Batch size
32
Batch size
128
Discount factor y
0.99
Batches per iteration
128
Exploration rate
0.1
Horizon h
500
Steps per iteration
50, 000
Depth A:
16
Replay buffer size
10,000
WIP levels were set to ensure mix of on-time and past due part completions. For each sample path in validation and testing, the environment was executed for 100,000 simulation minutes.
The generated 20 sequence steps (G20) factory systems were based upon data shared by Western Digital for real factory system. These systems were designed to resemble real systems while being partially randomly generated to allow for sharing the factory settings without divulging proprietary data. The G20 systems have the same (MTTF) and WIP levels as the B20 systems. The generation the G20 system factory files is described in the appendix. In all setups, the release of parts into the system was controlled by CONWIP based policy maintain relatively constant levels of WIP in the system by releasing each batch of parts when another completes. Due dates were set using set due date lead time for each head type, which specifies the time between when part was released and when was due. Machine failure and repair times were sampled from exponential distributions with specified mean times based data from Western Digital.
Each G20 system included 10 head types. The process routings for each head type were sampled with replacement from set of 24 stations. The processing times were sampled from a gamma distribution which was set to match the distribution of the balanced factory system. The number of machines in each station selected to match the level of demand at each station. The MTTF was again set to 100,000 minutes and 10,000 minutes with an MTTR of 102 minutes. The level of WIP was set to 30 release batches, where each released batch contained one part of each head type. For each sample path in validation and testing, the environment was executed for 500,000 simulation minutes. The longer execution times for these setups were set to account for a lower throughput compared to the B20 system.
In all setups, the release parts into the system was controlled by a CONWIP based policy to maintain relatively constant levels of WIP in the system by releasing each batch of parts when another completes. Due dates were set using a set due date lead time for each head type, which specifies the time between when part was released and when it was due. Machine failure and repair times were sampled from exponential distributions with specified mean times based on data from Western Digital.
13

---

Under review as a conference paper at ICLR 2022
Algorithm 1: PDQN training
Q= Load pretrained DQN;
Start and run environment for an initial burn-in period, using the greedy policy on the function;
predictron.batch.seee batches per iiteration; TrainPoliey False;
predictron wwt.m/trrnnnssnteeeneeeneeeeeeee
i 0;
while training do
Update state: 8 = 8';
Update allowed actions: ē=
Use greedy policy on Q function find action: à a=argmaz(Q($$e))
Take step: & f, é' step(ā);
Save to replay buffer: ReplayBuf fer. a, a,), Increase step counter: i+ =
TrainPPolicy then
Sample batch: 8, a, T,,,,'' s',,'' sample (ampleeaaaa Bu fer):
Calculate loss from Equation
Use Adam optimizer
batch loss;
i >= policy policy_steps sttppssseppppppp -per jiteration then
T Train False;
i 1 0:
end
else
S 8;
R; = P;
if step >= then
= 000000 * Ri-ntnl + 2 max max((Q(3)) (Q)))))))
(7
Append S.hk, Ep S.-hl to train_data;
step >= predictron train steps then
while train. data is not empty do
8, s.pp.l/s] getbateh (train. datapredictron)
Calculate k loss from Equation
S
Calculate À loss from Equation
Use Adam optimizer
k loss and loss;
To use consistency updates do:
Calculate cu loss from Equation]]
Use Adam optimizer cu loss;
end
T Train True;
i 0;
end
end
end
end
A.3 ADDITIONAL OBSERVATIONS
Here, more results from running the dispatching methods on the two factory systems introduced in section 4 are presented. The mean lateness and the number of completed parts are reported for the entire length of the simulation, except for the initial 2x WIP x the release batchsize number of parts, from which the results are discarded. The general trend that the RL-based methods outperform CR and FIFO in terms of mean lateness, where PDQN outperforms DQN in 4 of 6 setups. CR and FIFO have the highest number of parts completed in the 15 and 30 WIP level experiments. Note that the number of completed parts is low while the mean lateness is also low for both the DQN and PDQN methods. The reason for this might be that the objective function used penalizes the agent from completing parts later than their due time. As the CONWIP release policy is used, the agents
14

---

Under review as a conference paper at ICLR 2022
might learn to delay the completion of batches much possible delay the introduction of new parts for as long as possible. This way, the mean lateness can be low the majority of parts get completed fast, and the number of completed parts will be low, due the delayed completion of the last part of the batch, which would match the results seen.
Table 3: Test results for balanced 20 step factory, B20, on 100,000 simulation minutes. The results averaged over 50 different test runs and shown with + standard deviation of the sample mean over the different test runs.
CR
FIFO
DQN
PDQN
30 WIP BATCHES
MTTF
10,000
Sum of Lateness
2.8926
2.67e6++:.6866
1.1466=:8.00e3
1.2566++4.61e3
Mean Lateness
191.3++0.0
180.7=+0.28
80.9:t0.62
91.3==0.9
Completed parts
14798.713.900
14774.2:13.18
14146.8=12.52
13.70.6110078
MTTF
100,000
Sum of Lateness
2.76c6=11.97c3
2.6766=11.442
1.41c6+2.68e3
1.0566=7.92e3
Mean Lateness
184.1=0.13
178.4+0.12
104.7=0.28
73.1=0.39
Completed parts
14968.9+1.26
14943.7+1.12
13491.347.16
14437.5++2.97
Table Test results for generated step factory, G20, on 500,000 simulation minutes. The results are averaged over 50 different test runs and shown with + standard deviation of the sample mean over the different test runs. 2 different failure rates and different WIP levels are compared.
CR
FIFO
DQN
PDQN
15 WIP BATCHES
MTTF
10,000
Sum of Lateness
19.3e6::27.1e3
16.566
13.2e6#24.8e3
14.9c6+22.223
Mean Lateness
561.4410.4
533.4:10.47
489.941.229
464.6:10.96
Completed parts
34417.249.20
34540.5+7.01
26919.64:28.99
32060.0438.26
MTTF
100,000
Sum of Lateness
18.666+16.7e.
17.966+
13.556++22.5e3
13.0e6+:20.443
Mean Lateness
532.7++0.55
512.4+0.30
4.6.5++0668
449.3++.88
Completed parts
34926.113.72
34902.6-t2.07
31311.7+11.01
28828.8+:244
30 WIP BATCHES
MTTF
10,000
Sum of Lateness
58.5c6++1152e3
08.2e6444.0ee.
41.8c6199.2c3
26.2e61399.223
Mean Lateness
1730.8+2.55
1698.9+1.45
1501.6
+74.6-112.65
Completed parts
33798.6+22.551
34265.2+5.33
27831.6+:27.91
33895.5+44415
MTTF
100,000
Sum of Lateness
18.9e6=+102.5e3
57. $7.0c64135.86
43.5e6:187.7et
42.2e6:199.4e3
Mean Lateness
1707. 1707.9=2.45
1647.9 1647.9+1.05
1400.2+2.64
1390.7+:2.95
Completed parts
14508.6+16.75
34587.8+12.22
31057.7+26.64
30336.1#25.44
15


Reviews:
[{"review_id": "i-SEQD2ke1r", "reviewer": "Reviewer_ZLM1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\n- This work raises an important issue that has hardly been tackled in the RL literature, namely high delays in the reward signal. It has a strong motivation, and the proposed solution is sound in my opinion. \n- In terms of contribution, I think that making a connection between a typical operations research problem and a deep RL method constitutes a contribution in and of itself, as I am not aware of any previous work that does so. \n- The paper is clearly written, in a comprehensive manner that makes it easy to follow, may it be for an RL expert reader or for a scheduling one. I have minor comments in that regard, which I will make at the end of this review. \n\n*Weaknesses / Questions to the authors*\n\n- The authors cite Stricker et al. (2018) as an RL paper tackling scheduling tasks with Q-learning. Why is their method not used as a baseline in that paper? Indeed, PDQN is also a Q-learning algorithm and aims to solve the same type of problem. Another Q-learning method for delayed rewards is proposed in [3]...\n- The authors motivate the predictron submodule by « highly delayed rewards » (Sec. 3.3 and a few other places). How much is it « high »? Although I guess it is stochastic, is it bounded? Another concern is regarding the way time steps are set in the MDP: according to Eq. (8) time is discretized in order for the reward signal to fit with the MDP framework. But what is a typical value of $\\Delta_i:= t_i - t_{i-1}$? Would not it affect that of the reward delay, i.e. high $\\Delta_i$ implies lower delay value? Similarly, what is a typical value for $h$ in Sec. 3.3.? I guess this should depend on an average delay value. How would a high $h$ value affect the computational time of PDQN (higher $h$ implies long-term inference)? \n- How different are Eq. (11) and (12) from Eq. (5) and (6), respectively? If these are in some sense equivalent, the authors can synthesize their notation and refer to (5) and (6) in Sec. 3.3. instead of rewriting them.\n- Sec. 2 looks like a Preliminary section providing the problem setting, more than a Related work section. In that regard, a review of delayed RL works is missing. More precisely, high delay in the signal results in some challenging issues: although one could perform embedding and retrieve a non-delayed MDP as in [1], this greatly increases computational complexity. In my opinion, the predictron has a major advantage over augmentation methods that may be even less tractable than MCTS (such drawback of MCTS is indeed mentioned in Sec. 3.3).\n- In [2], the authors use a forward model to tackle execution delay issues. How does the predictron compare to this type of model? The use of eligibility traces seems to have similar effects. \n-  In Figs. 2-3, DQN seems to perform decently against PDQN. This discredits a bit the use of the predictron. How could we explain such good performance of DQN which, by the way, does not account for delay at all? Unexpectedly, although they are fashioned precisely to solve such scheduling problems, FIFO and CR perform much worse than those two deep RL methods. How can this be interpreted? ", "summary_of_the_paper": "This paper finds its motivation in scheduling for semiconductor manufacturing systems. It proposes a deep RL algorithm for tackling the highly delayed feedback usually encountered in this type of dynamic system. The authors introduce an algorithm called Predictron Deep Q-Network (PDQN) that plugs a predictron architecture into the loss of the Q-network. The efficiency of PDQN is tested on two scheduling problems and challenged against 3 natural baselines, CR, FIFO, and DQN, the first two baselines being standard for scheduling. ", "summary_of_the_review": "Although this work is well-motivated, I have two main concerns leading me to a « weak reject » notation. \n- First, on the theoretical side, the problem of delayed reward is not tackled frontally, but rather implicitly through the predictron. As such, delay is not formalized, although intrinsic to the problem considered. It weakens this paper’s contribution in my opinion, as one could question whether PDQN would necessarily work in a different type of domain (e.g., healthcare, as mentioned in Sec. 1), where the reward function has other dynamics. \n- Second, delayed reward and delay in general (delayed execution; delayed observations), have been addressed in RL. This span of relevant works is never mentioned nor discussed. I list below a few references that are welcome to be discussed/compared with PDQN.\n\n*Relevant references (all of them are missing in this paper)*\n\n[1] Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays\nand asynchronous cost collection. IEEE transactions on automatic control, 48(4):568–574, 2003.\n\n[2] Derman, Esther, Gal Dalal, and Shie Mannor. \"Acting in Delayed Environments with Non-Stationary Markov Policies.\" ICLR 2021.\n\n[3] Jeffrey S Campbell, Sidney N Givigi, and Howard M Schwartz. Multiple model q-learning for stochastic asynchronous rewards. Journal of Intelligent & Robotic Systems, 81(3-4):407–422, 2016.\n\n[4] Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18(1):83, 2009.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SagfGAQxSMY", "reviewer": "Reviewer_8ES3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Main Review\n\nPros:\n1. The problem is very interesting. And it is a real life application that could generate good real-life values. \n2. The code is released and has good reproducibility.\n\nCons:\n1. It is not clear why the authors decide to make the presented modifications to the original predictron algorithm.\nAnd it is hard to understand the intuition of the algorithm.\nThe modifications are not justified.\n\n2. There’s no proper reinforcement learning baselines.\nThe algorithm is only compared with some toy methods and the RL baselines DQN is quite outdated. If I didn’t misunderstand, at least predictron should be compared against?\nAnd there are also some new algorithms that seem to be applicable to the problem.\n\n", "summary_of_the_paper": "\n PDQN - A Deep Reinforcement Learning Method for Planning with Long Delays: Optimization of Manufacturing Dispatching\nIn this paper, the authors applied a reinforcement learning algorithm to the problem of manufacturing dispatching.\nThey used a model-based algorithm that is inspired on the algorithm of predictron.\n", "summary_of_the_review": "\n*Summary Of The Review\nI am worried about the novelty of the algorithm, and the claims are not supported by experiments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "F4K7Fu8pmfi", "reviewer": "Reviewer_QqV1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper tackles an important decision/scheduling problem in manufacturing industry, which seems to be a very promising real-world application of deep RL.\n- The proposed method outperformed the common heuristics used in industry in terms of cumulative lateness.\n- The use of model-based approach to address the long reward delay issue in this problem makes sense to me.\n\n\nWeaknesses and other comments\n- The paper should formally define the dynamic scheduling problem in manufacturing in mathematical terms. The current level in the introduction is not sufficient for a scientific paper. The true objective of interest should also be clearly defined. The current choice of reward is the total time past due over the parts during a unit time period, but later in the experiments, it appears that the throughput is also a metric of interest. The choice of reward should be motivated by the ultimate objective of the problem, not by certain behavior of the policy that we want to encourage.\n- Before the conclusion, it is mentioned that \"It is interesting, however unexpected, that FIFO is better than CR on almost all parameters in the G20 setup, and also achieves lower mean lateness in the B20 setup.\" Is CR typically expected to outperform FIFO? Does this observation mean that the simulator requires some calibration before being used for training the agent?\n- The baselines for the benchmarking should be more properly selected. For example, Section 2.1 discussed several recent works based on RL for the same problem. Why is not one of them selected as a baseline? The proposed method is basically DQN with the targets estimated by predictron based on rolling out a simulator. To justify this combination of techniques (especially the use of predictron), benchmarking should be performed against a version of DQN, say, with targeted estimated simply by MC rollouts. ", "summary_of_the_paper": "This paper tackles the dynamic scheduling problem in semiconductor manufacturing using an RL approach. Due to the processing actions taking different amount of time to complete, the problem has a long reward delay issue, which the paper addresses through using predictron to estimate the targets in DQN. Simulation experiments were performed on domains constructed using real-world data and against common heuristic baselines and DQN. Results show that the proposed method outperforms the baselines in terms of cumulative lateness of the parts. ", "summary_of_the_review": "This paper tackles an interesting and promising industrial application of RL. To address the long reward delay issue, a combination of DQN and predictron (for estimating targets) is proposed to train the agent, which makes sense. Empirical results are presented to show the advantage of the proposed method over common heuristics and vanilla DQN. The paper needs significant improvement in formal problem/objective definitions and the choice of baselines in benchmarking for a more convincing case. The simulation environment is not a standard one, so some calibration results should be presented to ensure the validity of the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6_pCM22DTLX", "reviewer": "Reviewer_uJaW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors develop a realistic, public shareable environment for manufacturing dispatching that is suitable for RL methods research.\n- The PDQN framework is novel and offers better performance for these tasks.\n\nWeaknesses:\n- The main baseline for the work is quite old—the authors use the 2015 vanilla DQN. The authors say they tried various improvements from Hessel et al., and they didn't seem to help, but they don't enumerate which improvements they tried (see question 1 below). From a methods perspective, I find the evaluation of the PDQN to not be rigorous enough—the authors only evaluate in the new environment they constructed and they only compare against vanilla DQN.\n- From an application perspective, the issue of sim2real problems does not receive _any_ attention that I can see. The authors do not go into detail about how they build their simulated environment—specifically how they choose incoming parts that need to be processed. They also do not discuss a real-world test.\n- The throughput issue seems like a substantial real-world problem. I can't imagine wanting to implement this system if throughput (which translates directly to the bottom line) could decrease by 15%.\n\nQuestions:\n1. The architecture the authors propose seems quite similar to a dueling DQN in the way that the Q-function estimation and the value function estimation are split—the difference being that the dueling DQN does not use a predictron and that the dueling DQN has a different aggregation architecture. The authors mention they tried the improvements of Hessel et al., but did they try the dueling architecture specifically? It would also be easy to integrate a predictron into the value-estimation network in a dueling DQN. (It would be interesting to know how much the predictron structure is specifically contributing.)\n2. The use of L2 regularization in the predictron is interesting—did the authors find regularization improved performance at convergence or did it decrease sample complexity? There has been recent interest in the use of L2 regularization, e.g., https://arxiv.org/abs/1810.00123\n\nDetailed comments:\n- Eq. 1 is often called the Mean Squared Bellman Error because it measures the extent to which Q satisfies the Bellman equations\n- “Q-learning is considered an off-policy method” -> “Q-learning is an off-policy method”. It’s definitional—doing Q-learning would not make - sense because there would be no exploration.\n- Figure 1 is many pages after it is referenced\n\nPost-rebuttal: It seems that the rebuttal discussion has been productive—the paper has been improved in many relevant ways. There are still outstanding issues—it seems like throughput and baselines are the largest of these.\n", "summary_of_the_paper": "The authors study a deep reinforcement learning approach to the problem of scheduling machines in a semiconductor factory. The authors' main contribution is a deep Q-learning based system that uses a predictron to estimate the value function (instead of max_a Q(s',a)). The intuition behind the domain is that there are many machine groups that can process parts. The goal is to select the order that the machines should process parts. The authors compare a double DQN to their predictron DQN (PDQN) to baselines that are currently in use: critical ratio and first-in-first-out, with the objective of minimizing lateness. They build simulators based on real data: one that is proprietary and one that they release. They find the RL systems reduce lateness by substantially (20-50%) relative to the baselines and that their PDQN performs better by around 30% compared to DQN. However, they note that the RL systems complete substantially fewer parts, as much as 15% fewer for PDQN and 25% for DQN, suggesting that lateness may not be a good reward function for this task.", "summary_of_the_review": "From a methodological perspective, I found the evaluation to not be rigorous enough. As an application perspective, there are critical application-specific issues that are not addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with solving the problem of scheduling machines in a semiconductor factory using an RL approach. As the different actions take a different amount of time to complete, the authors propose to use a predictron architecture to estimate the targets in DQN. The experimental results show that the proposed method outperforms the considered baselines on two scheduling problems.

After reading the authors' feedback and discussing their concerns, the reviewers agree that this paper is still not ready for publication.
In particular, the main issues are about the novelty/similarity with respect to related works, the lack of theoretical insights and formal definitions, the effectiveness of the presented benchmarking, lack of analysis of some unexpected results.

I encourage the authors to take into consideration the concerns raised by the reviewers when they will work on the updated version of their paper.
Decision: Reject