PDQN - A Deep Reinforcement Learning Method for Planning with Long Delays: Optimization of Manufacturing Dispatching

1 INTRODUCTION . This paper proposes a method based on Deep Reinforcement Learning for automated production scheduling in semiconductor manufacturing systems . In such systems , scheduling decisions must be made for processing operations . These systems involve machines that each perform operations to process a variety of different semiconductor devices . Each type of device requires a specific set of operations to be performed which varies depending on the type of device . These systems also include re-entrant connections , and the machines may be subject to machine failures , in which machines break down during processing and need to be repaired before resuming operations . In the industry , static dispatching policies such as Critical Ratio ( CR ) or First-In-First-Out ( FIFO ) are often used , together with manual adjustments at failing machines . Obtaining efficient scheduling and dispatch policies at every machine , especially in a system with machine failures , is a challenging and complex task . There are multiple key productivity indicators that are considered in manufacturing systems . Included in these are throughput , cycle time , and lateness with respect to due dates . Optimizing weighted combinations of these productivity indicators is a challenging task . In this paper we introduce a novel Reinforcement Learning algorithm for planning in such environments . Due to the complexity of the issue , in this paper we focus specifically on the task of reducing lateness with respect to due dates . It should be noted that this objective function , in itself , and in isolation , is a measure that manufacturing system researchers have studied for decades , with less than desired success . In each factory system modeled in this paper , there is a set of machines that are used to process boxes of semiconductor devices on sheets of silicon wafers . These boxes are referred to as parts . At each point in time that a machine becomes available , a dynamic scheduling decision ( dispatching decision ) must be made as to what that machine should do next . In this paper , this decision is modeled as a choice of which part to process next from the queue at that machine group . Here , this process is modeled as a Markov Decision Process ( MDP ) . An MDP includes a state , action , and reward where the state is the information describing the system , the action is a choice of which decision to make , and the reward is a signal returned after an action is taken which indicates the quality of that action . In the factory systems being modeled , the semiconductor devices produced are hard drive head chips which are used to read from and write to hard drives . There is a range of different head types produced in the facility . In our experiments , we simulate the first 20 processing steps for each head type to model the front of line in the facility . In recent years Deep Reinforcement Learning ( RL ) techniques have been demonstrated to show remarkable performance in a number of previously challenging domains such as complex games ( Mnih et al. , 2015 ; Silver et al. , 2016a ; b ; Berner et al. , 2019 ; Vinyals et al. , 2019 ; Schrittwieser et al. , 2020 ) . In addition , some work has been done in applying such techniques to manufacturing systems , e.g . Waschneck et al . ( 2018 ) . One of the main difficulties in this domain is delayed rewards with particularly long delays ; other domains with this feature include medical interventions in healthcare . Consequently , our focus in this paper is to develop a planning method that can account for extra long delays in rewards , together with highly stochastic dynamics . So , this paper presents the Predictron Deep Q-Network ( PDQN ) , a novel Deep RL technique that combines the Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) and predictron ( Silver et al. , 2017 ) methods to learn a policy for dispatching of parts in a simulated system modeled after a semiconductor manufacturing facility . The DQN is a model-free RL optimization algorithm that trains through experience to estimate Q-values which can be used to form a policy . The predictron is a model-based policy evaluation algorithm that can be rolled forward multiple ” imagined ” planning steps to predict future rewards and values . The PDQN uses the predictron as a trained background planning model to generate value estimates for use in fine-tuning a pre-trained DQN . By doing this , it is possible to incorporate background planning as part of the training process . This combination helps the algorithm account for the highly delayed rewards encountered in these factory systems . Inspiration for this setup comes in part from Dyna ( Sutton , 1990 ) in which a model is used to train a policy , the main difference is that the PDQN uses an abstract model to perform its planning . The PDQN is also closely related to methods such as Value Prediction Network ( VPN ) ( Oh et al. , 2017 ) and MuZero ( Schrittwieser et al. , 2020 ) , but is different from these papers and approaches by using background planning instead of decision time planning and by using an arbitrary number of steps in between each abstract state representation . Background planning has recently been shown to be the largest contributor to policy improvement when using planning in model-based RL ( Hamrick et al. , 2021 ) . We compare the PDQN with both DQN and two standard factory dispatching policies , CR and FIFO . 2 RELATED WORK . 2.1 DEEP REINFORCEMENT LEARNING IN PRODUCTION SCHEDULING . In recent years , work has been done in applying deep reinforcement learning to production scheduling tasks ( Cadavid et al. , 2019 ) . Stricker et al . ( 2018 ) presents a Q-learning with artificial neural network function approximation dispatching method . This method was demonstrated to outperform a First-In-First-Out on the task of maximizing utilization and minimizing lead time on a small simulated semiconductor manufacturing system . In Zhang et al . ( 2020 ) a method is similarly proposed to automatically generate priority dispatch rules using a deep reinforcement learning agent . Here , a Graph Neural Network-based scheme is used to embed the states . This work , however , only applies to job shop problems in which there are a fixed number of jobs to be completed . Therefore this approach would be inapplicable to the more realistic production scheduling problem encountered in semi-conductor manufacturing where new jobs are repeatedly being added into the system and production can continue indefinitely . In addition , the work in Zhang et al . ( 2020 ) only considers deterministic systems which don ’ t account for uncertainties such as machine failures . In Katsikopoulos & Engelbrecht ( 2003 ) an approach to addressing Markov decision processes with delays and asynchronous cost collection is considered . However that work assumes fixed or deterministic delays or stochastic delays which are independent of the state . In the systems considered in this paper the delays are dependent upon the policy so this approach may not be applicable . In Derman et al . ( 2021 ) they consider MDPs in which there are action delays such that actions are executed some number of steps after they are chosen . This is a separate issue than the one we are addressing with PDQN . In our case actions are executed immediately , the challenge in our case is that actions affect the return over many time steps . In Campbell et al . ( 2016 ) they consider the problem of applying Q-learning with stochastic time delays in the reward signal . This is again different from the issue we are addressing as in our problem . It is not that specific reward values are not received immediately it is that the effects of each action impact the return over ong horizons which makes learning and credit assignment difficult . Delayed feedback is also considered in Walsh et al . ( 2008 ) . However it is limited to fixed , constant delays , which is not applicable for our case . 2.2 DEEP Q-NETWORK . DQN Mnih et al . ( 2015 ) is a Q-learning method , which works by estimating the expected discounted future return for a given state and action pair . For DQNs this is accomplished by using a Neural Network function approximator . DQN uses mini-batch stochastic gradient descent to update the weights of the neural network based on the gradient of a loss function to minimize the expected value of the loss . For DQNs the Mean Squared Error ( MSE ) loss is used as seen in Equation 1 . L ( θ ) = E [ ( r + γmax a′ Q ( s′ , a′ ; θ̄ ) −Q ( s , a ; θ ) ) 2 ] ( 1 ) where θ represents the weight parameters of the neural network . θ̄ represents an earlier copy of θ , which is used to form a target network . The purpose of the target network is to estimate the expected value of the next state . The training is stabilized by only updating θ̄ after a number of training iterations , thereby giving the online network a stable target . An alternative to this is to use a soft update technique where the target is gradually updated towards the online network . Q-learning is considered an off-policy method , as it can train on data collected by a different policy . To get the best result , however , exploration and exploitation should be tuned according to the problem . A common solution is to use the -greedy policy on the Q-function , where is slowly decayed over time . Experienced data is stored as a set of { state , action , reward , next state } in an experience replay buffer and sampled according to some distribution , originally uniformly . 2.3 PREDICTRON . The predictron ( Silver et al. , 2017 ) is an architecture for model-based value estimation and policy evaluation . It consists of a fully abstract model which works by ” imagining ” a sequence of waypoints , each simultaneously describing an arbitrary number of steps into the future and an estimation of the value from the abstract waypoint state . It is strongly related to methods such as n-step TD-learning ( Watkins , 1989 ) and eligibility traces ( Sutton & Barto , 2018 ) . The predictron learns a representation function f which outputs the first abstract state . Furthermore , it learns K sets of functions , where each set includes a value function vk , a next abstract state function sk , a reward function for the transition to the next abstract state rk , a discount value function γk and an eligibility trace function λk . The predictron has two outputs describing the predicted returns ( preturns ) , g0 : K and gλ . Here g0 : K is the set of K preturns , with one k-preturn gk for each abstract step k , as seen in Equation 2 . The λ-preturn gλ , seen in Equation 3 , is the weighted average of the k-preturns , where the λ-weights are determined using the learned eligibility trace parameters . gk = r1 + γ1 ( r2 + γ2 ( . . .+ γk−1 ( rk + γkvk ) . . . ) ) ( 2 ) gλ = K∑ k=0 wkgk ( 3 ) where wk = ( 1− λk ) ∏k−1 j=0 λ j if k < K ∏K−1 j=0 λ j otherwise ( 4 ) The predictron is trained by minimizing the MSE loss for both g0 : K and gλ , as defined in Equations 5 and 6 . L0 : K = 1 2K K∑ k=0 ‖Ep [ g|s ] − Em [ gk|s ] ‖2 ( 5 ) Lλ = 1 2 ‖Ep [ g|s ] − Em [ gλ|s ] ‖2 ( 6 ) where Ep is the sampled sum of discounted rewards gained in the episode , and Em is the predicted value from the model . Thirdly , an optional loss is minimized , the consistency loss , improving the consistency between the k-preturns and the λ-preturn , as seen in Equation 7 . L = 1 2 K∑ k=0 ‖Em [ gλ|s ] − Em [ gk|s ] ‖2 ( 7 ) The predictron is the inspiration for both the VPN ( Oh et al. , 2017 ) and MuZero ( Schrittwieser et al. , 2020 ) , in the sense that they all use abstract state-space representations . The main difference is that VPN and MuZero also train a policy for control , whereas the predictron is purely policy evaluation . VPN and MuZero both use tree searches to conduct their planning , whereas the predictron uses an estimation of eligibility traces to different depths of its abstract version of the expected future .

Summary: The authors study a deep reinforcement learning approach to the problem of scheduling machines in a semiconductor factory. The authors' main contribution is a deep Q-learning based system that uses a predictron to estimate the value function (instead of max_a Q(s',a)). The intuition behind the domain is that there are many machine groups that can process parts. The goal is to select the order that the machines should process parts. The authors compare a double DQN to their predictron DQN (PDQN) to baselines that are currently in use: critical ratio and first-in-first-out, with the objective of minimizing lateness. They build simulators based on real data: one that is proprietary and one that they release. They find the RL systems reduce lateness by substantially (20-50%) relative to the baselines and that their PDQN performs better by around 30% compared to DQN. However, they note that the RL systems complete substantially fewer parts, as much as 15% fewer for PDQN and 25% for DQN, suggesting that lateness may not be a good reward function for this task.