Title: Residual Scheduling: A New Reinforcement Learning Approach to Solving Job Shop Scheduling Problem
Venue: NeurIPS 2023
Accepted: no
Keywords: reinforcement learning

job-shop scheduling problem

flexible job-shop scheduling problem

graph neural network
Abstract: Job-shop scheduling problem (JSP) is a mathematical optimization problem widely used in industries like manufacturing, and flexible JSP (FJSP) is also a common variant. Since they are NP-hard, it is intractable to find the optimal solution for all cases within reasonable times. Thus, it becomes important to develop efficient heuristics to solve JSP/FJSP. A kind of method of solving scheduling problems is construction heuristics, which constructs scheduling solutions via heuristics. Recently, many methods for construction heuristics leverage deep reinforcement learning (DRL) with graph neural networks (GNN). In this paper, we propose a new approach, named residual scheduling, to solving JSP/FJSP. In this new approach, we remove irrelevant machines and jobs such as those finished, such that the states include the remaining (or relevant) machines and jobs only. Our experiments show that our approach reaches state-of-the-art (SOTA) among all known construction heuristics on most well-known open JSP and FJSP benchmarks. In addition, we also observe that even though our model is trained for scheduling problems of smaller sizes, our method still performs well for scheduling problems of large sizes. Interestingly in our experiments, our approach even reaches zero gap for 49 among 50 JSP instances whose job numbers are more than 150 on 20 machines.

Abstract

Job-shop scheduling problem (JSP) is a mathematical optimization problem widely used in industries like manufacturing, and flexible JSP (FJSP) is also a common variant. Since they are NP-hard, it is intractable to find the optimal solution for all cases within reasonable times. Thus, it becomes important to develop efficient heuristics to solve JSP/FJSP. A kind of method of solving scheduling problems is construction heuristics, which constructs scheduling solutions via heuristics. Recently, many methods for construction heuristics leverage deep reinforcement learning (DRL) with graph neural networks (GNN). In this paper, we propose a new approach, named residual scheduling, to solving JSP/FJSP. In this new approach, we remove irrelevant machines and jobs such as those finished, such that the states include the remaining (or relevant) machines and jobs only. Our experiments show that our approach reaches state-of-the-art (SOTA) among all known construction heuristics on most well-known open JSP and FJSP benchmarks. In addition, we also observe that even though our model is trained for scheduling problems of smaller sizes, our method still performs well for scheduling problems of large sizes. Interestingly in our experiments, our approach even reaches zero gap for 49 among 50 JSP instances whose job numbers are more than 150 on 20 machines.

1 Introduction

The job-shop scheduling problem ( JSP ) is a mathematical optimization (MO) problem widely used in many industries, like manufacturing (Zhang et al., 2020; Waschneck et al., 2016). For example, a semiconductor manufacturing process can be viewed as a complex JSP problem (Waschneck et al., 2016), where a set of given jobs are assigned to a set of machines under some constraints to achieve some expected goals such as minimizing makespan which is focused on in this paper. While there are many variants of JSP (Abdolrazzagh-Nezhad and Abdullah, 2017), we also consider an extension called flexible JSP ( FJSP ) where job operations can be done on designated machines.

A generic approach to solving MO problems is to use mathematical programming, such as mixed 26 integer linear programming (MILP) and constraint satisfaction problem (CSP). Two popular generic 27

MO solvers for solving MO are OR-Tools (Perron and Furnon, 2019) and IBM ILOG CPLEX 28

Optimizer (abbr. CPLEX ) (Cplex, 2009). However, both JSP and FJSP, as well as many other MO 29

problems, have been shown to be NP-hard (Garey and Johnson, 1979; Lageweg et al., 1977). That 30

said, it is unrealistic and intractable to find the optimal solution for all cases within reasonable times. 31

These tools can obtain the optimal solutions if sufficient time (or unlimited time) is given; otherwise, 32

return best-effort solutions during the limited time, which usually have gaps to the optimum. When 33

problems are scaled up, the gaps usually grow significantly. 34

In practice, some heuristics (Gupta and Sivakumar, 2006; Haupt, 1989) or approximate methods 35

(Jansen et al., 2000) were used to cope with the issue of intractability. A simple greedy approach is to 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

use the heuristics following the so-called priority dispatching rule ( PDR ) (Haupt, 1989) to construct solutions. These can also be viewed as a kind of solution construction heuristics or construction heuristics . Some of PDR examples are First In First Out ( FIFO ), Shortest Processing Time ( SPT ), Most WorK Remaining ( MWKR ), and Most Operation Remaining ( MOR ). Although these heuristics are usually computationally fast, it is hard to design generally effective rules to minimize the gap to the optimum, and the derived results are usually far from the optimum.

Furthermore, a generic approach to automating the design of heuristics is called metaheuristics , such as tabu search (Dell'Amico and Trubian, 1993; Saidi-Mehrabad and Fattahi, 2007) , genetic algorithm (GA) (Pezzella et al., 2008; Ren and Wang, 2012), and PSO algorithms (Lian et al., 2006; Liu et al., 2011). However, metaheuristics still take a high computation time, and it is not ensured to obtain the optimal solution either.

Recently, deep reinforcement learning (DRL) has made several significant successes for some applications, such as AlphaGo (Silver et al., 2016), AlphaStar (Vinyals et al., 2019), AlphaTensor (Fawzi et al., 2022), and thus it also attracted much attention in the MO problems, including chip design (Mirhoseini et al., 2021) and scheduling problems (Zhang et al., 2023). In the past, several researchers used DRL methods as construction heuristics, and their methods did improve scheduling performance, illustrated as follows. Park et al. (2020) proposed a method based on DQN (Mnih et al., 2015) for JSP in semiconductor manufacturing and showed that their DQN model outperformed GA in terms of both scheduling performance (namely gap to the optimum on makespan) and computation time. Lin et al. (2019) and Luo (2020) proposed different DQN models to decide the scheduling action among the heuristic rules and improved the makespan and the tardiness over PDRs, respectively.

A recent DRL-based approach to solving JSP/FJSP problems is to leverage graph neural networks (GNN) to design a size-agnostic representation (Zhang et al., 2020; Park et al., 2021b,a; Song et al., 2023). In this approach, graph representation has better generalization ability in larger instances and provides a holistic view of scheduling states. Zhang et al. (2020) proposed a DRL method with disjunctive graph representation for JSP, called L2D ( Learning to Dispatch ), and used GNN to encode the graph for scheduling decision. Besides, Song et al. (2023) extended their methods to FJSP. Park et al. (2021b) used a similar strategy of (Zhang et al., 2020) but with different state features and model structure. Park et al. (2021a) proposed a new approach to solving JSP, called ScheduleNet , by using a different graph representation and a DRL model with the graph attention for scheduling decision. Most of the experiments above showed that their models trained from small instances still worked reasonably well for large test instances, and generally better than PDRs. Among these methods, ScheduleNet achieved state-of-the-art (SOTA) performance. There are still other DRL-based approaches to solving JSP/FJSP problems, but not construction heuristics. Zhang et al. (2022) proposes another approach, called Learning to Search (L2S), a kind of search-based heuristics.

In this paper, we propose a new approach to solving JSP/FJSP, a kind of construction heuristics, also based on GNN. In this new approach, we remove irrelevant machines and jobs, such as those finished, such that the states include the remaining machines and jobs only. This approach is named residual scheduling in this paper to indicate to work on the remaining graph.

Without irrelevant information, our experiments show that our approach reaches SOTA by outperforming the above mentioned construction methods on some well-known open benchmarks, seven for JSP and two for FJSP, as described in Section 4. We also observe that even though our model is trained for scheduling problems of smaller sizes, our method still performs well for scheduling problems of large sizes. Interestingly in our experiments, our approach even reaches zero gap for 49 among 50 JSP instances whose job numbers are more than 150 on 20 machines.

2 Problem Formulation

2.1 JSP and FJSP

A n × m JSP instance contains n jobs and m machines. Each job J j consists of a sequence of k j 84 operations { O j, 1 , . . . , O j,k j } , where operation O j,i must be started after O j,i -1 is finished. One 85 machine can process at most one operation at a time, and preemption is not allowed upon processing 86 operations. In JSP, one operation O j,i is allowed to be processed on one designated machine, denoted 87

one of the three machines { M 1 , M 2 , M 3 } in the table. A solution of a JSP instance is to dispatch all 90 operations O j,i to the corresponding machine M j,i at time τ ( s ) j,i , such that the above constraints are 91 satisfied. Two solutions of the above 3x3 JSP instance are given in Figure 1 (a) and (b). 92

(a)

(b)

(c)

(d)

M1

O11

O3,1

023

M1

O11

02,3

M

031

01:2

O2,3

M1

O

1:1

C

3.2

02,3

M2

O2,2

O1,3

M2

O2,2

01,3

M2

01,1

O2,2

M2

0

3.1

02,2

01,3

M3

O

02.1

O12

3.2

M3

02,1

03.2

01:2

M3

02,1

3.2

01,3

M3

2,1

012

0

23

6

891012

0

23

56

9

11

15

0

3

5

6

9

0

234

6

8

1112

While there are different expected goals, such as makespan, tardiness, etc., this paper focuses on 93 makespan. Let the first operation start at time τ = 0 in a JSP solution initially. The makespan of the 94 solution is defined to be T ( mksp ) = max( τ ( c ) j,i ) for all operations O j,i , where τ ( c ) j,i = τ ( s ) j,i + T ( op ) j,i 95 denotes the completion time of O j,i . The makespans for the two solutions illustrated in Figure 1 (a) 96 and (b) are 12 and 15 respectively. The objective is to derive a solution that minimizes the makespan 97 T ( mksp ) , and the solution of Figure 1 (a) reaches the optimal. 98

99

100

101

102

103

A n × m FJSP instance is also a n × m JSP instance with the following difference. In FJSP, all operations O j,i are allowed to be dispatched to multiple designated machines with designated processing times. Table 1 (b) illustrates a 3 × 3 FJSP instance, where multiple machines can be designated to be processed for one operation. Figure 1 (c) illustrates a solution of an FJSP instance, which takes a shorter time than that in Figure 1 (d).

104

2.2 Construction Heuristics

An approach to solving these scheduling problems is to construct solutions step by step in a greedy 105 manner, and the heuristics based on this approach is called construction heuristics in this paper. In 106 the approach of construction heuristics, a scheduling solution is constructed through a sequence of 107 partial solutions in a chronicle order of dispatching operations step by step, defined as follows. The 108 t -th partial solution S t associates with a dispatching time τ t and includes a partial set of operations 109 that have been dispatched by τ t (inclusive) while satisfying the above JSP constraints, and all the 110 remaining operations must be dispatched after τ t (inclusive). The whole construction starts with S 0 111 where none of operations have been dispatched and the dispatching time is τ 0 = 0 . For each S t , a set 112 of operations to be chosen for dispatching form a set of pairs of ( M , O ), called candidates C t , where 113 operations O are allowed to be dispatched on machines M at τ t . An agent (or a heuristic algorithm) 114 chooses one from candidates C t for dispatching, and transits the partial solution to the next S t +1 . If 115 there exists no operations for dispatching, the whole solution construction process is done and the 116 partial solution is a solution, since no further operations are to be dispatched. 117

(a)

S

0

(b)

S

1

(c)

S

2

(d)

S

3

(e)

S

4

(f)

S

5

(g)

S

6

(h)

S

7

(i)

S

8

M

M

M

M2

M2

M2

M3

M3

02.1

M3

2,1

Candidates

M

1,1

M

O3,1

M3

02,1

Candidates

M

1,1

M

3,1

Candidates

M2

2,2

M

M

江

O3,1

M

1

江

O3,1

M2

P2,2

M2

P2,2

M2

O2,2

M3

2

M3

2

M3

√

2.1

O

2

3

O

2

3

23

6

8

Candidates

M3

O1,2

M

3,1

Candidates

M3

1,2

Candidates

M

2,3

M

11

O3,1

O23

M

11

O3,1

O23

M1

O3,1

O23

M2

O2,2

M2

O2,2

1,3

M2

O2,2

1,3

M3

2,1

12

M3

2,1

O

12

M3

02,1

Q

12

3.2

O

23

6

8

9

23

6

8

9

12

0

23

6

8910

12

Candidates

M2

1,3

M3

3.2

Candidates

M3

3.2

Candidates

dispatched as in Figure 2 (a). The initial candidates C 0 are { ( M 1 , O 1 , 1 ) , ( M 3 , O 2 , 1 ) , ( M 1 , O 3 , 1 ) } . 120

122

Figure, where the dashed line in red indicate the time

τ

t

. The last one

S

9

, the same as the one in

Figure 1 (a), is a solution, since all operations have been dispatched, and the last operation ends at 123 time 12, the makespan of the solution. 124

125

126

127

128

For FJSP, the process of solution construction is almost the same except for that one operation have multiple choices from candidates. Besides, an approach based on solution construction can be also viewed as the so-called Markov decision process ( MDP ), and the MDP formulation for solution construction is described in more detail in the appendix.

129

130

131

132

133

134

3 Our Approach

In this section, we present a new approach, called residual scheduling , to solving scheduling problems. Weintroduce the residual scheduling in Subsection 3.1, describe the design of the graph representation in Subsection 3.2, propose a model architecture based on graph neural network in Subsection 3.3 and present a method to train this model in Subsection 3.4;

3.1 Residual Scheduling

In our approach, the key is to remove irrelevant information, particularly for operations, from states 135 (including partial solutions). An important benefit from this is that we do not need to include all 136 irrelevant information while training to minimize the makespan. Let us illustrate by the state for the 137 partial solution S 3 at time τ 3 = 3 in Figure 2 (d). All processing by τ 3 are irrelevant to the remaining 138 scheduling. Since operations O 1 , 1 and O 2 , 1 are both finished and irrelevant the rest of scheduling, 139 they can be removed from the state of S 3 . In addition, operation O 2 , 2 is dispatched at time 2 (before 140 τ 3 = 3 ) and its processing time is T ( op ) 2 , 1 = 4 , so the operation is marked as ongoing . Thus, the 141 operation can be modified to start at τ 3 = 3 with a processing time 4 -(3 -2) . Thus, the modified 142 state for S 3 do not contain both O 1 , 1 and O 2 , 1 , and modify O 2 , 2 as above. Let us consider two more 143 examples. For S 4 , one more operation O 2 , 2 is dispatched and thus marked as ongoing, however, the 144 time τ 4 remains unchanged and no more operations are removed. In this case, the state is almost the 145 same except for including one more ongoing operation O 2 , 2 . Then, for S 5 , two more operations O 3 , 1 146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

and O 2 , 2 are removed and the ongoing operation O 1 , 2 changes its processing time to the remaining time (5-3).

For residual scheduling, we also reset the dispatching time τ = 0 for all states with partial solutions modified as above, so we derive makespans which is also irrelevant to the earlier operations. Given a scheduling policy π , T ( mksp ) π ( S ) is defined to be the makespan derived from an episode starting from states S by following π , and T ( mksp ) π ( S, a ) the makespan by taking action a on S .

3.2 Residual Graph Representation

In this paper, our model design is based on graph neural network (GNN), and leverage GNN to extract the scheduling decision from the relationship in graph. In this subsection, we present the graph representation. Like many other researchers such as Park et al. (2021a), we formulate a partial solution into a graph G = ( V , E ) , where V is a set of nodes and E is a set of edges. A node is either a machine node M or an operation node O . An edge connects two nodes to represent the relationship between two nodes, basically including three kinds of edges, namely operation-to-operation ( O → O ), machine-to-operation ( M → O ) and operation-to-machine ( O → M ). All operations in the same job are fully connected as O → O edges. If an operation O is able to be performed on a machine M , there exists both O → M and M → O directed edges. In (Park et al., 2021a), they also let all machines be fully connected as M → M edges. However, our experiments in section 4 show that mutual M → M edges do not help much based on our Residual Scheduling. An illustration for graph representation of S 3 is depicted in Figure 3 (a).

(a) Graph for

S

3

(b) Graph embedding

(c) Score function

M1

1.2

(3)

M2

O2.1

02.2

2

h

(0)

G

HGIN(1)

h

G

1

HGIN(2)

h

(2)

G

HGIN(2)

h

(3)

MLP→Score(M,Oji)

M3

M

03.1

In the graph representation, all nodes need to include some attributes so that a partial solution S at the dispatching time τ can be supported in the MDP formulation (in the appendix). Note that many of the attributes below are normalized to reduce variance. For nodes corresponding to operations O j,i , we have the following attributes:

Status ϕ j,i : The operation status ϕ j,i is completed if the operation has been finished by τ , ongoing if the operation is ongoing (i.e., has been dispatched to some machine by τ and is still being processed at τ ), ready if the operation designated to the machine which is idle has not been dispatched yet and its precedent operation has been finished, and unready otherwise. For example, in Figure 3 (a), the gray nodes are completed , the red ongoing , the yellow ready and the white unready . In our residual scheduling, there exists no completed operations in all partial solutions, since they are removes for irrelevance of the rest of scheduling. The attribute is a one-hot vector to represent the current status of the operation, which is one of ongoing , ready and unready . Illustration for all states S 0 to S 8 are shown in the appendix.

Normalized processing time ¯ T ( op ) j,i : Let the maximal processing time be T ( op ) max = max ∀ j,i ( T ( op ) j,i ) . Then, ¯ T ( op ) j,i = T ( op ) j,i /T ( op ) max . In our residual scheduling, the operations that have been finished are removed in partial solutions and therefore their processing time can be ignored; the operations that has not been dispatched yet still keep their processing times the same; the operations that are ongoing change their processing times to the remaining times after the dispatching time τ t . As for FJSP, the operations that has not been dispatched yet may have several processing times on different machines, and thus we can simply choose the average of these processing times.

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

Normalized job remaining time ¯ T ( job ) j,i : Let the rest of processing time for job J j be T ( job ) j,i = ∑ ∀ i ′ ≥ i T ( op ) j,i ′ , and let the processing time for the whole job j be T ( job ) j = ∑ ∀ i ′ T ( op ) j,i ′ . In practice, T ( job ) j is replaced by the processing time for the original job j . Thus, ¯ T ( job ) j,i = T ( job ) j,i /T ( job ) j . For FJSP, since operations O j,i can be dispatched to different designated machines M l , say with the processing time T ( op ) j,i,l , we simply let T ( op ) j,i be the average of T ( op ) j,i,l for all M l .

For machine nodes corresponding to machines M l , we have the following attributes:

Machine status ϕ l : The machine status ϕ l is processing if some operation has been dispatched to and is being processed by M l at τ , and idle otherwise (no operation is being processed at τ ). The attribute is a one-hot vector to represent the current status, which is one of processing and idle .

Normalized operation processing time ¯ T ( mac ) l : On the machine M l , the processing time T ( mac ) l is T ( op ) j,i (the same as the normalized processing time for node O j,i ) if the machine status is processing , i.e., some ongoing operation O j,i is being processed but not finished yet, is zero if the machine status is idle . Then, this attribute is normalized to T ( op ) max and thus ¯ T ( mac ) l = T ( mac ) l /T ( op ) max .

Now, consider edges in a residual scheduling graph. As described above, there exists three relationship sets for edges, O → O , O → M and M → O . First, for the same job, say J j , all of its operation nodes for O j,i are fully connected. Note that for residual scheduling the operations finished by the dispatching time τ are removed and thus have no edges to them. Second, a machine node for M l is connected to an operation node for O j,i , if the operation O j,i is designated to be processed on the machine M l , which forms two edges O → M and M → O . Both contains the following attribute.

Normalized operation processing time ¯ T ( edge ) j,i,l : The attribute is ¯ T ( edge ) j,i,l = T ( op ) j,i /T ( op ) max . Here, T ( op ) j,i = T ( op ) j,i,l in the case of FJSP. If operation O j,i is ongoing (or being processed), T ( op ) j,i is the remaining time as described above.

3.3 Graph Neural Network

In this subsection, we present our model based on graph neural network (GNN). GNN are a family of deep neural networks (Battaglia et al., 2018) that can learn representation of graph-structured data, widely used in many applications (Lv et al., 2021; Zhou et al., 2020). A GNN aggregates information from node itself and its neighboring nodes and then update the data itself, which allows the GNN to capture the complex relationships within the data graph. For GNN, we choose Graph Isomorphism Network ( GIN ), which was shown to have strong discriminative power (Xu et al., 2019) and summarily reviewed as follows. Given a graph G = ( V , E ) and K GNN layers ( K iterations), GIN performs the k -th iterations of updating feature embedding h ( k ) for each node v ∈ V :

where h ( k ) v is the embedding of node v at the k -th layer, ϵ ( k ) is an arbitrary number that can be learned, and N b ( v ) is the neighbors of v via edges in E . Note that h (0) v refers to its raw features for input. MLP ( k ) is a Multi-Layer Perceptron ( MLP ) for the k -th layer with a batch normalization (Ioffe and Szegedy, 2015).

Furthermore, we actually use heterogeneous GIN , also called HGIN , since there are two types of nodes, machine and operation nodes, and three relations, O → O , O → M and M → O in the graph representation. Although we do not have cross machine relations M → M as described above, updating machine nodes requires to include the update from itself as in (1), that is, there is also one more relation M → M . Thus, HGIN encodes graph information between all relations by using the four MLPs as follows,

where R is one of the above four relations and MLP ( k ) R is the MLP for R . For example, for S 0 in Figure 2 (a), the embedding of M 1 in the ( k +1) -st iteration can be derived as follows.

Similarly, the embedding of O 1 , 1 in the ( k +1) -st iteration is: 229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

In our approach, an action includes the two phases, graph embedding phase and action selection phase. Let h ( k ) G denote the whole embedding of the graphs G , a summation of the embeddings of all nodes, h ( k +1) v . In the graph embedding phase, we use an HGIN to encode node and graph embeddings as described above. An example with three HGIN layers is illustrated in Figure 3 (b).

In the action selection phase, we select an action based on a policy, after node and graph embedding are encoded in the graph embedding phase. The policy is described as follows. First, collect all ready operations O to be dispatched to machines M . Then, for all pairs ( M , O ), feed their node embeddings ( h ( k ) M , h ( k ) O ) into a MLP Score ( M,O ) to calculate their scores as shown in Figure 3 (c). The probability of selecting ( M , O ) is calculated based on a softmax function of all scores, which also serves as the model policy π for the current state.

3.4 Policy-Based RL Training

In this paper, we propose to use a policy-based RL training mechanism that follows REINFORCE (Sutton and Barto, 2018) to update our model by policy gradient with a normalized advantage makespan with respect to a baseline policy π b as follows.

In this paper, we choose a lightweight PDR, MWKR, as baseline π b , which performed best for makespan among all PDRs reported from the previous work (Zhang et al., 2020). In fact, our experiment also shows that using MWKR is better than the other PDRs shown in the appendix. The model for policy π is parametrized by θ , which is updated by ∇ θ logπ θ A π θ ( S t , a t ) . Our algorithm based on REINFORCE is listed in the appendix.

4 Experiments

4.1 Experimental Settings and Evaluation Benchmarks

In our experiments, the settings of our model are described as follows. All embedding and hidden vectors in our model have a dimension of 256. The model contains three HGIN layers for graph embedding, and an MLP for the score function, as shown in Figure 3 (b) and (c). All MLP networks including those in HGIN and for score contain two hidden layers. The parameters of our model, such as MLP, generally follow the default settings in PyTorch (Paszke et al., 2019) and PyTorch Geometric (Fey and Lenssen, 2019). More settings are in the appendix.

Each of our models is trained with one million episodes, each with one scheduling instance. Each instance is generated by following the procedure which is used to generate the TA dataset (Taillard, 1993). Given ( N , M ), we use the procedure to generate an n × m JSP instance by conforming to the following distribution, n ∼ U (3 , N ) , m ∼ U (3 , n ) , and operation count k j = m , where U ( x, y ) represents a distribution that uniformly samples an integer in a close interval [ x, y ] at random. The details of designation for machines and processing times refer to (Taillard, 1993) and thus are omitted here. We choose (10,10) for all experiments, since (10,10) generally performs better than the other two as described in the appendix. Following the method described in Subsection 3.4, the model is updated from the above randomly generated instances. For testing our models for JSP and FJSP, seven JSP open benchmarks and two FJSP open benchmarks are used, as listed in the appendix.

The performance for a given policy method π on an instance is measured by the makespan gap G defined as

where T ( mksp ) π ∗ is the optimal makespan or the best-effort makespan, from a mathematical optimization tool, OR-Tools, serving as π ∗ . By the best-effort makespan, we mean the makespan derived with a

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

sufficiently large time limitation, namely half a day with OR-Tools. For comparison in experiments, we use a server with Intel Xeon E5-2683 CPU and a single NVIDIA GeForce GTX 1080 Ti GPU. Our method uses a CPU thread and a GPU to train and evaluate, while OR-Tools uses eight threads to find the solution.

4.2 Experiments for JSP

For JSP, we first train a model based on residual scheduling, named RS. For ablation testing, we also train a model, named RS+op, by following the same training method but without removing irrelevant operations. When using these models to solve testing instances, action selection is based on the greedy policy that simply chooses the action ( M,O ) with the highest score deterministically, obtained from the score network as in Figure 3 (c).

For comparison, we consider the three DRL construction heuristics, respectively developed in (Zhang et al., 2020) called L2D, (Park et al., 2021b) by Park et al., and (Park et al., 2021a), called ScheduleNet. We directly use the performance results of these methods for open benchmarks from their articles. For simplicity, they are named L2D, Park and SchN respectively in this paper. We also include some construction heuristics based PDR, such as MWKR, MOR, SPT and FIFO. Besides, to derive the gaps to the optimum in all cases, OR-Tools serve as π ∗ as described in (6).

Now, let us analyze the performances of RS as follows. Table 2 shows the average makespan gaps for each collection of JSP TA benchmarks with sizes, 15×15, 20×15, 20×20, 30×15, 30×20, 50×15, 50×20 and 100×20, where the best performances (the smallest gaps) are marked in bold. In general, RS performs the best, and generally outperforms the other methods for all collections by large margins, except for that it has slightly higher gaps than RS+op for the two collections, 15 × 15 and 20 × 20. In fact, RS+op also generally outperforms the rest of methods, except for that it is very close to SchN for two collections. For the other six open benchmarks, ABZ, FT, ORB, YN, SWV and LA, the performances are similar and thus presented in the appendix. It is concluded that RS generally performs better than other construction heuristics by large margins.

4.3 Experiments for FJSP

For FJSP, we also train a model based on residual scheduling, named RS, and a ablation version, 297 named RS+op, without removing irrelevant operations. We compares ours with one DRL construction 298 heuristics developed by (Song et al., 2023), called DRL-G, and four PDR-based heuristics, MOR, 299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

MWKR, SPT and FIFO. We directly use the performance results of these methods for open datasets according to the reports from (Song et al., 2023).

Table 3 shows the average makespan gaps in the four open benchmarks, MK, LA(rdata), LA(edata) and LA(vdata). From the table, RS generally outperforms all the other methods for all benchmarks by large margins, except for that RS+op is slightly better for the benchmark LA(vdata).

5 Discussions

In this paper, we propose a new approach, called residual scheduling, to solving JSP an FJSP problems, and the experiments show that our approach reaches SOTA among DRL-based construction heuristics on the above open JSP and FJSP benchmarks. We further discusses three issues: large instances, computation times and further improvement.

0.25

0.20

二

RS

p

e

0.15

L2D

G

0.10

0.05

0.00

15x15

20x15

20x20

30x15

30x20

50x15

50x20

100x15

100x20

150x15

150x20

200x15

200x20

Problemsize

First, from the above experiments particularly for TA benchmark for JSP, we observe that the average gaps gets smaller as the number of jobs increases, even if we use the same model trained with ( N,M ) = (10 , 10) . In order to investigate size-agnostics, we further generate 13 collections of JSP instances of sizes for testing, from 15 × 15 to 200 × 20, and generate 10 instances for each collection by using the procedure above. Figure 4 shows the average gaps for these collections for RS and L2D, and these collections are listed in the order of sizes in the x-axis. Note that we only show the results of L2D in addition to our RS, since L2D is the only open-source among the above DRL heuristics. Interestingly, using RS, the average gaps are nearly zero for the collections with sizes larger than 100 × 15, namely, 100 × 15, 100 × 20, 150 × 15, 200 × 15 and 200 × 20. Among the 50 JSP instances in the five collections, 49 reaches zero gaps. A strong implication is that our RS approach can be scaled up for job sizes and even reach the optimal for sufficient large job count.

Second, the computation times for RS are relatively small and has low variance like most of other construction heuristics. Here, we just use the collection of TA 100x20 for illustration. It takes about 30 seconds on average for both RS and RS+op, about 28 for L2D and about 444 for SchN. In contrast, it takes about 4000 seconds with high variance for OR-tools. The times for other collections are listed in more detail in the appendix.

Third, as proposed by Song et al. (2023), construction heuristics can further improve the gap by constructing multiple solutions based on the softmax policy, in addition to the greedy policy. They had a version constructing 100 solutions for FJSP, called DRL+100 in this paper. In this paper, we also implement a RS version for FJSP based on the softmax policy, as described in Subsection 3.3, and then use the version, called RS+100, to constructing 100 solutions. In Table 4, the experimental results show that RS+100 performs the best, much better than RS, DRL-G and DRL+100. An important property for such an improvement is that constructing multiple solutions can be done in parallel. That is, for construction heuristics, the solution quality can be improved by adding more computation powers.

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

References

Majid Abdolrazzagh-Nezhad and Salwani Abdullah. 2017. Job Shop Scheduling: Classification, Constraints and Objective Functions. International Journal of Computer and Information Engineering 11, 4 (2017), 429-434.

Joseph William Adams, Egon Balas, and Daniel J. Zawack. 1988. The Shifting Bottleneck Procedure for Job Shop Scheduling. Management science 34, 3 (1988), 391-401.

David L. Applegate and William J. Cook. 1991. A Computational Study of the Job-Shop Scheduling Problem. INFORMS Journal on Computing 3, 2 (1991), 149-156. https://doi.org/10.1287/ ijoc.3.2.149

Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. 2018. Relational inductive biases, deep learning, and graph networks. CoRR abs/1806.01261 (2018). arXiv:1806.01261 http://arxiv.org/abs/1806.01261

Dennis Behnke and Martin Josef Geiger. 2012. Test instances for the flexible job shop scheduling problem with work centers. Arbeitspapier/Research Paper/Helmut-Schmidt-Universität, Lehrstuhl für Betriebswirtschaftslehre, insbes. Logistik-Management (2012).

Paolo Brandimarte. 1993. Routing and scheduling in a flexible job shop by tabu search. Ann. Oper. Res. 41, 3 (1993), 157-183. https://doi.org/10.1007/BF02023073

IBM ILOG Cplex. 2009. V12. 1: User's Manual for CPLEX. International Business Machines Corporation 46, 53 (2009), 157.

Mauro Dell'Amico and Marco Trubian. 1993. Applying tabu search to the job-shop scheduling problem. Annals of Operations Research 41, 3 (1993), 231-252. https://doi.org/10.1007/ BF02023076

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. 2022. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature 610, 7930 (2022), 47-53.

Matthias Fey and Jan Eric Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric, In ICLR Workshop on Representation Learning on Graphs and Manifolds. CoRR abs/1903.02428. arXiv:1903.02428 http://arxiv.org/abs/1903.02428

R. Garey and David S. Johnson. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness . W. H. Freeman, USA.

Amit Kumar Gupta and Appa Iyer Sivakumar. 2006. Job shop scheduling techniques in semiconductor manufacturing. The International Journal of Advanced Manufacturing Technology 27, 11 (2006), 1163-1169.

Reinhard Haupt. 1989. A survey of priority rule-based scheduling. Operations-Research-Spektrum 11, 1 (1989), 3-16.

Johann Hurink, Bernd Jurisch, and Monika Thole. 1994. Tabu search for the job-shop scheduling problem with multi-purpose machines. Operations-Research-Spektrum 15 (1994), 205-215.

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning (ICML) (JMLR Workshop and Conference Proceedings, Vol. 37) , Francis R. Bach and David M. Blei (Eds.). JMLR.org, 448-456. http://proceedings.mlr.press/v37/ioffe15.html

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

Klaus Jansen, Monaldo Mastrolilli, and Roberto Solis-Oba. 2000. Approximation Algorithms for Flexible Job Shop Problems. In Latin American Symposium on Theoretical Informatics (Lecture Notes in Computer Science, Vol. 1776) , Gaston H. Gonnet, Daniel Panario, and Alfredo Viola (Eds.). Springer, 68-77. https://doi.org/10.1007/10719839_7

BJ Lageweg, JK Lenstra, and AHG Rinnooy Kan. 1977. Job-shop scheduling by implicit enumeration. Management Science 24, 4 (1977), 441-450.

Stephen Lawrence. 1984. Resouce constrained project scheduling: An experimental investigation of heuristic scheduling techniques (Supplement). Graduate School of Industrial Administration, Carnegie-Mellon University (1984).

Zhigang Lian, Bin Jiao, and Xingsheng Gu. 2006. A similar particle swarm optimization algorithm for job-shop scheduling to minimize makespan. Appl. Math. Comput. 183, 2 (2006), 1008-1017. https://doi.org/10.1016/j.amc.2006.05.168

Chun-Cheng Lin, Der-Jiunn Deng, Yen-Ling Chih, and Hsin-Ting Chiu. 2019. Smart Manufacturing Scheduling With Edge Computing Using Multiclass Deep Q Network. IEEE Trans. Ind. Informatics 15, 7 (2019), 4276-4284. https://doi.org/10.1109/TII.2019.2908210

Min Liu, Zhi-jiang Sun, Junwei Yan, and Jing-song Kang. 2011. An adaptive annealing genetic algorithm for the job-shop planning and scheduling problem. Expert Systems with Applications 38, 8 (2011), 9248-9255. https://doi.org/10.1016/j.eswa.2011.01.136

Shu Luo. 2020. Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Applied Soft Computing 91 (2020), 106208. https://doi.org/10.1016/j. asoc.2020.106208

Mingqi Lv, Zhaoxiong Hong, Ling Chen, Tieming Chen, Tiantian Zhu, and Shouling Ji. 2021. Temporal Multi-Graph Convolutional Network for Traffic Flow Prediction. IEEE Transactions on Intelligent Transportation Systems 22, 6 (2021), 3337-3348. https://doi.org/10.1109/ TITS.2020.2983763

Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim M. Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, Jiwoo Pak, Andy Tong, Kavya Srinivasa, Will Hang, Emre Tuncer, Quoc V. Le, James Laudon, Richard Ho, Roger Carpenter, and Jeff Dean. 2021. A graph placement methodology for fast chip design. Nature 594, 7862 (2021), 207-212.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529-533. https://doi.org/10.1038/nature14236

J.F. Muth and G.L. Thompson. 1963. Industrial Scheduling . Prentice-Hall.

In-Beom Park, Jaeseok Huh, Joongkyun Kim, and Jonghun Park. 2020. A Reinforcement Learning Approach to Robust Scheduling of Semiconductor Manufacturing Facilities. IEEE Transactions on Automation Science and Engineering 17, 3 (2020), 1420-1431. https://doi.org/10.1109/ TASE.2019.2956762

Junyoung Park, Sanjar Bakhtiyar, and Jinkyoo Park. 2021a. ScheduleNet: Learn to solve multi-agent scheduling problems with reinforcement learning. CoRR abs/2106.03051 (2021). arXiv:2106.03051 https://arxiv.org/abs/2106.03051

Junyoung Park, Jaehyeong Chun, Sang Hun Kim, Youngkook Kim, and Jinkyoo Park. 2021b. Learning to schedule job-shop problems: representation and policy learning using graph neural network and reinforcement learning. International Journal of Production Research 59, 11 (2021), 3360-3377. https://doi.org/10.1080/00207543.2020.1870013

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, HighPerformance Deep Learning Library. In Neural Information Processing Systems (NeurIPS) , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 8024-8035. https://proceedings.neurips.cc/paper/2019/ hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html

Laurent Perron and Vincent Furnon. 2019. OR-Tools . Google. https://developers.google. com/optimization/

Ferdinando Pezzella, Gianluca Morganti, and Giampiero Ciaschetti. 2008. A genetic algorithm for the Flexible Job-shop Scheduling Problem. Computers and Operations Research 35, 10 (2008), 3202-3212. https://doi.org/10.1016/j.cor.2007.02.014

Qing-dao-er-ji Ren and Yuping Wang. 2012. A new hybrid genetic algorithm for job shop scheduling problem. Computers and Operations Research 39, 10 (2012), 2291-2299. https://doi.org/ 10.1016/j.cor.2011.12.005

Mohammad Saidi-Mehrabad and Parviz Fattahi. 2007. Flexible job shop scheduling with tabu search algorithms. The International Journal of Advanced Manufacturing Technology 32, 5 (2007), 563-570.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484-489. https://doi.org/10.1038/nature16961

Wen Song, Xinyang Chen, Qiqiang Li, and Zhiguang Cao. 2023. Flexible Job-Shop Scheduling via Graph Neural Network and Deep Reinforcement Learning. IEEE Trans. Ind. Informatics 19, 2 (2023), 1600-1610. https://doi.org/10.1109/TII.2022.3189725

Robert H. Storer, S. David Wu, and Renzo Vaccari. 1992. New search spaces for sequencing problems with application to job shop scheduling. Management science 38, 10 (1992), 1495-1509.

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction (second ed.). The MIT Press. http://incompleteideas.net/book/the-book-2nd.html

Éric D. Taillard. 1993. Benchmarks for basic scheduling problems. european journal of operational research 64, 2 (1993), 278-285.

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 7782 (2019), 350-354. https://doi.org/10.1038/s41586-019-1724-z

Bernd Waschneck, Thomas Altenmüller, Thomas Bauernhansl, and Andreas Kyek. 2016. Production Scheduling in Complex Job Shops from an Industry 4.0 Perspective: A Review and Challenges in the Semiconductor Industry. In Proceedings of the 1st International Workshop on Science, Application and Methods in Industry 4.0 (i-KNOW) (CEUR Workshop Proceedings, Vol. 1793) , Roman Kern, Gerald Reiner, and Olivia Bluder (Eds.). CEUR-WS.org. http://ceur-ws.org/ Vol-1793/paper3.pdf

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How Powerful are Graph Neural Networks? (2019). https://openreview.net/forum?id=ryGs6iA5Km

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

Takeshi Yamada and Ryohei Nakano. 1992. A Genetic Algorithm Applicable to Large-Scale JobShop Problems. In Parallel Problem Solving from Nature 2, (PPSN-II) , Reinhard Männer and Bernard Manderick (Eds.). Elsevier, 283-292.

Cong Zhang, Wen Song, Zhiguang Cao, Jie Zhang, Puay Siew Tan, and Chi Xu. 2020. Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning. In Neural Information Processing Systems (NeurIPS) , Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, MariaFlorina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/ 2020/hash/11958dfee29b6709f48a9ba0387a2431-Abstract.html

Cong Zhang, Wen Song, Zhiguang Cao, Jie Zhang, Puay Siew Tan, and Chi Xu. 2022. Learning to Search for Job Shop Scheduling via Deep Reinforcement Learning. CoRR abs/2211.10936 (2022). https://doi.org/10.48550/arXiv.2211.10936 arXiv:2211.10936

Cong Zhang, Yaoxin Wu, Yining Ma, Wen Song, Zhang Le, Zhiguang Cao, and Jie Zhang. 2023. A review on learning to solve combinatorial optimisation problems in manufacturing. IET Collaborative Intelligent Manufacturing 5, 1 (2023), e12072. https://doi.org/10.1049/cim2.12072 arXiv:https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/cim2.12072

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020. Graph neural networks: A review of methods and applications. AI Open 1 (2020), 57-81. https://doi.org/10.1016/j.aiopen.2021.01.001

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the problem of learning a graph neural network based policy network as a construction heuristic for solving job shop scheduling problems. This paper proposed an idea called residual scheduling to remove irrelevant operations and machines from the graph based state representation. This has been shown experimentally to perform well on several benchmark job shop scheduling problem instances.\n\nSTRENGTHS:\nWith the fast advancement of deep learning technologies, it becomes increasingly interesting to develop deep neural network models that can effectively solve complex combinatorial optimization problems, including job shop scheduling problems. The newly developed deep learning system in this paper appears to be very effective and highly competitive in performance, compared to similar approaches from the literature.\n\nWEAKNESSES:\nThe idea of using graph neural networks or other forms of deep neural networks trained through reinforcement learning to solve job shop scheduling problems has been studied in many past research works. The main text of this paper lacks a comprehensive review of these research works, making it hard to clearly understand the key technical novelty and contribution of this paper, compared to other recently published works.\n\nThe development of the residual scheduling technique is not strongly motivated in this paper. It is hard to understand why it is essential to remove irrelevant operations and machines from the graph based state representation. The development of this technique also appears to be very highly intuitive and lacks thorough theoretical analysis. Hence, the technical contribution of this development remains largely questionable.\n\nThe authors stated that in order for the process of learning the construction heuristic to be formulated as an MDP, they introduced several attributes for the operation nodes and machine nodes in the graph based state representation. However, it remains largely unknown whether, with the introduced attributes, the state representation can satisfy the Markov property of the MDP. The importance of using any newly introduced attributes should be more thoroughly evaluated experimentally. The associated technical contributions should be clarified and strongly justified.\n\nSome other key aspects of the new system design should also be justified more. For example, the use of GIN needs to be supported with more convincing reasons. The focus on learning a construction heuristic rather than an improvement heuristic, which is gaining increasing popularity and attention, should be better justified and experimentally validated in this paper.\n\nQUESTIONS:\nWhat are the key novelties of this research work, compared to existing deep learning based methods for combinatorial optimization, including job shop scheduling?\n\nAt the theoretical level, why is it essential to remove irrelevant operations and machines from the graph based state representation?\n\nWith the newly introduced attributes, why will the state representation satisfy the Markov property of the MDP?\n\nWhat are the theoretical and empirical advantages of learning a construction heuristic, in comparison to learning an improvement heuristic?\n\nLIMITATIONS:\nI do not have any concerns regarding this question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

comments: SUMMARY:
The paper introduces a novel approach called residual scheduling for solving the Job-shop scheduling problem (JSP) and its variant, flexible JSP (FJSP), focusing on removing irrelevant machines and jobs from the consideration set. Despite these problems being NP-hard, the proposed method demonstrates state-of-the-art performance across standard benchmarks, even performing well when scaled to larger problem sizes, achieving a zero gap in 49 out of 50 instances with more than 150 jobs on 20 machines.

STRENGTHS:
The method developed claims that for the 98% of the cases, the zero gap is achieved for fairly large instances.

WEAKNESSES:
The zero gap is mentioned. Upon reading, readers find out that the gap refers to "makespan gap." Understandably, significant bulk of existing papers on job-shop focus on makespan rather than tardiness. Ignoring tardiness may lead to poorer on time delivery, which is a weakness in itself, but with respect to the "makespan gap" measure it is not clear whether zero gap will result in zero (or small gap) is tardiness is considered or, generally, if the standard duality or MIP gaps are used instead. The gap is misleading at best.

QUESTIONS:
The paper claims to beath SOTA heuristics. Lagrangian heuristic is definitely missing. How does the new method compare against Lagrangian heuristic? Understandably, Lagrangian relaxation has been used for a long time, the reviewer is asking about recent rather than historical Lagrangian heuristics, since their quality may differ.

LIMITATIONS:
In light of the above comments, the limitations are 1. only makespan is considered, 2. only one type of gap seems to be introduced and considered, and 3. major heuristics are not used/compared with.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed DRL based method to learn dispatching polices for (flexible) job-shop scheduling problems (JSP/FJSP). The main idea is to remove the completed operations from the state embedding, which is called residual scheduling, so as to improve the representation accuracy. The DRL agent uses a graph representation, which is processed by a Graph Neural Network (GNN) architecture. Experiments on JSP and FJSP benchmarks show that the proposed residual scheduling scheme outperforms recent DRL baselines.\n\nSTRENGTHS:\n1. The idea of residual scheduling makes great sense and is interesting.\n\n2. The method is generally applicable to both JSP and FJSP, which are important scheduling problems.\n\n3. Good empirical performance, comparing to recent DRL based scheduling methods.\n\nWEAKNESSES:\n1. The main weakness is that the technical contribution is incremental. While the redisual scheduling idea is interesting and novel, a large part of the proposed method is similar to existing works. Specifically, the graph representation and heterogeneous graph neural network in Section 3.2 and 3.3 is similar to the heterogeneous graph and heterogeneous GNN in (Song et al., 2023). This is not mentioned in Section 3, and the differences between the proposed method and existing works are not discussed.\n\n2. Some design choices need to be justified. Please see the below questions.\n\n3. Empirical evaluation needs to be improved. \n\n* It is unclear whether the models in other works (e.g. L2D, SchN, DRL-G) are retrained using the same dataset as the proposed method. If not, then directly comparing their performance (even on the same benchmark instances) is not fair due to different training data.\n\n* The discussion for Figure 4 is not surprising. It is well known that for JSP, problems with larger $n/m$ ratios are easier to solve (Taillard 1993). That is why the gaps on large problems in Figure 4 are smaller. Actually this is true for most algorithms, and cannot be claimed as a major advantage of the proposed method.\n\n* Training time is not reported.\n\n4. The authors made several inappropriate statements in the paper, mainly in introduction. The authors should be more precise about the related concepts.\n\n* In the first paragraph, it is better to describe JSP as a combinatorial optimization problem, instead of mathematical optimization.\n\n* In the second paragraph, Constraint Satisfaction Problem (CSP) should be not stated as a type of mathematical optimization. In addition, Constraint Programming (CP) here is more suitable than CSP. \n\n* In the fourth paragraph, the approaches that automate the design of heuristics should be hyperheuristics, instead of metaheuristics.\n\n4. The language needs to be improved. Besides, there are quite a few grammar errors and typos.\n\nQUESTIONS:\n1. Why use fully connected edges to link the operations in the same job? How about using only precedence relationship as a directed edge?\n\n2. The second graph below Figure 3, for flexible problems, how to define the ready status for an operation considering there could be multiple compatible machines?\n\n3. According to Figure 3(b), the graph embedding $h_{\\mathcal{G}}^{(k)}$ is not used in the action score prediction as in Figure 3(c). So what is $h_{\\mathcal{G}}^{(k)}$ used for?\n\n4. In the experiments, to generate number of jobs and machines, why use the uniform distribution $\\mathcal{U}(3, N)$ and $\\mathcal{U}(3,n)$ (should be $\\mathcal{U}(3,M)$ for machines)? The value 3 seems arbitrary.\n\nLIMITATIONS:\nThe authors did not give a particular discussion on the limitations. One limitation could be the computational efficiency. As shown in Table 13 in the appendix, the runtime increase rapidly with the number of jobs. This could affect both training and inference efficiency, and limit its applicability to larger problems.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a deep reinforcement learning-based constructive heuristic to solve the (Flexible) Job Shop Scheduling Problem. An instance of the problem is represented as a graph and fed into a Graph Neural Network-based model which outputs a score for each candidate (operation-machine) pair. The model is trained with the REINFORCE algorithm with as baseline a classic Priority Dispatching Rule-based heuristic. The novelty of the paper lies in the update of the state after each action: the graph is updated by removing the operations which have already been executed to focus on the most relevant information, which is the residual operations and remaining times of the ongoing ones. The proposed approach is experimentally shown to outperform RL-based constructive heuristics on classic JSSP and FJSSP benchmarks.\n\nSTRENGTHS:\n1. Sound and interesting idea of removing irrelevant information from the state \n1. The paper is fairly clear and I appreciated the illustrations Fig 1-3.\n1. The model was proposed for the JSSP and easily adapted to the Flexible JSSP\n1. The approach outperforms deep RL-based construction heuristics on classic benchmarks\n\nWEAKNESSES:\n1. Limited novelty: the main contribution is the definition of the residual state at each step of the construction process by removing irrelevant operations and resetting the time reference. This seems to me an incremental improvement of the approach L2D [1], which already proposed a similar state graph representation and the use of the Graph Isomorphism Network architecture for the JSSP.\n1. The proposed approach seems very specific to (F)JSSPs (state representation, baseline) and it's not clear what could be transferable to DRL heuristics for solving other optimization problems.\n1. In the experiments, the presented non-learning-based baselines seem pretty weak: only greedy (see more in Questions)\n1. Lots of English typos (I noted a few per page)\n\n[1] C Zhang et al, Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning, Neurips 2020\n\nQUESTIONS:\n1. Where dos the attributes of edges (L205) appear in the GNN model (Sec 3.3)?\n1. The paper claims: “Interestingly, using RS, the average gaps are nearly zero for the collections with sizes larger than 100 … A strong implication is that our RS approach can be scaled up for job sizes and even reach the optimal for sufficient large job count.” Do you have an idea of why the model would work better on unseen large instances versus instances of the same size as the training ones? Does OR-tools return the optimal solutions for these larger instances? Can it be that the quality of the reference solutions decreases and therefore the “optimality” gap becomes smaller?\n1. Are there stronger non-learning-based baselines for the JSPP other than the greedy PDR heuristics presented in Table 2? To be able to appreciate the performance of the proposed approach, it would be useful and more convincing to compare it to the best heuristics for this problem, beyond simple greedy ones. For example maybe [2]?\n1. Appendix, Algorithm 1, Line 6:  to compute this makespan at state s_t, given action a_t, do you do a rollout of the current policy \\pi_{\\theta} with the updated parameter \\theta? This would mean doing a rollout until the end of both the baseline and current policy at each step of the trajectory? How long did the training take?\n1. How is the average computation time computed? (Table 13) In particular, was the policy applied to each instance individually or were instances batched? \n1. The discussion L322 about the time it takes for RS/L2D/ScheduleNet versus OR-Tools can be a bit misleading: OR-tools is an exact solver. If spends a lot of time proving the optimality of the solution. Probably it could return a good quality solution much faster if used as a heuristic. In addition it runs on CPUs and not GPUs therefore just comparing the time is not really fair. \n\n[2] CY Zhang, A tabu search algorithm with a new neighborhood structure for the job shop scheduling problem, Computers & Operations Research, 2007\n\nLIMITATIONS:\nNot explicitly addressed by the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}