Title: Learning Memory-Enhanced Improvement Heuristics for Flexible Job Shop Scheduling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Flexible Job-Shop Scheduling Problem

Graph Neural Network

Reinforcement Learning

Combinatorial Optimization
Abstract: The rise of smart manufacturing under Industry 4.0 introduces mass customization and dynamic production, demanding more advanced and flexible scheduling techniques. The flexible job-shop scheduling problem (FJSP) has attracted significant attention due to its complex constraints and strong alignment with real-world production scenarios. Current deep reinforcement learning (DRL)-based approaches to FJSP predominantly employ constructive methods. While effective, they often fall short of reaching (near-)optimal solutions. In contrast, improvement-based methods iteratively explore the neighborhood of initial solutions and are more effective in approaching optimality. However, the flexible machine allocation in FJSP poses significant challenges to the application of this framework, including accurate state representation, effective policy learning, and efficient search strategies. To address these challenges, this paper proposes a $\textbf{M}$emory-enhanced $\textbf{I}$mprovement $\textbf{S}$earch framework with he$\textbf{t}$erogeneous gr$\textbf{a}$ph $\textbf{r}$epresentation—$\textit{MIStar}$. It employs a novel heterogeneous disjunctive graph that explicitly models the operation sequences on machines to accurately represent scheduling solutions. Moreover, a memory-enhanced heterogeneous graph neural network (MHGNN) is designed for feature extraction, leveraging historical trajectories to enhance the decision-making capability of the policy network. Finally, a parallel greedy search strategy is adopted to explore the solution space, enabling superior solutions with fewer iterations. Extensive experiments on synthetic data and public benchmarks demonstrate that $\textit{MIStar}$ significantly outperforms both traditional handcrafted improvement heuristics and state-of-the-art DRL-based constructive methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Deep Reinforcement Learning (DRL)-based improvement heuristic framework named MIStar to solve FJSP problems. The model is presented to address (1) complex state representation by a novel heterogeneous disjunctive graph, (2) effective policy learning by the proposed heterogeneous graph representation learning, and (3) efficient search strategies by a parallel greedy search strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The integration of memory mechanisms with heterogeneous graph neural networks is innovative and well-suited for FJSP.\n2. The proposed method outperforms the traditional heuristics and DRL-based constructive methods across both synthetic and real-world benchmarks.\n\nWeakness:\n1. The analysis of embedding is not presented, even the authors claim that the embeddings are significant for memorizing historical data and extracting state features. \n2. It is unclear whether the GIN and GAT are de facto significant and effective for topological information encoding and semantic information enhancing respectively, referring to many advanced GNN models. More analysis related to the module selection in operation node embedding should be provided.\n3. It is very common to witness changes in the functionality of devices (some devices are capable of doing two kinds of actions), alterations in the quantity of equipment, and sudden shifts in the priority of the job-shops in real smart factories. Although this is a technical work for solving FJSP only, I would cordially encourage the authors to discuss if MIStar can address those challenges or if the authors will consider these in the future works. \n4. Although the answer to open access was No, I would encourage the authors to open source the code or provide detailed instructions for dataset generation and modeling.\n\nQUESTIONS:\nPlease refer to the above weaknesses.\n\nLIMITATIONS:\nPlease refer to the above weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MIStar, a Deep Reinforcement Learning (DRL) framework for the Flexible Job Shop Scheduling Problem (FJSP). The key idea is to move away from constructive DRL methods, which build a schedule from scratch, and instead focus on improvement-based methods that iteratively refine a complete solution. To this end, the authors formulate the problem as a Markov Decision Process (MDP) where the state is a complete schedule and actions correspond to local moves in the Nopt2 neighborhood. \nThe main contributions of the paper are:\n1. A directed heterogeneous disjunctive graph representation for FJSP solutions, which explicitly includes machine nodes and their processing sequences using hyper-arcs.\n2. A Memory-enhanced Heterogeneous Graph Neural Network (MHGNN) that encodes the graph state and leverages a memory of previously visited solutions to enhance decision-making and avoid local optima.\n3. A parallel greedy search strategy that evaluates multiple candidate actions at each step to improve search efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe paper tackles the FJSP, a well-known NP-hard problem with significant industrial relevance. The framing of the problem as an improvement task learned via DRL is a promising direction. The authors correctly identify that most recent DRL work has focused on constructive methods, which often struggle to find high-quality solutions. Shifting the focus to learning improvement heuristics is a valuable contribution to the field of neural combinatorial optimization.\n- Well-Motivated Methodological Components: Each component of the proposed MIStar framework is well-motivated by a specific challenge in applying improvement methods to FJSP.\n\nWeaknesses\n\n- Ablation Study Missing: There is no evidence or ablation study provided to isolate the benefit of the memory-enhanced heterogeneous graph neural network (MHGNN) compared to standard GNNs without memory. It is unclear how much the memory module contributes to overall performance. The memory module is a core contribution, but its design raises questions. The state similarity is calculated using the Frobenius inner product of the operation-machine incidence matrix (Lt​). This matrix only captures which machine an operation is assigned to, but not the processing sequence on that machine. It is therefore possible for two schedules with very different makespans (e.g., a good sequence vs. a poor sequence on the same machine) to be considered identical or highly similar by this metric. This seems like a significant oversimplification that could lead to misleading historical information. The paper needs to provide a much stronger justification for why this simplified representation is sufficient.\n- The paper employs parallel greedy exploration runs, but it is not clear whether the computational budget is held constant across all compared models. If MIStar is allowed to explore more solutions in parallel, it may have an unfair advantage in solution quality.\n- The term \"agent\" (e.g., \"agent selects an action...\") is introduced without prior definition, which may reduce clarity for readers unfamiliar with the terminology.\n-The descriptions accompanying Table 2 are brief, and the tables themselves are dense and potentially overwhelming. The paper would benefit from additional visualizations to better communicate model comparisons and results.\n- The paper does not provide code for reproducing the results, limiting the ability of the community to validate and build upon this work.\nLimited Choice of \"Hand-crafted\" Baselines: While the inclusion of GD, BI, and FI is a good start, they are relatively basic local search algorithms. The field of metaheuristics for FJSP is mature, with highly effective algorithms like Tabu Search (e.g., the original Brandimarte paper ) and Genetic Algorithms. \nIncremental Novelty of Graph Representation: The paper introduces a \"novel directed heterogeneous disjunctive graph\". However, it is explicitly \"inspired by the heterogeneous disjunctive graph H [8]\", which is the graph representation used by the HGNN baseline.\n\nQUESTIONS:\nQuestions for the Authors:\n\nThe state similarity for the memory module is based on the operation-machine incidence matrix, which ignores the processing order on each machine. Could you please justify this design choice? Have you considered or experimented with more expressive representations for historical states that also encode sequence information, and if so, how did they perform?\n\nRegarding the experimental runtimes reported in Table 1 and Table 2: Are the reported times for the baseline methods (GD, FI, BI) the aggregate time for running searches from 100 different initial solutions? Is the reported time for MIStar for a single search trajectory, or is it also an aggregate? Please clarify how the \"search effort\" is normalized between MIStar and the rule-based baselines to ensure a fair comparison of both solution quality and runtime.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MIStar, a novel DRL-based improvement heuristic framework for solving the Flexible Job Shop Scheduling Problem (FJSP). Unlike prior approaches that primarily rely on constructive methods, MIStar starts from a complete schedule and iteratively refines it using learned improvement heuristics. Key innovations include: 1) A heterogeneous disjunctive graph representation that models operation sequences and machine assignments explicitly. 2) A memory-enhanced heterogeneous graph neural network (MHGNN) that utilizes past scheduling experiences to improve decision making and reduce local optima traps. 3) A parallel greedy search strategy for efficient exploration of the solution space. Experiments on synthetic and public benchmark datasets demonstrate that MIStar outperforms state-of-the-art DRL-based constructive methods and handcrafted improvement heuristics in both quality and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed method is well-motivated and carefully implemented. The combination of memory-augmented GNNs and improvement heuristics is novel and technically sound.\n\nThe methodology is presented in a well-structured manner, with comprehensive figures, examples, and ablation studies to support understanding.\n\nFJSP is a critical real-world problem, and improving its solution quality has significant industrial implications. This work contributes a viable learning-based improvement framework with promising generalization capabilities.\n\nTo the best of my knowledge, this is the first work that brings memory-enhanced GNNs into DRL-based improvement heuristics for FJSP, addressing challenges that existing methods often overlook.\n\nWhile strong comparisons are made against DRL-based constructive and rule-based improvement methods, the absence of comparisons with classic or recent dynamic scheduling algorithms limits the broader applicability assessment of MIStar.\n\nThe framework assumes static environments. In realistic smart manufacturing, events like new job insertions or machine breakdowns are common but not considered or simulated in experiments.\n\nEmbedding choices: The paper uses GIN and GAT for operation nodes and a hetero-GAT for machine embeddings, but ablation studies on alternative embedding strategies or modules (e.g., Transformer variants) are missing.\n\nThe reward function is based on makespan improvement and redundancy penalties, but no detailed reward curve or sensitivity analysis is presented, making it difficult to assess stability.\n\nWhile performance is reported in tables, error bars or statistical confidence measures are absent, making reproducibility and robustness harder to judge.\n\nQUESTIONS:\nHow would MIStar adapt to real-time job arrivals or unexpected machine unavailability? The current formulation assumes a fixed problem instance. Please clarify whether the memory mechanism and graph representation can support online updates or dynamic re-optimization.\n\nCan the authors provide a reward curve during training? Understanding how quickly the policy converges and how often it gets stuck in local optima would strengthen the interpretation of results and help in reproducing or fine-tuning the model.\n\nWhy was no ablation on embedding architecture included? Multiple GNN modules are used, including GIN and GAT. How crucial are each of these components? A component-wise breakdown would clarify their individual contributions.\n\nLIMITATIONS:\nNo support for unexpected events: The current model assumes a static schedule and does not consider dynamic changes such as new job insertions or maintenance events. This may limit deployment in real-world systems where dynamic events are common.\n\nNo error bars in reported metrics: Performance results lack standard deviation or confidence intervals, making robustness unclear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MIStar, a Memory-enhanced Improvement Search framework designed to tackle the Flexible Job Shop Scheduling Problem (FJSP). Traditional deep reinforcement learning approaches often fall short of optimal solutions for FJSP, while improvement-based methods, though effective, face challenges with state representation and efficient search strategies due to flexible machine allocation. MIStar addresses these issues by employing a novel heterogeneous disjunctive graph for accurate solution representation, a memory-enhanced heterogeneous graph neural network (MHGNN) for improved decision-making through historical data, and a parallel greedy search strategy for efficient exploration.\n\nSTRENGTHS AND WEAKNESSES:\n- Strength\n  - A DRL-based improvement heuristic framework specifically designed for the FJSP problem.\n- Weakness\n  - The benchmarking examples seem pretty small (40x10). It is challenging to ascertain the model's true scalability given that even conventional optimization tools like OR-Tools can achieve optimal solutions in such settings. For a more comprehensive evaluation, the authors should consider larger, practical scheduling problems often found in domains like systems and networking, which involve datasets with thousands or millions of nodes.\n  - Real-world FJSP often involves multiple objectives (e.g., minimizing total flow time, machine idle time) and additional constraints (e.g., machine breakdowns, dynamic job arrivals). The current MDP formulation, reward function design (which explicitly focuses on makespan improvement and visit penalties), and the Nopt2 neighborhood structure are tailored to makespan minimization. The adaptability of MIStar to these more complex and constrained scenarios is not explored.\n  - The paper does not delve into the impact of different initial solution generation methods on MIStar's performance or how robust the framework is to poor initial solutions.\n\nQUESTIONS:\n1. What is the run time of OR-Tools in Table 1?\n2. How do you plan to incorporate additional constraints (e.g., machine breakdowns) into your model?\n3. What is the actual training time for MIStar? Do you need to retrain the model for different benchmarks? How well can the model transfer knowledge from one benchmark to another?\n4. Given that MIStar also requires an initial solution, how does the quality of this initial solution affect MIStar's overall performance? Do you have any additional experimental results to illustrate this impact?\n5. Do you have scalability results for larger benchmarks where even OR-Tools might struggle to generate good solutions within a reasonable timeframe?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}