Title: RoME: Domain-Robust Mixture-of-Experts for MILP Solution Prediction across Domains
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning to Optimize

Machine Learning for Combinatorial Optimization

Mixed-Integer Linear Programming
Abstract: Mixed-Integer Linear Programming (MILP) is a fundamental and powerful framework for modeling complex optimization problems across diverse domains.
Recently, learning-based methods have shown great promise in accelerating MILP solvers by predicting high-quality solutions.
However, most existing approaches are developed and evaluated in single-domain settings, limiting their ability to generalize to unseen problem distributions.
This limitation poses a major obstacle to building scalable and general-purpose learning-augmented solvers.
To address this challenge, we introduce RoME, a domain-Robust Mixture-of-Experts (MoE) framework for predicting MILP solutions across domains.
RoME dynamically routes problem instances to specialized experts based on learned task embeddings.
The model is trained using a two-level distributionally robust optimization strategy: inter-domain to mitigate global shifts across domains, and intra-domain to enhance local robustness by introducing perturbations on task embeddings.
We reveal that cross-domain training not only enhances the model's generalization capability to unseen domains but also improves performancewithin each individual domain by encouraging the model to capture more general intrinsic combinatorial patterns.
Specifically, a single RoME model trained on three domains achieves an average improvement of $67.7\%$ then evaluated on five diverse domains.
We further test the pretrained model on MIPLIB in a zero-shot setting, demonstrating its ability to deliver measurable performance gains on challenging real-world instances where existing learning-based approaches often struggle to generalize.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a machine learning model designed to predict solutions for mixed-integer linear programs (MILPs) to serve as warm starts. Specifically, the model is a variant of the mixture-of-experts framework. The predicted solution is then used to initialize the solver, which subsequently solves the original MILP constrained to a neighborhood around the warm-start solution.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n- The paper provides an interesting framework (RoME) for the the prediction of MIP.\n\n**Weakness**\n\n- The theoretical foundation for the choice of the loss function used in training seems weak (see Questions).\n\n- As the main contribution of the paper is the RoME model, there should also be numerical study for the solution returned by the model, not just a combined effect of the model and solving a constrained MILP (see Questions)\n\nI am willing to re-evaluate my rating if my concerns are answered.\n\nQUESTIONS:\n- Question 1: What is the denominator in equation (2) summation over (x')? My concern is the feasible region of a MIP can have uncountably infinite solution.\n\n- Question 2: The paper seems to claim its method works on general MIP at first, but starting from line 123, it started to assume that MIP only contains binary variables. Can you clarify this point?\n\n- Question 3: What is your intuition for the \"fully factorized solution\" assumption in line 124? Besides the part that it makes it easy to design a loss function, is there any mathematical justification for that? Ignoring the part the definition of p(x|I) is not yet well-defined, it is doubtful that this assumption can be true, as given an instance I, the value of x_1 can change the feasible region of x_2. In addition, using this assumption, are we ignoring the continuous part of the solution? it seems like even if the continuous part give bad objective, we still assign high probability.\n\n- Question 4: The definition of x^(i)_j in line 129 is a bit confusing. Why there is no j in the right-hand side?\n\n- Question 5: In Append A.3 , It is said the the selection of k_0, k_1 and Delta have great influences. Given that in Table 3, different methods use different configuration of these values. Would you be able to provide tables/plots showing the impact of these values when varying?\n\n- Question 6: I do not understand why, in Figure 3, the lines plot do not start second 0, but rather at the 10th second (4 out of 5)? Is it because the solver takes a while to find its first feasible solution or something else?\n\n- Question 7: I am curious about the quality of the binary solution returned by RoME. I think it give us more insight if you can show the objective of the returned solution of RoME or how many of it was feasible (for all instances in your experiments). For the non-pure binary case, we can assign value for the continuous variables by the solving the LP when fixed all binary variables at the predicted solution.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper addresses the problem of domain generalization for MILP solution prediction using machine learning. The author proposes a general-purpose framework that can perform well across different MILP problem distributions,which leverages a Mixture-of-Experts model that dynamically selects experts based on learned task embeddings. Training is guided by a novel two-level DRO objective that mitigates domain shifts and promotes robustness. Results show that a single RoME model trained on multiple domains generalizes better than single-domain models.

STRENGTHS AND WEAKNESSES:
#Strengths
- This paper is well-written and easy to understand
- This paper is technically sound
- The experiments are extensive

#Weakness
- Although the model performs well on average, for some problem, a performance drop can be observed compared to Gurobi. This performance drop should be quantified and reported.
- The effect of the number of experts in the MoE architecture is not explored, which is essential to understand the scalability and architectural sensitivity of the proposed approach.
- The paper includes minor inconsistencies and typos (e.g., “T-SNE” should be “t-SNE” with lowercase “t”).

QUESTIONS:
1. Although the model performs well on average, for some problem, a performance drop can be observed compared to Gurobi. This performance drop should be quantified and reported.
2. The effect of the number of experts in the MoE architecture is not explored, which is essential to understand the scalability and architectural sensitivity of the proposed approach.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes a domain-Robust Mixture-of-Experts (RoME) framework for computing high-quality initial solutions to mixed-integer linear programs (MILPs) that can generalize to different types of MILPs. The framework leverages a two-stage MoE architecture (on both the encoder and decoder parts). In terms of the intra-domain generalization, the authors propose a fuzzy consistency loss that forces the routing results of the original and perturbed task embeddings to be the same, which can potentially make it generalize to unseen types of problems. For the inter-domain generalization, the authors use group-DRO that puts more focus on the hardest domain in the training mini-batch. 

The experiment's results look promising. One RoME model combined with Gurobi can either largely reduce Gurobi's running time or improve the quality of the solutions.

STRENGTHS AND WEAKNESSES:
Pros:

1. The entire RoME framework shows a good generalization ability to unseen types of problems. All the proposed gadgets contribute to the performance.
2. The paper is well written and very easy to follow.

Cons:

1. The reason why the fuzzy consistency loss works seems unclear. In the high-dimensional task embedding space, why can forcing the original and perturbed outputs to be the same help generalize? Do you have more insights into this part?

QUESTIONS:
1. See cons above.
2. In terms of the testing problem sizes, what's the gap between them and the real-world problem size?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper introduces RoME, a domain-robust Mixture-of-Experts framework designed to predict solutions for MILP problems across various domains. Traditional MILP solvers struggle with generalization across different problem distributions, but RoME addresses this by dynamically routing problem instances to specialized experts based on learned task embeddings. \n\n- RoME utilizes a Mixture-of-Experts (MoE) architecture combined with a two-level Distributionally Robust Optimization (DRO) strategy. The MoE consists of multiple expert networks, each specializing in different MILP distributions. For each problem instance, RoME uses a graph neural network (GNN) to extract embeddings and dynamically routes the instance to the most suitable expert. The DRO strategy include inter-domain robustness, achieved by minimizing the worst-case loss across domains, and intra-domain robustness, which stabilizes expert selection through perturbations on task embeddings. This approach allows RoME to adapt to the unique structural characteristics of each instance and improve generalization across domains.\n\n- Experiments shows that RoME significantly improves generalization and performance across multiple domains, including unseen ones, achieving a 67.7% average improvement over strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I think this is a good paper. It demonstrates the efficacy of ML-model-based methods to significantly boost the runtime of  traditional solvers.\n\nStrengths:\n- Experimental results are convincing. RoME demonstrates strong generalization capabilities across multiple domains, including unseen ones, which is a significant improvement over existing methods (e.g. ConPS).\n\n- The use of a two-level DRO strategy is quite neat and, according to the ablations in D.4, it looks like it indeed enhances the model's robustness to distributional shifts, improving its adaptability to new problem instances.\n\n- The MoE architecture allows for specialization among experts, leading to better performance on diverse MILP distributions.\n\nWeaknesses and Limitations:\n\n- The complexity of the model may pose challenges in terms of interpretability and understanding the decision-making process of the experts. Additionally, the model's performance heavily relies on the quality of the task embeddings and the effectiveness of the routing mechanism.\n\n- The need for a large and diverse training dataset to fully leverage the cross-domain capabilities of RoME could be a barrier in some applications.\n\n- The approach may not be easily applicable to domains with limited data availability or where the problem structure is not well-represented in the training set.\n\n- The method's effectiveness on domains with drastically different structures from the training set remains to be fully explored.\n\nQUESTIONS:\nPlease see \"Strengths And Weaknesses\"\n\nLIMITATIONS:\nPlease see \"Strengths And Weaknesses\"", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}