Title: SymILO: A Symmetry-Aware Learning Framework for Integer Linear Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: integer linear programming

symmetry

machine learning

neural networks
Abstract: Integer linear programs (ILPs) are commonly employed to model diverse practical problems such as scheduling and planning. 
Recently, machine learning techniques have been utilized to solve ILPs.  A straightforward idea is to train a model via supervised learning, with an ILP as the input and an optimal solution as the label. An ILP is symmetric if its variables can be permuted without changing the problem structure, resulting in numerous equivalent and optimal solutions. Randomly selecting an optimal solution as the label can introduce variability in the training data, which may hinder the model from learning stable patterns. In this work, we incorporate the intrinsic symmetry of ILPs and propose a novel training framework called SymILO. Specifically, we modify the learning task by introducing solution permutation along with neural network weights as learnable parameters and then design an alternating algorithm to jointly optimize the loss function.
We conduct extensive experiments on ILPs involving different symmetries and the computational results demonstrate that our symmetry-aware approach significantly outperforms three existing methods----achieving $50.3\\%$, $66.5\\%$, and $45.4\\%$ average improvements, respectively.

SymILO: A Symmetry-Aware Learning Framework for Integer Linear Optimization

Abstract

Integer linear programs (ILPs) are commonly employed to model diverse practical problems such as scheduling and planning. Recently, machine learning techniques have been utilized to solve ILPs. A straightforward idea is to train a model via supervised learning, with an ILP as the input and an optimal solution as the label. An ILP is symmetric if its variables can be permuted without changing the problem structure, resulting in numerous equivalent and optimal solutions. Randomly selecting an optimal solution as the label can introduce variability in the training data, which may hinder the model from learning stable patterns. In this work, we incorporate the intrinsic symmetry of ILPs and propose a novel training framework called SymILO. Specifically, we modify the learning task by introducing solution permutation along with neural network weights as learnable parameters and then design an alternating algorithm to jointly optimize the loss function. We conduct extensive experiments on ILPs involving different symmetries and the computational results demonstrate that our symmetry-aware approach significantly outperforms three existing methods--achieving 50 . 3% , 66 . 5% , and 45 . 4% average improvements, respectively.

1 Introduction

Integer linear programs (ILPs) are optimization problems with integer variables and a linear objective, and have a wide range of practical uses in various fields, such as production planning (Pochet & Wolsey, 2006; Chen, 2010), resource allocation (Liu & Fan, 2018; Watson & Woodruff, 2011), and transportation management (Luathep et al., 2011; Schöbel, 2001). An important property that often arises in ILPs is symmetry (Margot, 2003), which refers to a situation where permuting variables does not change the structure of an ILP.

Recently, there emerges many approaches equipping machine learning methods, supervised learning in particular, to help efficient solution identification for ILPs (Zhang et al., 2023). Among these approaches, an important category derived from the idea of predicting the optimal solution has demonstrated significant improvements (Han et al., 2023; Ding et al., 2020; Khalil et al., 2022; Nair et al., 2020). In this paper, we consider a classic supervised learning task that aims to train an ML model to predict an optimal solution for an ILP. Specifically, given a training dataset D = { ( s i , y i ) } N i =1 with y i denoting an optimal solution to instance s i , we hope to train a neural network

model f θ ( · ) to approximate the mapping from ILP instances to their optimal solutions, via minimizing the empirical risk defined on f θ ( s i ) and y i .

However, for an ILP s i with symmetry, there exist multiple optimal solutions including y i and its symmetric counterparts, any of which has an equal probability of being returned as a label. Training neural networks without taking symmetry into account is basically learning a model supervised by random outputs, leading to prediction models of inferior performance.

To address this issue, we propose to leverage the symmetry of ILPs to improve the model performance of predicting an optimal solution. Specifically, given input s i , we define a new empirical risk using f θ ( s i ) and π i ( y i ) , where π i ( · ) denotes the operation of permuting elements in y i into its symmetric counterpart. Along with ML model parameters, the permutation operators will also be optimized during training. To achieve this, we further develop a computationally affordable algorithm that alternates between optimization of model parameters and optimization of permutation operation. The distinct contributions of our work can be summarized as follows.

We propose a symmetry-aware framework (called SymILO) that introduces permutation operators as extra optimization variables to the classic training procedure.

We devise an alternating algorithm to solve the newly proposed problem, with a specific focus on updating the permutation operator for different symmetries.

We conduct comprehensive numerical studies on four typical benchmark datasets involving symmetries, and the results show that our proposed approach significantly outperforms existing methods.

2 Related works

Previous works on identifying high-quality solutions to ILPs via machine learning techniques mainly focus on reducing problem sizes. For example, Ding et al. (2020) propose to identify and predict a subset of decision variables that stay unchanged within the collected solutions. Li & Wu (2022) formulate MILPs as Markov decision processes and learn to reduce problem sizes via early-fixing.

It is noteworthy that the emergence of GNNs has had a significant impact on solving ILPs. Gasse et al. (2019) are the first to propose a bipartite-graph representation of ILPs and pass it to GNNs. Nair et al. (2020) adopt the same representation scheme and train GNNs to predict the conditional distribution of solutions, from which they further sample solutions. Rather than directly fixing variables, Han et al. (2023) conduct search algorithms in a neighborhood centered around an initial point generated from the predicted distribution. Other works based on GNNs (Sonnerat et al., 2022; Lin et al., 2019; Khalil et al., 2022; Wu et al., 2021) also illustrate great potential in improving the solving efficiency.

Limitations of the existing GNN-based approaches are also noticed. Nair et al. (2020); Han et al. (2023) try to address the multiple solution problem by learning the conditional distribution. Chen et al. (2022) introduce random features into the bipartite graph representation to differentiate variable nodes involving symmetries.

However, none of the existing learning-based approaches explicitly leverage the inherent symmetries in ILPs to achieve improvements. In contrast, works from mathematical optimization perspectives suggest that symmetry-handling algorithms exhibit great abilities in solving symmetry-involving ILPs (Pfetsch & Rehn, 2019). To name a few, such algorithms include orbital fixing (Ostrowski et al., 2011), tree pruning (Margot, 2002), and lexicographical ordering (Kaibel & Pfetsch, 2008).

3 Background and preliminaries

3.1 ILPs

An integer linear program (ILP) has a formulation as follows:

where x ∈ Z n are integer decision variables, and c ∈ R n , A ∈ R m × n , b ∈ R m are given coefficients.

3.2 Symmetry Group

Symmetry of ILPs is typically represented by groups. We start with some basic notations and most of which follow Margot (2009). Denoting the index set by I n = { 1 , 2 , . . . , n } , a permutation on I n is a bijective (one-to-one and onto) mapping π : I n - → I n . For example, an identity permutation maps the index set to itself as { π ( i ) = i } n i =1 and a cyclic permutation has rotational mapping rules { π ( i ) = i +1 } n -1 i =1 and π ( n ) = 1 . Schematic diagrams of these two permutations and other ones are shown in Figure 1. For brevity, we abuse the notation π a little bit and denote the permutation acting on a vector y ∈ R n by rearranging its coordinates, namely π ( y ) = [ y π (1) , y π (2) , . . . , y π ( n ) ] ⊤ . Let Q be the set of all feasible solutions of (1) and S n the set of all permutations on I n . Note that S n is referred to as the symmetric group , which should not be confused with the symmetry group discussed as follows.

Definition 3.1. A symmetry group of (1) is defined as the set of all permutations π that map Q onto itself, such that each feasible solution is mapped to another feasible solution with the same objective value, i.e.,

Next, we will delve into three commonly encountered symmetries, accompanied by typical example problems. Since not all variables in an ILP involve symmetry, we use q ≤ n to indicate the size of the symmetry group.

Symmetric group The symmetric group , denoted by S q , is the group that consists of all permutations ( q ! in total) on I q . Problems with this kind of symmetry include bin packing (Johnson, 1974) and optimal job scheduling Graham et al. (1979), etc. An example is illustrated in Appendix B.0.1.

Cyclic and dihedral groups As its name suggests, cyclic symmetry allows elements to be

𝑛

…

2

1

(c) Cyclic permutation

𝑛 - 1

…

2

1

𝑛

(d) Reflective permutation

(b) Swapping permutation

(a) Identity permutation

permuted to their right neighbors, cycling the right-most variables back to the left, e.g., a cyclic (or rotational) permutation ρ in Figure 1 (c). The elements of a cyclic group C q are powers of ρ , and | C q | = q . Problems with cyclic group often have characteristics of rotations or cycles, e.g., periodic event scheduling problem (Serafini & Ukovich, 1989).

Compared to the cyclic group, a dihedral group (denoted as D q ) additionally includes reflective permutations, which is illustrated in Figure 1 (d). Consequently, D q comprises a total of 2 q distinct permutations. A typical problem with such symmetry is the circular (or modular) golomb ruler problem (see Appendix B.0.2).

3.3 Classic supervised learning for solution prediction

A classic solution prediction task based on supervised learning is formulated as follows. Let S be the space of ILP instances and Y be the label (i.e., optimal solution) space. A model function f θ : S - →Y parameterized by θ ∈ Θ is used to learn a mapping from instances to optimal solutions. Let P ( S, Y ) be a distribution over S × Y . The performance of the model function is measured by a criterion called true risk : R ( f θ ) := E P ( S,Y ) [ ℓ ( f θ ( s ) , y )] , where ℓ : Y × Y - → R + is a given loss function, e.g., mean squared error or cross-entropy loss. An intuitive way to improve the model performance is to minimize the true risk . However, one cannot access all data from distribution P ( S, Y ) , which makes it impossible to calculate the true risk. Practically, one can obtain a set of (instance, solution) pairs called training data D = { ( s i , y i ) } N i =1 ⊆ ( S × Y ) N sampled from P ( S, Y ) , based on which define the empirical risk as

By minimizing the empirical risk , i.e., min θ ∈ Θ r , one aims to approximate the minimization of the true risk , under the assumption that the training data is a representative sample of the overall data distribution.

4 Methodology

4.1 Reformulation of the learning task

In Section 3.3, we introduce a classic supervised learning task for general ILPs, which aims at learning a mapping f θ from instances to optimal solutions. In this task, a dataset D = { ( s i , y i ) } is given, and the mapping f θ is learned by minimizing (3) with θ as decisions. However, for ILPs with symmetry, an ILP instance has multiple solutions (let Y i be the set of optimal solutions of i -th instance). As a consequence, the labels in this task have multiple choices, thus datasets choosing different optimal solutions as labels {D ′ = { ( s i , y ′ i ) } N i =1 , ∀ y ′ i ∈ Y i } are all valid for the learning task. Empirically, we observe that different D ′ can lead to distinct performance, which motivates us to consider the selection of labels for ILPs with symmetry.

We reformulate the learning task as follows. Firstly, we augment dataset D to dataset D s = { ( s i , y i , G i ) } , where G i is the symmetry group of i -th instance and π i ∈ G i . Secondly, we define the symmetry-aware empirical risk as

Then, the mapping f θ is learned by minimizing the symmetry-aware risk as min θ,π r s (both θ and π as decisions). In contrast to the original task, the symmetry-aware task uses symmetry information by introducing extra decisions { π i } N i =1 , so as to dynamically selecting proper optimal solutions as labels. There are important differences between the symmetry-aware empirical risk and the classic one:

Proposition 4.1. Let r ∗ and r ∗ s be the global minimal values of min θ r and min θ,π r s , respectively. Then, the following claims hold:

̸

Claim (i) always holds since min θ r is a special case of min θ,π r s when π 1 , . . . , π N are all identity permutations . Claim (ii) shows a significant advantage of r s compared to r . A non-rigorous proof is available in Appendix A.1.

4.2 An alternating minimization algorithm

The minimization of (4) is challenging due to the discrete nature of π . Motivated by the well-known block coordinate minimization algorithms (Mangasarian, 1994), we update θ and π alternately, i.e.,

Such an alternating mechanism divides the minimization of (4) into two sub-problems: a discrete optimization in (5) over sets { G i } N i =1 and a classic empirical risk minimization in (6). Repeatedly solving (6) to optimal is unrealistic, thus it is more practical to update θ by several gradient steps instead.

The sub-problem in (5) is further specified as shown in Section 4.2.1, according to the symmetry structures in the ILP instances.

We summarize the proposed alternating minimization algorithm in Algorithm 1. In the main loop, { π i } N i =1 are updated first (line 5), after which an inner loop (lines 6-10) is operated to update θ through a gradient-based method GD , e.g., Adam (Kingma & Ba, 2014). These two updates alternate until a preset maximum number of epochs K is reached. We finally note that Algorithm 1 can be easily adapted to a mini-batch version, in which the data can be randomly sampled from D s .

ILPs

𝑠 1

, 𝑠

2 , … , 𝑠𝑛

Bipartite

representation

Networks

𝑓 𝜃

Prediction

Symmetry-aware empirical risk

𝑚𝑖𝑛

1

𝑁 ෍

𝑖=1

𝑁

𝓁 𝑓 𝜃

𝑠𝑖

, 𝜋

𝑖

𝑦𝑖

Solutions

𝑦1, 𝑦

2 , … , 𝑦 𝑛

Symmetries

𝐺1, 𝐺

2 , … , 𝐺𝑛

Update

𝜋

Update

𝜃

Alternating updates

Data

Training

Model

Downstream

Tasks

Fix and optimize

Local Branching

…

…

…

𝒗𝟏

𝒗𝒏

𝒄𝟏

𝒄𝒎

𝑨𝟏𝟏

𝑨𝒎𝟏

𝑨𝟏𝒏

𝑨𝒎𝒏

0.9

0.1

…

0.2

4.2.1 Optimization over symmetry groups in (5)

In this section, we investigate the concrete formulations of the sub-problem in (5) for the symmetry groups mentioned in Section 3.2 (symmetric group, cyclic and dihedral groups), and devise algorithms to solve them.

Cyclic and dihedral groups The cardinality of a cyclic group C q is q , and it is 2 q for a dihedral group D q . For symmetry groups with such reasonably small size, a straightforward and effective way to solve (5) is to evaluate all possible permutations

Algorithm 1 Alternating optimization

1: Input: Dataset D s = { ( s i , y i , G i ) } N i =1 , K, T

2: Output:

θ

K

3: Initialize ML model parameter θ 1 ;

4: for k ← 1 to K - 1 do

5: { π k +1 } N ← arg min π ∈ G r s ( f k , { π i } N ; D s ) ;

i i =1 i i θ i =1 ˜ 1 k

6: θ ← θ ;

7: for t ← 1 to T - 1 do 8: ˜ θ t +1 ← GD( ˜ θ t , ∇ ˜ θ t r s ( ˜ θ t , { π k +1 i } N i =1 ; D s )) ;

9: end for

10: θ k +1 ← ˜ θ T ;

11: end for

and select the one that yields the minimum r s .

Symmetric group The cardinality of a symmetric group is factorially large, | S q | = q ! , so it is impractical to traverse all permutations. Since π 1 , . . . , π N are not coupled, we can separate them and solve the N sub-problems individually:

Without loss of generality, consider an ILP whose variables have a matrix form (e.g., see Appendix B.0.1), denoted by X ∈ Z p × q ( p · q < n ), and a symmetric group S q acting on its column coordinates. In this case, (7) is equivalent to solve the following binary linear program (BLP),

where P is a permutation matrix, ˆ X is the matrix form of f θ ( s i ) , and 1 is an all-one vector. We relax P to take continuous values between 0 and 1, and get a linear program (LP),

According to Proposition 4.2, one can solve (9), to get the optimal permutations for the original problem in (8). It can be done quite efficiently with the aid of off-the-shelf LP solvers, such as Gurobi Optimization, LLC (2023), CPLEX IBM (2020), etc.

Proposition 4.2. When ℓ is the squared error or binary cross-entropy loss, the optimal solution to (9) is also an optimal solution to (8). (See the proof in A.2.)

4.3 An overview of the SymILO framework

In this section, we summarize a novel learning framework (SymILO) that utilizes symmetry for solving ILPs. An overview is depicted in Figure 2, which consists of two parts: the upper row connected by green arrows delineates a graph neural network (GNN)-based workflow, and the lower row connected by red arrows outlines the training process.

For the GNN-based workflow, an ILP is first converted to a bipartite graph (see appendix C for details), which is then fed to a GNN model f θ (see appendix D for details), producing a predicted solution. Notably, the predicted solution is finally used in downstream tasks for refinement. Due to the complexity of solving ILPs, existing methods, such as Nair et al. (2020); Ding et al. (2020); Khalil et al. (2022); Han et al. (2023), often include a post-processing module taking the predicted solution as an initial point to identify higher-quality solutions. Our approach follows this routine and integrates certain downstream techniques. Section 5.1 specifies three downstream tasks.

For the training process, the data used to minimize the symmetry-aware empirical risk r s include the collected solution y i and the symmetry group G i of each instance s i . Both parameters θ of the GNN model and permutations { π i } N i =1 of each solution are optimized via an alternating algorithm mentioned in Algorithm 1. Given a trained model f θ K , the prediction f θ K ( s ′ ) for an unseen instance s ′ is used to guide the downstream tasks in identifying feasible solutions. Note that { π i } N i =1 are utilized only in the training phase but not in the inference phase.

5 Experimental settings

In this section, the experimental settings are presented. The corresponding source code is available at https://github.com/NetSysOpt/SymILO.

5.1 Downstream tasks and baselines

In our experiments, we pass the predictions of GNN models to three downstream tasks, namely fix and optimize, local branching, and node selection, to identify feasible solutions. For each downstream task, we choose one existing method as a baseline. The downstream tasks and their corresponding baselines (in parentheses) are shown below.

Fix and optimize (ND): 'Fix and optimize' refers to a strategy where one first 'fix' or set some variables to specific values and then 'optimize' the remaining variables to find better solutions. The baseline we choose is 'Neural Diving' ( ND ) proposed by Nair et al. (2020), a technique using a graph neural network to generate partial assignments for ILPs, which creates smaller sub-ILPs with the unassigned variables.

Local branching (PS): Local branching is a heuristic method that constructs a linear constraint based on a given initial solution to the original ILP instance. This constraint restrains the search space in a region around the initial solution. It can help guide the optimization process toward better solutions while balancing computational efficiency. Approaches based on this idea include Ding et al. (2020); Han et al. (2023); Chen et al. (2023) and we select the 'predict-and-search' ( PS ) framework proposed by Han et al. (2023) as a baseline.

Node selection (MIP-GNN): In branch and bound algorithms, node selection is a process of choosing the proper nodes to explore next. Effective node selection is crucial for the algorithm's success in solving optimization problems. 'MIP-GNN' ( MIP-GNN ) proposed by Khalil et al. (2022) uses GNN prediction to guide node selection and warm-starting, and is selected as another baseline.

5.2 Benchmark datasets

We evaluate the proposed framework on four ILP benchmarks with certain symmetry, which consists of (i) two problems with symmetric groups: the item placement problem (IP) and the steel mill slab problem (SMSP), (ii) the periodic event scheduling problem (PESP) with cyclic group, and (iii) a modified variant of PESP (PESPD) which has a dihedral group.

The first benchmark IP is from the NeurIPS ML4CO 2021 competition (Gasse et al., 2022). We use their source code to randomly generate instances with binary variables ranging from 208 to 1050 . Each instance has a symmetric group S 4 ∼ S 10 . We use 500 instances for our experiments, taking 400 as the training set and the remaining 100 for testing. The SMSP benchmark is from Schaus et al. (2011), and contains 380 problem instances. We randomly select 304 of them as training data and take the others as testing data. The instances of this benchmark have 22k ∼ 24k binary variables and nearly 10k constraints, with each of them having a symmetric group S 111 . The last two benchmarks are from

PESPlib Goerigk (2012), a collection of periodic timetabling problems inspired by real-world railway timetabling settings. Since PESPlib only provides a few instances, which are not sufficient to support neural network training, we randomly perturb the weights of the provided instances to generate more data (see Appendix G.3.1 for details). We respectively generate 500 instances for PESP and PESPD, taking 400 of them as training sets and 100 as testing sets. The symmetry groups of these two datasets are cyclic groups C 5 ∼ C 15 and dihedral groups D 5 ∼ D 15 , respectively. For all training sets, 30% instances are used for validation. The average numbers of variables and constraints, as well as the symmetry groups of each benchmark problem, are summarized in Appendix F.1. Besides, more details about their ILP formulations and corresponding symmetries are supplemented in Appendix G.

These benchmarks only include problem instances. We collect the corresponding solutions using an ILP solver CPLEX (IBM, 2020). However, solving ILP instances even with moderate sizes to optimal is extremely expensive. It is more practical to use high-quality solutions as the labels. Therefore, we run single-thread CPLEX for a time limit of 3,600 seconds and record the best solutions.

5.3 Training settings

All models are trained with a batch size 16 for 50 epochs. The Adam optimizer with a learning rate of 0 . 001 is used, and other hyperparameters of the optimizer are set to their default values. The model with the smallest loss on the validation set is used for subsequent evaluations. Other training settings, such as the loss function and neural architectures, follow the configurations in Han et al. (2023). More details about the hyper-parameter tuning for the downstream tasks and software resources are shown in Section E.

5.4 Evaluation metrics

To compare the prediction performance of the model trained on r and r s , we define the Topm % error for evaluation. In addition, another criterion relative primal gap is used to evaluate the final performance in identifying feasible solutions in different downstream tasks.

Topm % error: We use the distance between a rounded prediction and its nearest equivalent solution as the error. Specifically, given a prediction ˆ y and its label y , we define the equivalent solution closest to ˆ y as ˜ y = π ′ ( y ) , where π ′ = arg min π ∥ ˆ y -π ( y ) ∥ . Then, the Topm %error is defined as

where M is the index set of m % variables with largest values of | Round (ˆ y j ) -ˆ y j | . This error measures the minimum distance between the prediction and all solutions equivalent to the label. Compared to naive use of the distance ∑ i ∈ M | Round (ˆ y i ) -y i | , (10) can more accurately represent how close a prediction is to a feasible solution. Since for the naive distance, when Round (ˆ y ) equals any equivalent solution π ( y ) = y , the distance is greater than 0, while that of (10) is 0.

̸

Relative primal gap: We also feed the outputs of the models trained through r s to the downstream tasks mentioned in Section 5.1 to evaluate the quality of the predictions. All the three downstream approaches incorporate ILP solvers to search for solutions. We run these ILP solvers on a single thread for a maximum of 800 seconds. Since all the problems used in the experiments are NP-hard, identifying optimality is highly time-consuming. Thus the metric used in our experiments is relative primal gap

which measures the relative gap in the objective value of a feasible solution ˜ y to that of the best-known solution y ∗ , and ϵ is a small positive value to avoid the numerical issue. Additionally, let γ r and γ r s respectively be the primal gaps of models trained through r and r s , then an improvement gain of our approach is calculated as ( γ r -γ r s ) /γ r .

6 Numerical results

In this section, we present the comparison results on empirical risk r and symmetry-aware one r s . In addition, primal gaps of SymILO and baselines on three downstream tasks are reported.

6.1 On empirical risks and Topm % error

We denote training and test risks by r tr ( · ) = r ( · ; D tr ) and r te = r ( · ; D te ) , respectively, and similarly use r tr s and r te s for symmetry-aware risk. Let f ( k ) and f ( k ) s be the best classic model and symmetryaware model obtained at k -th epoch by training with r tr and r tr s , respectively. We plot both the training and test risks versus the number of epochs in Figure 3. As predicted in Proposition 4.1, when algorithms converge, the classic empirical risk r tr is always greater than symmetry-aware risk r tr s .

0

50

(a) IP

0.25

0.31

0.38

0.44

empirical risk

0

50

(b) SMSP

0.01

0.05

0.09

0.14

empirical risk

0

50

(c) PESP

0.33

1.22

2.11

3.00

empirical risk

0

50

(d) PESPD

1.33

1.89

2.44

3.00

empirical risk

As shown in Table 1, the symmetry-aware model f ( K ) s always predicts smaller Topm % errors in (10) compared to the classic model f ( K ) , demonstrating the usefulness of proposed empirical symmetry-aware risk in predicting solutions correctly.

Moreover, the time costs of minimizing different empirical risks r and r s for a mini batch are shown in Table 2. Here, t denotes the average time of solving the permutation decisions per instance. The reported times for r s include the optimization time t . The table illustrates that the alternate training strategy does not significantly increase the training duration, and the optimization step over π is executed efficiently.

6.2 Downstream results

The relative primal gaps of different downstream tasks at different solving time are shown in Figure 4, and the final values at 800 seconds are listed in Table 3. As Figure 4 shows, our proposed empirical risk significantly improves the performance of different downstream tasks over the primal gap in 800 seconds.

Note that the node selection task exhibits modest performance in comparison to other tasks; a possible reason is that it requires runtime interaction to call the callback functions provided by the CPLEX

10

200

400

600

800

Solving time (s)

(a) IP

0.1

0.15

0.2

0.25

0.3

0.35

Primal gap

10

200

400

600

800

Solving time (s)

(b) SMSP

0.2

0.4

0.6

0.8

1

1.2

Primal gap

10

200

400

600

800

Solving time (s)

(c) PESP

0

1

2

3

4

Primal gap

10

200

400

600

800

Solving time (s)

(d) PESPD

0

2

4

6

8

10

Primal gap

Python APIs, which can slow down the whole solving process. However, such a flaw does not affect the demonstration of the effectiveness of our proposed method.

For the primal gap at 800 seconds shown in Table 3, the models trained through r s significantly improve all downstream tasks. The performance gain of the model trained through r s is calculated by computing the relative gaps between our approach's gap improvements and that of the baselines. Average gains over the three downstream tasks are 50.3%, 66.5% and 45.4%, respectively. The overall results demonstrate the effectiveness of the proposed empirical risk r s . We also provide the corresponding p-values for the significance of improvements in Appendix F.2.

7 Limitations and conclusions

In conclusion, we propose SymILO, a novel symmetry-aware learning framework for enhancing the prediction of solutions for integer linear programs by incorporating symmetry into the training process. Our approach shows significant performance improvements over symmetry-agnostic methods on benchmark datasets. Despite the significant advancements presented in our symmetry-aware learning framework, SymILO, several limitations must be acknowledged. Firstly, while we provide realizations for three commonly encountered symmetry groups-symmetric, cyclic, and dihedral-the framework requires specific formulations for optimizing permutations, which limits its immediate applicability to other symmetry groups not discussed in this work. Secondly, for large-scale problem instances with extensive and complex symmetry groups, the sub-problems involved in optimizing permutations can significantly slow down the training process. Enhancing the computational efficiency of our alternating optimization algorithm for these cases remains a challenge and an area for future research.

Acknowledgments

This work was supported by the National Key R&D Program of China under grant 2022YFA1003900. Akang Wang also acknowledges support from the National Natural Science Foundation of China (Grant No. 12301416), the Shenzhen Science and Technology Program (Grant No. RCBS20221008093309021), the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515010306) and the Longgang District Special Funds for Science and Technology Innovation (LGKCSDPT2023002). Ruoyu Sun also acknowledges support from the Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project (No. HZQSWS-KCCYB2024016), the University Development Fund (UDF01001491) at the Chinese University of Hong Kong, Shenzhen, the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001), and the Guangdong Major Project of Basic and Applied Basic Research (2023B0303000001). Tsung-Hui Chang acknowledges support from the Shenzhen Science and Technology Program (Grant No. ZDSYS20230626091302006).

References

Chen, Y., Gao, W., Ge, D., and Ye, Y. Pre-trained mixed integer optimization through multi-variable cardinality branching. arXiv preprint arXiv:2305.12352 , 2023.

Chen, Z., Liu, J., Wang, X., Lu, J., and Yin, W. On representing mixed-integer linear programs by graph neural networks, 2022.

Chen, Z.-L. Integrated production and outbound distribution scheduling: review and extensions. Operations research , 58(1):130-148, 2010.

Dantzig, G. B. Linear inequalities and related systems . Number 38. Princeton university press, 1956.

Ding, J.-Y., Zhang, C., Shen, L., Li, S., Wang, B., Xu, Y., and Song, L. Accelerating primal solution findings for mixed integer programs based on solution prediction. In Proceedings of the aaai conference on artificial intelligence , volume 34, pp. 1452-1459, 2020.

Gargani, A. and Refalo, P. An efficient model and strategy for the steel mill slab design problem. In International Conference on Principles and Practice of Constraint Programming , pp. 77-89. Springer, 2007.

Gasse, M., Chetelat, D., Ferroni, N., Charlin, L., and Lodi, A. Exact combinatorial optimization with graph convolutional neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/pap er/2019/file/d14c2267d848abeb81fd590f371d39bd-Paper.pdf .

Gasse, M., Bowly, S., Cappart, Q., Charfreitag, J., Charlin, L., Chételat, D., Chmiela, A., Dumouchelle, J., Gleixner, A., Kazachkov, A. M., et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NeurIPS 2021 Competitions and Demonstrations Track , pp. 220-231. PMLR, 2022.

Goerigk, M. Pesplib-a benchmark library for periodic event scheduling, 2012.

Graham, R. L., Lawler, E. L., Lenstra, J. K., and Kan, A. R. Optimization and approximation in deterministic sequencing and scheduling: a survey. In Annals of discrete mathematics , volume 5, pp. 287-326. Elsevier, 1979.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www.guro bi.com .

Han, Q., Yang, L., Chen, Q., Zhou, X., Zhang, D., Wang, A., Sun, R., and Luo, X. A gnn-guided predict-and-search framework for mixed-integer linear programming. In The Eleventh International Conference on Learning Representations , 2023.

IBM, I. I. C. O. V20. 1: User's manual for cplex. IBM Corp , 2020.

Johnson, D. S. Fast algorithms for bin packing. Journal of Computer and System Sciences , 8(3): 272-314, 1974.

Kaibel, V. and Pfetsch, M. Packing and partitioning orbitopes. Mathematical Programming , 114(1): 1-36, 2008.

Khalil, E. B., Morris, C., and Lodi, A. Mip-gnn: A data-driven framework for guiding combinatorial solvers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 1021910227, 2022.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Li, L. and Wu, B. Learning to accelerate approximate methods for solving integer programming via early fixing. arXiv preprint arXiv:2207.02087 , 2022.

Lin, X., Hou, Z. J., Ren, H., and Pan, F. Approximate mixed-integer programming solution with machine learning technique and linear programming relaxation. In 2019 3rd International Conference on Smart Grid and Smart Cities (ICSGSC) , pp. 101-107, 2019. doi: 10.1109/ICSGSC.2019.00-11.

Liu, L. and Fan, Q. Resource allocation optimization based on mixed integer linear programming in the multi-cloudlet environment. IEEE Access , 6:24533-24542, 2018.

Luathep, P., Sumalee, A., Lam, W. H., Li, Z.-C., and Lo, H. K. Global optimization method for mixed transportation network design problem: a mixed-integer linear programming approach. Transportation Research Part B: Methodological , 45(5):808-827, 2011.

Mangasarian, O. L. Nonlinear programming . SIAM, 1994.

Margot, F. Pruning by isomorphism in branch-and-cut. Mathematical Programming , 94:71-90, 2002.

Margot, F. Exploiting orbits in symmetric ilp. Mathematical Programming , 98:3-21, 2003.

Margot, F. Symmetry in integer linear programming. 50 Years of Integer Programming 1958-2008: From the Early Years to the State-of-the-Art , pp. 647-686, 2009.

Nair, V., Bartunov, S., Gimeno, F., Von Glehn, I., Lichocki, P., Lobov, I., O'Donoghue, B., Sonnerat, N., Tjandraatmadja, C., Wang, P., et al. Solving mixed integer programs using neural networks. arXiv preprint arXiv:2012.13349 , 2020.

Ostrowski, J., Linderoth, J., Rossi, F., and Smriglio, S. Orbital branching. Mathematical Programming , 126:147-178, 2011.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/9015-pytorch-a n-imperative-style-high-performance-deep-learning-library.pdf .

Pfetsch, M. E. and Rehn, T. A computational comparison of symmetry handling methods for mixed integer programs. Mathematical Programming Computation , 11:37-93, 2019.

Pochet, Y. and Wolsey, L. A. Production planning by mixed integer programming , volume 149. Springer, 2006.

Schaus, P., Van Hentenryck, P., Monette, J.-N., Coffrin, C., Michel, L., and Deville, Y. Solving steel mill slab problems with constraint-based techniques: Cp, lns, and cbls. Constraints , 16:125-147, 2011.

Schöbel, A. A model for the delay management problem based on mixed-integer-programming. Electronic notes in theoretical computer science , 50(1):1-10, 2001.

Serafini, P. and Ukovich, W. A mathematical model for periodic scheduling problems. SIAM Journal on Discrete Mathematics , 2(4):550-581, 1989.

Sonnerat, N., Wang, P., Ktena, I., Bartunov, S., and Nair, V. Learning a large neighborhood search algorithm for mixed integer programs, 2022.

Watson, J.-P. and Woodruff, D. L. Progressive hedging innovations for a class of stochastic mixedinteger resource allocation problems. Computational Management Science , 8(4):355-370, 2011.

Wu, Y., Song, W., Cao, Z., and Zhang, J. Learning large neighborhood search policy for integer programming. Advances in Neural Information Processing Systems , 34:30075-30087, 2021.

Zhang, J., Liu, C., Li, X., Zhen, H.-L., Yuan, M., Li, Y., and Yan, J. A survey for solving mixed integer programming via machine learning. Neurocomputing , 519:205-217, 2023.

A Theoretical proofs

A.1 Proposition 4.1

Non-rigorous Proof: Consider the case where s 1 = s 2 , y 1 = y 2 , and ℓ is the mean squared error, and let ˆ y = f θ ( s 1 ) = f θ ( s 2 ) . Then,

̸

While for the symmetry-aware risk, since s 1 and s 2 corresponds to identical instances, there must exist permutations π ′ 1 and π ′ 2 , such that π ′ 1 ( y 1 ) = π ′ 2 ( y 2 ) . Consequently,

A.2 Proposition 4.2

Proof : When the loss function ℓ ( · , · ) is the squared error (SE) or the binary cross-entropy loss (BCE),

equations (13) to (14) hold for permutations of a matrix's rows and columns don not change its Frobenius norm, i.e., tr( P ⊤ X ⊤ XP ) = ∥ XP ∥ 2 F = ∥ X ∥ 2 F = tr( X ⊤ X ) . (14) and (15) show that these two loss functions are linear w.r.t P , with which (8) becomes a linear assignment problem Kuhn (1955). It is easy to verify that the constriant matrix of a linear assignment problem is totally unimodular--it satisfies the four conditions of Hoffman and Gale (see Page 252 in Dantzig (1956)), thus an optimal solution of the relaxed problem (9) must be integral as well, namely an optimal solution to problem (8).

B ILP examples with different symmetry group

Example B.0.1 . Consider a bin packing problem, in which there are three items I = { 1 , 2 , 3 } with sizes { a 1 = 1 , a 2 = 2 , a 3 = 3 } and three identical bins J = { 1 , 2 , 3 } with capacity B = 3 . Items are packed into bins, and it is required to use a minimum number of bins without exceeding the capacity. The specific formulation is as follows:

where y j = 1 denotes j -th bin is used and x ij = 1 denotes i -th item is placed in j -th bin.

3

1

2

3

1

2

3

1

2

…

Since all bins are identical, arbitrarily swapping them does not change the feasibility and the objective value, e.g., the different assignments shown in Figure 5 are all equivalent. Specifically, assume

is an optimal solution to the problem (16), then

are all equivalent solutions to X . Formally, this problem has a symmetric group S 3 = { (1 , 2 , 3) , (1 , 3 , 2) , (2 , 1 , 3) , (2 , 3 , 1) , (3 , 1 , 2) , (3 , 2 , 1) } w.r.t. its bin numbers J .

Example B.0.2 . Given a circle with circumference 8 , place 3 ticks at integer points around the circle such that all distances between inter-ticks along the circumference are distinct. The formulation of this problem is as follows:

x

min

1

,x

2

,x

3

̸

̸

̸

where S = { (1 , 2) , (1 , 3) , (2 , 3) } , x 1 , x 2 , x 3 ∈ { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 } denote the positions of each tick, d ij are distances between ticks i and j with auxiliary variables y ij . The constraints in this formulation are nonlinear, we linearize them by big-M methods. Equalities (17a) can be linearized by introducing auxiliary variables a ij ∈ { 0 , 1 } , ∀ ( i, j ) ∈ S as

when x i ≥ x j , a ij = 0 , otherwise a ij = 1 . Similarly, equalities (17b) are equivalent to

where m ij ∈ { 0 , 1 } , ∀ ( i, j ) ∈ S are auxiliary variables, with m ij = 0 when y ij ≤ 8 -y ij .

The not-equal constraints (17 c ) can be linearized by

̸

where K = { (1 , 2 , 1 , 3) , (1 , 2 , 2 , 3) , (1 , 3 , 2 , 3) } . By introducing auxiliary variables t ijkℓ ∈ { 0 , 1 } , ∀ ( i, j, k, ℓ ) ∈ K , we have d ij ≥ d kℓ + 1 if t ijkℓ = 1 , otherwise d ij ≤ d kℓ -1 , i.e., d ij = d kℓ .

Assume { x 1 = ¯ x 1 , x 2 = ¯ x 2 , x 3 = ¯ x 3 } is a feasible solution of this problem and let [ · ] T denote the mod -T operation, then it's easy to verify that { x i = [¯ x i + b ] 8 } 3 i =1 (rotation) and its reverse { x i = [(8 -¯ x i ) + b ] 8 } 3 i =1 , ∀ b ∈ Z (reflection of the rotation) are both equivalent feasible solutions. It is more intuitive to see the illustration in Figure 6, rotation and reflection acting on the ticks do not change their corresponding distances.

When representing x 1 , x 2 , x 3 by binary variables z ip ∈ { 0 , 1 } , ∀ i ∈ 1 , 2 , 3 , ∀ p ∈ { 1 , . . . , 8 } :

0

the modulo symmetry leads to a dihedral group D 8 along the p dimension of z ip . Specifically, let Z be a feasible solution with its ( i, p ) -th entry as the value of z ip , then any permutation π ∈ D 8 acting on the columns of Z yields another equivalent solution [ Z : π (1) , . . . , Z : π (8) ] .

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

rotation

reflection

C Bipartite graph representation for MILP

Gasse et al. (2019) proposed to represent a MILP (also works for ILP) by a bipartite graph G = ( V, C, E ) with two disjoint sets of nodes, V = { v 1 , v 2 , . . . , v n } and C = { c 1 , c 2 , . . . , c m } , denoting the decision variables and constraints in Problem (1), respectively. And E = { A jk | A jk = 0 , c j ∈ C, v k ∈ V } is the set of weighted edges connecting variable nodes and constraint nodes, where A is the coefficient matrix in Problem (1). Each node has a feature vector v k or c j describing the information of the variables or constraints. For example, in our experiments, these features include variable types (continuous or binary), variable positions, lower and upper bounds, right-hand side coefficients, constraint types ( = , ≤ , ≥ ), etc.

D Graph convolutional neural network

In the graph convolutional neural network (GCNN)-based approach proposed by Gasse et al. (2019), a bipartite graph G = ( V, C, E ) (with node features c (0) j = c j , v (0) k = v k , and edge features A jk ) is taken as the input. Stacked layers are applied to aggregate information from neighbors and update node embeddings. Each layer has two consecutive half convolutions computed as

where l ∈ { 0 , · · · , L } denotes the layer index, f ( l ) c and g ( l ) c are non-linear transformations gathering information from variable nodes and update on constraint nodes, f ( l ) v and g ( l ) v are on the contrary. All of these four transformations are two-layer perceptrons with ReLU activations. Lastly, another twolayer perceptron f out with sigmoid activation is used to convert the final embeddings to the predictions of integer variables by ˆ x k = Sigmoid ( f out ( v ( L ) k )) . We denote f θ as the GNN parameterized by θ , and the output vector for the discrete part as ˆ x = f θ ( G ) in the remaining sections.

E Detailed experimental settings

E.1 Hyper-parameter tuning

The experiments involved three downstream tasks: fix&optimize, local branching, and node selection. For the first two, we utilized grid search for hyperparameter tuning. In fix&optimize, we adjusted α , the fraction of variables to fix, exploring values from 0 . 1 to 0 . 9 . For local branching, we varied β , the percentage of variables in the local branching constraint, within the same range. For node selection, we adhered to default settings as stated in Khalil et al. (2022). In Table E.1, we report the optimal

̸

hyperparameters for each task and dataset, ensuring clarity and aiding in the reproducibility of our work.

E.2 Computational resources and software

All evaluations are performed under the same configuration. The evaluation machine has one AMD EPYC 7H12 64-Core Processor @ 2.60GHz, 256GB RAM, and one NVIDIA GeForce RTX 3080. CPLEX 22 . 2 . 0 and PyTorch 2.0.1 (Paszke et al., 2019) are utilized in our experiments. The time limit for running each experiment is set to 800 seconds since a tail-off of solution qualities was often observed after that.

F Other supplements

F.1 Dataset details

F.2 p-values for the significance of improvement

We use the paired-sample T-test in MATLAB and report p-values in the following table. A p-value less than 0 . 05 means the mean difference (improvement) is significant. It shows that our proposed framework is effective in finding a better solution.

G Problem formulation and their corresponding symmetry group

G.1 IP

There are I items, J bins, and K resource types. Each item i has a fixed resource requirement a ik for each resource type k . Each bin j has a fixed capacity b k for each resource type k . The goal is to

place all items in bins, while minimizing the imbalance of the resources used across all bins. This problem has a formulation as follows:

where x ij = 1 denotes assigning item i to bin j , d ik is normalized resource requirement for each item, y jk and z k are implicit decision variables to track the imbalance of the resources. Since each bin in this problem is identical (i.e., with the same capacity), reordering bins would not change a feasible solution's feasibility and objective value. This problem naturally has a symmetric group S | J | w.r.t. the ordering of bins J . Specifically, let X ∈ { 0 , 1 } | I |×| J | be a feasible solution of an IP instance with its ( i, j ) -th entry as the value of variable x ij . Then arbitrary permutation π ∈ S | J | acting on its columns { X : j , ∀ j ∈ J } yields an equivalent solution [ X : π (1) , X : π (2) , . . . , X : π ( | J | ) ] .

G.2 SMSP

Given order set O , and slab set S . Color set C , and slab weights Q = { u 0 = 0 , u 1 , u 2 , ..., u k } . u 0 denotes unused slab. The ILP formulation of SMSP used in our experiments is from (Gargani & Refalo, 2007) as

G.3 PESP

Periodic event scheduling problem involves determining optimal schedules for a set of events that occur repeatedly over a fixed period, such as bus or train departures. Consider a set of events E . For each event i ∈ E we would like to schedule a time t i ∈ { 1 , . . . , T -1 } , where T is the periodic length. Besides, a set of activities A ⊆ E × E connect events with each other. Each activity a ∈ A has a lower bound ℓ a ∈ N , an upper bound u a ∈ N , and a weight w a . The goal is to minimize the weighted sum of the slack y a of all activities, while ensuring all activity slacks are within [0 , u a -ℓ a ] .

It can be formulated as:

where [ · ] T denotes the modulo operation, which enforces the periodic nature. It is modeled as [ t j -t i ] T ≜ t j -t i + z a T by introducing additional implicit variables { z a ∈ N } . Due to the existence of modulo operation, we can regard { t i , ∀ i ∈ E} as frames in a clock with intervals [0 , . . . , T -1] . If all t i rotate the same angles simultaneously, then the activity slacks y a remain unchanged. In our experimentation, we substitute t i by t i = ∑ k ( k -1) · x ik and ∑ k x ik = 1 , where x ik ∈ { 0 , 1 } , ∀ i ∈ E , k ∈ { 1 , T } . Let X ∈ { 0 , 1 } |E|× T denotes a feasible solution with its ( i, k ) -th entry as the value of variable x ik , then this problem has a cyclic group C T w.r.t. the column indices of X , i.e., any permutation π ∈ C T acting on the columns of X yields an equivalent feasible solution [ X : π (1) , . . . , X : π ( T ) ] .

G.3.1 Data generation by perturbation

As mentioned above, a PESP instance has a set of events E and a set of activities A ⊆ E × E connecting events with each other. Each activity has a weight w a . The goal is to assign an appropriate time t i to each event i ∈ E to meet some constraints while minimizing the total time slack weighted by { w a , a ∈ A} . These weights heavily impact the time assignment. We perturb these weights to generate new instances by introducing Gaussian noises, i.e., w ′ a = w a + n a , where n a ∼ N ( µ = w a , σ = 0 . 1 ∗ w a ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Appendix A

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 5 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a code link in Section 5.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the p-values in Appendix F.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:The research presented in this paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not present any such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer to Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See the code link in Section 5.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes the SymILO framework for solving combinatorial optimization problems (using machine learning). Building on traditional supervised learning, this method leverages the inherent symmetry structure of the problem to guide the model in learning the true patterns. The author also show its performance on several datasets.

STRENGTHS:
1. The idea of considering symmetry is interesting and crucial, especially when using the optimal solutions as labels. Since the solution space could be symmetry-invariant, using one particular solution as a label may cause the model to fail in learning the intrinsic pattern.
2. The presentation is neat.

WEAKNESSES:
See questions.

QUESTIONS:
1. Although the idea is good, the permutation group, which plays a crucial role, is only vaguely explained in the paper. In most typical combinatorial optimization problems, the permutation group is subtle and may even vary with the instance. In the main body of the paper (Section 4), the authors' direct use of $ G_i $ is confusing. 
2. Following up on the previous point, the authors should explain how to identify the appropriate permutation group for a new problem and clarify whether selecting a specific permutation group significantly impacts the computational complexity during the optimization process.
3. Where in the paper are the details on "passing the outputs to an off-the-shelf solver" provided?
4. I expect the authors to explicitly formulate the problem as a MIP problem in the examples provided in Section B and specify the permutation group.

LIMITATIONS:
See questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article examines the problem of predicting solutions to MILPs with a large number of symmetric solutions. An integer linear program (ILP) is symmetric if its variables can be permuted without changing the structure of the problem. The authors propose a new formulation of supervised learning for problems with symmetry. It includes choosing the optimal permutation of the target solution in the training set. The authors provide a new algorithm that updates learning parameters and permutations alternately. The proposed paradigm can be applied to complement existing methods in which the output of the model is the prediction of the solution. Experiments are conducted together with SL methods for node selection, local branching, and Neural Diving.\n\nSTRENGTHS:\n1.\tThe motivational part is well explained, and examples of symmetry are provided to understand this problem. The theoretical preliminaries provide the reader with the necessary information to understand the symmetry group, permutation classes, etc.\n\n2.\tThe framework is theoretical and general, since it reformulates the supervised learning objective. Hence, it is applicable to any SL-based task, where the target is a solution for combinatorial optimization problem. It also contains proofs that symmetry-aware risk is theoretically preferable, if there exist permutations of instance’s optimal solutions.\n\n3.\tThe experimental design is solid, since the proposed method is combined with three popular deep learning-based frameworks (Neural diving, Node selection with GNNs, Predict-and-search) and for each of them shows a significant increase in quality when solving problems with a large number of symmetric solutions.\n\nWEAKNESSES:\n1.\tThere exist classical approaches (Dantzig-Wolfe decomposition, orbital branching, and etc) that are applied to tackle the symmetry in MILPs. So, the discussion if some of them can be theoretically utilized together with the proposed framework, or some comparison (for example ND + symILO vs ND + some classical method for symmetric problems) would benefit the paper.\n\n2.\tI would also suggest testing performance on other problems (probably with less symmetry) and comparing the computational time. This would trigger a discussion about whether the proposed structure should be applied if the degree of symmetry is unknown.\n\n3.\tIn limitations, the authors say that the sub-problems involved in optimizing permutations can significantly slow down the training process for large-scale problems. An analysis of how significant it will be is needed, as well as an analysis of the convergence and scalability of the alternating minimization algorithm (4.2).\n\nQUESTIONS:\n1.\tIn the vast majority of real cases, a symmetry group G is not known, we know only one of its subgroups. It would be interesting to compare the effect of adding a search for an optimal permutation over a symmetry group, with the same search over some subgroup.\n\n2.\tHow to check the symmetry of the new problems for real-world scenarios, if one wants to apply the proposed symmetry-aware framework? Does it work for non-symmetry problems? Prop 4.1 holds only for the known symmetry group.\n\n3.\tShould one apply the algorithm for symmetric problems where the number and the size of similar solutions is unknown?\n\n4.\thow to compute \\pi^' in line 267? Is argmin_\\pi a trivial problem?\n\n5.\tRefer to weaknesses\n\nLIMITATIONS:\nThe authors point to the potential slowdown on a large instance as a limitation. However, authors do not provide deeper analysis of it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to enhance solution prediction methods for ILPs that contain certain types of symmetry. The main methodological contribution of the paper is a loss function that considers symmetry, along with an optimization algorithm for it. In particular, the loss function applies a permutation to each optimal solution label, and these permutations are learned along with the solution predictor. This is done via an alternating algorithm that alternates between optimizing (graph) neural network weights with a classical loss function and optimizing this permutation. For symmetric groups where any permutation is allowed, the authors show that the latter problem can be efficiently solved via LP. This method is then combined with three downstream methods that use a solution predictor and tested on four sets of problems with symmetry, showing consistent improvements in obtaining better solutions in all cases.\n\nSTRENGTHS:\nThis method overall is a solid and clean approach to incorporate symmetry information in solution prediction methods in ILP. While the alternating algorithm seems quite natural in retrospect, it is a suitable approach particularly with the nice observation that optimizing the permutation for the symmetric group case can be done efficiently via LP. It is also not too complicated to implement. If one is solving a problem with known symmetry via a solution prediction approach, I see little drawback to using this method. The experimental setup is sound and sufficiently extensive, covering four sets of instances and three different methods to integrate this with. Even before reading the computational section, I could see that some computational improvement was expected since it allows the neural network model to not worry about permuted optimal solutions, and the computational section does confirm that with substantial improvements across the board. The paper is overall clear and easy to follow.\n\nWEAKNESSES:\nI do not see any major weaknesses in this paper. Perhaps a concern in terms of relevance is that work does apply to a rather narrow class of instances (those with the types of symmetry in the paper) and it also requires prior knowledge of which type of symmetry the problem has. However, this does include a number of important problems in discrete optimization, so I do not view this as a significant issue. Other than that, there are a few parts of the experimental setup that need to be clarified as discussed below, but they can be easily fixed.\n\nQUESTIONS:\n1. It would be helpful for the reader to know earlier in the Introduction that the predicted optimal solution will be coupled with downstreams approaches rather than used directly. Options are to mention this in the introduction, or alternatively move Figure 2 (which is very helpful) near the introduction, as it easier to go through the paper with the big picture in your head. If you do opt to move Figure 2 earlier, consider adding some context in the caption since the reader does not have much context at that point.\n\n2. Please add a description of the problems in the Appendix, preferrably (but not necessarily) with the MILP formulation used, and point out where the symmetry comes from. In addition, indicate how the perturbation for PESPlib was done.\n\n3. I tried to check the anonymous code repository, but it was expired.\n\n4. Could you add standard deviations / errors to Table 2?\n\n5. The Appendix says that the evaluation machine has a GPU, but I cannot clearly find if the training time reported in Table 2 is for GPUs. The reason why this matters is that this paper shows that the LP is not a bottleneck, but the picture might be different if Table 2 were CPU times. In particular, I am assuming that the $r_s$ time includes: GPU training time + CPU LP solves + communication overhead between GPU and CPU. Could you clarify this in the paper?\n\n6. The paper does not describe what is \"Tuned CPLEX\". In particular, an important question is, do you tune the symmetry breaking parameters in CPLEX? If you do not, would be able to include it in the paper? Please also mention in the paper how the tuning is done overall, and you might as well mention that CPLEX contains its own symmetry handling methods based on classical approaches. Another important set of parameters to tune are primal focus (e.g. heuristics), but I assume that is already included in your tuning.\n\n7. Typos: Line 269: Is $\\hat{x}$ supposed to be $\\hat{y}$? Line 420 has a typo: \"don not\".\n\nLIMITATIONS:\nThe paper properly indicates the main limitations of this work in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an ML-based framework, SymLo, for solving MILPs that leverages symmetries of ILPs to improve ML performance. SymLo takes into account symmetric groups on the solutions and formulates a learning task with respect to the model parameters and the permutations that belong to the symmetric group of each instance. The loss is challenging to minimize, and an alternating algorithm is proposed to mitigate the issues. In experiments, the method is evaluated on three different tasks and on four benchmarks.\n\nSTRENGTHS:\n1. The proposed formulation and method for symmetry-aware learning for ILPs is novel.\n\n2. Empirical results are promising, suggesting its usefulness for instances that have symmetric properties in the solutions. It is also applicable to different ILP learning tasks.\n\n3. The paper is engaging and well-written.\n\nWEAKNESSES:\nThis is the second time I have reviewed this paper. I am satisfied with the authors in addressing my previous comments.\n\nNonetheless, I still believe that evaluating the method on a third task would make the contribution of this work more solid and convincing.  The fix&optimize task and the local branching task are similar to each other. I am not surprised to see that it works on one if it already has worked for the other. Some tasks that you could consider: Initial Basis Selection [1][2], Backdoor variable predictions [3]\n\n[1] Fan, Zhenan, et al. \"Smart initial basis selection for linear programs.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Zhang, Yahong, et al. \"MILP-FBGen: LP/MILP Instance Generation with Feasibility/Boundedness.\" Forty-first International Conference on Machine Learning.\n\n[3] Ferber, Aaron, et al. \"Learning pseudo-backdoors for mixed integer programs.\" International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research. Cham: Springer International Publishing, 2022.\n\n2. The limitation of generalizing the method to larger instances was discussed with the other reviewers last time. It would be important to include such a discussion in the main paper to provide a more comprehensive understanding of the method's pros and cons.\n\nQUESTIONS:\nI don't have specific questions.\n\nLIMITATIONS:\nThe authors talk about some limitations of the work, but not all that are known.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}