Title: Learning Cuts via Enumeration Oracles
Venue: NeurIPS 2023
Accepted: yes
Keywords: Integer Programming

Cutting Planes

Optimization
Abstract: Cutting-planes are one of the most important building blocks for solving large-scale integer programming (IP) problems to (near) optimality. The majority of cutting plane approaches rely on explicit rules to derive valid inequalities that can separate the target point from the feasible set. Local cuts, on the other hand, seek to directly derive the facets of the underlying polyhedron and use them as cutting planes. However, current approaches rely on solving Linear Programming (LP) problems in order to derive such a hyperplane. In this paper, we present a novel generic approach for learning the facets of the underlying polyhedron by accessing it implicitly via an enumeration oracle in a reduced dimension. This is achieved by embedding the oracle in a variant of the Frank-Wolfe algorithm which is capable of generating strong cutting planes, effectively turning the enumeration oracle into a separation oracle. We demonstrate the effectiveness of our approach with a case study targeting the multidimensional knapsack problem (MKP).

Learning Cuts via Enumeration Oracles

Abstract

Cutting-planes are one of the most important building blocks for solving largescale integer programming (IP) problems to (near) optimality. The majority of cutting plane approaches rely on explicit rules to derive valid inequalities that can separate the target point from the feasible set. Local cuts , on the other hand, seek to directly derive the facets of the underlying polyhedron and use them as cutting planes. However, current approaches rely on solving Linear Programming (LP) problems in order to derive such a hyperplane. In this paper, we present a novel generic approach for learning the facets of the underlying polyhedron by accessing it implicitly via an enumeration oracle in a reduced dimension. This is achieved by embedding the oracle in a variant of the Frank-Wolfe algorithm which is capable of generating strong cutting planes, effectively turning the enumeration oracle into a separation oracle. We demonstrate the effectiveness of our approach with a case study targeting the multidimensional knapsack problem (MKP).

1 Introduction

In this paper, we deal with integer programs (IP)

where A ∈ Q m × n , b ∈ Q m , and c ∈ Q n . Let P := { x ∈ R n : Ax ≤ b } be the underlying polyhedron and its integer hull P I := conv( P ∩ Z n ) . We restrict attention to the case in which all variables are required to be integral, as the methods we will propose are more readily applicable to this case, but the general idea works for mixed-integer programs (MIP) as well.

Solving IPs is NP -hard in general, however, surprisingly fast algorithms exist in practice [1, 50]. The most successful approach to solving IPs is based on the branch-and-bound algorithm and its extensions. This algorithm involves breaking down the original problem into smaller subproblems that are easier to solve through a process known as branching. By repeatedly branching on subproblems, a search tree is obtained. The bounding step involves computing upper bounds for subproblems and pruning suboptimal nodes of the tree in order to avoid enumerating exponentially many subproblems. Upper bounds are generally computed with the help of Linear Programming (LP) relaxations

Because the integrality constraints are relaxed, optimal solutions of (LP) provide an upper bound for the original problem (IP).

Boro Sofranac

Sebastian Pokutta

Alternatively, cutting plane procedures iteratively solve LP-relaxations as long as the solution x ∗ is not integral (and thus x ∗ / ∈ P I ). To remove these solutions x ∗ from the relaxation's polyhedron, one adds cutting planes (or cuts ) ⟨ α, x ⟩ ≤ β with a ∈ Q n , β ∈ Q , and ⟨ α, x ∗ ⟩ > β . The search for such cutting planes or to determine that none exists is called the separation problem . The strongest cuts are those that define a facet , i.e., the face P I ∩ { x : ⟨ α, x ⟩ = β } has co-dimension 1 with respect to P I . When the cutting plane method is combined with branch-and-bound, the resulting algorithm is often called branch-and-cut . Gomory conducted foundational work in this field, demonstrating that pure cutting plane approaches can solve integer programs with rational data in a finite number of steps without the need for branching [32, 33, 34].

Gomory's initial approach to cutting planes suffered from numerical difficulties at that time, preventing pure cutting plane methods from being effective in practical applications. However, his proposed (Gomory) mixed integer (GMI) cuts are very efficient if combined with branch-and-bound (see the computational study in [11]) and still are one of the most important types of cutting planes used by contemporary solvers. As more GMI cuts are added to a problem, their incremental value tends to diminish. To address this issue, modern MIP and IP solvers use a range of techniques to generate cuts, e.g., mixed-integer-rounding (MIR) inequalities [55], knapsack covers [26, 38], flow covers [39], lift-and-project cuts [5], {0, 1 2 }-Chvátal-Gomory cuts [22], and others.

Most cutting plane separation algorithms rely on fixed formulas to derive valid inequalities that separate the target point x ∗ from the polyhedron P I . An alternative approach is to directly seek to derive the facets of P I that separate the point x ∗ . Notice that while the facets of the polyhedron P are explicitly known from the problem definition, the facets of its integer hull P I are unknown in general. While the facet-defining inequalities are intuitively the strongest cuts, they can be relatively expensive to explicitly compute, limiting their applicability in practice. Local cuts , a type of cutting planes that try to derive facets of P I , approach this problem by deriving facets of P I in a reduced dimension, and then lifting those cuts to obtain facets in the original dimension. In this paper, we will propose a new variant of the Frank-Wolfe algorithm with the goal of learning the (unknown) facets of P I (or at least valid inequalities) in a reduced dimension, which can then be lifted to the original dimension and be used as strong cutting planes. In our learning approach, the underlying polyhedron will only be accessed via an algorithmically simple linear optimization oracle, in contrast to existing approaches, which also need to solve LPs.

1.1 Related Work

Local cuts have first been introduced as 'Fenchel cuts' in Boyd [13, 14], who developed an algorithm to exactly separate inequalities for the knapsack polytope via the equivalence of separation and optimization. They were subsequently investigated extensively by Applegate et al. [3] for solving the traveling salesman problem (TSP). Buchheim et al. [20, 21] and Althaus et al. [2] adopted local cuts into their approaches for solving constrained quadratic 0-1 optimization problems and Steiner-tree problems, respectively. In [25], Chvátal et al. generalize the local cuts method to general MIP problems.

In the context of knapsack problems, after the aforementioned work of Boyd [13, 14], Boccia [12] introduced an approach based on local cuts, as stated by Kaparis and Letchford [47] 1 , who further refined the algorithm. Vasilyev presented an alternative approach with application to the generalized assignment problem in [63], see also the comprehensive computational study conducted by Avella et al. in [4]. In [64], Vasilyev et al. propose a new implementation of this approach, with the goal of making it more efficient. In [37], Gu presents an extension of the algorithm of Vasilyev et al. [64].

Existing works on the application of learning methods in solving IP (and more generally MIP) problems can in general be divided into two categories: learning decision strategies within the solvers, and learning heurisitcs to obtain feasible (primal) solutions. Examples of the former would be learning to select branching variables [6, 31, 48], learning to select branching nodes [41], learning to select cutting planes [61], learning to optimize the usage of primal heuristics [23, 42, 49]. A typical example of the latter case would be learning methods to develop large neighborhood search (LNS) heuristics [28, 60, 59]. Additionally, a number of works in the literature have focused on learning algorithms for solving specific IP problems [8, 44, 27, 51, 54, 56]. For a more detailed overview of using learning methods in IP, we refer the interested reader to [9].

1.2 Contribution

The contributions of this paper can be summarized as follows:

We present an efficient, LP-free separation framework that aims to learn local cuts for IPs through the solution of subproblems. We propose to use a variant of the Frank-Wolfe [29] algorithm to solve the associated separation problem. The resulting framework is general and - given the availability of a suitable lifting method - applicable to any IP.

We propose a new, dynamic stopping criterion for the application of Frank-Wolfe to the separation problem at hand. This new criterion, derived by exploiting duality information, directly evaluates the strength of the resulting cut and thus dramatically decreases the number of iterations.

We illustrate the benefit of our approach in a case study for the multidimensional knapsack (MKP) problem, demonstrating its effectiveness. Our computational results show that embedding our method in the academic solver SCIP leads to 31% faster solving times on the instances solved to optimality, on average.

The rest of this paper is organized as follows: In Section 2, the fundamental framework of local cuts and required notation are introduced. Section 3 presents our approach for the LP-less generation procedure for local cuts. Section 4 demonstrates how the aforementioned framework can be applied to solve the multidimensional knapsack problem. Computational experiments are presented in Section 5. Finally, Section 6 summarizes conclusions and future work.

2 Local Cuts

To describe the idea of local cuts, assume that P ⊂ R n , n > 0 , is a polytope, i.e., bounded, and full-dimensional. Then one considers a small subproblem with underlying polytope ˜ P , which is usually an orthogonal projection of P onto a lower-dimensional space. The polytope ˜ P is restricted to being non-empty and its dimension 0 < k ≤ n is chosen small enough such that integer optimization problems over ˜ P can be solved efficiently in practice, for example, by enumeration. Consider a projection ˜ x of the point to be separated x ∗ on R k . The procedure tries to generate a valid cut ⟨ ˜ α, x ⟩ ≤ ˜ β with ˜ α ∈ Q k , ˜ β ∈ Q , such that ⟨ ˜ α, ˜ x ⟩ > ˜ β , i.e., it cuts off ˜ x from ˜ P . This cut can be 'lifted' to the original space, which yields a cut ⟨ α, x ⟩ ≤ β that hopefully cuts off x ∗ . 2

The approach to generate ⟨ ˜ α, x ⟩ ≤ ˜ β , in the literature mentioned above, relies on the equivalence between optimization and separation [36] and can be very briefly explained as follows. By the Minkowski-Weyl Theorem, we can express ˜ P as the convex hull of its vertex set V . Let ˜ x 0 ∈ ˜ P be an interior point. Then consider the LP

The dual problem ( D ) is

Let ˜ α , ˜ β be an optimal solution of ( D ) . Then ⟨ ˜ α, x ⟩ ≤ ˜ β is a valid inequality for ˜ P , since by construction ⟨ v, ˜ α ⟩ ≤ ˜ β holds for all v ∈ V and thus by convexity for all points in ˜ P . The objective enforces that this cut is maximally violated by ˜ x if the optimal value is positive.

Since ˜ P may have an exponential number of vertices, problem ( D ) can be solved by a column generation algorithm (or cutting plane algorithm in the primal). In each iteration, one needs to solve the following pricing problem for the current point (ˆ α, ˆ β ) : Decide whether there exists v ∈ V with ⟨ v, ˆ α ⟩ > ˆ β . This can be done by maximizing ˆ α over the subproblem ˜ P , i.e., one can use a linear optimization oracle for the subproblem. This subproblem can contain integrality constraints, thereby requiring, again, IP techniques. Note that the most interesting case is where we operate on integer

hulls, i.e. ˜ P = ˜ P I to generate cuts for P = P I . In this way, local cuts can help solving an integer optimization problem over P . Hence, in the following sections, any reference to P , ˜ P holds for the integer case as well and our case study illustrates exactly that.

As mentioned above, the strongest cutting planes are those that define facets. The tilting method by Applegate et al. [3] produces such a facet. Buchheim et al. [20] introduced a different formulation that automatically produces a facet. Chvátal et al. [25] developed a formulation for general MIPs using linear optimization oracles. All three approaches use a sequence LPs at their heart; either for tilting a plane or through a column-generation procedure.

3 Learning Strong Cuts from Enumeration

The local cuts framework, applicable to general IPs, relies on a sequence of three operators: SEP, FACET and LIFT. SEP refers to a separation oracle separating the projected point ˜ x from ˜ P that returns a separating cut ⟨ ˜ α, x ⟩ ≤ ˜ β (or certifies that ˜ x ∈ ˜ P ). FACET further refines the cut until it represents a facet of ˜ P and lastly, LIFT transforms the resulting facet into the space of P such that it separates x ∗ from P with high probability. In some variants of local cuts, SEP and FACET may be combined into one step similar to [20], whereas in [25], the tilting process is a separate, concrete embodiment of FACET. Note that that facets of the subproblem, when lifted, result in the strongest cuts. In practice, it is often sufficient to find good valid inequalities of ˜ P . As mentioned before, the original approach for local cuts through duality requires an expensive column-generation method which is based on LPs. In this section, we derive an alternative and LP-less approach.

The general idea of our new approach is sketched in Figure 1: Given a point ˜ x ∈ R n that we intend to separate from P , we solve the following optimization problem:

with f ( y ) := 1 2 ∥ y -˜ x ∥ 2 . Observe that this is effectively the projection of ˜ x onto ˜ P under the ℓ 2 -norm and that ∇ f ( y ) = ( y -˜ x ) .

We solve (Separation) with a suitable variant of the Frank-Wolfe algorithm. The Frank-Wolfe algorithm [29] (also called: Conditional Gradients [53]) is a method to minimize a smooth convex function f over a compact convex domain P by only relying on a First-order Oracle (FO) for f , i.e., given a point x the oracle returns ∇ f ( x ) (and potentially f ( x ) ) as well as a Linear Minimization Oracle (LMO) ('oracle' for the remainder of this paper), i.e., given an objective vector c , the oracle returns v ∈ argmin x ∈ ˜ P ⟨ c, x ⟩ . The original Frank-Wolfe algorithm, provided with step sizes γ t > 0 , iteratively calls the LMO to determine v t ← argmin v ∈C ⟨∇ f ( y t ) , v ⟩ and updates the iterate to y t +1 ← y t + γ t ( v t -y t ) . There are various step-size strategies for γ t , but the actual choice is irrelevant for the discussion here; a common choice is γ t = 2 t +2 .

The main advantages of using Frank-Wolfe are (1) if there is a LP-less oracle, valid inequalities can be generated without solving LPs, (2) the computational overhead of the Frank-Wolfe steps compared to calls to the LMO are very light and, finally, (3) as we will show, for the case of (Separation), we can derive a new dynamic stopping criterion that can dramatically reduce the number of iterations. Note that we are not guaranteed to end up with facets, especially when the method is stopped early, however, valid inequalities that are 'close' to being a facet can still serve as strong cutting planes.

For our problem minimizing f , the Frank-Wolfe algorithm iteratively calls the oracle and updates its current iterate through a convex combination of the previous iterate and oracle's solution vertex. Step by step, the solution is thus expressed through a convex combination of vertices in ˜ P as shown in Figures 1a - 1c. At convergence, the hyperplane ⟨∇ f ( y ∗ ) , x ⟩ ≥ ⟨∇ f ( y ∗ ) , y ∗ ⟩ forms the desired cut.

3.1 Separation via Conditional Gradients

Let y ∗ ∈ ˜ P be an optimal solution to (Separation) and let x ∈ ˜ P be arbitrary. By convexity, it follows that 0 ≤ f ( x ) -f ( y ∗ ) ≤ ⟨∇ f ( x ) , x -y ∗ ⟩ ≤ max v ∈ ˜ P ⟨∇ f ( x ) , x -v ⟩ and the last quantity is referred to as Frank-Wolfe gap (at x ) . Moreover, the following lemma holds, which is a direct consequence of the first-order optimality condition.

v

1

v

0

v

2

˜

x

f

(

y

) =

∥

y

-

˜

x

∥

2

˜

P

(a)

v

1

v

0

v

2

˜

x

f

(

y

) =

∥

y

-

˜

x

∥

2

˜

P

y

k

∇

f

(

y

k

)

(b)

v

1

v

0

v

2

˜

x

f

(

y

) =

∥

y

-

˜

x

∥

2

˜

P

y

k

+

i

∇

f

(

y

k

+

i

)

(c)

Lemma 1 (First-order Optimality Condition) . Let y ∗ ∈ ˜ P . Then y ∗ is an optimal solution to min y ∈ ˜ P f ( y ) if and only if ⟨∇ f ( y ∗ ) , y ∗ -v ⟩ ≤ 0 for all v ∈ ˜ P (and in particular max v ∈ ˜ P ⟨∇ f ( y ∗ ) , y ∗ -v ⟩ = 0 ).

̸

Note that in the constrained case, it does not necessarily hold that ∇ f ( y ∗ ) = 0 , if y ∗ is an optimal solution. In fact, if the ˜ x that we want to separate is not contained in ˜ P , then f ( y ∗ ) > 0 and ∇ f ( y ∗ ) = 0 since y ∗ will lie on the boundary of ˜ P .

It turns out that we can naturally use an optimal solution y ∗ ∈ ˜ P to (Separation) to derive a separating hyperplane. By Lemma 1:

which holds for all v ∈ ˜ P . Moreover, if ˜ x ̸∈ ˜ P , then (Cut) is violated by ˜ x , i.e., ⟨∇ f ( y ∗ ) , y ∗ ⟩ > ⟨∇ f ( y ∗ ) , ˜ x ⟩ , since ⟨∇ f ( y ∗ ) , y ∗ -˜ x ⟩ ≥ f ( y ∗ ) -f (˜ x ) = f ( y ∗ ) > 0 .

Usually, however, we do not solve Problem (Separation) exactly, but rather up to some accuracy. In fact, the Frank-Wolfe algorithm often uses the Frank-Wolfe gap as a stopping criterion, minimizing the function until for some iterate y t it holds max v ∈ ˜ P ⟨∇ f ( y t ) , y t -v ⟩ ≤ ε for some target accuracy ε ; note that the Frank-Wolfe gap converges with the same rate (up to small constant factors) as the primal gap (see e.g., [46]). Given an accuracy ε > 0 , we obtain the valid inequality

for all v ∈ ˜ P , which also separates ˜ x from ˜ P if it is √ ε -far from ˜ P , i.e., ∥ y ∗ -˜ x ∥ > √ ε :

The accuracy ε is chosen depending on the application; see also [17] for a sensitivity analysis for conditional gradients.

3.1.1 A dynamic stopping criterion

It turns out, however, that in our case of interest, the above can be significantly improved by exploiting duality information. This allows us not only to stop the algorithm much earlier, but we also obtain a separating inequality directly from the associated stopping criterion and duality information.

The stopping criterion is derived from a few simple observations, which provide a new characterization of a point ˜ x that can be separated from ˜ P . Our starting point is the following standard expansion. Let v ∈ ˜ P be arbitrary and let y t be an iterate from above. Then,

which is equivalent to

Observe that the left hand-side is the Frank-Wolfe gap expression at iterate y t (except for the maximization over v ∈ ˜ P ) since ∇ f ( y t ) = y t -˜ x .

Necessary Condition. Let us first assume ∥ y t -v ∥ < ∥ ˜ x -v ∥ for all vertices v ∈ ˜ P in some iteration t . Then (1) yields

If v t is the Frank-Wolfe vertex in iteration t , we obtain:

Subtracting 1 2 ∥ ˜ x -y t ∥ 2 on both sides and re-arranging yields: 0 < 1 2 ∥ y ∗ -˜ x ∥ 2 , which proves that ˜ x ̸∈ ˜ P . Moreover, (1) also immediately provides a separating hyperplane: observe that (altTest) is actually a linear inequality in v and it holds for all v ∈ ˜ P since the maximum is achieved at a vertex. However, for the choice v = ˜ x the inequality is violated.

Sufficient Condition. Suppose that in each iteration t there exists a vertex ¯ v t ∈ ˜ P (not to be confused with the Frank-Wolfe vertex), so that ∥ y t -¯ v t ∥ ≥ ∥ ˜ x -¯ v t ∥ . In this case (1) ensures:

Thus, the Frank-Wolfe gap satisfies in each iteration t that

i.e., the Frank-Wolfe gap upper bounds the distance between the current iterate y t and point ˜ x in each iteration. Now, the Frank-Wolfe gap converges to 0 as the algorithm progresses, with iterates y t ∈ ˜ P , so that with the usual arguments (compactness and limits etc.) it follows that ˜ x ∈ ˜ P . In total, we obtain the following result.

Characterization 2 . The following are equivalent:

(Non-Membership) ˜ x ̸∈ ˜ P .

(Distance) There exists an iteration t , so that ∥ y t -v ∥ < ∥ ˜ x -v ∥ for all vertices v ∈ ˜ P .

(FW Gap) For some iteration t , max v ∈ ˜ P ⟨ y t -˜ x, y t -v ⟩ < 1 2 ∥ ˜ x -y t ∥ 2 .

In particular, Characterization 2.3 can be easily tested within the algorithm, since the Frank-Wolfe gap is computed anyways. Using this criterion significantly improves the performance of the algorithm. Moreover, the characterization above can also be combined with standard convergence guarantees to estimate the number of iterations required to either certify non-membership or membership (up to an ε -error): If we use the vanilla Frank-Wolfe algorithm, then by standard guarantees (see e.g., [16]) it is known that the Frank-Wolfe gap g t = max v ∈ ˜ P ⟨ y t -˜ x, y t -v ⟩ satisfies min 0 ≤ τ ≤ t g τ ≤ 4 LD 2 t +3 for appropriate positive constants L and D . Suppose that max v ∈ ˜ P ⟨ y t -˜ x, y t -v ⟩ ≥ 1 2 ∥ ˜ x -y t ∥ 2 holds for all iterations 0 ≤ t ≤ T . We want to estimate how long this can hold. If ˜ x ̸∈ ˜ P , then using the convergence guarantee yields:

Using L = 1 as f ( y ) = 1 2 ∥ ˜ x -y ∥ 2 and rearranging we obtain

i.e., after at most T iterations we have certified that ˜ x is not in ˜ P . Guarantees for more advanced Frank-Wolfe variants can be obtained similarly.

Algorithm 1 Lazy Away-Step Frank-Wolfe [18, 19] with explicit active set and early termination

3.2 Computational Aspects

A common trait of the local cuts framework is that ˜ P is accessed implicitly via an oracle returning vertices. By far the simplest black-box oracle for any bounded IP is enumeration , which simply evaluates all possible solutions x and picks the best one. If the IP is unbounded, then pure enumeration does not suffice any more and the oracle needs to take the unboundedness into account. For some problems, we can find problem-specific algorithms that only enumerate over feasible solutions or otherwise exploit the structure of the problem at hand to reduce the complexity of enumeration. Examples are the dynamic programming approach for knapsack problems, see Section 4.1, or directly enumerating n ! possible permutations of n items for the linear ordering problem (LOP).

Similarly to the enumeration oracle, the lifting routine can also avail of problem-specific structure in some cases. In the case of LOPs, the so-called trivial lifting lemma holds, that is, facet-defining inequalities of the LOP polytope in dimension n also define facets in dimension r > n [35], meaning that no lifting is needed at all in this case. For knapsack problems, we can again use a dynamic programming approach, see Section 4.2.

In general, to apply our method to a given class of IP probems, one needs three components: i) a projection P → ˜ P . ii) An oracle solving linear optimization problems over ˜ P to optimality; in order to be practical, the selected ˜ P should be such that the oracle runs reasonably fast. iii) A lifting method to lift cuts from ˜ P up to P .

In our implementation, we use the Lazy Away-Step Frank-Wolfe algorithm of [18, 19], which converges linearly for (Separation). We integrate the novel termination criterion from Characterization 2.3, leading to Algorithm 1. This algorithm should be thought of as a more advanced version of the vanilla Frank-Wolfe algorithm. This variant is motivated by the fact that Frank-Wolfe trends towards sparse solutions and hence the oracle will often return previously-seen vertices. Hence, instead of querying the expensive oracle, one stores all previous vertices in a active set whose size is controlled through so-called away steps. It provides superior convergence speed both in iterations and

wall-clock time, exploiting the strong convexity of our objective function of the separation problem; we refer the interested reader to [46, 52, 16] for an overview. Lazification, to be thought of as an advanced caching technique, further reduces the per-iteration cost by reusing previously computed LMO solutions. Lastly, we note that in our setting and case study presented in Section 4, the LMO always returns a vertex. Even though this is not a theoretical requirement for the results presented in this paper, we do not consider the alternative case for brevity.

4 Case Study: The Multidimensional Knapsack Problem

The multidimensional knapsack problem (MKP) is a well-known problem in combinatorial optimization and is strongly NP -hard. It has been used to address various practical resource allocation problems [30]. The problem involves maximizing the total profits of selected items, taking into account m resource capacity (knapsack) constraints. There are n items that contribute profits given by c ∈ Z n . The resource consumption of item j for the i th knapsack is given by a ij ∈ Z + ; this defines a matrix A = ( a ij ) ∈ Z m × n + . The capacities of the knapsacks are given by b ∈ Z m . We define binary variables x ∈ { 0 , 1 } n such that x j is equal to 1 if item j is selected and 0 otherwise. Then MKP can be expressed as an IP:

There exists abundant literature on the knapsack problems; we refer the interested reader to the recent survey by Hojny et al. [43].

We will also test our approach on the instances of the generalized assignment problem (GAP), see Section 5. GAP is a variant of MKP with applications in scheduling [43]. In addition to the constraints from the MKP problem, it is required that each of the n items be assigned to exactly one knapsack. The interested reader can find a survey, more details, and a comprehensive reference list in [4, 58].

In order for our approach to work, we need to provide two things: the oracle , presented in Section 4.1, and the lifting routine, presented in Section 4.2, cf. Section 2 and Section 3.

We consider each knapsack problem in turn and try to generate inequalities that are valid for each individual knapsack. This has the advantage that there are practically efficient oracles and more importantly efficient lifting processes. The disadvantage is that the cuts might be weaker, since they are valid for all integer solutions for all knapsack constraint instead of their intersection. An alternative would be to consider optimization oracles for the complete set of knapsack constraints as done by Gu [37]. However, then either lifting becomes more computationally demanding or one cannot use lifting.

4.1 The Linear Minimization Oracle

The process begins with a solution x ⋆ of the LP relaxation of the MKP. We create a reduced knapsack problem of dimension k ∈ Z + , k ≤ n by removing variables of each knapsack that have integral values (0/1) in the LP relaxation. The oracles now solve the knapsack problems (KP) for each constraint of the form max {⟨ c, x ⟩ : ⟨ w,x ⟩ ≤ C, x ∈ { 0 , 1 } n } , with c ∈ R k , w ∈ Z k + , C ∈ Z + . In practice, the dimension k is rather small (in our test sets, see Section 5, we observe an average k value of 9.6 with a maximal size of 26), allowing for efficient solution approaches. In our implementation, we use a LMO based on dynamic programming . We note that we also apply the preprocessing improvements described by Vasilyev et al. [64], before we run the oracle on the reduced problem.

Dynamic programming, as presented by Bellman in 1957 [7], was one of the earliest exact algorithms for solving KPs. Toth [62] presents additional improvements to the algorithm. More recently, Boyer et al. present massively-parallel implementations running on GPUs [15]. The space and time complexity of the dynamic programming algorithm for KP is O ( kC ) [15], where k is the number of items and C is the knapsack capacity. For this work, we reuse the single-threaded, CPU-based implementation of dynamic programming available in the open-source solver SCIP [10]. As we rely on existing implementations, we refer the interested reader to the above references for more details on this algorithm.

4.2 The Lifting Routine

Lifting knapsack constraints has been extensively studied in literature, see [43] for a short survey with comprehensive references. Therefore, we only briefly summarize the implemented methods here and refer the interested reader to [43] and references therein for more details.

Let { x ∈ { 0 , 1 } n : ∑ n j =1 a j x j ≤ a 0 } be one of the original knapsack constraints (corresponding to a single row in Ax ≤ b in (MKP)). Define [ n ] := { 1 , . . . , n } , F 0 := { j ∈ [ n ] : x ⋆ j = 0 } , and F 1 := { j ∈ [ n ] : x ⋆ j = 1 } . Then S := [ n ] \ ( F 0 ∪ F 1 ) are the variable indices in the reduced knapsack. The lifting procedure then lifts a given inequality ∑ j ∈ S α j x j ≤ α 0 valid for the reduced knapsack polytope { x ∈ { 0 , 1 } S : ∑ j ∈ S a j x j ≤ a 0 -∑ j ∈ F 1 a j } to a valid inequality for the original knapsack by computing new coefficients β j , j ∈ F 0 ∪ F 1 :

We implemented algorithms known as sequential up-lifting and sequential down-lifting , respectively. The implementation is based on dynamic programming as described by Vasilyev et al. [64].

5 Computational Experiments

We implemented the described methods in C/C++, using a developer version of SCIP 8.0.4 (githash 3dbcb38) and CPLEX 12.10 as LP-solver. All tests were performed on a Linux cluster with 3.5 GHz Intel Xeon E5-1620 Quad-Core CPUs, having 32 GB main memory and 10 MB cache. All computations were run single-threaded and with a time limit of one hour. To concentrate on the improvement of local cuts on the dual bound, we initialize all runs with the optimal value. We used ϵ = 1 × 10 -9 in Algorithm 1.

To demonstrate the advantage of using local cuts with the Frank-Wolfe approach, we run our implementation on the generalized assignment instances from the OR-Library available at http:// people.brunel.ac.uk/~mastjjb/jeb/orlib/gapinfo.html . These instances have also been used by Avella et al. [4].

The results are presented in Table 1. Here, 'default' are the default, factory settings of SCIP. The other settings are lcX-nc-Y , where X = 0 means that we only separate local cuts in the root node and X = 1 means that we separate local cuts in the whole tree; Y refers to whether we perform down lifting ( Y = downlift) or up- and down lifting ( Y = lifting). Note that for these settings, where local cuts are enabled, we turn the generation of all other cuts off, because this (somewhat surprisingly) showed better performance. The CPU time in seconds ('time') and separation time ('sep. time') as well as number of nodes ('#nodes') are given as shifted geometric means 3 . The numbers of generated cuts ('#cuts') are arithmetic means. Note that the iteration limit for the Frank-Wolfe algorithm is 10 000 in the root node and 1000 in the subtree. We also reduce the effect of cut filtering allowing for more cuts to enter the main LP. Moreover, we initialize the runs with the best know solution values as in [4].

The results show that the best version is lc1-nc-lifting , i.e., it helps to separate local cuts in every node and perform up- and down lifting. This version is roughly 31% faster than the default settings on the instances solved to optimality. Using only down lifting performs badly. In any case, on these

instances, using our local cuts method is a big advantage. Additional results are given in the appendix. In addition, we also ran an experiment in which we applied complemented mixed-integer rounding (CMIR) on the produced cut, which turned out to not be helpful and is therefore not reported in detail.

Some additional observations over all 45 instances in the test set are as follows: Variant lc1-nc-lifting called local cuts separation 16 870 . 6 times on average. The total time for FrankWolfe separation is about one third of the total time. The time spent in the oracle is 17 . 3 seconds on average compared to a total of 87 . 9 seconds for the complete Frank-Wolfe algorithm. On average 69 968 . 8 calls ended running into the iteration limit, 81 . 2 detected optimality with a zero gradient, 8652 . 9 stopped because the primal gap is small enough, and 143 155 . 9 stopped because of the termination criterion of Section 3.1.1. This demonstrates the effect of this criterion.

6 Conclusion and Future Work

In this paper, we presented a novel method to learn local cuts without relying on solutions of LPs in the process. To show the effectiveness of our approach, we selected the multidimensional knapsack problem as a case study and presented computational results to support our claims.

Solving LPs has proved to be notoriously hard to parallelize, with only minor performance improvements reported in literature to date [40, 45]. Thus, existing methods for deriving local cuts, which rely on solving LPs, typically run single-threaded, on CPUs. Our approach is quite fast, as demonstrated in the computational experiments for our target problem class, but also paves the way for exploring highly parallel implementations on heterogeneous hardware and compute accelerators. This is made possible by eliminating the dependence on LPs and instead relying on the Frank-Wolfe algorithm. One such option we would like to explore in the future is to derive a vectorized version of our Frank-Wolfe algorithm that could work on multiple separation problems at the same time, increasing the computational density of the operations performed and availing of massively parallel compute accelerators like GPUs in the process.

The presented method is generic and can be applied to any (M)IP. We have chosen one important problem class in this paper to demonstrate the method. A natural extension of this work would be to consider other important problem classes and evaluate the benefits of using our method on those problems - especially those with beneficial properties as outlined in Section 3.2.

References

T. Achterberg and R. Wunderling. Mixed integer programming: Analyzing 12 years of progress. In M. Jünger and G. Reinelt, editors, Facets of Combinatorial Optimization: Festschrift for Martin Grötschel , pages 449-481. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.

E. Althaus, T. Polzin, and S. V. Daneshmand. Improving linear programming approaches for the Steiner tree problem. In K. Jansen, M. Margraf, M. Mastrolilli, and J. D. P. Rolim, editors, Experimental and Efficient Algorithms , pages 1-14, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.

D. Applegate, R. Bixby, V. Chvátal, and W. Cook. TSP Cuts Which Do Not Conform to the Template Paradigm. In M. Jünger and D. Naddef, editors, Computational Combinatorial Optimization , volume 2241 of Lecture Notes in Computer Science , pages 261-303. Springer Berlin Heidelberg, 2001.

P. Avella, M. Boccia, and I. Vasilyev. A computational study of exact knapsack separation for the generalized assignment problem. Computational Optimization and Applications , 45(3):543-555, 2010.

E. Balas, S. Ceria, and G. Cornuéjols. A lift-and-project cutting plane algorithm for mixed 0-1 programs. Mathematical Programming , 58(1):295-324, 1993.

M.-F. Balcan, T. Dick, T. Sandholm, and E. Vitercik. Learning to branch. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 344-353. PMLR, 10-15 Jul 2018.

R. Bellman. Dynamic Programming . Princeton University Press, Princeton, NJ, USA, 1957.

I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio. Neural combinatorial optimization with reinforcement learning, 2017.

Y. Bengio, A. Lodi, and A. Prouvost. Machine learning for combinatorial optimization: A methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, Apr. 2021.

K. Bestuzheva, M. Besançon, W.-K. Chen, A. Chmiela, T. Donkiewicz, J. van Doornmalen, L. Eifler, O. Gaul, G. Gamrath, A. Gleixner, L. Gottwald, C. Graczyk, K. Halbig, A. Hoen, C. Hojny, R. van der Hulst, T. Koch, M. Lübbecke, S. Maher, F. Matter, E. Mühmer, B. Müller, M. E. Pfetsch, D. Rehfeldt, S. Schlein, F. Schlösser, F. Serrano, Y. Shinano, B. Sofranac, M. Turner, S. Vigerske, F. Wegscheider, P. Wellner, D. Weninger, and J. Witzig. Enabling research through the SCIP Optimization Suite 8.0. ACMTransactions on Mathematical Software , 2023. accepted for publication.

R. Bixby and E. Rothberg. Progress in computational mixed integer programming-a look back from the other side of the tipping point. Annals of Operations Research , 149(1):37-41, 2007.

M. Boccia. Using exact knapsack separation for the single-source capacitated facility location problem. tech. rep., Department of Engineering, University of Sannio , 2006.

A. E. Boyd. A pseudopolynomial network flow formulation for exact knapsack separation. Networks , 22(5):503-514, 1992.

E. A. Boyd. Generating Fenchel cutting planes for knapsack polyhedra. SIAM Journal on Optimization , 3(4):734-750, 1993.

V. Boyer, D. El Baz, and M. Elkihel. Solving knapsack problems on GPU. Computers & Operations Research , 39(1):42-47, 2012.

G. Braun, A. Carderera, C. W. Combettes, H. Hassani, A. Karbasi, A. Mokthari, and S. Pokutta. Conditional gradient methods. Preprint, arXiv, 2022. https://arxiv.org/abs/2211. 14103 .

G. Braun and S. Pokutta. Dual Prices for Frank-Wolfe Algorithms. Preprint, arXiv, 2021. https://arxiv.org/abs/2101.02087 .

G. Braun, S. Pokutta, and D. Zink. Lazifying conditional gradient algorithms. In Proceedings of the 34th International Conference on Machine Learning , pages 566-575, 2017.

G. Braun, S. Pokutta, and D. Zink. Lazifying conditional gradient algorithms. Journal of Machine Learning Research (JMLR) , 20(71):1-42, 2019.

C. Buchheim, F. Liers, and M. Oswald. Local cuts revisited. Operations Research Letters , 36(4):430-433, 2008.

C. Buchheim, F. Liers, and M. Oswald. Speeding up IP-based algorithms for constrained quadratic 0-1 optimization. Mathematical Programming , 124(1):513-535, 2010.

A. Caprara and M. Fischetti. {0, 1/2}-Chvátal-Gomory cuts. Mathematical Programming , 74(3):221-235, 1996.

A. Chmiela, E. B. Khalil, A. Gleixner, A. Lodi, and S. Pokutta. Learning to schedule heuristics in branch and bound. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

P. Chu and J. E. Beasley. A genetic algorithm for the multidimensional knapsack problem. Journal of Heuristics , 4:63-86, 1998.

V. Chvátal, W. Cook, and D. Espinoza. Local cuts for mixed-integer programming. Mathematical Programming Computation , 5(2):171-200, 2013.

H. P. Crowder, E. L. Johnson, and M. W. Padberg. Solving large-scale zero-one linear programming problems. Oper. Res. , 31:803-834, 1983.

H. Dai, E. B. Khalil, Y. Zhang, B. Dilkina, and L. Song. Learning combinatorial optimization algorithms over graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6351-6361, Red Hook, NY, USA, 2017. Curran Associates Inc.

J.-Y. Ding, C. Zhang, L. Shen, S. Li, B. Wang, Y. Xu, and L. Song. Accelerating primal solution findings for mixed integer programs based on solution prediction. Proceedings of the AAAI Conference on Artificial Intelligence , 34(02):1452-1459, Apr. 2020.

M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly , 3(1-2):95-110, 1956.

A. Fréville. The multidimensional 0-1 knapsack problem: An overview. European Journal of Operational Research , 155(1):1-21, 2004.

M. Gasse, D. Chételat, N. Ferroni, L. Charlin, and A. Lodi. Exact Combinatorial Optimization with Graph Convolutional Neural Networks . Curran Associates Inc., Red Hook, NY, USA, 2019.

R. Gomory. Outline of an algorithm for integer solutions to linear programs. Bulletin of the American Society , 64:275-278, 1958.

R. Gomory. An algorithm for the mixed integer problem. Technical Report RM-2597, The RAND Cooperation, 1960.

R. Gomory. Solving linear programming problems in integers. Combinatorial Analysis, R. Bellman and J. M. Hall, eds., Symposia in Applied Mathematics X, Providence, RI, 1960, American Mathematical Society , 10:211--215, 1960.

M. Grötschel, M. Jünger, and G. Reinelt. Facets of the linear ordering polytope. Mathematical Programming , 33:43-60, 1985.

M. Grötschel, L. Lovász, and A. Schrijver. Geometric Algorithms and Combinatorial Optimization . Springer, 1988.

H. Gu. Local cuts for 0-1 multidimensional knapsack problems. In R. Sarker, H. A. Abbass, S. Dunstall, P. Kilby, R. Davis, and L. Young, editors, Data and Decision Sciences in Action , pages 81-89, Cham, 2018. Springer.

Z. Gu, G. L. Nemhauser, and M. W. Savelsbergh. Sequence Independent Lifting in Mixed Integer Programming. Journal of Combinatorial Optimization , 4(1):109-129, 2000.

Z. Gu, G. L. Nemhauser, and M. W. P. Savelsbergh. Lifted flow cover inequalities for mixed 0-1 integer programs. Mathematical Programming , 85(3):439-467, 1999.

J. A. J. Hall. Towards a practical parallelisation of the simplex method. Computational Management Science , 7(2):139-170, 2010.

H. He, H. Daume III, and J. M. Eisner. Learning to search in branch and bound algorithms. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

G. Hendel. Adaptive large neighborhood search for mixed integer programming. Mathematical Programming Computation , 14(2):185-221, Nov. 2021.

C. Hojny, T. Gally, O. Habeck, H. Lüthen, F. Matter, M. E. Pfetsch, and A. Schmitt. Knapsack polytopes: a survey. Annals of Operations Research , 292(1):469-517, 2020.

A. Hottung and K. Tierney. Neural large neighborhood search for routing problems. Artificial Intelligence , 313:103786, 2022.

Q. Huangfu and J. A. J. Hall. Parallelizing the dual revised simplex method. Mathematical Programming Computation , 10(1):119-142, 2018.

M. Jaggi. Revisiting Frank-Wolfe: projection-free sparse convex optimization. In Proceedings of the 30th International Conference on Machine Learning , pages 427-435, 2013.

K. Kaparis and A. N. Letchford. Separation algorithms for 0-1 knapsack polytopes. Mathematical Programming , 124(1):69-91, 2010.

E. Khalil, P. L. Bodic, L. Song, G. Nemhauser, and B. Dilkina. Learning to branch in mixed integer programming. Proceedings of the AAAI Conference on Artificial Intelligence , 30(1), Feb. 2016.

E. B. Khalil, B. Dilkina, G. L. Nemhauser, S. Ahmed, and Y. Shao. Learning to run heuristics in tree search. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence . International Joint Conferences on Artificial Intelligence Organization, Aug. 2017.

T. Koch, A. Martin, and M. E. Pfetsch. Progress in academic computational integer programming. In M. Jünger and G. Reinelt, editors, Facets of Combinatorial Optimization: Festschrift for Martin Grötschel , pages 483-506. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.

W. Kool, H. van Hoof, and M. Welling. Attention, learn to solve routing problems! In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

S. Lacoste-Julien and M. Jaggi. On the global linear convergence of Frank-Wolfe optimization variants. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28 , pages 496-504. Curran Associates, 2015.

E. S. Levitin and B. T. Polyak. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics , 6(5):1-50, 1966.

D. Liu, A. Lodi, and M. Tanneau. Learning chordal extensions. Journal of Global Optimization , 81(1):3-22, Jan. 2021.

H. Marchand and L. A. Wolsey. Aggregation and Mixed Integer Rounding to Solve MIPs. Operations Research , 49(3):363-371, June 2001.

M. Nazari, A. Oroojlooy, L. Snyder, and M. Takac. Reinforcement learning for solving the vehicle routing problem. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

M. E. Pfetsch, G. Rinaldi, and P. Ventura. Optimal patchings for consecutive ones matrices. Mathematical Programming Computation , 14(1):43-84, 2022.

M. Posta, J. A. Ferland, and P. Michelon. An exact method with variable fixing for solving the generalized assignment problem. Computational Optimization and Applications , 52(3):629-644, 2012.

J. Song, r. lanka, Y. Yue, and B. Dilkina. A general large neighborhood search framework for solving integer linear programs. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 20012-20023. Curran Associates, Inc., 2020.

N. Sonnerat, P. Wang, I. Ktena, S. Bartunov, and V. Nair. Learning a large neighborhood search algorithm for mixed integer programs, 2021.

Y. Tang, S. Agrawal, and Y. Faenza. Reinforcement learning for integer programming: Learning to cut. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 9367-9376. PMLR, 13-18 Jul 2020.

P. Toth. Dynamic programming algorithms for the zero-one knapsack problem. Computing , 25(1):29-45, 1980.

I. L. Vasil'ev. A cutting plane method for knapsack polytope. Journal of Computer and Systems Sciences International , 48(1):70-77, 2009.

I. Vasilyev, M. Boccia, and S. Hanafi. An implementation of exact knapsack separation. Journal of Global Optimization , 66(1):127-150, 2016.

Appendix A

We ran two additional experiments. In the first, we try to repeat the experiments about the strength of local cuts in the root node from [47]. As a comparison, we use the implementation of the local cuts with the LP-based approach of [57] (see the beginning of Section 3 for an overview of the method). In the second experiment, we demonstrate what happens if we use local cuts in a branch-and-cut framework to solve multi-dimensional knapsack problems to optimality.

We start with a comparison of the gap-closed, which is defined as 100 -100 p -d r p -d lp , where p is the optimal primal value, d r is the dual bound at the end of the root node, and d lp is the dual bound of the first LP at the root node. Table 2 shows the results. We use the same multi-dimensional knapsack instances as Kaparis and Letchford: they were originally generated randomly by Chu and Beasley [24] and are available at http://people.brunel.ac.uk/~mastjjb/jeb/orlib/ mknapinfo.html . The instances are organized in blocks of 10, using n variables, m knapsack constraints, and a parameter α . Column 'KL' shows the gap-closed from [47]. Then the results of using the implementation of the LP-based approach (which is also used in [47]) and our Frank-Wolfe approach are presented. For each approach, we show the average of gap-closed, total running time in seconds, as well as separation time, number of calls, and generated violated cuts by the local cut separation over each instance block (of 10 instances). For both approaches, we turn off all other cuts and strong branching. Moreover, we use settings that allow 1000 rounds of local cuts, 10 000 iterations of the Frank-Wolfe algorithm in the root node, and reduce the effect of cut filtering, i.e., more cuts are added to the LP. In each round we separate local cuts for all knapsack constraints that are available. We also initialize the runs with the optimal value to remove the effects of primal heuristics.

The results show that the LP-based approach achieves similar gap-closed values as Kaparis and Letchford. One explanation for the differences is that the final gap depends on the particular points to be separated (but note that we do not generate rank-2 cuts). In comparison, the Frank-Wolfe approach is much faster, but also produces a smaller gap-closed. There are again several reasons for the differences: We limit the number of Frank-Wolfe iterations to 10 000 in the root node, which will leave some separation problems to be undecided and the Frank-Wolfe algorithm might fail to converge (for instance, for n = 500 , m = 5 , α = 75 , on average 34 . 8 Frank-Wolfe runs terminated in the iteration limit and 75 . 4 with the termination criterion explained in Section 2). Moreover, the Frank-Wolfe approach does not necessarily produce a facet, which can weaken the bounds.

Table 3 shows results for running a complete branch-and-cut with variants lc0-nc-lifting and lc1-nc-lifting . As a comparison, we use the default settings of SCIP, but reduce cut filtering (this produces slightly better results for these instances). The results show that for smaller instances,

there is no clear advantage of the Frank-Wolfe approach, but for larger sizes, it solves more instances and is faster.

Peer reviews:
comments: SUMMARY:
The paper proposes an efficient algorithm for learning local cuts used in integer programming relaxations. The algorithm uses a variant of the Frank-Wolfe algorithm and employs a stopping criterion for reducing the number of iterations during the application of the Frank-Wolfe algorithm. The paper showcases the effectiveness of the proposed algorithm by applying it to the multidimensional knapsack problem and reports its findings. 

I have read the authors' rebuttal.

STRENGTHS:
* The paper contributes to solving the generic IP problem efficiently.
* The paper includes experimental results showing the usefulness of the proposed algorithm.

WEAKNESSES:
The paper lack a discussion about the relevant of the studied problem to the NeurIPS community. I am not sure how much the results would appeal to the broad NeurIPS audience.

QUESTIONS:
Please include a discussion about the relevance of the problem to NeurIPS audience. I appreciate the theoretical work, but I am not sure about its relevance to the broad  NeurIPS audience. At least this warrants further discussion and motivation, which are currently lacking from the paper.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for generated lifted cuts from a smaller projected polytope that replaces the usually-expensive LP/IP-based separation routines with an optimization-based method that uses the Frank-Wolfe algorithm. The authors present theory showing that the proposed separation routine indeed produces strong cuts. They then validate their methods via a case study on the multidimensional knapsack problems and variants.\n\nSTRENGTHS:\nThe proposed method is novel and is a genuinely new and interesting algorithmic development in cutting plane generation. The paper is written clearly and precisely. The need for faster separation routines that do not rely on large LPs (or even MIPs) requiring expensive column generation routines is clear, and the authors make interesting progress towards a solution.\n\nWEAKNESSES:\nI think the Knapsack case study could’ve used some more details, including comparisons to existing methods of lifting in this domain. E.g., how does the authors’ Frank-Wolfe based method compare to the usual pipelines of lifting for knapsack inequalities – in particular sequential up-lifting/down-lifting for minimal cover inequalities? Also, how does the authors’ separation procedure itself compare to the usual separation routines for cover inequalities (e.g., those in “Lifted Cover Inequalities for 0-1 Integer Programs: Computation” by Gu, Nemhauser, and Savelsbergh)? Does it produce similar cuts? Stronger ones? Weaker ones? Is it faster or slower? Knapsack constraints are one of the main areas where lifting really shines since it can be done fairly efficiently, so some more comparison here would be super interesting. But, I still view this paper as providing a nice contribution without the deeper dive into lifting knapsack cover inequalities.\n\nA comparison to the equivalent local cuts implementation that SCIP uses seems important to include, but maybe SCIP doesn’t do anything like this? Tangentially, to what extent are local cuts actually used by general-purpose MIP solvers?\n\nQUESTIONS:\nMethodologically, how does the (high-level) approach differ from lift-and-project? Some brief discussion of the similarities/differences would be nice to include.\n\nDo the authors think this approach would scale beyond the kinds of multi-dimensional knapsack/GAP problems studied? Those problems conveniently have efficient lifting methods, but for general MIP lifting can be expensive (requiring solutions to MIPs). (I do understand that lifting is not the focus of this paper.)\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for generating separating cutting planes for integer programming problems that is based on the Frank-Wolfe method for optimizing over polyhedron. It falls roughly within the existing local/Fenchel cut framework: given a point, it identifies the closest point within the feasible region by repeatedly calling a (hopefully fast) optimization oracle, and then\n\nSTRENGTHS:\nOverall I quite like the paper and how it stitches together some disparate, existing ideas from the optimization literature.\n\nThe paper is generally very well-written. The paper is in an area of clear interest to the Operations Research and Mathematical Optimization community, and the subject matter is consistent with a number of other papers accepted to NeurIPS in the past. The application of Frank-Wolfe in this manner is interesting and non-trivial, as the authors apply some extensions to improve its practical performance (the \"FW Gap\" characterization being chief among them).\n\nWEAKNESSES:\nThe results in the computational study are slightly underwhelming. A 31% improvement in solving times is nice, but this only considers \"solved\" instances and the geo-mean solve times here are quite small to begin with (6s for \"default\"). The provided solve times on all instances (including ones that terminate without proving optimality) show a significantly smaller (though still positive) speedup. Given that MIP solvers such as SCIP tend to be engineered more for hard instances than easy ones, it is difficult to know how much to read into these results without additional detail.\n\nQUESTIONS:\n* L19: \"However, the by far most interesting case is the one we consider here\" (IP vs. MIP). This is a subjective statement that I imagine many researchers in the MIP community would (strongly!) disagree with. I'd suggest softening or removing, as this is not really necessary to justify the restriction to IP for the purposes of the paper.\n* L30: \"first p indices\" seems like it is left over from an earlier draft that considered MIP.\n* The first paragraph of Section 2 is fairly confusing, largely because you introduce two polyhedron (P and tilde{P}), but really only work with the second. As such, statements about \"valid cut\"s and \"original space\" are unclear without further qualification. I'd suggest rewriting it, and maybe adding some discussion about how P and tilde{P} should or do relate to each other: \"subproblem\" suggests some relationship (maybe containment in a projected space?), but is vague.\n* L101: You do not assume that tilde{P} is full-dimensional (just P, but see comment above), so there may not exist an interior point. \n* Figure 1: The figure and caption are a bit confusing: it's not clear what the tilde{P}_I label refers to; indices t and k are used variously in the caption and figure; there is no explicit reference to (c) in the caption, etc.\n* L151: Is the oracle required to return a solution vertex? If so, you should state that above. If not, does this affect any of the statements to follow?\n* L317: It is a bit confusing to the reader that you state a percentage solve time improvement in an unqualified manner, when it only applies to a subset of the instances in your test bed (the solved ones). Please be more explicit about this in the test (here, and also earlier in the contributions section).\n* Section: To bolster the takeaways from the computational study, it would be interesting to have more information about the relative performance on the unsolved instances. For instance: How many instances does each method solve to optimality within the time budget? What do the dual and primal/dual integrals look like for the various methods? How long does \"default\" SCIP spend in cut separation? A separate suggestion to isolate the dual effect of the new method would be to seed each method with the best known primal solution as a warm start.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* Context:\nThis paper deals with the subject of generating cuts for solving Integer Programs (MIP).\nRather than using a cut generating algorithm based on a formula (like Gomory cuts) to create hyperplanes separating the solution of the relaxed problem from the (integer) feasible domain, the papers attempts to generate \"Local cuts\".\nLocal cuts are cuts that are obtained by generating a cut in a subproblem with reduced dimension (so that any optimization problem required to generate the cut are easier to solve), and then, taking the cut for the subproblem and transforming it into a cut for the actual problem to solve.\n\n* Contribution:\nThis paper presents a local cuts generating algorithm, based on the Frank-Wolfe / conditional gradient algorithm.\nFrank-Wolfe is used to find the point in the subproblem polyhedron $\\tilde{P}$ that is the closest to the point to be separated $\\tilde{x}$, as this point will be likely to be on a facet.\nThe Frank-Wolfe algorithm has the advantage that it only require linear optimization oracle access over the subproblem to arrive at a solution, which can often be implemented efficiently.\nThe paper also demonstrates that using duality information, the Frank-Wolfe algorithm does not need to be run all the way to convergence and that the strength of the cut generated by intermediate iterates can be evaluated, leading to a speed-up.\n\nThe impact of using these cuts is evaluated on the multi-dimensional knapsack problem, by including the local cut generating algorithm into the open source SCIP solver.\n\nSTRENGTHS:\nThe paper presents a algorithmic improvement for solving an important problem, and does it in a clear fashion.\nIt is clearly delineated what exists in the current literature and what constitutes an actual contribution of the work.\nExplanations and motivations are straightforward to follow.\n\nWEAKNESSES:\n* From reading the paper, what is not obvious is how to take the contribution of this paper, and apply it to a different problem. After reading the paper, I understand how to apply this to a knapsack problem, but it's not obvious to me how I would do it if I had a more arbitrary MIP (for example one for verifying neural networks), or even a different type of problem solvable by integer programs (let's say a TSP). What would be beneficial would be to have somewhere clearly \"These are the requirements that you need for your problem to be solvable using this method\"\n\n* I think that what is missing is some appropriate baseline from the evaluation. As it is, the contribution of the paper seems to me to be \"here is an efficient way to do local cuts\", while the only baseline that is compared to is default SCIP. Is the performance difference that is seen due to using local cuts or using the local cuts presented in this paper? (I think that Table 3 in the appendix might have some results that are relevant to this, but this would benefit from being moved to the main part of the paper.)\n\nQUESTIONS:\n* [Section 2] Could you clarify the relation between the polytope $P \\subseteq \\mathbb{R}^n$ and the polyhedron $\\tilde{P} \\subseteq \\mathbb{R}^k$ ? Is it always a restriction (a subset of the variables in P, such that an element is in $\\tilde{P}$ if there exists an element in $P$ with those values?) or could it be something more general ( like an affine transformation of $\\tilde{P}$ ?) and have as only requirement \"if we have a cut in $\\tilde{P}$, we can find a cut in $P$? . As it is, it sounds quite abstract.\n\n* [Section 5] Downlifting vs up- and down lifting is not explained anywhere. Given that downlifting performs significantly worse on all benchmarks, it seems that it might not be worth it to include it in the main version of the paper to avoid confusion.\nSimilarly, CMIR seems to have very little impact so unless it can be explained, it might make sense to move it to the appendix.\n* Would it be possible to include some more information on what the default settings of SCIP looks like? Is it a reasonable baseline? Is there an indication on how many cuts (whether local or not local) it corresponds to?\n\nTypos / small comments:\n- Line 30, I don't understand what the notion of the first $p$ indices is. Is that a reference to some Mixed Integer Programming case, where only the first $p$ variables need to be integral?\n- Figure 1 legend, is it supposed to be the L_2 projection (rather than L_t) ?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

comments: SUMMARY:
The paper studies cut generation from operations research. While this is potentially a promising approach, I believe the paper falls out of scope of the conference. Optimization can be seen as part of machine learning machiery, but normally NeurIPS papers are expected to have some connection to machine learning, e.g., at the very least, the experiments could have been done on some machine learning problem. It is not common to reject a paper based on out of scope, however in this case the paper is purely operations research. I believe submiting to an operations research venue would be more appropriate, and it is likely that the impact of the work would be greater in such a community.

STRENGTHS:
Did not look in detail.

WEAKNESSES:
Did not look in detail.

QUESTIONS:
What is the motivation for submiting this paper to NeurIPS?

LIMITATIONS:
Did not look in detail.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies integer programming (IP) by proposing an alternative cutting-plane approach that makes use of the Frank-Wolfe (FW) algorithm for the separation sub-routine (i.e., separation of the target from the feasible set). The main idea is to use “local cuts” which aims at deriving the facets of $P_I$, i.e., the integer hull of the feasible set, $P$, of the relaxed linear program (LP). \n\nFacets of $P_I$ are the strongest cuts but they are unknown and finding the facets could be computationally expensive. The authors propose a separation procedure which does not need to solve an LP, and instead uses away-step FW variant to (approximately) solve the separation problem. \n\nTo demonstrate the performance of their approach, the authors compare their FW-based approach with the default SCIP solver, an optimization framework for solving mixed-integer programs, against solving the multi-dimensional knapsack problem. They also describe the implementation of the linear minimization oracle (LMO) of the FW algorithm using dynamic programming for the application at hand.\n\nSTRENGTHS:\n-\tAuthors provide an alternative to the cutting-plane approaches by simplified local cuts. Instead of exactly solving an LP, the authors make use of FW algorithm and dynamic programming to find facets of the underlying polyhedron in lower dimensions. They use known lifting techniques to obtain the full-dimensional cut.\n\n\n-\tThe stopping criterion for the algorithm is derived via fundamental properties of the FW algorithm and LMOs. Using a fundamental, technical intuition, they improve the performance of the separation routine by an additional stopping criterion.\n\n\n-\tThey compare their approach with SCIP with default settings as a benchmark and provide improved run times.\n\nWEAKNESSES:\n-\tThe fundamental differences and improvements with respect to existing cutting-plane methods to solve IP problem (specifically MKP as the authors provide it as an example) are not clarified. Computational/complexity improvements with respect to LP-based methods are not studied in technical details.\n\n\n-\tThe method is not comprehensively compared against (in practice) to existing methods for solving IPs, specifically MKPs. The authors provide a comparison between LP-based separation as implemented by [42] and the FW approach in the appendix. The FW approach is faster in run time but the gap closed is less than the LP-based approach. I am not sure whether it is enough to claim a clear advantage for the proposed algorithm. However, we should note that Table 3 shows promising results for the FW algorithm.\n\n\n-\tSince this paper does not propose a new theory and analysis, I don’t find the experimental evaluation comprehensive enough. The authors should have considered other problems, such as TSP, while including existing algorithms in their runs.\n\nQUESTIONS:\n-\tHow do you set $\\epsilon$ in your algorithm?\n-\tHave you compared your algorithm numerically with other proposed methods for solving MKP or other popular IP problem (other than the results in the appendices)?\n-\tHave you run experiments for other problem instances?\n-\tI am curious about the possible shortcomings of the FW approach. The linear minimization oracle for the FW method uses dynamic programming, which introduces an overhead for the computation of $v_t$. Is there any regime where the FW approach with the proposed LMO oracle becomes computationally expensive such that the run times are comparable to LP-based approaches?\n\nLIMITATIONS:\n-\tThe authors might need to consider more problems like MKP to validate the performance of their approach.\n-\tI would prefer to see several other methods to be included in the numerical tests.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}