SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daniel Thuerck
Quantagonia
Bad Homburg, Germany
daniel.thuerck@quantagonia.com
Boro Sofranac
boro.sofranac@quantagonia.com
Marc E. Pfetsch
Department of Mathematics, TU Darmstadt
pfetsch@mathematik.tu-darmstadt.de
Sebastian Pokutta
Zuse Institute Berlin and TU Berlin
pokutta@zib.de
Cutting-planes are one of the most important building blocks for solving large-scale integer programming (IP) problems to (near) optimality. The majority of cutting plane approaches rely on explicit rules to derive valid inequalities that can separate the target point from the feasible set. Local cuts, on the other hand, seek to directly derive the facets of the underlying polyhedron and use them as cutting planes. However, current approaches rely on solving Linear Programming (LP) problems in order to derive such a hyperplane. In this paper, we present a novel generic approach for learning the facets of the underlying polyhedron by accessing it implicitly via an enumeration oracle in a reduced dimension. This is achieved by embedding the oracle in a variant of the Frank-Wolfe algorithm which is capable of generating strong cutting planes, effectively turning the enumeration oracle into a separation oracle. We demonstrate the effectiveness of our approach with a case study targeting the multidimensional knapsack problem (MKP).
In this paper, we deal with integer programs (IP)
$$
\max  \left\{\langle c, x \rangle : A x \leq b, x \in \mathbb {Z} ^ {n} \right\}, \tag {IP}
$$
where  $A \in \mathbb{Q}^{m \times n}$ ,  $b \in \mathbb{Q}^m$ , and  $c \in \mathbb{Q}^n$ . Let  $P \coloneqq \{x \in \mathbb{R}^n : Ax \leq b\}$  be the underlying polyhedron and its integer hull  $P_I \coloneqq \operatorname{conv}(P \cap \mathbb{Z}^n)$ . We restrict attention to the case in which all variables are required to be integral, as the methods we will propose are more readily applicable to this case, but the general idea works for mixed-integer programs (MIP) as well.
Solving IPs is  $\mathcal{NP}$ -hard in general, however, surprisingly fast algorithms exist in practice [1, 50]. The most successful approach to solving IPs is based on the branch-and-bound algorithm and its extensions. This algorithm involves breaking down the original problem into smaller subproblems that are easier to solve through a process known as branching. By repeatedly branching on subproblems, a search tree is obtained. The bounding step involves computing upper bounds for subproblems and pruning suboptimal nodes of the tree in order to avoid enumerating exponentially many subproblems. Upper bounds are generally computed with the help of Linear Programming (LP) relaxations
$$
\max  \left\{\langle c, x \rangle : x \in P \right\}. \tag {LP}
$$
Because the integrality constraints are relaxed, optimal solutions of (LP) provide an upper bound for the original problem (IP).
Alternatively, cutting plane procedures iteratively solve LP-relaxations as long as the solution  $x^{*}$  is not integral (and thus  $x^{*} \notin P_{I}$ ). To remove these solutions  $x^{*}$  from the relaxation's polyhedron, one adds cutting planes (or cuts)  $\langle \alpha, x \rangle \leq \beta$  with  $a \in \mathbb{Q}^n$ ,  $\beta \in \mathbb{Q}$ , and  $\langle \alpha, x^{*} \rangle > \beta$ . The search for such cutting planes or to determine that none exists is called the separation problem. The strongest cuts are those that define a facet, i.e., the face  $P_{I} \cap \{x : \langle \alpha, x \rangle = \beta\}$  has co-dimension 1 with respect to  $P_{I}$ . When the cutting plane method is combined with branch-and-bound, the resulting algorithm is often called branch-and-cut. Gomory conducted foundational work in this field, demonstrating that pure cutting plane approaches can solve integer programs with rational data in a finite number of steps without the need for branching [32, 33, 34].
Gomory's initial approach to cutting planes suffered from numerical difficulties at that time, preventing pure cutting plane methods from being effective in practical applications. However, his proposed (Gomory) mixed integer (GMI) cuts are very efficient if combined with branch-and-bound (see the computational study in [11]) and still are one of the most important types of cutting planes used by contemporary solvers. As more GMI cuts are added to a problem, their incremental value tends to diminish. To address this issue, modern MIP and IP solvers use a range of techniques to generate cuts, e.g., mixed-integer-rounding (MIR) inequalities [55], knapsack covers [26, 38], flow covers [39], lift-and-project cuts [5],  $\{0,\frac{1}{2}\}$ -Chvátal-Gomory cuts [22], and others.
Most cutting plane separation algorithms rely on fixed formulas to derive valid inequalities that separate the target point  $x^{*}$  from the polyhedron  $P_{I}$ . An alternative approach is to directly seek to derive the facets of  $P_{I}$  that separate the point  $x^{*}$ . Notice that while the facets of the polyhedron  $P$  are explicitly known from the problem definition, the facets of its integer hull  $P_{I}$  are unknown in general. While the facet-defining inequalities are intuitively the strongest cuts, they can be relatively expensive to explicitly compute, limiting their applicability in practice. Local cuts, a type of cutting planes that try to derive facets of  $P_{I}$ , approach this problem by deriving facets of  $P_{I}$  in a reduced dimension, and then lifting those cuts to obtain facets in the original dimension. In this paper, we will propose a new variant of the Frank-Wolfe algorithm with the goal of learning the (unknown) facets of  $P_{I}$  (or at least valid inequalities) in a reduced dimension, which can then be lifted to the original dimension and be used as strong cutting planes. In our learning approach, the underlying polyhedron will only be accessed via an algorithmically simple linear optimization oracle, in contrast to existing approaches, which also need to solve LPs.
Local cuts have first been introduced as "Fenchel cuts" in Boyd [13, 14], who developed an algorithm to exactly separate inequalities for the knapsack polytope via the equivalence of separation and optimization. They were subsequently investigated extensively by Applegate et al. [3] for solving the traveling salesman problem (TSP). Buchheim et al. [20, 21] and Althaus et al. [2] adopted local cuts into their approaches for solving constrained quadratic 0-1 optimization problems and Steiner-tree problems, respectively. In [25], Chvátal et al. generalize the local cuts method to general MIP problems.
In the context of knapsack problems, after the aforementioned work of Boyd [13, 14], Boccia [12] introduced an approach based on local cuts, as stated by Kaparis and Letchford  $[47]^1$ , who further refined the algorithm. Vasilyev presented an alternative approach with application to the generalized assignment problem in [63], see also the comprehensive computational study conducted by Avella et al. in [4]. In [64], Vasilyev et al. propose a new implementation of this approach, with the goal of making it more efficient. In [37], Gu presents an extension of the algorithm of Vasilyev et al. [64].
Existing works on the application of learning methods in solving IP (and more generally MIP) problems can in general be divided into two categories: learning decision strategies within the solvers, and learning heuristics to obtain feasible (primal) solutions. Examples of the former would be learning to select branching variables [6, 31, 48], learning to select branching nodes [41], learning to select cutting planes [61], learning to optimize the usage of primal heuristics [23, 42, 49]. A typical example of the latter case would be learning methods to develop large neighborhood search (LNS) heuristics [28, 60, 59]. Additionally, a number of works in the literature have focused on learning algorithms for solving specific IP problems [8, 44, 27, 51, 54, 56]. For a more detailed overview of using learning methods in IP, we refer the interested reader to [9].
<sup>1</sup>We could not independently verify the claim as we could not access Boccia's paper online.
1.2 Contribution
The contributions of this paper can be summarized as follows:
1. We present an efficient, LP-free separation framework that aims to learn local cuts for IPs through the solution of subproblems. We propose to use a variant of the Frank-Wolfe [29] algorithm to solve the associated separation problem. The resulting framework is general and – given the availability of a suitable lifting method – applicable to any IP.
2. We propose a new, dynamic stopping criterion for the application of Frank-Wolfe to the separation problem at hand. This new criterion, derived by exploiting duality information, directly evaluates the strength of the resulting cut and thus dramatically decreases the number of iterations.
3. We illustrate the benefit of our approach in a case study for the multidimensional knapsack (MKP) problem, demonstrating its effectiveness. Our computational results show that embedding our method in the academic solver SCIP leads to  $31\%$  faster solving times on the instances solved to optimality, on average.
The rest of this paper is organized as follows: In Section 2, the fundamental framework of local cuts and required notation are introduced. Section 3 presents our approach for the LP-less generation procedure for local cuts. Section 4 demonstrates how the aforementioned framework can be applied to solve the multidimensional knapsack problem. Computational experiments are presented in Section 5. Finally, Section 6 summarizes conclusions and future work.
2 Local Cuts
To describe the idea of local cuts, assume that  $P \subset \mathbb{R}^n$ ,  $n > 0$ , is a polytope, i.e., bounded, and full-dimensional. Then one considers a small subproblem with underlying polytope  $\tilde{P}$ , which is usually an orthogonal projection of  $P$  onto a lower-dimensional space. The polytope  $\tilde{P}$  is restricted to being non-empty and its dimension  $0 < k \leq n$  is chosen small enough such that integer optimization problems over  $\tilde{P}$  can be solved efficiently in practice, for example, by enumeration. Consider a projection  $\tilde{x}$  of the point to be separated  $x^*$  on  $\mathbb{R}^k$ . The procedure tries to generate a valid cut  $\langle \tilde{\alpha}, x \rangle \leq \tilde{\beta}$  with  $\tilde{\alpha} \in \mathbb{Q}^k$ ,  $\tilde{\beta} \in \mathbb{Q}$ , such that  $\langle \tilde{\alpha}, \tilde{x} \rangle > \tilde{\beta}$ , i.e., it cuts off  $\tilde{x}$  from  $\tilde{P}$ . This cut can be "lifted" to the original space, which yields a cut  $\langle \alpha, x \rangle \leq \beta$  that hopefully cuts off  $x^*$ .<sup>2</sup>
The approach to generate  $\langle \tilde{\alpha}, x \rangle \leq \tilde{\beta}$ , in the literature mentioned above, relies on the equivalence between optimization and separation [36] and can be very briefly explained as follows. By the Minkowski-Weyl Theorem, we can express  $\tilde{P}$  as the convex hull of its vertex set  $V$ . Let  $\tilde{x}^0 \in \tilde{P}$  be an interior point. Then consider the LP
$$
\min  _ {\lambda , \gamma} \left\{\gamma : \sum_ {v \in V} v \lambda_ {v} + (\tilde {x} - \tilde {x} ^ {0}) \gamma = \tilde {x}, \sum_ {v \in V} \lambda_ {v} = 1, \lambda \geq 0 \right\}.
$$
The dual problem  $(D)$  is
$$
\max _ {\alpha , \beta} \left\{\langle \tilde {x}, \alpha \rangle - \beta : \langle v, \alpha \rangle \leq \beta \forall v \in V, \langle \tilde {x} - \tilde {x} ^ {0}, \alpha \rangle \leq 1 \right\}.
$$
Let  $\tilde{\alpha},\tilde{\beta}$  be an optimal solution of  $(D)$ . Then  $\langle \tilde{\alpha},x\rangle \leq \tilde{\beta}$  is a valid inequality for  $\tilde{P}$ , since by construction  $\langle v,\tilde{\alpha}\rangle \leq \tilde{\beta}$  holds for all  $v\in V$  and thus by convexity for all points in  $\tilde{P}$ . The objective enforces that this cut is maximally violated by  $\tilde{x}$  if the optimal value is positive.
Since  $\tilde{P}$  may have an exponential number of vertices, problem  $(D)$  can be solved by a column generation algorithm (or cutting plane algorithm in the primal). In each iteration, one needs to solve the following pricing problem for the current point  $(\hat{\alpha},\hat{\beta})$ : Decide whether there exists  $v\in V$  with  $\langle v,\hat{\alpha}\rangle >\hat{\beta}$ . This can be done by maximizing  $\hat{\alpha}$  over the subproblem  $\tilde{P}$ , i.e., one can use a linear optimization oracle for the subproblem. This subproblem can contain integrality constraints, thereby requiring, again, IP techniques. Note that the most interesting case is where we operate on integer
2The idea of local cuts is often confused with lift-and-project cuts, as the two methods share some high-level ideas, such as exploring solutions in a different space and using projection. However, they are quite different. Lift-and- project methods first lift, then generate a cut and project it back, while local cuts first project and then lift the cut. Moreover, the subproblems to generate cuts are significantly different.
hulls, i.e.  $\tilde{P} = \tilde{P}_I$  to generate cuts for  $P = P_I$ . In this way, local cuts can help solving an integer optimization problem over  $P$ . Hence, in the following sections, any reference to  $P$ ,  $\tilde{P}$  holds for the integer case as well and our case study illustrates exactly that.
As mentioned above, the strongest cutting planes are those that define facets. The tilting method by Applegate et al. [3] produces such a facet. Buchheim et al. [20] introduced a different formulation that automatically produces a facet. Chvátal et al. [25] developed a formulation for general MIPs using linear optimization oracles. All three approaches use a sequence LPs at their heart; either for tilting a plane or through a column-generation procedure.
3 Learning Strong Cuts from Enumeration
The local cuts framework, applicable to general IPs, relies on a sequence of three operators: SEP, FACET and LIFT. SEP refers to a separation oracle separating the projected point  $\tilde{x}$  from  $\tilde{P}$  that returns a separating cut  $\langle \tilde{\alpha}, x \rangle \leq \tilde{\beta}$  (or certifies that  $\tilde{x} \in \tilde{P}$ ). FACET further refines the cut until it represents a facet of  $\tilde{P}$  and lastly, LIFT transforms the resulting facet into the space of  $P$  such that it separates  $x^{*}$  from  $P$  with high probability. In some variants of local cuts, SEP and FACET may be combined into one step similar to [20], whereas in [25], the tilting process is a separate, concrete embodiment of FACET. Note that that facets of the subproblem, when lifted, result in the strongest cuts. In practice, it is often sufficient to find good valid inequalities of  $\tilde{P}$ . As mentioned before, the original approach for local cuts through duality requires an expensive column-generation method which is based on LPs. In this section, we derive an alternative and LP-less approach.
The general idea of our new approach is sketched in Figure 1: Given a point  $\tilde{x} \in \mathbb{R}^n$  that we intend to separate from  $P$ , we solve the following optimization problem:
$$
y ^ {*} = \underset {y \in \tilde {P}} {\operatorname {a r g m i n}} f (y), \quad (\text {S e p a r a t i o n})
$$
with  $f(y) \coloneqq \frac{1}{2} \| y - \tilde{x} \|^2$ . Observe that this is effectively the projection of  $\tilde{x}$  onto  $\tilde{P}$  under the  $\ell_2$ -norm and that  $\nabla f(y) = (y - \tilde{x})$ .
We solve (Separation) with a suitable variant of the Frank-Wolfe algorithm. The Frank-Wolfe algorithm [29] (also called: Conditional Gradients [53]) is a method to minimize a smooth convex function  $f$  over a compact convex domain  $P$  by only relying on a First-order Oracle (FO) for  $f$ , i.e., given a point  $x$  the oracle returns  $\nabla f(x)$  (and potentially  $f(x)$ ) as well as a Linear Minimization Oracle (LMO) ("oracle" for the remainder of this paper), i.e., given an objective vector  $c$ , the oracle returns  $v \in \operatorname{argmin}_{x \in \tilde{P}}\langle c, x \rangle$ . The original Frank-Wolfe algorithm, provided with step sizes  $\gamma_t > 0$ , iteratively calls the LMO to determine  $v_t \gets \operatorname{argmin}_{v \in \mathcal{C}}\langle \nabla f(y_t), v \rangle$  and updates the iterate to  $y_{t+1} \gets y_t + \gamma_t(v_t - y_t)$ . There are various step-size strategies for  $\gamma_t$ , but the actual choice is irrelevant for the discussion here; a common choice is  $\gamma_t = \frac{2}{t+2}$ .
The main advantages of using Frank-Wolfe are (1) if there is a LP-less oracle, valid inequalities can be generated without solving LPs, (2) the computational overhead of the Frank-Wolfe steps compared to calls to the LMO are very light and, finally, (3) as we will show, for the case of (Separation), we can derive a new dynamic stopping criterion that can dramatically reduce the number of iterations. Note that we are not guaranteed to end up with facets, especially when the method is stopped early, however, valid inequalities that are "close" to being a facet can still serve as strong cutting planes.
For our problem minimizing  $f$ , the Frank-Wolfe algorithm iteratively calls the oracle and updates its current iterate through a convex combination of the previous iterate and oracle's solution vertex. Step by step, the solution is thus expressed through a convex combination of vertices in  $\tilde{P}$  as shown in Figures 1a - 1c. At convergence, the hyperplane  $\langle \nabla f(y^*), x \rangle \geq \langle \nabla f(y^*), y^* \rangle$  forms the desired cut.
3.1 Separation via Conditional Gradients
Let  $y^{*} \in \tilde{P}$  be an optimal solution to (Separation) and let  $x \in \tilde{P}$  be arbitrary. By convexity, it follows that  $0 \leq f(x) - f(y^{*}) \leq \langle \nabla f(x), x - y^{*} \rangle \leq \max_{v \in \tilde{P}} \langle \nabla f(x), x - v \rangle$  and the last quantity is referred to as Frank-Wolfe gap (at  $x$ ). Moreover, the following lemma holds, which is a direct consequence of the first-order optimality condition.
(b)
Figure 1: We propose the following approach to separate a fractional point  $\tilde{x}$  from a full-dimensional polytope  $\tilde{P}$ : We solve  $\min_{y\in \tilde{P}}f(y)\coloneqq \frac{1}{2}\| y - \tilde{x}\| ^2$ , i.e., the  $L_{2}$  projection of  $\tilde{x}$  onto  $\tilde{P}$ , through a variant of the Frank-Wolfe algorithm. Starting from a random vertex (a), the algorithm iteratively computes the gradient of  $f$  at the current iterate  $y_{k}$  and uses an oracle to solve a linear integer optimization problem over  $\tilde{P}$ , building up an active set of vertices that form iterates through a convex combination (b). At convergence (c) the optimal solution  $y^{*} = y_{k + i}$  together with its gradient forms a cut that induces a facet of  $\tilde{P}$ :  $\nabla f(y_{k + i})^{\top}x\geq f(y_{k + i})^{\top}y_{k + i}$  (except for degenerate cases).
Lemma 1 (First-order Optimality Condition). Let  $y^{*} \in \tilde{P}$ . Then  $y^{*}$  is an optimal solution to  $\min_{y \in \tilde{P}} f(y)$  if and only if  $\langle \nabla f(y^{*}), y^{*} - v \rangle \leq 0$  for all  $v \in \tilde{P}$  (and in particular  $\max_{v \in \tilde{P}} \langle \nabla f(y^{*}), y^{*} - v \rangle = 0$ ).
Note that in the constrained case, it does not necessarily hold that  $\nabla f(y^{*}) = 0$ , if  $y^{*}$  is an optimal solution. In fact, if the  $\tilde{x}$  that we want to separate is not contained in  $\tilde{P}$ , then  $f(y^{*}) > 0$  and  $\nabla f(y^{*}) \neq 0$  since  $y^{*}$  will lie on the boundary of  $\tilde{P}$ .
It turns out that we can naturally use an optimal solution  $y^{*} \in \tilde{P}$  to (Separation) to derive a separating hyperplane. By Lemma 1:
$$
\langle \nabla f (y ^ {*}), y ^ {*} \rangle \leq \langle \nabla f (y ^ {*}), v \rangle , \tag {Cut}
$$
which holds for all  $v \in \tilde{P}$ . Moreover, if  $\tilde{x} \notin \tilde{P}$ , then (Cut) is violated by  $\tilde{x}$ , i.e.,  $\langle \nabla f(y^{\ast}), y^{\ast} \rangle > \langle \nabla f(y^{\ast}), \tilde{x} \rangle$ , since  $\langle \nabla f(y^{\ast}), y^{\ast} - \tilde{x} \rangle \geq f(y^{\ast}) - f(\tilde{x}) = f(y^{\ast}) > 0$ .
Usually, however, we do not solve Problem (Separation) exactly, but rather up to some accuracy. In fact, the Frank-Wolfe algorithm often uses the Frank-Wolfe gap as a stopping criterion, minimizing the function until for some iterate  $y_{t}$  it holds  $\max_{v\in \tilde{P}}\langle \nabla f(y_t),y_t - v\rangle \leq \varepsilon$  for some target accuracy  $\varepsilon$ ; note that the Frank-Wolfe gap converges with the same rate (up to small constant factors) as the primal gap (see e.g., [46]). Given an accuracy  $\varepsilon >0$ , we obtain the valid inequality
$$
\left\langle \nabla f (y _ {t}), y _ {t} \right\rangle - \varepsilon \leq \left\langle \nabla f (y _ {t}), v \right\rangle , \tag {approxCut}
$$
for all  $v\in \tilde{P}$ , which also separates  $\tilde{x}$  from  $\tilde{P}$  if it is  $\sqrt{\varepsilon}$ -far from  $\tilde{P}$ , i.e.,  $\| y^{*} - \tilde{x}\| >\sqrt{\varepsilon}$ :
$$
\langle \nabla f (y _ {t}), y _ {t} - \tilde {x} \rangle - \varepsilon \geq f (y _ {t}) - f (\tilde {x}) - \varepsilon \geq f (y ^ {*}) - \varepsilon > 0.
$$
The accuracy  $\varepsilon$  is chosen depending on the application; see also [17] for a sensitivity analysis for conditional gradients.
3.1.1 A dynamic stopping criterion
It turns out, however, that in our case of interest, the above can be significantly improved by exploiting duality information. This allows us not only to stop the algorithm much earlier, but we also obtain a separating inequality directly from the associated stopping criterion and duality information.
The stopping criterion is derived from a few simple observations, which provide a new characterization of a point  $\tilde{x}$  that can be separated from  $\tilde{P}$ . Our starting point is the following standard expansion. Let  $v \in \tilde{P}$  be arbitrary and let  $y_{t}$  be an iterate from above. Then,
$$
\left\| \tilde {x} - v \right\| ^ {2} = \left\| \tilde {x} - y _ {t} \right\| ^ {2} + \left\| y _ {t} - v \right\| ^ {2} - 2 \langle y _ {t} - \tilde {x}, y _ {t} - v \rangle ,
$$
$$
\langle y _ {t} - \tilde {x}, y _ {t} - v \rangle = \frac {1}{2} \| \tilde {x} - y _ {t} \| ^ {2} + \frac {1}{2} \| y _ {t} - v \| ^ {2} - \frac {1}{2} \| \tilde {x} - v \| ^ {2}. \tag {1}
$$
Observe that the left hand-side is the Frank-Wolfe gap expression at iterate  $y_{t}$  (except for the maximization over  $v \in \tilde{P}$ ) since  $\nabla f(y_{t}) = y_{t} - \tilde{x}$ .
Necessary Condition. Let us first assume  $\| y_t - v \| < \| \tilde{x} - v \|$  for all vertices  $v \in \tilde{P}$  in some iteration  $t$ . Then (1) yields
$$
\left\langle y _ {t} - \tilde {x}, y _ {t} - v \right\rangle <   \frac {1}{2} \| \tilde {x} - y _ {t} \| ^ {2}. \tag {altTest}
$$
If  $v_{t}$  is the Frank-Wolfe vertex in iteration  $t$ , we obtain:
$$
\begin{array}{l} \frac {1}{2} \left\| y _ {t} - \tilde {x} \right\| ^ {2} - \frac {1}{2} \left\| y ^ {*} - \tilde {x} \right\| ^ {2} = f (y _ {t}) - f (y ^ {*}) \\ \leq \max  _ {v \in P} \left\langle \nabla f (y _ {t}), y _ {t} - v \right\rangle = \left\langle \nabla f (y _ {t}), y _ {t} - v _ {t} \right\rangle = \left\langle y _ {t} - \tilde {x}, y _ {t} - v _ {t} \right\rangle <   \frac {1}{2} \| \tilde {x} - y _ {t} \| ^ {2}. \\ \end{array}
$$
Subtracting  $\frac{1}{2}\| \tilde{x} -y_t\|^2$  on both sides and re-arranging yields:  $0 < \frac{1}{2}\| y^{*} - \tilde{x}\|^{2}$ , which proves that  $\tilde{x}\notin \tilde{P}$ . Moreover, (1) also immediately provides a separating hyperplane: observe that (altTest) is actually a linear inequality in  $v$  and it holds for all  $v\in \tilde{P}$  since the maximum is achieved at a vertex. However, for the choice  $v = \tilde{x}$  the inequality is violated.
Sufficient Condition. Suppose that in each iteration  $t$  there exists a vertex  $\bar{v}_t \in \tilde{P}$  (not to be confused with the Frank-Wolfe vertex), so that  $\| y_t - \bar{v}_t \| \geq \| \tilde{x} - \bar{v}_t \|$ . In this case (1) ensures: