SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\langle y _ {t} - \tilde {x}, y _ {t} - \bar {v} _ {t} \right\rangle = \frac {1}{2} \| \tilde {x} - y _ {t} \| ^ {2} + \frac {1}{2} \| y _ {t} - \bar {v} _ {t} \| ^ {2} - \frac {1}{2} \| \tilde {x} - \bar {v} _ {t} \| ^ {2} \geq \frac {1}{2} \| \tilde {x} - y _ {t} \| ^ {2}.
$$
Thus, the Frank-Wolfe gap satisfies in each iteration  $t$  that
$$
\max  _ {v \in \tilde {P}} \left\langle \nabla f (y _ {t}), y _ {t} - v \right\rangle \geq \left\langle y _ {t} - \tilde {x}, y _ {t} - \bar {v} _ {t} \right\rangle \geq \frac {1}{2} \| \tilde {x} - y _ {t} \| ^ {2},
$$
i.e., the Frank-Wolfe gap upper bounds the distance between the current iterate  $y_{t}$  and point  $\tilde{x}$  in each iteration. Now, the Frank-Wolfe gap converges to 0 as the algorithm progresses, with iterates  $y_{t} \in \tilde{P}$ , so that with the usual arguments (compactness and limits etc.) it follows that  $\tilde{x} \in \tilde{P}$ . In total, we obtain the following result.
Characterization 2. The following are equivalent:
1. (Non-Membership)  $\tilde{x} \notin \tilde{P}$ .
2. (Distance) There exists an iteration  $t$ , so that  $\| y_t - v \| < \| \tilde{x} - v \|$  for all vertices  $v \in \tilde{P}$ .
3. (FW Gap) For some iteration  $t$ ,  $\max_{v\in \tilde{P}}\langle y_t - \tilde{x},y_t - v\rangle < \frac{1}{2}\| \tilde{x} -y_t\| ^2$
In particular, Characterization 2.3 can be easily tested within the algorithm, since the Frank-Wolfe gap is computed anyways. Using this criterion significantly improves the performance of the algorithm. Moreover, the characterization above can also be combined with standard convergence guarantees to estimate the number of iterations required to either certify non-membership or membership (up to an  $\varepsilon$ -error): If we use the vanilla Frank-Wolfe algorithm, then by standard guarantees (see e.g., [16]) it is known that the Frank-Wolfe gap  $g_{t} = \max_{v\in \tilde{P}}\langle y_{t} - \tilde{x},y_{t} - v\rangle$  satisfies  $\min_{0\leq \tau \leq t}g_{\tau}\leq \frac{4LD^2}{t + 3}$  for appropriate positive constants  $L$  and  $D$ . Suppose that  $\max_{v\in \tilde{P}}\langle y_t - \tilde{x},y_t - v\rangle \geq \frac{1}{2}\| \tilde{x} -y_t\|^2$  holds for all iterations  $0\leq t\leq T$ . We want to estimate how long this can hold. If  $\tilde{x}\notin \tilde{P}$ , then using the convergence guarantee yields:
$$
0 <   \frac {1}{2} \operatorname {d i s t} (\tilde {x}, \tilde {P}) ^ {2} \leq \min  _ {0 \leq \tau \leq t} \frac {1}{2} \| \tilde {x} - y _ {\tau} \| ^ {2} \leq \frac {4 L D ^ {2}}{t + 3}.
$$
Using  $L = 1$  as  $f(y) = \frac{1}{2}\| \tilde{x} -y\|^2$  and rearranging we obtain
$$
t \leq T := \frac {8 D ^ {2}}{\mathrm {d i s t} (\tilde {x} , \tilde {P}) ^ {2}} - 3,
$$
i.e., after at most  $T$  iterations we have certified that  $\tilde{x}$  is not in  $\tilde{P}$ . Guarantees for more advanced Frank-Wolfe variants can be obtained similarly.
Algorithm 1 Lazy Away-Step Frank-Wolfe [18, 19] with explicit active set and early termination
Input: Point  $y_0\in \tilde{P}$  , function  $f(y) = \frac{1}{2}\| y - \tilde{x}\| ^2$  , step-sizes  $\gamma_{t} > 0$  , tolerance  $\epsilon >0$  , oracle  $\Omega$    
Output: Valid cut  $\langle \tilde{\alpha},x\rangle \leq \tilde{\beta}$  for  $\tilde{P}$  with  $\langle \tilde{\alpha},\tilde{x}\rangle >\tilde{\beta}$  or false if  $\tilde{x}\in \tilde{P}$    
1:  $v_{0}\gets \Omega (\nabla f(y_{0}))$    
2:  $S\gets \{\{\gamma_0,v_0\} (1 - \gamma_0,y_0)\} ,\phi = \langle \nabla f(y_0),y_0 - v_0\rangle$    
3: for  $t = 0$  to  $t_\mathrm{max}$  do   
4: if  $\| f(y_t)\| <  \epsilon$  then   
5: return false   
6: end if   
7:  $(\lambda_L,v_L)\gets \min_{(\lambda ,v)\in S}\langle \nabla f(y_t),v\rangle ,(\lambda_A,v_A)\gets \max_{(\lambda ,v)\in S}\langle \nabla f(y_t),v\rangle$    
8: if  $\langle \nabla f(y_t),y_t - v_L\rangle \geq \max \{\langle \nabla f(y_t),v_A - y_t\rangle ,\frac{\phi}{2}\}$  then   
9:  $v_{t + 1}\gets v_L,\gamma_{\mathrm{max}}\gets 1$  {lazy step}   
10: else if  $\langle \nabla f(y_t),v_A - y_t\rangle \leq \max \{\langle \nabla f(y_t),x - v_L\rangle ,\frac{\phi}{2}\}$  then   
11:  $v_{t + 1}\gets v_A,\gamma_{\mathrm{max}}\gets \frac{\lambda_A}{1 - \lambda_A}$  {away step}   
12: else   
13:  $v_{t + 1}\gets \Omega (\nabla f(y_t)),\gamma_{\mathrm{max}}\gets 1$    
14: if  $\langle \nabla f(y_t),y_t - v_{t + 1}\rangle <  \frac{\phi}{2}$  then   
15:  $\phi \leftarrow \min \{\langle \nabla f(y_t),y_t - v_{t + 1}\rangle ,\frac{\phi}{2}\} ,\gamma_{\mathrm{max}}\leftarrow 0$  {dual step}   
16: end if   
17: end if   
18: if  $\langle y_t - \tilde{x},y_t - v_{t + 1}\rangle <  \frac{1}{2}\| \tilde{x} -y_t\| ^2$  then   
19: return cut  $\langle -\nabla f(y_t),x\rangle \leq \langle -\nabla f(y_t),v_{t + 1}\rangle$  {see Charac. 2.3}   
20: end if   
21:  $\alpha \gets \min \{\gamma_t,\gamma_{\mathrm{max}}\}$    
22:  $S\gets \{(\lambda (1 - \alpha),v):(\lambda ,v)\in S\} \cup \{(\alpha ,v_{t + 1})\}$    
23:  $y_{t + 1}\gets \sum_{(\lambda ,v)\in S}\lambda v$    
24: end for
3.2 Computational Aspects
A common trait of the local cuts framework is that  $\tilde{P}$  is accessed implicitly via an oracle returning vertices. By far the simplest black-box oracle for any bounded IP is enumeration, which simply evaluates all possible solutions  $x$  and picks the best one. If the IP is unbounded, then pure enumeration does not suffice any more and the oracle needs to take the unboundedness into account. For some problems, we can find problem-specific algorithms that only enumerate over feasible solutions or otherwise exploit the structure of the problem at hand to reduce the complexity of enumeration. Examples are the dynamic programming approach for knapsack problems, see Section 4.1, or directly enumerating  $n!$  possible permutations of  $n$  items for the linear ordering problem (LOP).
Similarly to the enumeration oracle, the lifting routine can also avail of problem-specific structure in some cases. In the case of LOPs, the so-called trivial lifting lemma holds, that is, facet-defining inequalities of the LOP polytope in dimension  $n$  also define facets in dimension  $r > n$  [35], meaning that no lifting is needed at all in this case. For knapsack problems, we can again use a dynamic programming approach, see Section 4.2.
In general, to apply our method to a given class of IP problems, one needs three components: i) a projection  $P \rightarrow \tilde{P}$ . ii) An oracle solving linear optimization problems over  $\tilde{P}$  to optimality; in order to be practical, the selected  $\tilde{P}$  should be such that the oracle runs reasonably fast. iii) A lifting method to lift cuts from  $\tilde{P}$  up to  $P$ .
In our implementation, we use the Lazy Away-Step Frank-Wolfe algorithm of [18, 19], which converges linearly for (Separation). We integrate the novel termination criterion from Characterization 2.3, leading to Algorithm 1. This algorithm should be thought of as a more advanced version of the vanilla Frank-Wolfe algorithm. This variant is motivated by the fact that Frank-Wolfe trends towards sparse solutions and hence the oracle will often return previously-seen vertices. Hence, instead of querying the expensive oracle, one stores all previous vertices in a active set whose size is controlled through so-called away steps. It provides superior convergence speed both in iterations and
wall-clock time, exploiting the strong convexity of our objective function of the separation problem; we refer the interested reader to [46, 52, 16] for an overview. Lazification, to be thought of as an advanced caching technique, further reduces the per-iteration cost by reusing previously computed LMO solutions. Lastly, we note that in our setting and case study presented in Section 4, the LMO always returns a vertex. Even though this is not a theoretical requirement for the results presented in this paper, we do not consider the alternative case for brevity.
4 Case Study: The Multidimensional Knapsack Problem
The multidimensional knapsack problem (MKP) is a well-known problem in combinatorial optimization and is strongly  $\mathcal{NP}$ -hard. It has been used to address various practical resource allocation problems [30]. The problem involves maximizing the total profits of selected items, taking into account  $m$  resource capacity (knapsack) constraints. There are  $n$  items that contribute profits given by  $c \in \mathbb{Z}^n$ . The resource consumption of item  $j$  for the  $i$ th knapsack is given by  $a_{ij} \in \mathbb{Z}_+$ ; this defines a matrix  $A = (a_{ij}) \in \mathbb{Z}_+^{m \times n}$ . The capacities of the knapsacks are given by  $b \in \mathbb{Z}^m$ . We define binary variables  $x \in \{0,1\}^n$  such that  $x_j$  is equal to 1 if item  $j$  is selected and 0 otherwise. Then MKP can be expressed as an IP:
$$
\max  \left\{\langle c, x \rangle : A x \leq b, x \in \{0, 1 \} ^ {n} \right\}. \tag {MKP}
$$
There exists abundant literature on the knapsack problems; we refer the interested reader to the recent survey by Hojny et al. [43].
We will also test our approach on the instances of the generalized assignment problem (GAP), see Section 5. GAP is a variant of MKP with applications in scheduling [43]. In addition to the constraints from the MKP problem, it is required that each of the  $n$  items be assigned to exactly one knapsack. The interested reader can find a survey, more details, and a comprehensive reference list in [4, 58].
In order for our approach to work, we need to provide two things: the oracle, presented in Section 4.1, and the lifting routine, presented in Section 4.2, cf. Section 2 and Section 3.
We consider each knapsack problem in turn and try to generate inequalities that are valid for each individual knapsack. This has the advantage that there are practically efficient oracles and more importantly efficient lifting processes. The disadvantage is that the cuts might be weaker, since they are valid for all integer solutions for all knapsack constraints instead of their intersection. An alternative would be to consider optimization oracles for the complete set of knapsack constraints as done by Gu [37]. However, then either lifting becomes more computationally demanding or one cannot use lifting.
4.1 The Linear Minimization Oracle
The process begins with a solution  $x^{\star}$  of the LP relaxation of the MKP. We create a reduced knapsack problem of dimension  $k \in \mathbb{Z}_{+}, k \leq n$  by removing variables of each knapsack that have integral values (0/1) in the LP relaxation. The oracles now solve the knapsack problems (KP) for each constraint of the form  $\max \left\{\langle c, x \rangle : \langle w, x \rangle \leq C, x \in \{0, 1\}^{n}\right\}$ , with  $c \in \mathbb{R}^k, w \in \mathbb{Z}_{+}^k, C \in \mathbb{Z}_{+}$ . In practice, the dimension  $k$  is rather small (in our test sets, see Section 5, we observe an average  $k$  value of 9.6 with a maximal size of 26), allowing for efficient solution approaches. In our implementation, we use a LMO based on dynamic programming. We note that we also apply the preprocessing improvements described by Vasilyev et al. [64], before we run the oracle on the reduced problem.
Dynamic programming, as presented by Bellman in 1957 [7], was one of the earliest exact algorithms for solving KPs. Toth [62] presents additional improvements to the algorithm. More recently, Boyer et al. present massively-parallel implementations running on GPUs [15]. The space and time complexity of the dynamic programming algorithm for KP is  $\mathcal{O}(kC)$  [15], where  $k$  is the number of items and  $C$  is the knapsack capacity. For this work, we reuse the single-threaded, CPU-based implementation of dynamic programming available in the open-source solver SCIP [10]. As we rely on existing implementations, we refer the interested reader to the above references for more details on this algorithm.
Table 1: Statistics for a branch-and-cut run with separation of local cuts for 45 generalized assignment problem instances (left) and 21 instances that were solved to optimization by all variants (right).
variant	# solved	time	sep. time	#cuts
default	29	98.2	-	-
lc0-nc-downlift	21	411.7	54.0	10858.6
lc0-nc-lifting	26	113.7	9.8	4933.6
lc1-nc-lifting	29	86.8	31.8	160244.4
variant	# solved	# nodes	time
default	21	614	6.0
lc0-nc-downlift	21	1522	33.7
lc0-nc-lifting	21	657	5.3
lc1-nc-lifting	21	185	4.2
4.2 The Lifting Routine
Lifting knapsack constraints has been extensively studied in literature, see [43] for a short survey with comprehensive references. Therefore, we only briefly summarize the implemented methods here and refer the interested reader to [43] and references therein for more details.
Let  $\{x\in \{0,1\} ^n:\sum_{j = 1}^n a_jx_j\leq a_0\}$  be one of the original knapsack constraints (corresponding to a single row in  $Ax\leq b$  in (MKP)). Define  $[n]\coloneqq \{1,\ldots ,n\}$ ,  $F_{0}\coloneqq \{j\in [n]:x_{j}^{\star} = 0\}$ , and  $F_{1}\coloneqq \{j\in [n]:x_{j}^{\star} = 1\}$ . Then  $S\coloneqq [n]\setminus (F_0\cup F_1)$  are the variable indices in the reduced knapsack. The lifting procedure then lifts a given inequality  $\sum_{j\in S}\alpha_{j}x_{j}\leq \alpha_{0}$  valid for the reduced knapsack polytope  $\{x\in \{0,1\} ^S:\sum_{j\in S}a_jx_j\leq a_0 - \sum_{j\in F_1}a_j\}$  to a valid inequality for the original knapsack by computing new coefficients  $\beta_{j}$ ,  $j\in F_0\cup F_1$ :
$$
\sum_ {j \in S} \alpha_ {j} x _ {j} + \sum_ {j \in F _ {1}} \beta_ {j} x _ {j} + \sum_ {j \in F _ {0}} \beta_ {j} x _ {j} \leq \alpha_ {0} + \sum_ {j \in F _ {1}} \beta_ {j}. \tag {2}
$$
We implemented algorithms known as sequential up-lifting and sequential down-lifting, respectively. The implementation is based on dynamic programming as described by Vasilyev et al. [64].
5 Computational Experiments
We implemented the described methods in C/C++, using a developer version of SCIP 8.0.4 (github 3dbcb38) and CPLEX 12.10 as LP-solver. All tests were performed on a Linux cluster with 3.5 GHz Intel Xeon E5-1620 Quad-Core CPUs, having 32 GB main memory and 10 MB cache. All computations were run single-threaded and with a time limit of one hour. To concentrate on the improvement of local cuts on the dual bound, we initialize all runs with the optimal value. We used  $\epsilon = 1 \times 10^{-9}$  in Algorithm 1.
To demonstrate the advantage of using local cuts with the Frank-Wolfe approach, we run our implementation on the generalized assignment instances from the OR-Library available at http:// people.brunel.ac.uk/~mastjjb/jeb/orlib/gapinfo.html. These instances have also been used by Avella et al. [4].
The results are presented in Table 1. Here, "default" are the default, factory settings of SCIP. The other settings are lcX-nc-Y, where  $X = 0$  means that we only separate local cuts in the root node and  $X = 1$  means that we separate local cuts in the whole tree;  $Y$  refers to whether we perform down lifting ( $Y = \text{downlift}$ ) or up- and down lifting ( $Y = \text{lifting}$ ). Note that for these settings, where local cuts are enabled, we turn the generation of all other cuts off, because this (somewhat surprisingly) showed better performance. The CPU time in seconds ("time") and separation time ("sep. time") as well as number of nodes ("#nodes") are given as shifted geometric means<sup>3</sup>. The numbers of generated cuts ("#cuts") are arithmetic means. Note that the iteration limit for the Frank-Wolfe algorithm is 10000 in the root node and 1000 in the subtree. We also reduce the effect of cut filtering allowing for more cuts to enter the main LP. Moreover, we initialize the runs with the best know solution values as in [4].
The results show that the best version is lc1-nc-lifting, i.e., it helps to separate local cuts in every node and perform up- and down lifting. This version is roughly  $31\%$  faster than the default settings on the instances solved to optimality. Using only down lifting performs badly. In any case, on these
3The shifted geometric mean of values  $t_1,\ldots ,t_r$  is defined as  $\left(\prod (t_i + s)\right)^{1 / r} - s$  with shift  $s = 1$  for time and  $s = 100$  for nodes in order to decrease the influence of easy instances in the mean values.
instances, using our local cuts method is a big advantage. Additional results are given in the appendix. In addition, we also ran an experiment in which we applied complemented mixed-integer rounding (CMIR) on the produced cut, which turned out to not be helpful and is therefore not reported in detail.
Some additional observations over all 45 instances in the test set are as follows: Variant lc1-nc-lifting called local cuts separation 16870.6 times on average. The total time for Frank-Wolfe separation is about one third of the total time. The time spent in the oracle is 17.3 seconds on average compared to a total of 87.9 seconds for the complete Frank-Wolfe algorithm. On average 69968.8 calls ended running into the iteration limit, 81.2 detected optimality with a zero gradient, 8652.9 stopped because the primal gap is small enough, and 143155.9 stopped because of the termination criterion of Section 3.1.1. This demonstrates the effect of this criterion.
In this paper, we presented a novel method to learn local cuts without relying on solutions of LPs in the process. To show the effectiveness of our approach, we selected the multidimensional knapsack problem as a case study and presented computational results to support our claims.
Solving LPs has proved to be notoriously hard to parallelize, with only minor performance improvements reported in literature to date [40, 45]. Thus, existing methods for deriving local cuts, which rely on solving LPs, typically run single-threaded, on CPUs. Our approach is quite fast, as demonstrated in the computational experiments for our target problem class, but also paves the way for exploring highly parallel implementations on heterogeneous hardware and compute accelerators. This is made possible by eliminating the dependence on LPs and instead relying on the Frank-Wolfe algorithm. One such option we would like to explore in the future is to derive a vectorized version of our Frank-Wolfe algorithm that could work on multiple separation problems at the same time, increasing the computational density of the operations performed and availing of massively parallel compute accelerators like GPUs in the process.
The presented method is generic and can be applied to any (M)IP. We have chosen one important problem class in this paper to demonstrate the method. A natural extension of this work would be to consider other important problem classes and evaluate the benefits of using our method on those problems - especially those with beneficial properties as outlined in Section 3.2.
[1] T. Achterberg and R. Wunderling. Mixed integer programming: Analyzing 12 years of progress. In M. Jünger and G. Reinelt, editors, *Facets of Combinatorial Optimization: Festschrift for Martin Grötschel*, pages 449–481. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.
[2] E. Althaus, T. Polzin, and S. V. Daneshmand. Improving linear programming approaches for the Steiner tree problem. In K. Jansen, M. Marggraf, M. Mastrolilli, and J. D. P. Rolim, editors, Experimental and Efficient Algorithms, pages 1-14, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.
[3] D. Applegate, R. Bixby, V. Chvátal, and W. Cook. TSP Cuts Which Do Not Conform to the Template Paradigm. In M. Jünger and D. Naddef, editors, Computational Combinatorial Optimization, volume 2241 of Lecture Notes in Computer Science, pages 261-303. Springer Berlin Heidelberg, 2001.
[4] P. Avella, M. Boccia, and I. Vasilyev. A computational study of exact knapsack separation for the generalized assignment problem. Computational Optimization and Applications, 45(3):543-555, 2010.
[5] E. Balas, S. Ceria, and G. Cornuéjols. A lift-and-project cutting plane algorithm for mixed 0-1 programs. Mathematical Programming, 58(1):295-324, 1993.
[6] M.-F. Balcan, T. Dick, T. Sandholm, and E. Vitercik. Learning to branch. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 344-353. PMLR, 10-15 Jul 2018.
[7] R. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1957.
[63] I. L. Vasil'ev. A cutting plane method for knapsack polytope. Journal of Computer and Systems Sciences International, 48(1):70-77, 2009.
[64] I. Vasilyev, M. Boccia, and S. Hanafi. An implementation of exact knapsack separation. Journal of Global Optimization, 66(1):127-150, 2016.
We ran two additional experiments. In the first, we try to repeat the experiments about the strength of local cuts in the root node from [47]. As a comparison, we use the implementation of the local cuts with the LP-based approach of [57] (see the beginning of Section 3 for an overview of the method). In the second experiment, we demonstrate what happens if we use local cuts in a branch-and-cut framework to solve multi-dimensional knapsack problems to optimality.
We start with a comparison of the gap-closed, which is defined as  $100 - 100\frac{p - d_r}{p - d_{lp}}$ , where  $p$  is the optimal primal value,  $d_r$  is the dual bound at the end of the root node, and  $d_{lp}$  is the dual bound of the first LP at the root node. Table 2 shows the results. We use the same multi-dimensional knapsack instances as Kaparis and Letchford: they were originally generated randomly by Chu and Beasley [24] and are available at http://people.brunel.ac.uk/~mastjjb/jeb/orlib/mknapinfo.html. The instances are organized in blocks of 10, using  $n$  variables,  $m$  knapsack constraints, and a parameter  $\alpha$ . Column "KL" shows the gap-closed from [47]. Then the results of using the implementation of the LP-based approach (which is also used in [47]) and our Frank-Wolfe approach are presented. For each approach, we show the average of gap-closed, total running time in seconds, as well as separation time, number of calls, and generated violated cuts by the local cut separation over each instance block (of 10 instances). For both approaches, we turn off all other cuts and strong branching. Moreover, we use settings that allow 1000 rounds of local cuts, 10000 iterations of the Frank-Wolfe algorithm in the root node, and reduce the effect of cut filtering, i.e., more cuts are added to the LP. In each round we separate local cuts for all knapsack constraints that are available. We also initialize the runs with the optimal value to remove the effects of primal heuristics.
The results show that the LP-based approach achieves similar gap-closed values as Kaparis and Letchford. One explanation for the differences is that the final gap depends on the particular points to be separated (but note that we do not generate rank-2 cuts). In comparison, the Frank-Wolfe approach is much faster, but also produces a smaller gap-closed. There are again several reasons for the differences: We limit the number of Frank-Wolfe iterations to 10000 in the root node, which will leave some separation problems to be undecided and the Frank-Wolfe algorithm might fail to converge (for instance, for  $n = 500$ ,  $m = 5$ ,  $\alpha = 75$ , on average 34.8 Frank-Wolfe runs terminated in the iteration limit and 75.4 with the termination criterion explained in Section 2). Moreover, the Frank-Wolfe approach does not necessarily produce a facet, which can weaken the bounds.
Table 3 shows results for running a complete branch-and-cut with variants lc0-nc-lifting and lc1-nc-lifting. As a comparison, we use the default settings of SCIP, but reduce cut filtering (this produces slightly better results for these instances). The results show that for smaller instances,