OKRidge: Scalable Optimal k-Sparse Ridge Regression
Authors: Jiachang Liu
Sam Rosen
Chudi Zhong
Cynthia Rudin
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Sparse Ridge Regression
Dynamical Systems

Abstract: We consider an important problem in scientific discovery, namely identifying sparse governing equations for nonlinear dynamical systems. This involves solving sparse ridge regression problems to provable optimality in order to determine which terms drive the underlying dynamics. We propose a fast algorithm, OKRidge, for sparse ridge regression, using a novel lower bound calculation involving, first, a saddle point formulation, and from there, either solving (i) a linear system or (ii) using an ADMM-based approach, where the proximal operators can be efficiently evaluated by solving another linear system and an isotonic regression problem. We also propose a method to warm-start our solver, which leverages a beam search. Experimentally, our methods attain provable optimality with run times that are orders of magnitude faster than those of the existing MIP formulations solved by the commercial solver Gurobi.

OKRidge: Scalable Optimal k-Sparse Ridge Regression
Jiachang Liu, Sam Rosen, Chudi Zhong, Cynthia Rudin
Duke University
(jiachang. 1iu, sam samerosen, chudi zzoog) @duke. edu, cynthia@cs Suntha
Abstract
We consider important problem in scientific discovery, namely identifying sparse governing equations for nonlinear dynamical systems. This involves solving sparse ridge regression problems to provable optimality in order to determine which terms drive the underlying dynamics. We propose a fast algorithm, OKRidge, for sparse ridge regression, using a novel lower bound calculation involving, first, a saddle point formulation, and from there, either solving (i) linear system (ii) using an ADMM based approach, where the proximal operators can efficiently evaluated by solving another linear system and isotonic regression problem. We also propose a method to warr-start our solver, which leverages beam search. Experimentally, our methods attain provable optimality with times that orders of magnitude faster than those the existing MIP formulations solved by the commercial solver Gurobi.
Introduction
We are interested in identifying sparse and interpretable governing differential equations arising from nonlinear dynamical systems. These scientific machine learning problems whose solution involves sparse linear regression. Specifically, these problems require the exact solution of sparse regression problems, with the most basic being sparse ridge regression:
minlly y x x1112 Aa | subject
||/311o k.
where & specifies the number of nonzero coefficients for the model. This formulation is general, but in the case nonlinear dynamical systems, the outcome is a derivative (usually time or spatial) of each dimension Here, assume that the practitioner has included the true variables, along with many other possibilities, and is looking to determine which terms (which transformations of the variables) are real and which not. This problem is NP-hard 44), and is more challenging in the presence of highly correlated features. Selection of correct features is vital in this context, as many solutions may give good results training data, but will quickly deviate from the true dynamics when extrapolating past the observed data due to the chaotic nature of complex dynamical systems.
Both heuristic and optimal algorithms have been proposed solve these problems. Heuristic methods include greedy sequential adding of features (25] 16 54 [22 or ensemble (31] methods. These methods are fast, but often get stuck in local minima, and there is way to assess solution quality due to the lack of a lower bound on performance. Optimal methods provide an alternative, but are slow since they must prove optimality. MIOSR 99. mixed-integer programming (MIP) approach, has been able to certify optimality of solutions given enough time. Slow solvers cause difficulty in performing cross valssdaiiitiinnn X2 (the l2 regularization coefficient) and & (sparsity level).
We aim to solve sparse ridge regression to certifiable optimality, but fraction of the run time. We present fast branch-and-bound (BnB) formulation, OKRidge. A crucial challenge is obtaining
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

tight and feasible lower bound for each node in the BnB tree. It is possible to calculate the lower bound via the SOS1 [9] big-M [TO] or the perspective formulations (also known as the rotated second-order cone constraints) 32 59 the mixed-integer problems can then be solved by MIP solver. However, these formulations
not consider the special mathematical structure of the regression problem. To calculate lower bound more efficiently, we first propose a new saddle point formulation for the relaxed sparse ridge regression problem. Based on the new saddle point formulation, we propose two novel methods to calculate the lower bound. The first method is extremely efficient and relies on solving only a linear system of equations. The second method is based on ADMM and can tighten the lower bound given by the first method. Together, these methods give us a tight lower bound, used to prune nodes and provide a small optimality gap. Additionally, propose method based on beam-search [58] to get near-optimal solution quickly, which can be a starting point for both our algorithm and other MIP formulations. Unlike previous methods, our method uses dynamic programming approach that previous solutions in the BnB tree can be used while exploring the current node, giving a massive speedup. In summary, our contributions are:
(I) We develop highly efficient customized branch-and-bowe bound framework for achieving optimality in k-sparse ridge regression, using a novel lower bound calculation and heuristic search.
(2) To compute the lower bound, we introduce new saddle point formulation, from which we derive two efficient methods (one based on solving a linear system and the other on ADMM).
(3) Our warm-start method is based beam- and implemented in dynamic programming fashion, avoiding redundant calculations. We prove that our warm-start method is an approximation algorithm with exponential factor tighter than previous work.
On benchmarks, OKRidge certifies optimality orders of magnitude faster than the commercial solver Gurobi. For dynamical systems, our method outperforms the tate-off-the-rt certifiable method by finding superior solutions, particularly in high-dimensional feature spaces.
Preliminary: Dual Formulation via the Perspective Function
There is an extensive literature on this topic, and longer review of related work is in Appendix A If we ignore the constant term y" y. we can rewrite the loss objective Equation as:
Gridge (3) = X X2
with p as the number features. We are interested in the following optimization problem:
min Lridg (3) s.t. (1 - 20)),;;----- 0 2 E 10,
(3)
B
where & the number of nonzero coefficients. With the sparsity constraint, the problem Na-hard. The constraint (1-2,)8 8, in Problem (3) can be reformulated with the SOSI, big-M, or the perspective formulation (with quadratic cone constraints), which then can be solved by a MIP solver. Since commercial solvers do not exploit the special structure of the problem, we develop customized branch-and- -nd--aunddd----dd--- framework.
For any function f()), the perspective function is g(a, b) = bf(l)) for the domain b > 0
32 34 26
and g(a, 6) = 0 otherwise. Applying to f(a) = a², we obtain another function g(a b) = - As
shown by [4) replacing the loss term 83 and constraint (1 - 2, 3, = with the perspective formula 83/23 in Problem (3) would not change the optimal solution. By the Fenchel conjugate 4 g(;:) can be rewritten as g(a, b) = max, ac- h. If we define new perspective loss as:
2 - Bic - 24
Fanchel (,, 22 = BT x x x Ao Σ
(4)
rige
then we can reformulate Problem
3 as:
n
min max
ridge
(Fenchhl 2,, s.t. Σ
2, S k, 2j E 1).
(5)
3
c
j=1

---

If we relax the binary constraint {0, 1) to the interval [0, 1] and swap max and min (no duality gap. as pointed by we obtain the dual formulation for the convex relaxation of Problem (5):
max min Fonchel riide (3, c) s.t. Σ Eash 2j E [0,
(6)
While 4 uses the perspective formulation for safe feature screening, we use it to calculate lower bound for Problem (3). However, directly solving the maxmin problem computationally challenging.
In Section 3.1 we propose two methods to achieve this in an efficient way.
3
Methodology
We propose custom BnB framework to solve Problem (3). We have 3 steps to process each node in the BnB tree. First, we calculate lower bound of the node, using two algorithms proposed in the next subsection. If the lower bound exceeds or equals the current best solution, does not lead to any optimal solution, so we prune the node. Otherwise, we go to Step 2, where the solution from Step
have proven that
perform beam-search find near-optimal solution. In Step and propose a branching strategy to create new nodes in the BnB tree. We continue until reaching the optimality gap tolerance. Below, we elaborate on each step. In Appendix E we provide visual illustrations of BuB and beam search as well as the complete pseudocodes of our algorithms.
3.1 Lower Bound Calculation
Tight Saddle Point Formulation
We first rewrite Equation 2 with new hyperparameter A:
P Crdge xB, 2) = QT 2y X (A2 + A) Σ
83
(7)
j=1
where Q^ = x-= AI. We restrict À [0, Amia (XT) X) where Xuin (-) denotes the minimum eigenvalue of matrix. We see that Qx is positive semidefinite, so the first term remains convex. This trick is related to the optimal perspective formulation [62 28 37). but we set the diagonal matrix diag(d) in (28) be AI. We call this trick eigen-perspective formulation. The optimal perspective formulation requires solving semidefinite programming (SDP) problems, which have been shown not scalable to high dimensions (28), and MI-SDP is not supported by Gurobi.
Solving Problem is equivalent to solving the following problem:
min _,
Lidge 2B, s.t. (1 (1-zz)] 22]]3 = 0, Σ 2j Si 2j 10,
(8)
j=1
We get a continuous relaxation of Problem 3 if we relax {0. 1) to [0,
We can now define new loss analogous to the loss defined in Equation
P
ridgo :=, 2y x3-x3++5 (A2 + A) Σ
Fenchel
(B,c,
2g)
(9)
==1
Then, the dual formulation analogous to Problem 6 is:
max min 9.2 Lridge A (3, 2, c) s.t. Σ
eFeenchel
< ki Zj E 10.
(10)
Solving Problem (10) provides us with a lower bound to Problem 8 More importantly, this lower bound becomes tighter as À increases. This novel formulation is the starting point for our work.
We next propose a reparametrization trick to simplify the optimization problem above. For the inner optimization problem in Problem (10), given any c, the optimality condition for 3 is (take the gradient with respect to B and set the gradient to 0):
de + (X y
(11)
Inspired by this optimality condition, we have the following theorem:
3

---

Theorem 3.1. If we reparameterize d
(X" Qxy) with a new parameter 7, then Problem (10) is equivalent to the following saddle point optimization problem:
max min ridge
x (7,,
S.T.
2j S k, 23 E (0, 1].
(12)
where
X y ) (x)ag()((((()))) ) = =- X2 x
Luudile
-- Qx7),
(13)
and dzag(z) ) is diagonal matrix with the diagonal.
To our knowledge, this is the first time this formulation is given. Solving the saddle point formulation to optimality in Problem (2) gives us a tight lower bound. However, this computationally hard.
Our insight is that we can solve Problem (22 approximately while still obtaining feasible lower bound. Let us define new function h(y) as short-hand for the inner minimization in Problem (12)
h(r) min ridge (1,, 2) s.t.
Σ 2j S k. 2g E [0, 1].
(14)
For any arbitrary E RP, h(r)) is a valid lower bound for Problem We should choose "y such that this lower bound h(y) is tight. Below, we provide two efficient methods to calculate such a ty
Fast Lower Bound Calculation
First, we provide fast way to choose 7 The choice of is motivated by the following theorem:
Theorem 3.2. The function h(()) defined in Equation
lower bounded by
1
h(()) 2 ---
-
(15)
bn
Furthermore, the right-hand size of Equation where in this case, h((r) evaluated at Y becomes
(55) is maximized if = =argmin, Cridge
h(9)) = Cridge + (A2 + AySumBottomp-klk
(16)
where SumBottom,-&ly x()) denotes the summation of the smallest - terms of a given set.
Here provide intuitive explanation of why h(f)) is valid lower bound. Note that the ridge regression loss is strongly convex. Assuming that the strongly convex parameter is μ (see Appendix B). by the strong convexity property, have that for any RP.
Cridge 2ridde (7) + Lridge (4) - 7) + 14 2 11 - 112..
(17)
Because y minimizes Lridge (-), we have V Lridge (+) = O. For the k-sparse vector y with ||>| < k. the minimum for the right-hand side Inequality (17) can be achieved if n; = Ω for the top terms of "s''.. This ensures the bound applies for all k-sparse 7. Thus, the k-sparse ridge regression loss is lower-bounded by
Lridge Zriidge (f) + H SumBottom ((13))
for y E RR with l/ < kt For ridge regression, the strong convexity y parameter can be chosen from [0, 2(2 Amin X"X)). If we let 2()2 + A). obtain h(H) in Theorem 3.2
The lower bound l((5) can be calculated extremely efficiently by solving ridge regression problem (solving the linear system (XT) X + 12 x y for 7) and adding the extra p - terms. However, this bound not the tightest we can achieve. In the next subsection, we discuss how apply ADMM to maximize f((r)) further based on Equation [14

---

Tight Lower Bound via ADMM
Let define p X y Yxy Starting from Problem 12 if we minimize in the inner optimization under the constraints 2j-1 2j S k and Ej E [0, 1 for Vj, we have 2j = 1 for the top
terms of p and 2j = 0 otherwise. Then, Problem T2) can be reformulated as follows:
- -min (F(x) + G(p)) s.t. +xy+p== + -- p
(18)
where F(7): =(?)==~"QQ Qxyy and G(p) == Sum ((r))) The solution to this problem is dense vector that can be used to provide lower bound on the original k-sparse problem. This problem
Qxxy
can be solved by the alternating direction method of multipliers (ADMM) [17]. Here, we apply the
iterative algorithm with the scaled dual variable [3]
y ++ argmin F(7) + 2
1Qx++ + p' - xx y + 22112
(19)
0 2aQx7 1+1 (1 2a)(p' -x y)
(20)
p² argmin G(p) + P
2 1 +p X" y 9'112
(21)
nt = q¹ + + 11 +p t+ x X"y, y,
(22)
where a is the relaxation factor, and is the step size.
It is known that ADMM suffers from slow convergence when the step size is not properly chosen. According to [33,, to ensure the optimized linear convergence rate bound factor, we can pick a = 1 and p
where Amax(-) denotes the largest eigenvalue of matrix, and Amin>o(-) denotes the smallest positive eigenvalue of matrix.
V Amaxx (Qainin
Having settled the choices for the relaxation factor a and the step size p. we are left with the task of solving Equation (I9) and Equation (2I) (also known as evaluating the proximal operators (521). Interestingly, Equation (19) can be evaluated by solving a linear system while Equation (2I can be evaluated by recasting the problem as an isotonic regression problem.
Theorem 3.3. Let F(y) = yx and G(p) X SomTTop (t Then the solution for the
problem 741 argmin, F(>) + 11l +p' -x-x + 2'112 is
y
,+1 (2 12)+ (x"y-p'-d) 0
(23)
Furthermore, let a = X y - 0-+1 q" and J be the indices of the top terms of {laz]}. The
solution for the problem pt = argmin, G(p) + & 1 p- x y + o" 1 is2is is = sign(a,) 0g.
where
è argmin
wy b,)² s.t. vs 21 if las1 |ii
(24)
"
ff J otherwise
a by
wj-
1
x(>2+t)
Wi
Problem
24 is an isotonic regression problem and can be efficiently solved in linear time 12
21v.
3.2 Beam-Search as a Heuristic
After finishing the lower bound calculation in Section 3.1 upper bound in the BnB tree. We discuss how to add features, keep good solutions, and use dynamic programming improve efficiency. Lastly, we give a theoretical guarantee on the quality of our solution.
next explain how to quickly reduce the
33 also considers matrix preconditioning when computing the step size, but this is computationally expensive when the number of features is large, so we ignore matrix rescaling by letting E be the identity matrix in Section VI Subsection A of (33.
5

---

Starting from the vector O, we add one coordinate at a time into our support until we reach a solution with support size k. At each iteration, we pick the coordinate that results in the largest decrease the ridge regression loss while keeping coefficients in the existing support fixed:
( (B)) | 3 + A
j E argmin min Cridge (3 + ae,)
j E argmax
(25)
j
where x3 denotes the j-th column of X, and the right-hand side uses an analytical solution for the line-search for a. This is similar to the sparse sinnlex algorithm [6. However, after adding feature, we adjust the coefficients restricted on the new support by minimizing the ridge regression loss.
The above idea does not handle highly correlated features well. Once a feature is added, it cannot be
removed [61]. To alleviate this problem, we use beam-search 58 43 at each stage of support expansion:
keeping the best B solutions
j E arg BottomB, (min Lridge (3 + aee));
(26)
where j" E arg BottomB, means j belongs to the set of solutions whose loss is one of the B smallest finetune the solution on the newly expanded support and choose the best B
losses. Afterwards,
solutions for the next stage of support expansion. A visual illustration of beam search can be found in Figure 6 in Appendix E which also contains the detailed algorithm.
Although many methods have been proposed for spanse ridge regression, none of them have been designed with the BnB tree structure in mind, Our approach is to take advantage of the search history of past nodes to speed up the search process for a current node. To achieve this, we follow a dynamic programming approach by saving the solutions of already explored support sets. Therefore, whenever we need to adjust coefficients on the new support during beam search, we can simply retrieve the coefficients from the history if support has been explored in the past. Essentially, we trade memory space for computational efficiency.
3.2.1 Provable Guarantee
Lastly, using similar methods to 29 we quantify the gap between our found heuristic solution e and the optimal solution 3 in Theorem 3.4 Compared with Theorem 5 in (29,, we improve the
factor in the exponent from Mi to mag (since M1 Max. where M1 and Max are defined in (29).
Theorem 3.4. Let us define a k-sparse vector pair domain to be Q == {xx y) E E ||0]|o S k. llo S k; ||æc - yyll- I k). Any M1 satisfying [(y S f(x) + (xx) (y (y-x) x)) x) My ly x for all (x. 3) E Q is called a restricted smooth parameter with support size and any m2x satisfying fly) Z f(x) + fxx (y- x) + man bly 2222 for all (x, y) E P2z is called a restricted strongly convex parameter with support size 2k. If ß is our heuristic solution by the beam-search method, and 3 is the optimal solution, then:
Gridge (Ba) S Cridde(@)) E (1 - e- emax//i) Lridtee (B*))
(27)
3.3 Branching and Queuing
Branching: The most common branching techniques include most-infeasible branching and strong branching 15 7 However, these two techniques require having fractional values for the binary variables 2j's 's which we do not compute in our framework. Instead, we propose new branching strategy based on heuristic solution 3 we branch the coordinate whose coefficient, if set would result in the largest increase in the ridge regression loss Lridge (See Appendix E for details):
j = argmax Lridge (3 -B,e)).
(28)
The intuition is that the coordinate with the largest increase in Lrigg potentially plays a significant role, want fix such coordinate early as possible in the BuB tree.
Queuing: Besides the branching strategy, we need a queue to pick node to explore among newly created nodes. Here, we breadth-first approach, evaluating nodes in the order they are created.

---

(p=0.1)
(p=0.5)
(p=0.9)
10a
10a
10a
time (sec)
10*
10a
10a
10
10-
100
500
5000
100
500
1000
3000
100
1000
3000
5000
1.00
1.00
1.00
0.75
0.75
0.75
gap
0.50
0.50
0.50
0.25
0.00
0..00
10.00
500
5000
100
3000
5000
# of features
# of features
of features
50S1 + warm start
big.M + warm start
persp + warm start
eigen persp. warm start
ouTs
Figure 1: Comparison of running time (top row) and optimality gap (bottom row) between our method and baselines, varying the number of features, for three correlation levels p = 0.1, 0.5, 0.9 (n 100000, & 10). Time is on the log scale. Our method is generally orders of magnitude faster than other approaches. Our method achieves the smallest optimality gap, especially when the feature correlation p is high.
(p=0.1)
(p=0.5)
(p=0.9)
10a
10a
10a
time (sec)
102
10a
102
10a-
10
10a
3000
6000
7000
3000
8000
/000
3000
4000
6000
/000
5
5
5
3000
7000
3000
5000
3000
6000
7000
of samples
of samples
# of samples
5051 warm start
big-M warm start
persp. warm start
eigen persp. warm start
ours
Figure 2: Comparison of running time (top row) and optimality gap (bottom row) between our method and baselines, varying sample sizes, for three correlation levels p == 0.1, 0.9 (p 3000, 10). Time is on the log scale. When 0.1 and = 0.5, OKRidge is generally orders of magnitude faster than other approaches. In the case p = 0.9, we achieve the smallest optimality gap as shown in the bottom row.
4 Experiments
We test the effectiveness of our OKRidge on synthetic benchmarks and sparse identification of nonlinear dynamical systems (SINDy) Our main focus is: assessing how well our proposed lower
bound calculation speeds up certification (Section 4.1). and evaluating solution quality of OKRidge
on challenging applications (Section 4.2) Additional extensive experiments are in Appendix G and
H Our algorithms are written in Python. Any improvements we see over commercial MIP solvers, which are coded in C/C++, are solely due to specialized algorithms.
7

---

Lorenz
Hopf
MHD
Coeffic coniiiicnnto Error True Positivity Rate
0.8
0.8
0.6
0.6
0.6
02
0.
0.0
0.0
10"
139
10
10
10-2
10-1
10-3
10"2
10-
104
Train RMSE
10a
10
10a
10A
13"
Test RMSE
10-1
102-
10"
1
10".
10-3
-
10"
10
10a
10
10"
10
Training Trajectory Length (s)
Training Trajertory Length
Training Trajectary Length (5)
MIOSR
SQ)
+
SSR
+
ESSISQ
ours
Figure 3: Results on discovering sparse differential equations. On various metrics, OKRidge outperforms all other methods, including MIOSR which uses a commercial (proprictary) MIP solver.
4.1 Assessing How Well Our Proposed Lower Bound Calculation Speeds Up Certification
Here, we demonstrate the speed of OKRidge for certifying optimality compared to existing MIPs We 1-hour time limit and an optimality gap of relative tolerance 10-4
solved by Gurobi 335
We use a value of 0.001 for X2. Our 4 baselines include MIPs with SOS1, big-M (M = 50 to
prevent cutting off optimal solutions), perspective [4) and sigen-perspective formulations (A Amia (X7 X)) (28). the main text, we use plots to present the results. In Appendix we present
the results in tables. Additionally, in Appendix G conduct perturbation studies X2 (A2 0.1
and A2 10) and M (M = 20 and M = 5). Finally, still in Appendix G we also compare OKRidge
with other MIPs including the MOSEK solver $SubsetSelectionI10 [11, and LOBNB 39
Similar to the data generation process in [ 48 we first sample v, IRP from Gaussian distribution (O Σ) with mean and covariance matrix Σ, where Σ, = أام- Variable p controls the feature correlation. Then, we create the coefficient vector 3 with nonzero entries, where B = if
j mod (p/k) 0. Next, construct the prediction yı .!. 3 + cx where E; i.i.d. NO ||x 11 SNR stands for signal-to-noise ratio (SNR), and we choose SNR to be in all our experiments.
SNR
In the first setting, fix the number of samples with 100000 and vary the number of features p E (100, 500, 1000, 3000, 5000} and correlation levels p E (0.1, 0.5, 0.9} (See Appendix G for 0.7). We warm-started the MIP solvers by our beam-search solutions. The results
0.3 and can be seen Figure T From both figures, we see that OKRidge outperforms all existing MIPs solved by Gurobi, usually by orders of magnitude.
In the second setting, we fix number of features to 3000 vary number of samples {3000, 4000, 5000, 6000, 7000} and the correlation levels E (0.1, 0.5. 0.9) (see Appendix G for p 0.3 and p 0.7). As in the first setting, we also warm-started the MIP solvers by our beam-search solutions. The results are in Figure When is close the correlation is high

---

(p 0.9), no methods can finish within the 1-hour time limit, but OKRidge prunes the search space well and achieves the smallest optimality gap. When becomes larger in the case of / = 0.1 and = 0.5, OKRidge runs orders of magnitude faster than all baselines.
4.2 Evaluating Solution Quality of OKRidge on Challenging Applications
On previous synthetic benchmarks, many heuristics (including our beam search method) can find the optimal solution without branch-and-bound. In this subsection, we work on more challenging scenarios (sparse identification of differential equations). We replicate the experiments in [9] using three dynamical systems from the PySINDy library [27 42]: Lorenz System, Hopf Bifurcation, and seggnetohydroyyamiin (MHD) model [24). The Lorenz System is 3-D system with the nonlinear differential equations:
dx/d ox + oy,
dy/d = pt y T2,
dz/dt = ry 8z
where we use standard parameters o = =00, p 28. The true sparsities for each dimension are (2, 3, 2). The Hopf Bifurcation is 2-D system with nonlinear differential equations:
dx/dl pa + wy Ax Ary²
dy/dt = wit + jy Ax²y Ay
where the standard parameters -0.05, 1, 1. The true sparsities for each dimension are (4, 4). Finally, the MHD is 6-D system with the nonlinear differential equations:
VVi/d 4V2 V33 4ByBs, dV2/ dt - 7VV + 7B, dB, /dt = 2B3V 2/2 - 2Vz22, dB2/dt = 5V - 5B&Vi,
dVs/ 3V, V22 -3B, B2,
dBs/dt = 9V, B2 -BB
The true sparsities for each dimension are 2,
We all monomial features (candidate funetions) up to 5th order interactions. This results in 56 functions for the Lorentz System, 21 for Hopf Bifurcation, and 462 for MHD. Due to the high-order interaction terms, features highly correlated, resulting in poor performance of heuristic methods.
4.2.1 Baselines and Experimental Setup
In addition MIOSR (which relies on the SOSI formulation), we also compare with three common baselines in the SINDy literature: STLSQ [4. SSR [16], and E-STLSQ [31]. The baseline SR3 225 not included since the previous literature 9 shows it performs poorly. We compare OKRidge with other baselines using the SINDy library [27 42). We follow the experimental setups in (9 for model selection, hyperparameter choices, and evaluation metrics (please see Appendix for details). In Appendix H. we provide additional experiments on Gurobi with different MIP formulations and comparing with more heuristic baselines.
4.2.2 Results
Figure 3 displays the results. OKRidge (red curves) outperforms all baselines, including MIOSR (blue curves), across evaluation metrics. On the Lorenz Sys- tem, all methods recover the true feature support when the training trajectory is long enough. When the train- ing trajectory length is short, i.e., the left part of each subplot, (or equivalently, when the number of samples small), OKRidge performs uniformly better than all other baselines. On the Hopf Bifurcation, all heuristic methods fail to recover the true support, resulting in poor performance. On the final MHD, OKRidge maintains the top performance and outperforms MIOSR on the true positivity rate. This demonstrates the effectiveness of OKRidge, which incurs lower runtimes and yields bet- ter metric scores under high-dimensional settings. The highest runtimes are incurred for the MHD (with 462 candidate functions/fcatures), which are shown in Figure
MHD
3000
Burning
2000
1000
Iraining Irajectory Length [s] + ours
MIDSR
Figure 4: Running time comparison be- tween OKRidge and MIOSR on the MHD system with 462 candidate functions. OKRidge significantly faster than the previous state of the art.

---

Limitations of OKRidge When the feature dimension is low (under 100s), Gurobi can solve the problem to optimality faster than OKRidge. This is observed on the synthetic benchmarks (p 100) and also on the Hopf Bifurcation (p 21). Since Gurobi is commercial proprietary solver, we cannot inspect the details of its sophisticated implementation. Gurobi may resort to an enumeration/brute-frre approach, which could be faster than spending time to calculate lower bounds in the BnB tree. This being said, OKRidge is still competitive with Gurobi the low-dimensional setting, and OKRidge scales favorably in high-dimensional settings.
5 Conclusion
We presented method for optimal sparse ridge regression that leverages a novel tight lower bound on the objective. We showed that the method is both faster and more accurate than existing approaches for learning differential equations key problem scientific discovery. This tool (unlike its main competitor) does not require proprietary software with expensive licenses and can have significant impact on various regression applications.
Code Availability
Implementations of OKRidge discussed in this paper are available at https: com/ jiachangliu/DKRidge
Acknowledgements
The authors gratefully acknowledge funding support from grants NSF IIS-2130250, NSF-NRT DGE- 2022040, NSF OAC-1835782, DOE DE-SC0023 194, and NIH/NIDA RO1 DA054994, The authors would also like to thank the anonymous reviewers for their insightful comments.
References
[1] Achterberg, T. Koch, and A. Martin. Branching rules revisited. Operations Research Letters, 33(1):42-54, 2005.
[2] D. Applegate, Bixby, Chvátal, and W. Cook. On the solution of traveling salesman problems. Documenta Mathematica, pages 645-656, 1998.
[3] M. ApS. Mosek optimizer API for python. Version, 9(17)66-4, 2022.
[4] A. Atamturk and A. Gómez. Safe screening rules 10-regression from perspective relaxations.
In International Conference on Machine Learning, pages 421-430. PMLR, 2020.
[5] A. Atamtürk, A. Gómez, and Han. Sparse and smooth signal estimation: Convexification of 10-formulations. Journal of Machine Learning Research, 22:52-1, 2021.
[6] A. Beck and Y. C. Eldar. Sparsity constrained nonlinear optimization: Optimality conditions and algorithms. SIAM Journal on Optimization, 23(3): (480):44001599 2013.
[7] P. Belotti, C. Kirches, S. Leyffer, J. Linderoth, J. Luedtke, and A. Mahajan. Mixed-integer nonlinear optimization. Acta Numerica, 22:1-131, 2013.
[8] D. Bertsekas. Convex optimization theory, volume 1. Athena Scientific, 2009.
[9] D. Bertsimas and W. Gurnee. Learning sparse nonlinear dynamics via mixee-integer optimiza- tion. Nonlinear Dynamics, Jan 2023.
[10] D. Bertsimas, A. King, and R. Mazumder. Best subset selection via modern optimization lens.
The Annals of Statistics, 44(2):813 852, 2016.
[11] D. Bertsimas, J. Pauphilet, and B. Van Parys. Sparse regression: Scalable algorithms and empirical performance. Statistical Science, 35(4):555-578, 2020.
[12] M. Best and N. Chakravarti. Active set algorithms for isotonic regression; unifying framework. Mathematical Programming, 47(1-3):455-439, 1990.
[13] Blumensath and M. E. Davies. Gradient pursuits. IEEE Transactions on Signal Processing, 56(6):2370-23822 2008
10

---

[14] T. Blumensath and M. E. Davies. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis, 27(3):265-22 2744, 2009.
[15] P. Bonami, J. Lee, S. Leyffer, and A. Wächter. More branch-and- bound experiments in convex nonlinear integer programming. Preprint ANLMMSS- P1949-0911, Argonne National Laboratory, Mathematics and Computer Science Division, 91, 2011.
[16] L. Boninsegna, F. Nüske, and C. Clementi. Sparse learning of stochastic dynamical equations. The Journal of Chemical Physics, 148((4):241723, June 2018.
[17] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1): 2011.
[18] S. Boyd, L. Xiao, and A. Mutapcic. Subgradient methods. lecture notes of EE392o, Stanford University, Autumn Quarter, 2004:2004-2000 2003.
[19] S. L. Brumton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 1 (1((11 13((1))3932-39953)
2016.
[20] S. Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3-4):231-355 2015.
[21] M. Busing. Monotone regression: A simple and fast o(n) PAVA implementation. Journal of Statistical Software, 102:1-25, 2022.
[22] T. Cai and Wang. Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Transuctions on Information Theory, 57(7):4680-4688 2011.
[23] P. M. Camerini, L.. Fratta, and F. Maffioli. On improving relaxation methods by modified gradient techniques. In Nondifferentiable Optimization, pages 26-34. Springer Berlin Heidelberg, 1975.
[24] V. Carbone and P. Veltri. Relaxation processes in magnetohydrroynamiin model. Astronomy and Astrophysics, 259(1))35--372, June 1992.
A triad-interaction
[25] K. Champion, P. Zheng, A. Y. Aravkin, S. L. Brunton, and J. N. Kutz. A unified sparse optimization framework to learn parsimonious physics-informed models from data. IEEE
Access, 8: 8:169259-169271, 2020.
[26] P. L. Combettes. Perspective functions: Properties, constructions, and examples. Vet-Valued and Variational Analysis, 26(2):247-264, 2018.
[27] B. de Silva, K. Champion, M. Quade, J.-C. Loiseau, J. Kutz, and S. Brunton. Pysindy: A python package for the sparse identification of nonlinear dynamical systems from data. Journal of Open Source Software, 5(49):2104, 2020.
[28] H. Dong, K. Chen, and J. Linderoth. Regularization relaxation: A conic optimization perspective of statistical variable selection. arXiv preprint arXiv:1510.0660 2015.
[29] E. R. Elenberg, R. Khanna, A. G. Dimakis, and S. Negahban. Restricted strong convexity implies weak submodularity. The Annals of Statistics, 46(6B):3539-3566 2018.
[30] A. Eriksson, T. Thanh Pham, T.-J. Chin, and I. Reid. The k-support norm and convex envelopes of cardinality and rank. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3349-3357, 2015.
[31] U. Fasel, N. Kutz, B. W. Brunton, and S. L. Brunton. Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 478(2260), Apr. 2022.
[32] A. Frangioni and C. Gentile. Perspective cuts for a class of convex 0-1 mixed integer programs. Mathematical Programming, 106(2):225-236, 2006.
[33] Giselsson and Boyd. Linear convergence and metric selection for douglas-rachford splitting and admm. IEEE Transactions on Automatic Control, 62(2):532-544, 2016.
[34] O. Günlük and Linderoth. Perspective reformulations of mixed integer nonlinear programs with indicator variables. Mathematical Programming, 124(1):183-205) 2010.
[35] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[36] W. H. Haemers. Interlacing eigenvalues and graphs. Linear Algebra and its Applications,
226:593-616, 1995.
11

---

[37] S. Han, A. Gómez, and A. Atamtürk. The equivalence of optimal perspective formulation and Shor's SDP for quadratic programs with indicator variables. Operations Research Letters, 1955):155119, 2022.
[38] H. Hazimeh and R. Mazumder. Fast best subset selection: Coordinate descent and local combinatorial optimization algorithms. Operations Research, 68(5):1517-1337, 2020.
[39] H. Hazimeh, R. Mazumder, and A. Saab. Sparse regression at scale: Branch-and- haun- rooted in first-order optimization. Mathematical Programmingg 196( (6661))3773888 2022.
[40] P. Jain, A. Tewari, and P. Kar. On iterative hard thresholding methods for high-dimensional m--estimation. Advances in Neural Information Processing Systems, 27, 2014.
[41] K. Kaheman, J. N. Kutz, and S. L. Brunton. SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 476(2242), Oct. 2020.
[42] A. A. Kaptanoglu, B. M. de Silva, U. Fasel, K. Kaheman, A. J. Goldschmidt, J. Callaham, C. B. Delahunt, Z. G. Nicolaou, K. Champion. J.-C. Loiseau, N. Kutz, and S. L. Brunton, Pysindy: A comprehensive python package for robust sparse system identification. Journal of Open Source Software, 7(69):3994, 2022.
[43] J. Liu, C. Zhong, B. Li, M. Seltzer, and C. Rudin. FasterRisk: Fast and accurate interpretable risk scores. In Advances in Neural Information Processing Systems, 2022.
[44] J. Liu, C. Zhong, M. Seltzer, and C. Rudin. Fast sparse classification for generalized linear and additive models. In Proceedings of Artificial Intelligence and Statistics (AISTATS), 2022.
[45] N. M. Mangan, J. N. Kutz, S. L. Brunton, and J. L. Proctor. Model selection for dynamical systems via sparse regression and information criteria. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473/2204):2017000, Aug. 2017.
[46] D. A. Messenger and D. M. Bortz. Weak sindy for partial differential equations. Journal of Computational Physics, 443: 10525, 2021.
[47] D. A. Messenger and M. Bortz. Weak sindy: Galerkin- based data-driven model selection. 2021,
Multiscale Modeling & Simulation,
[48] T. Moreau, M. Massias, A. Gramfort, P. Ablin, P.-A. Bannier, B. Charlier, M. Dagréou,
T. Dupré la Tour, G. Durif, C. F. Dantas, Q. Klopfenstein, J. Larsson, E. Lai, T. Lefort, B. Malézieux, B. Moufad, B. T. Nguyen, A. Rakotomamonjy, Z. Ramzi, J. Salmon, and S. Vaiter. Benchopt: Reproducible, efficient and collaborative optimization benchmarks. In Advances in Neural Information Processing Systems, 2022.
[49] B. K. Natarajan. Sparse approximate solutions to linear systems. SIAM Journal on Computing. 24(2):227 234, 1995.
[50] D. Needell and A. Tropp. Cosamp: Iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26(3):301-311, 2009.
[51] D. Needell and R. Vershynin. Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit. IEEE Journal of Selected Topics in Signal Processing, 4(2):310-316, 2010.
[52] N. Parikh, Boyd, et al. Proximal algorithms. Foundations and Trends Optimization, 1(3):127-33, 2014.
[53] M. Pilanci, M. J. Wainwright, and L. El Ghaoui. Sparse learning via boolean relaxations. Mathematical Programming, 151(1):63-88) 87, 2015.
[54] S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz. Data-driven discovery of partial differential equations. Science Advances, 3(4) 16ex)616,,, 2017.
[55] M. E. Sander, J. Puigcerver, J. Djolonga, G. Peyré, and M. Blondel. Fast, differentiable and sparse top-k: convex analysis perspective. In International Conference on Machine Learning, pages 29919-29936. PMLR, 2023.
[56] J. Tropp. Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10):2231-2244 2004.
[57] R. Vreugdenhil, V. A. Nguyen, A. Eftekhari, and P. M. Esfahani. Principal component hierarchy for sparse quadratic programs. In International Conference on Machine Learning, pages 10607-10616 16. PMLR, 2021.
12

---

[58] S. Wiseman and A. M. Rush. Sequence-tee -e-nneee----eeeenee learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1296-13088 Austin, Texas, Nov. 2016. Association for Computational Linguistics.
[59] W. Xie and X. Deng. Scalable algorithms for the sparse ridge regression. SIAM Journal on Optimization, 30(4):3355-33366 2020.
[60] G. Yuan, L. Shen, and W.-S. Zheng. A block decomposition algorithm for sparse optimization. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 275-285, 2020.
[61] T. Zhang. Adaptive forward- backwrd greedy algorithm for learning sparse representations. IEFE Transactions on Information Theory, 5767))4689-4400 2011.
[62] X. Zheng, X. Sun, and D. Li. Improving the performance of MIQP solvers for quadratic programs with cardinality and minimum threshold constraints: A semidefinite program approach. INFORMS Journal on Computing. 26(4):690-703, 2014.
[63] J. Zhu, C. Wen, J. Zhu, H. Zhang, and X. Wang. A polynomial algorithm for best-subset selection problem. Proceedings of the National Academy of Sciences, (7(52):33117-33111 2020.
13

---

Appendix to OKRidge Scalable Optimal k-Sparse Ridge Regression
Table of Contents
Related Work
B Background Concepts
17
C Derivations and Proofs
18
Proof of Theorem
18
Proof of Theorem 3.2
20
Proof Theorem
23
Proof Theorem 3.4
25
C.5
Derivation of g(a, b) max, ac via Fenchel Conjugate
30
D
Existing MIP Formulations for k-sparse Ridge Regression
32
E
Agorithmic Charts
34
Overall Branch-and-a bound Framework
34
Fast Lower Bound Calculation
F.3
Rehne the Lower Bound through the ADMM methad
37
1.4
(Optional) An alternative to the Lower Bound Calculation Subgradient Ascent
and the CMF method]
Beam Search Algorithm
39
E.6
Branching
41
E.7
A Note on the Connection with the Interlacing Eigenvalue Property
41
F
More Experimental Setup Details
42
FT
Computing Platform
42
Baselines, Licenses, and Links
42
Differential Equation Experiments
G
More Experimental Results Synthetic Benchmarks
44
G.
Results Summarization)
44
G.2
Comparison with SOS1, Big-M, Perspective, and Eigen-Perspectiv Perspective Formulations with different l Regularization
46
G.3
Comparison with the Big M Formulation with different M Values
106
G.4
Comparison with MOSEK Solver, SubsetSclection CO, and LOBNB
143
G.5
Experiments on the Case >>
173
G.6
Comparison between Fast Solve and ADMM
176
H
More Experimental Results Dynamical Systems
177
Results Summarization
177
HI2
Direct Comparison of OKRidge with MIOSR
178
H.3
Comparison of OKRidge with SOSI, Big-M, Perspective, and Figen- Perspective
Formulations]
179
H4
Comparison of Our Beam Search with More Heuristic Baselines
180
H.5
Comparison of OKRidge with MIPs Warnstarted by Our Beam Search
182
H.6
Comparison of OKRidge with MOSEK solver, SubsetSelectionCIO, and LOBNB
183
14

---

A Related Work
Sparse Identification of Nonlinear Dynamical Systems: The Sparse Identification of Nonlinear Dynamical Systems (SINDy) framework [19] has been widely adopted for discovering dynamical systems from observed data. The basic framework consists of approximating derivatives, picking a library of features based on positional data, and performing sparse regression on the resulting design matrix. SINDy has expanded to solving PDEs [54] 466 or implicit equations [41, noisy or low data
settings [3 47 and constrained problems [25] 9]. The basic framework can be summarized as
standard regression problem:
X = (X)=
(29)
where the goal is to find E = B Ba). with d being the number of dimensions of the dynamical system, X being the observed data, and θ is map from observed data candidate functions. The problem is typically solved independently across dimensions, yielding d sparse regression problems.
Many methods in the SINDy framework solve regression problems with greedy backwards selection approach. These methods have been shown to outperform LASSO (25) and Orthogonal Matching Pursuit [16] in learning dynamical systems. The backwards selection approach typically solves non-sparse regression problem. and then either removes all small coefficients below a threshold, or remove one feature time until a desired sparsity level is reached [54]. Although these methods have performed well with rcasonable run times, they struggle to identify true dynamics as the number of candidate features increases.
Advances in MIP formulations have led to solving SINDY problems with greater success and ability to verify optimality [9; however, they tend not to scale as well. The method we present also verifies optimality and is much faster than the MIP formula, particularly for larger number of features p.
Heuristic Methods: Greedy methods aim to solve the problem to near-optimallty, with few guar- antees. One direction is greedy pursuit (i.e., forward selection), where one coordinate at a time is added until the required support size reached [56 13
22 29] Another direction is iterative thresholding, where gradient descent steps alternate with projection steps to satisfy the support size requirement
50, 51
59 57]. Other approaches include randomized rounding on the solution of
the boolean-relaxed problem [53] swapping features 661.6 38 63 44, solving smaller problem
optimally on a working set [60]. These heuristic solutions may greatly improve the computational speed of MIP solvers when used as warm starts. Typically, the heuristic algorithm starts from scratch each time that it runs, which is slow when running it repeatedly throughout the BnB tree, Our insight is that the search history of heuristic methods in previously solved nodes can be used to speed up the heuristic method in the current node.
Optimal Methods and Lower Bound Calculation: In order to certify optimality for this NP- hard problem. need to perform branch-and-boume and calculate the lower bound for each node in the BnB tree. For the lower bound calculation, early works include the SOSI formulation or the big-M formulation [T0 9.. Both formulations
be implemented in a commercial solver. However, the SOSI formulation is not scalable to high dimensions, while the big-M formulation is sensitive to the hyperparameter "M" used to balance scalability and correctness. To circumvent this problem, SubsetSSlectioonCIo [I] formulate the least-squares term using the Fenchedl duality with dual variables. After the formulation, SubsetSelectionC1O applies callbacks to add cutting planes to get a lower bound. Although OKRidge also applies the Fenchel duality, we apply the Fenchel duality the fa regularization term. We are solving the problem in the feature space, while SubsetSelectionC10 solves the problem in the sample space. As pointed in (3,, the branch-and-cut method in SubsetSelectionCI0 II runs slow when the E2 regularization
small. Recently, the 59 5 371 of the l2 term has also been used. Through convex relaxation, the lower bound can be obtained by solving a quadratically constrained quadratic program (QCQP) with rotated arcond-order cone constraints. However, the conic formulation is still computationally intensive for large-scale datasets and has difficulty attaining optimal solutions. Our work builds upon the perspective formulation, but we propose an efficient way to calculate the lower bound through Fenchel duality. Another line of works focuses on optimal perspective formulation 62 28 37 This requires solving semidefinite programming (SDP) problem at each node, which has been shown not scalable to high dimensions [8]. What is more, MI-SDP is not supported by Gurobi. Our work is related to the optimal perspective formulation, but we set the
perspective formulation (32 34
62 28
15

---

diagonal matrix diag(d) = Amin ( (X' (X"X)I, this is not the optimal choice. is a good approximation and is supported by Gurobi through the QCQP formulation discussed above. Lastly, there is recent work called 10bnb [9 which also implements a customized branch-and-bound framework without the commercial solvers. However, 10bnb is solving the lo-regularized problem, which is not the same as the co-constaaened problem. To get a solution with the specified sparsity, 10bnb needs solve the problem multiple times with different lo regularizations.
I We call this the
formulation. Although
16

---

B Background Concepts
Definition B.1 (Strong Convexity (0)). function f :P R is strongly convex with parameter
> O if it satisfies the following subgradient inequality:
f(y) 2 f(x) + Vf(x)" x)
=11
(30)
for æ, E IRP If the gradient v. does not exist but the subgradient exists, then we have
f(y) 2 f + g y x
2 y/ 2112
(31)
for x; y E IR,, where E o and f is the subdifferential of f. Definition B.2 (Smoothness 20 A continuously differentiable function f R2 > R is smooth
with parameter 3 >> 0 if the gradeint V is 3-Liisccitz. Mathematically, this means that:
VV (x) vf(y) 2 S 3| 2ll
(32)
for æ. y E IRP.
Lemma B.3 (Smoothness Property (0) Lel f RP- R be a B-smooth function. Then, we have the following sandwich property (f) is "sandwiched" between two quadratic functions):
3 11y 2: (33)
f(x) + vx(x)) x) + 2 11 x 2 f(v) 2 f(z) + Vf(x) x)
2
for x. y E I.P
Definition B.4 (k-sparse Pair Domain 29 domain 22 C RP Rp is a k-sparse vector pair domain if it contains all pairs of k-sparse vectors that differ in at most ht entries, i.e.
2x = ((x, y) E R R':
||= < k; llyllo S k. | - yll S k)-
(34)
Definition B.5 (Restricted Strong Convexity 29 function f RP - IR is restricted strongly convex with parameter > on the k-sparse vector pair domain f2 if it satisfies the following inequality:
f(y) 2 +Z))) x) 2 x11
(35)
for (x, y) E 92k-
Definition B.6 (Restricted Smoothness 29 ). A function f R. E is restrictedly smooth with
parameter / >> O on the k-sparse vector pair domain 2g if it satisfies the following inequality:
f(x) + V((=)>y - x) + 2 fly x12 f(y)
(36)
for (x, y) E 9g-
Lemma B.7 (Gradient of Ridge Regression). At B E RP, the gradient of our ridge regression function is given by:
V Lridge = 2X" (X3 y) + +22,
(37)
for B E RP.
Proof.
(rrddee =
228 3) -V - V(2y x22)TXx3) + Xu V
-2X"X8-2X''= - 22200
=2XT - y) 2
17

---

Derivations and Proofs
C.1 Proof of Theorem
Theorem 3.1 If we reparameterize с = Io+x -
Qx7) y with a new parameter ?, then Problem 10 is equivalent to the following saddle point optimization problem:
max min ridge
Csaddle
x (7,, 2) s.l.
Σ S 23 2 10,
(38)
j=1
where
[soddle ridge (x,,2) -1
- QQx X +
X y- Qx7)² diag (x) Qx7),
(39)
and dzag(z) a diagonal matrix with on the diagonal.
Before we give the proof for Theorem 3.1 we first state one useful lemma below:
Lemma C.1. For an optimal solution c* to Problem 40 there exists some 3 such that c*
A2+A (X y Qx
Without using this lemma, there is some concern with reparametrizing with new parameter y when À = Amin (X" X). In this case, X x XI becomes singular and our reparametrization trick could miss some subspace. This issue of having X >I being singular would lead the concern that Problem (39) is relaxation of Problem (40), not equivalent. Fortunately, due to this lemma, the subspace which we miss via the reparametrization trick will not prevent Problem (39 from achieving the same optimal value as Problem (40).
We now prove this lemma by the method of contradiction.
Proof. Suppose the optimal solution to Problem (40) cannot be written by Equation (4 Through the nullit theorem linear algebra, we can find di E col(Qu) and d2 E ker Qx): so that x (X d1 + d2), where col(.) denotes the column space of matrix and ker(.) denotes the kernel of a matrix. If we let 33 = ad2 for some real value a, then the inner optimization of Problem (40) (ignore the last term involving 2j 's for now since 33 and z are separable in the objective function) becomes
(er(Qx) ). d2 *
CRonchal ridge
A 2X 2x
2
- - ((22 A) \2 1 (XT di + d2) 2X y)²
=
+ d2) -2x y)
= 2(d d3
Q2Q: + 2d B 2d 3.
Because 3 = adg, we have 3 E ker Q2) and B dj- Thus, the first two terms become 0, and we left with (Fenchel
-2a da d2 For the inner optimization of Problem (40), because we want to minimize over B, we can let a + xo. and we will get FFachhl
Lidge
2dl [
À - -c.. This is obviously
not the optimal value for Problem 440) (a simple counter-example is to let A XT y. and
easily show that min1,,2 ridge- X (B. z, e) is finite). Therefore, d2 must be 0, and the optimal solution
(ienchel
2 to Problem (40) must be represented by some &3 through the equation X T Qx3). O
Now that have this lemma, proofs so that readers can understand the logic from different angles. The first proof is based on using the lemma directly; the second proof is based on the sandwich argument.
prove Theorem 3.1 rigorously. We provide two alternative
Proof 1
Proof. For notational convenience, let use D, {z Eラ= 23 S k, E 10, 1]} as a shorthand for the domain of z.
18

---

Recall that Problem (T) is the following optimization problem:
max min CTenchel ridge
(3, 2, c)
(40)
3
subject to
Σ 2
2j [0,1]
where - (3, c) 33 2y x + (A2 + A) E)= (3 - 4
For any fixed if we take the derivative of ridge A (3, x, c) with respect to B and set the derivative to O, we obtain the optimality condition for 3:
(41)
x x Xx-" then 3 = y If we reparametrize with new parameter Y and let c satisfies the optimality condition for 3 in Equation (41) Therefore, whenever we reparametrize with с X XX - (x" X - >f) we can just let / = 7, and this ensures us that we can achieve the inner minimum of Equation (40) with respect to B.
Then, we can reduce Problem (40) to Problem 38 in the following way:
max min Fenchel 8 ridge-
(3, 2, c)
subject to 20 E [0
max min
rige
Fanchel
(3,
use Dr as the shorhand for the domain of
sED..3 3
2
min 3"QxB-2y" x3 X (A2 + A)
Σ
(Byc)
22)
2ED,,3
j=1
plug in the details for Feencel (B, 2, c)
Lidge A
c zラ)
= max min sED.
Y' 2n" 2y X7 + (A2 + A)
hg
j=1
# letting 3 ry achieves the inner minimum for B
2''Q7-13 - 2 + (A2 + A)(7 c eea diag(z)e)
max min
sED, # write the summation in vector forms; diag(z) denotes diagonal matrix with the diagonal
2
min T² #'Q77 - 2-y" Xx + (A2 + A)
(X"
x"y--Ax7)
7 =ED.
A2 + A
1 2
2
(xTy Qx7) diag(z)
de + (X y Q
4 A2
+
# apply the reparametrization c 2 XXx Qxy).
# remember that due to Lemma we will not miss the optimal solution c²
max min 7 ZED;
2ny "0p0"x777" Q=A (xT y Qxy)
A2 x (X y Qx7)* diag( (x)aa(()(("()) Qx7)
# distribute X2 + > inside the parentheses
1
max min zED,
''Q 2yj" X7 + 22 x x -22" 22
(X Qxy)" diag(z))))))))) (x" - X +
# distribute 2y inside the parentheses
max min ZED, maxmin
-QTQAy X2 + X
(XT y - -x>)) (
Qx7)
# simplify the linear algebra
1
-
(x" Q^7)² (iag(()(('') y - Qx7)
Ag + x
subject to E)= 2j S k, E [0.1]
19

---

Csadde max min ridee
(7, 2)
# subject to Σ/= 2g S k, E [0, 1]; apply the definition of Csaudle ridge x (7, 2)
This completes the derivation for Theorem 3.1
Proof
Proof. We will show max. min
FFenchel ridge (B, c, c)
max, mitly nidge A (7, and
Laaddle
P max. ming.2 ridge X (B, c) 2 max, min, ndge- > (7, 2) (both under the constrains Σ 2j S 2, [0, 1)) and therefore establish the equivalence of optimal values.
For the first inequality, let f², ,.e be the optimal solution to Problem 440 with the relation e²
2 (XTv y - Qxf*) (this is always true because of Lemma C.1. Furthermore, let y" = Then
Aa+A
we have
max min Abenchel 2
(33 2, c) = min ridge - ridge
Fenchel (B", , c²) = min ridee
x2)" S max min ridge x (7,, 2)
7
The derivation for the second equality roughly follows the derivation in the first proof we gave.
For the second inequality, let y be the optimal solution Problem 38 Let A2+A (XT y Qx9).
Furthermore, let B -y (notice that this & satisfies the optimality condition of Problem 40 Then we have
max min aaddlee ridge
ssaddle (7, z) min ridge
> (9,, x)
min Fenchel Iidge
(B, 2, c)
-inn ridge- (B, x, é)
Fenchel
$ max min 8,2
ridge
(Fenchel (3, x, c).
Similarly, the derivation for the second equality roughly follows the derivation (but this time in reverse order) in our first proof.
Using these two inequalities, we can establish the equivalence of optimal values between Problem 40 (Fenchel
and Problem (38 i.e., max, ming, ridge- (3, 2, c) maxy ming Lidge (7,
C.2
Proof of Theorem
Theorem
3.2
The function h(y) defined in Equation
(14 is lower bounded by
h((r) 2 - - A2 +
X y Qx7112.
(42)
Furthermore, the right-hand size of Equation where in this case, h((r) on the left-hand side of Equation (42) becomes
42 is maximized if f argmin Cridge (a),
1(()) Fridge (A) (22 + A (<(
where SumBottom,-Kk x denotes the summation of the smallest l terms of a given set.
Proof. We first derive Inequality
42
1(())= min Asaddle
ridge- -000--177- x),
# #D= = (2 k, (0,
=ED,
1
min sEP-
TTTQ
1T (1-)) diagt (z) (XT Qx7)
do +
# apply the definition of Lsaddle (7, 2)
min xED,
- "'QA"
d' diag(z)d
# let d = y x
d2 + x
20

---

min ZED,
Qux
1
Σ
dzzz # use summattin notation instead of matrix notation
A A
2 2-7-Qx
d
A2 + X
j=1
# notice that 2j E 10, 1}; letting 2j -forr each j gives lower bound
TTQYY
dº
# use matrix notation stead of summation notation
A2 + X
= - --7 X2 + A X
(X' -
apply d = (X" Q27))
This completes the derivation for Inequality 42)
The right-hand side of Inequality 42 is maximized if the gradient with respect y 0.
The gradient can be derived and simplified as:
0
(-
1 (X y Qx7)" QA7)) (ax'y--Qr)
0y
# gradient of the right-hand side of Inequality 42
0
(-TTQAT
A + (g 1x"xX" y xQxx+xx>>>> 7 QQa)))
# expand the quadratic term
-2Q
A2 + (-2Q) yx'''=('@00)
# apply the differential operator to each term
2
da + 1 x-
XT y
2Q%
# change the first term from Qx7 Q37 because Q2 =x=X-X XI is symmetric
O ((12 + A)y X y Qx7))
A2 + A
pull the common factor du+> QJ out of the parentheses
Q ((22 + A)y x (x" X -i))))
A2 +
# substitute Q X - Al for the last term
o (Q2 X (x"x \I)))
d + A
# add identity matrix in the first term
A2 X Q ((-XTy + (x + \21)))
# add an identity matrix in the first term
Therefore, we set (XT X + X2T) = XTy y. then the gradient of the right-hand side of In- equality 42 will be and the optimality condition is achieved. However, this means that TTy y. which gives us the minimizer the ridge regression loss Liidge (+).
A2T) T
Lastly, we show that when & )
argminy Fridge (7), h(()) Lridgo (4) + (A2 +
First, let derive a useful property. We claim that when Ý argminy Lridge (7), X Qx7 (A2 A)5. Below, prove this by showing that when subtract the two terms, we get 0.
X Qx (he A)4
=X y - (x= - -11 - (A2+ 2)
substitute Q = - XI # add the identity matrix I # aggregate the relevant terms in front of Fy
=XT y (xTx-X - -My- -
= -x y (x"x + xelly
21

---

=X' y (X" + 111) 121) #4 -rrgmmn,, Crdge = =X"x + A2I =X"y y -0.
# cancel oul X A2
Therefore, we have X y QxΩ = (A2 + >)S when y = argminy Fridge With this property, we can derive the final simplified formula for h(f)):
1(())=
min =ED=
caaddle (5)
# #D= = (z12z=1 3j S 2, [0. 1]
min sED=
-2-
d2 + (X y- Qx4) diag(z)) Qxf))
# aaply the definition of Lsaddle > (5/, 2)
min zED,
Qx²
A2 + A (22 + 2) (dag(()(())) 2)
# apply the property X y
Qxタ (22 + 2)7
min ZED,
Qx (A2 + 2)5² diag(z)
cancel out 22
min zEDz
- x (A2 A)5 dzag(z)
apply Qx X X XI
min
-x" x - 2n7 - (A2 + A)
7/23
sED;
j=1 # use summation instead of matrix notation
=- --- - -11 - (A2 + X) max
2
DED
move min(.) inside the negative sign and convert it to max
=- - -113 - ()2 + ^)T ooTopummt {
TopSum (-) denotes summation over the top terms
p
- A13 (A2 2)
3 -ottoomSumpp-tt x{93))
f=1
# BottomSum (.)
- denotes summation over the bottom p - terms
x - (A2 + A) Boltom k (65) (A2 + A)
j=1
# distribute into the parentheses
-- -x X1) (A2 + BottomSuu x63) (A2 + 7
# use matrix notation instead of sunmation notation
We note that -TXXX x - Ar - (A2 T) = fridge(F)). (f)) To show this, we subtract the two
terms and prove that the subtraction equal to below:
-x" ^1y (A2 + - Lridgo (f)
=- - A1y - (12 (22+))9 - (4" X" X 2y X + >22
# plug the formula for Cridge(F)
=-2(fT + x21 - x
simplify the linear algebra
= =-2 --2(FT(X")) x + (X X + -
xxA)
# #lug in the equation 7 = (X + - because &y is the minimizer of Fridge (.)
2 TT 8/ X4)
# cancel out X X + AaI
=0.
(12 + + Lridge we have: Using the property that - -X X A1 -22
h(9) -x²= X \1 (A2 + 1)7 5 (A2 + Bottooom
22

---

ridde (7) + (A2 + A/Bootom-Sump-ee9
This completes the proof for Theorem 3.2
C.3
Proof of Theorem 3.3
Theorem
3.3
Let F(7) = 7-7)=-70000 Qx7 and G(p) = Agth
1 & ((p))) Then the solution for the
problem
argminy (()) + 1 Q 1A +
ppppp''' -x"' y q'13
++ = ( I+ -
p¹ 9¹) (x"
(43)
p
Furthermore, let a = X + 1 q' and I be the indices of the top terms of {layl}. The
solution for the problem p² = argmin, G(p) 1/2 1 p- 1--x1p y + 4 is is = sign(a,) -g.
where
① argmin
(2y - bj)² s.t. 2 11 if ac 2 1011
(44)
Tj
|,
W/
1
p()2+X)
otherwise
wi
Problem is an isotonic regression problem and can be efficiently solved in linear time 12 21y.
Proof. Part I. We first show why Equation 43 is the analytic solution to the first optimization problem y = argmin, F(7) + QQx
X y + 1'.
The loss function we want to minimize (writing out F(7) = - Qxa explicitly)
p 1xx -'-x'' 1'11 2 -7 -- ( X"y+ Q
-=Q
XT 1'1l2).
# write out the square term as a sum of three terms
This loss function is minimized if we take the gradient with respect to y and set it to O:
0 2Qxx+ P (20 2Q' - X q"))
take the gradient w.r.f. 7 and set it to 0; ignore the last term because is a constant
=QQx / Q Q Q (p' X" 9²))
# distribute" 2 inside parentheses
x" y
# Q is symmetric # take out the common factor QJ # multiply by identity matrix # aggregate the coefficient for "
+ (p' x" q')))
-QT + + +p' x y + +q'))
= =Q[((2I + +0x7+++++++++ p(p' - -X y + +())
- ( + (p' x" y + q')).
# take out the common factor p
sufficient condition for the gradient to be 0 is to have
2 Qx) (p' - X"y )¹) = 0.
p
If we solve the above equation, we obtain Equation
43
2 Qx) (x pd d¹)
Part I.. Next, we show why [+1 = sag) 0y together with Equation 44 gives us the solution to the second optimization problem p argmin, G(p) 0 +p x 11..
23

---

If we write out G(p) =
1 SumTop, (Pp))) explicitly and use a = y - 0 q" as
x shorthand to represent the constant, the objective function can be rewritten as:
p = argmin
SumTop,
flppp al
X2 X
P
argmin
x + A) SumTop;((p))) + lp 11
p
# multiplying by doesn't I chunge the optimal solution
According Proposition 3.1 (b) in 30 the optimal solution p' has the property that sign(p)) sign(ay).
Inspired by this sign-preserving property, let us make a change of variable by writing Pj = sign(a)) (gg). Due to this change of variable, our optimal solution p'+1 can be obtained by solving the following optimization problem:
p = sign(a,) is
where argmin p(A2 + X)
SumTop,(( ((v7)) + Σ (0--1ay1)2
(45)
2=1
Note that in Problem (45). if we add the extra constraints 8y 2 O, this will not change the optimal solution. We show this by the argument of contradiction. Suppose there exists optimal solution and an index j with 8j < O. Then, if we flip the sign êj. then the objective loss Problem (45) is decreased. This contradicts with the assumption that ô is the optimal solution. Therefore, safe to add the extra constraints by 2 0 into Problem (45) without cutting off the optimal solution.
Thus, optimal solution p*+1 becomes:
p sign(a;) (az) viy
where e = argmin p(12 + A)
( + Σ (uz layl)2 s.t. oj 2 0 for Vj.
(46)
Next, according to Proposition 3.1 (c) 30
we have 6; 2 0j if [a: 2 |ar| in Problem (46). Thus, add these extra constraints into Problem (46) without changing the optimal solution and p'+! becomes:
Pj sign(a;))
2
where e argmin p(>₂ X)
SumTop, (( + (1g - 1ay1)3
(47)
Uy for Vj, and vi 2 Uy if |a layl- for Vi,j-
Note that in Problem 47 because of the extra constraints |a,| 2 layl for Vi. J. we can rewrite SumTop, ((3)))
Ejer us where I is the set of indices of the top terms of {aay13 Then the optimal solution p becomes:
p,+ sign(a;) 15
where argmin p(12 + A)
2
Σ S3
P
Ug
lay)2
(48)
JEJ
1=1
s.t. By 2 for Vj. and 2 0 if |as1
layl for Vi,
Next, we drop the constraints vj 2 for vj as this does not change the optimal solution, using the same argument of contradiction we have shown before. Therefore, our optimal solution p'+1 can be obtained in the following way:
sign (a;) if
where e = argmin p(3e A)
2
Σ
oz +
(1 - 1ay1)²
(49)
JEJ
1=1
s.t. U 2 22 if a. 2 la,1 for Vi.
24

---

We can obtain the formulation in Problem 44 follows:
by performing some linear algebra manipulations as
2
argmin
Σn
o
20 Dj - |ayl)² j=1
p()2 X)
JEJ
argmin
2
1 + (vy lay1)²)
+E(r
(1g 1 (15--lay) layl)2
(0)2 + X)
jEJ
JEJ # divide the summattin into two groups based on the set T
argmin
1j p(12 + +) +1
2 - 2|0,|u, + las1)
+ (o,--1ay))
JEJ
JEJ
combine the coefficients for vz
= =argmin
+1)
1/j
agj
|a,l" l2
lag1
+ (0s-layl
p(>2 + 2)
+
JEJ
p()++Aj
JEJ
# complete the square
2
2
aaj
+1
1/
+
Σ
la,-
-
p(22 + 2)
jEJ
p(a++Aj
j
# get rid of the constant terms which do not affect the optimal solution
|az) )
|a=1
arggmnn
Wi
Σ
wj
By
w/
W/
JEJ
J&3
# Wi
+ +1
J and wy = $
x0a+ti
argmin
wy (v
b,)2.
let by [a and combine the two summations into a single summation
Using this new property, obtain the final result stated in the theorem for optimal solution pi+1 to the second optimization problem:
Pj sign -6)
where 6 argmin Σ wy (2) -,,2 s.t. vs 2 Dj if lo:1 2 |ayl for Vi,
(50)
if I J
|a,1 W/
Wy
bj =
otherwise
p()2+A)
Therefore, the task of evaluating the proximal operator is essentially reduced to solving an isotonic regression problem. We would like to acknowledge that the connection between Sum (.p) and isotonic regression has also been recently discovered by (55) However, the context and applications are totally different. While [55] discovers this relationship in the context of training neural networks, we find this connection
the context of ADMM and perspective relaxation of k-sparse ridge regression. Mathematically, our objective function is also different as our objective function contains a linear term of p in the proximal operator evaluation.
C.4 Proof of Theorem
3.4
Theorem 3.4 Let us define a k-sparse vector pair domain to be Qs == {(x (x ER ||0||o S
k. Il hi ||a -yllo S k). Any Mj satisfying f(y) S f(x) + (x:) (y-a)) (y- ly 2
all (x, y) Ω is called restricted smooth parameter with support size and any m2k satisfying
25

---

x) man l for all (a, y) E S22x is called a restricted strongly convex parameter with support size 2k. If 3 is our heuristic solution by the beam-search method, and 3 is the optimal solution, then:
f(y) 2 f(z) + V
Cridge S Cridge s - --aa//Maa Criddee
(51)
Proof. The left-hand-side inequality (3)) < Cridgo (3) is obvious because the optimal loss should always be less than or equal to the loss found by our heuristic method. Therefore, we focus on the right-hand-side inquality Lridge (3) S (1 e -- MM (3"))
We first give an overview of our proof strategy, which is similar the proof in 29 but our decay factor m2x M in the exponent is much larger than theirs, which is m2x/M2x (we can obtain the definition of M2x by extending the definition of M1 to the support size 2k). Let Lridge and Lrddg (g++1) be the losses the (-th and (1 + 1)-th iterations of our beam search algorithm. We first derive a lower bound formula on the decrease of the loss function Lridge between two successive iterations, i.e,, Lridge (
( We will then show that this quantity of decrease can be expressed in terms of the parameters M1 and m2k defined in the statement of the theorem. Finally, we use the relationship between the decrease of the loss and M1 and m2x to derive the right-hand inequality in Equation (51).
Step 1: Decrease of Loss between Two Successive Iterations:
For an arbitrary coordinate j that is not part of the support of B², or more explicitly 5 = 0. if we change the coefficient on the j-th coordinate 3 from 0 to a, the loss becomes:
Cridge (B3 ae,))
# ej denotes a vector with j-th entry to be 1 and 0 otherwise
-yy X (B' + aee)))" 1ly-x(66 X(8' + ae,)) + 12 + ue;)"((3 + tae))
# aappyy the definition of ridge regression loss
-Iy X3 - aXe,)* '(y x33 x + (B + ae
+ +ae,)
# distribuue into the parentheses
-V x3 ax,)) (0y x33- ax,;) +2 ae,) )² ( + (ee)
# X.j denotes the j-th column of X
-Vy (3 X3¹ 20 - X331'" X.j + a² x T X, + X2 [[ g + 20033 ez + a²] # expand each quadratic term into three summation terms
-V X 4) X - 20 x38 j + a² x X ,j + x2 [[ g' + 2a.3 + a=]
# ( T B
BJ: eg takes the j-th component of g
-I - (3 (y- X33 - 20(y - X x, + 0 x T x, ; + X2 [(3') g + a²]
# #3 = 0 because are oplimizing
coordinate with coefficient equal to O
-Iy (g X13') + >2 B² + (113 + >2)a² 2(( - 3 collect relevant coefficients for o² and
-2 - x x x aa Cridno (B') + (113 + >2)a²²
# simplify the first two terms using Lridge(.)
= -Cridgo (B') (1 1 + 22)0² V VfLridge(B* a # the j-th partial derivative is V CBCrdge((*)) 2(X03 r) according to Lemmma B.7 with 33
0
a²+ v (i || 1133 + X2"
= -Cridge + (1 X2)
# extract out coefficient
V, Eridge (3t)
1 V Cridge (Bt)
-(1 X
( Cridgo (39)
= -Cridao + (I|x X22
a2 + 2
a
X .j 112 + A22
2 |X
X2
11 + X2
# add and subtract the same term
(0+
1 V Cridge (35)
V Cridge (3t)
2
= -Cridge (B²)) ( (1 + Az)
+Xa)
2
x
12
X22
2
+ X2
# complete the square
26

---

1 ( [V, ridge (33))2 C
V Ceidde (3') 2 ||
4 X A2 (11x, X2) a +
+ X2 # distribute into the parentheses
The first two terms are constant, and the third term is a quadratic function of a. The minimum loss achieve for Lridge (3' ae,) by optimizing the j-th coefficient is:
(V, Cridge (5*)2
min Cridge ae,) = Cridge (B*) 4 12 + A
and the maximum decrease of our ridge regression loss by optimizing a single coefficient is then:
max(Lridgo - ( + oee)) = fridge (g min ( + +aee) = (V Cridgo (g²)) a
X 11 + A2
(52)
where X,j is the j-th column of the design matrix X.
Now, for our beam-seardh method, the decrease of loss between two successive iterations will be greater than the decrease of loss by optimizing on single coordinate (because are fine-tuning the coefficients on the newly expanded support and picking the best expanded support among all support candidates in the beam search), have
Lridge aae))) max 1(V, ( (3)))²
(F"d ( 2 max (ax
|| X , 1 + A2
(53)
For an arbitrary of coordinates J', we have the following inequality:
1 (V Srgdde (3))
1
(V Fridge ()))
max 3 4 X A2
(54)
J
X
A2
J'ET'
Inequality holds because the largest decrease of loss among all possible coordinates is greater than or equal to the average decrease of loss among any arbitrary set of coordinates J'.
we choose J' = R := S \\ supp((3), where S* is the set coordinates for the optimal solution 3 and supp(-) denotes the support of the solution, we have
1 (V Cridge (B²))2
Eridgo(B)) - Cridge (B++1) N max
4
1
+ A2
y Cridgo (3))
S
4
X 5 1|2 + X2
jEERR
(V Fridge (3²))2
# pull to the front
S
12 + +
JESR
Lridge (
Σ
# because SR C 15 |8*1=
4k
X 1
+ >22
PESSR
Σ (V (B*))2
4k esx X| X2)
jESSR
# take the maximum of the denominator and pull the front
Lridge (B¹) (B 2 4k maxyess (|)X
1
Σ
Cndde (55)
X2)
jesn
the lower bound of decrease of loss between two successive iterations of our beam-search algorithm.
Inequality 55)
27

---

In Step 2 and Step below, we will provide bounds on maxy (I Ejesa (V, ( in terms of Mj and m2zk respectively.
x1 + X2) and
Step 2: Upper bound on maxy'ss (1l X X2)
We will show below that
M1 max X 12 A2).
2
According to the definition of the restricted smoothness of M1 (See Background Concepts Ap- pendix we have that
M1 www w1 112 Cridge (w22) Cridge (w1) V Seldge (w -w1)
(56)
2
for Vun, RP with l llo S 1, ||wllo S 1, and |w2 wr 1l1 S 1.
The right-hand side be rewritten
Cridge (22) Fridge (w), - V Cridgo (w) )a (w2 wi
X( 8112 + >ellwal (2) (1|Xw> 8112 + >el|wa ((:) - -
(xx"(xwx (Xw1 y) + ) w1)
write out Lridgo()) explicitly and substitute V Lridgo (20)
#1 (x (Xw1 - 4) + w1) according to Lemma B.7
=(I(Xw> Xw1) + X w- - y)112 + >ell( - wi) + ww (12
(||Xw y + w2|l [2:) 2(x (Xwi y) + \yw)1) (ww)))((==>>))))) w1)
We subtract and add identical terms inside the first parentheses
= =11Xw>> - Xwili + W w1 -112 + 2(X (wi) (X -y)
+ >ellu22 wi 11 - Ao w 12 + 2>2 =
(l ww 21 + A21 w1 (2) 2(x" (Xw1 - y) )2 (T (w2 w1)
# expand the terms inside for the first line into two lines
(I| w2 X Xwi 11 + |xw
y12 + 2(x" (Xw1 - =)))((w> -112-ww)) w1)
+ >21lu² - will 1/2
\2 |
12 + 2(\2 1d (w2 - -wi)
(|) X|w - y1 + >21|w1
w |
- -((X" T (Xwi - y) + x²2w1)) TT (w2 w
# rewrite (X ww Xw2)²² (Xwi y) (X" (X wi ")) wi)
=|| W2 Xw, 112 + >2|lw> wi 11
cancel out terms
-(w2 w1)" "0ag((II+ x X) W2
w1)
Cridpo (w2) Lridgo (w)) (w
v Cridgo (w) )T (w2 wi) (w2 wi (A2l+ -T X) [xXI+XX))(x2 wi)
Therefore, together with Inequality (56). we have that
Mi 2
wi 11 2 (w2 wr) " (A2I X X)( - -wr)
(57)
for any W2 wi E R and |/02 wi llo I 1. If we let W2 wi = Ej where ej is a vector with on the j-th entry and 0 otherwise, have
M A2 - 1
for Vjj p). Thus, can derive our upper bound the beginning of Step 2:
Mi
> max
| + A2)
(58)
2
J'ES"
Step 3: Lower bound on Eyesa Lridge (3
We will show that Eyeg (V Cridgo (3²))2 2m2x (Bri)go (3*))
According to the definition of the restricted strong convexity parameter 112k, we have:
( 38) +
m22
|
33113 < (rdggo(")) Cridgo (B")) )
2
28

---

where 3 E I2 is our heuristic solution at the t-th iteration, 3 is the optimal k-sparse ridge regression solution, and 11 = 2k: |||3 llo
k 2k, and || 33' lo S 2k
By rearranging some terms, we have:
Cridge (3") (ridgo(33))
< V Cridge
(B3) -B¹) "
122k 3 31
# always holds because of restricted strong convexity
V Lridgo TB3 + "aje, aje; B²) -
11
Σ aojej -
ej is a vector with on the j-th entry and otherwise; Oi is a fixed number with 05 = [8]
Lridge (
ajee)
122k
ayezll
# #oy is a fixed number with Oj = [3
2
1=1
j=1
V. Jriidgo (3")as
m22
03)
2)
# Combine into one summation
2
1=1
Σ JESR
(V (VyCridge( )a
122k 03)
Σ
1722 a3) (V)
jEsupp(88)
Σ
V (3") )aj
m22k
03
148 (
# Recall SR S* \ supp(33). We divide the indices 2,, supp( and the rest.
.......)) p. into three groups: SR.
Σ
V Vrridg (B")a, )a; m22k a2)
JESR
# V V (3") 0f j supp(B'), and aj ffr & S supp(
Σ J'EJ'
V Eridge Jap
122
a3.
Change the index notation from j to j'
I <max-
j (B))
112hk
2
JEJr
# Allow to take any values with new variable The inequality holds if we take the maximum with respect to v
Σ
(V Gridge (B)) each term can be maximized because it is a quadratic functtiiic
2m22 J'E.T
This gives us the lower bound:
Σ (V Eridgge (3+)2 2m2x(Gridgecff Fridge (f²))
(59)
J'E
Step 4: Final Bound for the Loss of Our Heuristic Solution
we plug Inequalities
(58) and (59 into the Inequality (55). we have:
Cridge (B*)) (Bidge(B
(V Cridge (3²))²
4k max X| A2)
jes
3 (ft) - (eridge((3t))
12kk
(Cridge (.iidg(((3)))
(60)
kM
Lastly, let's use Inequality (60) to derive the right-hand-side of Incquality 51 stated in the Theorem. Inequality (60) can be rewritten as:
(Cridge (BO) - Lridge(("")) (Cridge (1²+1)- - Lridge (B*)) 2
m22
(Cridgo((3)) (.3
kM,
29

---

which can be rearranged to the following expression:
(1 (1- 122k
KM1 )(ridge((")) Lridge ((")) (Cride Fridge ((r))
Therefore, by applying the above inequality k times, we have:
(1 (Crigg (0) ((+))g((3)) 2 (rriggo(*)) Lridge (3*))
m2k
kM
However, Lridge (0) 0. This gives us
Cridge (3%) I (1-
m2k kM
)b
Leidge (B*)
To get the final inequality, we start from well-known inequality result + x S e for any I E IR.
Let 1, then we have N -a/k i If we take both sides to the power of k, we have
(1 a Σ
e
we let a
makk Mr
(because mak
29
we have I =
Miz -1 for any k: 2 1), then we Mik
Mi
have
malk
jk
af1
(61)
kM
Because Cridge is negative number, we finally have
(3 - 1 1 (1
R: )k
m2k
Cridge (B*)
man
< (1
Cridge
C.5
Derivation of g(a, b) = max, ac : via Fenchel Conjugate
Recall function g R R IR U (+00) } is defined
4/4
if
g(a,b)
i-a=b=0
(62)
otherwise
The convex conjugate of g(: -) denoted as g" (:, :), is defined as 8
(" d) = sup (ac + bd g(a, b))
(63)
a.b
Since function g(a, b) is convex and lower semi-continuous, the biconjugate of g(a, b) equal the original function, ie., (g")* B Writing this explicitly, we have:
g(a, b) sup (ac bd g" d))
(64)
c.d
For now, let us first derive an analytic formula for the convex conjugate function g" (s.).
According to the definition of g(a, b) in Equation 62 have
ac bd
if if = 0.
bla,b, = ac + bd g(a, b)
otherwise
For any fixed and d, the optimality of the first case be achieved if the first derivatives of ac bd a² with respect to a and are both equal to O. This means that,
d + a²
= and
0.
30

---

If d c²
T: the optimality condition is achieved if we let and then + bd o² bc 2
5
2 cct b( o =.
Ifd + - =²² the optimality condition can not be achieved. However, if still let be have
ac + bd _/4 bi c b(d) = ( d)b. Then, we have two specific cases to discuss: 1) if
é + > 0, the supremum is oo we let b + oo; 2) + d < O, the supremum is we let 6-0.
With all these cases discussed, we have that
(c, d) sup h(a, c,
a,b
0
i + +
oo
otherwise
Now let revisit Equation (64). where g(a, supe,a ac + bdl - gc Below, we discuss three cases and derive the alternative formulas for g(a, under different scenarios: (1) c² d (2) d S O, and (3) 4 + d 0.
1. c + -d=00 0:
91(a,b)
sup ac bd - 9'(c, sup + b(-
a,d, 2 +d-0
2. T d < 0:
2
92(a,b) sup ac bd g" (²(ccd) sup ac bd
c.d, +dco
3. c + d > 0:
ga(a,b)= sup ac + bd g" (c, 00
o.d, o +db0
After discussing these three cases, we can obtain the final results by taking the maximum of the three c²
optimal values, iee,, g(a, b) max(g: (a, b), (2)g2((() b), (a, b))- The third case 5 d > 0 is not interesting and can be easily eliminated. The second case & + d < O is also not interesting because ac + bd = ac + b(-$) + b( + d) I ac - b(-$) for any b 2 O, which is the domain we are interested in. This gives g2(a, b) s 91 (a, 6). and the equality is only achieved when b = 0. Thus,
g(a. b) = max(gi (a, b), (2 b). gs(a
9l (a,
(65)
sup ac + b(
max ac
(66)
4
where in the last step we convert sup(.) to max(.) because the function quadratic and the optimum be achieved.
31

---

D Existing MIP Formulations for k-sparse Ridge Regression
We want solve the following k-sparse ridge regression problem:
min B.
B x x
XB+ XB
2y
A2 Σ
(67)
==1
subject to (1 (1-2 0,
Σ j=1
2j E 2j s
(0,10
However, we can not handle the constraint (1- = directly in a mixed-integer programming write before
(MIP) solver. Currently in the research community, there four formulations sending the problem to a MIP solver. They are (1) SOS1, (2) big-M, (3) perspective, and (4) optimal perspective. We write each formulation explicitly below:
SOSI formulation:
P X2
min 3
3 X T 2y X3
+
Σ
(8
(68)
j=1
subject to (1- 2y. E SOS1,
E
2j E 10.1),
j=1
where (a, b) E SOS1 means that at most one variable (either a b but not both) can be nonzero.
2. big-M formulation:
min 3,2
- X3 2x
X2
Σ B3
(69)
j=1
subject
M2, S B3 - Mzj,
Σ
2
S
2j E {0.1)
j=1
where M > O. For the big-M formulation, as we mentioned the main paper, it is very challenging choose the right value for M. If M is too big, the lower bound given by the relaxed problem is too loose; if M is too small, we risk cutting off the optimal solution.
3. perspective formulation:
min 8,2
x
2y
X3
A22 Σ
83
(70)
subject to
3 s32js
Σ 2 S
2j E (0,1)
where sy's are new variables which ensure that when 2, 0, the corresponding coeffi- cient 8, 0. The problem is a mixed-integer muadratically-conttraiii quadratic problem (QCQP). The problem can be solved by a commercial solver which accepts QCQP formula- tions.
4. eigen-perspective formulation:
P
min B.z.8
TT (x"x X
Aminn (X*x)I) 2y 233+ + (Ae Amin(X x))
Σ
83
f=1
(71)
subject to
83 S 8زスj1 853ز1
Σ 2 Sk
E (0.1).
j=1
where Amin() (-) denotes the smallest eigenvalue of a matrix. This formulation is similar to the QCQP formulation above, we increase the fg coefficient by subtracting Amin (XTX) from the diagonal of X X.
32

---

5. optimal perspective formulation:
min B
(x x, B) -2y" X
(72)
subject to
B - --3 =0, 88 S B3
Σ j=1
2/ {0, 1),
where A Z 0 means that A is positive semidefinite Such formulation is called a mixed- integer semidefinite programming (MI-SDP) problem and is proposed [62 28 37.
Currently, there is commercial solver that solve MI-SDP problems: Gurobi does not support SDP; MOSEK can solve SDP but does not support MI-SDP. As shown by (28), SDP is not scalable to high dimensions. We will confirm this observation in our experiments by solving the relaxed convex SDP problem (relaxing {0, 1} to [0, 1J) via MOSEK, the state-of-the of-thhe-rttttttt conic solver. After get the solution from the relaxed problem, find the indices corresponding to the top & largest 2j and retrieve a feasible solution on these features.
33

---

E Algorithmic Charts
E.1 Overall Branch-and-boumn Framework
We first provide the overall branch-and -a----d--boddd framework. Then we go into details on how to calculate the lower bound, how to obtain near-optimal solution and upper bound (via beam search), and how to do branching. A visualization of the branch-and-bowe tree can be found in Figure
Root
Step 1: use novel lower bound calculation algorithm
LiGuer
Step the beam Casperifuspir search algorithm
Node
Step pick variable to branch based on Rupper from Step using the branching strategy
Node
Node
Zg 1
Zg
Zr 1
Zy
Node
Node 5
Node
Node 7
Figure 5: Branch-and- ansbhound diagram.
34

---

Algorithm 1 OKKidgeeD,,k..g e, Taee Tmax)
Input: dataset D, sparsity constraint k, & coefficient X2, optimality gap e, iteration limit Titer for subgradient ascent, and time limit Tmax-
Output: sparse continuous coefficient vector B E R2 with || S k.
1: Ehest, lower Puppers Cbest 30; -00, 0, Dinitialize best optimality gap co, best lower bound
Cheat
10 -o xc, best solution to O and its loss to O.
2: f- initialize empty queue (FIFO) structure >FIFO corresponds breadth-ffrst search.
3: Nscloct = Navvi = 0 @CCordinates in Naelea can be selected during heuristic search and lower
bound calculation for node; Coordinates in Navoid must be avoided.
4: RootNode = CreateNode(P). k, Neclect N Navoid) DCreate the first node, which is the root node
in the BnB tree.
5: Q 2.push(RootNoo) (RootNode)
PPtt the root node in the queue. DSmallest depth where least one node is unsolved the BnB tree 7: WHILE Q + 0 and Telapsed < Tmax do D-eepp searching until the time limit is reached or there
6: lunsolved
is no node in the queue.
8:
Node (.geeNode()
9:
(oweerSeSSS))))) DGet the lower bound for this node through the fast method if Liower Sheee then Liower Node.lowerSolveeighhe >Get tighter lower bound for this
Liower
10:
node through the subgradient ascent
11:
if Llower hhess then continue >>Prune Node if its lower bound is higher than equal the
current best upper bound.
12:
if All nodes in depth lunsolved of the BnB tree solved then
13:
max,( {L", for q E (1,2.... ..., lunsolved)) DTighten the best lower bound; L the smallest lower bound of all nodes in depth 9
Lower
Chest =
14:
luncalved = luncolved +
D-Riise up the smallest unsolved depth by 1
15:
ébest (Cupper
Liewwe) (beat < 6 then return Bresa D. Optimaliity gap tolerance is reached, return the current best
C.estt
16:
solution
17:
Buppers Cupper Node.upperSolve() search, which an upper bound for our BnB tree.
pGet feasible sparse solution and its loss from beam
18:
if Lupper Obest then
upper
19:
Cupper uupper Luppers B upper
DUpdate our best solution and best upper bound
Chest (Chpper
Chonte Chest
21:
if Ebest then return Brest DOplimality gup tolerance is reuched, return the current best
solution
22:
j Node. Tande.getBranchCoord..ff
>Get which feature coordinate to branch ChildNodel = CreateNoded DD k. A2, (Node + Nsstee) U Kj].Node + Navoii) >New child
23:
node where feature j must be selected
24:
ChildNode2 = CreateNode(D, k, A2, Node - Nistects (Node - Navoii)) U (i)) NNew child
node where feature j must be avoided
25:
PPut the two newly generated nodes into our queue. return Bees PReturn the current best solution when time limit is reached there is node in the queue
Q.push( ChildNode ChildNode2)
26:
35

---

E.2 Fast Lower Bound Calculation
Algorithm 2 Node.lowerrSoveeast)) - Llower
Attributes in the Node class: dataset D {(z;, 1/1 1- sparsity constraint k. & coefficient X2, set of features that must be selected N set of features that must be avoided Navoid- Output: lower bound for the node Liower-
7 argmin =t B" X X X3 X2 1|11
Solve the ridge regression problem. We avoid using coordinates belonging to Navoid by fixing the coefficients on those coordinates to be 0
2: Lridg = 7 -"'X"XX
2y" Xx + 2211v11
Losss for the ridge regression.
3: Lexra (A2 + Ami X)) SumBottomp WWinal (15). for Vj E [1.....]] pl (Navoid
)) DFiist of all, we are restricted the coordinates not belonging to Navoid- Second,
coordinates in the set Nselect must be selected. If we work through the strong convexity argument in Section
3.1 the main paper, we would get this formula.
4: Llower
ridge
Lextra
5: return Liower
36

---

E.3
Refine the Lower Bound through the ADMM method
Next, we show how to get tighter lower bound through the ADMM method.
Algorithm 3 Node. Node.loweeSavveiighhAAMMMM - Llower
Attributes in the Node class: dataset D = {z v:)]== sparsity constraint k. &2 coefficient X2, set of features that must be selected Nselect and iteration limit Ther for ADMM (For simplicity of presentation, assume Navaid is empty. If Navaid F 0. X, y, x to focus solely on coordinates not belonging to Navoia).
do selection and relaheling step
Output: a lower bound for the node Llower.
7k argming B3 X problem. This will be used as a warm start for the ADMM method below. To avoid redundant computation, we can use the value for Y from
2y" X3+ >211 >initialize y by solving the ridge regression
2: p² -x QxY
>lnitialize p as stated in Problem 18 >Initialize the scaled dual variable q the ADMM. DCalculate the step size used in the ADMM
3: q' =0 -0
4: / == 2/v Amax (x) (QQx))((QQQ)))))))))))))))))))
5: for = 2, ...TTTeeer do
6: T ( (X p² p'-1--1 1'
>Update "y in the ADMM. DUpdate 0 in the ADMM. DThis line and the next lines are used to calculate variables used for setting up the isotonic regression problem.
7: 0f 2Qxx* + -
8:
a y'y-0''''
9:
Let J be the indices the top 1s - WWeteaa terms of |ayl with the constraint that any element
in I does not belong NSetect-
10:
Let Wy = j $ I and j & Nuelect and let Wy p(A++Aj+T otherwise.
Let bj = [d
11:
wyy
12:
Solve the problem è = argmin, Ej En - b,)² such that for Vi, & NNsloa, V; 2 Uz >We can decompose this optimization problem into two independent
la > larl. problems with coordinates belonging N (can be solved individually) and coordinates not belonging to Nselect (can be solved via standard isotonic regression).
p'= =iiinn O i
DUpdate p the ADMM. The symbol O denotes the element-wise
(Hadamard) product.
--'-1+ 15: end for
14:
9' q
p³ X
DUpdate q the ADMM.
Liower lim)
DCalcclate the lower bound.
17: return Llower
37

---

E.4
(Optional) An alternative to the Lower Bound Calculation via Subgradient Ascent and the CMF method
In addition to the ADMM method, we also provide an alternative algorithm showing how to get tighter lower bound through the subgradient ascent algorithm using the CMF method. Note that this subgradient method has been superseded by the ADMM method, which converges faster. We provide this method the Appendix for the sake of completeness and for future readers to consult.
Algorithm 4 Node. Nodee.loverSbileelggghhugrrrrrre - Liower
Attributes in the Node class: dataset {(x,, y/1 1-) sparsity constraint k. coefficient X2, set of features that must be selected Ne set of features that must be avoided Navoid, and iteration limit The for subgradient ascent Output: lower bound for the node Liower.
1: 7 argming,8; of je B X 3x3 X3
DSolve the ridge regression problem and avoid using coordinates in Navoid- This will be used as warm start for the subgradient ascent procedure below. To avoid redundant computation, we can use the value for from Node. lowerlStveSessett
1
2: V = YY
IInitialization for of at iteration step = 3: h- Lridgo (ppee)) Iget estimate of mazzh(()) if Buppe has not been solved at the root
node, solve Bupper by calling the upperSolve() function the root node.
4: Llower 11(7)
5: for t= ...Tier Tee do
6: 7 Node www.suubrddgddeennau..
v) >Perform one step of subgradient ascent using the
CMF method
7: Llower max(Liom hu()))
DLoss decreases stochastically during the subgradient ascent procedure. We keep the largest value of h((r)
8: end for
9:
return Liower
In the above algorithm, we call new sub algorithm called subGradAscentCMIF We provide the details on how to do this below. We try to maximize h(y) through iterative subgradient ascent. A more sophisticated algorithm (CMF method (23] can be applied to adaptively choose the step size, which has been shown to achieve faster convergence in practice. The computational procedure for the subgradient ascent algorithm using CMF method
[8. 23 as follows. Given > iteration
I 1, can be computed as:
i ridge
> (r, ((-"((111
w²
# get the subgradient
0y
7=9--1
8, max(0, (d'
w /||0 /110-21113
# 8t will be used as a smoothing between w and w
0' w' s,0'
# combine wi1 and w' using the smoothing factor
04 = (h' hh(r"~1))/ll0
# calculate the step size; h² is an estimate of max, h(7)); here we let h. Lridge ( # perform subgradient ascent
n' ~² a
where the beginning of the iteration (t 1), ve 0 and choose 1.5 as a constant for all iterations as recommended by the original paper. Below, we provide the detailed algorithm to perform subgradient ascent using the CMF method.
38

---

Algorithm 5 VodessubbrradAssseCCMMiCCC 4²) +
Attributes in the Node class: dataset D = { Vi ) sparsity constraint k. f2 coefficient X2, set of features that must be selected Ne set of features that must be avoided Navii Input: Current variable for the lower bound function k(-) for Equation [4] current ascent vector and ht is an estimate of maxy l1()).
Output: updated variable and updated ascent vector v.
1: 2 argming -ridge-X
Loaddle (1,;
Dgee
the minimizer for the current value yor optimization is performed under the constraint
1) S k. E [0,1], 2j = for E Neelea and 2j = 0fror j E NNslect
2: @C x /
(0y)
Vaget the subgradient
3: w = for j Navidd b We don't perform any ascent steps on the coordinates belong to Navoid-
4: s max(0,- -1.5v² w w/|[v/] /
DCalculate the smoothing factor bupdate the ascent vector by combining the current subgradient and previous
5:
+
ascent vector
6: (h" h(()) /
bcalculate the step size for the ascent step PPerform one step of ascent along the direction of the ascent vector
7: 7 = c00
8: return y, v
E,5 Beam Search Algorithm
We provide visualization of the beam search algorithm in Figure 6
1,2
1,2,3
1,3
1,2
1,2,4
2
1,4
1.2,5)
<start>
3
1,5
1,2,5
4,1
4,3,1
5
4,2
4,3
4.3,2)
4,3
4,3,5
4,5
Figure 6: We add into our support one feature at time by picking the feature which would result in the largest decrease in the objective function. After that, we finetune all coefficients on the support. We keep the top B solutions during each stage of support expansion.
39

---

Algorithm 6 Node. Node.upperSolve(B = 50) + Supper: Lupper
Attributes in the Node class: dataset D {<x ye Wim sparsity constraint & coefficient A2, a set of feature coordinates that must be selected Nselect, and a set of feature coordinates that must be avoided Naviid-
Input: beam search size B, which is set to be 50 the default value.
Output: a near-optimal solution Bupper with || Bupper = and the corresponding loss Lupper-
" x + A2|1 DInitialize the coefficients by 1: B argming,8, =0 for Nit B x" x"3x3 2y"
fitting on the mrust-be-selecter coordinates.
2:
{
bcollection of solutions at each iteration (initially we only have one solution) Dlnitialize the collection of found supports as empty
3: F 0
4: for = |Weeteal + 1,,,..; & do
5: Wimp {
6: for 3'' Wodo
Daach of these has support - Reeurrns B models with supp. t
7: (W",
ExpandSuppBy1(D,P (', B,Navoia)..
8: Wimp + Wimp UW
9:
end for
10: Reset W to be the B solutions in Wmpp with the smallest ridge regression loss values.
end for
12: Pick Supper from W with the smallest ridge regression loss.
13: Lupper = Grpper X X Bupper 2y" X Bupper + 22 12plBupperl1 ll
return Bupper. Cupper-
Algorithm ExpandSSup BaNavii) + W
Input: dataset D ((x,, )=) current solution B', set of found supports F, beam search size B,
and set of feature coordinates that must be avoided N Naviid-
Output: a collection of solutions W { with ||,3111o || for vt. All these solutions include the support of f plus one more feature. None of the solutions have the same support as any element of meaning we do not discover the same support set multiple times. We also output the updated F.
DNon-support of the given solution 2: Pick up to B coords (j's) in S with largest decreases in A Grdgg (-), call this set J'. DWW will use these supports, which include the support of w plus one more.
1: Let S < - = and & Navaiaa
3: W +0
4: for j E J' do
DOptimize on the top coordinates
5: if (uuppw))UR)) (j}) E then
6: continue
D WeWe' ve already seen this support, so skip.
7: end if
8:
DAdd new support to F. u" argmin, Lidge (D, u) with supp(u) supp(w')) [i}- DUpdate coefficients on the
F F F-FU (supp(w)) U (i})-
9:
newly expanded support
WU
beache the solution to the central pool
11: end for
return W and F.
40

---

E.6 Branching
Algorithm 8 Node.getBBranchCoordl + j
Attributes in the Node class: dataset D { 31 M= sparsity constraint k. &g coefficient Ag, set of feature coordinates that must be selected Neelects and a set feature coordinates that must be avoided Wavaid-
Output: a single coordinate j on which the node should branch the BnB tree.
1: j = argmaX -23 (X" X + 1 + ((X" X)s + X2) - 2y" X 3,e, Dselect the coordinate whose coefficient, we set to 0, would result in the greatest increase our ridge regression loss
2: return j
E.7 A Note on the Connection with the Interlacing Eigenvalue Property
During the branch- and-bound procedure, finding the smallest eigenvalue of any principle submatrix of X X is computationally expensive if we do it for every node. Fortunately, we can get lower bound for the smallest eigenvalue based on the following Interlacing Eigenvalue Property [36]]
Theorem E.1. Suppose A g is symmetric. Let B E m= with m <, be principal submatrix (obtained by deleting both i-th row and i-th column for some values of i). Suppose A has eigenvalues " in and B has eigenvalues an I ... I Omn Then
Tik mt for k 1,...
And if = n1 1,
71 S 01
72 S 50 ann Sm
Using this Theorem, we can let Amin (x X) algorithm. This saves lot of computational time because only have to calculate the smallest cigenvalue once in our root node. However, if the minimum eigenvalue were to be recalculated at each node efficiently, the lower bounds would be even tighter.
for every node in the branch-and-Be and-bundd tree in our
41

---

F More Experimental Setup Details
E.1 Computing Platform
All experiments were run on the 10x TensorEX TS2-673917-DPN Intel Xeon Gold 6226 Processor, 2. 2.7Ghz. We set the memory limit be 100GB.
F2 Baselines, Licenses, and Links
we used Gurobi and MOSEK. We implemented the SOS1, bigM, perspective, and eigen-perspective formulations in Gurobi. The Gurobi version is 10.0, which can be installed through conda ( //aa///ddaaaaaaaaaaaa gurobi). We used the Academic Site License. We implemented the perspective formulations and relaxed convex optimal perspective formulations in MOSEK. The MOSEK version is 10.0, which can be installed through conda hhtpps// htctle///aacccaaaa.oo/////////// We used the Personal Academic License.
Commerical Solvers For the MIP formulations listed Appendix D
org/ gurobi/
PySINDy, STLSQ, SSR, E-STLSQ We used the PySINDy library (https://github com/ cannapisslbb/yyiinyy to perform the differential equation experiments. The license MIT. The heuristic methods STLSQ, SSR, and E-STLSQ are implemented in PySINDy.
MIOSR For the differential equation experiments, the MIOSR code is available on GitHub(https The license for this code repository is MIT.
GitHub (https with commit version The license for this code repository is MIT j//blob// master/LICENSE.nnt Due to package compatibility, we use Gurobi 9.0 for SubsetSelec- tionCIO. See the SusbetSelectionCI0 package version specification at this link: https Vttth//woc..aaaaataaaaaaaaattttn
SubsetSelectionCI0 The SubsetSelectionCII code is available //tpghhhw.ccoo//gnnaaaaa///////a
"Expat" License htttpe//ggtuubb..c//aaaaaapappp
10bnb The 10bnb code is available on GitHub battps://githul com/ www.coommalaaaaeaaeeeeeeeeeeaaae htt///alsaaa/lobbbaa with commit version 4447775hhe666h/////1111edeff6666
The license for this code repository
MIT.
Sparse Simplex The Sparse Simplex code is available at the author's website (https://siten google. com/site//
taatee/mmirie ami rimck314/ /tsff/tttareeeee aauthuser=O). We ran the greedy sparse simplex algorithm. The license for this code repository is GNU General Public License 2.0.
Block Decomposition The block decomposition code is available the author's 's website (https) 7/yuangzh. github. io). The author didn't specify license for this code repository. However, we didn't modify the code and only used it for benchmarking purposes, which is allowed by the software community. Besides the block decompostion algorithm, the code repository also contains reimplementations for other algorithms, such as GP, SSP, OMP, proximal gradient 10c, QPM, ROMP. We used these reimplementations in the differential equation experiments as well. Please see [60 for details of these baselines.
F.3 Differential Equation Experiments
We compare the certifiable method MIOSR along with other basclines on solving dynamical system problems. In particular, for each dynamical system, we perform 20 trials with random initial conditions for 12 different trajectory lengths. Each dynamical system has observed positions every 0.002 seconds, so a larger trajectory length corresponds larger sample size n. Derivatives were approximated using a smoothing finite difference with window size of 9 points. The derivatives were calculated automatically by the PySINDY library. Each trajectory has 0.2% noise added to it to demonstrate robustness to noise. For the SINDy approach, true dynamics exist in the finite- dimensional candidate functions/features. We investigate which method recovers the correct dynamics more effectively.
42

---

F.3.1 Model Selection
Model selection was performed via validation with training data encompassing the first 2/3rds of trajectory and the final third used for validation. Predicted derivatives were compared to the validation set, with the best model being the one that minimizes AIC. for sparse regression in the setting of dynamical systems 45
F.3.2 Hyperparameter Choices
With respect to each trajectory, each algorithm was trained with various combinations of hyperpa- rameter choices. For both MIOSR and our method OKRidge, choices are Xg E {10-5, 10-3, 10-2, 0.05, 0.2), and the sparsity level hyperparameter choices are E 2,, 3, 4, 5). As in the experiments of MIOSR 19]. we set the time limit for each optimization to be 30 seconds for both MIOSR and our method OKRidge. For SSR and STLSQ. the ridge regression hyperparameter choices are X2 E {10-3 100 10 0.05, 0.2}. SSR and STLSQ may automatically pick sparsity level, but they require thresholding hyperparameter. We choose 50 thresholding hyperparameters with values between 10 and 10 depending on the system being need
ridge regression hyperparameter
fit. Together there are 250 different hyperparameter combinations length for for SSR and STLSQ. When fitting E-STLSQ, cross-validation is not performed as the chosen hyperparameters for STLSQ used instead. For SSR, MIOSR, and OKRidge, we cross validate on each dimension of the data and combine the strongest results into a final model. For STLSQ and E-STLSQ, post-processing is done and the same hyperparameters used for each dimension.
each trajectory
F.3.3 Evaluation Metries
Three metrics considered to evaluate each algorithm: true positivity rate, L2 coefficient error, and mou-ueean-squared eгror (RMSE). The true positivity rate is defined as the ratio of the number of correctly chosen features divided by the sum of the number of true features and incorrectly chosen features. This metric equals if and only if the correct form of the dynamical system is recovered. L2 coefficient error defined as the sum of squared errors of the predicted and the true coefficients. Finally, for each model fit, we produce 10 simulations with different initial conditions of the true model, and calculate the root-mean- coot-mean-squared error (RMSE) the predicted derivatives and the true derivatives. Besides these three metrics, we also report the running times, comparing OKRidge with other optimal ridge regression methods.
43

---

G More Experimental Results on Synthetic Benchmarks
G.I Results Summarization
&g Perturbation Study We first compare with SOSI, Big-M (Big-M50 means that the big-M value is set to 50), Perspective, and Eigen-Perspectiin
formulations with different lg regularizations, with and without using beam search as the warm start. We have three levels of &g regularizations: 0.001, 0.1, 10.0. The results are shown in Tables I 60 in Appendix G.2 When the method has "_" in the upper bound or gap column, means that method does not produce any nontrivial solution within the time limit.
Extensive results have shown that OKRidge outperforms all exisiting MIP formulations solved by the state-of-the-art MIP solver Gurobi. With the beam search solutions (usually they are already the optimal solutions) as the warm starts, Gurobi are faster than without using warm starts. However, OKRidge still outperforms all existing MIP formulations. If Gurobi can solve the problem within the time limit, OKRidge is usually usually orders of magnitude faster than all baselines. When Gurobi cannot solve the problem within the time limit, OKRidge can either solve the problem to optimality or achieve a much smaller optimality gap than all baselines, especially in the presence of high feature correlation and/or high feature dimension.
Big-M Perturbation Study We also conduct perturbation study on the big-M values. We pick three values: 50, 20, and practice, it difficult to select the right value because setting the big-M value too small will accidentally cut off the optimal solutions. Nonetheless, we still pick small big-M value (5; the optimal coefficients are all I's in the synthetic benchmarks) to do a thorough comparison. The results are shown in Tables 120 in Appendix G.3 When the method has "-" in the upper bound or gap column, means that method does not produce any nontrivial solution within the time limit.
The results show that although choosing small big-M value can often lead to smaller rumning time, OKRidge is still orders of magnitude faster than the big-M formulation, and achieves the smallest optimality gap when all algorithms reach the time limit. Additionally, we find that having a small big-M value will sometimes lead to increased running time on solving the root relaxation. Sometimes, when the root relaxation is not finished within the 1h time limit, Gurobi will provide a loose lower bound, resulting in large optimality gaps. Aside from accidentally cutting off the optimal solution, this is another reason against choosing a very small big-M value.
Comparison with MOSEK, SubsetSelectionC10 and 10bnb For the purpose of thorough com- parison, we also solve the problem with other MIP solvers/formulations The results are shown in Table it means that method does not produce any nontrivial solution within the time limit. For the MOSEK solver, when the upper bound and gap columns have "_" values, and the time is exactly 3600.00, this means that MOSEK runs out of memory (100GB).
Tables Table T35 in Appendix G.4] When the method has "_" in the upper bound or gap column,
First, we try to solve the problem using the commercial conic solver called MOSEK. We use MOSEK to solve the perspective formulation, with and without using beam search solutions as warm starts. The results indicate that Gurobi in general much faster and more memory efffcient than MOSEK in solving the perspective formulation. We also use MOSEK to solve the relaxed convex optimal perspective formulation (relaxing the binary constraint 2j E {0, 1} to the interval constraint žj E [0, 1). we only solve the relaxed convex SDP problem because MOSEK does not support MI-SDP. The results show that MOSEK has limited capacity in solving the perspective formulation. MOSEK also runs out of memory for large-scale problems (n = 100000, p 3000 p 5000). Besides, the experimental results show that convex SDP problem is not scalable to high dimensions. For the high-dimensional cases, OKRidge solving the MIP problem is usually much faster than MOSEK solving a convex SDP problem.
Second, we uses the SubsetSelectionC10 method, which solves the original sparse regression problem using branch-and-cut clazy-conttraint callbacks) Gurobi. However, as the results indicate, SubsetS- electionCIO produces wrong optimal solutions when l2 regularization is small and the number of samples large (n 100000). When SubsetSelectionCIO does not have errors, the running times or optimality gaps are much worse than those of OKRidge.
44

---

Lastly, we solve the sparse ridge regression problem using 10bnb. In contrast to other methods, 10bnb controlls the sparsity level implicitly through the én regularization. instead of the én constraint. 10bnb needs to try several different bo regularizations to find the desired sparsity level. For each to regularization coefficient, we set the time limit to be 180 seconds, and we stop the 10bnb algorithm after time limit of 7200 seconds (this is longer than 3600 seconds because we want to give enough time for 10bnb to try different to coefficients). Moreover, 10bnb imposes box constraint (similar to all coefficients. The box constraint tightens the lower bound and potentially
the big-M method) accelerates the pruning process, but it could also cut off the optimal solution the "M" value not chosen properly or large enough. To our best knowledge, there is no reliable way to calculate the "M" value. The 10bnb package internally estimates the "M" value in a heuristic way. Despite the box constraint giving 10bnb extra advantage, we still compare this bascline with OKRidge. As the results indicate, 10bnb often takes a long time to find the desired sparsity level by trying different to regularizations. In some cases, 10bnb finds solution with support size different than the required sparsity level (demonstrating the difficulty of finding good En coefficient.. other cases, 10bnb still has optimality gap after reaching the time limit while OKRidge finish certification within seconds.
Discussion on the Case p n For the application of discovering differential equations, the datasets usually have n 2 p. Therefore, the case p > nt is not the main focus of this paper. Nonetheless, we provide additional experiments give complete view of our algorithm.
We set the number of features = 3000 and the number of samples n = 100, 200, 500. The feature correlation is set to be p = 0.1 since this is already very challenging case. The results are shown in Tables 151 153 in Appendix G.5 The results show that OKRidge is competitive with other MIP methods. 10bnb is the only method that solve the problem when 500 within the time limit. However, this algorithm has difficulty in finding the desired sparsity level in the case of n = 100 and n 200 even with large time limit of 7200 seconds. In some cases, MOSEK gives the best optimality gap. However, when n 2 as we have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and 10bnb are not competitive in discovering differential equations (datasets have n 2 p). which the main focus of this paper.
In the future, we plan to extend OKRidge to develop a more scalable algorithm for the case > n.
Fast Solve ADMM In the main paper, we proposed two ways calculate the saddle point problem. The first approach, fast solve, is based on an analytical formula. The second approach is based the ADMM method. Intuitively, since the fast solve method is only maximizing lower bound the saddle point objective and ADMM is maximizing the saddle point objective directly, we expect that ADMM would give much tighter lower bounds than the fast solve approach does. Here, we give empirical evidence to support our claim. The results can be seen in Figures 7 and
45

---

G.2
Comparison with SOS1, Big-M, Perspective, and Eigen-Perspecttve Formulations with
different & Regularizations
G.2.1
Synthetic Benchmark with X2 0.001 with No Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
SOS1
2.10878
106
1.68
0.00
100
big-M50
-2.10874 106
0.39
0,00
100
persp.
-2.10878
106
0.08
0.00
100
eig. persp.
-2.10878
106
0.01
0.00
100
ours
-2.10878
106
0.29
0.00
500
SOS1
-2.12833
106
8.04
0.00
500
big-M50
-2.12828
106
36.80
0.00
500
persp.
-2.12833
106
20.29
0.00
500
eig- persp.
-2.12832
106
0.69
0.00
500
ours
-2.12833
106
0.37
0.00
1000
SOSI
22.11005
10
82.53
0.00
1000
big-M50
-2.11005
106
1911.41
0.00
1000
persp.
-2.11005
106
1143.11
0.00
1000
cig. persp.
-2.11005
10
4.11
0.00
1000
ours
-2.11005
106
0.58
0.00
3000
SOSI
-2.10230
10t
3600.15
0.59
3000
big-M50
-7.83869
10
3601.20
169.78
3000
persp.
-2.10331
106
2288.93
0.00
3000
cig. persp.
-2.10230
106
102.18
0.00
3000
-2.10230
106
1.73
0.00
5000
SOS1
-1.98939-
106
3600.13
6.43
5000
big-M50
- -1.87965
106
3603.25
12.65
5000
persp.
-2.09637
106
3600.62
1.00
5000
eig. persp.
3880.65
5000
ours
2.09635
106
4.22
0.00
Table 1: Comparison of time and optimality gap on the synthetic datasets (n T00000, k 10,
0.1, Ag = 0.001).
46

---

# of features
method
upper bound
time(s)
gap(%)
100
SOSI
-5.28726
106
3.37
0.00
100
big-M50
-5.28725
106
0.42
0.00
100
persp.
-5.28726
106
0.07
0.00
100
cig. persp.
-5.28726
106
0.02
0.00
100
ours
-5.28726
106
0.29
0.00
500
SOS1
-5.31852
106
9.97
0.00
500
big-M50
-5.31819
106
89.04
0.00
500
persp.
-5.31852
106
8.44
0.00
500
eig. persp.
-5.31552
106
0.99
0.00
500
ours
5.31852
10
0.41
0.00
1000
SOS1
-5.26974
106
88.42
0.00
1000
big-M50
5.26974
106
1528.98
0.00
1000
persp.
-..66974
106
54.96
0.00
1000
eig. persp.
-5.26974
106
300.72
0.00
1000
ours
-5.26974 106
0.52
0.00
3000
SOS1
-5.16793
106
3600.38
2.71
3000
big-M50
-3.72978
106
3602.55
42.31
3000
persp.
-5.27668
10
3600.33
0.59
3000
eig. persp.
-5.27668
106
100.58
0.00
3000
ours
-5.27668
106
1.75
0.00
5000
SOS1
-5.16888
106
3600.14
3.16
5000
big-M50
-4.16126
106
3602.53
27.95
5000
persp.
55.2712
106
3600.72
1.00
5000
eig. persp.
3890.12
5000
ours
5..7716
106
4.32
0.00
Table 2: Comparison of time and optimality gap on the synthetic datasets (n = T00000, k == 10,
= 0.3, Ag = 0.001).
47

---

of features
method
upper bound
time(s)
gap(%)
100
SOSI
1.10084 107
2.97
0.00
100
big-M50
-1.10084 107
0.41
0.00
100
persp.
-1.10084
107
0.09
0.00
100
cig. persp.
- -1.10884 107
0.11
0.00
100
ours
1.10084 107
0.29
0.00
500
SOS1
- 1.10518 107
9.65
0.00
500
big-M50
-1.10515
107
89.59
0.00
500
persp.
-1.10518
107
3600.01
0.08
500
cig. persp.
-1.10588
107
25.13
0.00
500
ours
- 11.15518 107
0.40
0.00
1000
SOS1
1.09508 107
92.16
0.00
1000
big-M50
-1.09506
107
1757.09
0.00
1000
persp.
-1.09508-
107
3600.04
0.20
1000
eig. persp.
-1.09508-
107
226.82
0.00
1000
ours
-1.09508
107
0.59
0.00
3000
SOS1
1110012 10?
3600.10
0.52
3000
big-M50
3615.51
3000
persp.
1.10012 107
3600.91
0.59
3000
eig. persp.
-1.10012 107
100.85
0.00
3000
ours
- 11100112 107
1.74
0.00
5000
SOS1
1.08838
10?
3600.13
2.05
5000
big-M50
3612.62
5000
persp.
1.09966
107
3600.17
1.00
5000
eig. persp.
3732.98
5000
ours
1.09966
107
4.00
0.00
Table 3: Comparison of time and optimality gap on the synthetic datasets (n = I00000, k 10,
p 0.5, Ag 0.001).
48

---

of features
method
upper bound
time(s)
gap(%)
100
SOSI
-2.43575
107
2.83
0.00
100
big-M50
-2.43573
107
0.46
0.00
100
persp.
-2.43575
107
0.21
0.00
100
cig. persp.
-2.43575
107
0.02
0.00
100
ours
-2.33575
107
0.32
0.00
500
SOS1
-2.44185
107
10.25
0.00
500
big-M50
-2.44188
107
92.53
0.00
500
persp.
-2.44185
107
114.80
0.00
500
cig. persp.
-2.44185
107
26.49
0.00
500
ours
-2.44185
107
0.46
0.00
1000
SOS1
-2.11989
107
88.38
0.00
1000
big-M50
-2.41987
107
1361.05
0.00
1000
persp.
-2.41989
107
3600.08
0.20
1000
eig. persp.
-2.41989
107
232.96
0.00
1000
ours
-2.11989
107
0.52
0.00
3000
SOS1
-2.43713 10?
3600.47
0.59
3000
big-M50
3640.12
3000
persp.
-2.43713 107
3600.70
0.59
3000
eig. persp.
3606.15
3000
ours
-2,43713 107
1.77
0.00
5000
SOS1
-2.42551
10?
3600.13
1.48
5000
big-M50
-2.43694
107
3643.28
1.00
5000
persp.
-2.43694
107
3604.07
1.00
5000
eig. persp.
3711.01
5000
ours
-2.43694 107
4.15
0.00
Table 4: Comparison of time and optimality gap on the synthetic datasets (n = I00000, k 10,
0.7, Ag 0.001).
49

---

of features
method
upper bound
time(s)
gap(%)
100
SOS1
9.11026
107
0.05
0.00
100
big-M50
-9.11026
107
0.13
0.00
100
persp.
-9.11026
107
0.27
0.00
100
cig. persp.
-9.11019
107
1.63
0.00
100
ours
-9.11026
107
0.30
0.00
500
SOS1
9.12047 10?
21.43
0.00
500
big-M50
-9.12047 107
94.03
0.00
500
persp.
-9.12047 107
176.04
0.00
500
eig. persp.
99.1204 107
85.92
0.00
500
ours
-9.12047 107
0.38
0.00
1000
SOS1
99.4075 107
1085.91
0.00
1000
big-M50
99.4075
107
3600.01
0,07
1000
persp.
9.04075 107
3600.01
0.20
1000
eig. persp.
99.4075
107
341.28
0.00
1000
ours
9..4075
107
0.52
0.00
3000
SOS1
9.09247 107
3600.05
0.98
3000
big-M50
3605.36
3000
persp.
9.12761 107
3600.05
0.59
3000
eig. persp-
3600.41
3000
ours
9.12761 107
20.74
0.00
5000
SOS1
9.00628 107
3600.16 16
2.38
5000
big-M50
3607.68
5000
persp.
-9.1274 107
3600.13
1.07
5000
eig. persp.
3810.32
5000
ours
9.12296 107
58.29
0.00
Table 5: Comparison of time and optimality gap on the synthetic datasets (n = T00000, k 10,
p 0.9, Ag 0.001).
50

---

G.2.2
Synthetic Benchmark with X2 0.001 with Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
SOS1 warm start
22.10878
106
0.05
0.00
100
big-M50 + warm start
-2.10878 10t
0.08
0.00
persp. + warm start
-2.10878
10
0.08
0.00
100
eig. persp. warm start
-2.10878
106
0.01
0.00
100
ours
-2.10878 10
0.29
0.00
500
SOS1 warm start
22.1283 106
6.31
0.00
500
big-M50 warm start
-2.12833 10
37.62
0.00
500
persp. + warm start
-2.12833 10
19.60
0.00
500
eig. persp. warm start
-2.12833
106
0.61
0.00
500
ours
-2.12833 106
0.37
0.00
1000
SOSI warm start
-2.11005 10
77.82
0.00
1000
big-M50 warm start
-2.11005- 10
28.51
0.00
1000
persp. + warm start
-2.11005 10
1148.21
0.00
1000
cig. persp. warm start
-2.11005 10
4.08
0.00
1000
ours
-2.11005 106
0.58
0.00
3000
SOSI warm start
-2.10230 10
2100.58
0.00
3000
big-M50 warm start
-2.10230
10
407.92
0.00
3000
persp. + warm start
-2.10231
106
2275.23
0.00
3000
cig. persp. warm start
-2.10230
10
102.19
0.00
3000
-2.10230 106
1.73
0.00
5000
SOS1 warm start
-2.09635 106
3601.23
0.95
5000
big-M50~ warm start
-2.09635
10
1380.38
0.00
5000
persp. + warm start
-2.09637
106
3600.68
1.00
5000
eig. persp. warm start
-2.09635
106
483.38
0.00
5000
-2.09635
106
4.22
0.00
Table 6: Comparison of time and optimality gap on the synthetic datasets (n
100000, k 10,
0.1, A2 0.001). All baselines use our beam search solution as warm start.
51

---

# of features
method
upper bound
time(s)
gap( ())
100
SOSI + warm start
-5.28726
106
0.05
0.00
100
big-M50- warm start
-5.28726
10
0.26
0.00
100
persp. warm start
-5.28726
10
0.07
0.00
100
cig. persp. warm start
-5.28726 10
0.02
0.00
100
ours
-5.28726 106
0.29
0.00
500
SOSI + warm start
-5.31852 10
9.61
0.00
500
big-M50 warm start
-5.31852 10
3.00
0.00
500
persp. warm start
-5.31852
10
8.06
0.00
500
cig. persp. warm start
-5.31852
10
0.63
0.00
500
ours
-5.31852 10
0.41
0.00
1000
SOS1 warm start
-5.26974 106
88.36
0.00
1000
big-M50~ + warm start
-5.26974 -00
13.79
0.00
1000
persp. warm start
-5.26974 10
54.26
0.00
1000
eig. persp. warm start
-5.26974 106
4.18
0.00
1000
ours
-5.26974 10t
0.52
0.00
3000
SOS1 warm start
55.27668 106
2740.42
0.00
3000
big-M50~ + warm start
-5.27668 100
313.22
0.00
3000
persp. warm start
-5.27668 10
144.09
0.00
3000
eig. persp. warm start
-5.27668 106
101.95
0.00
3000
ours
-5.27668 106
1.75
0.00
5000
SOSI warm start
-5.27126 106
3600.51
1.00
5000
big-M50~ warm start
-5.27126
106
1705.37
0.00
5000
persp. + warm start
-5.27126
10
543.39
0.00
5000
cig. persp. warm start
-5.27126
106
485.74
0.00
5000
ours
-5.27126
106
4.32
0.00
Table 7: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.3, Ag = 0.001). All baselines use our beam search solution as a warm start.
52

---

# of features
method
upper bound
time(s)
gap( ())
100
SOSI + warm start
-1.10084 107
0.05
0.00
100
big-M50- warm start
1..1004 107
0.06
0.00
100
persp. warm start
1.10084 107
0.09
0.00
100
cig. persp. warm start ours
1.10084 107
0.02
0.00
100
1.10084 107
0.29
0.00
500
SOSI + warm start
1.10518 107
7.68
0.00
500
big-M50 warm start
1.10518 107
11.59
0.00
500
persp. warm start
1110518
107
3600.02
0.08
500
cig. persp. warm start ours
1.10518
107
0.63
0.00
500
1.10518 107
0.40
0.00
1000
SOS1 warm start
1.09508 107
90.16
0.00
1000
big-M50~ + warm start
11.9508
107
21.16
0.00
1000
persp. warm start
-1.99588 107
9.85
0.00
1000
eig. persp. warm start
-
-1.09508
107
4.04
0.00
1000
ours
1.09508 107
0.59
0.00
3000
SOS1 warm start
-1.10012 107
3044.33
0.00
3000
big-M50~ + warm start
1111011 107
243.42
0.00
3000
persp. warm start
11100112
141.68
0.00
3000
eig. persp. warm start
- 1.10012 107
100.50
0.00
3000
ours
1.10012 107
1.74
0.00
5000
SOSI warm start
1.09966 107
3603.69
0.99
5000
big-M50~ warm start
-1.09966 107
3600.95
1.00
5000
persp. + warm start
-1.09966
107
3603.33
1.00
5000
cig. persp. warm start
-
-1.09966
107
445.26
0.00
5000
ours
-1.09966 107
4.00
0.00
Table 8: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.5, Àg = 0.001). All baselines use our beam search solution as a warm start.
53

---

# of features
method
upper bound
time(s)
gap( ()
100
SOSI + warm start
-2.43575 107
0.05
0.00
100
big-M50- warm start
-2.43575 107
0.05
0.00
100
persp. warm start
-2.43575
107
0.19
0.00
100
cig. persp. warm start
-2.33575 107
0.02
0.00
100
ours
-2.43575 107
0.32
0.00
500
SOSI + warm start
-2.44185 107
8.94
0.00
500
big-M50 warm start
-2.44185
4.02
0.00
500
persp. warm start
-2.44185 107
271.52
0.00
500
cig. persp. warm start
- 107
0.60
0.00
500
ours
22.41185 107
0.46
0.00
1000
SOS1 warm start
-2.41989 107
83.96
0.00
1000
big-M50~ + warm start
-2.11989
107
11.63
0.00
1000
persp. warm start
-2.11989
107
3600.13
0.20
1000
eig. persp. warm start
-2.41989
107
4.02
0.00
1000
ours
-2.11989 107
0.52
0.00
3000
SOS1 warm start
-2.43713 107
3601.04
0.59
3000
big-M50~ + warm start
-2.43713 -107
3608.31
0.59
3000
persp. warm start
-2.43713 -107
3600.34
0.59
3000
eig. persp. warm start
-2.43713- 107
110.94
0.00
3000
ours
-2.4371 107
1.77
0.00
5000
SOSI warm start
-2.43694 107
3600.18
1.00
5000
big-M50~ warm start
-2.43694
107
3670.94
1.00
5000
persp. + warm start
-2.43694
107
3600.18
1.00
5000
cig. persp. warm start
-2.43694
107
442.81
0.00
5000
ours
-2.43694 107
4.15
0.00
Table 9: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.7, Ag = 0.001). All baselines use our beam search solution as a warm start.
54

---

# of features
method
upper bound
time(s)
gap( ()
100
SOSI + warm start
-9.11026 107
0.04
0.00
100
big-M50- warm start
-9.11026 107
0.03
0.00
100
persp. warm start
-9.11026 107
0.26
0.00
100
cig. persp. warm start
-9.11026 107
0.02
0.00
100
ours
-9.1106 107
0.30
0.00
500
SOSI + warm start
-9.12047 107
48.24
0.00
500
big-M50 warm start
-9.12047 100
34.86
0.00
500
persp. warm start
-9.12047 107
3600.02
0.09
500
cig. persp. warm start
-9.12047 107
0.59
0.00
500
ours
9.12047 107
0.38
0.00
1000
SOS1 warm start
9.04075 107
1090.45
0.00
1000
big-M50~ + warm start
-9.04075 10
3600.01
0.08
1000
persp. warm start
-9.04075 107
3600.08
0.20
1000
eig. persp. warm start
-9.0407 107
4.15
0.00
1000
ours
-9.4077 107
0.52
0.00
3000
SOS1 warm start
-9.12761 10
3600.07
0.59
3000
big-M50~ + warm start
-9.12761 10
3612.88
0.59
3000
persp. warm start
-9.12761
3600.07
0.59
3000
eig. persp. warm start
-9.12761 107
100.26
0.00
3000
ours
9.12261 107
20.74
0.00
5000
SOSI warm start
-9.12916 107
3601.57
1.00
5000
big-M50 warm start
-9.12916 107
3622.52
1.00
5000
persp. warm start
-9.12916 107
3602.87
1.00
5000
cig. persp. warm start ours
-9.12916 107
439.82
0.00
5000
99.1296 107
58.29
0.00
Table 10: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.9, A 0.001). All baselines use our beam search solution as a warm start.
55

---

G.2.3 Synthetic Benchmark with X2 0.1 with No Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
SOS1
2.10878
106
3.04
0.00
100
big-M50
-2.10878
106
0,86
0,00
100
persp.
-2.10878
10c
0.07
0.00
100
eig. persp.
-2.10878
106
0.02
0.00
100
ours
-2.10878
10t
0.36
0.00
500
SOS1
-2.12833
106
8.13
0.00
500
big-M50
-2.12833
106
37.96
0.00
500
persp.
-2.12833
106
3.87
0.00
500
eig. persp.
-2.12833
106
0.63
0.00
500
ours
-2.12833
106
0.37
0.00
1000
SOSI
-2.11005
10t
81.40
0.00
1000
big-M50
-2.11005
10
1007.27
0.00
1000
persp.
-2.11005
10
1195.64
0.00
1000
cig. persp.
-2.11005
10
4.07
0.00
1000
ours
-2.11005
106
0.57
0.00
3000
SOSI
-2.10230
10th
3601.25
0.59
3000
big-M50
-7.99504
10
3608.36
164.51
3000
persp.
-2.10230
106
2055.12
0.00
3000
eig. persp.
-2.10230
106
103.29
0.00
3000
-2.10230
106
1.79
0.00
5000
SOS1
-1.98939-
106
3600.15
6.43
5000
big-M50
-9.69753
105
3619.50
118.35
5000
persp.
-2.99635
106
3600.72
1.00
5000
eig. persp.
-2.09635
106
584.03
0.00
5000
ours
-2.09635
106
4.20
0.00
Table 11: Comparison time and optimality gap on the synthetic datasets (n = 100000, k 10,
0.1, A2 = 0.1).
56

---

# of features
method
upper bound
time(s)
gap(%)
100
SOSI
-5.28726
106
3.37
0.00
100
big-M50
-4.60269
106
1.83
0.00
100
persp.
-5.28726
106
0.07
0.00
100
cig. persp.
-5.28726
106
0.11
0.00
100
ours
-5.28726
106
0.27
0.00
500
SOSI
-5.31852
10t
10,00
0.00
500
big-M50
-5.31827
106
85.52
0.00
500
persp.
-5.31852
106
8.41
0.00
500
cig. persp.
-5.31852
106
1.07
0,00
500
ours
-5.31852
106
0.41
0.00
1000
SOS1
5.269974 106
88.50
0.00
1000
big-M50
5..66974
106
1576.75
0.00
1000
persp.
-5.26974
106
91.53
0.00
1000
eig. persp.
-5.26974
106
4.11
0.00
1000
ours
-5.26974 106
0.52
0.00
3000
SOS1
55.16793
106
3600.25
2.71
3000
big-M50
-3.73049
106
3603.09
42.28
3000
persp.
-5.27668
106
3600.09
0.59
3000
eig. persp.
-5.27668
106
103.60
0.00
3000
ours
-5.27668
106
1.84
0.00
5000
SOS1
-5.16087
106
3600.13
3.16
5000
big-M50
-3.98069
106
3606.79
33.75
5000
persp.
-5.27126
10
3601.72
1.00
5000
eig. persp.
-5.27126
106
611.45
0.00
5000
ours
-5.27126
106
4.42
0.00
Table 12: Comparison of time and optimality gap the synthetic datasets (n 100000, k 10,
= 0.3, A2 0.1).
57

---

of features
method
upper bound
time(s)
gap(%)
100
SOSI
1.10084 107
2.97
0.00
100
big-M50
-1.10084 107
0.44
0.00
100
persp.
-1.10084
107
0.96
0.00
100
cig. persp.
- -1.10084 107
0.13
0.00
100
ours
1.10084 107
0.32
0.00
500
SOS1
- 1.10518 107
9.66
0.00
500
big-M50
-1.10515
107
86.52
0.00
500
persp.
-1.10518
107
3600.03
0.08
500
cig. persp.
-1.10518
107
0.62
0.00
500
ours
- 11.15518 107
0.40
0.00
1000
SOS1
1.09508 107
87.74
0.00
1000
big-M50
-1.09506
107
1539.25
0.00
1000
persp.
-1.09508-
107
3600.12
0.19
1000
eig. persp.
-1.09508.
107
335.72
0.00
1000
ours
-1.09508
107
0.53
0.00
3000
SOS1
1110012 10?
3600.36
0.51
3000
big-M50
3608.09
3000
persp.
1.10012 107
3600.08
0.59
3000
eig. persp.
-1.10012
107
103.03
0.00
3000
ours
- 11100112 107
1.67
0.00
5000
SOS1
1.08838
10?
3602.31
2.05
5000
big-M50
3649.14 14
5000
persp.
1.09966
107
3601.44
1.00
5000
eig. persp.
3758.62
5000
ours
1.09966
107
4.13
0.00
Table 13: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k 10,
p 0.5, A 0.1).
58

---

of features
method
upper bound
time(s)
gap(%)
100
SOSI
-2.43575
107
0.05
0.00
100
big-M50
-2.43573
107
0.41
0.00
100
persp.
-2.43575
107
0.43
0.00
100
cig. persp.
-2.43575
107
0.13
0.00
100
ours
-2.33575
107
0.34
0.00
500
SOS1
-2.44185
107
9.10
0.00
500
big-M50
-2.44167
107
96.59
0.00
500
persp.
-2.44185
107
148.21
0.00
500
cig. persp.
-2.44185
107
23.81
0.00
500
ours
-2.44185
107
0.44
0.00
1000
SOS1
-2.11989
107
89.99
0.00
1000
big-M50
-2.41986
107
1608.58
0.00
1000
persp.
-2.41989
107
3600.10
0.19
1000
eig. persp.
-2.41989
107
4.42
0.00
1000
ours
-2.11989
107
0.53
0.00
3000
SOS1
-2.42629 10?
3601.14
1.04
3000
big-M50
3605.20
3000
persp.
-2.43713 107
3601.16
0.59
3000
eig. persp.
3658.27
3000
ours
-2,43713 107
1.75
0.00
5000
SOS1
-2.42551
10?
3600.14
1.48
5000
big-M50
-2.43694
-17
3650.52
1.00
5000
persp.
-2.43694
107
3606.01
1.00
5000
eig. persp.
3750.44
5000
ours
-2.43694 107
4.22
0.00
Table 14: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
0.7, A 0.1).
59

---

of features
method
upper bound
time(s)
gap(%)
100
SOS1
9.11026
107
0.05
0.00
100
big-M50
-9.11026
107
0.18
0.00
100
persp.
-9.11026
107
0.30
0.00
100
cig. persp.
-9.11026
107
0.13
0.00
100
ours
-9.11026
107
0.31
0.00
500
SOS1
9.11247 107
22.15
0.00
500
big-M50
-9.12047
107
398.02
0.00
500
persp.
-9.12047 107
3600.01
0.08
500
eig. persp.
99.12046
107
84.16
0.00
500
ours
-9.12047 107
0.38
0.00
1000
SOS1
99.4075 107
1090.30
0.00
1000
big-M50
99.4075
107
3600.05
0.01
1000
persp.
9.04075
107
3600.08
0.20
1000
eig. persp.
99.4074
107
881.89
0.00
1000
ours
9..4075
107
0.52
0.00
3000
SOS1
9.11671 107
3600.55
0.71
3000
big-M50
3619.63
3000
persp.
9.12761 107
3600.05
0.59
3000
eig. persp-
3600.21
3000
ours
9.12761 107
20.88
0.00
5000
SOS1
9.02495 107
3602.25
2.17
5000
big-M50
3620.37
5000
persp.
-9.12900 107
3600.18
1.00
5000
eig. persp.
4028.26
5000
ours
-9.1296 107
58.07
0.00
Table 15: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10,
p 0.9, X 0.1).
60

---

G.2.4
Synthetic Benchmark with X2 0.1 with Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
SOS1 warm start
22.10878
106
0.05
0.00
100
big-M50 + warm start
-2.10878 10t
0.08
0.00
persp. + warm start
-2.10878
10
0.07
0.00
100
eig. persp. warm start
-2.10878
106
0.02
0.00
100
ours
-2.10878 10
0.36
0.00
500
SOS1 warm start
22.1283 106
5.04
0.00
500
big-M50 warm start
-2.12833 10
3.78
0.00
500
persp. + warm start
-2.12833 10
3.84
0.00
500
eig. persp. warm start
-2.12833
106
0.60
0.00
500
ours
-2.12833 106
0.37
0.00
1000
SOSI warm start
-2.11005 10
74.29
0.00
1000
big-M50 warm start
-2.11005 10
26.94
0.00
1000
persp. + warm start
-2.11005 10
1175.58
0.00
1000
cig. persp. warm start
-2.11005 10
4.18
0.00
1000
ours
-2.11005 106
0.57
0.00
3000
SOSI warm start
-2.10230 10
2144.33
0.00
3000
big-M50 warm start
-2.10330
10
193.85
0.00
3000
persp. + warm start
-2.10230
106
2108.24
0.00
3000
cig. persp. warm start
-2.10230
10
110.85
0.00
3000
-2.10230 106
1.79
0.00
5000
SOS1 warm start
-2.09635 106
3600.66
0.95
5000
big-M50~ warm start
-2.09635 10
1366.14
0.00
5000
persp. + warm start
-2.09635
106
3601.28
1.00
5000
eig. persp. warm start
-2.09635
106
460.24
0.00
5000
-2.09635 106
4.20
0.00
Table 16: Comparison time and optimality gap on the synthetic datasets (n
100000, k 10,
0.1, A2 0.1). All baselines use our beam search solution as warm start.
61

---

# of features
method
upper bound
time(s)
gap( ()
100
SOSI + warm start
-5.28726
106
0.05
0.00
100
big-M50- warm start
-5.28726
10
0.07
0.00
100
persp. warm start
-5.28726
10
0.07
0.00
100
cig. persp. warm start
-5.28726 10
0.02
0.00
100
ours
-5.28726 106
0.27
0.00
500
SOSI + warm start
-5.31852 10
9.77
0.00
500
big-M50 warm start
-5.31852 10
39.88
0.00
500
persp. warm start
-5.31852
10
8.26
0.00
500
cig. persp. warm start
-5.31852 10
0.63
0.00
500
ours
-5.31852 10
0.41
0.00
1000
SOS1 warm start
-5.26974 106
89.01
0.00
1000
big-M50~ + warm start
-5.26974 10
14.33
0.00
1000
persp. warm start
-5.26974 10
91.61
0.00
1000
eig. persp. warm start
-5.26974 106
4.09
0.00
1000
ours
-5.26974 10t
0.52
0.00
3000
SOS1 warm start
55.27668 106
2743.09
0.00
3000
big-M50~ + warm start
-5.27668 10
317.41
0.00
3000
persp. warm start
-5.27668 10
139.45
0.00
3000
eig. persp. warm start
-5.27668 106
111.91
0.00
3000
ours
-5.27668 106
1.84
0.00
5000
SOSI warm start
-5.27126 106
3600.80
1.00
5000
big-M50~ + warm start
-5.27126
106
1658.42
0.00
5000
persp. + warm start
-5.27126
10
546.50
0.00
5000
cig. persp. warm start
-5.27126
106
454.67
0.00
5000
ours
-5.27126
106
4.42
0.00
Table 17: Comparison of time and optimality gap the synthetic datasets (n 100000, k 10,
= 0.3, Aa 0.1). All baselines use our beam search solution as a warm start.
62

---

# of features
method
upper bound
time(s)
gap( ())
100
SOSI + warm start
-1.10084 107
0.05
0.00
100
big-M50- warm start
1..0004 107
0.06
0.00
100
persp. warm start
1.10084 107
0.90
0.00
100
cig. persp. warm start ours
1.10084 107
0.02
0.00
100
1.10084 107
0.32
0.00
500
SOSI + warm start
1.10518 107
9.26
0.00
500
big-M50 warm start
1.10518 107
2.28
0.00
500
persp. warm start
-1.15188
107
3600.02
0.08
500
cig. persp. warm start ours
1.10518
107
0.64
0.00
500
1.10518 107
0.40
0.00
1000
SOS1 warm start
1.09508 107
89.44
0.00
1000
big-M50~ + warm start
11.9508
107
14.11
0.00
1000
persp. warm start
-1.99588
107
10.09
0.00
1000
eig. persp. warm start
-
-1.09508
107
4.05
0.00
1000
ours
1.09508 107
0.53
0.00
3000
SOS1 warm start
-1.10012 107
3046.03
0.00
3000
big-M50~ + warm start
11110122 107
222.56
0.00
3000
persp. warm start
11100112
141.64
0.00
3000
eig. persp. warm start
1.10012 107
110.14
0.00
3000
ours
1.10012 107
1.67
0.00
5000
SOSI warm start
1.09966 107
3600.73
0.99
5000
big-M50~ + warm start
-1.09966 107
3606.43
1.00
5000
persp. + warm start
-1.09966
107
3600.38
1.00
5000
cig. persp. warm start
-
-1.09966
107
455.87
0.00
5000
ours
-1.09966 107
4.13
0.00
Table 18: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.5, Aa 0.1). All baselines use our beam search solution as a warm start.
63

---

# of features
method
upper bound
time(s)
gap( ())
100
SOSI + warm start
-2.43575 107
0.05
0.00
100
big-M50- warm start
-2.43575 107
0.06
0.00
100
persp. warm start
-2.43575
107
0.43
0.00
100
cig. persp. warm start
-2.33575
107
0.02
0.00
100
ours
-2.43575 107
0.34
0.00
500
SOSI + warm start
-2.44185 107
9.20
0.00
500
big-M50 warm start
-2.44185
107
4.00
0.00
500
persp. warm start
-2.44185 107
146.41
0.00
500
cig. persp. warm start
-2.4418- 107
0.62
0.00
500
ours
22.41185 107
0.44
0.00
1000
SOS1 warm start
-2.41989 107
88.02
0.00
1000
big-M50~ + warm start
-2.11989 107
12.62
0.00
1000
persp. warm start
-2.11989 107
1117.62
0.00
1000
eig. persp. warm start
-2.41989
107
4.05
0.00
1000
ours
-2.11989 107
0.53
0.00
3000
SOS1 warm start
-2.43713 107
3600.80
0.59
3000
big-M50~ + warm start
-2.43713 -00
3600.43
0.59
3000
persp. warm start
-2.43713 -107
3600.68
0.59
3000
eig. persp. warm start
-2.43713- 107
101.46
0.00
3000
ours
-2.4371 107
1.75
0.00
5000
SOSI warm start
-2.43694 107
3600.86
1.00
5000
big-M50~ + warm start
-2.43694 107
3616.71
1.00
5000
persp. + warm start
-2.43694
107
3603.18
1.00
5000
cig. persp. warm start
-2.43694
107
455.57
0.00
5000
ours
-2.43694 107
4.22
0.00
Table 19: Comparison of time and optimality gap the synthetic datasets (n 100000, k 10,
= 0.7, A2 0.1). All baselines use our beam search solution as a warm start.
64

---

# of features
method
upper bound
time(s)
gap( %
100
SOSI + warm start
-9.11026 107
0.04
0.00
100
big-M50- warm start
-9.11026 107
0.04
0.00
100
persp. warm start
-9.11026 107
0.29
0.00
100
cig. persp. warm start
-9.11026 107
0.02
0.00
100
ours
-9.1106 107
0.31
0.00
500
SOSI + warm start
-9.12047 107
52.72
0.00
500
big-M50 warm start
-9.12047 1007
14.34
0.00
500
persp. warm start
-9.12047 -107
166.73
0.00
500
cig. persp. warm start
-9.12047 107
0.60
0.00
500
ours
9.12047 107
0.38
0.00
1000
SOS1 warm start
9.04075 107
1068.59
0.00
1000
big-M50~ + warm start
-9.04075 10
3600.16
0.01
1000
persp. warm start
-9.04075 107
3600.01
0.20
1000
eig. persp. warm start
-9.0407 107
4.17
0.00
1000
ours
-9.4075 107
0.52
0.00
3000
SOS1 warm start
-9.12761 107
3600.39
0.59
3000
big-M50~ + warm start
-9.12761 10
3611.02
0.59
3000
persp. warm start
-9.12761
3600.23
0.59
3000
eig. persp. warm start
-9.12761 107
101.65
0.00
3000
ours
9.12261 107
20.88
0.00
5000
SOSI warm start
-9.12916 107
3603.57
1.00
5000
big-M50~ + warm start
-9.12916 107
3606.64
1.00
5000
persp. + warm start
-9.12916 107
3601.56
1.00
5000
cig. persp. warm start
-9.12916 107
454.63
0.00
5000
ours
-9.1296 107
58.07
0.00
Table 20: Comparison of time and optimality gap the synthetic datasets (n = 0.9, D 0.1). All baselines use our beam search solution as a warm start.
100000, k 10,
65

---

G.2.5 Synthetic Benchmark with X2 10.0 with No Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
SOS1
2.10868
106
3.37
0.00
100
big-M50
-2.10868
106
0.28
0,00
100
persp.
-2.10868
10c
2.06
0.00
100
eig. persp.
-2.10868
106
0.02
0.00
100
ours
-2.10868
10t
0.30
0.00
500
SOS1
-2.12823
106
6.58
0.00
500
big-M50
-2.12819
106
42.85
0.00
500
persp.
-2.12823
106
143.68
0.00
500
eig. persp.
-2.12823
106
0.65
0.00
500
ours
-2.12823
106
0.38
0.00
1000
SOSI
-2.10995
10
70.57
0.00
1000
big-M50
-2.10995
10
1072.15
0.00
1000
persp.
-2.10995
106
452.04
0.00
1000
cig. persp.
-2.10995
10
4.22
0.00
1000
ours
-2.10995
106
0.57
0.00
3000
SOSI
11.77788
10th
3600.81
12.60
3000
big-M50
-7.92671
10
3605.01
166.77
3000
persp.
-2.10220
106
3600.04
0.55
3000
eig. persp.
-2.10220
106
102.51
0.00
3000
-2.10220
106
1.68
0.00
5000
SOS1
-1.88929
106
3600.16
6.43
5000
big-M50
-7.89596
105
3617.52
168.16
5000
persp.
-2.09584
106
3601.36
1.00
5000
eig. persp.
-2.09625
106
475.37
0.00
5000
ours
-2.09625
106
5.09
0.00
Table 21: Comparison time and optimality gap on the synthetic datasets (n = 100000, k 10,
0.1, A2 10.0).
66

---

# of features
method
upper bound
time(s)
gap(%)
100
SOSI
-5.28716
10t
2.98
0.00
100
big-M50
-5.28713
106
0.40
0.00
100
persp.
-5.28716
106
0.77
0.00
100
cig. persp.
-5.28716
106
0.10
0.00
100
ours
-5.28716
106
0.30
0.00
500
SOS1
-5.31842
106
10.06
0.00
500
big-M50
55.18899
106
89.34
0.00
500
persp.
-5.31841
106
8.73
0.00
500
eig. persp.
-5.1142
106
0.60
0.00
500
ours
5.31842
10
0.41
0.00
1000
SOS1
-.266964
106
87.65
0.00
1000
big-M50
5.26964
106
1869.66
0.00
1000
persp.
-5.26964
106
97.81
0.00
1000
eig. persp.
-5.26964
106
226.32
0.00
1000
ours
5..66964
106
0.51
0.00
3000
SOS1
-5.27658
106
3600.46
0.55
3000
big-M50
-3.96610
106
3600.32
33.83
3000
persp.
-5.27658
106
3600.06
0.58
3000
eig. persp.
3615.07
3000
ours
-5.27658 106
1.76
0.00
5000
SOS1
55.1607
106
3603.93
3.16
5000
big-M50
-3.96757
106
3607.78
34.19
5000
persp.
55.27116
106
3600.43
1.00
5000
eig. persp-
-5.27116
10
614.83
0.00
5000
ours
-5.27116
106
4.32
0.00
Table 22: Comparison of time and optimality gap on the synthetic datasets 100000, k = 10,
= 0.3, A2 10.0).
67

---

of features
method
upper bound
time(s)
gap(%)
100
SOSI
1.10083
107
2.83
0.00
100
big-M50
-1.10083
107
0.44
0.00
100
persp.
-1.10083
107
0.46
0.00
100
cig. persp.
-1.10083
107
0.13
0.00
100
ours
-1.10083 107
0.32
0.00
500
SOS1
- 1.10517 107
11.73
0.00
500
big-M50
- 11.10515 107
91.37
0.00
500
persp.
-1.10517 107
3600.01
0.08
500
cig. persp.
-1.10517 107
0.63
0.00
500
ours
- 1.10517 107
0.41
0.00
1000
SOS1
1.09507 107
90.27
0.00
1000
big-M50
- 1.09505 107
1534.37
0.00
1000
persp.
-1.09507
107
3600.04
0.19
1000
eig. persp.
-1.09507
107
3.92
0.00
1000
ours
- -1.9557 107
0.55
0.00
3000
SOS1
1.10011
10?
3600.92
0.51
3000
big-M50
3603.26
3000
persp.
1.10011 107
3601.24
0.58
3000
eig. persp.
3646.60
3000
ours
1.10011 107
1.73
0.00
5000
SOS1
- 1.08825
107
3600.12
2.06
5000
big-M50
9962991
10
3608.37
15.34
5000
persp.
-1.09965
107
3602.42
1.00
5000
eig. persp.
3600.21
5000
ours
1.09965 107
4.24
0.00
Table 23: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10, p 0.5, A 10.0).
68

---

features
method
upper bound
time(s)
gap(%)
100
SOS1
-2.43574
10?
0.05
0.00
100
big-M50
-2.43574 107
0.42
0.00
100
persp.
-2.43574
107
0.55
0.00
100
eig. persp.
-2.43574
107
0.12
0.00
100
ours
-2.43574 107
0.30
0.00
500
SOS1
-2.4488
107
10.39
0.00
500
big-M50
-2.44778
107
85.12
0.00
500
persp.
-2.44184
107
248.24
0.00
500
cig. persp.
-2.44184
107
0.61
0.00
500
ours
-2.44184
107
0.44
0.00
1000
SOS1
-2.11988
107
91.02
0.00
1000
big-M50
-2.11986
107
1415.19
0.00
1000
persp.
-2.41988
107
3600.04
0.16
1000
cig. persp.
-2.41988
107
9.28
0.00
1000
ours
-2.41988
107
0.52
0.00
3000
SOS1
-2.43712 10?
3601.13
0.59
3000
big-M50
3615.34
3000
persp.
22.33712 107
3600.26
0.58
3000
cig. persp.
3669.39
3000
ours
-2,43712 107
1.81
0.00
5000
SOS1
-2.42550
10?
3600.13
1.48
5000
big-M50
-2.43693
107
3607.06
1.00
5000
persp.
-2.43693
107
3601.61
1.00
5000
eig. persp-
-2.43693
107
619.11
0.00
5000
ours
-2.43693
107
4.08
0.00
Table 24: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k 10,
= 0.7, A2 10.0).
69

---

of features
method
upper bound
time(s)
gap(%)
100
SOS1
9.11025
107
0.05
0.00
100
big-M50
-9.11025
107
0.17
0.00
100
persp.
-9.11025
107
0.29
0.00
100
cig. persp.
-9.11018
107
11.35
0.00
100
ours
-9.11025
107
0.32
0.00
500
SOS1
99.1204 107
13.62
0.00
500
big-M50
-9.12046
107
117.86
0.00
500
persp.
-9.12046
107
997.83
0.00
500
eig. persp.
-9.12046
107
84.29
0.00
500
ours
-9.12046 107
0.38
0.00
1000
SOS1
99.4074 107
1201.62
0.00
1000
big-M50
99.0404 107
3600.04
0.01
1000
persp.
9.04074 107
3600.03
0.20
1000
eig. persp.
-9.00074
107
343.13
0.00
1000
ours
-9.0407 10?
0.52
0.00
3000
SOS1
9.12760 107
3601.23
0.59
3000
big-M50
3611.90
3000
persp.
9.12755 107
3601.40
0.59
3000
eig. persp-
3625.51
3000
ours
9.12760 107
20.84
0.00
5000
SOS1
-..8300 107
3600.14
2.65
5000
big-M50
3615.45
5000
persp.
9..1241 107
3600.13
1.07
5000
eig. persp.
3819.54
5000
ours
9.12915 107
57.41
0.00
Table 25: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10,
p 0.9, A 10.0).
70

---

G.2.6 Synthetic Benchmark with X2 10.0 with Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
SOS1 warm start
-2.10868
106
0.05
0.00
100
big-M50 + warm start
-2.10868
10t
0.25
0.00
persp. + warm start
-2.10868
10
2.05
0.00
100
eig. persp. warm start
-2.10868
106
0.01
0.00
100
ours
-2.10868
10
0.30
0.00
500
SOS1 warm start
22.12823 106
6.54
0.00
500
big-M50 warm start
-2.12823
106
3.76
0.00
500
persp. + warm start
-2.12823
10
144.06
0.00
500
eig. persp. warm start
-2.12823
106
0.61
0.00
500
ours
-2.12823 106
0.38
0.00
1000
SOSI warm start
-2.19995 10
67.85
0.00
1000
big-M50 warm start
-2.10995 10
23.00
0.00
1000
persp. + warm start
-2.10995 10
436.41
0.00
1000
cig. persp. warm start
-2.10995
10
4.18
0.00
1000
ours
-2.10995 106
0.57
0.00
3000
SOSI warm start
-2.10220 10
2171.96
0.00
3000
big-M50 warm start
-2.10220
10
241.10
0.00
3000
persp. + warm start
-2.10220
106
3571.06
0.00
3000
cig. persp. warm start
-2.10220
10
99.06
0.00
3000
-2.10220 106
1.68
0.00
5000
SOS1 warm start
-2.09625 106
3602.96
0.95
5000
big-M50~ warm start
-2.09625
10
1309.75
0.00
5000
persp. + warm start
-2.09625
106
3601,45
0.98
5000
eig. persp. warm start
-2.09625
106
442.61
0.00
5000
-2.09625
106
5.09
0.00
Table 26: Comparison time and optimality gap on the synthetic datasets (n
100000, k 10,
0.1, A2 10.0). All baselines use our beam search solution as warm start.
71

---

# of features
method
upper bound
time(s)
gap( ())
100
SOSI + warm start
-5.28716 106
0.05
0.00
100
big-M50- warm start
-5.28716 10
0.25
0.00
100
persp. warm start
-5.28716 10
0.89
0.00
100
cig. persp. warm start
-5.28716 10
0.02
0.00
100
ours
-5.28716 10
0.30
0.00
500
SOSI + warm start
-5.31842 100
7.34
0.00
500
big-M50 warm start
-5.31842 10
2.93
0.00
500
persp. warm start
-5.31842
10
8.30
0.00
500
cig. persp. warm start
-5.31842 10
0.62
0.00
500
ours
-5.31842 10t
0.41
0.00
1000
SOS1 warm start
-5.26964 106
84.96
0.00
1000
big-M50~ + warm start
-5.26964 10
15.10
0.00
1000
persp. warm start
-5.26964 10
97.38
0.00
1000
eig. persp. warm start
-5.26964 106
4.10
0.00
1000
ours
-5.26964 10t
0.51
0.00
3000
SOS1 warm start
-5.27658 106
2809.30
0.00
3000
big-M50~ + warm start
-5.27658 106
315.30
0.00
3000
persp. warm start
-5.27658 10
152.21
0.00
3000
eig. persp. warm start
-5.27658
106
100.20
0.00
3000
ours
-5.77658 106
1.76
0.00
5000
SOSI warm start
-5.27116 106
3601.86
0.99
5000
big-M50 + warm start
-5.27116 106
1100.87
0.00
5000
persp. + warm start
-5.27116 10
633.11
0.00
5000
cig. persp. warm start
-5.27116-
10
460.01
0.00
5000
ours
-5.27116 106
4.32
0.00
Table 27: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.3, A 10.0). All baselines use our beam search solution as a warm start.
72

---

# of features
method
upper bound
time(s)
gap( ())
100
SOSI + warm start
-1.10083 107
0.05
0.00
100
big-M50- warm start
11.0083 107
0.06
0.00
100
persp. warm start
11.10083
107
0.46
0.00
100
cig. persp. warm start ours
1.10083 107
0.02
0.00
100
1.10083 107
0.32
0.00
500
SOSI + warm start
1-11511--1111 107
10.26
0.00
500
big-M50 warm start
1.10517 107
8.51
0.00
500
persp. warm start
1.10517 107
3600.01
0.08
500
cig. persp. warm start ours
1.10517 107
0.64
0.00
500
1.10517 107
0.41
0.00
1000
SOS1 warm start
1.09507 107
88.05
0.00
1000
big-M50~ + warm start
11.9507 107
14.03
0.00
1000
persp. warm start
-1.9507
107
10.25
0.00
1000
eig. persp. warm start ours
1.09507 107
4.13
0.00
1000
1.09507 107
0.55
0.00
3000
SOS1 warm start
11.10011 107
3008.66
0.00
3000
big-M50~ + warm start
1110011 -07
201.32
0.00
3000
persp. warm start
1.10011 -10
153.29
0.00
3000
eig. persp. warm start
-
1.10011 107
102.13
0.00
3000
ours
1.10011 107
1.73
0.00
5000
SOSI warm start
-1.9965 107
3601.18
0.99
5000
big-M50~ + warm start
-1.09965 107
3601.20
1.00
5000
persp. + warm start
-1.09965
107
3602.12
1.00
5000
cig. persp. warm start
-
-1.09965
107
499.65
0.00
5000
ours
-1.09965
107
4.24
0.00
Table 28: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.5, A 10.0). All baselines use our beam search solution as a warm start.
73

---

# of features
method
upper bound
time(s)
gap( ()
100
SOSI + warm start
-2.43574
107
0.05
0.00
100
big-M50- warm start
-2.43574 107
0.16
0.00
100
persp. warm start
-2.43574
107
0.51
0.00
100
cig. persp. warm start
-2.43574 107
0.02
0.00
100
ours
-2.43574 107
0.30
0.00
500
SOSI + warm start
-2.44184 107
8.95
0.00
500
big-M50 warm start
-2.44184 100
2.11
0.00
500
persp. warm start
-2.44184 107
215.86
0.00
500
cig. persp. warm start
-2.44184 107
0.61
0.00
500
ours
-2.4184 107
0.44
0.00
1000
SOS1 warm start
-2.41988 107
86.91
0.00
1000
big-M50~ + warm start
-2.41988 107
12.39
0.00
1000
persp. warm start
-2.41988
107
3600.03
0.08
1000
eig. persp. warm start
-2.41988
107
4.14
0.00
1000
ours
-2.11988 107
0.52
0.00
3000
SOS1 warm start
-2.43712 100
3601.60
0.59
3000
big-M50~ + warm start
-2.43712 -107
3600.43
0.59
3000
persp. warm start
-2.43712
3601.09
0.58
3000
eig. persp. warm start
-2.43712 107
113.06
0.00
3000
ours
2.43712 107
1.81
0.00
5000
SOSI warm start
-2.43693 107
3601.29
1.00
5000
big-M50~ warm start
-2.43693. 107
3624.24
1.00
5000
persp. + warm start
-2.43693
107
3600.19
1.00
5000
cig. persp. warm start
-2.43693
107
457.54
0.00
5000
ours
-2.43693 107
4.08
0.00
Table 29: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.7, Da 10.0). All baselines use our beam search solution as a warm start.
74

---

# of features
method
upper bound
time(s)
gap( (
100
SOSI + warm start
-9.11025 107
0.04
0.00
100
big-M50- warm start
-9.11025 107
0.04
0.00
100
persp. warm start
-9.11025
107
0.29
0.00
100
cig. persp. warm start
-9.11025 107
0.03
0.00
100
ours
-9.1125 107
0.32
0.00
500
SOSI + warm start
-9.12046 107
54.77
0.00
500
big-M50 warm start
-9.12046 10
36.63
0.00
500
persp. warm start
-9.12046 107
306.17
0.00
500
cig. persp. warm start
-9.12046 107
0.59
0.00
500
ours
9.12046 107
0.38
0.00
1000
SOS1 warm start
-9.4074 107
1178.82
0.00
1000
big-M50~ + warm start
-9.04074 107
3600.19
0.06
1000
persp. warm start
-9.04074 107
3600.04
0.19
1000
eig. persp. warm start
-9.04074 107
4.15
0.00
1000
ours
-9.0077 107
0.52
0.00
3000
SOS1 warm start
-9.12760 107
3600.52
0.59
3000
big-M50~ + warm start
-9.12760 107
3606.60
0.59
3000
persp. warm start
-9.12760 -107
3600.05
0.59
3000
eig. persp. warm start
-9.12760 107
103.22
0.00
3000
ours
-9.1266 107
20.84
0.00
5000
SOSI warm start
-9.12915 107
3602.40
1.00
5000
big-M50 + warm start
-9.12915 107
3606.47
1.00
5000
persp. + warm start
-9.12915
107
3600.16
1.00
5000
cig. persp. warm start
-9.12999
107
490.05
0.00
5000
ours
-9.1291 107
57.41
0.00
Table 30: Comparison of time and optimality gap the synthetic datasets (n = 0.9, A = 10.0). All baselines use our beam search solution as a warm start.
100000, k 10,
75

---

G.2.7 Synthetic Benchmark with X2 0.001 with No Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
SOS1
5.42178
10
3600.10
39.82
3000
big-M50
66.4585
104
3600.07
19.16
3000
persp.
-4.05327
104
3600.06
87.02
3000
eig- persp.
-5.42702
104
3600.07
39.69
3000
ours
66.34585
104
3600.49
19.45
4000
SOS1
8.64294
10
3601.19
15.37
4000
big-M50
- 8.64294
104
3600.07
15.32
4000
persp.
8.29421
10
3600.06
20.22
4000
eig- persp.
8.29301
10a
3600.08
6.80
4000
ours
8.64294 104
20.25
0.00
5000
SOSI
9985324
10a
3600.10
24.46
5000
big-M50
- -1.9576
10
3600.81
10.25
5000
persp.
1.09576
10a
3600.07
11.91
5000
eig. persp.
1.09576
10a
3600.07
0.05
5000
ours
-1 -1.09576 10
20.51
0.00
6000
SOSI
1.15043
10
3600.09
21.96
6000
big-M50
- 1..27643
10a
3600.08
9.93
6000
persp.
-1.27643
10
3600.14
9.93
6000
eig. persp.
-1.27643
10
3600.06
12.64
6000
ours
-1.77643
10
1.70
0.00
7000
SOS1
11.50614
1
3600.86
8.32
7000
big-M50
-1.50614
105
3600.09
8.29
7000
persp.
-1.50614
10
3600.21
8.33
7000
eig. persp.
-1.50614
10
106.73
0.00
7000
ours
-1.50614
10
1.67
0.00
Table 31: Comparison time and optimality gap on the synthetic datasets (p = 3000, k 10, 0.1,
d = 0.001).
76

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-1 -1.50599 10a
3601.34
26.68
3000
big-M50
-1.59509
10
3600.08
19.07
3000
persp.
-1.48091
10
3600.99
28.74
3000
cig. persp.
-1.35700
10a
3600.07
40.51
3000
ours
-1.59509
10
3600.67
19.53
4000
SOSI
-2.09337
10a
3600.08
19.87
4000
big-M50
-..17774
10a
3600.07
14.78
4000
persp.
-2.17574
10
3600.06
14.91
4000
cig. persp.
-..02936
108
3600.06
23.67
4000
ours
-2.17574
10a
100.13
0.00
5000
SOS1
-2.71986
10
3600.29
12.00
5000
big-M50
-2.71986
105
3600.07
11.22
5000
persp.
-2.71986
105
3600.40
11.98
5000
eig. persp.
-2.71986
10
2152.71
0.00
5000
ours
-2.71986
10a
20.80
0.00
6000
SOS1
-3.16761
10
3600.70
9.96
6000
big-M50
3.16761
10
3600.09
9.96
6000
persp.
-3.16761
10
3600.06
9.96
6000
eig. persp.
-3
3.16761
10
1968.93
0.00
6000
ours
3.16761
10
20.70
0.00
7000
SOSI
-3.72610
10a
3601.25
8.33
7000
big-M50
-3.72610
10
3601.19
8.33
7000
persp.
-3.72610-
10
3600.14
8.33
7000
cig. persp.
-3.72610
1
2906.29
0.00
7000
ours
-3.72610
10
1.98
0.00
Table 32: Comparison of time and optimality gap on the synthetic datasets (p 3000, k= 10, p 0.3.
Ag = 0.001).
77

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-3.24748
10a
3600.06
22.83
3000
big-M50
-3.33604
10
3600.08
18.79
3000
persp.
-3.03254
10
3600.06
31.54
3000
cig. persp.
-3.04485
108
3600.80
31.01
3000
ours
-3.33004
10
3600.58
19.57
4000
SOSI
- 4.49670
10a
3600.55
16.16
4000
big-M50
-4.33261
10a
3600.07
14.98
4000
persp.
-4.50269
10
3600.06
15.99
4000
cig. persp.
4.30194
10a
3600.06
21.59
4000
ours
4.53261
10a
3600.25
0.63
5000
SOS1
5.57342
10
3600.12
13.18
5000
big-M50
5.63007
105
3601.19
11.68
5000
persp.
-5.63007
10
3600.57
11.75
5000
eig. persp.
5.63007
10
3600.05
0.29
5000
ours
-5.63007
10a
36.97
0.00
6000
SOS1
6.51710
10
3600.08
11.01
6000
big-M50
-6.57799
105
3600.34
9.98
6000
persp. -6.57799
10"
3600.15
9.97
6000
eig. persp.
-6.57799
10
1685.67
0.00
6000
ours
6.57799
10
20.75
0.00
7000
SOSI
-7.46135
10
3600.71
12.13
7000
big-M50
-7.72267
10a
3600.11
8.33
7000
persp.
-7.72267
10
3600.50
8.32
7000
cig. persp.
-7.72267
105
3600.27
10.37
7000
ours
-7.72267
10
21.02
0.00
Table 33: Comparison of time and optimality gap the synthetic datasets (p 3000, k 10, p 0.5.
Ag = 0.001).
78

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
- 7.34855 10a
3600.26
20.63
3000
big-M50
-7.41187
108
3600.05
18.15
3000
persp.
-6.86643
10
3600.74
29.10
3000
cig. persp.
-7.02765
109
3600.06
26.14
3000
ours
-7.41187
10
3601.14
19.60
4000
SOSI
-1.00137
106
3601.49
15.39
4000
big-M50
-1.00103
10
3600.07
15.19
4000
persp.
-9.96124
10
3600.10
15.98
4000
cig. persp.
99.90959
10a
3600.06
17.00
4000
ours
-..0287
10
3600.37
1.88
5000
SOS1
-1.23058
106
3600.25
12.96
5000
big-M50
-1.24027
10
3601.10
11.88
5000
persp.
-1.23565
106
3601.00
12.49
5000
eig. persp.
-1.22403-1
106
3600.05
2.31
5000
ours
-124427
10
466.42
0.00
6000
SOS1
-1.44944 106
3600.09
10.36
6000
big-M50
-1.45417
10
3600.11
9.78
6000
persp.
-1.45417 -
3600.55
9.82
6000
eig. persp.
-1 -1.45417 106
3430.19
0.00
6000
ours
- -1.4541 106
57.67
0.00
7000
SOSI
-1 -1.68757 106
3600.20
9.45
7000
big-M50
-1.70483
106
3601.55
8.34
7000
persp.
-1.70483
106
3600.96
8.13
7000
cig. persp.
-1.70483
106
3432.42
0.00
7000
ours
-1.70483
106
21.42
0.00
Table 34: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.7,
Ag = 0.001).
79

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-2.77903
106
3600.07
19.87
3000
big-M50
-2.78521
106
3600.15
15.90
3000
persp.
-2.76075
106
3600.05
20.66
3000
cig. persp.
-2.74468
106
3600.81
21.37
3000
ours
-2.78521
106
3600.01
19.60
4000
SOSI
-3.75203-
106
3600.12
15.10
4000
big-M50
-3.75202
10
3600.06
13.33
4000
persp.
-3.72754-
106
3600.07
15.88
4000
cig. persp.
-3.72921
106
3600.51
16.56
4000
ours
-3.75203-
10
3600.42
3.50
5000
SOS1
4.61204 106
3600.15
12.27
5000
big-M50
4.61877
10
3600.19
11.37
5000
persp.
4.60317
106
3600.36
12.48
5000
eig. persp.
4.60063
106
3600.26
13.32
5000
ours
4.61877
10
3600.57
0.93
6000
SOS1
5.41709
106
3600.27
10.45
6000
big-M50
-5.43826
106
3600.06
9.77
6000
persp.
-5.41956
106
3600.06
10.39
6000
eig. persp.
-5.40204
106
3600.06
11.80
6000
ours
5.43826
106
3600.65
0.33
7000
SOSI
6.35797
106
3600.07
8.51
7000
big-M50
6.36767
106
3600.38
8.22
7000
persp.
66.55166
106
3600.24
8.62
7000
cig. persp.
-6.36156
106
3610.76
0.53
7000
ours
6.36767
106
3600.78
0.11
Table 35: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.9,
Ag = 0.001).
80

---

G.2.8
Synthetic Benchmark with X2 0.001 with Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
SOSI warm start
6.34585 10
3600.96
19,46
3000
big-M50 warm start
66.3458 10
3601.20
19.16
3000
persp. + warm start
-6.34585
104
3600.10
19.45
3000
eig. persp. warm start
6.34585
10
3600.13
19.46
3000
ours
6.34585 10*
3600.49
19.45
4000
SOS1 warm start
8.64294 10
3600.56
15.37
4000
big-M50 warm start
88.44994 10
3600.87
15.32
4000
persp. + warm start
-8.64294 10*
3600.07
14.33
4000
eig. persp. warm start
-8.64294 10
3600.12
1.64
4000
ours
8.64294 10
20.25
0.00
5000
SOSI warm start
1.09576 10
3600.38
11.91
5000
big-M50 warm start
1.09576
10
3601.86
10.25
5000
persp. + warm start
1.09576
10
3600.07
11.91
5000
cig. persp. warm start
1.09576
10
3389.68
0.00
5000
ours
-1.09576 105
20.51
0.00
6000
SOSI warm start
1.27643 10
3600.46
9.92
6000
big-M50 warm start
-1.27643
10
3601.14
9.93
6000
persp. + warm start
-1.27643
10
3600.05
9.93
6000
cig. persp. warm start
-1.27643
-005
3600.07
12.65
6000
-1.27643 10
1.70
0.00
7000
SOS1 warm start
-1.50614 105
3600.07
5.84
7000
big-M50 warm start
-1.50614 10
3600.13
8.29
7000
persp. + warm start
-1.50614 10
3600.17
8.33
7000
eig. persp. warm start
-1.50614..
10
101.37
0.00
7000
-1.50614 10
1.67
0.00
Table 36: Comparison time and optimality gap on the synthetic datasets (p = 3000, k= 10, 0.1, d = 0.001). All baselines use our beam search solution as warm start.
81

---

# of samples
method
upper bound
time(s)
gap(
3000
SOSI + warm start
-1.59509
10
3600.74
19.53
3000
big-M50 warm start
-1.59509
10
3600.60
19.07
3000
persp. + warm start
-1.59509
10
3601.09
19.53
3000
eig. persp. + warm start
-1.59509-
10
3601.11
19.54
3000
ours
-1.99509 10
3600.67
19.53
4000
SOSI + warm start
-2.1757- 10
3600.07
15.28
4000
big-M50~ warm start
-2.17574 10
3600.10
14.78
4000
persp.
-2.17574 10
3601.12
14.91
4000
cig. persp. warm start
-2.17574 10
3600.11
15.34
4000
ours
-2.17574 10
100.13
0.00
5000
SOS1 warm start
-2.71986 105
3600.44
12.00
5000
big-M50~ warm start
-2.71986 10
3600.06
11.22
5000
persp.
-2.71986 10
3600.14
11.35
5000
eig. persp. warm start
-2.71986
105
2118.55
0.00
5000
ours
-2.71986 10
20.80
0.00
6000
SOS1 warm start
-3.16761 105
3600.73
9.96
6000
big-M50~ warm start
3.16761 10
3600.07
9.96
6000
persp. + warm
-3.16761 10
3601.28
9.96
6000
eig. persp. warm start
-3.16761 105
3600.50
11.79
6000
ours
33.6766 105
20.70
0.00
7000
SOSI warm start
-3.72610 105
3600.61
8.33
7000
big-M50 warm start
-3.72610 10
3600.08
8.33
7000
persp. + warm
-3.72610 10
3600.08
8.33
7000
cig. persp. warm start
-3.72610
10
3600.81
9.45
7000
ours
-3.72610 105
1.98
0.00
Table 37: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.3.
Ag = 0.001). All baselines use our beam search solution as a warm start.
82

---

# of samples
method
upper bound
time(s)
gap %
3000
SOSI + warm start
-3.33604 10
3600.69
19.57
3000
big-M50 warm start
-3.33604 10
3600.17
18.79
3000
persp. + warm start
-3.33604 10
3600.06
19.57
3000
eig. persp. + warm start
-3.33604 10
3600.06
19.58
3000
ours
-3.3600 10
3600.58
19.57
4000
SOSI + warm start
4.53261 10
3600.07
15.24
4000
big-M50~ warm start
4.53261
10
3600.17
14.98
4000
persp.
-.53261
10
3600.08
15.23
4000
cig. persp. warm start
-.53261
10
3600.09
15.41
4000
ours
4.53261 10
3600.25
0.63
5000
SOS1 warm start
5.63007 10
3600.25
12.04
5000
big-M50~ warm start
-5.63007 110
3600.29
11.70
5000
persp.
-5.63007 10
3600.76
11.75
5000
eig. persp. warm start
-5.63007
105
3600.39
0.37
5000
ours
-5.63007 10
36.97
0.00
6000
SOS1 warm start
6.57799 105
3600.29
9.98
6000
big-M50~ warm start
66.7779 10
3600.94
9.98
6000
persp. + warm
6.57799 10
3601.41
9.57
6000
eig. persp. warm start
6.57799 105
1420.54
0.00
6000
ours
6.57799 10
20.75
0.00
7000
SOSI warm start
-7.72267 105
3600.06
8.33
7000
big-M50 warm start
-7.72267
100
3600.55
8.33
7000
persp. + warm
-7.72267
100
3600.39
7.86
7000
cig. persp. warm start
-7.72267
10
3600.81
10.37
7000
ours
-7.72267
105
21.02
0.00
Table 38: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.5,
Ag = 0.001). All baselines use our beam search solution as a warm start.
83

---

# of samples
method
upper bound
time(s)
gap %
3000
SOSI + warm start
-7.41187 10
3600.08
19.60
3000
big-M50 warm start
-7.41187 10
3600.76
18.15
3000
persp. + warm start
-7.41187
10
3600.32
19.60
3000
eig. persp. + warm start
-7.41187 10
3600.07
19.60
3000
ours
-7.41187 10
3601.14
19.60
4000
SOSI + warm start
- -1.00287 106
3600.07
15.22
4000
big-M50~ warm start
11.0287 10
3600.07
14.98
4000
persp.
1.00287
3600.06
15.20
4000
cig. persp. warm start
- 1.00287 10
3600.07
15.61
4000
ours
1.00287 10
3600.37
1.88
5000
SOS1 warm start
-1.24027 106
3600.06
12.08
5000
big-M50~ warm start
-1.24027 100
3600.05
11.88
5000
persp.
-1.24027 106
3600.07
12.07
5000
eig. persp. warm start
-1.24027 106
3600.20
0.97
5000
ours
-1.24027 10
466.42
0.00
6000
SOS1 warm start
-1.45417 100
3601.20
10.00
6000
big-M50~ warm start
-1.45417 10
3600.45
9.78
6000
persp. + warm
-1.4517 10
3600.04
9.82
6000
eig. persp. warm start
-
-1.45417 106
2727.13
0.00
6000
ours
11.4417 106
57.67
0.00
7000
SOSI warm start
-1.70483 10
3600.69
8.34
7000
big-M50 warm start
-1.70483 106
3601.38
8.34
7000
persp. + warm
-1.70483
10
3601.09
8.13
7000
cig. persp. warm start
-1.70483
10
1517.88
0.00
7000
ours
-1.70483
10
21.42
0.00
Table 39: Comparison of time and optimality gap the synthetic datasets (p = 3000, k= 10, p 0.7,
Ag = 0.001). All baselines use our beam search solution as a warm start.
84

---

# of samples
method
upper bound
time(s)
gap( (%)
3000
SOSI + warm start
-2.78521 10
3600.34
19.60
3000
big-M50 warm start
-2.78521
10
3600.74
15.90
3000
persp. + warm start
-2.78521
10
3600.06
19.60
3000
eig. persp. + warm start
-2.78521
10
3600.44
19.61
3000
ours
-2.78521 10
3600.01
19.60
4000
SOSI + warm start
-3.75203 10
3600.06
15.10
4000
big-M50~ warm start
-3.75203 10
3600.07
13.33
4000
persp.
-3.75203
106
3600.06
15.12
4000
cig. persp. warm start
-3.75203
10
3600.29
15,85
4000
ours
-3.75203 10
3600.42
3,50
5000
SOS1 warm start
4..1877 106
3600.07
12.11
5000
big-M50~ warm start
4.61877
106
3601.36
11.37
5000
persp.
-.61187 106
3600.06
12.10
5000
eig. persp. warm start
4.61877 106
3604.20
12.88
5000
ours
4.61877 10
3600.57
0.93
6000
SOS1 warm start
5.43826 10
3600.07
10.02
6000
big-M50~ warm start
5.43826 100
3600.22
9.77
6000
persp. + warm
5.43826
10
3601.43
10.01
6000
eig. persp. warm start
55.43826
106
3600.95
0.85
6000
ours
5.43826 106
3600.65
0.33
7000
SOSI warm start
6.36767 10
3600.06
8.35
7000
big-M50 warm start
6.36767 106
3600.12
8.22
7000
persp. + warm
6.36767 10
3600.06
8.34
7000
cig. persp. warm start
6.36767
1
3600.28
0.35
7000
ours
6.36767 106
3600.78
0.11
Table 40: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.9,
Ag = 0.001). All baselines use our beam search solution as a warm start.
85

---

G.2.9 Synthetic Benchmark with X2 0.1 with No Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
SOS1
6.34575
10
3600.08
19.36
3000
big-M50
6.34575
104
3600.07
19.14
3000
persp.
4.81135
104
3600.14
57.24
3000
eig- persp.
-4.80471
104
3600.06
57.76
3000
ours
6.34575
104
3600.33
19.31
4000
SOS1
8.64284
10
3600.20
15.37
4000
big-M50
8.64284
104
3600.23
15.33
4000
persp.
8.29378
10*
3600.27
20.19
4000
eig- persp.
8.64284
104
3600.49
1.61
4000
ours
8.64284
104
20.81
0.00
5000
SOSI
1.09575
10a
3600.96
11.91
5000
big-M50
- 1.09575
10
3600.08
11.60
5000
persp.
1.09575
10a
3600.07
11.90
5000
eig. persp.
1.09575
10a
3600.18
0.23
5000
ours
-1 -1.09575
10
20.65
0.00
6000
SOSI
-.27744
10a
3600.09
9.92
6000
big-M50
1..77642
10a
3601.02
9.92
6000
persp.
-1.27642
100
3600.93
9.92
6000
eig. persp.
-1.27642
10
3600.74
12.65
6000
ours
-1.77642
103
1.70
0.00
7000
SOS1
11.50613
10
3600.12
8.32
7000
big-M50
11.00613
105
3600.45
8.32
7000
persp.
-1.50613
10
3600.20
8.29
7000
eig. persp. ours
-1.50613
10
113.48
0.00
7000
-1.50613
10
1.67
0.00
Table 41: Comparison time and optimality gap on the synthetic datasets (p = 3000, k 10, 0.1,
d = 0.1).
86

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
- 11.52666
10a
3600.76
24.80
3000
big-M50
-1.59508
10
3601.50
19.07
3000
persp.
-1.28190
10
3600.48
48.57
3000
cig. persp.
-1.34180
10a
3600.42
42.09
3000
ours
-1.59508
10
3600.27
19.40
4000
SOSI
-2.04336
10a
3600.51
22.74
4000
big-M50
-..17573
10a
3600.23
14.78
4000
persp.
-2.17573
10
3600.79
14.88
4000
cig. persp.
-2.99228
10
3600.20
19.95
4000
ours
-..17573
10a
99.53
0.00
5000
SOS1
-2.66411
10
3600.59
14.34
5000
big-M50
-2.11985
105
3600.09
11.22
5000
persp.
-2.71985
10
3600.21
11.96
5000
eig. persp.
-2.71985
109
2019.93
0.00
5000
ours
-2.71985
10a
21.01
0.00
6000
SOS1
-3.16760
10
3600.27
9.96
6000
big-M50
-3.16760
10
3600.07
9.96
6000
persp.
-3.16760
10
3600.07
9.96
6000
eig. persp.
-3.16760-11
10
1196.41
0.00
6000
ours
-3.16700
10
20.91
0.00
7000
SOSI
-3.55314
10
3600.32
13.60
7000
big-M50
-3.72609
10a
3600.80
8.33
7000
persp.
-3.72609
10
3600.06
8.31
7000
cig. persp.
-3.72609
10
3600.07
9.46
7000
ours
-3.72609
10
1.73
0.00
Table 42: Comparison of time and optimality gap on the synthetic datasets (p 3000, k= 10, p 0.3.
Ag = 0.1).
87

---

of samples
method
upper bound
time(s)
gap (6)
3000
SOSI
-3.23979
10a
3601.13
23.04
3000
big-M50
-3.33603
10
3601.02
18.78
3000
persp.
-2.94042
10
3600.41
35.54
3000
cig. persp.
-3.07151
108
3600.07
29.86
3000
ours
-3.33603
10
3600.68
19.44
4000
SOSI
4.35299
10a
3600.08
20.00
4000
big-M50
-..3320
10a
3600.12
14.95
4000
persp.
-4.50184
10
3600.07
15.99
4000
cig. persp.
- 4.41599
10a
3600.06
18.45
4000
ours
4.53260
10a
3600.41
0.62
5000
SOS1
5.63006
10
3600.09
12.04
5000
big-M50
5..3000
10
3600.07
11.69
5000
persp.
55.63006
10
3600.16
11.74
5000
eig. persp.
-5.63006
10
3600.43
0.46
5000
ours
-5.63006
10a
37.26
0.00
6000
SOS1
6.57798
105
3600.98
9.98
6000
big-M50
-6.57798
105
3600.53
9.98
6000
persp. -6.57798
105
3600.13
9.96
6000
eig. persp.
-6.57798
10
1737.91
0.00
6000
ours
6.57798
10
20.64
0.00
7000
SOSI
-7.46134
10
3600.11
12.13
7000
big-M50
-7.72266
10a
3600.96
8.33
7000
persp.
-7.72266
10
3600.23
8.31
7000
cig. persp.
-7.72266
10
3601.08
10.36
7000
ours
-7.72266
10
20.91
0.00
Table 43: Comparison of time and optimality gap on the synthetic datasets (p 3000, k= 10, p 0.5.
Ag = 0.1).
88

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-7 -7.36308 10a
3601.07
20.30
3000
big-M50
-7.41186
10
3600.75
18.15
3000
persp.
-7.11307
10
3600.38
24.54
3000
cig. persp.
-7.14103
109
3600.06
24.13
3000
ours
-7.41186
10
3600.59
19.46
4000
SOSI
-1.00107
106
3600.08
15,42
4000
big-M50
-1.00287
10
3600.07
14.98
4000
persp.
-9.8556
10
3600.06
15.67
4000
cig. persp.
9.88486
10a
3600.06
17.29
4000
ours
1.00287
10
3600.08
1.88
5000
SOS1
11.23516
106
3600.08
12.54
5000
big-M50
11.24027
10
3600.07
11.88
5000
persp.
-1.23565
106
3600.08
12.47
5000
eig. persp.
-1.22304
106
3600.46
14.19
5000
ours
-124027
10
469.43
0.00
6000
SOS1
-1.45416
106
3600.07
10.00
6000
big-M50
-1.45416
10
3601.17
9.78
6000
persp.
-1.45416
106
3601.30
9.82
6000
eig. persp.
-1 -1.45416-1 106
3600.05
0.03
6000
ours
-1.45416 106
59.16
0.00
7000
SOSI
-1.69945
106
3600.08
8.68
7000
big-M50
-1.70483
106
3602.06
8.34
7000
persp.
-1.70483
106
3600.92
8.13
7000
cig. persp.
-1.70483
106
1537.97
0.00
7000
ours
-1.70483
106
21.12
0.00
Table 44: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.7,
Ag = 0.1).
89

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-2.77826
106
3600.41
19.82
3000
big-M50
-2.78468
106
3600.31
15.93
3000
persp.
-2.76199
106
3600.06
20.50
3000
cig. persp.
-2.76520
106
3600.06
20.47
3000
ours
-2.78521
106
3600.23
19.46
4000
SOSI
-3.74434
106
3600.35
15.33
4000
big-M50
-3.75203
10
3600.08
13.35
4000
persp.
-3.73180
106
3600.06
15.73
4000
cig. persp.
-3.74150
106
3600.69
16.17
4000
ours
-3.75203
10
3601.10
3.50
5000
SOS1
4.60905
106
3600.07
12.34
5000
big-M50
4.61877
10
3600.08
11.40
5000
persp.
-4.44788
106
3600.79
13.84
5000
eig. persp.
4.59650
106
3600.06
13.42
5000
ours
4.61877
10
3600.40
0.93
6000
SOS1
5.40939
106
3600.07
10.60
6000
big-M50
-5.43226
106
3600.23
9.79
6000
persp.
-5.42336
106
3600.65
10.30
6000
eig. persp.
-5.43107
106
3600.99
11.20
6000
ours
5.43826
106
3600.26
0.33
7000
SOSI
6.33762
106
3600.07
8.86
7000
big-M50
6.36767
106
3600.07
8.22
7000
persp.
-6.26888
106
3600.06
10.04
7000
cig. persp.
-6.34367
106
3600.05
10.10
7000
ours
6.36767
106
3600.58
0.11
Table 45: Comparison of time and optimality gap the synthetic datasets (p = 3000, k= 10, p 0.9,
Ag = 0.1).
90

---

G.2.10
Synthetic 2 Benchmark with 22 = 0.1 with Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
SOSI warm start
6.34575 10
3600.07
19.36
3000
big-M50 warm start
66.3475 10
3600.12
19.14
3000
persp. + warm start
66.3455 104
3600.14
19.27
3000
eig. persp. warm start
6.34575
10
3600.13
19.45
3000
ours
66.3475 104
3600.33
19.31
4000
SOS1 warm start
8.64284 10
3600.36
15.36
4000
big-M50 warm start
8..4288 10
3600.05
15.33
4000
persp. + warm start
-8.64284 100
3600.52
14.30
4000
eig. persp. warm start
-8.64284
10a
3600.17
1.64
4000
ours
8.64284 10
20.81
0.00
5000
SOSI warm start
1.09575 10
3600.99
11.91
5000
big-M50 warm start
1.09575
10
3601.88
11.60
5000
persp. + warm start
1.09575
10
3600.14
11.90
5000
cig. persp. warm start
1.09575
10
2971.48
0.00
5000
ours
-1.09575 105
20.65
0.00
6000
SOSI warm start
1.27642 10
3600.07
9.92
6000
big-M50 warm start
11.27642
10
3600.62
9,93
6000
persp. + warm start
-1.27642 10
3600.06
9.92
6000
cig. persp. warm start
-1.27642 -105
3601.19
12.65
6000
-1.27642 -05
1.70
0.00
7000
SOS1 warm start
-1.50613 105
3600.13
5.84
7000
big-M50 warm start
-1.00613
10
3600.07
8.32
7000
persp. + warm start
-1.50613
10
3600.19
8.29
7000
eig. persp. warm start
-1.50613
105
103.79
0.00
7000
ours
-1.50613 10
1.67
0.00
Table 46: Comparison time and optimality gap on the synthetic datasets (p 3003000, k= 10, 0.1,
d = 0.1). All baselines use our beam search solution as a warm start.
91

---

# of samples
method
upper bound
time(s)
gap( %)
3000
SOSI + warm start
-1.59508
10
3600.06
19.45
3000
big-M50 warm start
-1.59508
10
3600.22
19.07
3000
persp. + warm start
-1.59508
10
3600.06
19.40
3000
eig. persp. + warm start
-1.59508
10
3600.14
19.52
3000
ours
-1.99508 10
3600.27
19.40
4000
SOSI + warm start
-2.17573 10
3600.40
15.28
4000
big-M50~ warm start
-2.17533
10
3600.10
14.78
4000
persp.
-2.17573
10
3600.06
14.88
4000
cig. persp. warm start
-2.17573 100
3600.10
15.34
4000
ours
-2.17573 1
99.53
0.00
5000
SOS1 warm start
-2.71985 100
3601.08
12.00
5000
big-M50~ warm start
-2.71985 10
3600.07
11.22
5000
persp.
-2.71985 105
3600.06
11.34
5000
eig. persp. warm start
-2.71985
105
1642.59
0.00
5000
ours
-2.71985 10
21.01
0.00
6000
SOS1 warm start
-3.16760 10
3600.83
9.96
6000
big-M50~ warm start
3..6776 10
3600.07
9.96
6000
persp. + warm
-3.16760 10
3600.05
9.96
6000
eig. persp. warm start
-3.16760 10
3600.06
11.77
6000
ours
-3.16760 105
20.91
0.00
7000
SOSI warm start
-3.72609 105
3600.67
8.33
7000
big-M50 warm start
-3.72609 1
3601.58
8.33
7000
persp. + warm
-3.72609
10
3601.01
8.31
7000
cig. persp. warm start
-3.72609
10
3600.55
9.45
7000
ours
-3.72609 10
1.73
0.00
Table 47: Comparison of time and optimality gap the synthetic datasets (p = 3000, k= 10, p 0.3.
Ag = 0.1). All baselines use our beam search solution as a warm start.
92

---

# of samples
method
upper bound
time(s)
gap( %)
3000
SOSI + warm start
-3.33603 10
3600.06
19,49
3000
big-M50 warm start
-3.33603 10
3601.53
18.78
3000
persp. + warm start
-3.33603 10
3600.07
19.47
3000
eig. persp. + warm start
-3.33603 10
3600.06
19.56
3000
ours
-3.33603 10
3600.68
19.44
4000
SOSI + warm start
4.53260 10
3600.06
15.24
4000
big-M50~ warm start
4.53260
10
3600.07
15.04
4000
persp.
-.533660
10
3600.07
15.20
4000
cig. persp. + warm start
-.53360
10
3601.61
15.40
4000
ours
4.53260
10
3600.41
0.62
5000
SOS1 warm start
5.63006 10
3600.47
12.04
5000
big-M50~ warm start
-5.63006
10
3601.70
11.69
5000
persp.
-5.63006 10
3600.06
11.74
5000
eig. persp. warm start
-5.63006
10
3600.50
0.47
5000
ours
-5.63006
10
37.26
0.00
6000
SOS1 warm start
6.57798
10
3601.18
9.98
6000
big-M50~ warm start
66.7798
100
3600.18
9.98
6000
persp. + warm
6..7798
10
3600.06
9.56
6000
eig. persp. warm start
6.57798
105
1376.14
0.00
6000
ours
6.57798 105
20.64
0.00
7000
SOSI warm start
-7.72266 105
3600.10
8.33
7000
big-M50 warm start
-7.72266
100
3600.63
8.33
7000
persp. + warm
-7.72266
1
3600.34
7.86
7000
cig. persp. warm start
-7.72266
10
3600.44
10.37
7000
ours
-7.72266 10
20.91
0.00
Table 48: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.5,
Ag = 0.1). All baselines use our beam search solution as a warm start.
93

---

# of samples
method
upper bound
time(s)
gap( %)
3000
SOSI + warm start
-7.41186 10
3600.06
19.51
3000
big-M50 warm start
-7.41186
10
3601.56
18.15
3000
persp. + warm start
-7.41186
10
3600.87
19.52
3000
eig. persp. + warm start
-7.41186
10
3600.05
19.59
3000
ours
-7.41186 10
3600.59
19.46
4000
SOSI + warm start
- -1.00287 106
3600.07
15.21
4000
big-M50~ warm start
-1.00287 10
3600.09
14.98
4000
persp.
1.00287-109
3600.07
15.18
4000
cig. persp. warm start
- 1.00287 10
3600.06
15.61
4000
ours
1.00287 10
3600.08
1.88
5000
SOS1 warm start
-1.24027 106
3601.15
12.08
5000
big-M50~ warm start
-1.24027 -00
3600.40
11.88
5000
persp.
-1.24027 106
3601,42
12.05
5000
eig. persp. warm start
-1.24027 106
3600.19
0.97
5000
ours
-1.24027 10
469.43
0.00
6000
SOS1 warm start
-1.45416 106
3600.09
10.00
6000
big-M50~ warm start
-1.45416 10
3600.05
9.78
6000
persp. + warm
-1.45416
3600.10
9.82
6000
eig. persp. warm start
-1 -1.45416 106
3600.16
0.22
6000
ours
-1.45416 100
59.16
0.00
7000
SOSI warm start
-1.70483 10
3600.74
8.34
7000
big-M50 warm start
-1.70483
106
3600.07
8.34
7000
persp. + warm
-1.70483
1
3600.06
8.13
7000
cig. persp. warm start
-1.70483
106
1417.14
0.00
7000
ours
-1.70483
10
21.12
0.00
Table 49; Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10 p 0.7,
Ag = 0.1). All baselines use our beam search solution as a warm start.
94

---

# of samples
method
upper bound
time(s)
gap (%)
3000
SOSI + warm start
-2.78521 10
3600.07
19.52
3000
big-M50 warm start
-2.78521
10
3600.07
15.91
3000
persp. + warm start
-2.78521
106
3600.06
19.50
3000
eig. persp. + warm start
-2.78521
10
3600.06
19.60
3000
ours
-2.78521 10
3600.23
19.46
4000
SOSI + warm start
-3.75203 10
3600.07
15.10
4000
big-M50~ warm start
-3.75203 10
3600.27
13.35
4000
persp.
-3.75203
106
3600.51
15.10
4000
cig. persp. warm start
-3.75203
10
3600.07
15,85
4000
ours
-3.75203 10
3601.10
3,50
5000
SOS1 warm start
4..1877 106
3600.07
12.11
5000
big-M50~ warm start
4.61877 106
3600.12
11.40
5000
persp.
-.61187 -06
3600.06
12.09
5000
eig. persp. warm start
4.61877 106
3600.35
12.88
5000
ours
4.61877 10
3600.40
0.93
6000
SOS1 warm start
5.43826 106
3600.37
10.02
6000
big-M50~ warm start
5.43826 100
3600.07
9.79
6000
persp. + warm
5.43826
10
3600.07
10.00
6000
eig. persp. warm start
55.43826
106
3600.08
11.05
6000
ours
5.43826 106
3600.26
0.33
7000
SOSI warm start
6.36767 10
3600.50
8.35
7000
big-M50 warm start
6.36767 106
3600.07
8.22
7000
persp. + warm
6.36767 10
3600.06
8.33
7000
cig. persp. warm start
6.36767 10
3600.05
0.38
7000
ours
6.36767 106
3600.58
0.11
Table 50: Comparison of time and optimality gap the synthetic datasets 3000, k= 10, p 0.9,
Ag = 0.1). All baselines use our beam search solution as a warm start.
95

---

G.2.11 Synthetic 2 Benchmark with 22 = 10.0 with No Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
SOS1
6.33576
10
3600.08
18.27
3000
big-M50
6.33576
104
3600.31
18.47
3000
persp.
66.3576
104
3600.20
17.53
3000
eig- persp.
-4.46130
104
3601.33
69.85
3000
ours
6.33576
104
3601.41
4.54
4000
SOS1
8.63287
10
3600.28
15.23
4000
big-M50
8.63287 104
3600.85
15.21
4000
persp.
8.63287 10
3600.15
12.69
4000
eig- persp.
8.58997 10a
3600.10
1.74
4000
ours
8. 8.63287 104
20.33
0.00
5000
SOSI
1.09474 10
3600.08
11.86
5000
big-M50
- 1.09474 10
3600.28
11.85
5000
persp.
1.09474
10
3600.80
10.44
5000
eig. persp.
1.09474
10T
3600.06
13.55
5000
ours
-1 -1.09474 10
20.71
0.00
6000
SOSI
-1277442
10
3600.19
9.90
6000
big-M50
1..77442
10a
3600.37
9.90
6000
persp.
-1.27542
1
3600.14
9.03
6000
eig. persp.
-1.27542
10
2609.62
0.00
6000
ours
-1.77542
103
1.69
0.00
7000
SOS1
-1.42695
10
3600.12
14.24
7000
big-M50
-1.50512
105
3600.56
8.31
7000
persp.
-1.50512
10
3600.15
7.73
7000
eig. persp. ours
-1.50512-
10
107.90
0.00
7000
-1.50512
10
1.70
0.00
Table 51: Comparison time and optimality gap on the synthetic datasets (p = 3000, k= 10, = 0.1,
d = 10.0).
96

---

of samples
method
upper bound
time(s)
gap (6)
3000
SOSI
- -1.22806
10a
3601.20
23.64
3000
big-M50
-1.59409
10
3600.72
18.52
3000
persp.
-1.59409-
10
3600.14
17.84
3000
cig. persp.
-1.25897
109
3600.88
51.38
3000
ours
-1.59409
10
3600.62
4.68
4000
SOSI
-2.17472
10a
3600.70
15.13
4000
big-M50
-2.17472
10a
3600.07
14.64
4000
persp.
-2.17472
10
3600.05
12.93
4000
cig. persp.
-2.03767
10
3600.20
23.07
4000
ours
-2.17472
10a
86.90
0.00
5000
SOS1
-2.71885
10
3600.32
11.94
5000
big-M50
-2.71885
105
3603.74
11.16
5000
persp.
-2.71885
10
3600.05
10.84
5000
eig. persp.
-2.71885
109
1969.80
0.00
5000
ours
-2.71885
10a
20.75
0.00
6000
SOS1
-3.10076
10
3600.08
12.27
6000
big-M50
-3.16660
10
3600.07
9.93
6000
persp.
-3.16660
10
3600.06
9.24
6000
eig. persp.
3.16660
10
2020.73
0.00
6000
ours
-3.16600
10
21.05
0.00
7000
SOSI
-3.72509
10
3600.17
8.31
7000
big-M50
-3.72509
10a
3601.17
8.31
7000
persp.
-3.72509
10
3600.13
7.78
7000
cig. persp.
-3.72509
10
2139.72
0.00
7000
ours
-3.72509
10
1.71
0.00
Table 52: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.3.
Ag = 10.0).
97

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-3.31006
10a
3600.07
19.45
3000
big-M50
-3.33503
10
3600.07
18.43
3000
persp.
-3.30991
10
3600.07
18.99
3000
cig. persp.
-2.97162
10a
3600.08
34.18
3000
ours
-3.33503
10
3600.25
6.21
4000
SOSI
-4 4.36975
10a
3600.35
19.35
4000
big-M50
4.53119
10a
3600.16
14.84
4000
persp.
-4.50480
10
3600.38
12.96
4000
cig. persp.
4.38770
10a
3600.11
19.13
4000
ours
4.53159
10a
3600.95
0.35
5000
SOS1
5.37846
10
3600.98
17.20
5000
big-M50
5.62900
10
3600.16
11.59
5000
persp.
55.22006
10
3600.07
10.92
5000
eig. persp.
-5.62906
10
3600.19
12.71
5000
ours
-5.62906
10a
31.47
0.00
6000
SOS1
6.51605
105
3600.45
10.98
6000
big-M50
6657799
105
3600.16
9.95
6000
persp. -6.57698
10"
3601.04
9.39
6000
eig. persp.
-6.57698
103
2550.81
0.00
6000
ours
6.57698
10
21.24
0.00
7000
SOSI
-7.65939
10
3600.07
9.20
7000
big-M50
-7.72166
10a
3600.12
8.31
7000
persp.
-7.72166
10
3600.13
7.90
7000
cig. persp.
-7.72166
105
3600.06
10.13
7000
ours
-7.72166
10
20.37
0.00
Table 53: Comparison of time and optimality gap on the synthetic datasets (p 3000, k= 10, p == 0.5.
Ag = 10.0).
98

---

of samples
method
upper bound
time(s)
gap(%)
3000
SOSI
-7 -7.35981
10a
3600.07
19.40
3000
big-M50
-7.41086
10
3600.65
17.88
3000
persp.
-7.25990
10
3600.38
20.81
3000
cig. persp.
-7.18339
108
3600.06
23.35
3000
ours
-7.41086
10
3600.48
8.32
4000
SOSI
-9.95525
10a
3600.07
15,44
4000
big-M50
-1.00277
10
3600.09
14.85
4000
persp.
-9.91636
10
3601.08
14.29
4000
cig. persp.
9..2005
10a
3600.10
16.78
4000
ours
-..0277
10
3600.62
1.66
5000
SOS1
-1.23548
106
3600.08
12.44
5000
big-M50
-1.24017
106
3600.25
11.82
5000
persp.
-1.23555
100
3600.42
11.18
5000
eig. persp.
-1.21712
106
3600.05
14.73
5000
ours
-124017
10
386.72
0.00
6000
SOS1
-1.42629
106
3600.81
12.11
6000
big-M50
-1.45406
10
3600.07
9.75
6000
persp.
-1.45406
106
3600.07
9.44
6000
eig. persp.
-1.45406
106
2941.57
0.00
6000
ours
-1.45406
106
54.13
0.00
7000
SOSI
-1.69139
106
3600.49
9.17
7000
big-M50
-1.70473
106
3600.81
8.32
7000
persp.
-1.70473
106
3600.06
7.85
7000
cig. persp.
-1.70473
106
1585.19
0.00
7000
ours
-1.70473
106
21.30
0.00
Table 54: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.7,
Ag = 10.0).
99

---

of samples
method
upper bound
time(s)
gap
3000
SOSI
-2.78026
106
3600.08
18.78
3000
big-M50
-2.7850
106
3600.17
15.95
3000
persp.
-2.75178
106
3600.06
20.10
3000
cig. persp.
-2.73809
106
3600.11
21.60
3000
ours
-2.78507
106
3600.78
9.79
4000
SOSI
-3.74696
106
3600.08
15.12
4000
big-M50
-3.75192
10
3600.06
13.19
4000
persp.
-3.73412
106
3600.07
14.42
4000
cig. persp.
-3.73458
106
3600.81
16.31
4000
ours
-3.75192
10
3600.32
3.23
5000
SOS1
4.60115 106
3600.29
12.47
5000
big-M50
4.61867
106
3600.17
11.32
5000
persp.
4.60427
106
3600.07
11.73
5000
eig. persp.
4.58981
106
3600.07
13.56
5000
ours
4.61867
10
3600.30
0.89
6000
SOS1
5.42828
106
3600.08
10.18
6000
big-M50
-5.43816
106
3600.12
9.75
6000
persp.
-5.41942
106
3600.07
9.90
6000
eig. persp.
-5.41142
106
106
3600.42
11.59
6000
ours
-5.43816
106
3600.22
0.32
7000
SOSI
6.33201
106
3600.05
8.93
7000
big-M50
6.36757
106
3600.32
8.20
7000
persp.
-6.35270
106
3600.06
8.23
7000
cig. persp.
-6.36342
106
3600.05
0.46
7000
ours
6.36757
106
3600.06
0.10
Table 55: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.9,
Ag = 10.0).
100

---

G.2.12
Synthetic 2 Benchmark with 22 = 10.0 with Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
SOSI warm start
6.33576 10
3601.40
18.27
3000
big-M50 warm start
-6.33576 10
3600.12
18.47
3000
persp. + warm start
66.3376 10
3601.92
17.53
3000
eig. persp. warm start
6.33576 10
3600.22
19.61
3000
ours
6.33576 10*
3601.41
4.54
4000
SOS1 warm start
8.63287 10*
3600.08
15.23
4000
big-M50 warm start
8.63287 10
3600.29
15.21
4000
persp. + warm start
8.63287 10*
3600.30
12.69
4000
eig. persp. warm start
8.63287
10
3600.38
1.32
4000
ours
8.63287 10
20.33
0.00
5000
SOSI warm start
1.09444 10
3600.06
11.86
5000
big-M50 warm start
1.09444 10
3600.15
11.85
5000
persp. + warm start
1.09474
10
3601.20
10.44
5000
cig. persp. warm start
1.09474
10
3601.16
13.51
5000
ours
1.09474 105
20.71
0.00
6000
SOSI warm start
1.27542 10
3600.07
9.90
6000
big-M50 warm start
11.25542 10
3601.39
9.89
6000
persp. + warm start
-1.27542 10
3600.21
9.03
6000
cig. persp. warm start
-1.27542 -
3601.11
12.70
6000
-1.27542 105
1.69
0.00
7000
SOS1 warm start
-1.50512 105
3600.28
5.83
7000
big-M50 warm start
-1.50512
10
3601.02
8.31
7000
persp. + warm start
-1.50512
10
3600.16
7.73
7000
eig. persp. warm start
-1.50512
105
104.66
0.00
7000
-1.50512 10
1.70
0.00
Table 56: Comparison time and optimality gap on the synthetic datasets (p = 3000, k= 10, 0.1,
d = 10.0). All baselines use our beam search solution as warm start.
101

---

# of samples
method
upper bound
time(s)
gap %
3000
SOSI + warm start
-1.59409
10
3601.55
18.52
3000
big-M50 warm start
-1.59409 10
3600.67
18.52
3000
persp. + warm start
-1.59409
10
3600.26
17.84
3000
eig. persp. + warm start
-1.59409
10
3600.07
19.55
3000
ours
-1.59409 10
3600.62
4.68
4000
SOSI + warm start
-2.17472 10
3600.06
15.13
4000
big-M50~ warm start
-2.17472 10
3600.08
14.64
4000
persp.
-2.17472 10
3600.78
12.93
4000
cig. persp. + warm start
-2.17472 1
3600.35
2.79
4000
ours
-2.17472 10
86.90
0.00
5000
SOS1 warm start
-2.71885 105
3601.07
11.94
5000
big-M50~ warm start
-2.71885
10
3600.07
11.16
5000
persp.
-2.71885
10
3600.22
10.25
5000
eig. persp. warm start
-2.71885
105
3109.70
0.00
5000
ours
-2.71885 10a
20.75
0.00
6000
SOS1 warm start
-3.16660 105
3600.92
9.93
6000
big-M50~ warm start
-3.16660 10
3600.06
9.93
6000
persp. + warm
-3.16660 10
3600.04
9.24
6000
eig. persp. warm start
-3.16660 10
3601.01
11.80
6000
ours
-3.16660 10
21.05
0.00
7000
SOSI warm start
-3.72509 105
3600.29
8.31
7000
big-M50 warm start
-3.72509
10
3601.61
8.31
7000
persp. + warm
-3.72509
10
3600.12
7.78
7000
cig. persp. warm start
-3.72509
10
3600.07
9.46
7000
ours
-3.72509
10
1.71
0.00
Table 57: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, 0.3.
Ag = 10.0). All basclines use our beam search solution as a warm start.
102

---

# of samples
method
upper bound
time(s)
gap( %
3000
SOSI + warm start
-3.33503
10
3600.95
18.55
3000
big-M50 warm start
-3.33503
10
3600.08
18.43
3000
persp. + warm start
-3.33503
10
3600.40
18.09
3000
eig. persp. + warm start
-3.33503
10
3600.06
19.55
3000
ours
-3.33503
10
3600.25
6.21
4000
SOSI + warm start
4.53159 10
3600.06
15,09
4000
big-M50~ warm start
4.53159
10
3600.06
14.84
4000
persp.
-4.53159 10
3673.11
12.29
4000
cig. persp. warm start
-4.53159
10
3600.09
15.34
4000
ours
4.53159
10
3600.95
0.35
5000
SOS1 warm start
5.62900 10
3600.07
11.98
5000
big-M50~ warm start
-5.62906
10
3600.08
11.59
5000
persp.
-5.62906
10
3600.06
10.92
5000
eig. persp. warm start
-5.62906
105
3600.31
0.24
5000
ours
-5.62906
10
31.47
0.00
6000
SOS1 warm start
6.57698
10
3600.88
9.95
6000
big-M50~ warm start
6.57698
10
3600.14
9.95
6000
persp. + warm
66.7698
10
3600.06
9.02
6000
eig. persp. warm start
6.57698
10
1703.35
0.00
6000
ours
6.57698 105
21.24
0.00
7000
SOSI warm start
-7.72166 105
3600.06
8.32
7000
big-M50 warm start
-7.72166
10
3601.52
8.31
7000
persp. + warm
-7.72166
10
3600.16
7.90
7000
cig. persp. warm start
-7.72166
10
3600.41
10.39
7000
ours
-7.72166 10
20.37
0.00
Table 58: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, p 0.5,
Ag = 10.0). All basclines use our beam search solution as a warm start.
103

---

# of samples
method
upper bound
time(s)
gap( %
3000
SOSI + warm start
-7.41086 10
3600.63
18.58
3000
big-M50 warm start
-7.41086
10
3601.32
17.88
3000
persp. + warm start
-7.41086
10
3600.59
18.35
3000
eig. persp. + warm start
-7.41086
10
3600.49
19.55
3000
ours
-7.41086 10
3600.48
8.32
4000
SOSI + warm start
-1.00277 106
3600.31
15,06
4000
big-M50~ warm start
-1.00277 10
3600.44
14.85
4000
persp.
-1.00277
3600.06
13.03
4000
cig. persp. + warm start
-1.00277 10
3600.21
15.54
4000
ours
11.0277 10
3600.62
1.66
5000
SOS1 warm start
-1.24017 106
3600.99
12.01
5000
big-M50~ warm start
-1.24017
3600.27
11.82
5000
persp.
-1.24017
3600.75
10.76
5000
eig. persp. warm start
-1.24017 106
3600.07
12.59
5000
ours
-122011 10
386.72
0.00
6000
SOS1 warm start
-1.45406 100
3600.07
9.97
6000
big-M50~ warm start
-1.45406 106
3600.26
9.75
6000
persp. + warm
-1.5506 -006
3600.31
9,44
6000
eig. persp. warm start
-1.45406
106
2855.45
0.00
6000
ours
-1.45406 106
54.13
0.00
7000
SOSI warm start
-1.70473 10
3601.18
8.32
7000
big-M50 warm start
-1.70473 106
3600.20
8.32
7000
persp. + warm
-1.70473
10
3600.18
7.85
7000
cig. persp. warm start
-1.70473
10
1264.32
0.00
7000
ours
11.70473 10
21.30
0.00
Table 59: Comparison of time and optimality gap the synthetic datasets (p = 3000, k= 10, p 0.7,
Ag = 10.0). All basclines use our beam search solution as a warm start.
104

---

# of samples
method
upper bound
time(s)
gap( %
3000
SOSI + warm start
-2.78507 10t
3601.20
18.57
3000
big-M50 warm start
-2.78507 10
3600.07
15.95
3000
persp. + warm start
-2.78507 10
3600.06
18.67
3000
eig. persp. + warm start
-2.78507 10
3600.36
19.55
3000
ours
-2.78507 10
3600.78
9.79
4000
SOSI + warm start
-3.75192 10
3600.30
14.97
4000
big-M50~ warm start
-3.75192
3600.06
13.19
4000
persp.
-3.75192.10
3600.05
13.88
4000
cig. persp. warm start
-3.75192 10
3600.06
15.77
4000
ours
-3.75192 10
3600.32
3.23
5000
SOS1 warm start
-4.61867 106
3601.14
12.04
5000
big-M50~ warm start
-4.61867
10
3600.06
11.32
5000
persp.
-4.61867 10
3600.06
11.39
5000
eig. persp. warm start
-4.61867 10
3600.07
12.85
5000
ours
-4.61867 10
3600.30
0.89
6000
SOS1 warm start
55.43816 100
3600.06
9.98
6000
big-M50~ warm start
55.43816 10
3600.39
9.75
6000
persp. + warm
-5.43816 10
3600.10
9.52
6000
eig. persp. warm start
55.48816 100
3600.13
11.05
6000
ours
5.43816 106
3600.22
0.32
7000
SOSI warm start
6.36757 10
3600.54
8.33
7000
big-M50 warm start
6.36757 106
3600.10
8.20
7000
persp. + warm
6.36757
3600.13
7.98
7000
cig. persp. warm start
6.36757
10
3600.25
0.34
7000
ours
6.36757 106
3600.06
0.10
Table 60: Comparison of time and optimality gap the synthetic datasets (p = 3000, k= 10, p 0.9,
Ag = 10.0). All basclines use our beam search solution as a warm start.
105

---

G.3
Comparison with the Big-M Formulation with different M Values
G.3.1
Synthetic Benchmark with X2 0.001 with No Warmstart
# of features
method
upper bound
time(s)
gap (%)
100
big-M50
22.08844
10
0.39
0.00
100
big-M20
-2.10878
10
0.44
0,00
100
big-M5
-2.10878
10t
0.32
0,00
100
ours
-2.10878
10
0.29
0.00
500
big-M50
-2.12828
106
36.80
0.00
500
big-M20
-2.12833
106
81.36
0.00
500
big-M5
-2.12833
10
52.95
0,00
500
ours
-2.12833
106
0.37
0.00
1000
big- big-M50
-2.11005
10
1911.41
0.00
1000
big-M20
-2.11005
106
1260.21
0,00
1000
big-M5
-2.11005
106
771.63
0.00
1000
ours
-2.11005
106
0.58
0.00
3000
big-M50
-7.83869
10a
3601.20
169.78
3000
big-M20
-1.14989
10
3603.59
83.91
3000
big-M5
-2.10230
1
1998.40
0,00
3000
ours
-2.10230
106
1.73
0.00
5000
big-M50
-1.87965
106
3603.25
12.65
5000
big-M20
-9.85913
10
3602.87
114.78
5000
big-M5
3618.45
5000
-2.09635
106
4.22
0.00
Table 61: Comparison of time and optimality gap on the synthetic datasets 100000, k 10,
0.1, X2 0.001).
106

---

# of features
method
upper bound
time(s)
gap( (%)
100
big-M50
-5.28725
106
0.42
0.00
100
big-M20
-5.28726
106
0,45
0.00
100
big-M5
-3.28726
10
0.36
0.00
100
-5.28726
106
0.29
0.00
500
big-M50
-5.31819
10
89.04
0.00
500
big-M20
-5.31845
106
75.86
0.00
500
big-M5
-5.31852
106
51.98
0.00
500
ours
-3.31852
10c
0.41
0.00
1000
big-M50
-5.26974
106
1528.98
0.00
1000
big-M20
-5.26973
106
1180.33
0.00
1000
big-M5
-5.26974
10
794.14
0.00
1000
ours
-5.26974
106
0.52
0.00
3000
big-M50
-3.72978
10
3602.55
42.31
3000
big-M20
-3.97937
106
3604.47
33.38
3000
big-M5
3660.43
3000
ours
5527668
10
1.75
0.00
5000
big-M50
-4.16126
10
3602.53
27.95
5000
big-M20
-4.20081
10
3601.38
26.74
5000
big-M5
-5.27126
106
3600.14
1.00
5000
ours
-5.27126
106
4.32
0.00
Table 62: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
p 0.3, A 0.001).
# of features
method
upper bound
time(s)
gap(% (4)
100
big-M50
-1.10084
107
0.41
0.00
100
big-M20
-1.10084
107
0.37
0.00
100
big-M5
-1.10084
107
0.33
0.00
100
ours
-1.10084
107
0.29
0.00
500
big-M50
1.10515
107
89.59
0.00
500
big-M20
1.10518
107
79.56
0.00
500
big-M5
-1.10518
107
54.59
0.00
500
ours
-1.10518
107
0.40
0.00
1000
big-M50
1.09506
107
1757.09
0.00
1000
big-M20
-1.09508
107
1133.80
0.00
1000
big-M5
-1.09508
107
744.15
0.00
1000
ours
-1.09508
107
0.59
0.00
3000
big-M50
3615.51
3000
big-M20
3635.15
3000
big-M5
3648.05
3000
ours
11.10012 107
1.74
0.00
5000
big-M50
3612.62
5000
big-M20
-1.01062 107
3612.19
9.90
5000
big-M5
3605.20
5000
ours
1..9966 107
4.00
0.00
Table 63: Comparison of time and optimality gap on the synthetic datasets (n 100000, k = 10, 0 0.5, >2 0.001).
107

---

# of features
method
upper bound
time(s)
gap (%)
100
big-M50
22.3553
107
0.46
0.00
100
big-M20
-2.43575
107
0.38
0,00
100
big-M5
-2.43575
10
0.31
0.00
100
ours
-2.43575
107
0.32
0,00
500
big-M50
-2.4418
107
92.53
0.00
500
big-M20
-2.4418
107
75.08
0.00
500
big-M5
-2.44185
107
52.27
0,00
500
ours
-2.44185
107
0.46
0.00
1000
big-M50
-2.11987
107
1361.05
0,00
1000
big-M20
-2.41988
107
974.43
0.00
1000
big-M5
-2.11989
107
681.46
0.00
1000
2..11989
107
0.52
0.00
3000
big-M50
3640.12
3000
big-M20
3637.70
3000
big-M5
2.43713
107
3600.23
0.56
3000
2.43713
107
1.77
0.00
5000
big-M50
-2.43694
107
3643.28
1.00
5000
big-M20
-2.43694
107
3602.42
1.00
5000
big-M5
3631.91
5000
ours
-2.43694
107
4.15
0.00
Table 64: Comparison of time and optimality gap on the synthetic datasets (n 100000, k = 10,
p 0.7, A2 0.001).
# of features
method
upper bound
time(s)
gap (%)
100
big-M50
-9.11026
107
0.13
0.00
100
big-M20
-9.11026
107
0.13
0.00
100
big-M5
-9.11026
107
0.10
0.00
100
ours
-9.11026
107
0.30
0.00
500
big-M50
99.12447
107
94.03
0.00
500
big-M20
9.12047
107
90.67
0.00
500
big-M5
-9.12047
-107
185.31
0,00
500
ours
-9.12047
107
0.38
0.00
1000
big-M50
-9.04075 107
3600.01
0.07
1000
big-M20
-9.00055
107
3600.50
0.20
1000
big-M5
-9.04075
107
2081.96
0.00
1000
ours
-9.04075
107
0.52
0.00
3000
big-M50
3605,36
3000
big-M20
3623.06
3000
big-M5
3.11.34
3000
ours
-9.12761 107
20.74
0.00
5000
b50
3607.68
5000
big-M20
3616.13
5000
big-M5
3627.90
5000
-9.12916 107
58.29
0.00
Table 65: Comparison time and optimality gap on the synthetic datasets (n 100000, k = 10, p 0.9, X2 0.001).
108

---

G.3.2
Synthetic Benchmark with X2 0.001 with Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
-2.10878
106
0.08
0.00
100
big-M20 warm start
-2.10878
10
0.07
0.00
100
big-M5 warm start
-2.10878
106
0.05
0.00
100
-2.0887
106
0.29
0.00
500
big-M50 + warm start
-2.12833
10
37.62
0.00
500
big-M20 warm start
-2.12833
106
2.82
0.00
500
big-M5 warm start
-2.12833
106
3.00
0.00
500
ours
-2.12833
10
0.37
0.00
1000
big-M50 warm start
-2.11005
106
28.51
0.00
1000
big-M20 + warm start
22.11005
10
17.68
0.00
1000
big-M5 warm start
-2.11005
106
25.46
0.00
1000
ours
-2.11005
10
0.58
0.00
3000
big-M50- warm start
-2.10230
106
407.92
0.00
3000
big-M20 warm start
-2.10230
106
255.31
0.00
3000
big-M5 + warm start
-2.10230
106
400.99
0.00
3000
ours
-2.10230
106
1.73
0.00
5000
big-M50 + warm start
-2.09635 - 106
1380.38
0.00
5000
big-M20 warm start
-2.09635
10
1107.68
0.00
5000
big-M5 warm start ours
-2.09635
106
1673.77
0.00
5000
-2.09635
106
4.22
0.00
Table 66: Comparison of time and optimality gap the synthetic datasets (n 100000, k 10,
= 0.1, A2 = 0.001). All baselines use our beam search solution as a warm start.
109

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
55.28726
106
0.26
0.00
100
big-M20 warm start
-5.28726
106
0.17
0.00
100
big-M5 warm start
-5.28726
106
0.05
0.00
100
-5.28726
106
0.29
0.00
500
big-M50 + warm start
-5.31852
10c
3.00
0.00
500
big-M20 warm start
-5.31852
106
12.76
0.00
500
big-M5 warm start
-5.31852
106
1.94
0.00
500
ours
-5.31852
106
0.41
0.00
1000
big-M50 warm start
-5.26974 106
13.79
0.00
1000
big-M20 + warm start
5.26974
106
18.04
0.00
1000
big-M5 warm start
-5.26974
106
12.19
0.00
1000
ours
-5.26974 106
0.52
0.00
3000
big-M50- warm start
-5.77668 10
313.22
0.00
3000
big-M20 warm start
-5.27668
106
461.86
0.00
3000
big-M5 + warm start
-5.27668
106
245.59
0.00
3000
ours
-5.27668
106
1.75
0.00
5000
big-M50 warm start
-5.27126
106
1705.37
0.00
5000
big-M20 warm start
-5.27126
10
1666.20
0.00
5000
big-M5 warm start
-5.27126
106
3601.58
1.00
5000
ours
-5.27126
106
4.32
0.00
Table 67: Comparison of time and optimality gap on the synthetic datasets (n
100000, k 10,
0.3, A2 0.001). All baselines use our beam search solution as a warm start.
# of features
method
upper bound
time(s)
gap(%)
100
big-M50- + warm start
1.10084
107
0.06
0.00
100
big-Mi big-M20- + warm start
-1.10084
107
0.06
0.00
100
big-M5 + warm start
-1.10084
107
0.04
0.00
100
ours
-1.10084
107
0.29
0.00
500
big-M50 + warm start
1.10518
107
11.59
0.00
500
big-M20 + warm start
1.10518
107
6.55
0.00
500
big-M5 + warm start
1.10518
-10
1.97
0.00
500
ours
-1.10518
107
0.40
0.00
1000
big-M50~ + warm start
1.09508
107
21.16
0.00
1000
big-M20 + warm start
- 1.09508
107
14.95
0.00
1000
big-M5 + warm start
- 1.09508
107
12.99
0.00
1000
ours
-1.95508
107
0.59
0.00
3000
big-M50 + warm start
- -1.10012
107
243.42
0.00
3000
big-M20 + warm start
-1.10012
107
301.69
0.00
3000
big-M5 + start
-1.10012
107
242.64
0.00
3000
ours
-1.10012 107
1.74
0.00
5000
big-M50 warm
1..9966 107
3600.95
1.00
5000
big-M20 + warm start
-1.09966
107
3600.28
1.00
5000
big-M5 + warm start
-1.09966
107
3602.10
570161330.64
5000
ours
-1.09966
107
4.00
0.00
Table 68: Comparison time and optimality gap on the synthetic datasets (n 100000, k = 10.
p 0.5, X₂ 0.001). All baselines
beam search solution warm start. The method big-M5 has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within 1h.
110

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50- + warm start
-2.43575
107
0.05
0.00
100
big-M20 + warm start
-2.43575
107
0.26
0.00
100
big-M5 + warm start
-2.43575
107
0.05
0.00
100
ours
-2.43575
107
0.32
0.00
500
big-M50 + warm start
-2.44185
107
4.02
0.00
500
big-M20 + warm start
-2.44185
107
2.15
0.00
500
big-M5
start
-2.44185
107
20.44
0.00
500
ours
-2.44185
107
0.46
0.00
1000
big-M50 warm start
-2.41989
107
11.63
0.00
1000
big-M20 + warm start
-2.41989
107
13.11
0.00
1000
big-M5 + warm start
-2.41989
107
10.49
0.00
1000
-2.41989
107
0.52
0.00
3000
big-M50 + warm start
-2.43713
-07
3608.31
0.59
3000
big-M20- warm
-2.43713
107
3600.14
0.59
3000
big-M5 + warm start
-2.43713
-10
3600.95
0.56
3000
ours
-2.43713
-107
1.77
0.00
5000
big-M50 + warm start
2.43694
107
3670.94
1.00
5000
big-M20 + warm start -2.33694
107
3604.91
1.00
5000
big-M5 + warm start
-2.43694
100
3646.34
600317552.81
5000
ours
-2.43694
107
4.15
0.00
Table 69: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10.
0.7, X2 = 0.001). All baselines use our beam search solution as warm start. The method big-M5 has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within lh.

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50- + warm start
-9.11026
107
0.03
0.00
100
big-M20 + warm start
-9.11026
107
0.03
0.00
100
big-M5 + warm start
-9.11026
107
0.04
0.00
100
ours
-9.11026
107
0.30
0.00
500
big-M50 + warm start
-9.12047 107
34.86
0.00
500
big-M20 + warm start
-9.12047 107
28.75
0.00
500
big-M5
start
-9.12047 107
18.67
0.00
500
ours
-9.12047 107
0.38
0.00
1000
big-M50 warm start
9.04075 107
3600.01
0.08
1000
big-M20 + warm start
9.04075
107
3600.04
0.20
1000
big-M5 + warm start
99.4075
107
1662.95
0.00
1000
99.40075 107
0.52
0.00
3000
big-M50 + warm start
9.12761 107
3612.88
0.59
3000
big-M20- warm
-9.12761
107
3600.12
0.59
3000
big-M5 + warm start
-9.12761
107
3613.07
222997912.50
3000
ours
-9.12761 107
20.74
0.00
5000
big-M50 + warm start
-9.12916 107
3622.52
1.00
5000
big-M20 + warm start
-9.12916
107
3606.87
1.00
5000
big-M5 + warm start
-9.12916
100
3600.86
618133887.09
5000
ours
-9.12916 107
58.29
0.00
Table 70: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10.
0.9, X2 = 0.001). All baselines use our beam search solution as warm start. The method big-M5 has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within lh.
112

---

G.3.3 Synthetic Benchmark with X2 0.1 with No Warmstart
# of features
method
upper bound
time(s)
gap( (%)
100
big-M50
2.10878
106
0.86
0.00
100
big-M20
-2.10878
10t
0.41
0.00
100
big-M5
-2.10878
10
0.32
0.00
100
-2.00878
106
0.36
0.00
500
big-M50
-2.12833
10
37.96
0.00
500
big-M20
-2.12833
106
27.57
0.00
500
big-M5
-2.12833
106
55.79
0.00
500
ours
-2.12833
10
0.37
0.00
1000
big-M50
-2.11005
106
1007.27
0.00
1000
big-M20
-2.11005
10
373.44
0.00
1000
big-M5
-2.11005
10
74.44
0,00
1000
ours
-2.11005
10
0.57
0.00
3000
big- big-M50
-7.99504
105
3608.36
164.51
3000
big-M20
-1.15295
106
3603.16
83.42
3000
big-M5
-2.10230
106
1289.57
0.00
3000
ours
-2.10230
10
1.79
0.00
5000
big-M50
-9.69753
10a
3619.50
118.35
5000
big-M20
-9.85912
10
3611.26
114.78
5000
big-M5
3610.20
5000
ours
2.09635
10
4.20
0.00
Table 71: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
= 0.1, A2 0.1).
113

---

# of features
method
upper bound
time(s)
gap( (
100
big-M50
4.60269
106
1.83
0.00
100
big-M20
-5.28726
106
0.40
0.00
100
big-M5
-3.28726
10
0.34
0.00
100
-5.28726
106
0.27
0.00
500
big-M50
-5.31827
10
85.52
0.00
500
big-M20
-5.31844
106
72.87
0.00
500
big-M5
-5.31852
106
52.81
0.00
500
ours
-3.31852
10c
0.41
0.00
1000
big-M50
-5.26974
106
1576.75
0.00
1000
big-M20
-5.26974
106
1112.97
0.00
1000
big-M5
-5.26974
10t
791.17
0.00
1000
ours
-5.26974
106
0.52
0.00
3000
big-M50
-3.73049
10
3603.09
42,28
3000
big-M20
-3.98562
106
3602.15
33.17
3000
big-M5
3634.94
3000
ours
5.27668
10
1.84
0.00
5000
big-M50
-3.98069
10
3606.79
33.75
5000
big-M20
-4.58351
10
3615.50
16.16
5000
big-M5
-5.27126
106
3600.14
1.00
5000
ours
-5.27126
106
4.42
0.00
Table 72: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
p = 0.3, 2 0.1).
# of features
method
upper bound
time(s)
gap (9)
100
big-M50
-1.10084
107
0.44
0.00
100
big-M20
-1.10084
107
0.37
0.00
100
big-M5
-1.10084
107
0.33
0.00
100
ours
-1.10084
107
0.32
0.00
500
big-M50
1.10515
107
86.52
0.00
500
big-M20
1.10518
107
77.61
0.00
500
big-M5
-1.10518
107
51.24
0.00
500
ours
-1.10518
107
0.40
0.00
1000
big-M50
1.09506
107
1539.25
0.00
1000
big-M20
-1.09508
107
1118.41
0.00
1000
big-M5
-1.09508
107
893.44
0.00
1000
ours
-1.09508
107
0.53
0.00
3000
big-M50
3608.00
3000
big-M20
3614.30
3000
big-M5
3610.50
3000
ours
11.10012 107
1.67
0.00
5000
big-M50
3649.14 14
5000
big-M20
-1.01130- 107
3600.71
9.83
5000
big-M5
3628.44
5000
ours
-1.09966 107
4.13
0.00
Table 73: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10,
0 0.5, A2 0.1).
114

---

# of features
method
upper bound
time(s)
gap (%)
100
big-M50
2243553
107
0.41
0.00
100
big-M20
-2.43575
107
0.39
0,00
100
big-M5
-2.43575
10
0.30
0.00
100
ours
-2.43575
107
0.34
0,00
500
big-M50
-2.4167
107
96.59
0.00
500
big-M20
-2.4418
107
72.41
0.00
500
big-M5
-2.44185
107
54.87
0,00
500
ours
-2.44185
107
0.44
0.00
1000
big-M50
-2.11986
107
1608.58
0,00
1000
big-M20
-2.41988
107
984.16
0.00
1000
big-M5
-2.41989
107
639.70
0.00
1000
2.411989
107
0.53
0.00
3000
big-M50
3605.20
3000
big-M20
3617.57
3000
big-M5
2.43713
107
3600.08
0.56
3000
2.43713
107
1.75
0.00
5000
big-M50
-2.43694
107
3650.52
1.00
5000
big-M20
-2.43694
107
3624.42
1.00
5000
big-M5
3650.91
5000
ours
-2.43694
107
4.22
0.00
Table 74: Comparison of time and optimality gap on the synthetic datasets (n 100000, k = 10,
0.7, A2 0.1).
# of features
method
upper bound
time(s)
gap (%)
100
big-M50
-9.11026
107
0.18
0.00
100
big-M20
-9.11026
107
0.11
0.00
100
big-M5
-9.11026
107
0.12
0.00
100
ours
-9.11026
107
0.31
0.00
500
big-M50
99.12047
107
398.02
0.00
500
big-M20
9.12047 107
113.34
0.00
500
big-M5
-9.12047
107
187.63
0,00
500
ours
-9.12047
107
0.38
0.00
1000
big-M50
-9.04075 107
3600.05
0.01
1000
big-M20
-9.00055
107
3600.24
0.20
1000
big-M5
-9.04075
107
2116.59
0.00
1000
ours
-9.04075 107
0.52
0.00
3000
big-M50
3619.63
3000
big-M20
3613.39
3000
big-M5
3609.80
3000
ours
-9.12761 107
20.88
0.00
5000
bi0
3620.37
5000
big-M20
3603.08
5000
big-M5
3623.71
5000
-9.12916 107
58.07
0.00
Table 75: Comparison time and optimality gap on the synthetic datasets (n 100000, k = 10, p 0.9, Xa 0.1).
115

---

G.3.4
Synthetic Benchmark with X2 0.1 with Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
-2.10878
106
0.08
0.00
100
big-M20 warm start
-2.10878
10
0.17
0.00
100
big-M5 warm start
-2.18878
106
0.05
0.00
100
-2.0878
106
0.36
0.00
500
big-M50 + warm start
-2.12833
10
3.78
0.00
500
big-M20 warm start
-2.12833
106
2.75
0.00
500
big-M5 warm start
-2.12833
106
3.10
0.00
500
ours
-2.12833
10
0.37
0.00
1000
big-M50 warm start
-2.11005
106
26.94
0.00
1000
big-M20 + warm start
22.11005
10
18.08
0.00
1000
big-M5 warm start
-2.11005
106
16.21
0.00
1000
ours
-2.11005
10
0.57
0.00
3000
big-M50- warm start
-2.10230
106
193.85
0.00
3000
big-M20 warm start
-2.10230
106
255.55
0.00
3000
big-M5 + warm start
-2.10230
106
283.37
0.00
3000
ours
-2.10230
106
1.79
0.00
5000
big-M50 + warm start
-2.09635 - 106
1366.14
0.00
5000
big-M20 warm start
-2.09635
10
1114.05
0.00
5000
big-M5 warm start
-2.09635
106
1660.23
0.00
5000
ours
-2.09635 106
4.20
0.00
Table 76: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.1, A2 = 0.1). All baselines use our beam search solution as a warm start.
116

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
-5.28726
106
0.07
0.00
100
big-M20 + warm start
-5.28726
106
0.07
0.00
100
big-M5 warm start
-5.28726
106
0.05
0.00
100
-5.28726
106
0.27
0.00
500
big-M50 warm start
-5.31852
10
39.88
0.00
500
big-M20 warm start
-5.31852
106
3.02
0.00
500
big-M5 + warm start
-5.31852
10
1.86
0.00
500
ours
-5.31852
106
0.41
0.00
1000
big-M50 warm start
-5.26974 106
14.33
0.00
1000
big-M20 + warm start
-5.26974
106
14.92
0.00
1000
big-M5 warm start
-5.26974
106
13.35
0.00
1000
ours
-5.26974
106
0.52
0.00
3000
big-M50 warm start
-5.27668
106
317.41
0.00
3000
big-M20 warm start
-5.27668
106
470.35
0.00
3000
big-M5 + warm start
-5.27668
106
249.02
0.00
3000
-5.27668
106
1.84
0.00
5000
big-M50- warm start
55.27126
10
1658.42
0.00
5000
big-M20 warm start
-5.27126
106
3607.98
1.00
5000
big-M5 15 warm start
-5.27126
106
14
1.00
5000
ours
-5.27126
10
4.42
0.00
Table 77: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.3, A2 = 0.1). All baselines use our beam search solution as a warm start.
# of features
method
upper bound
time(s)
gap(% %
100
big-M50 + warm
1.10084
107
0.06
0.00
100
big-M20 + warm start
-1.10084
107
0.06
0.00
100
big-M5 + warm start
-1.10084
107
0.04
0.00
100
ours
-1.10084 107
0.32
0.00
500
big-M50 + warm start
-1.10518
107
2.28
0.00
500
big-M20- + warm start
1.10518
107
6.61
0.00
500
big-M5 + warm start
11.10188
107
13.48
0.00
500
ours
1.10518
107
0.40
0.00
1000
big-M50 + warm start
-
1.09508
107
14.11
0.00
1000
big-M20 + warm start
-1.09508
107
13.83
0.00
1000
big-M5 + warm start
-1.09508
107
12.46
0.00
1000
ours
-1.09508
107
0.53
0.00
3000
big-M50 + warm start
1.10012
107
222.56
0.00
3000
big-M20 + warm start
-1.10012
107
290.65
0.00
3000
big-M5 + warm start
-1.10012
107
251.83
0.00
3000
ours
-1.10012
107
1.67
0.00
5000
big-M50 + warm start
-1.09966
107
3606.43
1.00
5000
big-M20 + warm start
-1.09966
107
3611.47
1.00
5000
big-M5 + start
-1.09966
107
3631.72
570161381.99
5000
ours
-1.09966
107
4.13
0.00
Table 78: Comparison time and optimality gap on the synthetic datasets 100000, K 10.
0.5, A2 = 0.1). All baselines use our beam search solution as a warm start. The method big-M5
has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within 1h.
117

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm
-2.43575
107
0.06
0.00
100
big-M20 + warm start -2.43575
10"
0.24
0.00
100
big-M5 + warm start
-2.43575
107
0.04
0.00
100
ours
-2.43575
107
0.34
0.00
500
big-M50 + warm start
-2.44185
107
4.00
0.00
500
big-M20- + warm
-2.4418
-07
8.60
0.00
500
big-M5 + warm start
-2.44185
107
19.75
0.00
500
ours
-2.44185
-177
0.44
0.00
1000
big-M50 + warm start
-2.41989
107
12.62
0.00
1000
big-M20 + warm start
-2.41989
107
12.12
0.00
1000
big-M5 + warm start
-2.41989
107
9.30
0.00
1000
ours
-2.41989
107
0.53
0.00
3000
big-M50 + warm start
-2.43713
107
3600.43
0.59
3000
big-M20 + warm start -2.43713
107
3600.05
0.59
3000
big-M5 + warm start
-2.43713
107
3600.15
0.56
3000
ours
-2.43713 107
1.75
0.00
5000
big-M50 + warm start
-2.43694
107
3616.71
1.00
5000
big-M20 + warm start
-2.43694
107
3615.82
1.00
5000
big-M5 + warm start -2.43694
107
3626.45
600317577.18
5000
ours
-2.43694
107
4.22
0.00
Table 79: Comparison of time and optimality gap the synthetic datasets (n 100000, K 10.
0.7, A2 = 0.1). All baselines use our beam search solution as a warm start. The method big-M5
has some large optimality gaps because Gurobi couldn't ' finish solving the root relaxation within 1h.
# of features
method
upper bound
time(s)
gap(%)
100
big-M50 + warm start
-9.11026
107
0.04
0.00
100
big-M20- + warm start
-9.11026
107
0.03
0.00
100
big-M5 + warm start
-9.11026
107
0.03
0.00
100
-9.11026 107
0.31
0.00
500
big-M50 + warm start
9.12047 107
14.34
0.00
500
big-M20 warm
99.1244 107
28.63
0.00
500
big-M5 + warm start
-9.12047
107
18.60
0.00
500
ours
-9.12047 -07
0.38
0.00
1000
big-M50 + warm start
9.04075 107
3600.16
0.01
1000
big-M20 + warm start
9.04075 107
3600.11
0.20
1000
big-M5 + start
99.4075 107
1687.23
0.00
1000
ours
-9.04075 107
0.52
0.00
3000
big-M50 + warm start
-9.2276 10"
3611.02
0.59
3000
big-M20 + warm start
-9.12761
107
3601.67
0.59
3000
big-M5 + warm start
-9.12761
107
3602.09
222997914.92
3000
ours
-9.12761
107
20.88
0.00
5000
big-M50 + warm start
-9.12916 107
3606.64
1.00
5000
big-M20 + warm start
-9.1291
107
3610.82
1.00
5000
big-M5 + warm start
9.12916
107
3632.24
618133893.79
5000
ours
-9.1216
107
58.07
0.00
Table 80: Comparison of time and optimality gap the synthetic datasets (n 100000, K 10.
= 0.9, X2 = 0.1). All baselines use our beam search solution as a warm start. The method big-M5
has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within 1h.
118

---

G.3.5 Synthetic Benchmark with X2 10.0 with No Warmstart
# of features
method
upper bound
time(s)
gap(%) (
100
big-M50
2.10868
106
0.28
0.00
100
big-M20
-2.10868
10t
0.44
0.00
100
big-M5
-2.10868
10
0.32
0.00
100
-2.10868
106
0.30
0.00
500
big-M50
-2.12819
10
42.85
0.00
500
big-M20
-2.12823
106
28.73
0.00
500
big-M5
-2.12823
106
57.10
0.00
500
ours
-2.12823
10
0.38
0.00
1000
big-M50
-2.10995
106
1072.15
0.00
1000
big-M20
-2.10995
10
451.79
0.00
1000
big-M5
-2.10995
10
904.01
0.00
1000
ours
-2.10995
106
0.57
0.00
3000
big-M50
-7.92671
10
3605.01
166.77
3000
big-M20
-9.76712
10
3602.25
116.51
3000
big-M5
3623.24
3000
2.10220
106
1.68
0.00
5000
big-M50
-7.89596
10a
3617.52
168.16
5000
big-M20
-9.77684
10
3612.02
116.57
5000
big-M5
-1.87959
10
3600.23
12.65
5000
ours
-2.09625
10
5.09
0.00
Table 81: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
p = 0.1, A2 10.0).
119

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50
5.28713
106
0.40
0.00
100
big-M20
-5.28716
106
0.39
0.00
100
big-M5
-5.28716
10
0.35
0.00
100
-5.28716
106
0.30
0.00
500
big-M50
-5.31819
10
89.34
0.00
500
big-M20
-5.31835
106
73.89
0,00
500
big-M5
-5.31842
106
52.55
0.00
500
ours
-3.31842
10
0.41
0.00
1000
big-M50
-5.26964
106
1869.66
0.00
1000
big-M20
-5,26964
10t
1241.04
0.00
1000
big-M5
-5.26964
10
711.84
0.00
1000
ours
-5.26964
106
0.51
0.00
3000
big-M50
-3.96610
10
3600.32
33,83
3000
big-M20
-3.98698
106
3601.87
33.13
3000
big-M5
3604.15
3000
5.27658
106
1.76
0.00
5000
big-M50
-3.96757
10
3607.78
34.19
5000
big-M20
-4.37277 106
3610.74
21.76
5000
big-M5
-5.27116
106
3600.14
1.00
5000
ours
-3.27116
-00
4.32
0.00
Table 82: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
p 0.3, A2 10.0).
# of features
method
upper bound
time(s)
gap( (%)
100
big- big-M50
1.10083
107
0.44
0.00
100
big-M20
-1.10083
107
0.37
0,00
100
big-M5
-1.10083
107
0.33
0.00
100
ours
-1.10083
107
0.32
0.00
500
big-M50
-1. 1.10515
107
91.37
0.00
500
big-M20
-1.10517
107
74.71
0.00
500
big-M5
-1.10517
107
99.37
0,00
500
ours
-1.10517 107
0.41
0.00
1000
big-M50
1.09505
107
1534.37
0.00
1000
big-M20
1.09507
107
1102.78
0.00
1000
big-M5
-1.09507
107
814.92
0.00
1000
-1.09507 107
0.55
0.00
3000
big-M50
3603.26
3000
big-M20
3606.10
3000
big-M5
3606.02
3000
1.10011 107
1.73
0.00
5000
big-M50
9.62941
10
3608.37
15.34
5000
big-M20
-1.00932
107
3602.48
10.04
5000
big-M5
3633.80
5000
ours
11.99955 107
4.24
0.00
Table 83: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10, 0 0.5, >2 10.0).
120

---

# of features
method
upper bound
time(s)
gap( (%)
100
big-M50
2243554
107
0.42
0.00
100
big-M20
-2.43574
107
0.36
0,00
100
big-M5
-2.43574
107
0.37
0.00
100
ours
-2.33574
107
0.30
0,00
500
big-M50
-2.4178
107
85.12
0.00
500
big-M20
-2.4418
107
70.81
0.00
500
big-M5
-2.44184
107
53.85
0,00
500
ours
-2.44184
107
0.44
0.00
1000
big-M50
22.1196
107
1415.19
0,00
1000
big-M20
-2.41987
107
878.61
0.00
1000
big-M5
-2.1198
107
613.98
0.00
1000
2..1198
107
0.52
0.00
3000
big-M50
3615.34
3000
big-M20
3632.97
3000
big-M5
2.43712
107
3600,25
0.56
3000
2.43712
107
1.81
0.00
5000
big-M50
-2.43693
107
3607.06
1.00
5000
big-M20
-2.43693
1077
3603.18
1.00
5000
big-M5
3626.02
5000
ours
-2.43693
107
4.08
0.00
Table 84: Comparison of time and optimality gap on the synthetic datasets (n 100000, k = 10,
p 0.7, A2 10.0).
# of features
method
upper bound
time(s)
gap( (%)
100
big-M50
-9.11025
107
0.17
0.00
100
big-M20
-9.11025
107
0.12
0.00
100
big-M5
-9.11025
107
0.10
0.00
100
ours
-9.11025
107
0.32
0.00
500
big-M50
-9.12046
107
117.86
0.00
500
big-M20
-9.2204
107
128.18
0.00
500
big-M5
-9.12046
107
184.18
0.00
500
ours
-9.12046
107
0.38
0.00
1000
big-M50
-9.04074 107
3600.04
0.01
1000
big-M20
-9.0404
107
3600.17
0.20
1000
big-M5
-9.04074
107
2000.72
0.00
1000
ours
-9.04074 107
0.52
0.00
3000
big-M50
3611,90
3000
big-M20
3625.57
3000
big-M5
3607.35
3000
ours
-9.12760 107
20.84
0.00
5000
b50
3615.45
5000
big-M20
3613.78
5000
big-M5
3613.28
5000
-9.12915 107
57.41
0.00
Table 85: Comparison time and optimality gap on the synthetic datasets (n = 100000, k = 10,
p 0.9, X₂ 10.0).
121

---

G.3.6
Synthetic Benchmark with X2 10.0 with Warmstart
# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
-2.10868
106
0.25
0.00
100
big-M20 warm start
-2.10868
10
0.17
0.00
100
big-M5 warm start
-2.10868
106
0.05
0.00
100
-2.00668
106
0.30
0.00
500
big-M50 + warm start
-2.12823
10
3.76
0.00
500
big-M20 warm start
-2.12823
106
2.68
0.00
500
big-M5 warm start
-2.12823
106
2.17
0.00
500
ours
-2.12823
106
0.38
0.00
1000
big-M50 warm start
-2.10995
106
23.00
0.00
1000
big-M20 + warm start
-2.10995
10
18.30
0.00
1000
big-M5 warm start
-2.10995
106
26.24
0.00
1000
ours
-2.10995
106
0.57
0.00
3000
big-M50- warm start
-2.10220
106
241.10
0.00
3000
big-M20 warm start
-2.10220
106
302.29
0.00
3000
big-M5 + warm start
-2.10220
106
273.21
0.00
3000
ours
-2.10220
106
1.68
0.00
5000
big-M50 + warm start
-2.09625 - 10
1309.75
0.00
5000
big-M20 warm start
-2.09625
10
1142.75
0.00
5000
big-M5 warm start
-2.09625
106
1636.89
0.00
5000
ours
-2.09625
106
5.09
0.00
Table 86: Comparison of time and optimality gap the synthetic datasets (n
100000, k 10,
= 0.1, A2 = 10.0). All baselines use our beam search solution as a warm start.
122

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
-5.28716
106
0.25
0.00
100
big-M20 + warm start
-5.28716
106
0.05
0.00
100
big-M5 warm start
-5.28716
106
0.05
0.00
100
-5.28716 106
0.30
0.00
500
big-M50 + warm start
-5.31842 10
2.93
0.00
500
big-M20 warm start
-5.31842
106
2.99
0.00
500
big-M5 + warm start
-5.31842
10
1.86
0.00
500
ours
-5.31842 106
0.41
0.00
1000
big-M50 warm start
-5.26964 106
15.10
0.00
1000
big-M20 + warm start
-5.26964
106
14.76
0.00
1000
big-M5 warm start
-5.26964
106
11.79
0.00
1000
ours
-5.26964 106
0.51
0.00
3000
big-M50 warm start
-5.27658
106
315.30
0.00
3000
big-M20 warm start
-5.27658
106
390.22
0.00
3000
big-M5 + warm start
-5.27658
106
249.80
0.00
3000
-5.27658
106
1.76
0.00
5000
big-M50- warm start
-5.27116
10
1100.87
0.00
5000
big-M20 warm start
-5.27116
106
1014.36
0.00
5000
big-M5 5 warm start
-5.27116
106
3627.13
1.00
5000
ours
-5.27116
10
4.32
0.00
Table 87: Comparison of time and optimality gap on the synthetic datasets (n
100000, k 10,
= 0.3, A2 10.0). All baselines use our beam search solution as a warm start.
# of features
method
upper bound
time(s)
gap(% %
100
big-M50 + warm
1.10083
107
0.06
0.00
100
big-M20 + warm start
-1.10083
107
0.05
0.00
100
big-M5 + warm start
-1.10083
107
0.05
0.00
100
ours
-1.10083 107
0.32
0.00
500
big-M50 + warm start
-1.
11.1017
107
8.51
0.00
500
big-M20- + warm start
1.10517 107
6.01
0.00
500
big-M5 + warm start
11.1051
107
13.97
0.00
500
ours
1.10517 107
0.41
0.00
1000
big-M50 + warm start
1.09507 107
14.03
0.00
1000
big-M20 + warm start
-1
1.09507
107
15.08
0.00
1000
big-M5 + warm start -1.09507
107
11.81
0.00
1000
ours
-1.09507 107
0.55
0.00
3000
big-M50 + warm start
1.10011 -10
201.32
0.00
3000
big-M20 + warm start
-1.10011 107
292.08
0.00
3000
big-M5 + warm start
-1.10011
107
241.63
0.00
3000
ours
-1.10011
107
1.73
0.00
5000
big-M50 + warm start
-1.09955.
107
3601.20
1.00
5000
big-M20 + warm start
-1.09965
107
3623.71
1.00
5000
big-M5 + start
-1.09965
107
3629.56
570166517.91
5000
ours
-1.09965
107
4.24
0.00
Table 88: Comparison time and optimality gap on the synthetic datasets 100000, K 10.
= 0.5, A2 10.0). All baselines use our beam search solution as warm start. The method big-M5
has some large optimality gaps because Gurobi couldn't finish solving the root relaxation within 1h.
123

---

# of features
method
upper bound
time(s)
gap(%)
100
big-M50 warm start
-2.43574
107
0.16
0.00
100
big-M20 + warm start
-2.43574
10"
0.17
0.00
100
big-M5 + warm start
-2.43574
107
0.04
0.00
100
ours
-2.43574 107
0.30
0.00
500
big-M50 + warm start
22.44184 107
2.11
0.00
500
big-M20- + warm
-2.44184
107
11.80
0.00
500
big-M5 + warm start
-2.44184
107
1.69
0.00
500
ours
-2.44184
107
0.44
0.00
1000
big-M50 + warm start
-2.41988
107
12.39
0.00
1000
big-M20 + warm start
-2.11988
107
13.36
0.00
1000
big-M5 + warm start
-2.41988
107
10.59
0.00
1000
ours
-2.41988
107
0.52
0.00
3000
big-M50 + warm start
-2.43712
107
3600.43
0.59
3000
big-M20 + warm start -2.43712
107
3600.08
0.59
3000
big-M5 + warm start
-2.43712
100
3600.72
0.56
3000
ours
-2.43712
107
1.81
0.00
5000
big-M50 + warm start
-2.43693
107
3624.24
1.00
5000
big-M20 + warm start
-2.43693
107
3603.69
1.00
5000
big-M5 + warm start
-2.43693
107
3646.36
600320013.98
5000
ours
-2.43693
107
4.08
0.00
Table 89: Comparison of time and optimality gap the synthetic datasets (n 100000, K 10.
0.7, A2 10.0). All baselines use our beam search solution as warm start. The method big-M5
has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within 1h.
# of features
method
upper bound
time(s)
gap(%)
100
big-M50 + warm start
-9.11025
107
0.04
0.00
100
big-M20- + warm start
-9.11025
107
0.04
0.00
100
big-M5 + warm start
-9.11025
107
0.04
0.00
100
-9.11025
107
0.32
0.00
500
big-M50 + warm start
9.12046
107
36.63
0.00
500
big-M20 warm
-9.1204
107
28.85
0.00
500
big-M5 + warm start
-9.12046
107
18.35
0.00
500
ours
-9.12046 -107
0.38
0.00
1000
big-M50 + warm start
9.04074 107
3600.19
0.06
1000
big-M20 + warm start
9.04074 107
3600.33
0.20
1000
big-M5 + start
-9.04074 107
1545.53
0.00
1000
ours
-9.04074 107
0.52
0.00
3000
big-M50 + warm start
99.12760 10"
3606.60
0.59
3000
big-M20 + warm start
-9.12760
-0T
3603.80
0.59
3000
big-M5 + warm start
-9.12760
107
3605.53
222998156.52
3000
ours
-9.12760
117
20.84
0.00
5000
big-M50 + warm start
-9.12915 107
3606.47
1.00
5000
big-M20 + warm start
-9.12915
107
3602.91
1.00
5000
big-M5 + warm start
9..2915 107
3634.46
618134564.44
5000
ours
99.1215
107
57.41
0.00
Table 90: Comparison of time and optimality gap the synthetic datasets (n 100000, K 10.
= 0.9, A2 10.0). All baselines use our beam search solution as warm start. The method big-M5
has some large optimality gaps because Gurobi couldn't 't finish solving the root relaxation within 1h.
124

---

G.3.7 Synthetic Benchmark with X2 0.001 with No Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50
6.34585 10
3600.07
19.16
3000
big-M20
6.34585
10*
3600.07
18.11
3000
big-M5
6.34585
10*
2252.12
0.00
3000
6.34585 101
3600.49
19.45
4000
big-M50
8.64294
10*
3600.07
15.32
4000
big-M20
8.64294
10*
3600.09
14.37
4000
big-M5
8.64294
10*
1536.42
0.00
4000
ours
8.64294 10*
20.25
0.00
5000
big-M50
-1.09576
10
3600.81
10.25
5000
big-M20
-1.09576
10a
3600.09
11.72
5000
big-M5
-1.09576
10a
1154.40
0.00
5000
ours
-1.09576
10
20.51
0.00
6000
big-M50
-1.27643
10
3600.08
9.93
6000
big-M20
-1.27643
10
3600.59
9.92
6000
big-M5
-1.27643
10
1018.24
0.00
6000
ours
-1.27643
10
1.70
0.00
7000
big-M50
-1.50614
10
3600.09
8.29
7000
big-M20
-1.50614
105
3600.07
5.79
7000
big-M5
-1.50614
10
851.91
0,00
7000
ours
-1.50614
10
1.67
0.00
Table 91: Comparison of time and optimality gap the synthetic datasets (p 3000, k= 10, 0.1,
A = 0.001).
125

---

of samples
method
upper bound
time(s)
gap %6)
3000
big-M50
1.59509
10
3600.08
19.07
3000
big-M20
-1.59509
10
3600.67
17.35
3000
big-M5
-1.59509
10
3600.06
6.40
3000
ours
-1.59509
10
3600.67
19.53
4000
big-M50
-2.17574
10
3600.07
14.78
4000
big-M20
-2.17574
10
3600.14
14.34
4000
big-M5
-2.17574
10
3600.40
4.63
4000
ours
-2.17574
105
100.13
0.00
5000
big-M50
-2.71986
10
3600.07
11.22
5000
big-M20
-2.71986
10
3600.07
11.12
5000
big-M5
-2.71986
10
2568.73
0.00
5000
ours
-2.71986
10a
20.80
0.00
6000
big-M50
-3.16761
10
3600.09
9.96
6000
big-M20
-3.16761
10
3600.06
9.95
6000
big-M5
-3.16761
10
1834.08
0.00
6000
-3.16761
105
20.70
0.00
7000
big-M50
-3.72610-
10
3601.19
8.33
7000
big-M20
-3.72610
10
3600.40
8.13
7000
big-M5
-3.72610
1
1682.96
0.00
7000
ours
-3.72610
10
1.98
0.00
Table 92: Comparison of time and optimality gap on the synthetic datasets (p = 3000, = 10, = 0.3.
d = 0.001).
of samples
method
upper bound
time(s)
gap(%)
3000
big-M50
-3.33604 10
3600.08
18.79
3000
big-M20
-3.31111
10
3600.31
16.99
3000
big-M5
-3.33604
10
3600.23
7.53
3000
ours
-3.33604
105
3600.58
19.57
4000
big-M50
4.53261
10
3600.07
14.98
4000
big-M20
4.53261
10s
3600.48
13.50
4000
big-M5
-4.53261
10
3600.17
6.23
4000
ours
-4.53261-
105
3600.25
0.63
5000
big-M50
5.63007
10
3601.19
11.68
5000
big-M20
5.63007
105
3600.05
11.25
5000
big-M5
5.63007
10
3600.07
3,85
5000
ours
5.63007
10
36.97
0.00
6000
big-M50
6.57799
10a
3600.34
9.98
6000
big-M20
6.57799
10
3600.43
9,36
6000
big-M5
6.57799
105
3600.05
2.87
6000
66.77799
10
20.75
0.00
7000
big-M50
-7.72267
10
3600.11
8.33
7000
big-M20
-7.72267
105
3600.96
7.85
7000
big-M5
-7.72267
10a
3600.06
2.40
7000
ours
-7.72267
10a
21.02
0.00
Table 93: Comparison of time and optimality gap on the synthetic datasets (p = 3000, = 10, 0.5.
A = 0.001).
126

---

of samples
method
upper bound
time(s)
gap(
3000
big-M50
7.41187 10
3600.05
18.15
3000
big-M20
7.41187
10
3600.20
14.15
3000
big-M5
-7.41187
-105
3600.22
5.94
3000
ours
7..1118 10
3601.14
19.60
4000
big-M50
11.0103
10
3600.07
15.19
4000
big-M20
1.00287
10th
3600.21
11.81
4000
big-M5
-1.0227-
10
3600.11
5,06
4000
ours
-1.00287
10
3600.37
1.88
5000
big-M50
- 1.24027
10
3601.10
11.88
5000
big-M20
-1.24027
106
3601.01
10.26
5000
big-M5
-1.24027
10t
3600.19
4.34
5000
ours
-1.24027
10t
466.42
0.00
6000
big-M50
1..5417
10
3600.11
9.78
6000
big-M20
- 1..5417
10t
3600.08
9.11
6000
big-M5
-1.45417
106
3600.19
3.94
6000
-1.45417 106
57.67
0.00
7000
big-M50
-1.70483
10
3601.55
8.34
7000
big-M20
-1.70483
10
3600.10
7.77
7000
big-M5
-1.70483
10
3600.38
3.39
7000
ours
-1.70483
10
21.42
0.00
Table 94: Comparison of time and optimality gap on the synthetic datasets (p = 3000, = 10, = 0.7,
d = 0.001).
of samples
method
upper bound
time(s)
gap(%)
3000
big-M50
-2.78521
10
3600.15 15
15,90
3000
big-M20
-2.78521
10
3600.05
10.37
3000
big-M5
-2.78521
10
3600.36
3.69
3000
ours
-2.78521
106
3600.01
19.60
4000
big-M50
-3.75202
10
3600.06
13.33
4000
big-M20
-3.75203
10
3600.28
8.64
4000
big-M5
-3.75132
106
3600.37
3.18
4000
ours
-3.75203
10
3600.42
3.50
5000
big-M50
-4.61877
106
3600.19
11.37
5000
big-M20
4.61877
10
3600.05
7.62
5000
big-M5
4.61877
106
3600.06
2.72
5000
ours
-4.1177
106
3600.57
0.93
6000
big-M50
5.43826
10
3600.06
9.77
6000
big-M20
5.43826
10t
3600.07
6.92
6000
big-M5
5.43826
106
3600.15
2.50
6000
5..3826
106
3600.65
0.33
7000
big-M50
6.36767
106
3600.38
8.22
7000
big-M20
66.6767
106
3600.05
6.06
7000
big-M5
-6.66777
10t
3600.21
2.23
7000
ours
66.66767
1
3600.78
0.11
Table 95: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k= 10, 0.9,
A = 0.001).
127

---

G.3.8
Synthetic Benchmark with X2 0.001 with Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50 warm start
6.34585 10
3601.20
19.16
3000
big-M20 warm start
-6.3458
104
3600.06
18.11
3000
big-M5 + warm start
6..4585
10
2435.86
0.00
3000
6.34585 104
3600.49
19.45
4000
big-M50 warm start
8..4294 10*
3600.87
15.32
4000
big-M20 warm start
8.64294
10
3600.12
14.37
4000
big-M5 + warm start
8.64294
10*
1618.73
0.00
4000
ours
8.64294 10*
20.25
0.00
5000
big-M50~ warm start
-1.09576 10
3601.86
10.25
5000
big-M20 warm start
-1.09576-
10a
3601.59
11.72
5000
big-M5 warm start
-1.09576
10a
1530.18
0.00
5000
ours
-1.09576
10
20.51
0.00
6000
big-M50- warm start
1..27643 105
3601.14
9.93
6000
big-M20 warm start
-1.27643
10
3600.64
9.82
6000
big-M5 + warm start
-1.27643
105
972.83
0.00
6000
ours
-1.27643
10
1.70
0.00
7000
big-M50 warm start
-1.50614 10
3600.13
8.29
7000
big-M20 warm start
-1.50614
105
3600.94
5.81
7000
big-M5 warm start
-1.50614
10
824.89
0.00
7000
ours
-1.50614
105
1.67
0.00
Table 96: Comparison of time and optimality gap the synthetic datasets (p 30 10, 0.1,
A = 0.001). All baselines use our beam search solution as a warm start.
128

---

# of samples
method
upper bound
time(s)
gap( %6)
3000
big-M50 warm start -
-1.59509
10
3600.60
19.07
3000
big-M20 warm start
-1.59509.
10
3600.07
17.35
3000
big-M5 warm start
-1.59509
10
3600.05
6.40
3000
ours
-1.59509
10
3600.67
19.53
4000
big-M50 warm start
-2.17574
10
3600.10
14.78
4000
big-M20 warm start
-2.17574
10
3600.68
14.34
4000
big-M5 + warm start
-2.17574
106
3600.04
4.63
4000
ours
-2.17574
105
100.13
0.00
5000
big-M50 warm start
-2.71986
109
3600.06
11.22
5000
big-M20 warm start
-2.71986
10
3601.05
11.12
5000
big-M5 warm start
-2.71986
10
2571.91
0.00
5000
ours
-2.71986
10
20.80
0.00
6000
big- big-M50 warm start
-3.16761 10
3600.07
9.96
6000
big-M20 warm start
-3.16761
10
3600.65
9.95
6000
big-M5 warm start
-3.16761
-10
1861.02
0.00
6000
-3.16761
-10
20.70
0.00
7000
big-M50 warm start
-3.72610- -00
3600.08
8.33
7000
big-M20 warm start
-3.72610
100
3600.41
8.13
7000
big-M5 M5 + warm start
-3.72610
10
1752.62
0.00
7000
ours
-3.72610- 100
1.98
0.00
Table 97: Comparison of time and optimality gap on the synthetic datasets (p = 3000, 10, 0.3,
d = 0.001). All baselines use our beam search solution as a warm start.
# of samples
method
upper bound
time(s)
gap(%) (%)
3000
big-M50 warm start
-3.33604 10
3600.17
18.79
3000
big-M20 warm start
-3.33604
10
3600.12
16.11
3000
big-M5 warm start
-3.33604
10
3600.20
7.53
3000
ours
33.3604
10
3600.58
19.57
4000
big-M50 warm start
4.53261
10
3600.17
14.98
4000
big-M20 warm start
4.53261
10
3600.77
13.50
4000
big-M5 + warm start
-..3266
10
3600.23
6.23
4000
ours
-4.53261-
10
3600.25
0.63
5000
big-M50 warm start
5.63007 105
3600.29
11.70
5000
big-M20 warm start
5.63007
105
3600.07
11.25
5000
big-M5 warm start
5.63007
10a
3600.32
3.38
5000
ours
5.63007
10
36.97
0.00
6000
big-M50 warm start
6..7799 10
3600.94
9.98
6000
big-M20 warm start
-6.57799
10
3600.28
9.36
6000
big-M5 warm start
6.57799
105
3600.06
2.89
6000
ours
-6.77799
105
20.75
0.00
7000
big-M50 warm start
-7.72267 10
3600.55
8.33
7000
big-M20 warm start
-7.72267
105
3600.07
7.85
7000
big-M5 + warm start
-7.72267
10
3600.65
2.40
7000
ours
-7.72267 105
21.02
0.00
Table 98: Comparison of time and optimality gap on the synthetic datasets (p 3000, k= 10, 0.5.
A = 0.001). All baselines use our beam search solution as warm start.
129

---

# of samples
method
upper bound
time(s)
gap %)
3000
big-M50 warm start
-7.41187
10
3600.76
18.15
3000
big-M20 warm start
-7.41187
10
3600.29
14.15
3000
big-M5 warm start
-7.41187
10
3600.04
5.94
3000
ours
-7.41187 10
3601.14
19.60
4000
big-M50 warm start
1.00287 10
3600.07
14.98
4000
big-M20 warm start
1.00287
10
3600.10
11.81
4000
big-M5 + warm start
-1.00287
10
3600.04
5,06
4000
ours
-1.00287 10"
3600.37
1.88
5000
big-M50 warm start
11.24027 10
3600.05
11.88
5000
big-M20 warm start
11.24027
10%
3600.49
10.26
5000
big-M5 warm start
- 11.24027
106
3600.24
4.34
5000
ours
- 11.24027 106
466.42
0.00
6000
big- big-M50 warm start
- -1.45417 10
3600.45
9.78
6000
big-M20 warm start
- 1-45417 10
3600.26
9.06
6000
big-M5 warm start
-1.45417 10%
3600.07
3.94
6000
- -1.45417 106
57.67
0.00
7000
big-M50 warm start
-1.70483 106
3601.38
8.34
7000
big-M20 warm start
-1.70483
106
3600.06
7.77
7000
big-M5 M5 + warm start
-1.70483
10
3600.05
3.39
7000
ours
-1.70483
10
21.42
0.00
Table 99: Comparison of time and optimality gap on the synthetic datasets (p 3000, == 10, 0.7,
d = 0.001). All baselines use our beam search solution as a warm start.
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50 warm start
-2.78521
10
3600.74
15.90
3000
big-M20 warm start
-2.78521
10
3600.24
10.37
3000
big-M5 warm start
-2.78521
10
3600.32
3.69
3000
ours
-2.78521
10
3600.01
19.60
4000
big-M50 warm start
-3.75203 10
3600.07
13.33
4000
big-M20 warm start
-3.75203
106
3600.24
8.64
4000
big-M5 + warm start
-3.75203
106
3600.16
3.16
4000
ours
-3.75203
10
3600.42
3.50
5000
big-M50 warm start
4.61877 106
3601.36
11.37
5000
big-M20 warm start
4.61877
10
3600.23
7.62
5000
big-M5 warm start
-4.11877
106
3600.07
2.72
5000
ours
-..1187 106
3600.57
0.93
6000
big-M50 warm start
5.43826
10"
3600.22
9.77
6000
big-M20 warm start
5.43826
106
3601.07
6.92
6000
big-M5 warm start
5.43826
10
3600.43
2.50
6000
5.43826
106
3600.65
0.33
7000
big-M50 warm start
6.36767 106
3600.12
8.22
7000
big-M20 warm start
6.36767
106
3600.57
6.06
7000
big-M5 + warm start
66.36767
106
3600.04
2.23
7000
ours
6.36767 106
3600.78
0.11
Table 100: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10, = 0.9, > 0.001). All baselines use our beam search solution as warm start.
130

---

G.3.9 Synthetic Benchmark with X2 0.1 with No Warmstart
# of samples
method
upper bound
time(s)
gap (%)
3000
big-M50
6.34575 10
3600.07
19.14
3000
big-M20
6.34575
10*
3600.11
18.10
3000
big-M5
6.34575
10*
2452.71
0.00
3000
6.34575 104
3600.33
19.31
4000
big-M50
8.64284
10*
3600.23
15.33
4000
big-M20
8.64284
10*
3600.11
14.04
4000
big-M5
8.64284
10*
1675.19
0.00
4000
ours
8.64284 10*
20.81
0.00
5000
big-M50
-1.09575
10
3600.08
11.60
5000
big-M20
-1.09575
100
3600.07
9.94
5000
big-M5
-1.09575
10
1316.91
0.00
5000
ours
-1.09575
10
20.65
0.00
6000
big-M50
11.27642
10
3601.02
9.92
6000
big-M20
-1.27642
10
3600.06
9.92
6000
big-M5
-1.27642
10
1040.84
0.00
6000
ours
-1.27642
10
1.70
0.00
7000
big-M50
-1.50613
10a
3600.45
8.32
7000
big-M20
-1.50613
1
3600.10
5.74
7000
big-M5
-1.50613
10
901.46
0.00
7000
ours
-1.50613
105
1.67
0.00
Table 101: Comparison of time and optimality gap the synthetic datasets (p 3000, k 10.
= 0.1, D2 = 0.1).
131

---

of samples
method
upper bound
time(s)
gap (%)
3000
big-M50
1.59508
10
3601.50
19.07
3000
big-M20
-1.99508
10
3600.41
17.35
3000
big-M5
-1.59508
10
3600.05
6.40
3000
ours
-1.59508
10
3600.27
19.40
4000
big-M50
-2.17573
10
3600.23
14.78
4000
big-M20
-2.17573
10
3600.11
14.34
4000
big-M5
-2.17573
10
3600.36
4.63
4000
ours
-2.17573
105
99.53
0.00
5000
big-M50
-2.71985
10
3600.09
11.22
5000
big-M20
-2.71985
10
3600.06
11.29
5000
big-M5
-2.71985
10
2986.61
0.00
5000
ours
-2.71985
10a
21.01
0.00
6000
big-M50
-3.16760
10
3600.07
9.96
6000
big-M20
-3.16760
10
3601.06
9.95
6000
big-M5
-3.16760
10
2025.19
0.00
6000
-3.16760
105
20.91
0.00
7000
big-M50
-3.72609
10a
3600.80
8.33
7000
big-M20
-3.72609
10
3600.81
8.12
7000
big-M5
-3.72609
10
1452.72
0,00
7000
ours
-3.72609
10
1.73
0.00
Table 102: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10,
p = 0.3, 22 0.1).
of samples
method
upper bound
time(s)
gap (%)
3000
big-M50
-3.33603 10
3601.02
18.78
3000
big-M20
-3.33603
10
3600.70
16.11
3000
big-M5
-3.33603
10
3600.22
7.55
3000
ours
-3.33603
105
3600.68
19.44
4000
big-M50
4.53260
10s
3600.12
14.95
4000
big-M20
4.53260
10s
3600.25
13.47
4000
big-M5
-4.53260
10
3600.05
6.23
4000
ours
-4.53260
10
3600.41
0.62
5000
big-M50
5.63006
10
3600.07
11.69
5000
big-M20
5.63006
105
3601.43
11.26
5000
big-M5
5.63006
10
3600.31
3.51
5000
ours
5.63006
1
37.26
0.00
6000
big-M50
6.57798
10a
3600.53
9.98
6000
big-M20
6.57798
10
3600.62
9,36
6000
big-M5
6.57798
10
3600.05
2.82
6000
66.7798
105
20.64
0.00
7000
big-M50
-7.72266
10
3600.96
8.33
7000
big-M20
-7.72266
105
3600.57
7.80
7000
big-M5
-7.72266
10
3600.14
2.34
7000
ours
-7.72266
10a
20.91
0.00
Table 103: Comparison of time and optimality gap on the synthetic datasets (p 3000, k = 10,
p = 0.5, X2 0.1).
132

---

of samples
method
upper bound
time(s)
gap(
3000
big-M50
7.41186
10
3600.75
18.15
3000
big-M20
-..1186
10
3600.78
14.15
3000
big-M5
-7.41186
10
3600.34
5.92
3000
ours
-7.4118
10
3600.59
19.46
4000
big-M50
1.00287
10
3600.07
14.98
4000
big-M20
1.00287
10t
3600.06
12.06
4000
big-M5
- -1.0227-
10
3600.36
5,06
4000
ours
-1.00287
106
3600.08
1.88
5000
big-M50
- 1.24027
10
3600.07
11.88
5000
big-M20
-1.24027
106
3600.08
10.26
5000
big-M5
-1.24027
10t
3600.22
4.34
5000
ours
-1.24027
10t
469.43
0.00
6000
big-M50
1..4546
10
3601.17
9.78
6000
big-M20
- 11.4416
10
3600.60
9.11
6000
big-M5
-1.45416
100
3600.20
3.94
6000
-1.45416
106
59.16
0.00
7000
big-M50
-1.70483
10
3602.06
8.34
7000
big-M20
-1.70483
10
3600.61
7.77
7000
big-M5
-1.70483
10
3600.22
3.39
7000
ours
-1.70483
10
21.12
0.00
Table 104: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10,
p = 0.7, 22 0.1).
of samples
method
upper bound
time(s)
gap(%)
3000
big-M50
-2.78468
10
3600.31
15.93
3000
big-M20
-2.78521
10
3600.05
10.59
3000
big-M5
-2.78521
106
3600.09
3.69
3000
ours
-2.78521
106
3600.23
19.46
4000
big-M50
-3.75203
10
3600.08
13.35
4000
big-M20
-3.75203
10
3600.04
8.60
4000
big-M5
-3.75203
106
3600.14
3.16
4000
ours
-3.75203
10
3601.10
3.50
5000
big-M50
44.61877
106
3600.08
11.40
5000
big-M20
4.61877
10
3600.05
7.62
5000
big-M5
4.61877
106
3600.16
2.72
5000
ours
-..1177
106
3600.40
0.93
6000
big-M50
5.43826
10
3600.23
9.79
6000
big-M20
5.43826
10t
3600.11
6,92
6000
big-M5
5.43826
106
3600.17
2.50
6000
5..3826
106
3600.26
0.33
7000
big-M50
6.36767
106
3600.07
8.22
7000
big-M20
6.36767
106
3600.24
6.06
7000
big-M5
-6.66777
10t
3600.05
2.25
7000
ours
66.6667
1
3600.58
0.11
Table 105: Comparison of time and optimality gap on the synthetic datasets (p -3000, k = 10,
p = 0.9, X2 0.1).
133

---

G.3.10
Synthetic 2 Benchmark with 22 = 0.1 with Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50 warm start
66.3575 10
3600.12
19.14
3000
big-M20 warm start
-6.3457
104
3600.22
18.10
3000
big-M5 + warm start
6.34575
104
2424.12
0.00
3000
6.34575 104
3600.33
19.31
4000
big-M50 warm start
8.64284 10*
3600.05
15.33
4000
big-M20 warm start
8.64284
10
3600.18
14.04
4000
big-M5 + warm start
8.64284
10*
1702.95
0.00
4000
ours
8.64284 10*
20.81
0.00
5000
big-M50~ warm start
-1.09575 10
3601.88
11,60
5000
big-M20 warm start
-1.09575
10a
3600.31
9.94
5000
big-M5 warm start
-1.09575
10a
1284.19
0.00
5000
ours
-1.09575 10
20.65
0.00
6000
big-M50- warm start
1..27642 10
3600.62
9.93
6000
big-M20 warm start
-1.27642
10
3600.15
9.92
6000
big-M5 + warm start
-1.27642 - 105
1004.13
0.00
6000
ours
-1.27642
10
1.70
0.00
7000
big-M50 warm start
-1.50613 10
3600.07
8.32
7000
big-M20 warm start
-1.50613
10
3600.06
5.74
7000
big-M5 warm start
-1.50613
10
794.91
0.00
7000
ours
-1.50613 10
1.67
0.00
Table 106: Comparison of time and optimality gap the synthetic datasets (p 3000, k 10, = 0.1, 22 = 0.1). All baselines use our beam search solution as a warm start.
134

---

# of samples
method
upper bound
time(s)
gap
3000
big-M50 warm start
1.59508
10
3600.22
19.07
3000
big-M20 warm start
-1.59508
10
3600.22
17.30
3000
big-M5 warm start
-1.59508
10
3600.05
6.35
3000
ours
-
-1.59508
10
3600.27
19.40
4000
big-M50 warm start
-2.17573
10
3600.10
14.78
4000
big-M20 warm start -2.17573
10
3600.06
14.34
4000
big-M5 + warm start
-2.17573
10
3600.04
4.63
4000
ours
-2.17573
105
99.53
0.00
5000
big-M50 warm start
-2.71985
109
3600.07
11.22
5000
big-M20 warm start
-2.71985
10
3600.10
11.29
5000
big-M5 warm start
-2.71985
10a
2980.53
0.00
5000
ours
-2.71985
10s
21.01
0.00
6000
big- big-M50 warm start
-3.16760 10
3600.07
9.96
6000
big-M20 warm start
-3.16760
10
3600.74
9.95
6000
big-M5 warm start
-3.16760
10
2034.39
0.00
6000
-3.16760
105
20.91
0.00
7000
big-M50 warm start
-3.72609 10
3601.58
8.33
7000
big-M20 warm start
-3.72609.
10
3600.38
8.12
7000
big-M5 M5 + warm start
-3.72609
10
1513.64
0.00
7000
ours
-3.72609
10
1.73
0.00
Table 107: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10, p = 0.3, 22 0.1). All baselines use our beam search solution as warm start.
# of samples
method
upper bound
time(s)
gap(%) (%)
3000
big-M50 warm start
-3.33603 105
3601.53
18.78
3000
big-M20 warm start
-3.33603
10
3600.81
16.11
3000
big-M5 warm start
-3.33603
10
3600.22
7.55
3000
ours
-3.33603
10
3600.68
19.44
4000
big-M50 warm start
-.53220
10
3600.07
15.04
4000
big-M20 warm start
-4.53260
10
3600.26
13.47
4000
big-M5 + warm start
-4.53260
10
3600.21
6.23
4000
ours
-4.53260-
10
3600.41
0.62
5000
big-M50 warm start
5..33006
105
3601.70
11.69
5000
big-M20 warm start
5.63006
105
3600.25
11.26
5000
big-M5 warm start
5.63006
10a
3600.05
3.43
5000
ours
5.63006
10
37.26
0.00
6000
big-M50 warm start
6..7798
10
3600.18
9.98
6000
big-M20 warm start
66.57798
10
3600.55
9.36
6000
big-M5 warm start
6.57798
105
3600.04
2.89
6000
6657799
105
20.64
0.00
7000
big-M50 warm start
-7.72266
10
3600.63
8.33
7000
big-M20 warm start
-7.72266
105
3600.06
7.80
7000
big-M5 + warm start -7.72266
10
3600.15
2.46
7000
ours
-7.72266
105
20.91
0.00
Table 108: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10,
= 0.5, A 0.1). All baselines use beam search solution as warm start.
135

---

# of samples
method
upper bound
time(s)
gap %)
3000
big-M50 warm start
-7.41186
10
3601.56
18.15
3000
big-M20 warm start
-7.41186
10
3600.35
14.15
3000
big-M5 warm start
-7.41186
10
3600.27
5.92
3000
ours
-7.41186
10
3600.59
19.46
4000
big-M50 warm start
-1.00287 10
3600.09
14.98
4000
big-M20 warm start
-
1.00287
106
3600.12
12.06
4000
big-M5 + warm start
-1.00287
10
3600.07
5.06
4000
ours
-1.00287 10"
3600.08
1.88
5000
big-M50 warm start
11.24027 10
3600.40
11.88
5000
big-M20 warm start
11.24027
10%
3600.98
10.26
5000
big-M5 warm start
- 11.24027
106
3600.22
4.34
5000
ours
11.24027 106
469.43
0.00
6000
big- big-M50 warm start
- -1.45416 10
3600.05
9.78
6000
big-M20 warm start
- 1..4446 10
3600.06
9.06
6000
big-M5 warm start
-1.45416
-10%
3600.19
3.94
6000
-1.45416 106
59.16
0.00
7000
big-M50 warm start
-1.70483
106
3600.07
8.34
7000
big-M20 warm start
-1.70483
106
3600.28
7.77
7000
big-M5 M5 + warm start
-1.70483
10
3600.24
3.39
7000
ours
-1.70483
10
21.12
0.00
Table 109: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10, p = 0.7, A2 0.1). All baselines use our beam search solution as warm start.
# of samples
method
upper bound
time(s)
gap(%) (%)
3000
big-M50 warm start
-2.78521
10
3600.07
15.91
3000
big-M20 warm start
-2.78521
10
3600.71
10.37
3000
big-M5 warm start
-2.78521
106
3600.08
3.69
3000
ours
-2.78521
10
3600.23
19.46
4000
big-M50 warm start
-3.75203 10
3600.27
13.35
4000
big-M20 warm start
-3.75203
106
3600.05
8.60
4000
big-M5 + warm start
-3.75203
106
3600.09
3.16
4000
ours
-3.75203
10
3601.10
3.50
5000
big-M50 warm start
4.61877 106
3600.12
11.40
5000
big-M20 warm start
4.61877
10
3600.06
7.62
5000
big-M5 warm start
-4.61877
106
3600.09
2.72
5000
ours
-..1187 106
3600.40
0.93
6000
big-M50 warm start
5.43826
10"
3600.07
9.79
6000
big-M20 warm start
5.43826
106
3600.33
6.97
6000
big-M5 warm start
5.43826
10
3600.24
2.50
6000
5.43826
106
3600.26
0.33
7000
big-M50 warm start
6.36767 106
3600.07
8.22
7000
big-M20 warm start
6.36767
106
3600.67
6.06
7000
big-M5 + warm start
66.36767
106
3600.07
2.25
7000
ours
6.36767
106
3600.58
0.11
Table 100: Comparison of time and optimality gap on the synthetic datasets (p 3000, k = 10, = 0.9, X2 0.1). All baselines use beam search solution as warm start.
136

---

G.3.11 Synthetic 2 Benchmark with 22 = 10.0 with No Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50
6.33576 10
3600.31
18.47
3000
big-M20
6.33576 10*
3600.43
17.82
3000
big-M5
6.33576
10*
2349.33
0.00
3000
6.33576 104
3601.41
4.54
4000
big-M50
8.63287
10*
3600.85
15,21
4000
big-M20
8.63287
10*
3600.06
13.93
4000
big-M5
8.63287
10*
1577.17
0.00
4000
ours
8.63287 10*
20.33
0.00
5000
big-M50
-1.09474
10
3600.28
11.85
5000
big-M20
-1.09474
10a
3600.55
10.17
5000
big-M5
-1.09474
10a
1627.54
0.00
5000
ours
-1.09474
10
20.71
0.00
6000
big-M50
11.27542
10
3600.37
9.90
6000
big-M20
-1.27542
10
3600.09
9.81
6000
big-M5
-1.27542
10
1071.17
0.00
6000
ours
-1.27542
10
1.69
0.00
7000
big-M50
-1.50512
10a
3600.56
8.31
7000
big-M20
-1.50512
105
3600.07
8.30
7000
big-M5
-1.50512
10
856.04
0.00
7000
ours
-1.50512
105
1.70
0.00
Table 111: Comparison of time and optimality gap the synthetic datasets (p 3000, k 10.
= 0.1, A2 = 10.0).
137

---

of samples
method
upper bound
time(s)
gap (%)
3000
big-M50
-1.9409
10
3600.72
18.52
3000
big-M20
-1.59409
10
3600.07
17.23
3000
big-M5
-1.59409
10
3600.44
5.92
3000
ours
-1.59409
10
3600.62
4.68
4000
big-M50
-2.17472
10
3600.07
14.64
4000
big-M20
-2.17472
10
3600.82
14.15
4000
big-M5
-2.17472
10
3600.05
4.05
4000
ours
-2.17472
105
86.90
0.00
5000
big-M50
-2.71885
10
3603.74
11.16
5000
big-M20
-2.71885
10
3601.16
11.14
5000
big-M5
-2.71885
10
2522.90
0.00
5000
ours
-2.71885
10a
20.75
0.00
6000
big-M50
-3.16660
10
3600.07
9.93
6000
big-M20
-3.16660
10a
3600.26
9.92
6000
big-M5
-3.16660
10
1893.58
0.00
6000
-3.16660
105
21.05
0.00
7000
big-M50
-3.72509
10
3601.17
8.31
7000
big-M20
-3.72509-
10
3600.31
8.29
7000
big-M5
-3.72509
10
1472.84
0.00
7000
ours
-3.72509
10
1.71
0.00
Table 112: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10,
p = 0.3, 22 10.0).
of samples
method
upper bound
time(s)
gap (ap(%)
3000
big-M50
-3.33503 10
3600.07
18.43
3000
big-M20
-3.33503
10
3600.06
16.05
3000
big-M5
-3.33503
10
3600.22
7.54
3000
ours
-3.33503
105
3600.25
6.21
4000
big-M50
-.53119
10
3600.16
14.84
4000
big-M20
-.53159
10s
3600.90
13.44
4000
big-M5
-4.53159
10
3600.21
6.24
4000
ours
-4.53159
10
3600.95
0.35
5000
big-M50
5.62906
105
3600.16
11,59
5000
big-M20
5.62906
105
3600.07
11.13
5000
big-M5
5.62906
10
3600.36
3,65
5000
ours
5.62906
10
31.47
0.00
6000
big-M50
6.57698
1
3600.16
9.95
6000
big-M20
6.57698
10
3600.52
9,33
6000
big-M5
6.57698
105
3600.37
2.90
6000
66.76998
105
21.24
0.00
7000
big-M50
-7.72166
10
3600.12
8.31
7000
big-M20
-7.72166
105
3600.05
7.81
7000
big-M5
-7.72166
10
3600.05
2.33
7000
ours
-7.72166
10a
20.37
0.00
Table 113: Comparison of time and optimality gap on the synthetic datasets (p 3000, k = 10,
p = 0.5, 22 10.0).
138

---

of samples
method
upper bound
time(s)
gap (6)
3000
big-M50
7.41086
10
3600.65
17.88
3000
big-M20
-7.41086
10
3600.14 14
14.12
3000
big-M5
-7.41086
10
3600.35
5.94
3000
ours
-7.41086
10
3600.48
8.32
4000
big-M50
1.00277
10
3600.09
14.85
4000
big-M20
1.00277
-008
3600.47
11.77
4000
big-M5
-1.00277
10
3600.05
5,04
4000
ours
-1.00277
10
3600.62
1.66
5000
big-M50
- 11.24017
10
3600.25
11.82
5000
big-M20
-1.24017
106
3600.06
10.25
5000
big-M5
-1.24017
10
3600.22
4.34
5000
ours
-1.24017
10t
386.72
0.00
6000
big-M50
-1.45406
10
3600.07
9.75
6000
big-M20
- -1.49199
10
3600.53
9.46
6000
big-M5
-1.45406
106
3600.20
3.95
6000
-1.45406
106
54.13
0.00
7000
big-M50
-1.70473
10
3600.81
8.32
7000
big-M20
-1.70473
10
3600.06
7.68
7000
big-M5
-1.70473
10
3600.20
3.41
7000
ours
-1.70473
10
21.30
0.00
Table 114: Comparison of time and optimality gap on the synthetic datasets (p 3000, k = 10,
p = 0.7, A2 = 10.0).
of samples
method
upper bound
time(s)
gap(%)
3000
big-M50
-2.78507 10
3600.17 17
15.95
3000
big-M20
-2.78507
10
3600.56
10.36
3000
big-M5
-2.78507
106
3600.09
3.69
3000
ours
-2.78507
106
3600.78
9.79
4000
big-M50
-3.75192
10
3600.06
13.19
4000
big-M20
-3.75192
10
3600.05
8.60
4000
big-M5
-3.75192
106
3600.41
3.16
4000
ours
-3.75192
10
3600.32
3.23
5000
big-M50
4.61867
106
3600.17
11.32
5000
big-M20
4.61867
10
3600.56
7.66
5000
big-M5
4.61867
106
3600.22
2.72
5000
ours
-4.11867
106
3600.30
0.89
6000
big-M50
5.43816
10
3600.12
9.75
6000
big-M20
5.43816
10t
3600.33
6.91
6000
big-M5
5.43816
106
3600.20
2.50
6000
55.33816
106
3600.22
0.32
7000
big-M50
6.36757
106
3600.32
8.20
7000
big-M20
66.3677
106
3600.05
6.05
7000
big-M5
-6.36757
10t
3600.37
2.25
7000
ours
-6.36757
1
3600.06
0.10
Table 115: Comparison of time and optimality gap on the synthetic datasets (p 3000, k = 10,
p = 0.9, X2 10.0).
139

---

G.3.12
Synthetic 2 Benchmark with 22 = 10.0 with Warmstart
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50 warm start
6.33576 10
3600.12
18.47
3000
big-M20 warm start
-6.33576
104
3600.44
17.82
3000
big-M5 + warm start
6.33576
104
2370.86
0.00
3000
6.33576 104
3601.41
4.54
4000
big-M50 warm start
8.63287 10*
3600.29
15.21
4000
big-M20 warm start
8.63287 10
3600.09
13.93
4000
big-M5 + warm start
8.63287 10*
1870.35
0.00
4000
ours
8.63287 10*
20.33
0.00
5000
big-M50~ warm start
-1.09474 10
3600.15
11.85
5000
big-M20 warm start
-1.09474
10a
3600.43
10.17
5000
big-M5 warm start
-1.09474
10
1146.28
0.00
5000
ours
-1.09474 10
20.71
0.00
6000
big-M50- warm start
1..27542 10
3601.39
9.89
6000
big-M20 warm start
-1.27542
10
3600.08
9.81
6000
big-M5 + warm start
-1.27542 - 105
1038.32
0.00
6000
ours
-1.27542
10
1.69
0.00
7000
big-M50 warm start
-1.50512 10
3601.02
8.31
7000
big-M20 warm start
-1.50512
10
3600.99
7.98
7000
big-M5 warm start
-1.50512
10
853.59
0.00
ours Table 116: Comparison of time and optimality gap the synthetic datasets (p 3000, k = 10,
7000
-1.50512 105
1.70
0.00
= 0.1, 22 = 10.0). All baselines use our beam search solution as a warm start.
140

---

# of samples
method
upper bound
time(s)
gap(%) (%)
3000
big-M50 warm start -
-1.59409
10
3600.67
18.52
3000
big-M20 warm start
-1.59409
10
3601.34
17.23
3000
big-M5 warm start
-1.59409-
10
3600.06
5.92
3000
ours
-1.59409
10
3600.62
4.68
4000
big-M50 warm start
-2.17472
10
3600.08
14.64
4000
big-M20 warm start
-2.17472
10
3600.35
14.15
4000
big-M5 + warm start
-2.17472
106
3600.06
3.77
4000
ours
-2.17472
105
86.90
0.00
5000
big-M50 warm start
-2.71885
109
3600.07
11.16
5000
big-M20 warm start
-2.71885
10
3600.19
11.09
5000
big-M5 warm start
-2.71885
10
2700.47
0.00
5000
ours
-2.71885
10s
20.75
0.00
6000
big- big-M50 warm start
-3.16660 10
3600.06
9.93
6000
big-M20 warm start
-3.16660
10
3600.74
9.92
6000
big-M5 warm start
-3.16660
10
1742.17
0.00
6000
-3.16660
10
21.05
0.00
7000
big-M50 warm start
-3.72509 10
3601.61
8.31
7000
big-M20 warm start
-3.72509-
10
3600.25
8.11
7000
big-M5 15 + warm start
-3.72509
10
1466.07
0.00
7000
ours
-3.72509
10
1.71
0.00
Table 117: Comparison of time and optimality gap on the synthetic datasets (p (p= 3000, k 10, p = 0.3, A2 10.0). All baselines use our beam search solution as a warm start.
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50 warm start
-3.33503
10
3600.08
18.43
3000
big-M20 warm start
-3.33503-
10
3600.16
16.05
3000
big-M5 warm start
-3.33503
10
3600.22
7.54
3000
ours
33.3550
10
3600.25
6.21
4000
big-M50 warm start
-4.53199
10
3600.06
14.84
4000
big-M20 warm start
-4.33159
10
3600.21
13.44
4000
big-M5 + warm start
-4.53159
10
3600.22
6.24
4000
ours
-4.53159-
10
3600.95
0.35
5000
big-M50 warm start
5.629906
105
3600.08
11.59
5000
big-M20 warm start
5.62906
105
3601.71
11.22
5000
big-M5 warm start
5.62906
10
3600.13
3.66
5000
ours
5.62906
10
31.47
0.00
6000
big-M50 warm start
6..7698
1
3600.14
9.95
6000
big-M20 warm start
6..7698
10a
3600.97
9.33
6000
big-M5 warm start
6.57698
105
3600.14
2.91
6000
66.77698
105
21.24
0.00
7000
big-M50 warm start
-7.72166
10
3601.52
8.31
7000
big-M20 warm start
-7.72166
105
3600.30
7.81
7000
big-M5 + warm start -7.72166 10
3600.46
2.33
7000
ours
-7.72166
105
20.37
0.00
Table 118: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10,
= 0.5, A2 10.0). All baselines use our beam search solution as warm start.
141

---

# of samples
method
upper bound
time(s)
gap( %))
3000
big-M50 warm start
-7.41086
10
3601.32
17.88
3000
big-M20 warm start
-7.41086
10
3600.23
14.06
3000
big-M5 warm start
-7.41086
10
3600.11
5.94
3000
ours
-7.41086
10
3600.48
8.32
4000
big-M50 warm start
-1.00277 10
3600.44
14.85
4000
big-M20 warm start
- 1.00277
10
3600.09
11.77
4000
big-M5 + warm start
-1.00277
10
3600.26
5.04
4000
ours
-1.00277 10"
3600.62
1.66
5000
big-M50 warm start
- 11.24017 10
3600.27
11.82
5000
big-M20 warm start
-1.24017
10%
3600.69
10.25
5000
big-M5 warm start
- 11.24017
106
3600.22
4.34
5000
ours
- 11.24017 106
386.72
0.00
6000
big- big-M50 warm start
- -1.45406 10
3600.26
9.75
6000
big-M20 warm start
- -1.45406 10"
3600.07
9.09
6000
big-M5 warm start
-1.45406
10
3600.18
3.95
6000
- -1.45406 106
54.13
0.00
7000
big-M50 warm start
-1.70473 106
3600.20
8.32
7000
big-M20 warm start
-1.70473
106
3600.12
7.68
7000
big-M5 M5 + warm start
-1.70473
10
3600.04
3.41
7000
ours
-1.70473
106
21.30
0.00
Table 119: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k 10,
p = 0.7, A2 10.0). All baselines use our beam search solution as a warm start.
# of samples
method
upper bound
time(s)
gap(%)
3000
big-M50 warm start
-2.78507
10
3600.07
15.95
3000
big-M20 warm start
-2.78507
106
3600.04
10.36
3000
big-M5 warm start
-2.78507
108
3600.09
3.69
3000
ours
-2.78507 10
3600.78
9.79
4000
big-M50 warm start
-3.75192 10
3600.06
13.19
4000
big-M20 warm start
-3.75192
106
3600.11
8.61
4000
big-M5 + warm start
-3.75192
106
3600.15
3.16
4000
ours
-3.75192 10
3600.32
3.23
5000
big-M50 warm start
4.61867 106
3600.06
11.32
5000
big-M20 warm start
4.61167 10
3600.79
7.64
5000
big-M5 warm start
-4.11867
106
3600.30
2.72
5000
ours
4..1186 106
3600.30
0.89
6000
big-M50 warm start
5.43816
108
3600.39
9.75
6000
big-M20 warm start
5.43816
106
3600.05
6.91
6000
big-M5 warm start
5.43816
10
3600.05
2.50
6000
5.43816
106
3600.22
0.32
7000
big-M50 warm start
6.36757
106
3600.10
8.20
7000
big-M20 warm start
6.36757
106
3600.28
6.05
7000
big-M5 + warm start
66.36757
1
3600.37
2.25
7000
ours
6.36757
106
3600.06
0.10
Table 120: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10,
= 0.9, X 10.0). All baselines use our beam search solution warm start.
142

---

G.4
Comparison with MOSEK Solver, SubsetSSlectionCI1,, and LOBNB
G.4.1
Synthetic Benchmark with X2 0.001
# of features
method
upper bound
support size
time(s)
gap(%)
100
SubsetSelectCIO
-4.79406 10a
2
2.46
0.00
100
SubsetSelectCIO + warm start
-4.79406- 10t
2
2.72
0.00
100
LOBnB
-2.10878- 106
10
1420.85
0.00
100
MSK persp.
-2.10878- 106
10
932.06
0.00
100
MSK persp. + warm start
-2.10878 10
10
898.00
0.00
100
Convex MSK opt. persp.
-2.10878 106
10
0.10
0.00
100
ours
-2.10878 10c
10
0.29
0.00
500
SubsetSelectCIO
-4.79201 10
2
3,91
0.00
500
SubsenSelectCIO + warm start
-4.79201 10
2
3.59
0.00
500
L0BnB
-2.12833 10
10
1754.22
0.00
500
MSK persp.
-2.12833 106
10
3606.30
0.09
500
MSK persp. + warm start
-2.12833 106
10
3600.23
0.09
500
Convex MSK opt. persp.
-2.12833 106
10
6.79
0.01
500
ours
-2.12833 106
10
0.37
0.00
1000
SubsetSelectC1O
-4.71433 105
2
4.36
0.00
1000
SubsetSelectCIO + warm start
-4.71433
10S
2
4.39
0.00
1000
LÖBnB
-2.11005 106
10
2114.74
0.00
1000
MSK persp.
3673.42
1000
MSK persp. + warm start
-2.11005 106
10
3608.90
0.00
1000
Convex MSK opt. persp.
-2.11005 106
10
54.02
0.01
1000
ours
-2.11005 10
10
0.58
0.00
3000
SubsetSelectC1O
-4.72136 105
2
5.51
0.00
3000
SubsetSelectCIO + warm start
-4.72136
105
2
9.19
0.00
3000
LOBnB
-2.10230 106
10
2302.34
1.20
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
3600.00
3000
Convex MSK opt. persp.
-2.10230 106
10
1759.45
0.02
3000
ours
-2.10230 106
10
1.73
0.00
5000
SubsetSelectCIO
4.62647 10
2
12.04
0.00
5000
SubsetSelectIIO + warm start
4.62647 105
2
12.66
0.00
5000
L.0nBB
2.09635 106
10
2643.23
5.18
5000
MSK persp.
3600.00
5000
MSK persp. warm start
3600.00
5000
Convex MSK opt. persp.
2. 2.06335 106
10
3760.81
0.05
5000
ours
2.09635 10
10
4.22
0.00
Table 121: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
p 0.1, 22 0.001).
143

---

of features
method
upper bound
support size
time(s)
gap(%) %
100
SubsetSelectCIO
-2.22792 10t
2
2.52
0.00
100
SubsetSelectC10 + warm start
-2.52792 106
2
2.71
0.00
100
LOBnB
-5.28726- 106
10
3182.66
0.00
100
MSK persp.
-5.28726 10t
10
3600.08
0.01
100
MSK persp. + warm start
-5.28726 10
10
3600.67
0.01
100
Convex MSK opt. persp.
-5.28726 106
10
0.11
0.00
100
ours
-5.28726 10"
10
0.29
0.00
500
SubsetSelectCIO
-2.52985 106
2
3.43
0.00
500
SubsetSelectCIO + warm start
-2.52985 10t
2
3.26
0.00
500
L0BnB
-5.31852- 106
10
3977.43
1.65
500
MSK persp.
-5.31852- 106
10
3631.98
0.10
500
MSK persp. + warm start
-5.31852 10t
10
3607. 49
0.10
500
Convex MSK opt. persp.
-5.31852 106
10
7.77
0.01
500
ours
-5.31852- 106
10
0.41
0.00
1000
SubsetSelectCIO
-2.50201 106
2
4.67
0.00
1000
SubsetSelectC1O + warm -2.50201 106
2
4.99
0.00
1000
L0BnB
-5.26974 106
10
4224.81
2.82
1000
MSK persp.
3607 .94
1000
MSK persp. warm start
5.26974 106
10
3601.79
0.20
1000
Convex MSK opt. persp.
-5.26974 106
10
54.48
0.01
1000
ours
-5.26974 106
10
0.52
0.00
3000
SubsetSelectCIO
-2.51861 106
2
8.73
0.00
3000
SubsetSelectC10 + warm start
-2.51861
106
2
8.50
0.00
3000
L0BnB
-5.27668 106
10
4945.04
0.35
3000
MSK persp.
3600.00
3000
MSK persp. + warm
3600.00
3000
Convex MSK opt. persp.
5.277668 10
10
1724.71
0.02
3000
ours
55.77668 106
10
1.75
0.00
5000
SubsetSelectC1O
-2.49167 106
2
7.60
0.00
5000
SubsetSelectC10 + warm start
-2.49167
106
2
13.32
0.00
5000
LOBnB
55.27126 106
10
5589.24
0.49
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
5.27126 106
10
3660.81
0.05
5000
ours
5.27126 106
10
4.32
0.00
Table 122: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
0.3, A2 = 0.001).
144

---

# of features
method
upper bound
support size
time(s)
gap(%) (%)
100
SubsetSelectCIO
-7.42601 10
2
2.18
0.00
100
SubsetSelectC1O + start
-7.42601 10t
2
2.61
0.00
100
L.0nBB
-1.10084 107
10
3464.13
0.00
100
MSK persp.
-1.10084 107
10
3605.37
0.01
100
MSK persp. + warm
-1.10084- 107
10
3602.75
0.01
100
Convex MSK opt. persp.
-1.10084 10"
10
0.14
0.00
100
ours
-1.10084 107
10
0.29
0.00
500
SubsetSelectCIO
-7.42485- 10t
2
3.33
0.00
500
SubsenSelectC10 + warm start
-7.42485 10t
2
3.59
0.00
500
L0BnB
-1.10518 107
10
4143.57
0.77
500
MSK persp.
-1.10518- 107
10
3606.04
0.10
500
MSK persp. + warm start
-1.10518 107
10
3600.96
0.10
500
Convex MSK opt. persp.
-1.10518 107
10
7.89
0.01
500
ours
-1.10518 107
10
0.40
0.00
1000
SubsetSelectC1O
-7.35715 106
2
3.71
0.00
1000
SubsetSelectC10 + warm start
-7.35715 106
2
4.39
0.00
1000
L0BnB
-1.09508- 107
10
1931.45
0.65
1000
MSK persp.
3708.29
1000
MSK persp. + warm start
-1 -1.09508 107
10
3692.66
0.00
1000
Convex MSK opt. persp.
-1.09508 107
10
56.04
0.01
1000
ours
11.9500 107
10
0.59
0.00
3000
SubsetSelectC1O
-7.42176 106
2
8.32
0.00
3000
SubsetSelectCIO + warm start
-7.42176 10
2
8.23
0.00
3000
LOBnB
-1.10012 107
10
5903.24
1.27
3000
MSK persp.
3600.00
3000
MSK persp. + warm
-1.10012 107
10
4174.40
0.00
3000
Convex MSK opt. persp.
-1.10012 10?
10
1848.58
0.02
3000
ours
-1.00112 107
10
1.74
0.00
5000
SubsetSelectC1O
-7.37395 106
2
8.42
0.00
5000
SubsetSelectC10 + warm start
-7.37395 106
2
11.87
0.00
5000
LOBnB
-1.9996 107
10
6029.87
1.11
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
-1.09966 107
10
6700.63
0.00
5000
Convex MSK opt. persp.
-1.09966 107
10
3657.70
0.06
5000
ours
-1
1109966 107
10
4.00
0.00
Table 123: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0.5, Xg = 0.001).
145

---

# of features
method
upper bound
support size
time(s)
gap(%) (%)
100
SubsetSelectCIO
-2.01567 10"
2
2.50
0.00
100
SubsetSelectCIO + warm start
-2.01567 107
2
2.64
0.00
100
LOBnB
-2.43575 10?
10
2716.56
0.00
100
MSK persp.
-2.43575 107
10
3602.86
0.01
100
MSK persp. + warm start
-2.43555 107
10
3600.41
0.01
100
Convex MSK opt. persp.
-2.43575- 107
10
0.12
0.00
100
ours
-2.43575 107
10
0.32
0.00
500
SubsetSelectCIO
-2.01420 107
2
3.26
0.00
500
SubsetSelectCIO- + warm start
-2.01420- 107
2
3.40
0.00
500
L0BnB
-2.44185 107
10
3747.19
0.00
500
MSK persp.
-2.24367 107
10
4608.22
8.94
500
MSK persp. + warm start
-2.44185 107
10
3623.31
0.10
500
Convex MSK opt. persp.
-2.44185 107
10
8.07
0.01
500
-2.44185 107
10
0.46
0.00
1000
SubsetSelectCIO
-1.99762 107
2
3.67
0.00
1000
SubsetSelectC1O + warm -1.99762 107
2
4.71
0.00
1000
L0BnB
-2.11994 107
11
4179.51
0.01
1000
MSK persp.
3620.44
1000
MSK persp. + warm start
-2.11989 107
10
3686.70
0.20
1000
Convex MSK opt. persp.
-2.41989 107
10
57.29
0.01
1000
ours
-2.11989 107
10
0.52
0.00
3000
SubsetSelectC1O
-2.01741 107
2
9.04
0.00
3000
SubsetSelectC10 + warm start
-2.01741
107
2
9.24
0.00
3000
L0BnB
-2.22599 107
9
7445.35
0.00
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
-2.43713 107
10
3888.61
0.00
3000
Convex MSK opt. persp.
-2.43713 -07
10
1929.94
0.02
3000
ours
2.43713 107
10
1.77
0.00
5000
SubsetSclectCIO
2.00985 107
2
13.61
0.00
5000
SubsetSelectC1O + warm start
-2.00985 107
2
12.21
0.00
5000
L0BnB
-2.43694 107
10
4485.99
23.68
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
2.43694 107
10
3748.60
0.06
5000
ours
-2.36994 107
10
4.15
0.00
Table 124: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0.7, A2 = 0.001).
146

---

of features
method
upper bound
support size
time(s)
gap(%) (%)
100
SubsetSelectCIO
-8.64519- 107
2
2.27
0.00
100
SubsetSelectC10 + warm start
-8.64519 107
2
2.79
0.00
100
LOBnB
-9.11185 10?
93
6954.00
0.04
100
MSK persp.
-8.96852 107
10
3601.43
1.59
100
MSK persp. + warm start
-9.11026 107
10
3601.28
0.01
100
Convex MSK opt. persp.
99.1102 10?
10
0.13
0.00
100
ours
99.11026 107
10
0.30
0.00
500
SubsetSelectCIO
8.63791 107
2
2.89
0.00
500
SubsetSelectCIO + warm start
8.63791 107
2
3.47
0.00
500
L0BnB
9.10885 107
9
7562.05
0.14
500
MSK persp.
3605.36
500
MSK persp. + warm start
9.12047 107
10
3630.75
0.10
500
Convex MSK opt. persp.
9..2207 107
10
7.26
0.01
500
ours
9.12047 107
10
0.38
0.00
1000
SubsetSelectCIO
8.56943 107
2
4.58
0.00
1000
SubsetSelectC10 + warm start
8.56943 107
2
4.73
0.00
1000
L0BnB
8.23906 107
1
7300.29
9.17
1000
MSK persp.
3646.29
1000
MSK persp. + warm start
9.04075 107
10
3613.37
0.00
1000
Convex MSK opt. persp.
9..4075 107
10
67.12
0.01
1000
ours
9.04075 107
10
0.52
0.00
3000
SubsetSelectC1O
8.66243 107
2
8.50
0.00
3000
SubsetSelectCIO + warm start
8.66243 107
2
8.12
0.00
3000
LOBnB
9.12781 107
12
1706.18
7.80
3000
MSK persp.
3600.00
3000
MSK persp. + warm
9.12761 107
10
3998.60
0.00
3000
Convex MSK opt. persp.
9.12761 107
10
2038.29
0.02
3000
ours
9.12761 107
10
20.74
0.00
5000
SubsetSelectC1O
8.64926 107
2
12.63
0.00
5000
SubsetSelectC10 + warm start
8.64926 107
2
11.94
0.00
5000
LOBnB
8..2072 107
1
7472.42
9.32
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
9.12916 107
10
3722.78
3.21
5000
ours
9.12916 107
10
58.29
0.00
Table 125: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
0.9, A2 = 0.001).
147

---

G.4.2
Synthetic Benchmark with X2 0.1
# of features
method
upper bound
support size
time(s)
gap(% %6
100
SubsetSclectCIO
3600.00
100
SubsetSelectI10 + warm start
3600.00
100
L0BnB
-2.10878 106
10
1415.23
0.00
100
MSK persp.
-2.10878 10t
10
928.86
0.00
100
MSK persp. + warm start
-2.10878 106
10
912.78
0.00
100
Convex MSK opt. persp.
-2.10878 106
10
0.10
0.00
100
ours
-2.10878 106
10
0.36
0.00
500
SubsetSelectCIO
-..19239 108
3.17
0.00
500
SubsetSelectCIO + warm start
-1.19239 10
1
3.39
0.00
500
L0BnB
-2.12833 106
10
1973.90
0.00
500
MSK persp.
-2.12830 106
10
3606.82
0.10
500
MSK persp. + warm start
-2.12833 106
10
3604.01
0.10
500
Convex MSK opt. persp.
-2.12833 106
10
6.49
0.01
500
ours
-2.12833 10
10
0.37
0.00
1000
SubsetSelectC10
-4.71433 105
2
4,47
0.00
1000
SubsetSelectCIO+ + warm start
-4.71433
105
2
4.68
0.00
1000
LOBnB
-2.11005
106
10
2101.63
0.00
1000
MSK persp.
3602.15
1000
MSK persp. + warm start
-2.11005 106
10
3600.98
0.20
1000
Convex MSK opt. persp.
-2.11005 10
10
58.36
0.01
1000
ours
-2.11005 106
10
0.57
0.00
3000
SubsetSelectCIO
44.72135 105
2
8.81
0.00
3000
SubsetSelectC10 + warm start
-4.72135 105
2
8.98
0.00
3000
L0BnB
-2.10230 106
10
2515.21
0.00
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
-2.10330 106
10
3660.82
0.00
3000
Convex MSK opt. persp.
-2.10230 106
10
1765.70
0.02
3000
ours
2.10030 106
10
1.79
0.00
5000
SubsetSelectCIO
4.62647 105
2
12.26
0.00
5000
SubsetSelectCIO + warm start
4.62647
105
2
12.06
0.00
5000
LOBnB
-2.09635 106
10
2666.21
3.42
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
2.09635- 106
10
3685.08
0.06
5000
ours
2.09635
106
10
4.20
0.00
Table 126: Comparison of time and optimality gap on the synthetic datasets (n 100000,
10.
0.1, A2 = 0.1).
148

---

# of features
method
upper bound
support size
time(s)
gap(%) (%)
100
SubsetSelectCIO
-..22792 10
2
2.32
0.00
100
SubsetSelectCIO + warm start
-2.52792 10
2
2.46
0.00
100
LOBnB
-5.28726 106
10
3182.16
0.00
100
MSK persp.
-5.28724 106
10
3601.53
0.01
100
MSK persp. + warm start
-5.28726 106
10
3600.02
0.01
100
Convex MSK opt. persp.
-5.28726 106
10
0.10
0.00
100
ours
-5.28726 106
10
0.27
0.00
500
SubsetSelectCIO
-2.52985 106
2
3.54
0.00
500
SubsetSelectCIO- + warm start
-2.52985 10t
2
3.26
0.00
500
L0BnB
-5.31852 106
10
3918.77
1.65
500
MSK persp.
-5.31852- 106
10
3601.88
0.10
500
MSK persp. + warm start
-5.31852 10"
10
3602.35
0.10
500
Convex MSK opt. persp.
-5.31852 106
10
8.26
0.01
500
-5.31852- 106
10
0.41
0.00
1000
SubsetSelectCIO
-2.50201 106
2
4.49
0.00
1000
SubsetSelectC1O + warm -2.50201 106
2
4.12
0.00
1000
L0BnB
-5.26974 106
10
4052.64
1.67
1000
MSK persp.
3600.90
1000
MSK persp. + warm start
-5.26974- 106
10
3600.10
0.20
1000
Convex MSK opt. persp.
-5.26974- 106
10
55.50
0.01
1000
ours
-5.26974 10
10
0.52
0.00
3000
SubsetSelectC1O
-2.51861 106
2
5.94
0.00
3000
SubsetSelectC10 + warm start
-2.51861
106
2
8.61
0.00
3000
L0BnB
-5.27668 106
10
4807.82
2.18
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
-5.27668 106
10
3744.12
0.00
3000
Convex MSK opt. persp.
-5.27668 106
10
1795.54
0.02
3000
ours
-5.27668 106
10
1.84
0.00
5000
SubsetSclectCIO
-2.49167 106
2
13.36
0.00
5000
SubsetSelectC1O + warm start
-2.49167
106
2
12.61
0.00
5000
L0BnB
5..77126 10
10
5667.77
0.00
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
5.27126 106
10
3606.96
0.05
5000
ours
55.27126 106
10
4.42
0.00
Table 127: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0.3, A2 = 0.1).
149

---

# of features
method
upper bound
support size
time(s)
gap(%) %6)
100
SubsetSelectCIO
-7.42601 10
2
2.39
0.00
100
SubsetSelectCIO + warm start
-7.42601 10
2
2.80
0.00
100
LOBnB
-1.10084-10 10?
10
3398.37
0.00
100
MSK persp.
-1.09018- 107
10
3599.98
0.99
100
MSK persp. + warm start
-1.10084- 107
10
3603.80
0.01
100
Convex MSK opt. persp.
-1.10084 107
10
0.11
0.00
100
ours
-1.10084 107
10
0.32
0.00
500
SubsetSelectCIO
-7.42485- 106
2
2.97
0.00
500
SubsetSelectCIO- + warm start
-7.42485 106
2
3.22
0.00
500
L0BnB
-1.10518 107
10
4364.09
0.77
500
MSK persp.
3605.36
500
MSK persp. + warm start
-1.10518 107
10
3610.84
0.00
500
Convex MSK opt. persp.
-1.10518 107
10
7.25
0.01
500
ours
-1.15518 107
10
0.40
0.00
1000
SubsetSelectCIO
-7.35115 106
2
4.35
0.00
1000
SubsetSelectCIO + warm start
-7.35715 106
2
4.97
0.00
1000
L0BnB
-1.09508. 107
10
1882.56
0.65
1000
MSK persp.
3609.88
1000
MSK persp. + warm start
-1 -1.09508 107
10
3616.06
0.00
1000
Convex MSK opt. persp.
-1.0950 107
10
63.89
0.01
1000
ours
-1.99508 107
10
0.53
0.00
3000
SubsetSelectCIO
-7.42176 106
2
5.59
0.00
3000
SubsetSelectCIO + warm start
-7.42176 10
2
8.42
0.00
3000
L0BnB
-1.10012- 107
10
5588.51
1.24
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
-1.10012 107
10
4173.41
0.00
3000
Convex MSK opt. persp.
-1.10012 107
10
1779.87
0.02
3000
ours
-1.10012 107
10
1.67
0.00
5000
SubsetSelectC1O
-7.37395 106
2
8.72
0.00
5000
SubsetSelectC10 + warm start
-7.37395 106
2
12.24
0.00
5000
LOBnB
-1.9996 107
10
6428.17
1.09
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
-1.09966 107
10
6106.85
0.00
5000
Convex MSK opt. persp.
- -1.09966 107
10
3665.75
0.06
5000
ours
- -1.09966 107
10
4.13
0.00
Table 128: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0.5, X2 = 0.1).
150

---

# of features
method
upper bound
support size
time(s)
gap(%) (%)
100
SubsetSelectCIO
-2.01567 10"
2
2.23
0.00
100
SubsetSelectCIO + warm start
-2.01567 107
2
3.02
0.00
100
LOBnB
-2.43575- 10?
10
2673.71
0.00
100
MSK persp.
-2.41233 107
10
3601.54
0.98
100
MSK persp. + warm start
-2.43575- 107
10
3600.03
0.01
100
Convex MSK opt. persp.
-2.43575- 107
10
0.14
0.00
100
ours
-2.43575 107
10
0.34
0.00
500
SubsetSelectCIO
-2.01420 107
2
3.58
0.00
500
SubsetSelectCIO- + warm start
-2.01420- 107
2
3.34
0.00
500
L0BnB
-2.4418- 107
10
3602.76
0.00
500
MSK persp.
-2.44185- 107
10
3603.68
0.10
500
MSK persp. + warm start
-2.44185 107
10
3604.92
0.10
500
Convex MSK opt. persp.
-2.44185 107
10
7.37
0.01
500
-2.44185 107
10
0.44
0.00
1000
SubsetSelectCIO
-1.99762 107
2
4.47
0.00
1000
SubsetSelectC1O + warm -1.99762 107
2
4.56
0.00
1000
L0BnB
-2.11994 107
11
4204.09
0.01
1000
MSK persp.
3610.81
1000
MSK persp. + warm start
-2.11989 107
10
3609.15
0.00
1000
Convex MSK opt. persp.
-2.41989 107
10
56.82
0.01
1000
ours
-2.11989 107
10
0.53
0.00
3000
SubsetSelectC1O
-2.01741 107
2
5,42
0.00
3000
SubsetSelectC10 + warm start
-2.01741
107
2
8.40
0.00
3000
L0BnB
-2.22599 107
9
7207.55
0.00
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
-2.43713 107
10
3980.12
0.00
3000
Convex MSK opt. persp.
-2.43713 107
10
1799.08 .0
0.02
3000
ours
2.43713 107
10
1.75
0.00
5000
SubsetSclectCIO
2.00985 107
2
12.64
0.00
5000
SubsetSelectC1O + warm start
-2.00985 107
2
13.96
0.00
5000
L0BnB
-2.43694 107
10
4979.94 .4
21.63
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
2.43694 107
10
3606.45
0.06
5000
ours
-2.36994 107
10
4.22
0.00
Table 129: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0.7, A2 = 0.1).
151

---

# of features
method
upper bound
support size
time(s)
gap(%) (%)
100
SubsetSelectCIO
88.1116 10"
1
2.42
0.00
100
SubsetSelectCIO + warm start
88.1116 107
1
2.56
0.00
100
LOBnB
-9.11185 10"
93
6919.82
0.04
100
MSK persp.
-8.94411 107
10
3600.44
1.89
100
MSK persp. + warm start
99.11026 107
10
3600.46
0.01
100
Convex MSK opt. persp.
99.1026 10"
10
0.18
0.00
100
ours
99.1026 107
10
0.31
0.00
500
SubsetSelectCIO
8.63791 107
2
3.47
0.00
500
SubsetSelectCIO- + warm start
8.63791 107
2
3.56
0.00
500
L0BnB
9.10885 107
9
7359.92
0.14
500
MSK persp.
3601.63
500
MSK persp. + warm start
9.12047 107
10
3602.72
0.00
500
Convex MSK opt. persp.
9.12047 107
10
8.27
0.01
500
ours
9.12047 107
10
0.38
0.00
1000
SubsetSelectCIO
8.56943 107
2
4.83
0.00
1000
SubsetSelectCIO + warm start
8.56943 107
2
4.74
0.00
1000
LOBnB
8.23906 107
1
7326.90
9.06
1000
MSK persp.
3641.35
1000
MSK persp. + warm start
9.04075 107
10
3611.36
0.00
1000
Convex MSK opt. persp.
-9.04075 107
10
61.25
0.01
1000
ours
-9.04075 107
10
0.52
0.00
3000
SubsetSelectCIO
8.66243 107
2
8.37
0.00
3000
SubsetSelectCIO + warm start
8.66243 107
2
9.68
0.00
3000
L0BnB
9.12271 107
12
2205.35
5.43
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
9.12761 107
10
3661.26
0.00
3000
Convex MSK opt. persp.
99.1266 107
10
2087.06 .66
0.02
3000
ours
9.12761 107
10
20.88
0.00
5000
SubsetSelectC1O
8.64926 107
2
12.48
0.00
5000
SubsetSelectC10 + warm start
8.64926 107
2
12.73
0.00
5000
LOBnB
-..2072 107
1
7201.18
9.32
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
9.12916 107
10
8314.78
0.00
5000
Convex MSK opt. persp.
9.12916 107
10
3691.61
3.40
5000
ours
9.12916 107
10
58.07
0.00
Table 130: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0.9, X2 = 0.1).
152

---

G.4.3 Synthetic Benchmark with X2 10.0
# of features
method
upper bound
support size
time(s)
gap(
100
SubsetSelectCIO
-1.62628 106
10
3693.06
55.44
100
SubsetSelectCIO + warm start
-1.62806 10t
10
3672.85
55.27
L0BnB
-2.10868 10c
10
1432.58
0.00
100
MSK persp.
-2.10868 106
10
926.33
0.00
100
MSK persp. + warm start
-2.10868 10th
10
919.61
0,00
100
Convex MSK opt. persp.
-2.10868 106
10
0.10
0.00
100
ours
-2.10868 106
10
0.30
0.00
500
SubsetSelectCIO
-1.33331 10
10
3776.44
91.62
500
SubsetSelectC10 warm start
-4.87330 105
7
1654.44
0.00
500
L0BnB
-2.12823 106
10
1929.01
0.00
500
MSK persp.
-2.12822 106
10
3601.80
0.09
500
MSK persp. + warm
-2.12823 106
10
3609.69
0.09
500
Convex MSK opt. persp.
-2.12823 106
10
7.05
0.01
500
ours
-2.22823 106
10
0.38
0.00
1000
SubsetSelectCI0
-7.92745 105
10
3709.35
219.40
1000
SubsetSelectC1O + warm start
-1.33080
10c
10
3702.48
90.26
1000
L0BnB
-2.10995 10
10
2118.04
0.00
1000
MSK persp.
3601.31
1000
MSK persp. + warm start
-2.19995 106
10
3600.48
0.20
1000
Convex MSK opt. persp.
-2.10995 106
10
54.57
0.01
1000
ours
-2.10995 106
10
0.57
0.00
3000
SubsetSelectCIO
-7.89940 10
10
3633.47
219.21
3000
SubsetSelectCIO + warm start
-1.31406
106
10
4260.43
91.89
3000
LOBnB
-2.10220 10
10
2318.67
4.73
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
3600.00
3000
Convex MSK opt. persp.
2.10220 10
10
1669.36 36
0.02
3000
ours
-2.10220
106
10
1.68
0.00
5000
SubsetSelectCIO
-9.76808 105
10
3838.13
157.56
5000
SubsetSelectCIO + warm start
-1.30652
106
10
4063.53
92.56
5000
I.0BnB
-2.09625 106
10
2700.89
3.42
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
2.09625- 10
10
3669.10
0.06
5000
ours
2.09625
106
10
5.09
0.00
Table 131: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k= 10,
0.1, A2 10.0).
153

---

# of features
method
upper bound
support size
time(s)
gap(%)
100
SubsetSelectCIO
-4.75666 106
10
3695.63
33.24
100
SubsenSelectCIO + warm start
-3.72965 106
10
1008.71
0.00
100
L.BnB
-5.28716 106
10
3186.48
0.00
100
MSK persp.
-5.28716- 106
10
3600.15
0.01
100
MSK persp. + warm start
-5.28716 10t
10
3600.07
0.01
100
Convex MSK opt. persp.
-5.28716 106
10
0.16
0.00
100
ours
-5.28716 106
10
0.30
0.00
500
SubsetSelectC1O
-4.61560- 106
10
3761.41
38.34
500
SubsetSelectC10 + warm start
-4.00602-. 106
10
3621.18
59.39
500
L0BnB
-5.31842- 10t
10
3958.53
1.65
500
MSK persp.
-5.31842 106
10
3604.61
0.09
500
MSK persp. warm start
-5.31842 106
10
3604.65
0.09
500
Convex MSK opt. persp.
-5.31842 106
10
7.05
0.01
500
ours
-5.31842 106
10
0,41
0.00
1000
SubsetSelectCIO
3.05644 106
6
3348.60
0.00
1000
SubsetSelectCIO+ + warm start
3..5760 106
10
3658.16
59.79
1000
L.0nBB
5..69964 106
10
4043.93
1.62
1000
MSK persp.
3600.44
1000
MSK persp. + warm start
5.26964 106
10
3600.14
0.20
1000
Convex MSK opt. persp.
5.26964 106
10
54.61
0.01
1000
ours
5.26964 108
10
0.51
0.00
3000
SubsetSelectC1O
-4.74347 106
10
3927.16
33.42
3000
SubsetSelectC10 + warm start
3600.00
3000
L0BnB
5.27658 106
10
4906.10
0.35
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
3600.00
3000
Convex MSK opt. persp.
5.27658
106
10
1962.60
0.02
3000
ours
5.27658
106
10
1.76
0.00
5000
SubsetSelectC10
3600.00
5000
SubsetSelectCIO + warm start
3600.00
5000
LOBnB
7200.00
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
3600.00
5000
Convex MSK opt. persp.
5.27116 106
10
3651.58
0.05
5000
ours
55.27116
106
10
4.32
0.00
Table 132: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
0 0.3, X
10.0).
154

---

# of features
method
upper bound
support size
time(s)
gap(%)
100
SubsetSelectCIO
-1.00851 10"
10
3728.97
30.84
100
SubsetSelectC1O + warm start
-1.02880- 107
10
3654.11
28.26
100
L0BnB
-1.10083 107
10
3417.76
0.26
100
MSK persp.
-1.10083- 107
10
1286.52
0.00
100
MSK persp. + warm start
-1.10083 107
10
1786.69
0.00
100
Convex MSK opt. persp.
-1.10083 107
10
0.16
0.00
100
ours
-1.10083 107
10
0.32
0.00
500
SubsetSelectC1O
9.42612 106
10
3685.22
40.77
500
SubsetSelectCIO + warm start
-9.92278 106
10
3685.15
33.72
500
L0BnB
-1.10517- 107
10
4308.04
0.77
500
MSK persp.
-1.10517 107
10
3601.09
0.09
500
MSK persp. + warm start
-1.10517- 107
10
3602.10
0.09
500
Convex MSK opt. persp.
-1.10517 107
10
7.72
0.01
500
ours
11.1011 107
10
0.41
0.00
1000
SubsetSelectCIO
1..0437 107
10
3649.62
30.84
1000
SubsetSelectC1O + warm start
6.62765 106
2
3147.50
0.00
1000
L0BnB
-1.09507 107
10
1834.54
0.65
1000
MSK persp.
3600.46
1000
MSK persp. + warm start
1.09507 107
10
3601.83
0.20
1000
Convex MSK opt. persp.
-
1..9507 107
10
57.71
0.01
1000
ours
11.95077 107
10
0.55
0.00
3000
SubsetSelectCIO
3600.00
3000
SubsetSelect110 + warm start
3600.00
3000
L0BnB
1.10011 107
10
5263.44
1.10
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
11.1001 107
10
4076.24
0.00
3000
Convex MSK opt. persp.
-1.10011 107
10
1538.48 48
0.02
3000
ours
-1.10011 107
10
1.73
0.00
5000
SubsetSelectC10
3600.00
5000
SubsetSelectC10 + warm start
3600.00
5000
LOBnB
- 11.09965 107
10
6222.74
1.11
5000
MSK persp.
3600.00
5000
MSK persp. warm start
3600.00
5000
Convex MSK opt. persp.
- 11.09965 107
10
3742.63
0.05
ours Table 133: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
5000
-1.09965 107
10
4.24
0.00
p 0.5, Xg
10.0).
155

---

# of features
method
upper bound
support size
time(s)
gap(%)
100
SubsetSelectCIO
-2.36231 10"
10
3732.54
23.59
100
SubsenSelectC1O + warm start
-2.32147 107
10
3623.89
25.77
100
L0BnB
-2.43574 107
10
2671.45
0.00
100
MSK persp.
-2.39754- 10"
10
3601.78
1.61
100
MSK persp. + warm start
-2.43574 107
10
3604.40
0.01
100
Convex MSK opt. persp.
-2.43574 -07
10
0.13
0.00
100
ours
-2.43574 107
10
0.30
0.00
500
SubsetSelectC1O
-2.77569 107
10
3601.0 .02
28.83
500
SubsetSelectCIO + warm start
- 107
10
3624.13
26.12
500
L0BnB
-2.4114 107
10
3822.46
0.00
500
MSK persp.
3605.48
500
MSK persp. + warm start
22.4184 107
10
3603.64 64
0.00
500
Convex MSK opt. persp.
-2.41184 107
10
7.40
0.01
500
ours
-2.4118 107
10
0.44
0.00
1000
SubsetSelectCIO
-2.8872 107
7
2623.29
0.00
1000
SubsetSelectC10+ + warm start
-2.22752 107
10
4344.29
24.76
1000
L0BnB
-2.11993 107
11
3983.89
0.31
1000
MSK persp.
3606.66
1000
MSK persp. warm start
-2.41988 107
10
3608.58
0.00
1000
Convex MSK opt. persp.
-2.41988 107
10
63.01
0.01
1000
ours
-2.41988 107
10
0.52
0.00
3000
SubsetSelectCIO
-2.34295 107
10
3767.28
24.75
3000
SubsetSelect110 + warm start
3600.00
3000
L0BnB
-2.2259 107
9
7332.12
0.00
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
-2.43712 107
10
4135.58
0.00
3000
Convex MSK opt. persp.
-2.43712 107
10
1968. 86
0.02
3000
ours
-2.43712 107
10
1.81
0.00
5000
SubsetSelectC10
3600.00
5000
SubsetSelectC10 + warm start
3600.00
5000
LOBnB
-2.43693 107
10
4661.71
23.68
5000
MSK persp.
3600.00
5000
MSK persp. warm start
3600.00
5000
Convex MSK opt. persp.
22.3693 107
10
3769.92
0.06
ours Table 134: Comparison of time and optimality gap on the synthetic datasets (n 100000, k= 10,
5000
-2.43693 107
10
4.08
0.00
p 0.7, A
10.0).
156

---

# of features
method
upper bound
support size
time(s)
gap(%)
100
SubsetSelectCIO
8.97109 10?
10
3602.39
21.73
100
SubsetSelectCIO + warm start
-9.05496 10?
10
3626.15
20.60
100
L0BnB
-9.11184 107
93
7102.10
0.04
100
MSK persp.
-8.99265 10?
10
3600.18
1.32
100
MSK persp. + warm start
-9.11025 107
10
3600.47
0.01
100
Convex MSK opt. persp.
-9.11025 107
10
0.13
0.00
100
ours
99.1005 107
10
0.32
0.00
500
SubsetSelectIIO
9.00037 107
10
4088.17
21.67
500
SubsetSelectCIO + warm start
99.0223 107
10
3817.48
21.37
500
L0BnB
99.1108 107
11
843.53
6.21
500
MSK persp.
3604.01
500
MSK persp. + warm start
9.12046 107
10
3604.99
0.00
500
Convex MSK opt. persp.
9.12046 107
10
7.57
0.01
500
ours
-9.12046 107
10
0.38
0.00
1000
SubsetSelectCIO
3600.00
1000
SubsetSelectC1O + warm start
3600.00
1000
L0BnB
8.23898 107
7369.81
9.08
1000
MSK persp.
3604.50
1000
MSK persp. + warm start
9.04074 107
10
3688.88
0.00
1000
Convex MSK opt. persp.
-9.0074 107
10
62.54
0.01
1000
ours
99.4074 107
10
0.52
0.00
3000
SubsetSelectCIO
3600.00
3000
SubsetSelectCIO + warm start
3600.00
3000
LOBnB
-9.1266 107
10
1883.50
6.26
3000
MSK persp.
3600.00
3000
MSK persp. + warm start
9.12760 107
10
3764.99
0.00
3000
Convex MSK opt. persp.
-9.12760 107
10
1999.52 52
0.02
3000
ours
-9.12760 107
10
20.84
0.00
5000
SubsetSelectCIO
3600.00
5000
SubsetSelectCIO + warm start
3600.00
5000
L0BnB
8.32064 107
7664.70
9.32
5000
MSK persp.
3600.00
5000
MSK persp. + warm start
9.12915 107
10
8218.76
0.00
5000
Convex MSK opt. persp.
9.12915 107
10
3654.46
3.94
5000
ours
9.11915- 107
10
57.41
0.00
Table 135: Comparison of time and optimality gap on the synthetic datasets (n 100000, k 10,
0 0.9, A2
10.0).
157

---

G.4.4
Synthetic Benchmark with X2 0.001
# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
-1.26607 10*
2
1.93
0.00
3000
SubsetSelectCIO+ + warm start
-1.26607 10a
2
2.02
0.00
3000
L0BnB
-6.34585 104
10
1200.55
0.00
3000
MSK persp.
-5.27912 101
10
3601.00
43.41
3000
MSK persp. + warm start
-6.34585 -104
10
3601.81
19.30
3000
Convex MSK opt. persp.
-4.55044 10a
10
1573.44
66.10
3000
ours
-6.34585 10
10
3600.49
19.45
4000
SubsetSelectCIO
-188997 10*
2
1.90
0.00
4000
SubsetSelectC1O+ warm start
-1.88907 104
2
2.06
0.00
4000
L0BnB
-8.64294 104
10
579.39
0.00
4000
MSK persp.
-8.64294 10a
10
3602.16
15.37
4000
MSK persp. + warm start
-8.64294 10a
10
3600.84
15.37
4000
Convex MSK opt. persp.
-8.64294 10a
10
1398.74
2.40
4000
ours
-8.64294 10*
10
20.25
0.00
5000
SubsetSelectC10
2.49336 101
2
2.08
0.00
5000
SubsetSelectCIO + warm start
2.49336 104
2
1.96
0.00
5000
L0BnB
-1.09576 105
10
1292.85
0.00
5000
MSK persp.
-1.9957 10
10
3601.60
11.91
5000
MSK persp. + warm start
-1.09576 105
10
3600.55
11.91
5000
Convex MSK opt. persp.
-1.9576 10
10
1496.25
1.03
5000
ours
-109557 10
10
20.51
0.00
6000
SubsetSelectCIO
3600.00
6000
SubsetSelectCIO- + warm start
-.833063 104
2
2.22
0.00
6000
LOBnB
11.27643 10
10
1379.36
0.00
6000
MSK persp.
3600.07
6000
MSK persp. + warm start
-1.27643 10
10
3600.30
9.93
6000
Convex MSK opt. persp. ours
-1.27643 105
10
1318.81
0.62
6000
-1.27643 105
10
1.70
0.00
7000
SubsetSelectCIO
-3.24781 104
2
2.06
0.00
7000
SubsetSelectCIO warm start
-3.24781 10a
2
2.22
0.00
7000
L0BnB
-1.50614. 105
10
1105.58
0.00
7000
MSK persp.
-2.07297 104
3
3600.06
687.08
7000
MSK persp. + warm start
-1.50614 10s
10
3599.93
8.33
7000
Convex MSK opt. persp.
-1.50614 105
10
1455.27
0.45
7000
ours
-1.50614 105
10
1.67
0.00
Table 136: Comparison of time and optimality gap the synthetic datasets (p X2 0.001).
3000, K
10,
0
158

---

# of samples
method
upper bound
support size
time(s)
gap( %6)
3000
SubsetSelectC1O
-7.20755 10*
2
1.99
0.00
3000
SubsetSelectCIO warm start
-7.20753 10*
2
2.08
0.00
3000
L0BnB
-1.59509 10
10
3023.58
0.00
3000
MSK persp.
-1.48826 -00
10
3601.42
27.95
3000
MSK persp. + warm start
-1.59509 105
10
3601.71
19.38
3000
Convex MSK opt. persp.
-1.13813 10s
10
1627.92
67.04
3000
ours
-1.59509 10.
10
3600.67
19.53
4000
SubsetSelectCIO
-1.03099- 10
2
2.07
0.00
4000
SubsetSelectCIO+ + warm start
-1.03099- 10
2
1.81
0.00
4000
L0BnB
-2.17574 105
10
3396.85
0.00
4000
MSK persp.
-2.17574 105
10
3600.26
15.27
4000
MSK persp. + warm start
-2.17544 105
10
3601.91
15.27
4000
Convex MSK opt. persp.
-2.17574 105
10
1509.12
2.41
4000
ours
-2.17574 105
10
100.13
0.00
5000
SubsetSelectCIO
-1.00339 10
2
1.98
0.00
5000
SubsetSelecrCIO+ + warm start
-1.30339 10
2
2.01
0.00
5000
L0BnB
-2.11986
105
10
531.44
0.00
5000
MSK persp.
-2.71986 105
10
3600.74
12.00
5000
MSK persp. + warm start
-2.71986 105
10
3600.26
12.00
5000
Convex MSK opt. persp.
-2.71986 105
10
1514.31
1.04
5000
ours
-2.71986 105
10
20.80
0.00
6000
SubsetSelectC10
-1.50202 105
2
2.02
0.00
6000
SubsetSelectCIOO+ + warm start
-150200 105
2
1.96
0.00
6000
LOBnB
-3.16761 1
10
3098.88
0.00
6000
MSK persp.
-7.83273 104
1
3603.79
344.72
6000
MSK persp. + warm start
-3.16761 1
10
3604.35
9,96
6000
Convex MSK opt. persp.
-3.16761 105
10
1445.46
0.62
6000
ours
-3.16761 1
10
20.70
0.00
7000
SubsetSelectC10
-1.75217 10
2
2.12
0.00
7000
SubsetSelectCIO+ + warm start
1.75211 100
2
2.12
0.00
7000
LOBnB
-3.72610 105
10
3130.56
0.00
7000
MSK persp.
3599.99
7000
MSK persp. + warm start
3.72610 105
10
3600.24
8.33
7000
Convex MSK opt. persp.
3.72610 10
10
1520.22
0.45
7000
ours
-3.72610 100
10
1.98
0.00
Table 137: Comparison time and optimality gap on the synthetic datasets (p 3000, K 0.3, A2 0.001).
10,
159

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectC1O
-2.19279 10a
2
1.96
0.00
3000
SubsetSelectCIO start
-2.19279 10
2
1.85
0.00
3000
L0BnB
-3.27442 10
8
7379.57
5.16
3000
MSK persp.
-2.83751 10a
10
3601.66
40.41
3000
MSK persp. + warm
-3.33604 10
10
3601.38
19.43
3000
Convex MSK opt. persp.
-2.77416 1105
10
1878.24
43.38
3000
ours
-3.33604 105
10
3600.58
19.57
4000
SubsetSelectC10
-3.03679 105
2
1.87
0.00
4000
SubsetSelectCIO + warm start
-3.03679 10
2
1.89
0.00
4000
LOBnB
-4.33261 10
10
4149.90
32.94
4000
MSK persp.
4.53261 10
10
3602.72
15.24
4000
MSK persp. + warm start
4.53261 105
10
3599.91
15.24
4000
Convex MSK opt. persp.
4.53261 10
10
1445.07
2.53
4000
ours
4.53261 10
10
3600.25
0.63
5000
SubsetSelectC1O
4.12220 105
3
1.79
0.00
5000
SubsetSelecrCIO + warm start
-4.12520 10
3
2.03
0.00
5000
LOBnB
-5.63007 100
10
3522.00
0.00
5000
MSK persp.
-5.63007 10
10
3600.76
12.04
5000
MSK persp. + warm start
-5.63007 10
10
3603.42
12.04
5000
Convex MSK opt. persp.
-5.63007 10
10
1618.41
1.04
5000
ours
5.63007 105
10
36.97
0.00
6000
SubsetSelectC1O
4.42065 105
2
1.94
0.00
6000
SubsetSelectC1O+ + warm start
4.42065 105
2
2.12
0.00
6000
L0BnB
6.57799 105
10
3327.45
0.00
6000
MSK persp.
3605.66
6000
MSK persp. + warm start
6.57799 105
10
3605.89
9.98
6000
Convex MSK opt. persp.
6.57799 10
10
1523.01
0.62
6000
ours
6.57799 105
10
20.75
0.00
7000
SubsetSelectCIO
-5.16405 1
2
2.11
0.00
7000
SubsetSelectC1O + warm start
55.1440 105
2
2.14
0.00
7000
LOBnB
-7.72267 105
10
3567.05
0.00
7000
MSK persp.
3600.27
7000
MSK persp. + warm start
-7.72267 105
10
3600.18
8.33
7000
Convex MSK opt. persp.
-7.72267 105
10
1681.35
0.45
7000
ours
-7.72267 105
10
21.02
0.00
Table 138: Comparison time and optimality gap on the synthetic datasets (p 3000,
10,
0.5, Ag = 0.001).
160

---

# of samples
method
upper bound
support size
time(s)
sap(%)
3000
SubsetSelectC1O
6.06571 105
2
1.88
0.00
3000
SubsetSelectCIO+ warm start
-6.06571 10s
2
1.88
0.00
3000
L0BnB
-5.52986 10
1
7308.45
34.21
3000
MSK persp.
-6.89430- 105
10
3600.74 74
28.43
3000
MSK persp. + warm start
-7.41187 10
10
3601.61
19.46
3000
Convex MSK opt. persp.
-6.84471 10s
10
1652.72
29.14
3000
ours
-7.41187 105
10
3601.14
19.60
4000
SubsetSelectCIO
-8.24785 10
2
1.85
0.00
4000
SubsetSelectC1O + warm start
-8.44785 105
2
1.86
0.00
4000
L0BnB
-7.53893 105
1
7293.04
32.01
4000
MSK persp.
-9.95041 10S
10
3601.67
16.12
4000
MSK persp. + warm start
-1.00287 106
10
3602.00
15.21
4000
Convex MSK opt. persp.
-1.00287 106
10
1442.72
2.82
4000
ours
-1.00287 106
10
3600.37
1.88
5000
SubsetSelectC10
-1.02490 106
2
2.05
0.00
5000
SubsetSelectCIO++ + warm start
-1.02490 106
2
1.92
0.00
5000
L0BnB
-1.23555 10
9
7387.5
4.90
5000
MSK persp.
-1.24027 106
10
3604.54
12.07
5000
MSK persp. + warm start
-1.24027 106
10
3600.77
12.07
5000
Convex MSK opt. persp.
-1.24027 10
10
1654.86
1.06
5000
ours
-1.24027 106
10
466.42
0.00
6000
SubsetSelectCIO
-1.20146 106
2
1.96
0.00
6000
SubsetSelectCIO + warm start
-1.20146 10
2
2.05
0.00
6000
L0BnB
-1.44860 106
9
7265.23
4.30
6000
MSK persp.
-1.14100 1006
2
3603.46
40.19
6000
MSK persp. + warm start
-1.45417 1
10
3604.24
10.00
6000
Convex MSK opt. persp.
-1.45417 106
10
1667.19
0.63
6000
ours
-1.45417 106
10
57.67
0.00
7000
SubsetSelectCIO
-1.46875 1
3
1.86
0.00
7000
SubsetSelectC1O + warm start
-1.46875 106
3
2.09
0.00
7000
LOBnB
-1.99804 106
9
7320.83
2.92
7000
MSK persp.
-1.32733 106
2
3600.13
39.15
7000
MSK persp. + warm start
-1.70483 106
10
3600.05
8.34
7000
Convex MSK opt. persp.
-1.70483
106
10
1760.17
0.45
7000
-1.70483 106
10
21.42
0.00
Table 139: Comparison time and optimality gap on synthetic datasets (p 3000,
10,
0.7, A2 = 0.001).
161

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
-..33352 10
2
1.90
0.00
3000
SubsetSelectCIO start
-2.63352 106
2
2.04
0.00
3000
L.BnB
-2.55637 106
1
7236.54
18.84
3000
MSK persp.
-2.72590 106
10
3599.87
22.06
3000
MSK persp. + warm
-2.72590 106
10
3600.80
22.06
3000
Convex MSK opt. persp.
-2.72998 106
10
2311.76
21.67
3000
ours
-2.78521 108
10
3600.01
19.60
4000
SubsetSelectCIO
-3.38912 10
1
1.79
0.00
4000
SubsetSelectC100 + warm start
3.38912 106
1
1.89
0.00
4000
L0BnB
33.45201 106
7291.93
15.48
4000
MSK persp.
3609.13
4000
MSK persp. + warm start
33.75203 106
10
3605.54
15.12
4000
Convex MSK opt. persp.
33.75202 106
10
1481.77
3.23
4000
ours
-3.75203 106
10
3600.42
3.50
5000
SubsetSelectCIO
-4.37679 10t
2
1.99
0.00
5000
SubsetSelectCIO+ + warm start
-4.37679 106
2
2.07
0.00
5000
L0BnB
-4.23095 10
1
7316.72
13.65
5000
MSK persp.
-4.60534 10t
10
3602.67
12.43
5000
MSK persp. + warm start
-4.61877 106
10
3601.35
12.11
5000
Convex MSK opt. persp.
4.61877 106
10
1537.61
1.22
5000
ours
4.61877 10
10
3600.57
0.93
6000
SubsetSelectC10
5.15897 106
2
2.23
0.00
6000
SubsetSelectCIO + warm start
5.15897 10
2
2.24
0.00
6000
LOBnB
-4.97742 106
1
7330.86
13.05
6000
MSK persp.
3603.01
6000
MSK persp. + warm start
-5.43826 106
10
3602.04
0.00
6000
Convex MSK opt. persp.
-5.43826
106
10
1652.62
0.65
6000
ours
-5.48826 108
10
3600.65
0.33
7000
SubsetSelectC10
6.02832 106
2
1.89
0.00
7000
SubsetSelectC100+ + warm start
6.02832 106
2
2.27
0.00
7000
LOBnB
5.82331 106
7356.27
11.85
7000
MSK persp.
3600.06
7000
MSK persp. + warm start
6.36767 106
10
3600.21
8.35
7000
Convex MSK opt. persp.
6.36767
106
10
1821.91
0.45
7000
ours
6.36767 106
10
3600.78
0.11
Table 140: Comparison of time and optimality gap the synthetic datasets (p 3000, K 0.9, Ag = 0.001).
10,
162

---

G.4.5
Synthetic Benchmark with X2 0.1
# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
-1.83147 10*
10
4229.80
313.93
3000
SubsetSelectCIO+ + warm start
-5.70956 104
10
3602.11
32.77
3000
LOBnB
-6.34575 104
10
1201.41
0.00
3000
MSK persp.
-6.34575 104
10
3601.57
17.90
3000
MSK persp. + warm start
-6.34575 104
10
3600.12
17.89
3000
Convex MSK opt. persp.
-6.34575 10a
10
1444.07
17.85
3000
ours
-6.34575 10
10
3600.33
19.31
4000
SubsetSelectCIO
-3.10931 10*
10
3872.26
235.85
4000
SubsetSelectC1O+ warm start
-7.28764 104
10
4101.55
43.29
4000
L0BnB
-8.64284 104
10
566.25
0.00
4000
MSK persp.
-8.64284 10a
10
3602.29
15.03
4000
MSK persp. + warm start
-8.64284 10a
10
3601.14
15.05
4000
Convex MSK opt. persp.
-8.64284 10a
10
1458.63
2.40
4000
ours
-8.64284 10*
10
20.81
0.00
5000
SubsetSelectC10
4.06480 101
10
3939.75
222.71
5000
SubsetSelectCIO + warm start
9.16380 104
10
3804.66
43.14
5000
L0BnB
-1.99575 105
10
1314.94
0.00
5000
MSK persp.
-1.09575 10
10
3603.48
11.78
5000
MSK persp. + warm start
-1.09575 105
10
3602.79
11.78
5000
Convex MSK opt. persp.
-1.9555- 10.
10
1478.60
1.03
5000
ours
1.09577 10
10
20.65
0.00
6000
SubsetSelectCI0
3600.00
6000
SubsetSelectCIO- + warm start
1..21549 105
10
3850.54
26.10
6000
LOBnB
-1.27642 10
10
1536.59
0.00
6000
MSK persp.
3606.50
6000
MSK persp. + warm start
-1.27642 10.
10
3603.98
9.85
6000
Convex MSK opt. persp. ours
-1.27642
105
10
1303.39
0.62
6000
-1.27642 105
10
1.70
0.00
7000
SubsetSelectCIO
-5.44763 104
10
4067.63
224.61
7000
SubsetSelectCIO++ warm start
-1.42693
105
10
3615.01
26.20
7000
L0BnB
-1.00613 105
10
1064.72
0.00
7000
MSK persp.
3599.90
7000
MSK persp. warm start
1.50613 10
10
3600.13
8.28
7000
Convex MSK opt. persp.
1.50613 10.
10
1405.96
0.45
7000
ours
11.0613 105
10
1.67
0.00
Table 141: Comparison of time and optimality on the synthetic datasets (p 3000, k = 10,
0.1, A2 0.1).
163

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
-..13982 105
10
3755.86
67.28
3000
SubsetSelectCIO start
-1.59508 10s
10
3857.37
19.54
3000
L.BnB
-1.59508 105
10
1491.90
0.00
3000
MSK persp.
-1.59508 105
10
3600.12
17.99
3000
MSK persp. + warm
-1.99508 10S
10
3600.37
17.99
3000
Convex MSK opt. persp.
-1.95508 10
10
1454.84
17.99
3000
ours
-1.95508 105
10
3600.27
19.40
4000
SubsetSelectCIO
3600,00
4000
SubsetSelectCIO+ + warm start
-.17773 10a
10
3711.79
20.70
4000
L0BnB
-2.1753
105
10
3399.35
0.00
4000
MSK persp.
-2.17533 -105
10
3601.17
14.95
4000
MSK persp. + warm start
-2.17573 105
10
3602.25
14.95
4000
Convex MSK opt. persp.
-2.17573 105
10
1510.77
2.40
4000
ours
-2.17533 105
10
99.53
0.00
5000
SubsetSelectCIO
-1.92108 10
10
3602.76
69.69
5000
SubsetSelectCIO+ + warm start
-2.71985 10
10
3733.49
19.85
5000
L0BnB
-2.11985 105
10
532.76
0.00
5000
MSK persp.
-2.71985 105
10
3601.74
11.88
5000
MSK persp. + warm start
-2.71985 105
10
3602.31
11.88
5000
Convex MSK opt. persp.
-2.71985 105
10
1576.92
1.04
5000
ours
-2.71985 105
10
21.01
0.00
6000
SubsetSelectC10
-2.03206 105
10
4031.62
87.30
6000
SubsetSelectCIO + warm start
33.00827
105
10
3780.98
22.45
6000
LOBnB
-3.16760 10
10
3082.87
0.00
6000
MSK persp.
3600.17
6000
MSK persp. + warm start
-3.16760 10s
10
3599.98 .9
9.90
6000
Convex MSK opt. persp.
33.16760
10
10
1487.94 94
0.62
6000
ours
-3.16760 105
10
20.91
0.00
7000
SubsetSelectC10
2.44730 105
10
3992.94
82.05
7000
SubsetSelectC100+ + warm start
3.37947
105
10
4164.11
31.83
7000
LOBnB
-3.72609 105
10
3126.74
0.00
7000
MSK persp.
3600.16
7000
MSK persp. + warm start
-3.72609 105
10
3604.63
8.28
7000
Convex MSK opt. persp.
3.72609
105
10
1555.38
0.45
7000
ours
-3.72609 105
10
1.73
0.00
Table 142: Comparison of time and optimality gap the synthetic datasets (p 3000, k
10,
0.3, Ag = 0.1).
164

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
2.799591 105
9
3686.43
42.67
3000
SubsetSelectCIO start
-3.11853 10
10
3738.09
27.91
3000
L.BnB
-3.27441 105
8
7347.92
4.34
3000
MSK persp.
-3.33603 105
10
3600.83
18.03
3000
MSK persp. + warm start
-3.33603 10
10
3600.62
18.03
3000
Convex MSK opt. persp.
-3.18968 10
10
1341.73
23,47
3000
ours
-3.33603 105
10
3600.68
19.44
4000
SubsetSelectCIO
3600,00
4000
SubsetSelectCIO+ + warm start
4.44307 10a
10
3651.78
23.07
4000
L0BnB
-4.53260
105
10
4150.43
32.94
4000
MSK persp.
-4.33260
105
10
3602.80
14.94
4000
MSK persp. + warm start
-4.33260
105
10
3601.24
14.94
4000
Convex MSK opt. persp.
-..53260
105
10
1444.17
2.53
4000
ours
4.53260
105
10
3600.41
0.62
5000
SubsetSelectCIO
-4.69039 10
8
3951.25
43.95
5000
SubsetSelectCIO+ + warm start
-5.44684
10
9
4020.24
23.96
5000
L0BnB
-5.63006 105
10
3516.88
0.00
5000
MSK persp.
-5.63006 10
10
3603.31
11.92
5000
MSK persp. + warm start
-5.63006 105
10
3601.57
11.92
5000
Convex MSK opt. persp.
-5.63006 105
10
1572.75
1.04
5000
ours
5.63006 105
10
37.26
0.00
6000
SubsetSelectC10
2.80001 105
1
2.28
0.00
6000
SubsetSelectCIO + warm start
6.44963
105
10
3724.51
22.59
6000
LOBnB
-6.57798 10
10
3317.18
0.00
6000
MSK persp.
3604.37
6000
MSK persp. + warm start
6.57798 10s
10
3606.26
9.92
6000
Convex MSK opt. persp.
66.77998
10
10
1657.28
0.62
6000
ours
66.7799 105
10
20.64
0.00
7000
SubsetSelectC10
3.48812 105
1
2.12
0.00
7000
SubsetSelectC100+ + warm start
-3.45235 105
1
2.06
0.00
7000
LOBnB
-7.72266 105
10
3589.79
0.00
7000
MSK persp.
3599.94
7000
MSK persp. + warm start
-7.72266 105
10
3600.05
8.29
7000
Convex MSK opt. persp.
-7.72266
105
10
1603.99
0.45
7000
ours
-7.72266 105
10
20.91
0.00
Table 143: Comparison of time and optimality gap the synthetic datasets (p 3000, K
10,
0.5, Ag = 0.1).
165

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
66.06455 105
10
4096.61
28.35
3000
SubsetSelectCIO start
-7.31489
10s
10
3621.77
21.19
3000
L.BnB
-5.52981
10
1
7310.65
34.21
3000
MSK persp.
-7.39018
105
10
3600.41
18.44
3000
MSK persp. + warm
-7.41186
10
10
3600.40
18.10
3000
Convex MSK opt. persp.
-7.07443
10
10
1698.29
23.74
3000
ours
-7.41186 105
10
3600.59
19.46
4000
SubsetSelectCIO
-9.43345 10
10
3677.02
28.22
4000
SubsetSelectC1O0+ + warm start
-9.93033 105
10
3684.08
21.81
4000
L0BnB
-7.53887 105
1
7333.72
31.99
4000
MSK persp.
-9.98256 10S
10
3600.92
15,46
4000
MSK persp. + warm start
-1.00287 106
10
3602.18
14.92
4000
Convex MSK opt. persp.
-1.00287 106
10
1574.91
2.82
4000
ours
-100227 106
10
3600.08
1.88
5000
SubsetSelectCIO
3600.00
5000
SubsetSelectCIO + warm start
1.22297 106
10
3997.81
21.00
5000
L0BnB
-1.23555 106
9
7414.89
5.14
5000
MSK persp.
-1.24027 10t
10
3601.11
11.96
5000
MSK persp. + warm start
-1.24027 106
10
3601.56
11.96
5000
Convex MSK opt. persp.
-1.24027 106
10
1638.36
1.06
5000
ours
-1.24027 10
10
469.43
0.00
6000
SubsetSelectC10
-9.41611 105
1
59.04
0.00
6000
SubsetSelectCIO + warm start
-9.78724 105
1
2.14
0.00
6000
LOBnB
-1.4460 106
9
7352.27
4.07
6000
MSK persp.
3602.12
6000
MSK persp. + warm start
-1.45416 106
10
3600.04
9.94
6000
Convex MSK opt. persp.
-1.45416
106
10
1653.65
0.63
6000
ours
-1.45416 108
10
59.16
0.00
7000
SubsetSelectC10
-1.14528 106
1
2.13
0.00
7000
SubsetSelectC100+ + warm start
-1.14528
106
1
2.11
0.00
7000
LOBnB
-169800
106
9
7503.37
3.09
7000
MSK persp.
3600.28
7000
MSK persp. + warm start
-1.70483 106
10
3600.18
8.30
7000
Convex MSK opt. persp.
-1.70483 106
10
1715.12
0.45
7000
ours
-
-1.70883 106
10
21.12
0.00
Table 144: Comparison of time and optimality gap the synthetic datasets (p 3000, K
10,
0.7, Ag = 0.1).
166

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectC1O
-2.7050 10s
5
3696.31
23.15
3000
SubsetSelectCIO start
-2.76623 10c
7
4030.96
20.43
3000
L0BnB
-2.55637 106
1
7339.86
18.82
3000
MSK persp.
-2.75384 10
10
3600.44
19.50
3000
MSK persp. + warm
-2.75384 10
10
3600.07
19.50
3000
Convex MSK opt. persp.
-2.72647 106
10
1999.54
20.67
3000
ours
-2.78521 10
10
3600.23
19.46
4000
SubsetSelectC10
-3.65950 106
9
3841.51
23.56
4000
SubsetSelectCIO + warm start
-3.74433
10
10
3649.94
20.76
4000
LOBnB
-3.45200
106
1
7481.26
15.51
4000
MSK persp.
-3.72627
106
10
3601.44
15.63
4000
MSK persp. + warm start
-3.75203
10
10
3601.81
14.84
4000
Convex MSK opt. persp.
-3.75202
10
10
1543.65
3.23
4000
ours
-3.75003 106
10
3601.10
3.50
5000
SubsetSelectC10
3600.00
5000
SubsetSelectCIO++ + warm start
4.61877 106
10
3681.64
20.02
5000
L0BnB
-4.23094 106
1
7531.59
13.78
5000
MSK persp.
-4.60533 106
10
3605.23
12.32
5000
MSK persp. + warm start
-4.61877 106
10
3601.43
11.99
5000
Convex MSK opt. persp.
-4.1187 10
10
1633.96
1.22
5000
ours
-4.61877 10
10
3600.40
0.93
6000
SubsetSelectCIO
5.38482 106
10
3762.10 10
21.45
6000
SubsetSelectCIO + warm start
5.42848 10
10
3891.96
20.47
6000
L0BnB
-4.97741 106
1
7211.19
12.81
6000
MSK persp.
-4.81401 106
1
3601.53
24.21
6000
MSK persp. + warm start
-5.43826 106
10
3601.18
9.95
6000
Convex MSK opt. persp.
-5.43826 106
10
1535.09
0.65
6000
ours
-5.43826 106
10
3600.26
0.33
7000
SubsetSelectCIO
5.66568 1
1
2.34
0.00
7000
SubsetSelectC1O + warm start
5.66568
106
1
2.16
0.00
7000
LOBnB
5.82321 106
7336.27
11.85
7000
MSK persp.
3600.04
7000
MSK persp. + warm start
6.36767 10
10
3599.94
8.31
7000
Convex MSK opt. persp.
6.36767 106
10
1779.25
0.45
7000
ours
6.36767 106
10
3600.58
0.11
Table 145: Comparison time and optimality gap on the synthetic datasets (p 3000,
10,
0.9, Ag = 0.1).
167

---

G.4.6
Synthetic Benchmark with X2 10.0
# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectCIO
-4.22001 10*
10
3915.82
53.90
3000
SubsetSelectCIO+ + warm start
-3.58868 10a
10
3913.17
111.25
3000
LOBnB
-6.33576 10a
10
1198.54
0.00
3000
MSK persp.
-6.33576 104
10
3601.44
8.58
3000
MSK persp. + warm start
-6.33576 10*
10
3600.20
8.58
3000
Convex MSK opt. persp.
-6.33576 10a
10
1334.48
9.04
3000
ours
-6.33576 10
10
3601.41
4.54
4000
SubsetSelectCIO
-7.36488 10*
10
4165.15
41.79
4000
SubsetSelectC1O+ warm start
-5.42345 104
10
3789.71
92.55
4000
L0BnB
-8.63287 104
10
1579.27
0.00
4000
MSK persp.
-8.63287 10a
10
3600.31
7.64
4000
MSK persp. + warm start
-8.63287 104
10
3601.55
7.64
4000
Convex MSK opt. persp.
-8.62287 104
10
1478.13
2.24
4000
ours
8.63287 10*
10
20.33
0.00
5000
SubsetSelectCIO
3600,00
5000
SubsetSelectCIO+ + warm start
66.1010 10a
10
3623.84
112.94
5000
L0BnB
11.09474 105
10
1296.29
0.00
5000
MSK persp.
-1.09474 100
10
3601.04
6.75
5000
MSK persp. + warm start
-1.09474 105
10
3602.25
6.75
5000
Convex MSK opt. persp.
-1.09474 105
10
1532.63
1.00
5000
ours
-1.09474 105
10
20.71
0.00
6000
SubsetSelectCIO
9.91460 10*
10
3945.11
54.60
6000
SubsetSelectCIO+ + warm start
-8.28163 104
10
3702.54
85.08
6000
LOBnB
-1.77542 10
10
1268.61
0.00
6000
MSK persp.
-1.27542 10
10
3602.43
6.47
6000
MSK persp. + warm start
-1.27542 105
10
3607.80
6.47
6000
Convex MSK opt. persp.
-1.27542 105
10
1314.11
0.61
6000
ours
-1.27542 105
10
1.69
0.00
7000
SubsetSelectC10
-1.17343 105
10
3661.02
53.46
7000
SubsetSelectCIO+ + warm start
-9.54749
10*
10
3758.42
88.62
7000
LOBnB
-1.00512 10
10
1063.19
0.00
7000
MSK persp.
-1.50512 105
10
3600.60
5.68
7000
MSK persp. + warm start
-1.50512 105
10
3600.79
5.68
7000
Convex MSK opt. persp.
-
-1.50512 105
10
1480.64
0.45
7000
ours
-1.50512 105
10
1.70
0.00
Table 146: Comparison of time and optimality gap the synthetic datasets (p
3000, k
10,
0.1, A2 = 10.0).
168

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectC1O
-1.39974 105
10
3764.95
36.22
3000
SubsetSelectCIO+ warm start
-1.44268 10s
10
3937.05
32.16
3000
L0BnB
-1.59409 10
10
1563.82
0.00
3000
MSK persp.
-1.59409 105
10
3601.01
9.02
3000
MSK persp. + warm start
-1.59409 10
10
3600.73
9.02
3000
Convex MSK opt. persp.
-1.59409 10s
10
1511.63
9.58
3000
ours
-1.59409 105
10
3600.62
4.68
4000
SubsetSelectCIO
-1.22403 10
10
3763.38
43.97
4000
SubsetSelectC1O + warm start
-2.8871- 105
10
4023.55
25.82
4000
L0BnB
-2.17472 105
10
3126.91
0.00
4000
MSK persp.
-2.17472 10S
10
3605.75
7.91
4000
MSK persp. + warm start
-2.17472 105
10
3601.48
7.91
4000
Convex MSK opt. persp.
-2.17472 105
10
1478.15
2.24
4000
ours
-2.1747 105
10
86.90
0.00
5000
SubsetSelectC10
-2.60617 105
10
3976.98
25.08
5000
SubsetSelectCIO++ + warm start
-2.19315 105
10
3615.40
48.64
5000
L0BnB
-2.71885 105
10
1571.26
0.00
5000
MSK persp.
-2.71885 105
10
3602.81
7.26
5000
MSK persp. + warm start
-2.71885 105
10
3603.13
7.26
5000
Convex MSK opt. persp.
-2.71885 10.
10
1509.28
1.01
5000
ours
-2.71885 10
10
20.75
0.00
6000
SubsetSelectCIO
-2.62606 105
10
3828.76
44.93
6000
SubsetSelectCIO + warm start
-284430 105
10
3662.07
33.87
6000
L0BnB
-3.16660 105
10
3085.01
0.00
6000
MSK persp.
-3.16660 105
10
3601.61
6.73
6000
MSK persp. + warm start
-3.16660 105
10
3605.65
6.73
6000
Convex MSK opt. persp.
-3.16660 10
10
1471.81
0.61
6000
ours
3..666 10
10
21.05
0.00
7000
SubsetSelectCIO
-2.66322 10
10
3700.24
50.35
7000
SubsetSelectC10 + warm start
-2.97673
105
10
3747.76
49.67
7000
LOBnB
-3.72509
105
10
3125.79
0.00
7000
MSK persp.
-3.72509
105
10
3605.45
5.87
7000
MSK persp. + warm start
-3.72509
100
10
3601.55
5.87
7000
Convex MSK opt. persp.
-3.72509
105
10
1549.05
0.45
7000
-3.72509
105
10
1.71
0.00
Table 147: Comparison time and optimality gap on the synthetic datasets (p 3000, K 10,
0.3, A2 = 10.0).
169

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectC1O
-3.12174 105
10
3636.16
27.78
3000
SubsetSelectCIO+ + warm start
-3.27648 10
10
3759.74
21.75
3000
L0BnB
-3.27321 10
8
7307.65
4.14
3000
MSK persp.
-3.33503 105
10
3600.22
9.25
3000
MSK persp. + warm start
-3.33503 10
10
3600.89
9.25
3000
Convex MSK opt. persp.
-3.33503 10s
10
1475.73
9.93
3000
ours
-3.33503 105
10
3600.25
6.21
4000
SubsetSelectCIO
-3.96358 10
10
3655.16
37.96
4000
SubsetSelectC1O + warm start
-4.44111 105
10
3679.12
23.13
4000
L0BnB
-4.33159 105
10
4152.36
32.87
4000
MSK persp.
-4.53159 105
10
3602.35
8.06
4000
MSK persp. + warm start
-..3319 105
10
3600.09
8.06
4000
Convex MSK opt. persp.
-4.53159 105
10
1513.22
2.34
4000
ours
-4.33159 10
10
3600.95
0.35
5000
SubsetSelectC10
-4.93471 105
10
3709.4 44
36.82
5000
SubsetSelectCIO + warm start
-5.27817 105
10
3730.92
27.92
5000
L0BnB
-5.62906 105
10
3521.21
0.00
5000
MSK persp.
-5.62906 105
10
3602.11
7.22
5000
MSK persp. + warm start
-5.62906 105
10
3603.85
7.44
5000
Convex MSK opt. persp.
-5.62906 10.
10
1530.90
1.02
5000
ours
-5.62906 10
10
31.47
0.00
6000
SubsetSelectCIO
5.71812 105
10
3620.92
38.27
6000
SubsetSelectCIO + warm start
-6.34165 105
10
3675.07
24.67
6000
L0BnB
-6.57698 105
10
3313.82
0.00
6000
MSK persp.
-6.57698 105
10
3605.13
6.87
6000
MSK persp. + warm start
-6.57698 105
10
3608.94 94
6.87
6000
Convex MSK opt. persp.
-6.57698 105
10
1642.87
0.62
6000
ours
-6.57698 105
10
21.24
0.00
7000
SubsetSelectCIO
-7.32796 1
10
3784.82
26.02
7000
SubsetSelectC10 + warm start
-7.45541
105
10
3662.49
23.87
7000
LOBnB
-7.72166
1005
10
3081.93
0.68
7000
MSK persp.
-7.72166 105
10
3604.29
5.98
7000
MSK persp. + warm start
-7.72166
105
10
3606.92
5.98
7000
Convex MSK opt. persp.
-7.72166
105
10
1526.71
0.45
7000
-7.72166
105
10
20.37
0.00
Table 148: Comparison time and optimality gap on the synthetic datasets (p 3000,
10,
0.5, À2 = 10.0).
170

---

# of samples
method
upper bound
support size
time(s)
gap(%)
3000
SubsetSelectC1O
-6.9357 105
10
3727.41
27.12
3000
SubsetSelectCIO+ + warm start
-7.35099
10s
10
3678.19
20.59
3000
L0BnB
-5.52435
10
1
7287.00
33.44
3000
MSK persp.
-7.41086 105
10
3602.90
9.56
3000
MSK persp. + warm start
-7.41086
100
10
3600.62
9.56
3000
Convex MSK opt. persp.
-7.41086
10s
10
1525.68
10.22
3000
ours
-7.41086
105
10
3600.48
8.32
4000
SubsetSelectCIO
99.55426 10a
10
3756.83
25.29
4000
SubsetSelectC1O + warm start
-9.86629 105
10
3731.48
22.60
4000
L0BnB
-7.53327 105
1
7269.26
31.51
4000
MSK persp.
-1.00277 106
10
3600.55
8.17
4000
MSK persp. + warm start
-1.00277 106
10
3601.59
8.17
4000
Convex MSK opt. persp.
-1.00277 106
10
1555.56
2.60
4000
ours
-1.00277 106
10
3600.62
1.66
5000
SubsetSelectC10
-1.14948 106
10
3626.73
29.45
5000
SubsetSelectCIO++ + warm start
-1.23402 106
10
3607.12
20.58
5000
L0BnB
-1.23544 106
9
7434.60
4.95
5000
MSK persp.
-1.24017 106
10
3602.18
7.35
5000
MSK persp. + warm start
-1.24017. 1066
10
3604.42
7.35
5000
Convex MSK opt. persp.
-1.24017 10t
10
1606.64
1.03
5000
ours
-1.24017 106
10
386.72
0.00
6000
SubsetSelectCIO
-1.36967 106
10
3743.26
27.64
6000
SubsetSelectCIO + warm start
-1.43335 10
10
3802.62
21.97
6000
L0BnB
-1.44849 106
9
7430.07
4.18
6000
MSK persp.
-1.45406 106
10
3608.33
6.98
6000
MSK persp. + warm start
-1.45406 10
10
3604.65
6.98
6000
Convex MSK opt. persp.
-1.45406
106
10
1665.16
0.62
6000
ours
-1.45406 106
10
54.13
0.00
7000
SubsetSelectCIO
-1.00283 1
10
3661.63
27.20
7000
SubsetSelectC10 + warm start
-1.68426
106
10
3897.62
21.05
7000
LOBnB
-1.69016 10
8
7280.69
3.21
7000
MSK persp.
-1.70473 106
10
3603.26
6.07
7000
MSK persp. + warm start
-1.70473 106
10
3603.98
6.07
7000
Convex MSK opt. persp.
-1.70473
106
10
1677.45
0.45
7000
-1.70473 106
10
21.30
0.00
Table 149: Comparison time and optimality gap on synthetic datasets (p 3000,
10,
0.7, À2 = 10.0).
171

---

# of samples
method
upper bound
support size
time(s)
sap(%)
3000
SubsetSelectC1O
-276699 106
10
3644.99
20.66
3000
SubsetSelectCIO warm start
-2.78489 106
10
3685.28
19.62
3000
L0BnB
-2.55553 106
1
7230.61
17.51
3000
MSK persp.
-2.78334 106
10
3601.82
10.13
3000
MSK persp. + warm start
-2.78334 106
10
3600.26
10.13
3000
Convex MSK opt. persp.
-2.77971 106
10
1638.17
10.70
3000
ours
-2.8800 106
10
3600.78
9.79
4000
SubsetSelectCIO
-3.71695 106
10
3601.35
21.65
4000
SubsetSelectCIO+ + warm start
-3.74577 10
10
3638.03
20.71
4000
L0BnB
-3.45116 106
1
7514.37
15.03
4000
MSK persp.
-3.74752 106
10
3602.22
8.70
4000
MSK persp. + warm start
-3.75192. 106
10
3602.31
8.57
4000
Convex MSK opt. persp.
-3.75191 106
10
1564.01
3.01
4000
ours
-3.75192 106
10
3600.32
3.23
5000
SubsetSelectCIO
-4.57617 10t
10
3864.92
21.14
5000
SubsetSelectCIO+ + warm start
-4.60856 106
10
3764.48
20.29
5000
L0BnB
-4.23012 10
1
7233.86
13.08
5000
MSK persp.
-4.61866 10t
10
3603.72
7.81
5000
MSK persp. + warm start
-4.61867 106
10
3603.77
7.90
5000
Convex MSK opt. persp.
4.61867 106
10
1492.40
1.19
5000
ours
4.61867 10
10
3600.30
0.89
6000
SubsetSelectC10
-5.1166 106
10
3718.93
22.96
6000
SubsetSelectCIOO+ + warm start
-5.22880 10
10
3662.21
20.47
6000
LOBnB
-4.97660 8
1
7210.21
12.38
6000
MSK persp.
-5.43816 106
10
3606.21
7.09
6000
MSK persp. + warm start
-5.43816 10
10
3602.03
7.09
6000
Convex MSK opt. persp.
-5.43816 106
10
1527.55
0.64
6000
ours
-5.43816 106
10
3600.22
0.32
7000
SubsetSelectC10
6.30345 10
10
3670.08
20.83
7000
SubsetSelectCIO + warm start
6.34815
106
10
3714.08
19.98
7000
LOBnB
-5.82238 106
1
7236.55
11.40
7000
MSK persp.
3604.74
7000
MSK persp. + warm start
6.36757 106
10
3607.94
6.15
7000
Convex MSK opt. persp.
6.36757 10s
10
1747.23
0.45
7000
ours
6.36757 106
10
3600.06
0.10
Table 150: Comparison time and optimality gap on the synthetic datasets (p 3000, K
10,
0.9, A2 = 10.0).
172

---

G.5
Experiments on the Case >> n
# of samples
method
upper bound
support size
time(s)
gap(%)
100
SOS1 + warm start
-2.76902 10
10
3600.94
13.02
100
big-M50 + start
-2.76902 100
10
3600.58
13.02
100
big-M20 + warm start
-2.76902 10
10
3600.47
13.02
100
big-M5 + warm start
-2.76902 10a
10
3600.17
13.02
100
persp. warm start
-2.76902 10
10
3602.22
13.02
100
eig. persp. warm start
-2.76902 10a
10
3600.65
13.02
100
Convex opt. persp. + warm start
-2.51480 -103
10
1706.56
24.44
100
MSK persp. warm start
-2.71927 -00
10
3599.92
15.08
100
SubsetSelectCIOO+ warm start
-1.68157 -103
10
4350.20
86.11
100
LOBnB ours
7.78758 102
7217.66
255.64
100
2.76902 10
10
3600.22
13.02
200
SOS1 + warm start
-4.15945 103
10
3600.08
17.20
200
big-M50 + warm start
-4.15945 10a
10
3600.05
17.20
200
big-M20 + warm start
- 10S
10
3600.11
17.20
200
big-M5 warm start
-4.15945 10
10
3600.22
17.20
200
persp. warm start
4.15945 10
3600.20
17.20
200
eig. persp. warm start
4.15945 103
10
3600.24
17.20
200
Convex opt. persp. warm start
-411110- 1
10
1800.00
18.57
200
MSK persp. + warm start
-..15945 10
3599.97
17.20
200
SubsetSelectCIO warm start
-2.7436 103
10
3819.10
77.69
200
LOBnB ours
-1.20299 10
7208.65
240.10
200
4.15945 10
10
3601.81
17.20
500
SOSI + warm start
- 1.00615 104
10
3600.51
20.46
500
big-M50 + warm start
1.00615 10aH
10
3600.28
20.46
500
big-M20 + start
-1.06155 10
10
3600.07
20.46
500
big-M5 warm start
-1.00615 10a
10
3600.11
20.46
500
persp. + warm start
-1.00615 10S
10
3600.37
20.46
500
eig. persp. warm start
-1.00615 10a
10
3600.10
20.46
500
Convex opt. persp. warm start
-
-1.00115 10a
10
1744.05
20.46
500
MSK persp. + warm start
-1.00115 10
10
3600.21
20.46
500
SubsetSelectCI0+ warm start
2.29
500
LOBnB
-1.00615 10
10
1834.75
0.00
500
ours
-1.00615 10a
3600.23
20.46
Table 151: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10,
p 0.1, A2 0.001). All baselines use our beam search solution as a warm start whenever possible. The results in the case of >>> n show that OKRidge is competitive with other MIP methods. 10bnb can solve the problem when case of 100 and n 200. MOSEK sometimes gives the best optimality gap. However, as have shown in the previous subsections, MOSEK not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and 10bnb are not competitive in discovering differential equations (datasets have n 2 p), which is the main focus of this paper. In the future, we plan to extend OKRidge to develop more scalable algorithm for the case >>
500 but has difficulty in finding the desired sparsity level in the
173

---

# of samples
method
upper bound
support size
time(s)
gap(%)
100
SOS1 + warm start
2276622 10
10
3601.59
13.09
100
big-M50 + warm start
-2.76721 10x
10
3600.66
13.09
100
big-M20 + warm start
-2.76721 10a
10
3600.32
13.09
100
big-M5 warm start
-2.76721 100
3602.22
13.09
100
persp. warm start
-2.76711 100
10
3600.09
13.09
100
cig. persp. warm start
-2.76721 100
10
3600.03
13.09
100
Convex opt. persp. warm start
-2.51378 103
2128.32
24.44
100
MSK persp. warm start
-2.71018 103
10
3599.92
15.36
100
SubsetSelectCIO+ warm start
-7.59815 10
10
3651.10
311.88
100
L0BnB
-7.78035 102
7218.43
255.36
100
-2.76721 103
10
3600.35
13.08
200
SOSI + warm start
4.15848 10s
10
3600.46
17.23
200
big-M50 + start
4.158488 100
10
3600.42
17.23
200
big-M20 + warm start
4.15848 103
10
3600.30
17.23
200
big-M5 + warm start
4.15848 10S
10
3600.10
17.23
200
persp.
start
4.15848 103
10
3600.12
17.23
200
eig. persp. warm start
4.15848 103
10
3600.47
17.23
200
Convex opt. persp. + warm start
4.11040 103
10
2159.71
18.55
200
MSK persp. start
4.15848 103
10
3599.93
17.11
200
SubsetSelectCIO warm start
33.1406 103
10
3632.89
55.23
200
LOBnB ours
-1.20245 103
7208.71
237.38
200
-4.15848 10
10
3600.13
17.22
500
SOS1 + warm start
11.0605 10*
10
3600.08
20.47
500
big-M50 + warm start
1.00605 104
10
3600.08
20.47
500
big-M20 + warm start
-1.0005 10t
10
3600.80
20.47
500
big-M5 + warm start
-1.00605 10a
10
3600.52
20.47
500
persp. warm start
-1.00605 10a
3600.04
20.47
500
eig. persp. warm start
-1.00055 10a
10
3600.06
20.47
500
Convex opt. persp. warm start
-1.00605 10a
10
2059.28
20.41
500
MSK persp. + warm start
-1.00605 10a
3600.08
20.33
500
SubsetSelectCI0 warm start
-7.47048 103
10
3748.65
62.24
500
LOBnB ours
-1.00605
10
10
1835.06
0.00
500
-1.00605
104
10
3601.58
20.46
Table 152: Comparison of time and optimality gap on the synthetic datasets (p 3000, k 10,
warm start whenever possible. The results in the case of >> n show that OKRidge is competitive with other MIP methods. 10bnb can solve the problem when
p 0.1, A2 0.1). All baselines use our beam search solution
500 but has difficulty in finding the desired sparsity level in the case of n 100 and n 200. MOSEK sometimes gives the best optimality gap. However, as we have shown in the previous subsections, MOSEK not scalable high dimensions. Additionally, the results in differential equations show that MOSEK and 10bnb are not competitive in discovering differential equations (datasets have 2 p), which the main focus of this paper. the future, we plan to extend OK Ridge to develop more scalable algorithm for the case >>>
174

---

of samples
method
upper bound
support size
time(s)
gap
100
SOSI + warm start
22.11800
10
10
3600.36
19.31
100
big-M50 + warm start
22.61800 10a
10
3600.18
19.31
100
big-M20 + warm start
-2.61800 10
10
3600.40
19.31
100
big-M5 warm start
-2.61800 103
10
3602.50
19.31
100
persp. + warm start
-2.61800 103
10
3600.47
19.54
100
cig. persp. warm start
-2.61800 10S
10
3600.19
19.54
100
Convex opt. persp. warm start
-2.40846 103
10
1968.34
25.24
100
MSK persp. + warm start
-2.61800 103
10
3599.93
10.09
100
SubsetSelectCIO++ warm start
-1.79729 10
10
3646.28
74.12
100
LOBnB
-7.11936
102
7212.43
236.77
100
ours
22.1806 103
10
3600.78
19.09
200
SOSI warm start
4.06441 103
10
3600.07
19.69
200
big-M50 + warm start
4.06441 103
10
3600.92
19.69
200
big-M20- + warm start
4.06441 10a
10
3600.61
19.69
200
big-M5 warm start
-4.0441 103
10
3600.07
19.69
200
persp. + warm start
4.06441 103
10
3600.03
11.80
200
eig. persp. warm start
4.06441 10a
3600.79
11.80
200
Convex opt. persp. warm start
-4.01212 10
10
2175.61
16.91
200
MSK persp. warm start
-4.06411 10S
10
3599.92
9.05
200
SubsetSelectC1O warm start
-2.8229 10
3704.01
81.71
200
LOBnB ours
1.15079 103
7207.23 .25
219.55
200
4.06441 10s
10
3600.92
19.45
500
SOS1 warm start
9.96316 103
10
3601.58
21.39
500
big-M50 + warm start
-9.9636 10
10
3602.06
21.39
500
big-M20 + warm start
-9.9636 10
3602.58
21.39
500
big-M5
start
-9.96316 103
10
3600.48
21.28
500
persp. + warm start
-9.96316 10s
10
3600.14 14
21.65
500
eig. persp. + warm start
-9.9336 10x
10
3600.12
13.16
500
Convex opt. persp. warm start
-9.9336 10
10
2145.98
16.65
500
MSK persp. warm start
9..6336 103
10
3600.03
11.45
500
SubsetSelcctCIO+ warm start
7.48575 1
10
3793.20
61.91
500
LOBnB ours
9996616
10
10
1483.07
0.00
500
-9.96316
103
10
3600.96
21.14
Table 153: Comparison time and optimality gap on the synthetic datasets (p 3000,
10.
0.1, A2 10.0). All baselines use our beam search solution as warm start whenever possible. competitive with other MIP methods. When the number of samples is 100, OKRidge is able to find slightly better solution through the branch-and- bound process, but other solvers are stuck with the warm start solution. 10bnb can solve the problem when
The results in the case of p >> n show that OKRidge
500 but has difficulty in finding the desired sparsity level in the case of 100 and 200. MOSEK sometimes gives the best optimality gap. However, as have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and 10bnb are not competitive in discovering differential equations (datasets have 2 p), which is the main focus of this paper. In the future, we plan to extend OKRidge to develop more scalable algorithm for the case p >> n..
175

---

G.6 Comparison between Fast Solve and ADMM
In this subsection, we provide the empirical evidence that the ADMM method is more effective than the fast solve method in terms of the computing tighter lower bound, which helps speed up the certification process.
(p=0.1)
(p=0.5)
(p=0.9)
10a
time (occ)
10-
10a
10a
10a
10a
10-
109
109
100
500
1000
3000
5000
100
500
1300
3000
5000
100
500
1000
3300
5000
0.2
0.7
0.2
0.1
0.0
0.0
0.0
1000
3000
1000
3000
5000
3000
5000
features
features
# features
(fast salve only)
(fast solve ADMM)
Figure 7: Comparison of running time (top row) and optimality gap (bottom row) between using fast solve (Equation (16)) alone and using fast solve and ADMM (Equations (19)- 22 to calculate the lower bounds in the BnB tree with three correlation levels p 0.1, 0.5, 0.9 (n 10000, k 10).
(p-0.1)
(p-0.5)
(0-0.9)
time (sec)
20?
102
10a
101
10
10-
5000
6000
7000
6000
7000
15
15.
15
8.
5-
5-
5-
7000
7000
3000
# of samples
# of samples
# of samples
ours (fast solve only)
(fast ARMM)
Figure 8: Comparison of running time (top row) and optimality gap (bottom row) between using fast solve (Equation (16) alone and using fast solve and ADMM (Equations (19)- (22) to calculate the lower bounds in the BnB tree with three correlation levels 0.1, 0.5, 0.9 (p 3000, k 10).
176

---

H More Experimental Results on Dynamical Systems
H.1 Results Summarization
Comparison with MIOSR We first show a direct comparison between OKRidge and MIOSR (based on SOSI formulation). The results are in Appendix HH.2 where show OKRidge outperforms MIOSR in terms of both solution quality and running time.
Comparison with Different MIP Formualtions Next, we compare OKRidge with other MIP formulations (SOSI, big-M, perspective, and cigen-perspective) as done on the synthetic benchmarks. These formulations have never been applied the differential equation experiments. Similar the synthetic benchmarks, we show that OKRidge obtain better solution quality and smaller running time. The results are shown in Appendix H.3
Comparison between Beam Search and More Heuristics In addition, we also compare our beam search method with many more heuristic methods from the sparse regression literature. Beam search outperforms all methods in terms of solution qualities in presence of highly correlated features. Block Decomposition algorithm also achieves high-quality solutions, but is much slower than beam search. The results are shown in Appendix H.4
Comparison with Different MIP Formulations Warm Started by Beam Search Due to the superiority of beam search, we give beam search solutions as warm starts Gurobi with different MIP formulations and repeat the experiments. Our warm start improve the solution quality and running time, but OKRidge still outperforms other MIP formulations in terms of running time. The results are shown in Appendix H.5
Comparison with Other MIP Solvers and Formulations Lastly, we solve the same problem using the MOSEK solver, 10bnb, and SubsetSelectionCIO. For this application, 10bnb has a time limit of 10 seconds for each Lu regularization and 60 seconds for the total time limit. All other algorithms have total time limit of seconds as mentioned in the experimental setup subsection. Although they are not competitive Gurobi experiments for the purpose of thorough comparison. Results Appendix H.6 show that OKRidge outperforms these MIP solvers/formulations.
have shown on the synthetic benchmarks, still conduct the
177

---

H.2 Direct Comparison of OKRidge with MIOSR
In the appendix, we directly compare MIOSR (based on the SOSI formulation) and our method on the differential equation experiments. The results are shown in Figure
Although MIOSR is faster than OKRidge the Lorenz system, OKRidge is able find better solutions. See Appendix H.. and Figure T0 where the eigen-perspective formulation doesn't have this problem and can produce high-quality solutions.
the Hopf system, MIOSR faster than OKRidge. However, the number features is small (p 21), and the running time is negligible compared to Lorenz and MHD systems.
On the MHD system (p 462), OKRidge outperforms MIOSR terms of both solution qualities and running times. This shows that OKRidge is much more scalable to high dimensions than MIOSR.
Lorenz
Hopf
MHD
Coofficion Bror Thie Posiivity Rate
1.00
1.00
100
0.75
0.50
0.0
0.50
0.5
0.25
0.25
10-1
10a
10"
10
10"4
10-1
10-2
Thain RMSE
10
6x
Text RMSE
10-4
10-
10-2
101
4030
1000
3000
Rurning
10-
2000
1000
10"
10a
10a
10a
10"
10'
Training Trajectory Length
Training Trajectory Length (s)
Training Tha Length
MICSR
Figure 9: Comparison between our method and MIOSR on the experiments of discovering sparse differential equations. We obtain better performance the true positivity rate, La coefficient error, and test RMSE.
178

---

H.3
Comparison of OKRidge with SOS1, Big-M, Perspective, and Eigen-Perspective
Formulations
Besides the MIOSR baseline (which is based on the SOSI formualtion), we also compare with other MIP formulations, including Big-M (M
50), perspective, and sgeen-erspeectiee formulations. The results are shown in Figure TO The big-M method produces errors on the Lorenz system, so its result not shown the left column. We have discussed about the SOS1 formulation in the last subsection. On the Lorenz system, OKRidge is faster than the perspective and digen-perspective formulations On the Hopf system (p = 21), OKRidge is slower than Gurobi, but OKRidge takes only 20 seconds. On the high-dimensional MHD system, OKRidge is significantly faster than all MIP formulations.
Lorenz
Hopf
MHD
Ls Coeffiiient Errer thue rosavity nate
1:00
1.00
1.00
0.75
0.75
0.75
0.25
0.00
0.00
0.00
10"
10a
10
10r1
10-3
1077
10"1
10
20
10 7-
10
Thair RIASE
301
102
Test AMSE
100
10"1
102
10-2
101
2000
Running Time
4000
1500
500
1000
10a
101
16P
10t
107
107
Training extory Lengtii
Training Trejectory Length
Training Trajectury Lenguh
S0S1 bigM
persp
eig. persp.
ours
Figure 10: Comparison between our method and other MIP formulations solved by Gurobi on the experiments discovering sparse differential equations. We obtain better performance on the true positivity rate, L2 coefficient error, and test RMSE. SOSI is faster than ours the Lorenz system, but the solution quality worse. OKRidge is slower than Gurobi on the Hopf system (p=21), but the time difference is not significant. On the eigh-dimensional MHD system, OKRidge is significantly faster than Gurobi with all exisiting MIP formulations.
179

---

H.4 Comparison of Our Beam Search with More Heuristic Baselines
Next, we compare our proposed beam search method with more heuristic methods from the sparse regression literature. The results are shown in Figure [T] Since many heuristic method implementa- tions do not consider the ridge regression, we set & regularization for all algorithms fair comparison. Since the features are highly correlated, most of the existing heuristic methods do not produce high-quality solutions except the Block Decomposition algorithm. However, our beam search method is significantly faster than the Block Decomposition algorithm.
for
Moreover, for the purpose of completeness, we also compare with LASSO, the classical sparse learning method. Other researchers have found that LASSO performs poorly in this context. We confirm this claim experimentally. The result is shown in Figure 12
Lorenz
Hopf
MHD
1.00
0.75
0.75
0.75
0.25
0.00
0.00
101
Oxeffiiiet
10-1
101
10
10.7
10
10
102
Train RMSC
10"
201
1072
101
107
Test RMsE
10
10-3
162
10
10-2
165
10"2
Punning Time
150-
600
100-
400
107
10
10"
10
10"
10'
Training Trajectory Length
Training Trajectory Length
Traning Trajectory Length
gp cosamp
ssp
proximal gradlent 10c
romp
BlockDecAlg ours (beam search)
omp
apmr
GreedySpprsseiimplx
Figure 11: Comparison between method and other heuristic methods the experiments of discovering sparse differential equations. We obtain better performance on true positivity rate, L2 coefficient error, and test RMSE. The Block Decomposition algorithm also produces high-quality solutions but is significantly slower than our beam search.
180

---

Lorenz
Hopf
MHD
La Coafficent Errer True Positivity Rete
1.-
102
10
66-
0.6
0.6
62-
02-
0.2
10
10"4
10
10-1
10-3
10r2
101
Train RMSE
10
10"
10
10-1-
10
Test RMSE
10-2
101
10
302
10A
Training Trajectory Length
Trainin Trajectory Length (S)
Training Trajectory Length (5)
only)
+ ours
Figure 12: Results on discovering sparse differential equations. Lasso does not do well in this application. Beam search lags behind OKRidge on the MHD system. This shows that beam search alone is not sufficient, and it is necessary to use BnB to obtain the optimal solution.
181

---

H.5
Comparison of OKRidge with MIPs Warmstarted by Our Beam Search
natural question ask is whether Gurobi would be faster than OKRidge in terms of running speed if we give the high-qualit, solutions from last subsection as warm starts to Gurobi Here, we compare with other formulations warmstarted by our beam search solutions. The results are shown in Figure 13 By comparing Figure T0 with Figure 13 we see that our beam search solutions, if used as warm starts, improve Gurobi's speed. However, OKRidge is still significantly faster than all MIP formulations on the high-dimensional MHD system.
Lorenz
Hopf
MHD
C. Chafffie Coafficnnt eiter True Pestivty 8228
0.75
0.75
0.75
0.00
0.5
0.0
0.00
0.00
107
10a
10-34
30
10"
3x
Thain RMSS
10
167
4x.00
10
100
Taxt (MMS
10
10-2-
10"4
107
Ruaving time
1900
1000
2000
1000
10
10'
10f
Traniin Trajectory Longth (s)
The ning Trajectory Longth (s)
Tranning Traje Trajettoyy Langth (5)
bigM
persp.
persp.
Figure 13: Comparison between our method and other MIP formulations solved by Gurobi, which was warmstarted by our beam search solutions the experiments of discovering sparse differential equations. OKRidge slower than Gurobi on the Hopf system (p = 21), the running time only 20 seconds. On the Lorenz and MHD systems, OKRidge is faster than other MIP formulations, especially on the high-dimensional MHD system (p = 462).
182

---

H.6 Comparison of OKRidge with MOSEK solver, SubsetSSelctionCII, and LOBNB
Lastly, we compare with other MIP solvers/formulatinn,, including the MOSEK solver, SubsetSelec- tionCIO, and 10bnb. The results are shown in Figure 14
MOSEK (perspective formualtion) produces good solutions the Hopf dataset, but could not produce optimal solutions on the Lorenz system. MOSEK (optimal formulation) produes suboptimal solutions the Lorenz and Hopf systems. However, convex SDP problem. This demonstrates why we need to tackle the NP hard problem instead of the relaxed convex problem. MOSEK runs of memory on the MHD dataset, so don't show the results on the MHD dataset.
are using MOSEK to solve the relaxed
SubsetSellctionCIO doesn't 't do well the Lorenz and MHD systems. On many instances, SubsetSe- lectionCIO would have errors.
10bnb performs poorly on these datasets. One major reason is that 10bnb could not retrieve the exact support size (k: 1, 2,2,,, 3, 4, 5) because the sparsity is controlled implicitly through &n regularization. It takes long time to run these differential experiments. 10bnb couldn't 't find good lo regularization which produces the right sparsity level within the time limit.
Please see Appendix G. for detailed discussion of the drawbacks on using MOSEK, SubsetSelec- tionCIO, and 10bnb.
Lorenz
Hopf
MHD
G Coeffriiet Eirer Thue rostiity nate
1,00
1.00
1.00
0.75
0.75
0.75
0.30
0.25
8.00
0.00
0.00
109
30*
109
10"1
20
10-42
10a2
10-1
Train RMSE
102
304
30'
10a
Too AMSE
10-1
107
301
30-2
Bunninn Time
3000
2000
500
2000
30'
10
10a
10:
10'
10*
Training ectory Length
Training Trajectory Length
Training Trajectory Lengch
persp.
subsetselectCIO
MSK persp. start
subsetselertChi eeettie
Convex MSK opt persp.
Figure 14: Comparison between our method and other MIP solvers or formulations on the experiments of discovering sparse differential equations. We obtain better performance the true positivity rate, La coefficient error, and test RMSE.
183


Reviews:
[{"review_id": "6CvwqJRZRN", "reviewer": "Reviewer_fUoX", "summary": "The authors propose a new framework for computing the k-sparse ridge regression. Specifically, the framework considers the Lagrangian dual and relaxes the lower bound to an ADMM problem with an isotonic proximal operator, which has a linear time solution. Combined with beam search and branch-and-bound, the authors demonstrate the framework's effectiveness in the experiment. Further, they claim to prove a tighter approximation bound based on the beam-search result.", "questions": "1. The proposed method is an algorithm for general sparse ridge regression with experiments on the dynamic system. There is no other utilization of the property of the dynamic system here other than the data itself. So I don't think the title is appropriate.\n2. Appendix L562: missing z.\n3. Appendix L567: The page overflows, so part of the equations are hidden.\n4. Appendix L588: The conclusion (Therefore $\\gamma=$) is your assumption.", "limitations": "NA", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "The paper's main contribution is the relation between the k-sparse problem with an isotonic proximal operator, which can be computed in linear time ($O(n log k)$ for sorting the coefficients.) I believe this technique is new and may have a significant impact on other works with k-sparse constraints. Actually, the framework can be generalized to any strongly convex and Lipschitz C^1 optimization with k-sparse constraints. The proposed Lagrangian dual lower-bound and isolation of the binary variables only depends on the k-sparse constraints, not the ridge regression itself. In addition, the proofs are clearly written with annotations on every inequality, which helps a lot in verification.", "weaknesses": "Despite the proofs being clearly annotated, there are still major bugs in the main theorems. However, I think these bugs may be corrected by weakening the theoretical results, and these errors may not impact the practical value of the work. Specifically, the $L_{\\text{ridge-}\\lambda}^\\text{saddle}$ problem should be a lower bound of equation (12), not an equivalent problem. To prove equivalence, we need to prove that the set of optimality conditions are the same, and the error can be easily checked by taking stationary condition on c in eq (10), which leads to the missing slackness condition $2\\beta_j=c_jz_j$ which doesn't appear in the stationary condition of the new formulation in eq (12) unless other assumptions are made. And I think the subtle error in the derivation is that the proof of Theorem 3.1 is not rigorous, i.e., not every step in the L569 derivation is an equivalence relation. The equation (40) may not be one-to-one, and the step of taking $\\beta=\\gamma$ with additional constraint eq (40), reparameterization of $c$, and moving $\\gamma$ from the minimum operator to the maximum operator may not be rigorous. There may be a way to prove it more rigorously, and in the worst case, the derivation becomes a relaxation, which is still helpful. However, we will need to see the more formal derivation first.\n\nSecond, there are also bugs in the authors' proof for the beam search in Theorem 3.4. In Section 3.2, the authors select the new support by the Gauss-Southwell rule of largest improvement. However, when combined with beam search, it is required to select the candidates with the lowest objective value, which is a conflict. As we see in Algorithm 1 and Algorithm 7 in the Appendix, the actual implementation generates the next step using the B largest improvements for each candidate and chooses the B best candidates with the best objective value, which differs from the proof in Theorem 3.4. On the other hand, the theorem and proofs on the beam search solution are independent to the number of candidates $B$ in the authors' proof, so there must be some error or hidden assumption (e.g., $B\\geq p^k$) here.\n\nHowever, I still consider this draft as a good paper and recommend it for acceptance. Just that these bugs need to be addressed, or the Theorems need to be weakened. However, please address these main issues in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Theorem 3.1}$\nWe have answered this question in the general response.\nWe give an alternative proof for the equivalence of the optimal values between Problem (10) and Problem (12) by showing the two-way inequalities.\nLet us know if you have further questions on Theorem 3.1.\n\n$\\textbf{Beam Search Description}$\nYou are right that we first propose to select the support by the Gauss-Southwell rule of largest improvement.\nHowever, if you keep reading to the next paragraph (Line 167-170), we pointed out the limitation of this approach and introduced beam search.\nWe think the confusion arose due to the absence of a mathematical formula after Line 170, which might have led readers to focus solely on Equation (25).\nFor revision, maybe we can insert after Line 170 an extra sentence and another formula like the one below:\n\nFor each solution $\\beta$ from the previous stage of support expansion, we select $B$ new features as potential support expansion candidates \n\n\\begin{align*}\n    j^* \\in \\arg \\text{BottomB}_j   \\left( \\min\\_\\alpha  L\\_{\\text{ridge}} ( \\beta + \\alpha e_j ) \\right),\n\\end{align*}\n\nwhere $j^* \\in \\arg \\text{BottomB}_j$ means $j^*$ belongs to the set of solutions whose loss is one of the B smallest losses.\nAfterwards, we finetune the solution on the newly expanded support and choose the top B solutions for the next stage of support expansion.\n\n$\\textbf{Theorem 3.4}$\nWe apologize for the confusion.\nTheorem 3.4 indeed provides a bound for the beam search solution.\nWe think the confusion was caused by the sentence inside the parentheses on Line 187 (\"starting from 0 and adding a variable k times to get support k\").\nWe were trying to describe beam search in a simplified way.\nHowever, you are right in that this description does not capture the full picture of beam search.\nAdding one variable at a time is part of the beam search algorithm, but we also need to select the best solution after each stage of support expansion.\nIt is possible that the best solutions of two neighboring stages could have supports that differ by 2 features instead of 1 feature.\nTherefore, the simplified description inside the parentheses is not precise.\nWe will delete this sentence during revision.\n\nThis has no impact on the correctness of the theorem for the beam search algorithm.\nTo show you that we are indeed proving for the beam search algorithm, please look at Line 661-664 and Equation (52) in Appendix 52.\nThe crucial argument in that proof is that the decrease of loss between best solutions of two successive stages (stage $t$ and stage $t+1$) of beam search is greater than that of minimizing a single coordinate on the best solution of the early stage (stage $t$).\n\nWith regards to the question on beam size, you are right that our bound does not involve the beam size.\nWe spent lots of time trying to incorporate the beam size into the formula but made no breakthrough.\nAlthough the beam size is not part of the theoretical bound, we do observe empirically that larger beam size leads to better solution quality.\nThe reason we are able to achieve a tighter theoretical bound than [29] is largely due to step 2 (Line 672-684) in our proof, where we show we can actually bound the term using $M_1$ instead of $M_{2k}$.\nTo summarize, the bound in our theorem has potential for improvement by incorporating the beam size $B$.\nHowever, at this point, we are unsure how to achieve this enhancement (and it remains an open question whether it is feasible to incorporate the beam size into the bound).\n\n$\\textbf{Title Change}$\nWe will listen to your suggestion and remove from the title \"for Learning Dynamical Systems\".\nWe still plan to use applications in learning dynamical systems as a strong motivation for this paper and also use the experimental results to demonstrate the effectiveness and potential scientific impact of our method.\nThe reason, as we explained in the introduction, is that (Line 23) selection of the correct features is key to discovering the true dynamics.\nEmpirically, due to the highly-correlated features formed by high-order polynomials, most existing heuristic methods failed to find the correct features (see Figure 9 on page 177 of the Appendix).\nTherefore, finding optimal sparsity-constrained solution is really needed in this context.\nFurthermore, sparse ridge regression is used extensively in the SINDy literature, making it a solid choice for application.\n\n$\\textbf{Appendix L562}$ Thank you, we will fix this typo: $(\\beta_j c_j - \\frac{c_j^2}{4}) \\rightarrow (\\beta_j c_j - \\frac{c_j^2}{4}z_j)$\n\n$\\textbf{Appendix L567}$ Thank you, we will make this more clear by rewriting \"reparameterize $c$ with $\\gamma$\" $\\rightarrow$ \"reparameterize $c$ with $\\gamma$ by having $c = \\frac{2}{\\lambda_2 + \\lambda} [X^T y - (X^T X - \\lambda I)\\gamma]$\".\n\n$\\textbf{Appendix 588}$ Thank you, this is an accidental writing error. We will fix this by rewriting \"we have $\\hat{\\gamma} = \\arg\\min_{\\gamma} L_{\\text{ridge}}(\\gamma)$\" $\\rightarrow$ \"we have $X^T y - Q_{\\lambda} \\hat{\\gamma} = (\\lambda_2 + \\lambda) \\hat{\\gamma}$ when $\\hat{\\gamma} = \\arg\\min_{\\gamma} L_{\\text{ridge}}(\\gamma)$\"."}, {"review_id": "eT5BjGiJQ6", "reviewer": "Reviewer_MGha", "summary": "In this manuscript the authors consider the ridge regression with a hard sparsity constraints. Since the problem is posed in the fixed design setting, the paper is on the optimization theory, while the main motivation for its study authors find in the applications related to data-driven methods for dynamical systems. Using the work of Atamturk and Gomez 2020 on max-min relaxation of the original problem, novel lower bounds are developed (one in closed form and the other using ADMM to improve the tightness). These results coupled with Beam-Search of Wiseman and Rush 2016 are used within Bound-and Branch framework to propose an algorithm with a theoretical bound on the optimality gap. Experiments are run on synthetic data to compare with SOTA methods for the studied problem, as well as on datasets from Brunton, Proctor and Kutz 2016 on toy problems in dynamical systems to showcase the application in the sparse identification of (continuous-time deterministic) equations of motion from data under the (strong) assumptions that the finite-dimensional basis of a function space is know and that dataset contains states and their time derivatives.", "questions": "Please elaborate on the claim in the line 57/58. What is the meaning of outperforms? Seams that on the validation set it performs similarly as MIOSR.", "limitations": "Apart of currently discussed limitations, if the title is kept as is, the authors need more discussion as noted above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written, apart of issues stated below. All the important concepts are properly introduced in the main body and extensively elaborated upon in the appendix. Main contribution well presented and the novelty of the new ideas w.r.t. existing work is made very clear. Up to my knowledge, relevant references are cited. The theoretical results are correct, except minor issues explained below. Numerical experiments are very detailed. ", "weaknesses": "In the following I give the main weaknesses of this paper:\n\n __(A) Title vs content:__ The paper is content wise purely work on optimization. Namely, the link to ML and to learning of dynamical systems is only through the problem which is an empirical risk minimization for linear model with $\\ell_2$ regularization and $\\ell_0$ constraints. Hence, there is no really any learning of dynamical systems as the title suggests. Therefore, since I find that the paper is in its current form of interest to ML community, I propose the authors to remove from the title “for Learning Dynamical Systems”.  If not, then much more than elaborated discussion needs to be done to justify the title. In particular:\n\n1. Assumptions under which the system is learned from data providing explaining when they are met in practice. My opinion is that relying just on SINDy approach is very limited since this aspect is quite vague.\n\n2. Comparison with other SOTA methods to learn dynamical systems from data is needed (meaning those solving the true problem and not an empirical one) with the discussion on benefits and limitations of the current proposal. Also some insights in how statistical learning theory of the proposed method might be developed. \n\n3. At least one experiment on the performance of the method on the real dataset of challenging dynamical system (unknown non-polynomial nonlinearities in ODE). \n\n\n__(B) Error in Thm. 3.1.:__ When $Q_\\lambda$ is singular, i.e. when $\\lambda = \\lambda_{\\min}(X^\\top X)$, not every $c$ is in the range of $X^\\top$, and, hence, reparametrization doesn’t hold, yielding that (12) is not equivalent to (10). Consequently, when $\\sigma_{\\min}(X)=0$, e.g. when number of features exceeds number of samples,  Thm. 3.1. on which all else is based doesn’t hold. This aspect needs to be checked and clarified. \n\n__(C) Notion of optimality:__ In the paper, starting from the title and abstract, the word _optimal_ is used. I believe that many of such uses are not entirely clear. To me it seams that sometimes it means just feasible k-sparse solution. Although upper bound and lower bound guarantees are shown, in the paper I do not find theoretical guarantee for the convergence of the proposed method to the optimal solution stated. Can the authors, please, elaborate on this? \n\n\nTo conclude, the current evaluation score I have given reflect the aforementioned issues. Depending on the authors reply, I am willing to reconsider it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Review Correction}$ There is one minor correction we want to point out with regards to your summary section.\nIn the differential equation experiments, only states are given in the dataset, not the time derivatives.\nDerivatives were approximated using a smoothing finite difference with a window size of 9 points.\nThe derivatives were calculated automatically by the PySINDy library.\nYou can find this information in Appendix F \"More Experimental Setup Details\".\n\n$\\textbf{Title vs content}$\nWe will listen to your suggestion and remove from the title \"for Learning Dynamical Systems\".\nWe still plan to use applications in learning dynamical systems as a strong motivation for this paper and also use the experimental results to demonstrate the effectiveness and potential scientific impact of our method.\nThe reason, as we explained in the introduction, is that (Line 23) selection of the correct features is key to discovering the true dynamics.\nEmpirically, due to the highly-correlated features formed by high-order polynomials, most existing heuristic methods failed to find the correct features (see Figure 9 on page 177 of the Appendix).\nTherefore, finding optimal sparsity-constrained solution is really needed in this context.\nFurthermore, sparse ridge regression is used extensively in the SINDy literature, making it a solid choice for application.\n\nWe agree with you that some discussion should be given upon the assumption of the SINDy approach.\nDuring the revision, we can add that the SINDy approach assumes that the true dynamics exist in the finite-dimensional candidate functions/features.\nAs the reviewer commented, our work focuses on the optimization side of sparse ridge regression.\nOne benefit of OKRidge is that it can be used to push the limit of the SINDy approach.\nComparison with methods that do not go under the SINDy approach (sparse linear regression) does not align with our central focus and requires a complete new work in a different direction.\n\n$\\textbf{Theorem 3.1}$\nPlease look at our answer in the general response.\nWe believe even when $Q_{\\lambda}$ is singular, Problem 10 and Problem 12 are still equivalent in the sense that they have the same optimal values.\nThe insight is that the optimal solution $c$ for Problem 10 must be reparametrized by some $\\beta$ with Equation (11) $c = \\frac{2}{\\lambda_2 + \\lambda} (X^T y - Q_{\\lambda} \\beta)$.\nPlease read our proof in the general response and see if you agree with us.\nIf the proof looks ok, then we will insert this lemma into our main paper during revision.\n\n$\\textbf{Notion of optimality}$\nWe apologize for the potential confusion. There is a difference between 0 optimality gap in a mathematical sense and in a more practical and engineering sense.\nStrictly speaking, your second interpretation is correct.\nSince the problem is nonconvex and NP-hard, the solution is only certifiably optimal when the upper bound equals to the lower bound.\nIn practice, however, commercial solvers will terminate the algorithm once the optimality gap is within some tolerance level (like the one we used $10^{-4}$ in all experiments for fair comparison; this is also the default value in Gurobi).\nWe loosely interpret such solutions as optimal solutions when a very small tolerance level is used.\nAnother case to consider is that when the problem is too hard, the optimality gap may not go below the tolerance level within a reasonable amount of time.\nIn this case, we have to stop the algorithm after reaching some pre-defined time limit.\nThe final solution obtained in this scenario can only be called a feasible solution but we should not call this solution optimal.\nMaybe the reviewer has the second scenario in mind.\nWe will revise the manuscript to give more background on practical MIP terminology and stopping criteria in the appendix to clarify the notion of optimality and stopping rules.\nOur contribution is that we have developed a customized branch-and-bound framework that either certifies the optimal solution much faster or provides a much smaller or comparable optimality gap.\nThese two contributions (faster certification and smaller optimality gap) are the standard judging criteria of an algorithm in the MIP literature.\n\n$\\textbf{Convergence of BnB}$\nSince the problem is NP-hard, we cannot talk about convergence rate to the optimal solution in the same way as in the context of convex optimization, such as $O(1/k)$ or $O(1/k^2)$ convergence rate.\nThe speed of the algorithm depends heavily on how efficiently we find the upper bound and lower bound.\nHowever, it is not hard to see that the generic branch-and-bound framework will terminate in a finite number of nodes (although in the worst case, the number of nodes will grow exponentially with respect to the problem size).\nA naive approach to prove this is through exhaustive enumeration.\nIn this case, we need to consider all $\\binom{p}{k}$ feature combinations to select the best support.\nTherefore, $\\binom{p}{k}$ is the maximum number of nodes we need to process.\nThe branch-and-bound framework focuses heavily on intelligent enumeration.\nIf the lower bound is tight, we can prune most of the search space, which can greatly speed up our certification process.\nComputing a tight lower bound efficiently is a major focus of our work.\n\n$\\textbf{Meaning of``outperform''}$\nIn Figure 3, if we look at the training and test losses, we agree with you sometimes the improvement over MIOSR is not significant.\nHowever, if we look at the metric of the true positive rate, then our method is doing better than MIOSR.\nTrue positive rate measures how often we recover the true dynamics (true features).\nEven though MIOSR might achieve a small training/test loss, the loss is still not optimal, which results in MIOSR not discovering the true dynamics.\nIn practice, this means OKRidge does a better job in discovering the true physics than MIOSR.\nNote also that MIOSR requires an expensive license to access whereas ours is free."}, {"review_id": "ZOM2hJuJk8", "reviewer": "Reviewer_GTWb", "summary": "The authors propose OKRidge, a fast algorithm to solve k-sparse ridge regression problems with provable optimality.\nThe proposed method is a customized branch-and-bound approach, leveraging the specific structure of the ridge regression problem in contrast to black-box solvers such as Gurobi.\nNamely, OKRidge first uses a new saddle point formulation specifically for efficiently computing lower bounds (either \"roughly\" by solving a rather simple linear system of more tightly with an ADMM scheme.\nSecond, a beam search heuristic is proposed to find a near-optimal solution, with the global BnB structure in mind. The near-optimality of such solutions is provably guaranteed.\nThird, relatively straightforward branching and queuing strategies are also proposed to \"complete the package\", building a full custom solver.\nLast, the authors benchmark OKRidge against a wide variety of other approaches, not only limited to MIP formulations. Both performance of the solutions, and compute time are evaluated on synthetic benchmarks and on sparse identification of nonlinear dynamical systems.", "questions": "I already included a few questions in the previous section.\nTo clarify, I have a few more questions:\n1) How do you estimate \\lambda_min and \\lambda_max, to determine \\rho for ADMM? Is it critical to estimate it carefully?\n2) It might be a silly question but do you have any insight on how to pick the optimality gap \\epsilon in practice? Would it make sense to pick it adaptatively?\n3) What about the scenario where you don't \"know\" k a priori but would like to find which level of sparsity offers a good trade-off? Can OKRidge be easily adapted to solve the problem for different levels and not have to rebuild the solution from scratch for every k?\n4) In the end, while learning dynamical systems is an important problem of its own, OKRidge should be useful for other problems which can be cast as a sparse ridge regression. Why so much emphasis on dynamical systems?", "limitations": "Some limitations of OKRidge are clearly identified and pinpointed towards the end of the manuscript.\nPotential negative societal impacts were not discussed, but I don't foresee any.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I think that the significance of this work lies in the careful and smart design and integration of multiple components, building into what seems to be a very efficient custom solver for a relatively broad class of problems.\nWhile each of this components, taken individually might not be ground-breaking, the method as a whole seems to build into a tool that has important significance.\nAlthough I am mostly an outsider and may very well miss some important relevant references, the authors are doing a very good job at placing their contribution in the context of a very abundant existing literature.\nThe clarity of the paper is also to be praised. The appendices, although extremely voluminous (in part due to the hundreds of tables reported from the experiments), are completely self-contained.\nThe proofs of the different theorems read well and are somewhat insightful. The Figures and algorithms summarized in Appendix E are also very helpful to bring all the pieces together.\nFor the quality, although I am not familiar with large parts of the work, I was not able to find mistakes in the proofs and derivations. To be completely fair (maybe due to the very good quality of the presentation), some derivations can be a bit tedious, but seem relatively straightforward (which is not meant to downplay the significance of the contributions in any way). ", "weaknesses": "As stated in the previous section, it could be argued that none of the contributions, taken in isolation are \"novel enough\". But I don't believe that would be a fair argument.\nOn the other hand, while the authors did a great job at presenting their method clearly and carefully, I find that it could have been helpful or interesting to include a little more comments on their results.\nFor instance, it is great that provable guarantees are derived for the heuristic solution found by their beam-search (section 3.2.1) but it could have been nice to understand a bit more how tight those bounds are (or if they are insightful at all).\nMore importantly, although the experimental results look good, do the authors consider this heuristic as an important limitation? Are there problems (or datasets) for which the sub-optimality of the beam-search is problematic?\nAlso, it would be great to understand, in practice, where the compute time is being spent as it might be a good way to understand where the computational bottlenecks of OKRidge are and where there is potential room for further improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Tightness of the bound for the beam search solution}$\nThe bound is not as tight as one might desire because the bound does not incorporate the beam size.\nWe spent lots of time on improving the bound but made no breakthrough.\nTherefore, the bound in our theorem has potential for improvement by incorporating the beam size $B$.\nHowever, at this point, we are unsure how to achieve this enhancement (and it remains an open question whether it is feasible to incorporate the beam size into the bound).\n\n$\\textbf{When might beam search fail}$\nBeam search is a heuristic method and is not guaranteed to produce optimal solutions.\nThis is why we need the branch-and-bound framework.\nBeam search is already doing very well in the regime of highly-correlated features as shown in the synthetic benchmarks and differential experiments.\nHowever, since you ask when beam search might fail, we conjecture that beam search might fail in finding the optimal solution in the regime of super-correlated features.\nAs a pathological example, suppose we have lots of features, which can be divided into different groups based on their correlation levels.\nInside each group, features are super-correlated and are almost the same and only differed in several samples.\nIn this case, beam search may have a hard time because there exist many near-optimal solutions.\n(This is also the case where MIOSR will be extremely slow.)\nFor most practical applications, beam search should work well.\nNote, we have already compared with many existing heuristics in Figure 9 (page 177) of the Appendix.\n\n$\\textbf{Where to improve}$\nBeam search is doing OK with respect to time according to our profiling.\nCurrently, lower bound calculation through ADMM is still taking more time than beam search.\nSince ADMM involves repeatedly solving linear systems (See Equation 23), we are thinking about using Joel A. Tropp's randomized linear algebra method to solve the large-scale linear system approximately to save time.\nNote, we are already making a Cholesky factorization at the beginning of the ADMM to save some computational time according to Stephen P. Boyd's book.\nWe hope randomized linear algebra could give us another boost of performance.\n\n$\\lambda_{\\text{min}}$  $\\textbf{and}$  $\\lambda_{\\text{max}}$  $\\textbf{estimation}$\nWe use numpy to calculate all eigenvalues and pick $\\lambda_{\\text{min}}$ and $\\lambda_{\\text{max}}$ afterwards.\nHowever, this eigenvalue calculation is only performed at the root node in the BnB tree.\nAfter we calculate the step size for ADMM at the root node, we use this step size for ADMM for all the nodes in the BnB tree.\nTo the rest of the nodes, such a step size is an approximation, but it works well in all our experiments.\nFor the smallest eigenvalue used in $Q_{\\lambda}$, please look at Appendix E7 on the Interlacing Eigenvalue Property.\n\n$\\textbf{How to choose tolerance level $\\epsilon$?}$\nIt is hard to provide a definitive answer or guidance.\nAny reasonable small number (such as $10^{-4}$ used in all our experiments; this is also the default value used in Gurobi) is probably fine.\n\n$\\textbf{What to do when we don't know $k$ a priori?}$ \nIn our differential equation experiments, $k$ is not known a priori and is treated as a hyperparameter chosen via cross-validation. We opt to use the hyperparameters that minimize $AIC_c$ developed specifically for sparse regression. Please see the reference [46] and details in Appendix F.3. This is the approach we recommend.\n\n$\\textbf{Why emphasis on dynamical systems?}$\nYou are right that our method can be applied broadly to other applications involving sparse ridge regression.\nThe reason, as we explained in the introduction, is that (Line 23) selection of the correct features is key to discovering the true dynamics.\nEmpirically, due to the highly-correlated features formed by high-order polynomials, most existing heuristic methods failed to find the correct features (see Figure 9 on page 177 of the Appendix).\nTherefore, finding optimal sparsity-constrained solution is really needed in this context and may have an impact in real scientific discovery.\nBesides, we know the true dynamics so that it is easy to verify the correctness of the solution to make sure our code base is solid.\nFurthermore, sparse ridge regression is used extensively in the SINDy literature, making it a solid choice for application.\nNevertheless, we plan to extend this work and branch out to other applications."}, {"review_id": "BTMT9koS2T", "reviewer": "Reviewer_X2ir", "summary": "This paper proposes a novel algorithm to solve sparse ridge regression problems -- which are standard ridge regression problems with an additional l0 sparsity constraint (so this problem is NP Hard in general). The algorithm is a branch and bound based approach that uses a novel, tighter, lower bound computed by solving a saddle point reformulation of the problem using ADMM. The reformulation is done using a perspective transform approach along with a reparameterization trick. Additionally, beam search over the chosen non-zero coordinates is used as a heuristic to obtain better upper bounds while expanding the branch and bound tree. A provable guarantee on the optimality gap of the solution obtained from this algorithm is also provided. Experiments on synthetic data as well as data from dynamical systems like the Hopf bifurcation, the Lorenz system and a non-linear magnetohydrodynamical model demonstrate the efficacy of this algorithm compared to existing open source and commericial benchmarks. ", "questions": "1. How does this method compare to relaxing the sparsity constraint to a lasso regularization, and approximating a sparse solution from the solution of the lasso?\n2. Have the authors conducted experiments using the simpler (but not as tight) lower bound obtained by solving the ridge regression linear system? If so, were there significant differences between this and the baseline? Further, are there any theoretical guarantees that can be shown for this simpler bound?", "limitations": "Yes, the authors addressed limitations of OKRidge in the case of smaller feature dimensions.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well very written and clear.\n2. I greatly appreciate the detailed and helpful annotations in each step of the proofs as well as the algorithms.\n3. The main result is obtained by combining a novel reformulation of the sparse ridge regression problem with ADMM to obtain a tighter lower bound that can be used in branch and bound.\n4. Thorough experiments were conducted to study and analyze the performance of the described algorithm.\n", "weaknesses": "1. Perhaps the authors could include experiments on more applications that arise from challenging physical systems beyond the three presented in the paper, since the beam search alone was sufficient for the synthetic problem.\n2. The improvement over MIOSR in the case of the sparse differential equations like Lorenz, Hopf and MHD is not very significant (Figure 3), however it still matches the performance of using a proprietary solver.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Is beam search sufficient}$ Please see Figure 2 in our rebuttal pdf.\nFor the MHD system, beam search alone is not sufficient for discovering the correct dynamical systems.\nThus, it is necessary to use BnB to get the optimal solution in this high-dimensional and highly correlated setting.\n\n$\\textbf{Improvement over MIOSR is not significant}$ If we look at the training and test losses, we agree with you sometimes the improvement over MIOSR is not significant.\nHowever, if we look at the metric of the true positive rate, then our method is doing better than MIOSR.\nTrue positive rate measures how often we recover the true dynamics (true features).\nEven though MIOSR might achieve a small training loss, the loss is not optimal, which results in MIOSR not discovering the true dynamics.\nIn practice, this means MIOSR cannot help us discover the true physics from the data.\n(Again, note that MIOSR has a major disadvantage of being proprietary software with an expensive license.)\n\n$\\textbf{Results on Lasso}$ Please see Figure 2 in our rebuttal pdf.\nLasso does not do well in the application of discovering sparse differential equations because the features are highly correlated.\nThe poor performance of Lasso has been discussed in previous SINDy literature: 1. \"A unified sparse optimization framework to learn parsimonious physics-informed models from data\" (Section 2.1) by Kathleen Champion, Peng Zheng, Aleksandr Y. Aravkin, Steven L. Brunton, J. Nathan Kutz. 2. \"Benchmarking sparse system identification with low-dimensional chaos.\" by Alan A. Kaptanoglu, Lanyue Zhang, Zachary G. Nicolaou, Urban Fasel, and Steven L. Brunton.\n\n$\\textbf{Comparison between fast solve and ADMM}$ Please see Figure 1 in our rebuttal pdf.\nThe results show that using the simpler method (fast solve) cannot calculate a tight lower bound, which results in longer running time and larger optimality gap.\nAlso, we would like to clarify one point in our OKRidge framework: we use fast solve and ADMM together.\nIf you look at Algorithm 1 on page 34 in the Appendix, we run ADMM to tighten the lower bound if the fast solve cannot help us prune the current node.\n\n$\\textbf{Theory behind the simpler method}$ We are not very sure what you mean by \"theoretical guarantee\". In the main paper, we made some mathematical connections by showing that the simpler method of calculating the lower bound is actually trying to maximize another lower bound of Problem 12 (See Theorem 3.2).\nThe simpler method can also be intuitively understood from the perspective of strong convexity (See Line 122-134).\nDoes this answer your question?\nIf not, could you please clarify?"}, {"review_id": "J6xxUHDAZu", "reviewer": "Reviewer_ono9", "summary": "This paper proposes a method, namely OKRidge, to solve spare ridge regression problems and identify terms of differential equations of nonlinear dynamical systems. In particular, OKRidge is formulated as a branch-and-bound algorithm, and this paper provides a lower bound calculation.", "questions": "$\\mathbf{1}.$ Please see $\\mathbf{1}.$ in $\\mathbf{Weakness}$ section.", "limitations": "This paper includes a limitation statement, and I do not have any concerns regarding a potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "$\\mathbf{1}.$ The proposed method is novel, mainly on the formulation of the optimization problem as prescribed by Eq (12), as well as the approach to solve the problem.\n\n$\\mathbf{2}.$ The empirical analysis is thorough and results are promising compared with various solvers/algorithms, including commercial solver, e.g., GuRoBi. The experiments very well supports the proposal of the algorithm with adequate evidence.\n\n$\\mathbf{3}.$ The proposed method is very complete, which covers a major branch-and-bound algorithm with novel lower bound approach and a heuristic to search in a tree. ", "weaknesses": "$\\mathbf{1}.$ Despite the novel formulation of the optimization problem (as well as the lower bound derivation), it might be still combination of known approach to derive the package of approach, including BnB, dynamic programming and heuristics. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Our method is not novel due to the use of branch-and-bound}$ Branch-and-bound is the foundation of all commercial MIP solvers (one variant is branch-and-cut, but it is still based on top of branch-and-bound).\nWithout using branch-and-bound, we cannot certify optimality for this NP-hard problem.\nWe hope the reviewer sees the effectiveness of our novel lower bound formulation and calculation and not use the fact of us relying upon branch-and-bound as a reason for rejection.\n (One wouldn't, for instance, reject a paper with a novel neural architecture because neural networks have been used before.)\n\n$\\textbf{Novelty of dynamic programming}$ Dynamic programming is a general method.\nHowever, we believe dynamic programming has not been applied in combination of beam search in a branch-and-bound framework for sparse ridge regression."}, {"review_id": "Fk77ZRxghq", "reviewer": "Reviewer_kX5s", "summary": "The paper proposes a lower bound calculation method and warm start method for sparse ridge regression via a branch-and-bound approach. Starting from a saddle point formulation, the lower bound on the original objective can be given by either a linear system or ADMM problem which can be efficiently solved. The warm-start is given by the beam search method. Experiments are provided to demonstrate the better efficiency of the proposed approach over baselines.", "questions": "See above", "limitations": "Limitations are addressed in the paper.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. propose a scalable method for sparse ridge regression.\n2. demonstrate the feasibility of the proposed method in practice.", "weaknesses": "1. The main contribution lies in efficient certifiable lower bound which however is just part of the whole algorithm. It would be better if the whole algorithm can be explicitly given in the main text.\n2. There is no analysis that gives the complexity of the whole algorithm to achieve a acceptable optimality gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Whole algorithm should be explicitly given in the main text}$ Detailed algorithms are already given in the original manuscript.\nPlease see Algorithm 1-8 on page 34-40 in Appendix E.\nThe whole algorithm doesn't fit in 9 pages in the main paper, but we tried to organize it well in the Appendix.\nNeurIPS often gives an extra page that will help.\nDuring revision, we can put the overall algorithm (Algorithm 1) in the main paper.\n\n$\\textbf{No complexity analysis of the algorithm to achieve an acceptable optimality gap}$\nSince the problem is NP-hard, one cannot easily talk about convergence rate to the optimal solution in the same way as in the context of convex optimization, such as $O(1/k)$ or $O(1/k^2)$ convergence rate.\nWe have already shown that most existing heuristic methods for producing sparse linear models failed to find the correct features (see Figure 9 on page 177 of the Appendix).\nThese heuristic methods may enjoy fast convergence rate to a locally stationary solution, but they cannot be used reliably in discovering true dynamical systems.\nTherefore, using branch-and-bound to certify global optimality is really needed in this context.\n\nThe speed of BnB depends heavily on how efficiently we find the upper bound and lower bound.\nThe branch-and-bound framework focuses heavily on intelligent enumeration.\nIf the lower bound is tight, we can prune most of the search space, which can greatly increase our certification process.\nComputing a tight lower bound efficiently is a major focus of our work."}, {"review_id": "BZACN9QLg8", "reviewer": "Reviewer_Nrfj", "summary": "This paper proposes a novel branch-and-bound technique called OKRidge to solve sparse Ridge regression problems to learn differential equations. The paper makes 3 contributions:\n1) A saddle point formulation of the optimization problem at hand which allows the use of a branch-and-bound algorithm leveraging the special structure (sparsity) of the problem.\n2) This efficient branch-and-bound algorithm constructs tight lower-bounds thanks to the saddle-point formulation.\n3) A beam search that leverages previous searches for past nodes to speed up the search process of the current node. ", "questions": "- As a non-expert in the field of differential equation learning, it is unclear to me why $z_j \\in \\{0, 1\\}$. Is it a generic formulation or does this integer constraint arise often in this field? A clarification in the intro would help the reader understand why it is necessary and why you compare yourselves to MIP solvers.\n\n- How does OKRidge scale with respect to the number of features?\n\n- \"Our algorithms are written in Python, Any improvements we see over commercial MIP solvers, which are coded in C/C++, are solely due to our specialized algorithms\" (l. 204): how can a Python routine (even very efficient) can beat an algorithm written in C/C++? Are you using a JIT compiler like Numba to compile critical parts of the code? If not, have you tried? It could significantly increase the runtime speed.", "limitations": "I see one unaddressed limitation to the method: while OKRidge scales easily to hundreds of thousands of samples, it does not seem to scale as well on the number of features. For instance, as stated in l.132, deriving the lower bound requires inverting a $p\\times p$ matrix, which does not scale well for $p \\gg 10000$. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. Apart from some details (see question section below) that might be unclear to the non-expert readers, the context of the paper is clear and the literature review is comprehensive.\n\nBesides, several extensive experiments have been made. Given the number of hyperparameters in the method ($\\lambda$, $\\lambda_2$) and the hidden parameters that might impact the convergence speed of the proposed methodology (correlation $\\rho$, setup size $n$ and $p$, $\\textrm{SNR}$), the sensitivity analysis carried out in the experiments section is welcome.", "weaknesses": "The paper does not have major weaknesses. \n\nBesides, as a suggestion, since your experiments assess the convergence speed as well as the quality of the sub-optimality gaps yielded by the solvers, you can use the Benchopt framework for more impactful figures: https://github.com/benchopt/benchopt . For instance, figure 1 could be summarized into a one-row figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Use Benchopt framework to do plotting}$ Thank you for this suggestion!\nLooking at Benchopt, we find that none of the available benchmarks support non-relaxed sparsity constraints.\nThis is a helpful library that could be useful for the broad ML community.\nWe can cite this work during revision.\nActually, when we extend this work to some other applications, this is a good benchmark library to build upon, and we can make a pull request on GitHub for this repository so that Benchopt includes a cardinality-constrained ridge regression benchmark.\n\n$\\textbf{Why}$ $z_j \\in \\\\{0, 1\\\\}$ $\\textbf{?}$ $z_j$ is a binary variable to indicate whether we choose to select feature $j$ or not (i.e., $z_j = 1$ vs. $z_j = 0$).\nThese binary variables are used to enforce the sparsity constraint.\nThere are approaches other than the MIP approach to produce sparse models (such as the baselines in Figure 3 on page 8 of the main paper or Figure 9 on page 177 in the appendix), but they are not competitive.\nGetting to know the true dynamics is important for scientific discovery.\nThis is why we focus on the MIP approach and try to come up with novel lower bound formulation and calculation.\n\n$\\textbf{How scalable is OKRidge with respect to the number of features?}$\nIn the submitted manuscript, the largest number of features we have tried is 7000.\nAs the number of features increases, solving the linear system (Equation 23) will take longer.\nHowever, if you look at results in the main paper and appendix, we are already much faster than the commercial solver Gurobi with all existing MIP formulations.\nHence, for our current code base, we enjoy either better or comparable scalability than commercial solver Gurobi.\nSee below on our plan to make OKRidge more scalable.\nNote that licenses for Gurobi software are extremely expensive.\n\n$\\textbf{How to make OKRidge more scalable with respect to the number of features?}$\nAs we have mentioned above, solving the linear system (Equation 23) will take longer time as the feature dimension increases.\nOne promising solution to further stretch the scalability of our method is to apply Joel A. Tropp's randomized linear algebra method to solve the linear system.\nNote, we are already making a Cholesky factorization at the beginning of the ADMM to save some computational time according to Stephen P. Boyd's book.\nWe hope randomized linear algebra could give us another boost of performance.\n\n$\\textbf{Why can we outperform an algorithm implemented in C/C++?}$\nThe most important reason is that our proposed algorithm exploits the special mathematical structure (our fast solve and ADMM methods) of this problem. In contrast, the commercial solver relies on a generic MIP algorithm, which is proprietary and closed source.\nThe paper finds a new way of calculating the lower bound in order to prune the search space more effectively, specifically for sparse ridge regression.\nWe use numpy, which uses BLAS and LAPACK to do the heavy linear algebra computation.\nPython is slow when there are loops (our ADMM method is an iterative method and involves loops).\nPython is also slow because many of its functions have wrappers and needs to do lots of checking for types and bounds.\nHowever, because our algorithm is efficient, these two disadvantages are reduced to a minimal negative effect.\nOf course, coding everything in C++ or substituting part of the code in Numba (which we haven't tried) could make our code go even faster.\nThis current work is to first demonstrate that our idea works even without resorting to C++ or Numba compiled code."}]

Author rebuttal:
First of all, we just want to thank the reviewers for their thoughtful comments.
We will use this general response to address an important theoretical question and will address specific issues to each reviewer.
Requested experimental results on 1) comparison between fast solve and ADMM and 2) comparison between Lasso, beam search, and OKRidge are also included as a PDF document as part of this general response.

$\textbf{Correctness of Theorem 3.1}$
We still believe that Theorem 3.1 should hold.

Reviewer MGha asks a very good question: what will happen if $Q_{\lambda}$ is singular?
If $Q_{\lambda}$ is singular, our reparametrization trick could miss some subspace and therefore Problem (12) may become a relaxation of Problem (10), as Reviewer fUoX worries.
Fortunately, during revision, we can insert the following lemma: for an optimal solution $c^*$ to Problem (10), there exists some $\beta^*$ so that $c^*=\frac{2}{\lambda_2 + \lambda}(X^Ty - Q_{\lambda}\beta^*)$.
This lemma implies that the subspace we miss will not prevent Problem (12) from achieving the same optimal value as Problem (10).

We prove this lemma by the method of contradiction.
Suppose the optimal solution $c$ to Problem (10) cannot be written by Equation (11).
Through the rank-nullity theorem in linear algebra, we can find $d_1 \in col(Q_{\lambda})$ and $d_2 \in ker(Q_{\lambda}), d_2 \neq 0$ so that $c = \frac{2}{\lambda_2 + \lambda}(X^Ty-d_1 + d_2)$, where $col(\cdot)$ denotes the column space of a matrix and $ker(\cdot)$ denotes the kernel of a matrix.
If we let $\beta = \alpha d_2$ for some real value $\alpha$, then the inner optimization of Problem (10) (ignore the last term involving $z_j$'s for now since $\beta$ and $z$ are separable in the objective function) becomes
\begin{align*}
    L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c) &= \beta^T Q_{\lambda} \beta - ((\lambda_2+\lambda)c - 2 X^Ty)^T \beta \\\\
    &= \beta^T Q_{\lambda} \beta - ((\lambda_2+\lambda)\frac{2}{\lambda_2 + \lambda} (X^T y-d_1 +d_2) - 2 X^Ty)^T \beta\\\\
    &= \beta^T Q_{\lambda} \beta - (2 (X^T y-d_1 +d_2) - 2 X^Ty)^T \beta \\\\
    &= \beta^T Q_{\lambda} \beta + 2 (d_1 -d_2)^T \beta \\\\
    &= \beta^T Q_{\lambda} \beta + 2 d_1^T \beta -2 d_2^T \beta.
\end{align*}
Because $\beta =\alpha d_2$, we have $\beta \in ker(Q_{\lambda})$ and $\beta \perp d_1$.
Thus, the first two terms become $0$, and we are left with $L_{\text{ridge}-\lambda}^{\text{Fenchel}} = - 2d_2^T \beta = - 2\alpha \lVert d_2\rVert_2^2$.
For the inner optimization of Problem (10), because we want to minimize over $\beta$, we can let $\alpha \rightarrow \infty$, and we will get $L_{\text{ridge}-\lambda}^{\text{Fenchel}} \rightarrow - \infty$.
This is obviously not the optimal value for Problem (10) (a simple counter-example is to let $c = \frac{2}{\lambda_2 + \lambda} X^T y$, and we can easily show that $\min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c)$ is finite).
Therefore, $d_2$ must be 0, and the optimal solution $c$ to Problem (10) must be represented by some $\beta$ through the equation $c=\frac{2}{\lambda_2 + \lambda}(X^Ty - Q_{\lambda}\beta)$.

Now that we have this lemma, to clear the questions of Reviewer Nrfj, we can prove Theorem 3.1 with an alternative approach.
We will show $\max_{c} \min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c) \leq \max_{\gamma} \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\gamma, z)$ and $\max_{c} \min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c) \geq \max_{\gamma} \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\gamma, z)$ (both under the constrains $\sum_{j=1}^p z_j \leq k, z_j \in [0, 1]$) and therefore establish the equivalence of optimal values.

For the first inequality, let $\beta^*, c^*$ be the optimal solution to Problem 10 with the relation $c^*=\frac{2}{\lambda_2 + \lambda}(X^Ty - Q_{\lambda}\beta^*)$.
Furthermore, let $\gamma^* = \beta^*$.
Then we have
\begin{align*}
    \max_{c} \min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c) = \min_{z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta^*, z, c^*) = \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\gamma^*, z) \leq \max_{\gamma} \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\gamma, z)
\end{align*}
The derivation for the second equality roughly follows the derivation in Appendix C1 on page 18-19.

For the second inequality, let $\hat{\gamma}$ be the optimal solution to Problem 12.
Let $\hat{c} = \frac{2}{\lambda_2 + \lambda}(X^Ty - Q_{\lambda}\hat{\gamma})$
Furthermore, let $\hat{\beta} = \hat{\gamma}$ (notice that this $\hat{\beta}$ satisfies the optimality condition of Problem 10).
Then we have
\begin{align*}
    \max_{\gamma} \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\gamma, z) &=
    \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\hat{\gamma}, z) \\\\
    &= \min_{z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\hat{\beta}, z, \hat{c}) \\\\
    &= \min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, \hat{c}) \\\\
    &\leq \max_{c} \min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c)
\end{align*}
Similarly, the derivation for the second equality roughly follows the derivation (but this time in reverse order) in Appendix C1 on page 18-19.

Using these two inequalities, we can establish the equivalence of optimal values between Problem 10 and Problem 12, $\textit{i.e.}$, $\max_{c} \min_{\beta, z} L_{\text{ridge}-\lambda}^{\text{Fenchel}}(\beta, z, c) = \max_{\gamma} \min_{z} L_{\text{ridge}-\lambda}^{\text{saddle}}(\gamma, z)$.

Meta-review: The authors develops a fast algorithm for solving k-sparse ridge regression problems. The proposed method is based on branch-and-bound approach and leverages the specific structure of the ridge regression problem. The paper is well written, the experiments are illuminating, and the ideas developed have many use cases in data analysis and scientific discovery.
Decision: Accept (spotlight)