SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $j^{*} \in \operatorname{arg~BottomB}_{j}$  means  $j^{*}$  belongs to the set of solutions whose loss is one of the B smallest losses. Afterwards, we finetune the solution on the newly expanded support and choose the best B solutions for the next stage of support expansion. A visual illustration of beam search can be found in Figure 6 in Appendix E, which also contains the detailed algorithm.
Although many methods have been proposed for sparse ridge regression, none of them have been designed with the BnB tree structure in mind. Our approach is to take advantage of the search history of past nodes to speed up the search process for a current node. To achieve this, we follow a dynamic programming approach by saving the solutions of already explored support sets. Therefore, whenever we need to adjust coefficients on the new support during beam search, we can simply retrieve the coefficients from the history if a support has been explored in the past. Essentially, we trade memory space for computational efficiency.
3.2.1 Provable Guarantee
Lastly, using similar methods to [29], we quantify the gap between our found heuristic solution  $\hat{\beta}$  and the optimal solution  $\beta^{*}$  in Theorem 3.4. Compared with Theorem 5 in [29], we improve the factor in the exponent from  $\frac{m_{2k}}{M_{2k}}$  to  $\frac{m_{2k}}{M_1}$  (since  $M_1\leq M_{2k}$ , where  $M_{1}$  and  $M_{2k}$  are defined in [29]).
Theorem 3.4. Let us define a  $k$ -sparse vector pair domain to be  $\Omega_k \coloneqq \{(x, y) \in \mathbb{R}^p \times \mathbb{R}^p : \| x \|_0 \leq k, \| y \|_0 \leq k, \| x - y \|_0 \leq k\}$ . Any  $M_1$  satisfying  $f(y) \leq f(x) + \nabla f(x)^T (y - x) + \frac{M_1}{2} \| y - x \|_2^2$  for all  $(x, y) \in \Omega_1$  is called a restricted smooth parameter with support size 1, and any  $m_{2k}$  satisfying  $f(y) \geq f(x) + \nabla f(x)^T (y - x) + \frac{m_{2k}}{2} \| y - x \|_2^2$  for all  $(x, y) \in \Omega_{2k}$  is called a restricted strongly convex parameter with support size  $2k$ . If  $\hat{\beta}$  is our heuristic solution by the beam-search method, and  $\beta^*$  is the optimal solution, then:
$$
\mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right) \leq \mathcal {L} _ {\text {r i d g e}} (\hat {\boldsymbol {\beta}}) \leq \left(1 - e ^ {- m _ {2 k} / M _ {1}}\right) \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right). \tag {27}
$$
3.3 Branching and Queuing
Branching: The most common branching techniques include most-infeasible branching and strong branching [2, 1, 15, 7]. However, these two techniques require having fractional values for the binary variables  $z_{j}$ 's, which we do not compute in our framework. Instead, we propose a new branching strategy based on our heuristic solution  $\hat{\beta}$ : we branch on the coordinate whose coefficient, if set to 0, would result in the largest increase in the ridge regression loss  $\mathcal{L}_{\mathrm{ridge}}$  (See Appendix E for details):
$$
j ^ {*} = \underset {j} {\operatorname {a r g m a x}} \mathcal {L} _ {\text {r i d g e}} \left(\hat {\boldsymbol {\beta}} - \hat {\boldsymbol {\beta}} _ {j} \boldsymbol {e} _ {j}\right). \tag {28}
$$
The intuition is that the coordinate with the largest increase in  $\mathcal{L}_{\mathrm{ridge}}$  potentially plays a significant role, so we want to fix such a coordinate as early as possible in the BnB tree.
Queuing: Besides the branching strategy, we need a queue to pick a node to explore among newly created nodes. Here, we use a breadth-first approach, evaluating nodes in the order they are created.
Figure 1: Comparison of running time (top row) and optimality gap (bottom row) between our method and baselines, varying the number of features, for three correlation levels  $\rho = 0.1, 0.5, 0.9$  ( $n = 100000$ ,  $k = 10$ ). Time is on the log scale. Our method is generally orders of magnitude faster than other approaches. Our method achieves the smallest optimality gap, especially when the feature correlation  $\rho$  is high.
Figure 2: Comparison of running time (top row) and optimality gap (bottom row) between our method and baselines, varying sample sizes, for three correlation levels  $\rho = 0.1, 0.5, 0.9$  ( $p = 3000$ ,  $k = 10$ ). Time is on the log scale. When  $\rho = 0.1$  and  $\rho = 0.5$ , OKRidge is generally orders of magnitude faster than other approaches. In the case  $\rho = 0.9$ , we achieve the smallest optimality gap as shown in the bottom row.
We test the effectiveness of our OKRidge on synthetic benchmarks and sparse identification of nonlinear dynamical systems (SINDy)[19]. Our main focus is: assessing how well our proposed lower bound calculation speeds up certification (Section 4.1), and evaluating solution quality of OKRidge on challenging applications (Section 4.2). Additional extensive experiments are in Appendix G and H. Our algorithms are written in Python. Any improvements we see over commercial MIP solvers, which are coded in  $\mathrm{C / C + + }$  , are solely due to our specialized algorithms.
Figure 3: Results on discovering sparse differential equations. On various metrics, OKRidge outperforms all other methods, including MIOSR which uses a commercial (proprietary) MIP solver.
4.1 Assessing How Well Our Proposed Lower Bound Calculation Speeds Up Certification
Here, we demonstrate the speed of OKRidge for certifying optimality compared to existing MIPs solved by Gurobi [35]. We set a 1-hour time limit and an optimality gap of relative tolerance  $10^{-4}$ .
We use a value of 0.001 for  $\lambda_{2}$ . Our 4 baselines include MIPs with SOS1, big-M ( $M = 50$  to prevent cutting off optimal solutions), perspective [4], and eigen-perspective formulations ( $\lambda = \lambda_{\min}(\mathbf{X}^T\mathbf{X}))$  [28]. In the main text, we use plots to present the results. In Appendix G, we present the results in tables. Additionally, in Appendix G, we conduct perturbation studies on  $\lambda_{2}$  ( $\lambda_{2} = 0.1$  and  $\lambda_{2} = 10$ ) and  $M$  ( $M = 20$  and  $M = 5$ ). Finally, still in Appendix G, we also compare OKRidge with other MIPs including the MOSEK solver [3], SubsetSelectionCIO [11], and L0BNB [39].
Similar to the data generation process in [11, 48], we first sample  $x_{i} \in \mathbb{R}^{p}$  from a Gaussian distribution  $\mathcal{N}(\mathbf{0}, \Sigma)$  with mean 0 and covariance matrix  $\Sigma$ , where  $\Sigma_{ij} = \rho^{|i - j|}$ . Variable  $\rho$  controls the feature correlation. Then, we create the coefficient vector  $\beta^{*}$  with  $k$  nonzero entries, where  $\beta_{j}^{*} = 1$  if  $j \mod (p / k) = 0$ . Next, we construct the prediction  $y_{i} = x_{i}^{T}\beta^{*} + \epsilon_{i}$ , where  $\epsilon_{i} \stackrel{i.i.d.}{\sim} \mathcal{N}(0, \frac{\|X\beta^{*}\|_{2}^{2}}{\mathrm{SNR}})$ . SNR stands for signal-to-noise ratio (SNR), and we choose SNR to be 5 in all our experiments.
In the first setting, we fix the number of samples with  $n = 100000$  and vary the number of features  $p \in \{100,500,1000,3000,5000\}$  and correlation levels  $\rho \in \{0.1,0.5,0.9\}$  (See Appendix G for  $\rho = 0.3$  and  $\rho = 0.7$ ). We warm-started the MIP solvers by our beam-search solutions. The results can be seen in Figure 1. From both figures, we see that OKRidge outperforms all existing MIPs solved by Gurobi, usually by orders of magnitude.
In the second setting, we fix the number of features to  $p = 3000$  and vary the number of samples  $n \in \{3000, 4000, 5000, 6000, 7000\}$  and the correlation levels  $\rho \in \{0.1, 0.5, 0.9\}$  (see Appendix G for  $\rho = 0.3$  and  $\rho = 0.7$ ). As in the first setting, we also warm-started the MIP solvers by our beam-search solutions. The results are in Figure 2. When  $n$  is close to  $p$  or the correlation is high
$(\rho = 0.9)$ , no methods can finish within the 1-hour time limit, but OKRidge prunes the search space well and achieves the smallest optimality gap. When  $n$  becomes larger in the case of  $\rho = 0.1$  and  $\rho = 0.5$ , OKRidge runs orders of magnitude faster than all baselines.
4.2 Evaluating Solution Quality of OKRidge on Challenging Applications
On previous synthetic benchmarks, many heuristics (including our beam search method) can find the optimal solution without branch-and-bound. In this subsection, we work on more challenging scenarios (sparse identification of differential equations). We replicate the experiments in [9] using three dynamical systems from the PySINDy library [27, 42]: Lorenz System, Hopf Bifurcation, and magnetohydrodynamical (MHD) model [24]. The Lorenz System is a 3-D system with the nonlinear differential equations:
$$
d x / d t = - \sigma x + \sigma y, \quad d y / d t = \rho x - y - x z, \quad d z / d t = x y - \beta z
$$
where we use standard parameters  $\sigma = 10, \beta = 8/3, \rho = 28$ . The true sparsities for each dimension are  $(2,3,2)$ . The Hopf Bifurcation is a 2-D system with nonlinear differential equations:
$$
d x / d t = \mu x + \omega y - A x ^ {3} - A x y ^ {2}, \quad d y / d t = - \omega x + \mu y - A x ^ {2} y - A y ^ {3}
$$
where we use the standard parameters  $\mu = -0.05, \omega = 1, A = 1$ . The true sparsities for each dimension are (4, 4). Finally, the MHD is a 6-D system with the nonlinear differential equations:
$$
d V _ {1} / d t = 4 V _ {2} V _ {3} - 4 B _ {2} B _ {3}, \quad d V _ {2} / d t = - 7 V _ {1} V _ {3} + 7 B _ {1} B _ {2}, \quad d V _ {3} / d t = 3 V _ {1} V _ {2} - 3 B _ {1} B _ {2},
$$
$$
d B _ {1} / d t = 2 B _ {3} V _ {2} - 2 V _ {3} B _ {2}, \quad d B _ {2} / d t = 5 V _ {3} B _ {1} - 5 B _ {3} V _ {1}, \quad d B _ {3} / d t = 9 V _ {1} B _ {2} - 9 B _ {1} V _ {2}.
$$
The true sparsities for each dimension are  $(2, 2, 2, 2, 2, 2)$ .
We use all monomial features (candidate functions) up to 5th order interactions. This results in 56 functions for the Lorentz System, 21 for Hopf Bifurcation, and 462 for MHD. Due to the high-order interaction terms, features are highly correlated, resulting in poor performance of heuristic methods.
4.2.1 Baselines and Experimental Setup
In addition to MIOSR (which relies on the SOS1 formulation), we also compare with three common baselines in the SINDy literature: STLSQ [54], SSR [16], and E-STLSQ [31]. The baseline SR3 [25] is not included since the previous literature [9] shows it performs poorly. We compare OKRidge with other baselines using the SINDy library [27, 42]. We follow the experimental setups in [9] for model selection, hyperparameter choices, and evaluation metrics (please see Appendix F for details). In Appendix H, we provide additional experiments on Gurobi with different MIP formulations and comparing with more heuristic baselines.
Figure 3 displays the results. OKRidge (red curves) outperforms all baselines, including MIOSR (blue curves), across evaluation metrics. On the Lorenz System, all methods recover the true feature support when the training trajectory is long enough. When the training trajectory length is short, i.e., the left part of each subplot, (or equivalently, when the number of samples is small), OKRidge performs uniformly better than all other baselines. On the Hopf Bifurcation, all heuristic methods fail to recover the true support, resulting in poor performance. On the final MHD, OKRidge maintains the top performance and outperforms MIOSR on the true positivity rate. This demonstrates the effectiveness of OKRidge, which incurs lower runtimes and yields better metric scores under high-dimensional settings. The highest runtimes are incurred for the MHD (with 462 candidate functions/features), which are shown in Figure 3.
Figure 4: Running time comparison between OKRidge and MIOSR on the MHD system with 462 candidate functions. OKRidge is significantly faster than the previous state of the art.
Limitations of OKRidge When the feature dimension is low (under 100s), Gurobi can solve the problem to optimality faster than OKRidge. This is observed on the synthetic benchmarks  $(p = 100)$  and also on the Hopf Bifurcation  $(p = 21)$ . Since Gurobi is a commercial proprietary solver, we cannot inspect the details of its sophisticated implementation. Gurobi may resort to an enumeration/brute-force approach, which could be faster than spending time to calculate lower bounds in the BnB tree. This being said, OKRidge is still competitive with Gurobi in the low-dimensional setting, and OKRidge scales favorably in high-dimensional settings.
We presented a method for optimal sparse ridge regression that leverages a novel tight lower bound on the objective. We showed that the method is both faster and more accurate than existing approaches for learning differential equations – a key problem in scientific discovery. This tool (unlike its main competitor) does not require proprietary software with expensive licenses and can have a significant impact on various regression applications.
Implementations of OKRidge discussed in this paper are available at https://github.com/jiachangliu/OKRidge.
The authors gratefully acknowledge funding support from grants NSF IIS-2130250, NSF-NRT DGE-2022040, NSF OAC-1835782, DOE DE-SC0023194, and NIH/NIDA R01 DA054994. The authors would also like to thank the anonymous reviewers for their insightful comments.
[1] T. Achterberg, T. Koch, and A. Martin. Branching rules revisited. Operations Research Letters, 33(1):42-54, 2005.
[2] D. Applegate, R. Bixby, V. Chvátal, and W. Cook. On the solution of traveling salesman problems. Documenta Mathematica, pages 645-656, 1998.
[3] M. ApS. Mosek optimizer API for python. Version, 9(17):6-4, 2022.
[4] A. Atamturk and A. Gómez. Safe screening rules for 10-regression from perspective relaxations. In International Conference on Machine Learning, pages 421-430. PMLR, 2020.
[5] A. Atamturk, A. Gómez, and S. Han. Sparse and smooth signal estimation: Convexification of 10-formulations. Journal of Machine Learning Research, 22:52-1, 2021.
[6] A. Beck and Y. C. Eldar. Sparsity constrained nonlinear optimization: Optimality conditions and algorithms. SIAM Journal on Optimization, 23(3):1480-1509, 2013.
[7] P. Belotti, C. Kirches, S. Leyffer, J. Linderoth, J. Luedtke, and A. Mahajan. Mixed-integer nonlinear optimization. Acta Numerica, 22:1-131, 2013.
[8] D. Bertsekas. Convex optimization theory, volume 1. Athena Scientific, 2009.
[9] D. Bertsimas and W. Gurnee. Learning sparse nonlinear dynamics via mixed-integer optimization. Nonlinear Dynamics, Jan 2023.
[10] D. Bertsimas, A. King, and R. Mazumder. Best subset selection via a modern optimization lens. The Annals of Statistics, 44(2):813-852, 2016.
[11] D. Bertsimas, J. Pauphilet, and B. Van Parys. Sparse regression: Scalable algorithms and empirical performance. Statistical Science, 35(4):555-578, 2020.
[12] M. J. Best and N. Chakravarti. Active set algorithms for isotonic regression; a unifying framework. Mathematical Programming, 47(1-3):425-439, 1990.
[13] T. Blumensath and M. E. Davies. Gradient pursuits. IEEE Transactions on Signal Processing, 56(6):2370-2382, 2008.
[58] S. Wiseman and A. M. Rush. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296-1306, Austin, Texas, Nov. 2016. Association for Computational Linguistics.
[59] W. Xie and X. Deng. Scalable algorithms for the sparse ridge regression. SIAM Journal on Optimization, 30(4):3359-3386, 2020.
[60] G. Yuan, L. Shen, and W.-S. Zheng. A block decomposition algorithm for sparse optimization. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 275-285, 2020.
[61] T. Zhang. Adaptive forward-backward greedy algorithm for learning sparse representations. IEEE Transactions on Information Theory, 57(7):4689-4708, 2011.
[62] X. Zheng, X. Sun, and D. Li. Improving the performance of MIQP solvers for quadratic programs with cardinality and minimum threshold constraints: A semidefinite program approach. INFORMS Journal on Computing, 26(4):690-703, 2014.
[63] J. Zhu, C. Wen, J. Zhu, H. Zhang, and X. Wang. A polynomial algorithm for best-subset selection problem. Proceedings of the National Academy of Sciences, 117(52):33117-33123, 2020.
Appendix to OKRidge Scalable Optimal k-Sparse Ridge Regression
A Related Work 15
B Background Concepts 17
C Derivations and Proofs 18
C.1 Proof of Theorem 3.1 18
C.2 Proof of Theorem 3.2 20
C.3 Proof of Theorem 3.3 23
C.4 Proof of Theorem 3.4 25
C.5 Derivation of  $g(a,b) = \max_cac - \frac{c^2}{4} b$  via Fenchel Conjugate 30
D Existing MIP Formulations for  $k$ -sparse Ridge Regression 32
E Algorithmic Charts 34
E.1 Overall Branch-and-bound Framework 34
E.2 Fast Lower Bound Calculation 36
E.3Refine the Lower Bound through the ADMM method. 37
E.4 (Optional) An alternative to the Lower Bound Calculation via Subgradient Ascent and the CMF method 38
E.5 Beam Search Algorithm 39
E.6 Branching 41
E.7 A Note on the Connection with the Interlacing Eigenvalue Property 41
F More Experimental Setup Details 42
F.1 Computing Platform 42
F.2Baselines,Licensees,and Links 42
F.3 Differential Equation Experiments 42
G More Experimental Results on Synthetic Benchmarks 44
G.1 Results Summarization 44
G.2 Comparison with SOS1, Big-M, Perspective, and Eigen-Perspective Formulations with different  $\ell_2$  Regularizations 46
G.3 Comparison with the Big-M Formulation with different  $M$  Values 106
G.4 Comparison with MOSEK Solver,SubsetSelectionCIO,and L0BNB. 143
G.5 Experiments on the Case  $p > > n$  173
G.6 Comparison between Fast Solve and ADMM 176
H More Experimental Results on Dynamical Systems 177
H.1 Results Summarization 177
H.2 Direct Comparison of OKRidge with MIOSR 178
H.3 Comparison of OKRidge with SOS1, Big-M, Perspective, and Eigen-Perspective Formulations 179
H.4 Comparison of Our Beam Search with More Heuristic Baselines 180
H.5 Comparison of OKRidge with MIPs Warmstarted by Our Beam Search 182
H.6 Comparison of OKRidge with MOSEK solver, SubsetSelectionCIO, and L0BNB 183
Sparse Identification of Nonlinear Dynamical Systems: The Sparse Identification of Nonlinear Dynamical Systems (SINDy) framework [19] has been widely adopted for discovering dynamical systems from observed data. The basic framework consists of approximating derivatives, picking a library of features based on positional data, and performing sparse regression on the resulting design matrix. SINDy has expanded to solving PDEs [54, 46] or implicit equations [41], noisy or low data settings [31, 47], and constrained problems [25, 9]. The basic framework can be summarized as a standard regression problem:
$$
\dot {\boldsymbol {X}} = \Theta (\boldsymbol {X}) \boldsymbol {\Xi}, \tag {29}
$$
where the goal is to find  $\Xi = [\beta_1, \dots, \beta_d]$ , with  $d$  being the number of dimensions of the dynamical system,  $\mathbf{X}$  being the observed data, and  $\Theta$  is a map from observed data to candidate functions. The problem is typically solved independently across dimensions, yielding  $d$  sparse regression problems.
Many methods in the SINDy framework solve regression problems with a greedy backwards selection approach. These methods have been shown to outperform LASSO [25] and Orthogonal Matching Pursuit [16] in learning dynamical systems. The backwards selection approach typically solves a non-sparse regression problem, and then either removes all small coefficients below a threshold, or remove one feature at a time until a desired sparsity level is reached [54]. Although these methods have performed well with reasonable run times, they struggle to identify true dynamics as the number of candidate features increases.
Advances in MIP formulations have led to solving SINDy problems with greater success and ability to verify optimality [9]; however, they tend not to scale as well. The method we present also verifies optimality and is much faster than the MIP formula, particularly for larger number of features  $p$ .
Heuristic Methods: Greedy methods aim to solve the problem to near-optimality, with few guarantees. One direction is greedy pursuit (i.e., forward selection), where one coordinate at a time is added until the required support size is reached [56, 13, 50, 51, 22, 29]. Another direction is iterative thresholding, where gradient descent steps alternate with projection steps to satisfy the support size requirement [14, 40, 11, 59, 57]. Other approaches include randomized rounding on the solution of the boolean-relaxed problem [53], swapping features [61, 6, 38, 63, 44], or solving a smaller problem optimally on a working set [60]. These heuristic solutions may greatly improve the computational speed of MIP solvers when used as warm starts. Typically, the heuristic algorithm starts from scratch each time that it runs, which is slow when running it repeatedly throughout the BnB tree. Our insight is that the search history of heuristic methods in previously solved nodes can be used to speed up the heuristic method in the current node.
Optimal Methods and Lower Bound Calculation: In order to certify optimality for this NP-hard problem, we need to perform branch-and-bound and calculate the lower bound for each node in the BnB tree. For the lower bound calculation, early works include the SOS1 formulation or the big-M formulation [10, 9]. Both formulations can be implemented in a commercial solver. However, the SOS1 formulation is not scalable to high dimensions, while the big-M formulation is sensitive to the hyperparameter "M" used to balance scalability and correctness. To circumvent this problem, SubsetSelectionCIO [11] formulate the least-squares term using the Fenchel duality with dual variables. After the formulation, SubsetSelectionCIO applies callbacks to add cutting planes to get a lower bound. Although OKRidge also applies the Fenchel duality, we apply the Fenchel duality on the  $\ell_2$  regularization term. We are solving the problem in the feature space, while SubsetSelectionCIO solves the problem in the sample space. As pointed in [39], the branch-and-cut method in SubsetSelectionCIO [11] runs slow when the  $\ell_2$  regularization is small. Recently, the perspective formulation [32, 34, 62, 28, 4, 59, 5, 37] of the  $\ell_2$  term has also been used. Through convex relaxation, the lower bound can be obtained by solving a quadratically constrained quadratic program (QCQP) with rotated second-order cone constraints. However, the conic formulation is still computationally intensive for large-scale datasets and has difficulty attaining optimal solutions. Our work builds upon the perspective formulation, but we propose an efficient way to calculate the lower bound through Fenchel duality. Another line of works focuses on optimal perspective formulation [62, 28, 37]. This requires solving a semidefinite programming (SDP) problem at each node, which has been shown not scalable to high dimensions [28]. What is more, MI-SDP is not supported by Gurobi. Our work is related to the optimal perspective formulation, but we set the
diagonal matrix  $\mathrm{diag}(d) = \lambda_{\min}(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{I}$ . We call this the eigen-perspective formulation. Although this is not the optimal choice, it is a good approximation and is supported by Gurobi through the QCQP formulation as discussed above. Lastly, there is a recent work called 10bnb [39], which also implements a customized branch-and-bound framework without the commercial solvers. However, 10bnb is solving the  $l_0$ -regularized problem, which is not the same as the  $\ell_0$ -constrained problem. To get a solution with the specified sparsity, 10bnb needs to solve the problem multiple times with different  $\ell_0$  regularizations.
B Background Concepts
Definition B.1 (Strong Convexity [20]). A function  $f: \mathbb{R}^p \to \mathbb{R}$  is strongly convex with parameter  $\alpha > 0$  if it satisfies the following subgradient inequality:
$$
f (\boldsymbol {y}) \geq f (\boldsymbol {x}) + \nabla f (\boldsymbol {x}) ^ {T} (\boldsymbol {y} - \boldsymbol {x}) + \frac {\alpha}{2} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} \tag {30}
$$
for  $\pmb{x}, \pmb{y} \in \mathbb{R}^p$ . If the gradient  $\nabla f$  does not exist but the subgradient exists, then we have