Title: OKRidge: Scalable Optimal k-Sparse Ridge Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Sparse Ridge Regression

Dynamical Systems
Abstract: We consider an important problem in scientific discovery, namely identifying sparse governing equations for nonlinear dynamical systems. This involves solving sparse ridge regression problems to provable optimality in order to determine which terms drive the underlying dynamics. We propose a fast algorithm, OKRidge, for sparse ridge regression, using a novel lower bound calculation involving, first, a saddle point formulation, and from there, either solving (i) a linear system or (ii) using an ADMM-based approach, where the proximal operators can be efficiently evaluated by solving another linear system and an isotonic regression problem. We also propose a method to warm-start our solver, which leverages a beam search. Experimentally, our methods attain provable optimality with run times that are orders of magnitude faster than those of the existing MIP formulations solved by the commercial solver Gurobi.

OKRidge: Scalable Optimal k-Sparse Ridge Regression

Abstract

Weconsider an important problem in scientific discovery, namely identifying sparse governing equations for nonlinear dynamical systems. This involves solving sparse ridge regression problems to provable optimality in order to determine which terms drive the underlying dynamics. We propose a fast algorithm, OKRidge, for sparse ridge regression, using a novel lower bound calculation involving, first, a saddle point formulation, and from there, either solving (i) a linear system or (ii) using an ADMM-based approach, where the proximal operators can be efficiently evaluated by solving another linear system and an isotonic regression problem. We also propose a method to warm-start our solver, which leverages a beam search. Experimentally, our methods attain provable optimality with run times that are orders of magnitude faster than those of the existing MIP formulations solved by the commercial solver Gurobi.

1 Introduction

We are interested in identifying sparse and interpretable governing differential equations arising from nonlinear dynamical systems. These are scientific machine learning problems whose solution involves sparse linear regression. Specifically, these problems require the exact solution of sparse regression problems, with the most basic being sparse ridge regression:

where k specifies the number of nonzero coefficients for the model. This formulation is general, but in the case of nonlinear dynamical systems, the outcome y is a derivative (usually time or spatial) of each dimension x . Here, we assume that the practitioner has included the true variables, along with many other possibilities, and is looking to determine which terms (which transformations of the variables) are real and which are not. This problem is NP-hard [49], and is more challenging in the presence of highly correlated features. Selection of correct features is vital in this context, as many solutions may give good results on training data, but will quickly deviate from the true dynamics when extrapolating past the observed data due to the chaotic nature of complex dynamical systems.

Both heuristic and optimal algorithms have been proposed to solve these problems. Heuristic methods include greedy sequential adding of features [25, 16, 54, 22] or ensemble [31] methods. These methods are fast, but often get stuck in local minima, and there is no way to assess solution quality due to the lack of a lower bound on performance. Optimal methods provide an alternative, but are slow since they must prove optimality. MIOSR [9], a mixed-integer programming (MIP) approach, has been able to certify optimality of solutions given enough time. Slow solvers cause difficulty in performing cross-validation on λ 2 (the ℓ 2 regularization coefficient) and k (sparsity level).

We aim to solve sparse ridge regression to certifiable optimality, but in a fraction of the run time. We present a fast branch-and-bound (BnB) formulation, OKRidge. A crucial challenge is obtaining a

tight and feasible lower bound for each node in the BnB tree. It is possible to calculate the lower bound via the SOS 1 [9], big-M [10], or the perspective formulations (also known as the rotated second-order cone constraints) [32, 4, 59]; the mixed-integer problems can then be solved by a MIP solver. However, these formulations do not consider the special mathematical structure of the regression problem. To calculate a lower bound more efficiently, we first propose a new saddle point formulation for the relaxed sparse ridge regression problem. Based on the new saddle point formulation, we propose two novel methods to calculate the lower bound. The first method is extremely efficient and relies on solving only a linear system of equations. The second method is based on ADMM and can tighten the lower bound given by the first method. Together, these methods give us a tight lower bound, used to prune nodes and provide a small optimality gap. Additionally, we propose a method based on beam-search [58] to get a near-optimal solution quickly, which can be a starting point for both our algorithm and other MIP formulations. Unlike previous methods, our method uses a dynamic programming approach so that previous solutions in the BnB tree can be used while exploring the current node, giving a massive speedup. In summary, our contributions are:

(1) We develop a highly efficient customized branch-and-bound framework for achieving optimality in k -sparse ridge regression, using a novel lower bound calculation and heuristic search.

(2) To compute the lower bound, we introduce a new saddle point formulation, from which we derive two efficient methods (one based on solving a linear system and the other on ADMM).

(3) Our warm-start method is based on beam-search and implemented in a dynamic programming fashion, avoiding redundant calculations. We prove that our warm-start method is an approximation algorithm with an exponential factor tighter than previous work.

On benchmarks, OKRidge certifies optimality orders of magnitude faster than the commercial solver Gurobi. For dynamical systems, our method outperforms the state-of-the-art certifiable method by finding superior solutions, particularly in high-dimensional feature spaces.

2 Preliminary: Dual Formulation via the Perspective Function

There is an extensive literature on this topic, and a longer review of related work is in Appendix A. If we ignore the constant term y T y , we can rewrite the loss objective in Equation (1) as:

with p as the number of features. We are interested in the following optimization problem:

where k is the number of nonzero coefficients. With the sparsity constraint, the problem is NP-hard. The constraint (1 -z j ) β j in Problem (3) can be reformulated with the SOS1, big-M, or the perspective formulation (with quadratic cone constraints), which then can be solved by a MIP solver. Since commercial solvers do not exploit the special structure of the problem, we develop a customized branch-and-bound framework.

For any function f ( a ) , the perspective function is g ( a, b ) := bf ( a b ) for the domain b > 0 [32, 34, 26] and g ( a, b ) = 0 otherwise. Applying to f ( a ) = a 2 , we obtain another function g ( a, b ) = a 2 b . As shown by [4], replacing the loss term β 2 j and constraint (1 -z j ) β j = 0 with the perspective formula β 2 j /z j in Problem (3) would not change the optimal solution. By the Fenchel conjugate [4], g ( · , · ) can be rewritten as g ( a, b ) = max c ac -c 2 4 b . If we define a new perspective loss as:

then we can reformulate Problem (3) as:

If we relax the binary constraint { 0 , 1 } to the interval [0 , 1] and swap max and min (no duality gap, as pointed by [4]), we obtain the dual formulation for the convex relaxation of Problem (5):

While [4] uses the perspective formulation for safe feature screening, we use it to calculate a lower bound for Problem (3). However, directly solving the maxmin problem is computationally challenging. In Section 3.1, we propose two methods to achieve this in an efficient way.

3 Methodology

We propose a custom BnB framework to solve Problem (3). We have 3 steps to process each node in the BnB tree. First, we calculate a lower bound of the node, using two algorithms proposed in the next subsection. If the lower bound exceeds or equals the current best solution, we have proven that it does not lead to any optimal solution, so we prune the node. Otherwise, we go to Step 2, where we perform beam-search to find a near-optimal solution. In Step 3, we use the solution from Step 2 and propose a branching strategy to create new nodes in the BnB tree. We continue until reaching the optimality gap tolerance. Below, we elaborate on each step. In Appendix E, we provide visual illustrations of BnB and beam search as well as the complete pseudocodes of our algorithms.

3.1 Lower Bound Calculation

Tight Saddle Point Formulation

We first rewrite Equation (2) with a new hyperparameter λ :

where Q λ := X T X -λ I . We restrict λ ∈ [0 , λ min ( X T X )] , where λ min ( · ) denotes the minimum eigenvalue of a matrix. We see that Q λ is positive semidefinite, so the first term remains convex. This trick is related to the optimal perspective formulation [62, 28, 37], but we set the diagonal matrix diag ( d ) in [28] to be λ I . We call this trick the eigen-perspective formulation. The optimal perspective formulation requires solving semidefinite programming (SDP) problems, which have been shown not scalable to high dimensions [28], and MI-SDP is not supported by Gurobi.

Solving Problem (3) is equivalent to solving the following problem:

We get a continuous relaxation of Problem (3) if we relax { 0 , 1 } to [0 , 1] .

We can now define a new loss analogous to the loss defined in Equation (4):

Then, the dual formulation analogous to Problem (6) is:

Solving Problem (10) provides us with a lower bound to Problem (8). More importantly, this lower bound becomes tighter as λ increases. This novel formulation is the starting point for our work.

We next propose a reparametrization trick to simplify the optimization problem above. For the inner optimization problem in Problem (10), given any c , the optimality condition for β is (take the gradient with respect to β and set the gradient to 0 ):

Inspired by this optimality condition, we have the following theorem:

Theorem 3.1. If we reparameterize c = 2 λ 2 + λ ( X T y -Q λ γ ) with a new parameter γ , then Problem (10) is equivalent to the following saddle point optimization problem:

and diag ( z ) is a diagonal matrix with z on the diagonal.

To our knowledge, this is the first time this formulation is given. Solving the saddle point formulation to optimality in Problem (12) gives us a tight lower bound. However, this is computationally hard.

Our insight is that we can solve Problem (12) approximately while still obtaining a feasible lower bound. Let us define a new function h ( γ ) as short-hand for the inner minimization in Problem (12):

For any arbitrary γ ∈ R p , h ( γ ) is a valid lower bound for Problem (3). We should choose γ such that this lower bound h ( γ ) is tight. Below, we provide two efficient methods to calculate such a γ .

Fast Lower Bound Calculation

First, we provide a fast way to choose γ . The choice of γ is motivated by the following theorem:

Theorem 3.2. The function h ( γ ) defined in Equation (14) is lower bounded by

Furthermore, the right-hand size of Equation (15) is maximized if γ = ˆ γ = argmin α L ridge ( α ) , where in this case, h ( γ ) evaluated at ˆ γ becomes

where SumBottom p -k ( · ) denotes the summation of the smallest p -k terms of a given set.

Here we provide an intuitive explanation of why h (ˆ γ ) is a valid lower bound. Note that the ridge regression loss is strongly convex. Assuming that the strongly convex parameter is µ (see Appendix B), by the strong convexity property, we have that for any γ ∈ R p ,

Because ˆ γ minimizes L ridge ( · ) , we have ∇L ridge (ˆ γ ) = 0 . For the k -sparse vector γ with ∥ γ ∥ 0 ≤ k , the minimum for the right-hand side of Inequality (17) can be achieved if γ j = ˆ γ j for the top k terms of ˆ γ 2 l 's. This ensures the bound applies for all k -sparse γ . Thus, the k -sparse ridge regression loss is lower-bounded by

for γ ∈ R p with ∥ γ ∥ 0 ≤ k . For ridge regression, the strong convexity µ parameter can be chosen from [0 , 2( λ 2 + λ min ( X T X ))] . If we let µ = 2( λ 2 + λ ) , we obtain h (ˆ γ ) in Theorem 3.2.

The lower bound h (ˆ γ ) can be calculated extremely efficiently by solving the ridge regression problem (solving the linear system ( X T X + λ 2 I ) γ = X T y for γ ) and adding the extra p -k terms. However, this bound is not the tightest we can achieve. In the next subsection, we discuss how to apply ADMM to maximize h ( γ ) further based on Equation (14).

Tight Lower Bound via ADMM

Let us define p := X T y -Q λ γ . Starting from Problem (12), if we minimize z in the inner optimization under the constraints ∑ p j =1 z j ≤ k and z j ∈ [0 , 1] for ∀ j , we have z j = 1 for the top k terms of p 2 j and z j = 0 otherwise. Then, Problem (12) can be reformulated as follows:

where F ( γ ) := γ T Q λ γ and G ( p ) := 1 λ 2 + λ SumTop k ( { p 2 j } ) . The solution to this problem is a dense vector that can be used to provide a lower bound on the original k -sparse problem. This problem can be solved by the alternating direction method of multipliers (ADMM) [17]. Here, we apply the iterative algorithm with the scaled dual variable q [33]:

where α is the relaxation factor, and ρ is the step size.

It is known that ADMM suffers from slow convergence when the step size is not properly chosen. According to [33], to ensure the optimized linear convergence rate bound factor, we can pick α = 1 and ρ = 2 √ λ max ( Q λ ) λ min > 0 ( Q λ ) 1 , where λ max ( · ) denotes the largest eigenvalue of a matrix, and λ min > 0 ( · ) denotes the smallest positive eigenvalue of a matrix.

Having settled the choices for the relaxation factor α and the step size ρ , we are left with the task of solving Equation (19) and Equation (21) (also known as evaluating the proximal operators [52]). Interestingly, Equation (19) can be evaluated by solving a linear system while Equation (21) can be evaluated by recasting the problem as an isotonic regression problem.

Theorem 3.3. Let F ( γ ) = γ T Q λ γ and G ( p ) = 1 λ 2 + λ SumTop k ( { p 2 j } ) . Then the solution for the problem γ t +1 = argmin γ F ( γ ) + ρ 2 ∥ Q λ γ + p t -X T y + q t ∥ 2 2 is

Furthermore, let a = X T y -θ t +1 -q t and J be the indices of the top k terms of {| a j |} . The solution for the problem p t +1 = argmin p G ( p )+ ρ 2 ∥ θ t +1 + p -X T y + q t ∥ 2 2 is p t +1 j = sign ( a j ) · ˆ v j ,

where

Problem (24) is an isotonic regression problem and can be efficiently solved in linear time [12, 21].

3.2 Beam-Search as a Heuristic

After finishing the lower bound calculation in Section 3.1, we next explain how to quickly reduce the upper bound in the BnB tree. We discuss how to add features, keep good solutions, and use dynamic programming to improve efficiency. Lastly, we give a theoretical guarantee on the quality of our solution.

Starting from the vector 0 , we add one coordinate at a time into our support until we reach a solution with support size k . At each iteration, we pick the coordinate that results in the largest decrease in the ridge regression loss while keeping coefficients in the existing support fixed:

where X : j denotes the j -th column of X , and the right-hand side uses an analytical solution for the line-search for α . This is similar to the sparse-simplex algorithm [6]. However, after adding a feature, we adjust the coefficients restricted on the new support by minimizing the ridge regression loss.

The above idea does not handle highly correlated features well. Once a feature is added, it cannot be removed [61]. To alleviate this problem, we use beam-search [58, 43], keeping the best B solutions at each stage of support expansion:

where j ∗ ∈ arg BottomB j means j ∗ belongs to the set of solutions whose loss is one of the B smallest losses. Afterwards, we finetune the solution on the newly expanded support and choose the best B solutions for the next stage of support expansion. A visual illustration of beam search can be found in Figure 6 in Appendix E, which also contains the detailed algorithm.

Although many methods have been proposed for sparse ridge regression, none of them have been designed with the BnB tree structure in mind. Our approach is to take advantage of the search history of past nodes to speed up the search process for a current node. To achieve this, we follow a dynamic programming approach by saving the solutions of already explored support sets. Therefore, whenever we need to adjust coefficients on the new support during beam search, we can simply retrieve the coefficients from the history if a support has been explored in the past. Essentially, we trade memory space for computational efficiency.

3.2.1 Provable Guarantee

Lastly, using similar methods to [29], we quantify the gap between our found heuristic solution ˆ β and the optimal solution β ∗ in Theorem 3.4. Compared with Theorem 5 in [29], we improve the factor in the exponent from m 2 k M 2 k to m 2 k M 1 (since M 1 ≤ M 2 k , where M 1 and M 2 k are defined in [29]).

Theorem 3.4. Let us define a k -sparse vector pair domain to be Ω k := { ( x , y ) ∈ R p × R p : ∥ x ∥ 0 ≤ k, ∥ y ∥ 0 ≤ k, ∥ x -y ∥ 0 ≤ k } . Any M 1 satisfying f ( y ) ≤ f ( x )+ ∇ f ( x ) T ( y -x )+ M 1 2 ∥ y -x ∥ 2 2 for all ( x , y ) ∈ Ω 1 is called a restricted smooth parameter with support size 1, and any m 2 k satisfying f ( y ) ≥ f ( x ) + ∇ f ( x ) T ( y -x ) + m 2 k 2 ∥ y -x ∥ 2 2 for all ( x , y ) ∈ Ω 2 k is called a restricted strongly convex parameter with support size 2 k . If ˆ β is our heuristic solution by the beam-search method, and β ∗ is the optimal solution, then:

3.3 Branching and Queuing

Branching: The most common branching techniques include most-infeasible branching and strong branching [2, 1, 15, 7]. However, these two techniques require having fractional values for the binary variables z j 's, which we do not compute in our framework. Instead, we propose a new branching strategy based on our heuristic solution ˆ β : we branch on the coordinate whose coefficient, if set to 0 , would result in the largest increase in the ridge regression loss L ridge (See Appendix E for details):

The intuition is that the coordinate with the largest increase in L ridge potentially plays a significant role, so we want to fix such a coordinate as early as possible in the BnB tree.

Queuing: Besides the branching strategy, we need a queue to pick a node to explore among newly created nodes. Here, we use a breadth-first approach, evaluating nodes in the order they are created.

(p=0.1)

(p=0.5)

(p=0.9)

103

103

103

(sec)

101

101.

101

time

10-1

10-1

10-1

100

500

1000

3000

5000!

100

500

1000

3000

50001

100

500

1000

3000

5000

1.00

1.00

1.00

0.75

0.75

0.75

0.50

10.50

10.50

0.25

i0.25

0.25

0.00

'0.00

'0.00

100

500

1000

3000

5000

100

500

1000

3000

5000

100

500

1000

3000

5000

#of features

#of features

#of features

SOS1+warm start

big-M+ warm start

persp.+warm start

ours

(p=0.1)

(p=0.5)

(6:0=d)

103

103

103

(sec)

102

102

102

time

101

101

101

3000

4000

5000

6000

70001

3000

4000

5000

6000

70001

3000

4000

5000

6000

7000

20

20

20

15

15

15

10

10

10

5

5

5

0

0

0

3000

4000

5000

6000

7000

3000

4000

5000

6000

7000

3000

4000

5000

6000

7000

#of samples

#of samples

#of samples

SOS1+ warm start

big-M+warm start

persp.+ warm start

ours

4 Experiments

We test the effectiveness of our OKRidge on synthetic benchmarks and sparse identification of nonlinear dynamical systems (SINDy)[19]. Our main focus is: assessing how well our proposed lower bound calculation speeds up certification (Section 4.1), and evaluating solution quality of OKRidge on challenging applications (Section 4.2). Additional extensive experiments are in Appendix G and H. Our algorithms are written in Python. Any improvements we see over commercial MIP solvers, which are coded in C/C++, are solely due to our specialized algorithms.

Hopf

MHD

True Positivity Rate

1.0

1.0

1.0

0.8

0.8

0.8

0.6-

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

L2 Coefficient Error

100

100

101

10

2

10-1

10-1

10

-3

10-2

104

Train RMSE

101

10-2

100

101

10-1

100

103

Test RMSE

10-1

101

101

10-2

100

10-3

T

10-1

100

101

100

101

100

101

Training Trajectory Length (s)

Training Trajectory Length (s)

Training Trajectory Length (s)

MIOSR

STLSQ

SSR

E-STLSQ

ours

4.1 Assessing How Well Our Proposed Lower Bound Calculation Speeds Up Certification

Here, we demonstrate the speed of OKRidge for certifying optimality compared to existing MIPs solved by Gurobi [35]. We set a 1-hour time limit and an optimality gap of relative tolerance 10 -4 .

We use a value of 0 . 001 for λ 2 . Our 4 baselines include MIPs with SOS1, big-M ( M = 50 to prevent cutting off optimal solutions), perspective [4], and eigen-perspective formulations ( λ = λ min ( X T X ) ) [28]. In the main text, we use plots to present the results. In Appendix G, we present the results in tables. Additionally, in Appendix G, we conduct perturbation studies on λ 2 ( λ 2 = 0 . 1 and λ 2 = 10 ) and M ( M = 20 and M = 5 ). Finally, still in Appendix G, we also compare OKRidge with other MIPs including the MOSEK solver [3], SubsetSelectionCIO [11], and L0BNB [39].

Similar to the data generation process in [11, 48], we first sample x i ∈ R p from a Gaussian distribution N ( 0 , Σ) with mean 0 and covariance matrix Σ , where Σ ij = ρ | i -j | . Variable ρ controls the feature correlation. Then, we create the coefficient vector β ∗ with k nonzero entries, where β ∗ j = 1 if j mod ( p/k ) = 0 . Next, we construct the prediction y i = x T i β ∗ + ϵ i , where ϵ i i.i.d. ∼ N (0 , ∥ X β ∗ ∥ 2 2 SNR ) . SNR stands for signal-to-noise ratio (SNR), and we choose SNR to be 5 in all our experiments.

In the first setting, we fix the number of samples with n = 100000 and vary the number of features p ∈ { 100 , 500 , 1000 , 3000 , 5000 } and correlation levels ρ ∈ { 0 . 1 , 0 . 5 , 0 . 9 } (See Appendix G for ρ = 0 . 3 and ρ = 0 . 7 ). We warm-started the MIP solvers by our beam-search solutions. The results can be seen in Figure 1. From both figures, we see that OKRidge outperforms all existing MIPs solved by Gurobi, usually by orders of magnitude.

In the second setting, we fix the number of features to p = 3000 and vary the number of samples n ∈ { 3000 , 4000 , 5000 , 6000 , 7000 } and the correlation levels ρ ∈ { 0 . 1 , 0 . 5 , 0 . 9 } (see Appendix G for ρ = 0 . 3 and ρ = 0 . 7 ). As in the first setting, we also warm-started the MIP solvers by our beam-search solutions. The results are in Figure 2. When n is close to p or the correlation is high

( ρ = 0 . 9 ), no methods can finish within the 1-hour time limit, but OKRidge prunes the search space well and achieves the smallest optimality gap. When n becomes larger in the case of ρ = 0 . 1 and ρ = 0 . 5 , OKRidge runs orders of magnitude faster than all baselines .

4.2 Evaluating Solution Quality of OKRidge on Challenging Applications

On previous synthetic benchmarks, many heuristics (including our beam search method) can find the optimal solution without branch-and-bound. In this subsection, we work on more challenging scenarios (sparse identification of differential equations). We replicate the experiments in [9] using three dynamical systems from the PySINDy library [27, 42]: Lorenz System, Hopf Bifurcation, and magnetohydrodynamical (MHD) model [24]. The Lorenz System is a 3-D system with the nonlinear differential equations:

where we use standard parameters σ = 10 , β = 8 / 3 , ρ = 28 . The true sparsities for each dimension are (2 , 3 , 2) . The Hopf Bifurcation is a 2-D system with nonlinear differential equations:

where we use the standard parameters µ = -0 . 05 , ω = 1 , A = 1 . The true sparsities for each dimension are (4 , 4) . Finally, the MHD is a 6-D system with the nonlinear differential equations:

The true sparsities for each dimension are (2 , 2 , 2 , 2 , 2 , 2) .

We use all monomial features (candidate functions) up to 5th order interactions. This results in 56 functions for the Lorentz System, 21 for Hopf Bifurcation, and 462 for MHD. Due to the high-order interaction terms, features are highly correlated, resulting in poor performance of heuristic methods.

4.2.1 Baselines and Experimental Setup

In addition to MIOSR (which relies on the SOS1 formulation), we also compare with three common baselines in the SINDy literature: STLSQ [54], SSR [16], and E-STLSQ [31]. The baseline SR3 [25] is not included since the previous literature [9] shows it performs poorly. We compare OKRidge with other baselines using the SINDy library [27, 42]. We follow the experimental setups in [9] for model selection, hyperparameter choices, and evaluation metrics (please see Appendix F for details). In Appendix H, we provide additional experiments on Gurobi with different MIP formulations and comparing with more heuristic baselines.

4.2.2 Results

Figure 3 displays the results. OKRidge (red curves) outperforms all baselines, including MIOSR (blue curves), across evaluation metrics. On the Lorenz System, all methods recover the true feature support when the training trajectory is long enough. When the training trajectory length is short, i.e., the left part of each subplot, (or equivalently, when the number of samples is small), OKRidge performs uniformly better than all other baselines. On the Hopf Bifurcation, all heuristic methods fail to recover the true support, resulting in poor performance. On the final MHD, OKRidge maintains the top performance and outperforms MIOSR on the true positivity rate. This demonstrates the effectiveness of OKRidge, which incurs lower runtimes and yields better metric scores under high-dimensional settings. The highest runtimes are incurred for the MHD (with 462 candidate functions/features), which are shown in Figure 4.

Limitations of OKRidge When the feature dimension is low (under 100s), Gurobi can solve the problem to optimality faster than OKRidge. This is observed on the synthetic benchmarks ( p = 100 ) and also on the Hopf Bifurcation ( p = 21 ). Since Gurobi is a commercial proprietary solver, we cannot inspect the details of its sophisticated implementation. Gurobi may resort to an enumeration/brute-force approach, which could be faster than spending time to calculate lower bounds in the BnB tree. This being said, OKRidge is still competitive with Gurobi in the low-dimensional setting, and OKRidge scales favorably in high-dimensional settings.

5 Conclusion

We presented a method for optimal sparse ridge regression that leverages a novel tight lower bound on the objective. We showed that the method is both faster and more accurate than existing approaches for learning differential equations - a key problem in scientific discovery. This tool (unlike its main competitor) does not require proprietary software with expensive licenses and can have a significant impact on various regression applications.

Code Availability

Implementations of OKRidge discussed in this paper are available at https://github.com/ jiachangliu/OKRidge .

References

T. Achterberg, T. Koch, and A. Martin. Branching rules revisited. Operations Research Letters , 33(1):42-54, 2005.

D. Applegate, R. Bixby, V. Chvátal, and W. Cook. On the solution of traveling salesman problems. Documenta Mathematica , pages 645-656, 1998.

M. ApS. Mosek optimizer API for python. Version , 9(17):6-4, 2022.

A. Atamturk and A. Gómez. Safe screening rules for l0-regression from perspective relaxations. In International Conference on Machine Learning , pages 421-430. PMLR, 2020.

A. Atamtürk, A. Gómez, and S. Han. Sparse and smooth signal estimation: Convexification of l0-formulations. Journal of Machine Learning Research , 22:52-1, 2021.

A. Beck and Y. C. Eldar. Sparsity constrained nonlinear optimization: Optimality conditions and algorithms. SIAM Journal on Optimization , 23(3):1480-1509, 2013.

P. Belotti, C. Kirches, S. Leyffer, J. Linderoth, J. Luedtke, and A. Mahajan. Mixed-integer nonlinear optimization. Acta Numerica , 22:1-131, 2013.

D. Bertsekas. Convex optimization theory , volume 1. Athena Scientific, 2009.

D. Bertsimas and W. Gurnee. Learning sparse nonlinear dynamics via mixed-integer optimization. Nonlinear Dynamics , Jan 2023.

D. Bertsimas, A. King, and R. Mazumder. Best subset selection via a modern optimization lens. The Annals of Statistics , 44(2):813-852, 2016.

D. Bertsimas, J. Pauphilet, and B. Van Parys. Sparse regression: Scalable algorithms and empirical performance. Statistical Science , 35(4):555-578, 2020.

M. J. Best and N. Chakravarti. Active set algorithms for isotonic regression; a unifying framework. Mathematical Programming , 47(1-3):425-439, 1990.

T. Blumensath and M. E. Davies. Gradient pursuits. IEEE Transactions on Signal Processing , 56(6):2370-2382, 2008.

T. Blumensath and M. E. Davies. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis , 27(3):265-274, 2009.

P. Bonami, J. Lee, S. Leyffer, and A. Wächter. More branch-and-bound experiments in convex nonlinear integer programming. Preprint ANL/MCS-P1949-0911, Argonne National Laboratory, Mathematics and Computer Science Division , 91, 2011.

L. Boninsegna, F. Nüske, and C. Clementi. Sparse learning of stochastic dynamical equations. The Journal of Chemical Physics , 148(24):241723, June 2018.

S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning , 3(1):1-122, 2011.

S. Boyd, L. Xiao, and A. Mutapcic. Subgradient methods. lecture notes of EE392o, Stanford University, Autumn Quarter , 2004:2004-2005, 2003.

S. L. Brunton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences , 113(15):3932-3937, 2016.

S. Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8(3-4):231-357, 2015.

F. M. Busing. Monotone regression: A simple and fast o ( n ) PAVA implementation. Journal of Statistical Software , 102:1-25, 2022.

T. T. Cai and L. Wang. Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Transactions on Information Theory , 57(7):4680-4688, 2011.

P. M. Camerini, L. Fratta, and F. Maffioli. On improving relaxation methods by modified gradient techniques. In Nondifferentiable Optimization , pages 26-34. Springer Berlin Heidelberg, 1975.

V. Carbone and P. Veltri. Relaxation processes in magnetohydrodynamics - A triad-interaction model. Astronomy and Astrophysics , 259(1):359-372, June 1992.

K. Champion, P. Zheng, A. Y. Aravkin, S. L. Brunton, and J. N. Kutz. A unified sparse optimization framework to learn parsimonious physics-informed models from data. IEEE Access , 8:169259-169271, 2020.

P. L. Combettes. Perspective functions: Properties, constructions, and examples. Set-Valued and Variational Analysis , 26(2):247-264, 2018.

B. de Silva, K. Champion, M. Quade, J.-C. Loiseau, J. Kutz, and S. Brunton. Pysindy: A python package for the sparse identification of nonlinear dynamical systems from data. Journal of Open Source Software , 5(49):2104, 2020.

H. Dong, K. Chen, and J. Linderoth. Regularization vs. relaxation: A conic optimization perspective of statistical variable selection. arXiv preprint arXiv:1510.06083 , 2015.

E. R. Elenberg, R. Khanna, A. G. Dimakis, and S. Negahban. Restricted strong convexity implies weak submodularity. The Annals of Statistics , 46(6B):3539-3568, 2018.

A. Eriksson, T. Thanh Pham, T.-J. Chin, and I. Reid. The k-support norm and convex envelopes of cardinality and rank. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3349-3357, 2015.

U. Fasel, J. N. Kutz, B. W. Brunton, and S. L. Brunton. Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 478(2260), Apr. 2022.

A. Frangioni and C. Gentile. Perspective cuts for a class of convex 0-1 mixed integer programs. Mathematical Programming , 106(2):225-236, 2006.

P. Giselsson and S. Boyd. Linear convergence and metric selection for douglas-rachford splitting and admm. IEEE Transactions on Automatic Control , 62(2):532-544, 2016.

O. Günlük and J. Linderoth. Perspective reformulations of mixed integer nonlinear programs with indicator variables. Mathematical Programming , 124(1):183-205, 2010.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.

W. H. Haemers. Interlacing eigenvalues and graphs. Linear Algebra and its Applications , 226:593-616, 1995.

S. Han, A. Gómez, and A. Atamtürk. The equivalence of optimal perspective formulation and Shor's SDP for quadratic programs with indicator variables. Operations Research Letters , 50(2):195-198, 2022.

H. Hazimeh and R. Mazumder. Fast best subset selection: Coordinate descent and local combinatorial optimization algorithms. Operations Research , 68(5):1517-1537, 2020.

H. Hazimeh, R. Mazumder, and A. Saab. Sparse regression at scale: Branch-and-bound rooted in first-order optimization. Mathematical Programming , 196(1):347-388, 2022.

P. Jain, A. Tewari, and P. Kar. On iterative hard thresholding methods for high-dimensional m -estimation. Advances in Neural Information Processing Systems , 27, 2014.

K. Kaheman, J. N. Kutz, and S. L. Brunton. SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 476(2242), Oct. 2020.

A. A. Kaptanoglu, B. M. de Silva, U. Fasel, K. Kaheman, A. J. Goldschmidt, J. Callaham, C. B. Delahunt, Z. G. Nicolaou, K. Champion, J.-C. Loiseau, J. N. Kutz, and S. L. Brunton. Pysindy: A comprehensive python package for robust sparse system identification. Journal of Open Source Software , 7(69):3994, 2022.

J. Liu, C. Zhong, B. Li, M. Seltzer, and C. Rudin. FasterRisk: Fast and accurate interpretable risk scores. In Advances in Neural Information Processing Systems , 2022.

J. Liu, C. Zhong, M. Seltzer, and C. Rudin. Fast sparse classification for generalized linear and additive models. In Proceedings of Artificial Intelligence and Statistics (AISTATS) , 2022.

N. M. Mangan, J. N. Kutz, S. L. Brunton, and J. L. Proctor. Model selection for dynamical systems via sparse regression and information criteria. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 473(2204):20170009, Aug. 2017.

D. A. Messenger and D. M. Bortz. Weak sindy for partial differential equations. Journal of Computational Physics , 443:110525, 2021.

D. A. Messenger and D. M. Bortz. Weak sindy: Galerkin-based data-driven model selection. Multiscale Modeling & Simulation , 19(3):1474-1497, 2021.

T. Moreau, M. Massias, A. Gramfort, P. Ablin, P.-A. Bannier, B. Charlier, M. Dagréou, T. Dupré la Tour, G. Durif, C. F. Dantas, Q. Klopfenstein, J. Larsson, E. Lai, T. Lefort, B. Malézieux, B. Moufad, B. T. Nguyen, A. Rakotomamonjy, Z. Ramzi, J. Salmon, and S. Vaiter. Benchopt: Reproducible, efficient and collaborative optimization benchmarks. In Advances in Neural Information Processing Systems , 2022.

B. K. Natarajan. Sparse approximate solutions to linear systems. SIAM Journal on Computing , 24(2):227-234, 1995.

D. Needell and J. A. Tropp. Cosamp: Iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis , 26(3):301-321, 2009.

D. Needell and R. Vershynin. Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit. IEEE Journal of Selected Topics in Signal Processing , 4(2):310-316, 2010.

N. Parikh, S. Boyd, et al. Proximal algorithms. Foundations and Trends in Optimization , 1(3):127-239, 2014.

M. Pilanci, M. J. Wainwright, and L. El Ghaoui. Sparse learning via boolean relaxations. Mathematical Programming , 151(1):63-87, 2015.

S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz. Data-driven discovery of partial differential equations. Science Advances , 3(4):e1602614, 2017.

M. E. Sander, J. Puigcerver, J. Djolonga, G. Peyré, and M. Blondel. Fast, differentiable and sparse top-k: a convex analysis perspective. In International Conference on Machine Learning , pages 29919-29936. PMLR, 2023.

J. Tropp. Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory , 50(10):2231-2242, 2004.

R. Vreugdenhil, V. A. Nguyen, A. Eftekhari, and P. M. Esfahani. Principal component hierarchy for sparse quadratic programs. In International Conference on Machine Learning , pages 10607-10616. PMLR, 2021.

S. Wiseman and A. M. Rush. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1296-1306, Austin, Texas, Nov. 2016. Association for Computational Linguistics.

W. Xie and X. Deng. Scalable algorithms for the sparse ridge regression. SIAM Journal on Optimization , 30(4):3359-3386, 2020.

G. Yuan, L. Shen, and W.-S. Zheng. A block decomposition algorithm for sparse optimization. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Data Mining , pages 275-285, 2020.

T. Zhang. Adaptive forward-backward greedy algorithm for learning sparse representations. IEEE Transactions on Information Theory , 57(7):4689-4708, 2011.

X. Zheng, X. Sun, and D. Li. Improving the performance of MIQP solvers for quadratic programs with cardinality and minimum threshold constraints: A semidefinite program approach. INFORMS Journal on Computing , 26(4):690-703, 2014.

J. Zhu, C. Wen, J. Zhu, H. Zhang, and X. Wang. A polynomial algorithm for best-subset selection problem. Proceedings of the National Academy of Sciences , 117(52):33117-33123, 2020.

Appendix to OKRidge Scalable Optimal k-Sparse Ridge Regression

Table of Contents

A Related Work

Sparse Identification of Nonlinear Dynamical Systems: The Sparse Identification of Nonlinear Dynamical Systems (SINDy) framework [19] has been widely adopted for discovering dynamical systems from observed data. The basic framework consists of approximating derivatives, picking a library of features based on positional data, and performing sparse regression on the resulting design matrix. SINDy has expanded to solving PDEs [54, 46] or implicit equations [41], noisy or low data settings [31, 47], and constrained problems [25, 9]. The basic framework can be summarized as a standard regression problem:

where the goal is to find Ξ = [ β 1 , . . . , β d ] , with d being the number of dimensions of the dynamical system, X being the observed data, and Θ is a map from observed data to candidate functions. The problem is typically solved independently across dimensions, yielding d sparse regression problems.

Many methods in the SINDy framework solve regression problems with a greedy backwards selection approach. These methods have been shown to outperform LASSO [25] and Orthogonal Matching Pursuit [16] in learning dynamical systems. The backwards selection approach typically solves a non-sparse regression problem, and then either removes all small coefficients below a threshold, or remove one feature at a time until a desired sparsity level is reached [54]. Although these methods have performed well with reasonable run times, they struggle to identify true dynamics as the number of candidate features increases.

Advances in MIP formulations have led to solving SINDy problems with greater success and ability to verify optimality [9]; however, they tend not to scale as well. The method we present also verifies optimality and is much faster than the MIP formula, particularly for larger number of features p .

Heuristic Methods: Greedy methods aim to solve the problem to near-optimality, with few guarantees. One direction is greedy pursuit (i.e., forward selection), where one coordinate at a time is added until the required support size is reached [56, 13, 50, 51, 22, 29]. Another direction is iterative thresholding, where gradient descent steps alternate with projection steps to satisfy the support size requirement [14, 40, 11, 59, 57]. Other approaches include randomized rounding on the solution of the boolean-relaxed problem [53], swapping features [61, 6, 38, 63, 44], or solving a smaller problem optimally on a working set [60]. These heuristic solutions may greatly improve the computational speed of MIP solvers when used as warm starts. Typically, the heuristic algorithm starts from scratch each time that it runs, which is slow when running it repeatedly throughout the BnB tree. Our insight is that the search history of heuristic methods in previously solved nodes can be used to speed up the heuristic method in the current node.

Optimal Methods and Lower Bound Calculation: In order to certify optimality for this NPhard problem, we need to perform branch-and-bound and calculate the lower bound for each node in the BnB tree. For the lower bound calculation, early works include the SOS1 formulation or the big-M formulation [10, 9]. Both formulations can be implemented in a commercial solver. However, the SOS1 formulation is not scalable to high dimensions, while the big-M formulation is sensitive to the hyperparameter 'M' used to balance scalability and correctness. To circumvent this problem, SubsetSelectionCIO [11] formulate the least-squares term using the Fenchedl duality with dual variables. After the formulation, SubsetSelectionCIO applies callbacks to add cutting planes to get a lower bound. Although OKRidge also applies the Fenchel duality, we apply the Fenchel duality on the ℓ 2 regularization term. We are solving the problem in the feature space, while SubsetSelectionCIO solves the problem in the sample space. As pointed in [39], the branch-and-cut method in SubsetSelectionCIO [11] runs slow when the ℓ 2 regularization is small. Recently, the perspective formulation [32, 34, 62, 28, 4, 59, 5, 37] of the ℓ 2 term has also been used. Through convex relaxation, the lower bound can be obtained by solving a quadratically constrained quadratic program (QCQP) with rotated second-order cone constraints. However, the conic formulation is still computationally intensive for large-scale datasets and has difficulty attaining optimal solutions. Our work builds upon the perspective formulation, but we propose an efficient way to calculate the lower bound through Fenchel duality. Another line of works focuses on optimal perspective formulation [62, 28, 37]. This requires solving a semidefinite programming (SDP) problem at each node, which has been shown not scalable to high dimensions [28]. What is more, MI-SDP is not supported by Gurobi. Our work is related to the optimal perspective formulation, but we set the

diagonal matrix diag ( d ) = λ min ( X T X ) I . We call this the eigen-perspective formulation. Although this is not the optimal choice, it is a good approximation and is supported by Gurobi through the QCQP formulation as discussed above. Lastly, there is a recent work called l0bnb [39], which also implements a customized branch-and-bound framework without the commercial solvers. However, l0bnb is solving the l 0 -regularized problem, which is not the same as the ℓ 0 -constrained problem. To get a solution with the specified sparsity, l0bnb needs to solve the problem multiple times with different ℓ 0 regularizations.

B Background Concepts

Definition B.1 ( Strong Convexity [20]) . A function f : R p → R is strongly convex with parameter α > 0 if it satisfies the following subgradient inequality:

for x , y ∈ R p . If the gradient ∇ f does not exist but the subgradient exists, then we have

for x , y ∈ R p , where g ∈ ∂f and ∂f is the subdifferential of f .

Definition B.2 ( Smoothness [20]) . A continuously differentiable function f : R p → R is smooth with parameter β > 0 if the gradeint ∇ f is β -Lipschitz. Mathematically, this means that:

for x , y ∈ R p .

Lemma B.3 ( Smoothness Property [20]) . Let f : R p → R be a β -smooth function. Then, we have the following sandwich property ( f ( y ) is 'sandwiched' between two quadratic functions):

for x , y ∈ R p .

Definition B.4 ( k -sparse Pair Domain [29]) . A domain Ω k ⊂ R p × R p is a k -sparse vector pair domain if it contains all pairs of k -sparse vectors that differ in at most k entries, i.e ,

Definition B.5 ( Restricted Strong Convexity [29]) . A function f : R p → R is restricted strongly convex with parameter α > 0 on the k -sparse vector pair domain Ω k if it satisfies the following inequality:

for ( x , y ) ∈ Ω k .

Definition B.6 ( Restricted Smoothness [29]) . A function f : R p → R is restrictedly smooth with parameter β > 0 on the k -sparse vector pair domain Ω k if it satisfies the following inequality:

for ( x , y ) ∈ Ω k .

Lemma B.7 ( Gradient of Ridge Regression ) . At β ∈ R p , the gradient of our ridge regression function is given by:

for β ∈ R p .

Proof.

C Derivations and Proofs

C.1 Proof of Theorem 3.1

Theorem 3.1. If we reparameterize c = 2 λ 2 + λ ( X T y -Q λ γ ) with a new parameter γ , then Problem (10) is equivalent to the following saddle point optimization problem:

and diag ( z ) is a diagonal matrix with z on the diagonal.

Before we give the proof for Theorem 3.1, we first state one useful lemma below:

Lemma C.1. For an optimal solution c ∗ to Problem (40) , there exists some β ∗ such that c ∗ = 2 λ 2 + λ ( X T y -Q λ β ∗ ) .

Without using this lemma, there is some concern with reparametrizing c with a new parameter γ when λ = λ min ( X T X ) . In this case, X T X -λ I becomes singular and our reparametrization trick could miss some subspace. This issue of having X T X -λ I being singular would lead to the concern that Problem (39) is a relaxation of Problem (40), not equivalent. Fortunately, due to this lemma, the subspace which we miss via the reparametrization trick will not prevent Problem (39) from achieving the same optimal value as Problem (40).

We now prove this lemma by the method of contradiction.

Proof. Suppose the optimal solution c to Problem (40) cannot be written by Equation (41). Through the rank-nullity theorem in linear algebra, we can find d 1 ∈ col ( Q λ ) and d 2 ∈ ker ( Q λ ) , d 2 = 0 so that c = 2 λ 2 + λ ( X T y -d 1 + d 2 ) , where col ( · ) denotes the column space of a matrix and ker ( · ) denotes the kernel of a matrix. If we let β = α d 2 for some real value α , then the inner optimization of Problem (40) (ignore the last term involving z j 's for now since β and z are separable in the objective function) becomes

̸

Because β = α d 2 , we have β ∈ ker ( Q λ ) and β ⊥ d 1 . Thus, the first two terms become 0 , and we are left with L Fenchel ridge -λ = -2 d T 2 β = -2 α ∥ d 2 ∥ 2 2 . For the inner optimization of Problem (40), because we want to minimize over β , we can let α →∞ , and we will get L Fenchel ridge -λ →-∞ . This is obviously not the optimal value for Problem (40) (a simple counter-example is to let c = 2 λ 2 + λ X T y , and we can easily show that min β , z L Fenchel ridge -λ ( β , z , c ) is finite). Therefore, d 2 must be 0 , and the optimal solution c to Problem (40) must be represented by some β through the equation c = 2 λ 2 + λ ( X T y -Q λ β ) .

Now that we have this lemma, we can prove Theorem 3.1 rigorously. We provide two alternative proofs so that readers can understand the logic from different angles. The first proof is based on using the lemma directly; the second proof is based on the sandwich argument.

Proof 1

Proof. For notational convenience, let us use D z := { z | ∑ p j =1 z j ≤ k, z j ∈ [0 , 1] } as a shorthand for the domain of z .

Recall that Problem (10) is the following optimization problem:

where L Fenchel ridge -λ ( β , z , c ) = β T Q λ β -2 y T Xβ +( λ 2 + λ ) ∑ p j =1 ( β j c j -c 2 j 4 ) z j .

For any fixed c , if we take the derivative of L Fenchel ridge -λ ( β , z , c ) with respect to β and set the derivative to 0 , we obtain the optimality condition for β :

If we reparametrize c with a new parameter γ and let c = 2 λ 2 + λ [ X T y -( X T X -λ I ) γ ] , then β = γ satisfies the optimality condition for β in Equation (41). Therefore, whenever we reparametrize c with c = 2 λ 2 + λ [ X T y -( X T X -λ I ) γ ] , we can just let β = γ , and this ensures us that we can achieve the inner minimum of Equation (40) with respect to β .

Then, we can reduce Problem (40) to Problem (38) in the following way:

max c min β , z L Fenchel ridge -λ ( β , z , c ) # subject to ∑ p j =1 z j ≤ k, z j ∈ [0 , 1] =max c min z ∈D z , β L Fenchel ridge -λ ( β , z , c ) # use D z as the shorhand for the domain of z =max c min z ∈D z , β β T Q λ β -2 y T Xβ +( λ 2 + λ ) p ∑ j =1 ( β j c j -c 2 j 4 z j ) # plug in the details for L Fenchel ridge -λ ( β , z , c ) =max c min z ∈D z γ T Q λ γ -2 y T Xγ +( λ 2 + λ ) p ∑ j =1 ( γ j c j -c 2 j 4 z j ) # letting β = γ achieves the inner minimum for β =max c min z ∈D z γ T Q λ γ -2 y T Xγ +( λ 2 + λ )( γ T c -1 4 c T diag ( z ) c ) # write the summation in vector forms; diag ( z ) denotes a diagonal matrix with z on the diagonal =max γ min z ∈D z γ T Q λ γ -2 y T Xγ +( λ 2 + λ ) ( γ T 2 λ 2 + λ ( X T y -Q λ γ ) -1 4 2 λ 2 + λ ( X T y -Q λ γ ) T diag ( z ) 2 λ 2 + λ ( X T y -Q λ γ ) ) # apply the reparametrization c = 2 λ 2 + λ ( X T y -Q λ γ ) . # remember that due to Lemma C.1, we will not miss the optimal solution c ∗ =max γ min z ∈D z γ T Q λ γ -2 y T Xγ +2 γ T ( X T y -Q λ γ ) -1 λ 2 + λ ( X T y -Q λ γ ) T diag ( z )( X T y -Q λ γ ) # distribute λ 2 + λ inside the parentheses =max γ min z ∈D z γ T Q λ γ -2 y T Xγ +2 γ T X T y -2 γ T Q λ γ -1 λ 2 + λ ( X T y -Q λ γ ) T diag ( z )( X T y -Q λ γ # distribute 2 γ inside the parentheses =max γ min z ∈D z -γ T Q λ γ -1 λ 2 + λ ( X T y -Q λ γ ) T diag ( z )( X T y -Q λ γ ) # simplify the linear algebra =max γ min z -γ T Q λ γ -1 λ 2 + λ ( X T y -Q λ γ ) T diag ( z )( X T y -Q λ γ ) # subject to ∑ p j =1 z j ≤ k, z j ∈ [0 , 1]

This completes the derivation for Theorem 3.1.

Proof 2

Proof. We will show max c min β , z L Fenchel ridge -λ ( β , z , c ) ≤ max γ min z L saddle ridge -λ ( γ , z ) and max c min β , z L Fenchel ridge -λ ( β , z , c ) ≥ max γ min z L saddle ridge -λ ( γ , z ) (both under the constrains ∑ p j =1 z j ≤ k, z j ∈ [0 , 1] ) and therefore establish the equivalence of optimal values.

For the first inequality, let β ∗ , c ∗ be the optimal solution to Problem (40) with the relation c ∗ = 2 λ 2 + λ ( X T y -Q λ β ∗ ) (this is always true because of Lemma C.1). Furthermore, let γ ∗ = β ∗ . Then we have

The derivation for the second equality roughly follows the derivation in the first proof we gave.

For the second inequality, let ˆ γ be the optimal solution to Problem (38). Let ˆ c = 2 λ 2 + λ ( X T y -Q λ ˆ γ ) . Furthermore, let ˆ β = ˆ γ (notice that this ˆ β satisfies the optimality condition of Problem (40)). Then we have

Similarly, the derivation for the second equality roughly follows the derivation (but this time in reverse order) in our first proof.

Using these two inequalities, we can establish the equivalence of optimal values between Problem (40) and Problem (38), i.e. , max c min β , z L Fenchel ridge -λ ( β , z , c ) = max γ min z L saddle ridge -λ ( γ , z ) .

C.2 Proof of Theorem 3.2

Theorem 3.2. The function h ( γ ) defined in Equation (14) is lower bounded by

Furthermore, the right-hand size of Equation (42) is maximized if γ = ˆ γ = argmin α L ridge ( α ) , where in this case, h ( γ ) on the left-hand side of Equation (42) becomes

where SumBottom p -k ( · ) denotes the summation of the smallest p -k terms of a given set.

Proof. We first derive Inequality (42).

This completes the derivation for Inequality (42).

The right-hand side of Inequality (42) is maximized if the gradient with respect to γ is 0 .

The gradient can be derived and simplified as:

Therefore, if we set ( X T X + λ 2 I ) γ = X T y , then the gradient of the right-hand side of Inequality (42) will be 0 , and the optimality condition is achieved. However, this means that γ = ( X T X + λ 2 I ) -1 X T y , which gives us the minimizer of the ridge regression loss L ridge ( · ) .

Lastly, we show that when ˆ γ = argmin γ L ridge ( γ ) , h (ˆ γ ) = L ridge (ˆ γ ) + ( λ 2 + λ ) SumBottom p -k ( { ˆ γ 2 j } ) .

First, let us derive a useful property. We claim that when ˆ γ = argmin γ L ridge ( γ ) , X T y -Q λ ˆ γ = ( λ 2 + λ )ˆ γ . Below, we prove this by showing that when we subtract the two terms, we get 0 .

Therefore, we have X T y -Q λ ˆ γ = ( λ 2 + λ )ˆ γ when ˆ γ = argmin γ L ridge ( γ ) . With this property, we can derive the final simplified formula for h (ˆ γ ) :

We note that -ˆ γ T ( X T X -λ I )ˆ γ -( λ 2 + λ )ˆ γ T ˆ γ = L ridge (ˆ γ ) . To show this, we subtract the two terms and prove that the subtraction is equal to 0 below:

Using the property that -ˆ γ T ( X T X -λ I )ˆ γ -( λ 2 + λ )ˆ γ T ˆ γ = L ridge (ˆ γ ) , we have:

This completes the proof for Theorem 3.2.

C.3 Proof of Theorem 3.3

Theorem 3.3. Let F ( γ ) = γ T Q λ γ and G ( p ) = 1 λ 2 + λ SumTop k ( { p 2 j } ) . Then the solution for the problem γ t +1 = argmin γ F ( γ ) + ρ 2 ∥ Q λ γ + p t -X T y + q t ∥ 2 2 is

Furthermore, let a = X T y -θ t +1 -q t and J be the indices of the top k terms of {| a j |} . The solution for the problem p t +1 = argmin p G ( p )+ ρ 2 ∥ θ t +1 + p -X T y + q t ∥ 2 2 is p t +1 j = sign ( a j ) · ˆ v j , where

Problem (44) is an isotonic regression problem and can be efficiently solved in linear time [12, 21].

Proof. Part I . We first show why Equation (43) is the analytic solution to the first optimization problem γ t +1 = argmin γ F ( γ ) + ρ 2 ∥ Q λ γ + p t -X T y + q t ∥ 2 2 .

The loss function we want to minimize is (writing out F ( γ ) = γ T Q λ γ explicitly)

# write out the square term as a sum of three terms

This loss function is minimized if we take the gradient with respect to γ and set it to 0 :

A sufficient condition for the gradient to be 0 is to have

If we solve the above equation, we obtain Equation (43):

Part II . Next, we show why p t +1 j = sign ( a j ) · ˆ v j together with Equation (44) gives us the solution to the second optimization problem p t +1 = argmin p G ( p ) + ρ 2 ∥ θ t +1 + p -X T y + q t ∥ 2 2 .

If we write out G ( p ) = 1 λ 2 + λ SumTop k ( { p 2 j } ) explicitly and use a = X T y -θ t +1 -q t as a shorthand to represent the constant, the objective function can be rewritten as:

# multiplying by 2 ρ doesn't change the optimal solution

According to Proposition 3.1 (b) in [30], the optimal solution p t +1 has the property that sign ( p t +1 j ) = sign ( a j ) .

Inspired by this sign-preserving property, let us make a change of variable by writing p j = sign ( a j ) · v j . Due to this change of variable, our optimal solution p t +1 can be obtained by solving the following optimization problem:

Note that in Problem (45), if we add the extra constraints v j ≥ 0 , this will not change the optimal solution. We can show this by the argument of contradiction. Suppose there exists an optimal solution ˆ v and an index j with ˆ v j < 0 . Then, if we flip the sign of ˆ v j , then the objective loss in Problem (45) is decreased. This contradicts with the assumption that ˆ v is the optimal solution. Therefore, we are safe to add the extra constraints v j ≥ 0 into Problem (45) without cutting off the optimal solution.

Thus, our optimal solution p t +1 becomes:

Next, according to Proposition 3.1 (c) in [30], we have ˆ v i ≥ ˆ v l if | a i | ≥ | a l | in Problem (46). Thus, we add these extra constraints into Problem (46) without changing the optimal solution and p t +1 becomes:

Note that in Problem (47), because of the extra constraints | a i | ≥ | a j | for ∀ i, j , we can rewrite SumTop k ( { v 2 j } ) = ∑ j ∈J v 2 j where J is the set of indices of the top k terms of {| a j |} 's. Then the optimal solution p t +1 becomes:

Next, we drop the constraints v j ≥ 0 for ∀ j as this does not change the optimal solution, using the same argument of contradiction we have shown before. Therefore, our optimal solution p t +1 can be obtained in the following way:

We can obtain the formulation in Problem (44) by performing some linear algebra manipulations as follows:

w

j

Using this new property, we can obtain the final result stated in the theorem for our optimal solution p t +1 to the second optimization problem:

Therefore, the task of evaluating the proximal operator is essentially reduced to solving an isotonic regression problem. We would like to acknowledge that the connection between SumTop k ( · ) and isotonic regression has also been recently discovered by [55]. However, the context and applications are totally different. While [55] discovers this relationship in the context of training neural networks, we find this connection in the context of ADMM and perspective relaxation of k-sparse ridge regression. Mathematically, our objective function is also different as our objective function contains a linear term of p in the proximal operator evaluation.

C.4 Proof of Theorem 3.4

Theorem 3.4. Let us define a k -sparse vector pair domain to be Ω k := { ( x , y ) ∈ R p × R p : ∥ x ∥ 0 ≤ k, ∥ y ∥ 0 ≤ k, ∥ x -y ∥ 0 ≤ k } . Any M 1 satisfying f ( y ) ≤ f ( x )+ ∇ f ( x ) T ( y -x )+ M 1 2 ∥ y -x ∥ 2 2 for all ( x , y ) ∈ Ω 1 is called a restricted smooth parameter with support size 1, and any m 2 k satisfying

f ( y ) ≥ f ( x ) + ∇ f ( x ) T ( y -x ) + m 2 k 2 ∥ y -x ∥ 2 2 for all ( x , y ) ∈ Ω 2 k is called a restricted strongly convex parameter with support size 2 k . If ˆ β is our heuristic solution by the beam-search method, and β ∗ is the optimal solution, then:

Proof. The left-hand-side inequality L ridge ( β ∗ ) ≤ L ridge ( ˆ β ) is obvious because the optimal loss should always be less than or equal to the loss found by our heuristic method. Therefore, we focus on the right-hand-side inquality L ridge ( ˆ β ) ≤ (1 -e -m 2 k /M 1 ) L ridge ( β ∗ ) .

We first give an overview of our proof strategy, which is similar to the proof in [29], but our decay factor m 2 k /M 1 in the exponent is much larger than theirs, which is m 2 k /M 2 k (we can obtain the definition of M 2 k by extending the definition of M 1 to the support size 2 k ). Let L ridge ( β t ) and L ridge ( β t +1 ) be the losses at the t -th and ( t +1) -th iterations of our beam search algorithm. We first derive a lower bound formula on the decrease of the loss function L ridge ( · ) between two successive iterations, i.e. , L ridge ( β t ) -L ridge ( β t +1 ) . We will then show that this quantity of decrease can be expressed in terms of the parameters M 1 and m 2 k defined in the statement of the theorem. Finally, we use the relationship between the decrease of the loss and M 1 and m 2 k to derive the right-hand inequality in Equation (51).

Step 1: Decrease of Loss between Two Successive Iterations :

For an arbitrary coordinate j that is not part of the support of β t , or more explicitly β t j = 0 , if we change the coefficient on the j -th coordinate β t from 0 to α , the loss becomes:

# distribute into the parentheses

The first two terms are constant, and the third term is a quadratic function of α .

The minimum loss we can achieve for L ridge ( β t + α e j ) by optimizing on the j -th coefficient is:

and the maximum decrease of our ridge regression loss by optimizing a single coefficient is then:

where X : j is the j -th column of the design matrix X .

Now, for our beam-search method, the decrease of loss between two successive iterations will be greater than the decrease of loss by optimizing on a single coordinate (because we are fine-tuning the coefficients on the newly expanded support and picking the best expanded support among all support candidates in the beam search), we have

For an arbitrary set of coordinates J ′ , we have the following inequality:

Inequality 54 holds because the largest decrease of loss among all possible coordinates is greater than or equal to the average decrease of loss among any arbitrary set of coordinates J ′ .

If we choose J ′ = S R := S ∗ \ supp ( β t ) , where S ∗ is the set of coordinates for the optimal solution β ∗ and supp ( · ) denotes the support of the solution, we have

# take the maximum of the denominator and pull it to the front

Inequality (55) is the lower bound of decrease of loss between two successive iterations of our beam-search algorithm.

In Step 2 and Step 3 below, we will provide bounds on max j ′ ∈S R ( ∥ X : j ′ ∥ 2 2 + λ 2 ) and ∑ j ′ ∈S R ( ∇ j ′ L ridge ( β t )) 2 in terms of M 1 and m 2 k , respectively.

We will show below that M 1 2 ≥ max j ′ ∈S R ( ∥ X : j ′ ∥ 2 2 + λ 2 ) .

According to the definition of the restricted smoothness of M 1 (See Background Concepts in Appendix B), we have that

for ∀ w 1 , w 2 ∈ R p with ∥ w 1 ∥ 0 ≤ 1 , ∥ w 2 ∥ 0 ≤ 1 , and ∥ w 2 -w 1 ∥ 0 ≤ 1 .

The right-hand side can be rewritten as

⇒L ridge ( w 2 ) -L ridge ( w 1 ) -∇L ridge ( w 1 ) T ( w 2 -w 1 ) = ( w 2 -w 1 ) T ( λ 2 I + X T X )( w 2 -w 1 ) .

Therefore, together with Inequality (56), we have that

for any w 2 -w 1 ∈ R p and ∥ w 2 -w 1 ∥ 0 ≤ 1 . If we let w 2 -w 1 = e j where e j is a vector with 1 on the j -th entry and 0 otherwise, we have

for ∀ j ∈ [1 , ..., p ] . Thus, we can derive our upper bound at the beginning of Step 2:

Step 3: Lower bound on ∑ j ′ ∈S R ( ∇ j ′ L ridge ( β t )) 2

We will show that ∑ j ′ ∈J ′ ( ∇ j ′ L ridge ( β t )) 2 ≥ 2 m 2 k ( L ridge ( β t ) -L ridge ( β ∗ )) . According to the definition of the restricted strong convexity parameter m 2 k

, we have:

where β t ∈ R p is our heuristic solution at the t -th iteration, β ∗ is the optimal k -sparse ridge regression solution, and ∥ β t ∥ 0 = t ≤ k < 2 k , ∥ β ∗ ∥ 0 = k < 2 k , and ∥ β ∗ -β t ∥ 0 ≤ 2 k By rearranging some terms, we have:

This gives us the lower bound:

Step 4: Final Bound for the Loss of Our Heuristic Solution

If we plug Inequalities (58) and (59) into the Inequality (55), we have:

Lastly, let's use Inequality (60) to derive the right-hand-side of Inequality (51) stated in the Theorem. Inequality (60) can be rewritten as:

which can be rearranged to the following expression:

Therefore, by applying the above inequality k times, we have:

However, L ridge ( 0 ) = 0 . This gives us

To get the final inequality, we start from a well-known inequality result 1 + x ≤ e x for any x ∈ R .

a a -a/k

Let x = -k > -1 , then we have 1 -k ≤ e . If we take both sides to the power of k , we have (1 -a k ) k ≤ e -a .

If we let a = m 2 k M 1 (because m 2 k M 1 < 1 [29] we have x = -a k = -m 2 k M 1 k > -1 for any k ≥ 1 ), then we have

Because L ridge ( β ∗ ) is a negative number, we finally have

C.5 Derivation of g ( a, b ) = max c ac -c 2 4 b via Fenchel Conjugate

Recall our function g : R × R → R ∪ { + ∞} is defined as:

The convex conjugate of g ( · , · ) , denoted as g ∗ ( · , · ) , is defined as [8]:

Since function g ( a, b ) is convex and lower semi-continuous, the biconjugate of g ( a, b ) is equal to the original function, i.e. , ( g ∗ ) ∗ = g [8]. Writing this explicitly, we have:

For now, let us first derive an analytic formula for the convex conjugate function g ∗ ( · , · ) .

According to the definition of g ( a, b ) in Equation (62), we have

For any fixed c and d , the optimality of the first case can be achieved if the first derivatives of ac + bd -a 2 b with respect to a and b are both equal to 0 . This means that,

If d = -c 2 4 , the optimality condition is achieved if we let a = bc 2 , and then ac + bd -a 2 b = bc 2 c + b ( -c 2 4 ) -c 2 4 b = 0 .

̸

If d = -c 2 4 , the optimality condition can not be achieved. However, if we still let a = bc 2 , we have ac + bd -a 2 b = bc 2 c + b ( d ) -c 2 4 b = ( c 2 4 + d ) b . Then, we have two specific cases to discuss: 1) if c 2 4 + d > 0 , the supremum is ∞ if we let b → ∞ ; 2) if c 2 4 + d < 0 , the supremum is 0 if we let b → 0 .

With all these cases discussed, we have that

Now let us revisit Equation (64), where g ( a, b ) = sup c,d ac + bd -g ∗ ( c, d ) . Below, we discuss three cases and derive the alternative formulas for g ( a, b ) under different scenarios: (1) c 2 4 + d = 0 , (2) c 2 4 + d ≤ 0 , and (3) c 2 4 + d > 0 .

After discussing these three cases, we can obtain the final results by taking the maximum of the three optimal values, i.e. , g ( a, b ) = max( g 1 ( a, b ) , g 2 ( a, b ) , g 3 ( a, b )) . The third case c 2 4 + d > 0 is not interesting and can be easily eliminated. The second case c 2 4 + d < 0 is also not interesting because ac + bd = ac + b ( -c 2 4 ) + b ( c 2 4 + d ) ≤ ac + b ( -c 2 4 ) for any b ≥ 0 , which is the domain we are interested in. This gives us g 2 ( a, b ) ≤ g 1 ( a, b ) , and the equality is only achieved when b = 0 . Thus,

where in the last step we convert sup( · ) to max( · ) because the function is quadratic and the optimum can be achieved.

D Existing MIP Formulations for k -sparse Ridge Regression

We want to solve the following k -sparse ridge regression problem:

However, we can not handle the constraint (1 -z j ) β j = 0 directly in a mixed-integer programming (MIP) solver. Currently in the research community, there are four formulations we can write before sending the problem to a MIP solver. They are (1) SOS1, (2) big-M, (3) perspective, and (4) optimal perspective. We write each formulation explicitly below:

1. SOS1 formulation:

where M > 0 . For the big-M formulation, as we mentioned in the main paper, it is very challenging to choose the right value for M . If M is too big, the lower bound given by the relaxed problem is too loose; if M is too small, we risk cutting off the optimal solution.

perspective formulation:

where s j 's are new variables which ensure that when z j = 0 , the corresponding coefficient β j = 0 . The problem is a mixed-integer quadratically-constrained quadratic problem (QCQP). The problem can be solved by a commercial solver which accepts QCQP formulations.

eigen-perspective formulation:

where λ min ( · ) denotes the smallest eigenvalue of a matrix. This formulation is similar to the QCQP formulation above, but we increase the ℓ 2 coefficient by subtracting λ min ( X T X ) I from the diagonal of X T X .

where ( a, b ) ∈ SOS 1 means that at most one variable (either a or b but not both) can be nonzero.

big-M formulation:

optimal perspective formulation:

where A ⪰ 0 means that A is positive semidefinite. Such formulation is called a mixedinteger semidefinite programming (MI-SDP) problem and is proposed in [62, 28, 37]. Currently, there is no commercial solver that can solve MI-SDP problems: Gurobi does not support SDP; MOSEK can solve SDP but does not support MI-SDP. As shown by [28], SDP is not scalable to high dimensions. We will confirm this observation in our experiments by solving the relaxed convex SDP problem (relaxing { 0 , 1 } to [0 , 1] ) via MOSEK, the state-of-the-art conic solver. After we get the solution from the relaxed problem, we find the indices corresponding to the top k largest z j 's and retrieve a feasible solution on these features.

E Algorithmic Charts

E.1 Overall Branch-and-bound Framework

We first provide the overall branch-and-bound framework. Then we go into details on how to calculate the lower bound, how to obtain a near-optimal solution and upper bound (via beam search), and how to do branching. A visualization of the branch-and-bound tree can be found in Figure 5.

Root

Llower

Step1:usenovellower

boundcalculationalgorithm

searchalgorithm

Step3:picka

1

Z3

=

：0

variabletobranch

based on βupper

fromStep2using

the new branching

Node3

strategy

Zs = 1

Z5= 0

Z,=1

Z7

=0

Algorithm 1 OKRidge( D , k , λ 2 , ϵ , T iter , T max ) → β

Input: dataset D , sparsity constraint k , ℓ 2 coefficient λ 2 , optimality gap ϵ , iteration limit T iter for subgradient ascent, and time limit T max .

Output: a sparse continuous coefficient vector β ∈ R p with ∥ β ∥ 0 ≤ k .

1: ϵ best , L best lower , β best upper , L best upper = ∞ , -∞ , 0 , 0 ▷ Initialize best optimality gap to ∞ , best lower bound to -∞ , best solution to 0 and its loss to 0.

2: Q ← initialize an empty queue (FIFO) structure ▷ FIFO corresponds to a breadth-first search.

3: N select = ∅ , N avoid = ∅ ▷ Coordinates in N select can be selected during heuristic search and lower bound calculation for a node; Coordinates in N avoid must be avoided.

4: RootNode = CreateNode ( D , k, λ 2 , N select , N avoid ) ▷ Create the first node, which is the root node in the BnB tree.

5: Q = Q . push ( RootNode ) ▷ Put the root node in the queue.

6: l unsolved = 1 ▷ Smallest depth where at least one node is unsolved in the BnB tree

7: WHILE Q ̸ = ∅ and T elapsed < T max do ▷ Keep searching until the time limit is reached or there is no node in the queue.

8: Node = Q .getNode ()

9: L lower = Node.lowerSolveFast() ▷ Get the lower bound for this node through the fast method

11: if L lower ≥ L upper best then continue ▷ Prune Node if its lower bound is higher than or equal to the current best upper bound.

10: if L lower < L upper best then L lower = Node.lowerSolveTight() ▷ Get a tighter lower bound for this node through the subgradient ascent

12: if All nodes in depth l unsolved of the BnB tree are solved then

13: L best lower = max q ( {L q , for q ∈ { 1 , 2 , ..., l unsolved } ) ▷ Tighten the best lower bound; L q is the smallest lower bound of all nodes in depth q

14: l unsolved = l unsolved +1

▷

Raise up the smallest unsolved depth by 1

16: if ϵ best < ϵ then return β best ▷ Optimality gap tolerance is reached, return the current best solution

15: ϵ best = ( L best upper -L best lower ) / |L best upper |

17: β upper , L upper = Node.upperSolve() ▷ Get feasible sparse solution and its loss from beam search, which is an upper bound for our BnB tree.

18: if L upper < L best upper then

19: L best upper , β best upper = L upper , β upper ▷ Update our best solution and best upper bound

20: ϵ best = ( L best upper -L best lower ) / |L best upper |

21: if ϵ best < ϵ then return β best ▷ Optimality gap tolerance is reached, return the current best solution

22: j = Node.getBranchCoord ( β upper ) ▷ Get which feature coordinate to branch

23: ChildNode1 = CreateNode ( D , k, λ 2 , ( Node →N select ) ∪ { j } , Node →N avoid ) ▷ New child node where feature j must be selected

24: ChildNode2 = CreateNode ( D , k, λ 2 , Node →N select , ( Node →N avoid ) ∪ { j } ) ▷ New child node where feature j must be avoided

25: Q .push ( ChildNode1 , ChildNode2 ) ▷ Put the two newly generated nodes into our queue.

26: return β best ▷ Return the current best solution when time limit is reached or there is no node in the queue

E.2 Fast Lower Bound Calculation

Attributes in the Node class: dataset D = { ( x i , y i ) } n i =1 , sparsity constraint k , ℓ 2 coefficient λ 2 , set of features that must be selected N select, set of features that must be avoided N avoid . Output: a lower bound for the node L lower .

1: γ = argmin β ,β j =0 if j ∈N avoid β T X T Xβ -2 y T Xβ + λ 2 ∥ β ∥ 2 2 ▷ Solve the ridge regression problem. We avoid using coordinates belonging to N avoid by fixing the coefficients on those coordinates to be 0 .

2: L ridge = γ T X T Xγ -2 y T Xγ + λ 2 ∥ γ ∥ 2 2

▷

Loss for the ridge regression.

3: L extra = ( λ 2 + λ min ( X T X )) · SumBottom p -|N avoid |-k ( { β 2 j , for ∀ j ∈ [1 , ..., p ] \ ( N avoid ∪ N select ) } ) ▷ First of all, we are restricted to the coordinates not belonging to N avoid. Second, coordinates in the set N select must be selected. If we work through the strong convexity argument in Section 3.1 in the main paper, we would get this formula.

4: L lower = L ridge + L extra

5: return L lower

E.3 Refine the Lower Bound through the ADMM method

Next, we show how to get a tighter lower bound through the ADMM method.

Algorithm 3 Node.lowerSolveTightADMM() → L lower

̸

Attributes in the Node class: dataset D = { ( x i , y i ) } n i =1 , sparsity constraint k , ℓ 2 coefficient λ 2 , set of features that must be selected N select, and iteration limit T iter for ADMM (For simplicity of presentation, we assume N avoid is empty. If N avoid = ∅ , we can do a selection and relabeling step on X , y , Q λ , N select to focus solely on coordinates not belonging to N avoid ). Output: a lower bound for the node L lower .

1: γ 1 = argmin β β T X T Xβ -2 y T Xβ + λ 2 ∥ β ∥ 2 2 ▷ Initialize γ by solving the ridge regression problem. This will be used as a warm start for the ADMM method below. To avoid redundant computation, we can use the value for γ from Node.lowerSolveFast().

2: p 1 = X T y -Q λ γ

3: q 1 = 0

4: ρ = 2 / √ λ max ( Q λ ) λ min > 0 ( Q λ )

5: for t = 2, 3, ... T iter . do

6: γ t = ( 2 ρ I + Q λ ) -1 ( X T y -p t -1 -q t -1 )

7: θ t = 2 Q λ γ t + p t -1 -X T y

▷ Initialize p as stated in Problem 18. in the ADMM. ▷ Calculate the step size used in the ADMM

▷ Initialize the scaled dual variable q

▷ Update γ in the ADMM.

▷ Update θ in the ADMM.

8: a = X T y -θ t -q t -1 ▷ This line and the next 3 lines are used to calculate variables used for setting up the isotonic regression problem.

9: Let J be the indices of the top k -|N select | terms of | a j | with the constraint that any element in J does not belong to N select .

10: Let w j = 1 if j / ∈ J and j / ∈ N select and let w j = 2 ρ ( λ 2 + λ )+1 otherwise.

11: Let b j = | a j | w j .

12: Solve the problem ˆ v = argmin v ∑ j w j ( v j -b j ) 2 such that for ∀ i, l / ∈ N select , v i ≥ v l if | a i | ≥ | a l | . ▷ We can decompose this optimization problem into two independent problems with coordinates belonging to N select (can be solved individually) and coordinates not belonging to N select (can be solved via standard isotonic regression).

13: p t = sign ( a ) ⊙ ˆ v ▷ Update p in the ADMM. The symbol ⊙ denotes the element-wise

(Hadamard) product.

14: q t = q t -1 + θ t + p t -X T y

15: end for

16: L lower = h ( γ T iter )

17: return L lower

▷ Update q in the ADMM.

▷ Calculate the lower bound.

E.4 (Optional) An alternative to the Lower Bound Calculation via Subgradient Ascent and the CMF method

In addition to the ADMM method, we also provide an alternative algorithm showing how to get a tighter lower bound through the subgradient ascent algorithm using the CMF method. Note that this subgradient method has been superseded by the ADMM method, which converges faster. We provide this method in the Appendix for the sake of completeness and for future readers to consult.

Algorithm 4 Node.lowerSolveTightSubgradient() → L lower

Attributes in the Node class: dataset D = { ( x i , y i ) } n i =1 , sparsity constraint k , ℓ 2 coefficient λ 2 , set of features that must be selected N select, set of features that must be avoided N avoid, and iteration limit T iter for subgradient ascent

Output: a lower bound for the node L lower .

1: γ = argmin β ,β j =0 if j ∈N avoid β T X T Xβ -2 y T Xβ + λ 2 ∥ β ∥ 2 2 ▷ Solve the ridge regression problem and avoid using coordinates in N avoid. This will be used as a warm start for the subgradient ascent procedure below. To avoid redundant computation, we can use the value for γ from Node.lowerSolveFast()

t

2: v = γ ▷ Initialization for v at iteration step t = 1 3: h ∗ = L ridge ( β root upper ) ▷ get an estimate of max γ h ( γ ) ; if β root upper has not been solved at the root node, solve β root by calling the upperSolve() function at the root node.

upper

4: L lower = h ( γ )

5: for t = 2, 3, ... T iter do

6: γ , v = Node.subgradAscentCMF ( γ , v ) ▷ Perform one step of subgradient ascent using the CMF method

7: L lower = max( L lower , h ( γ )) ▷ Loss decreases stochastically during the subgradient ascent procedure. We keep the largest value of h ( γ )

8: end for

9: return L lower

In the above algorithm, we call a new sub algorithm called subGradAscentCMF. We provide the details on how to do this below. We try to maximize h ( γ ) through iterative subgradient ascent. A more sophisticated algorithm (CMF method [23]) can be applied to adaptively choose the step size, which has been shown to achieve faster convergence in practice. The computational procedure for the subgradient ascent algorithm using the CMF method [18, 23] is as follows. Given γ t -1 at iteration t -1 , γ t can be computed as:

w t = ∂ L saddle ridge -λ ( γ , Z ( γ t -1 )) ∂ γ ∣ ∣ ∣ ∣ γ = γ t -1 # get the subgradient s t = max(0 , -d t ( v t -1 ) T w t / ∥ v t -1 ∥ 2 2 ) # s t will be used as a smoothing factor between v t -1 and w t v t = w t + s t v t -1 # combine v t -1 and w t using the smoothing factor α t = ( h ∗ -h ( γ t -1 )) / ∥ v t ∥ 2 2 # calculate the step size; h ∗ is an estimate of max γ h ( γ ) ; here we let h ∗ = L ridge ( β root upper ) γ t = γ t -1 + α t v t # perform subgradient ascent where at the beginning of the iteration ( t = 1 ), v 1 = 0 and we choose d t = 1 . 5 as a constant for all iterations as recommended by the original paper. Below, we provide the detailed algorithm to perform subgradient ascent using the CMF method.

Algorithm 5 Node.subgradAscentCMF( γ , v , h ∗ ) → γ , v

Attributes in the Node class: dataset D = { ( x i , y i ) } n i =1 , sparsity constraint k , ℓ 2 coefficient λ 2 , set of features that must be selected N select, set of features that must be avoided N avoid

Input: Current variable γ for the lower bound function h ( · ) for Equation 14, current ascent vector v , and h ∗ is an estimate of max γ h ( γ ) .

Output: updated variable γ and updated ascent vector v .

1: ˆ z = argmin z L saddle ridge -λ ( γ , z ) ▷ get the minimizer for z at the current value for γ ; optimization is performed under the constraint ∑ p j =1 z j ≤ k, z j ∈ [0 , 1] , z j = 1 for j ∈ N select and z j = 0 for j ∈ N select 2: w = ( ∂ L saddle ridge -λ ( γ , ˆ z ) ) / ( ∂ γ ) ▷ get the subgradient 3: w j = 0 for j ∈ N avoid ▷ We don't perform any ascent steps on the coordinates belong to N avoid . 4: s = max(0 , -1 . 5 v T w / ∥ v ∥ 2 2 ) ▷ Calculate the smoothing factor 5: v = w + s v ▷ update the ascent vector by combining the current subgradient and previous ascent vector 6: α = ( h ∗ -h ( γ )) / ∥ v ∥ 2 2 ▷ calculate the step size for the ascent step 7: γ = γ + α v ▷ Perform one step of ascent along the direction of the ascent vector 8: return γ , v

E.5 Beam Search Algorithm

We provide a visualization of the beam search algorithm in Figure 6

1,2

1,2,3

1

1,3

1

1,2

1,2,4

2

1,4

(1,2,5

<start>

3

1,5

1,2,5

4

4,1

4,3,1

5

4

4,2

4,3

4,3,2

4,3

4,3,5

4,5

Algorithm 6 Node.upperSolve( B = 50 ) → β upper , L upper

Attributes in the Node class: dataset D = { ( x i , y i ) } n i =1 , sparsity constraint k , ℓ 2 coefficient λ 2 , a set of feature coordinates that must be selected N select, and a set of feature coordinates that must be avoided N avoid .

Input: beam search size B , which is set to be 50 as the default value.

Output: a near-optimal solution β upper with ∥ β upper ∥ 0 = k and the corresponding loss L upper .

1: β ′ = argmin β ,β j =0 for j / ∈N select β T X T Xβ -2 y T Xβ + λ 2 ∥ β ∥ 2 2 ▷ Initialize the coefficients by

fitting on the must-be-selected coordinates. 2: W = { β ′ } ▷ collection of solutions at each iteration (initially we only have one solution) 3: F = ∅ ▷ Initialize the collection of found supports as an empty set 4: for t = |N select | +1 , ..., k do 5: W tmp ←∅ 6: for β ′ ∈ W do ▷ Each of these has support t -1 7: ( W ′ , F ) ← ExpandSuppBy1 ( D , β ′ , F , B, N avoid ) . ▷ Returns ≤ B models with supp. t 8: W tmp ←W tmp ∪ W ′ 9: end for 10: Reset W to be the B solutions in W tmp with the smallest ridge regression loss values. 11: end for 12: Pick β upper from W with the smallest ridge regression loss. 13: L upper = β T upper X T Xβ upper -2 y T Xβ upper + λ 2 ∥ β upper ∥ 2 2 14: return β upper , L upper .

Algorithm 7 ExpandSuppBy1 ( D , β ′ , F , B, N avoid ) → W , F

Input: dataset D = { ( x i , y i ) } n i =1 , current solution β ′ , set of found supports F , beam search size B , and a set of feature coordinates that must be avoided N avoid . Output: a collection of solutions W = { β t } with ∥ β t ∥ 0 = ∥ β ′ ∥ 0 +1 for ∀ t . All of these solutions include the support of β ′ plus one more feature. None of the solutions have the same support as any element of F , meaning we do not discover the same support set multiple times. We also output the updated F . 1: Let S c ←{ j | β ′ j = 0 and j / ∈ N avoid } ▷ Non-support of the given solution 2: Pick up to B coords ( j 's) in S c with largest decreases in ∆ j L ridge ( · ) , call this set J ′ . ▷ We will use these supports, which include the support of w plus one more. 3: W ← ∅ 4: for j ∈ J ′ do ▷ Optimize on the top B coordinates 5: if ( supp ( w ) ∪ { j } ) ∈ F then 6: continue ▷ We've already seen this support, so skip. 7: end if 8: F ← F ∪ { supp ( w ) ∪ { j }} . ▷ Add new support to F . 9: u ∗ ∈ argmin u L ridge ( D , u ) with supp ( u ) = supp ( w ′ ) ∪ { j } . ▷ Update coefficients on the newly expanded support 10: W ← W ∪ { u ∗ } ▷ cache the solution to the central pool 11: end for 12: return W and F .

E.6 Branching

Attributes in the Node class: dataset D = { ( x i , y i ) } n i =1 , sparsity constraint k , ℓ 2 coefficient λ 2 , a set of feature coordinates that must be selected N select, and a set of feature coordinates that must be avoided N avoid .

Output: a single coordinate j on which the node should branch in the BnB tree.

1: j ∗ = argmax j / ∈N select -2 β T ( X T X + λ 2 I ) β j e j +(( X T X ) jj + λ 2 ) β 2 j +2 y T X β j e j ▷ select the coordinate whose coefficient, if we set to 0 , would result in the greatest increase in our ridge regression loss

2: return j ∗

E.7 A Note on the Connection with the Interlacing Eigenvalue Property

During the branch-and-bound procedure, finding the smallest eigenvalue of any principle submatrix of X T X is computationally expensive if we do it for every node. Fortunately, we can get a lower bound for the smallest eigenvalue based on the following Interlacing Eigenvalue Property [36]:

Theorem E.1. Suppose A ∈ R n × n is symmetric. Let B ∈ R m × m with m < n be a principal submatrix (obtained by deleting both i -th row and i -th column for some values of i ). Suppose A has eigenvalues γ 1 ≤ ... ≤ γ n and B has eigenvalues α 1 ≤ ... ≤ α m . Then

Using this Theorem, we can let λ = λ min ( X T X ) for every node in the branch-and-bound tree in our algorithm. This saves us a lot of computational time because we only have to calculate the smallest eigenvalue once in our root node. However, if the minimum eigenvalue were to be recalculated at each node efficiently, the lower bounds would be even tighter.

And if m = n -1 ,

F More Experimental Setup Details

F.1 Computing Platform

All experiments were run on the 10x TensorEX TS2-673917-DPN Intel Xeon Gold 6226 Processor, 2.7Ghz. We set the memory limit to be 100GB.

F.2 Baselines, Licenses, and Links

Commerical Solvers For the MIP formulations listed in Appendix D, we used Gurobi and MOSEK. We implemented the SOS1, bigM, perspective, and eigen-perspective formulations in Gurobi. The Gurobi version is 10.0, which can be installed through conda ( https://anaconda.org/gurobi/ gurobi ). We used the Academic Site License. We implemented the perspective formulations and relaxed convex optimal perspective formulations in MOSEK. The MOSEK version is 10.0, which can be installed through conda ( https://anaconda.org/MOSEK/mosek ). We used the Personal Academic License.

PySINDy, STLSQ, SSR, E-STLSQ We used the PySINDy library ( https://github.com/ dynamicslab/pysindy ) to perform the differential equation experiments. The license is MIT. The heuristic methods STLSQ, SSR, and E-STLSQ are implemented in PySINDy.

MIOSR For the differential equation experiments, the MIOSR code is available on GitHub( https: //github.com/wesg52/sindy_mio_paper ). The license for this code repository is MIT.

SubsetSelectionCIO The SubsetSelectionCIO code is available on GitHub ( https: //github.com/jeanpauphilet/SubsetSelectionCIO.jl ) with commit version 8b03d1fba9262b150d6c0f230c6bf9e8ee57f92e. The license for this code repository is MIT 'Expat' License ( https://github.com/jeanpauphilet/SubsetSelectionCIO.jl/blob/ master/LICENSE.md ). Due to package compatibility, we use Gurobi 9.0 for SubsetSelectionCIO. See the SusbetSelectionCIO package version specification at this link: https: //github.com/jeanpauphilet/SubsetSelectionCIO.jl/blob/master/Project.toml .

l0bnb The l0bnb code is available on GitHub ( https://github.com/alisaab/l0bnb ) with commit version 54375f8baeb64baf751e3d0effc86f8b05a386ce. The license for this code repository is MIT.

Sparse Simplex The Sparse Simplex code is available at the author's website ( https://sites. google.com/site/amirbeck314/software?authuser=0 ). We ran the greedy sparse simplex algorithm. The license for this code repository is GNU General Public License 2.0.

Block Decomposition The block decomposition code is available at the author's website ( https: //yuangzh.github.io ). The author didn't specify a license for this code repository. However, we didn't modify the code and only used it for benchmarking purposes, which is allowed by the software community. Besides the block decompostion algorithm, the code repository also contains reimplementations for other algorithms, such as GP, SSP, OMP, proximal gradient l0c, QPM, ROMP. We used these reimplementations in the differential equation experiments as well. Please see [60] for details of these baselines.

F.3 Differential Equation Experiments

We compare the certifiable method MIOSR along with other baselines on solving dynamical system problems. In particular, for each dynamical system, we perform 20 trials with random initial conditions for 12 different trajectory lengths. Each dynamical system has observed positions every 0.002 seconds, so a larger trajectory length corresponds to a larger sample size n . Derivatives were approximated using a smoothing finite difference with a window size of 9 points. The derivatives were calculated automatically by the PySINDY library. Each trajectory has 0.2% noise added to it to demonstrate robustness to noise. For the SINDy approach, true dynamics exist in the finitedimensional candidate functions/features. We investigate which method recovers the correct dynamics more effectively.

F.3.1 Model Selection

Model selection was performed via cross-validation with training data encompassing the first 2/3rds of a trajectory and the final third used for validation. Predicted derivatives were compared to the validation set, with the best model being the one that minimizes AIC c for sparse regression in the setting of dynamical systems [45].

F.3.2 Hyperparameter Choices

With respect to each trajectory, each algorithm was trained with various combinations of hyperparameter choices. For both MIOSR and our method OKRidge, the ridge regression hyperparameter choices are λ 2 ∈ { 10 -5 , 10 -3 , 10 -2 , 0 . 05 , 0 . 2 } , and the sparsity level hyperparameter choices are k ∈ { 1 , 2 , 3 , 4 , 5 } . As in the experiments of MIOSR [9], we set the time limit for each optimization to be 30 seconds for both MIOSR and our method OKRidge. For SSR and STLSQ, the ridge regression hyperparameter choices are λ 2 ∈ { 10 -5 , 10 -3 , 10 -2 , 0 . 05 , 0 . 2 } . SSR and STLSQ may automatically pick a sparsity level, but they require a thresholding hyperparameter. We choose 50 thresholding hyperparameters with values between 10 -3 and 10 1 . 5 , depending on the system being fit. Together there are 250 different hyperparameter combinations we need to run on each trajectory length for for SSR and STLSQ. When fitting E-STLSQ, cross-validation is not performed as the chosen hyperparameters for STLSQ are used instead. For SSR, MIOSR, and OKRidge, we cross validate on each dimension of the data and combine the strongest results into a final model. For STLSQ and E-STLSQ, no post-processing is done and the same hyperparameters are used for each dimension.

F.3.3 Evaluation Metrics

Three metrics are considered to evaluate each algorithm: true positivity rate, L 2 coefficient error, and root-mean-squared error (RMSE). The true positivity rate is defined as the ratio of the number of correctly chosen features divided by the sum of the number of true features and incorrectly chosen features. This metric equals 1 if and only if the correct form of the dynamical system is recovered. L 2 coefficient error is defined as the sum of squared errors of the predicted and the true coefficients. Finally, for each model fit, we produce 10 simulations with different initial conditions of the true model, and calculate the root-mean-squared error (RMSE) of the predicted derivatives and the true derivatives. Besides these three metrics, we also report the running times, comparing OKRidge with other optimal ridge regression methods.

G More Experimental Results on Synthetic Benchmarks

G.1 Results Summarization

ℓ 2 Perturbation Study We first compare with SOS1, Big-M (Big-M50 means that the big-M value is set to 50 ), Perspective, and Eigen-Perspective formulations with different ℓ 2 regularizations, with and without using beam search as the warm start. We have three levels of ℓ 2 regularizations: 0 . 001 , 0 . 1 , 10 . 0 . The results are shown in Tables 1-60 in Appendix G.2. When the method has '-' in the upper bound or gap column, it means that method does not produce any nontrivial solution within the time limit.

Extensive results have shown that OKRidge outperforms all exisiting MIP formulations solved by the state-of-the-art MIP solver Gurobi. With the beam search solutions (usually they are already the optimal solutions) as the warm starts, Gurobi are faster than without using warm starts. However, OKRidge still outperforms all existing MIP formulations. If Gurobi can solve the problem within the time limit, OKRidge is usually usually orders of magnitude faster than all baselines. When Gurobi cannot solve the problem within the time limit, OKRidge can either solve the problem to optimality or achieve a much smaller optimality gap than all baselines, especially in the presence of high feature correlation and/or high feature dimension.

Big-M Perturbation Study We also conduct perturbation study on the big-M values. We pick three values: 50, 20, and 5. In practice, it is difficult to select the right value because setting the big-M value too small will accidentally cut off the optimal solutions. Nonetheless, we still pick a small big-M value (5; the optimal coefficients are all 1 's in the synthetic benchmarks) to do a thorough comparison. The results are shown in Tables 66-120 in Appendix G.3. When the method has '-' in the upper bound or gap column, it means that method does not produce any nontrivial solution within the time limit.

The results show that although choosing a small big-M value can often lead to smaller running time, OKRidge is still orders of magnitude faster than the big-M formulation, and achieves the smallest optimality gap when all algorithms reach the time limit. Additionally, we find that having a small big-M value will sometimes lead to increased running time on solving the root relaxation. Sometimes, when the root relaxation is not finished within the 1h time limit, Gurobi will provide a loose lower bound, resulting in large optimality gaps. Aside from accidentally cutting off the optimal solution, this is another reason against choosing a very small big-M value.

Comparison with MOSEK, SubsetSelectionCIO, and l0bnb For the purpose of thorough comparison, we also solve the problem with other MIP solvers/formulations. The results are shown in Tables 121-Table 135 in Appendix G.4. When the method has '-' in the upper bound or gap column, it means that method does not produce any nontrivial solution within the time limit. For the MOSEK solver, when the upper bound and gap columns have '-' values, and the time is exactly 3600 . 00 , this means that MOSEK runs out of memory (100GB).

First, we try to solve the problem using the commercial conic solver called MOSEK. We use MOSEK to solve the perspective formulation, with and without using beam search solutions as warm starts. The results indicate that Gurobi is in general much faster and more memory-efficient than MOSEK in solving the perspective formulation. We also use MOSEK to solve the relaxed convex optimal perspective formulation (relaxing the binary constraint z j ∈ { 0 , 1 } to the interval constraint z j ∈ [0 , 1] ). we only solve the relaxed convex SDP problem because MOSEK does not support MI-SDP. The results show that MOSEK has limited capacity in solving the perspective formulation. MOSEK also runs out of memory for large-scale problems ( n = 100000 , p = 3000 or p = 5000 ). Besides, the experimental results show that convex SDP problem is not scalable to high dimensions. For the high-dimensional cases, OKRidge solving the MIP problem is usually much faster than MOSEK solving a convex SDP problem.

Second, we uses the SubsetSelectionCIO method, which solves the original sparse regression problem using branch-and-cut (lazy-constraint callbacks) in Gurobi. However, as the results indicate, SubsetSelectionCIO produces wrong optimal solutions when ℓ 2 regularization is small and the number of samples is large ( n = 100000 ). When SubsetSelectionCIO does not have errors, the running times or optimality gaps are much worse than those of OKRidge.

Lastly, we solve the sparse ridge regression problem using l0bnb. In contrast to other methods, l0bnb controlls the sparsity level implicitly through the ℓ 0 regularization, instead of the ℓ 0 constraint. l0bnb needs to try several different ℓ 0 regularizations to find the desired sparsity level. For each ℓ 0 regularization coefficient, we set the time limit to be 180 seconds, and we stop the l0bnb algorithm after a time limit of 7200 seconds (this is longer than 3600 seconds because we want to give enough time for l0bnb to try different ℓ 0 coefficients). Moreover, l0bnb imposes a box constraint (similar to the big-M method) on all coefficients. The box constraint tightens the lower bound and potentially accelerates the pruning process, but it could also cut off the optimal solution if the 'M' value is not chosen properly or large enough. To our best knowledge, there is no reliable way to calculate the 'M' value. The l0bnb package internally estimates the 'M' value in a heuristic way. Despite the box constraint giving l0bnb an extra advantage, we still compare this baseline with OKRidge. As the results indicate, l0bnb often takes a long time to find the desired sparsity level by trying different ℓ 0 regularizations. In some cases, l0bnb finds a solution with support size different than the required sparsity level (demonstrating the difficulty of finding a good ℓ 0 coefficient). In other cases, l0bnb still has an optimality gap after reaching the time limit while OKRidge can finish certification within seconds.

Discussion on the Case p ≫ n For the application of discovering differential equations, the datasets usually have n ≥ p . Therefore, the case p ≫ n is not the main focus of this paper. Nonetheless, we provide additional experiments to give a complete view of our algorithm.

We set the number of features p = 3000 and the number of samples n = 100 , 200 , 500 . The feature correlation is set to be ρ = 0 . 1 since this is already a very challenging case. The results are shown in Tables 151-153 in Appendix G.5. The results show that OKRidge is competitive with other MIP methods. l0bnb is the only method that can solve the problem when n = 500 within the time limit. However, this algorithm has difficulty in finding the desired sparsity level in the case of n = 100 and n = 200 even with a large time limit of 7200 seconds. In some cases, MOSEK gives the best optimality gap. However, when n ≥ p as we have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and l0bnb are not competitive in discovering differential equations (datasets have n ≥ p ), which is the main focus of this paper.

In the future, we plan to extend OKRidge to develop a more scalable algorithm for the case p ≫ n .

Fast Solve vs. ADMM In the main paper, we proposed two ways to calculate the saddle point problem. The first approach, fast solve, is based on an analytical formula. The second approach is based on the ADMM method. Intuitively, since the fast solve method is only maximizing a lower bound of the saddle point objective and ADMM is maximizing the saddle point objective directly, we expect that ADMM would give much tighter lower bounds than the fast solve approach does. Here, we give empirical evidence to support our claim. The results can be seen in Figures 7 and 8.

G.2 Comparison with SOS1, Big-M, Perspective, and Eigen-Perspective Formulations with different ℓ 2 Regularizations

G.2.1 Synthetic 1 Benchmark with λ 2 = 0 . 001 with No Warmstart

G.2.2 Synthetic 1 Benchmark with λ 2 = 0 . 001 with Warmstart

G.2.3 Synthetic 1 Benchmark with λ 2 = 0 . 1 with No Warmstart

G.2.4 Synthetic 1 Benchmark with λ 2 = 0 . 1 with Warmstart

G.2.5 Synthetic 1 Benchmark with λ 2 = 10 . 0 with No Warmstart

G.2.7 Synthetic 2 Benchmark with λ 2 = 0 . 001 with No Warmstart

G.2.9 Synthetic 2 Benchmark with λ 2 = 0 . 1 with No Warmstart

G.2.11 Synthetic 2 Benchmark with λ 2 = 10 . 0 with No Warmstart

G.3 Comparison with the Big-M Formulation with different M Values

G.3.1 Synthetic 1 Benchmark with λ 2 = 0 . 001 with No Warmstart

G.3.2 Synthetic 1 Benchmark with λ 2 = 0 . 001 with Warmstart

G.3.3 Synthetic 1 Benchmark with λ 2 = 0 . 1 with No Warmstart

G.3.4 Synthetic 1 Benchmark with λ 2 = 0 . 1 with Warmstart

G.3.5 Synthetic 1 Benchmark with λ 2 = 10 . 0 with No Warmstart

Table 86: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10, ρ = 0 . 1 , λ 2 = 10 . 0 ). All baselines use our beam search solution as a warm start.

G.3.7 Synthetic 2 Benchmark with λ 2 = 0 . 001 with No Warmstart

Table 96: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0 . 1 , λ 2 = 0 . 001 ). All baselines use our beam search solution as a warm start.

G.3.9 Synthetic 2 Benchmark with λ 2 = 0 . 1 with No Warmstart

G.4 Comparison with MOSEK Solver, SubsetSelectionCIO, and L0BNB

Table 121: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10, ρ = 0 . 1 , λ 2 = 0 . 001 ).

Table 126: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10, ρ = 0 . 1 , λ 2 = 0 . 1 ).

Table 131: Comparison of time and optimality gap on the synthetic datasets (n = 100000, k = 10, ρ = 0 . 1 , λ 2 = 10 . 0 ).

Table 136: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0 . 1 , λ 2 = 0 . 001 ).

Table 141: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0 . 1 , λ 2 = 0 . 1 ).

Table 146: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0 . 1 , λ 2 = 10 . 0 ).

G.5 Experiments on the Case p >> n

Table 151: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0 . 1 , λ 2 = 0 . 001 ). All baselines use our beam search solution as a warm start whenever possible. The results in the case of p >> n show that OKRidge is competitive with other MIP methods. l0bnb can solve the problem when n = 500 but has difficulty in finding the desired sparsity level in the case of n = 100 and n = 200 . MOSEK sometimes gives the best optimality gap. However, as we have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and l0bnb are not competitive in discovering differential equations (datasets have n ≥ p ), which is the main focus of this paper. In the future, we plan to extend OKRidge to develop a more scalable algorithm for the case p >> n .

G.6 Comparison between Fast Solve and ADMM

In this subsection, we provide the empirical evidence that the ADMM method is more effective than the fast solve method in terms of the computing a tighter lower bound, which helps speed up the certification process.

(p=0.1)

(p=0.5)

(p=0.9)

103

103

103

time (sec)

102

102

102

101

101

101

10°

10°

10°

100

500

1000

3000

50001

100

500

1000

3000

5000!

100

500

1000

3000

5000

0.2

0.2

0.2

gap

0.1

0.1

0.1

0.0

0.0

0.0

100

500

1000

3000

5000

100

500

1000

3000

5000

100

500

1000

3000

5000

#offeatures

#offeatures

#of features

ours (fast solve only)

ours(fastsolve+ADMM)

(p=0.1)

(p=0.5)

(p=0.9)

103

103

103

(sec)

102

102

102

time

101

101

101

3000

4000

5000

6000

70001

3000

4000

5000

6000

7000!

3000

4000

5000

6000

7000

20

20

20

15

15

15

deb

10

10

10

5

5

5

0

0

0

3000

4000

5000

6000

7000

3000

4000

5000

6000

7000

3000

4000

5000

6000

7000

#ofsamples

#ofsamples

#ofsamples

ours(fastsolveonly)

ours(fastsolve+ADMM)

H More Experimental Results on Dynamical Systems

H.1 Results Summarization

Comparison with MIOSR We first show a direct comparison between OKRidge and MIOSR (based on SOS1 formulation). The results are in Appendix H.2, where we show OKRidge outperforms MIOSR in terms of both solution quality and running time.

Comparison with Different MIP Formualtions Next, we compare OKRidge with other MIP formulations (SOS1, big-M, perspective, and eigen-perspective) as done on the synthetic benchmarks. These formulations have never been applied to the differential equation experiments. Similar to the synthetic benchmarks, we show that OKRidge obtain better solution quality and smaller running time. The results are shown in Appendix H.3.

Comparison between Beam Search and More Heuristics In addition, we also compare our beam search method with many more heuristic methods from the sparse regression literature. Beam search outperforms all methods in terms of solution qualities in presence of highly correlated features. Block Decomposition algorithm also achieves high-quality solutions, but is much slower than beam search. The results are shown in Appendix H.4

Comparison with Different MIP Formulations Warm Started by Beam Search Due to the superiority of beam search, we give beam search solutions as warm starts to Gurobi with different MIP formulations and repeat the experiments. Our warm start improve the solution quality and running time, but OKRidge still outperforms other MIP formulations in terms of running time. The results are shown in Appendix H.5.

Comparison with Other MIP Solvers and Formulations Lastly, we solve the same problem using the MOSEK solver, l0bnb, and SubsetSelectionCIO. For this application, l0bnb has a time limit of 10 seconds for each ℓ 0 regularization and 60 seconds for the total time limit. All other algorithms have a total time limit of 30 seconds as mentioned in the experimental setup subsection. Although they are not as competitive as Gurobi as we have shown on the synthetic benchmarks, we still conduct the experiments for the purpose of thorough comparison. Results in Appendix H.6 show that OKRidge outperforms these MIP solvers/formulations.

H.2 Direct Comparison of OKRidge with MIOSR

In the appendix, we directly compare MIOSR (based on the SOS1 formulation) and our method on the differential equation experiments. The results are shown in Figure 9.

Although MIOSR is faster than OKRidge on the Lorenz system, OKRidge is able to find better solutions. See Appendix H.3 and Figure 10 where the eigen-perspective formulation doesn't have this problem and can produce high-quality solutions.

On the Hopf system, MIOSR is faster than OKRidge. However, the number of features is small ( p = 21 ), and the running time is negligible compared to Lorenz and MHD systems.

On the MHD system ( p = 462 ), OKRidge outperforms MIOSR in terms of both solution qualities and running times. This shows that OKRidge is much more scalable to high dimensions than MIOSR.

Hopf

MHD

L2 Coefcient Error True Positivity Rate

1.00

1.00

1.00

0.75

0.75

0.75 -

0.50

0.50

0.50

0.25

0.25

0.25

0.00

0.00

0.00

100

10-1

100

10

10-2

10-1

10-3

10

-3

10-2

2 ×101

Train RMSE

10-

5 ×10-2

101

4×10-2

6×100

103

Test RMSE

10-1

10-1.

10-2

100

10-3

Running Time

4000

20

1000

0000

500

10

2000

1000

0

0

100

101

100

101

100

101

Training Trajectory Length (s)

Training Trajectory Length (s)

Training Trajectory Length (s)

H.3 Comparison of OKRidge with SOS1, Big-M, Perspective, and Eigen-Perspective Formulations

Besides the MIOSR baseline (which is based on the SOS1 formualtion), we also compare with other MIP formulations, including Big-M ( M = 50 ), perspective, and eigen-perspective formulations. The results are shown in Figure 10. The big-M method produces errors on the Lorenz system, so its result is not shown on the left column. We have discussed about the SOS1 formulation in the last subsection. On the Lorenz system, OKRidge is faster than the perspective and eigen-perspective formulations. On the Hopf system ( p = 21 ), OKRidge is slower than Gurobi, but OKRidge takes only 20 seconds. On the high-dimensional MHD system, OKRidge is significantly faster than all MIP formulations.

102

Hopf

MHD

Coefficient Error True Positivity Rate

1.00

1.00

1.00

0.75

0.75

0.75

0.50-

0.50

0.50

0.25

0.25

0.25

0.00

0.00

100

100

100

10

-1

10-

10

10-1

10-2

7

10

-3

10-2

10-

Train RMSE

10-

100

10-

101

101

Test RMSE

10-1

100

10-2

101

10-

10-3

2000

40

Running Time

4000

1500

30

3000

1000

20

2000

500

10

1000

0

0

100

101

100

101

100

101

Training Trajectory Length (s)

Training Trajectory Length (s)

Training Trajectory Length (s)

SOS1

bigM

persp.

I eig. persp.

ours

H.4 Comparison of Our Beam Search with More Heuristic Baselines

Next, we compare our proposed beam search method with more heuristic methods from the sparse regression literature. The results are shown in Figure 11. Since many heuristic method implementations do not consider the ridge regression, we set ℓ 2 regularization for all algorithms to be 0 for a fair comparison. Since the features are highly correlated, most of the existing heuristic methods do not produce high-quality solutions except the Block Decomposition algorithm. However, our beam search method is significantly faster than the Block Decomposition algorithm.

Moreover, for the purpose of completeness, we also compare with LASSO, the classical sparse learning method. Other researchers have found that LASSO performs poorly in this context. We confirm this claim experimentally. The result is shown in Figure 12.

Hopf

MHD

L2 Coeficient ErrorTruePositivityRate

1.00

1.00

1.00

0.75

0.75

0.75

0s'0

0.50

0.50

0.25

0.25

0.25

0.00

0.00

100

101

102

100

100

10-2

10-3

10

101

Train RMSE

10-

100

101

10-1

10°

102

Test RMSE

10-

101

101

100

10-1

10

800

Running Time

150

40

600

100

400

50

20

200

0

0

0

100

101

100

101

100

101

Training Trajectory Length (s)

Training Trajectory Length (s)

Training Trajectory Length (s)

十

ssp

proximal_gradient_10c

romp

BlockDecAlg_c

cosamp

omp

qpm

GreedySparseSimplex

ours (beam search)

Hopf

MHD

True Positivity Rate

1.0

1.0

1.0

0.8-

0.8

0.8

0.6-

0.6

0.6

42

0.4

0.4

0.2

0.2

0.0

0.0

0.0

CoefficientError

10°

100

100

10

10-1

10

-2

10-1

10-2

10-3

102

103

Train RMSE

10°

102

101

10-1

103

Test RMSE

10-1

10°

10

10-2

10-1

100

10-3

100

101

100

101

100

101

Training Trajectory Length (s)

TrainingTrajectoryLength(s)

Training Trajectory Length (s)

ours (beam search only)

sino I

H.5 Comparison of OKRidge with MIPs Warmstarted by Our Beam Search

A natural question to ask is whether Gurobi would be faster than OKRidge in terms of running speed if we give the high-quality solutions from last subsection as warm starts to Gurobi . Here, we compare with other formulations warmstarted by our beam search solutions. The results are shown in Figure 13. By comparing Figure 10 with Figure 13, we see that our beam search solutions, if used as warm starts, improve Gurobi's speed. However, OKRidge is still significantly faster than all MIP formulations on the high-dimensional MHD system.

Hopf

MHD

LCoefcient ErrorTruePositityRate

1.00

1.00

1.00

0.75

0.75

0.75

0.50-

0.50

0.50

0.25

0.25

0.25

0.00

100

0.00

0.00

100

10-1

10-1

10-2

10-3

10-

3×101

Train RMSE

2 ×101

10

6×10-2

101

4 × 10-2

6×100

4×100

3×10

100

Test RMSE

101

10-1

100

10-3

2000

Running Time

1500

30

4000

20

3000

1000

2000

500

10

1000

0

100

101

100

101

0

100

101

Training Trajectory Length (s)

Training Trajectory Length (s)

Training Trajectory Length (s)

SOS1 + warm start

bigM+warm start

persp.+ warm start

I eig. persp.+ warm start

ours

H.6 Comparison of OKRidge with MOSEK solver, SubsetSelectionCIO, and L0BNB

Lastly, we compare with other MIP solvers/formulations, including the MOSEK solver, SubsetSelectionCIO, and l0bnb. The results are shown in Figure 14.

MOSEK (perspective formualtion) produces good solutions on the Hopf dataset, but could not produce optimal solutions on the Lorenz system. MOSEK (optimal formulation) produes suboptimal solutions on the Lorenz and Hopf systems. However, we are using MOSEK to solve the relaxed convex SDP problem. This demonstrates why we need to tackle the NP hard problem instead of the relaxed convex problem. MOSEK runs out of memory on the MHD dataset, so we don't show the results on the MHD dataset.

SubsetSelectionCIO doesn't do well on the Lorenz and MHD systems. On many instances, SubsetSelectionCIO would have errors.

l0bnb performs poorly on these datasets. One major reason is that l0bnb could not retrieve the exact support size ( k = 1 , 2 , 3 , 4 , 5 ) because the sparsity is controlled implicitly through ℓ 0 regularization. It takes a long time to run these differential experiments. l0bnb couldn't find a good ℓ 0 regularization which produces the right sparsity level within the time limit.

Please see Appendix G.1 for a detailed discussion of the drawbacks on using MOSEK, SubsetSelectionCIO, and l0bnb.

Hopf

MHD

Coefficient Error True Positivity Rate

1.00

1.00 -

1.00

0.75

0.75

0.75

0.50

0.50

0.50

0.25

0.25

0.25

0.00

0.00

100

10°

10°

10

10

10

10-

10

10

-3

10

7

Train RMSE

10

100

102

10

10

10

10

Test RMSE

10-

10°

101

10-

10

10

Running Time

3000

1500

6000

2000

1000

4000

1000

500

2000

0

0

100

101

100

101

100

101

Training Trajectory Length (s)

Training Trajectory Length (s)

Training Trajectory Length (s)

MSK persp.

l0bnb

subsetselectClO+warmstart

MSK persp.+ warm start

subsetselectCio

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel branch-and-bound technique called OKRidge to solve sparse Ridge regression problems to learn differential equations. The paper makes 3 contributions:\n1) A saddle point formulation of the optimization problem at hand which allows the use of a branch-and-bound algorithm leveraging the special structure (sparsity) of the problem.\n2) This efficient branch-and-bound algorithm constructs tight lower-bounds thanks to the saddle-point formulation.\n3) A beam search that leverages previous searches for past nodes to speed up the search process of the current node.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. Apart from some details (see question section below) that might be unclear to the non-expert readers, the context of the paper is clear and the literature review is comprehensive.\n\nBesides, several extensive experiments have been made. Given the number of hyperparameters in the method ($\\lambda$, $\\lambda_2$) and the hidden parameters that might impact the convergence speed of the proposed methodology (correlation $\\rho$, setup size $n$ and $p$, $\\textrm{SNR}$), the sensitivity analysis carried out in the experiments section is welcome.\n\nWEAKNESSES:\nThe paper does not have major weaknesses. \n\nBesides, as a suggestion, since your experiments assess the convergence speed as well as the quality of the sub-optimality gaps yielded by the solvers, you can use the Benchopt framework for more impactful figures: https://github.com/benchopt/benchopt . For instance, figure 1 could be summarized into a one-row figure.\n\nQUESTIONS:\n- As a non-expert in the field of differential equation learning, it is unclear to me why $z_j \\in \\{0, 1\\}$. Is it a generic formulation or does this integer constraint arise often in this field? A clarification in the intro would help the reader understand why it is necessary and why you compare yourselves to MIP solvers.\n\n- How does OKRidge scale with respect to the number of features?\n\n- \"Our algorithms are written in Python, Any improvements we see over commercial MIP solvers, which are coded in C/C++, are solely due to our specialized algorithms\" (l. 204): how can a Python routine (even very efficient) can beat an algorithm written in C/C++? Are you using a JIT compiler like Numba to compile critical parts of the code? If not, have you tried? It could significantly increase the runtime speed.\n\nLIMITATIONS:\nI see one unaddressed limitation to the method: while OKRidge scales easily to hundreds of thousands of samples, it does not seem to scale as well on the number of features. For instance, as stated in l.132, deriving the lower bound requires inverting a $p\\times p$ matrix, which does not scale well for $p \\gg 10000$.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

comments: SUMMARY:
The paper proposes a lower bound calculation method and warm start method for sparse ridge regression via a branch-and-bound approach. Starting from a saddle point formulation, the lower bound on the original objective can be given by either a linear system or ADMM problem which can be efficiently solved. The warm-start is given by the beam search method. Experiments are provided to demonstrate the better efficiency of the proposed approach over baselines.

STRENGTHS:
1. propose a scalable method for sparse ridge regression.
2. demonstrate the feasibility of the proposed method in practice.

WEAKNESSES:
1. The main contribution lies in efficient certifiable lower bound which however is just part of the whole algorithm. It would be better if the whole algorithm can be explicitly given in the main text.
2. There is no analysis that gives the complexity of the whole algorithm to achieve a acceptable optimality gap.

QUESTIONS:
See above

LIMITATIONS:
Limitations are addressed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10

comments: SUMMARY:
This paper proposes a method, namely OKRidge, to solve spare ridge regression problems and identify terms of differential equations of nonlinear dynamical systems. In particular, OKRidge is formulated as a branch-and-bound algorithm, and this paper provides a lower bound calculation.

STRENGTHS:
$\mathbf{1}.$ The proposed method is novel, mainly on the formulation of the optimization problem as prescribed by Eq (12), as well as the approach to solve the problem.

$\mathbf{2}.$ The empirical analysis is thorough and results are promising compared with various solvers/algorithms, including commercial solver, e.g., GuRoBi. The experiments very well supports the proposal of the algorithm with adequate evidence.

$\mathbf{3}.$ The proposed method is very complete, which covers a major branch-and-bound algorithm with novel lower bound approach and a heuristic to search in a tree.

WEAKNESSES:
$\mathbf{1}.$ Despite the novel formulation of the optimization problem (as well as the lower bound derivation), it might be still combination of known approach to derive the package of approach, including BnB, dynamic programming and heuristics.

QUESTIONS:
$\mathbf{1}.$ Please see $\mathbf{1}.$ in $\mathbf{Weakness}$ section.

LIMITATIONS:
This paper includes a limitation statement, and I do not have any concerns regarding a potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm to solve sparse ridge regression problems -- which are standard ridge regression problems with an additional l0 sparsity constraint (so this problem is NP Hard in general). The algorithm is a branch and bound based approach that uses a novel, tighter, lower bound computed by solving a saddle point reformulation of the problem using ADMM. The reformulation is done using a perspective transform approach along with a reparameterization trick. Additionally, beam search over the chosen non-zero coordinates is used as a heuristic to obtain better upper bounds while expanding the branch and bound tree. A provable guarantee on the optimality gap of the solution obtained from this algorithm is also provided. Experiments on synthetic data as well as data from dynamical systems like the Hopf bifurcation, the Lorenz system and a non-linear magnetohydrodynamical model demonstrate the efficacy of this algorithm compared to existing open source and commericial benchmarks.\n\nSTRENGTHS:\n1. The paper is well very written and clear.\n2. I greatly appreciate the detailed and helpful annotations in each step of the proofs as well as the algorithms.\n3. The main result is obtained by combining a novel reformulation of the sparse ridge regression problem with ADMM to obtain a tighter lower bound that can be used in branch and bound.\n4. Thorough experiments were conducted to study and analyze the performance of the described algorithm.\n\nWEAKNESSES:\n1. Perhaps the authors could include experiments on more applications that arise from challenging physical systems beyond the three presented in the paper, since the beam search alone was sufficient for the synthetic problem.\n2. The improvement over MIOSR in the case of the sparse differential equations like Lorenz, Hopf and MHD is not very significant (Figure 3), however it still matches the performance of using a proprietary solver.\n\nQUESTIONS:\n1. How does this method compare to relaxing the sparsity constraint to a lasso regularization, and approximating a sparse solution from the solution of the lasso?\n2. Have the authors conducted experiments using the simpler (but not as tight) lower bound obtained by solving the ridge regression linear system? If so, were there significant differences between this and the baseline? Further, are there any theoretical guarantees that can be shown for this simpler bound?\n\nLIMITATIONS:\nYes, the authors addressed limitations of OKRidge in the case of smaller feature dimensions.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose OKRidge, a fast algorithm to solve k-sparse ridge regression problems with provable optimality.\nThe proposed method is a customized branch-and-bound approach, leveraging the specific structure of the ridge regression problem in contrast to black-box solvers such as Gurobi.\nNamely, OKRidge first uses a new saddle point formulation specifically for efficiently computing lower bounds (either \"roughly\" by solving a rather simple linear system of more tightly with an ADMM scheme.\nSecond, a beam search heuristic is proposed to find a near-optimal solution, with the global BnB structure in mind. The near-optimality of such solutions is provably guaranteed.\nThird, relatively straightforward branching and queuing strategies are also proposed to \"complete the package\", building a full custom solver.\nLast, the authors benchmark OKRidge against a wide variety of other approaches, not only limited to MIP formulations. Both performance of the solutions, and compute time are evaluated on synthetic benchmarks and on sparse identification of nonlinear dynamical systems.\n\nSTRENGTHS:\nI think that the significance of this work lies in the careful and smart design and integration of multiple components, building into what seems to be a very efficient custom solver for a relatively broad class of problems.\nWhile each of this components, taken individually might not be ground-breaking, the method as a whole seems to build into a tool that has important significance.\nAlthough I am mostly an outsider and may very well miss some important relevant references, the authors are doing a very good job at placing their contribution in the context of a very abundant existing literature.\nThe clarity of the paper is also to be praised. The appendices, although extremely voluminous (in part due to the hundreds of tables reported from the experiments), are completely self-contained.\nThe proofs of the different theorems read well and are somewhat insightful. The Figures and algorithms summarized in Appendix E are also very helpful to bring all the pieces together.\nFor the quality, although I am not familiar with large parts of the work, I was not able to find mistakes in the proofs and derivations. To be completely fair (maybe due to the very good quality of the presentation), some derivations can be a bit tedious, but seem relatively straightforward (which is not meant to downplay the significance of the contributions in any way).\n\nWEAKNESSES:\nAs stated in the previous section, it could be argued that none of the contributions, taken in isolation are \"novel enough\". But I don't believe that would be a fair argument.\nOn the other hand, while the authors did a great job at presenting their method clearly and carefully, I find that it could have been helpful or interesting to include a little more comments on their results.\nFor instance, it is great that provable guarantees are derived for the heuristic solution found by their beam-search (section 3.2.1) but it could have been nice to understand a bit more how tight those bounds are (or if they are insightful at all).\nMore importantly, although the experimental results look good, do the authors consider this heuristic as an important limitation? Are there problems (or datasets) for which the sub-optimality of the beam-search is problematic?\nAlso, it would be great to understand, in practice, where the compute time is being spent as it might be a good way to understand where the computational bottlenecks of OKRidge are and where there is potential room for further improvement.\n\nQUESTIONS:\nI already included a few questions in the previous section.\nTo clarify, I have a few more questions:\n1) How do you estimate \\lambda_min and \\lambda_max, to determine \\rho for ADMM? Is it critical to estimate it carefully?\n2) It might be a silly question but do you have any insight on how to pick the optimality gap \\epsilon in practice? Would it make sense to pick it adaptatively?\n3) What about the scenario where you don't \"know\" k a priori but would like to find which level of sparsity offers a good trade-off? Can OKRidge be easily adapted to solve the problem for different levels and not have to rebuild the solution from scratch for every k?\n4) In the end, while learning dynamical systems is an important problem of its own, OKRidge should be useful for other problems which can be cast as a sparse ridge regression. Why so much emphasis on dynamical systems?\n\nLIMITATIONS:\nSome limitations of OKRidge are clearly identified and pinpointed towards the end of the manuscript.\nPotential negative societal impacts were not discussed, but I don't foresee any.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript the authors consider the ridge regression with a hard sparsity constraints. Since the problem is posed in the fixed design setting, the paper is on the optimization theory, while the main motivation for its study authors find in the applications related to data-driven methods for dynamical systems. Using the work of Atamturk and Gomez 2020 on max-min relaxation of the original problem, novel lower bounds are developed (one in closed form and the other using ADMM to improve the tightness). These results coupled with Beam-Search of Wiseman and Rush 2016 are used within Bound-and Branch framework to propose an algorithm with a theoretical bound on the optimality gap. Experiments are run on synthetic data to compare with SOTA methods for the studied problem, as well as on datasets from Brunton, Proctor and Kutz 2016 on toy problems in dynamical systems to showcase the application in the sparse identification of (continuous-time deterministic) equations of motion from data under the (strong) assumptions that the finite-dimensional basis of a function space is know and that dataset contains states and their time derivatives.\n\nSTRENGTHS:\nThe paper is very well written, apart of issues stated below. All the important concepts are properly introduced in the main body and extensively elaborated upon in the appendix. Main contribution well presented and the novelty of the new ideas w.r.t. existing work is made very clear. Up to my knowledge, relevant references are cited. The theoretical results are correct, except minor issues explained below. Numerical experiments are very detailed.\n\nWEAKNESSES:\nIn the following I give the main weaknesses of this paper:\n\n __(A) Title vs content:__ The paper is content wise purely work on optimization. Namely, the link to ML and to learning of dynamical systems is only through the problem which is an empirical risk minimization for linear model with $\\ell_2$ regularization and $\\ell_0$ constraints. Hence, there is no really any learning of dynamical systems as the title suggests. Therefore, since I find that the paper is in its current form of interest to ML community, I propose the authors to remove from the title “for Learning Dynamical Systems”.  If not, then much more than elaborated discussion needs to be done to justify the title. In particular:\n\n1. Assumptions under which the system is learned from data providing explaining when they are met in practice. My opinion is that relying just on SINDy approach is very limited since this aspect is quite vague.\n\n2. Comparison with other SOTA methods to learn dynamical systems from data is needed (meaning those solving the true problem and not an empirical one) with the discussion on benefits and limitations of the current proposal. Also some insights in how statistical learning theory of the proposed method might be developed. \n\n3. At least one experiment on the performance of the method on the real dataset of challenging dynamical system (unknown non-polynomial nonlinearities in ODE). \n\n\n__(B) Error in Thm. 3.1.:__ When $Q_\\lambda$ is singular, i.e. when $\\lambda = \\lambda_{\\min}(X^\\top X)$, not every $c$ is in the range of $X^\\top$, and, hence, reparametrization doesn’t hold, yielding that (12) is not equivalent to (10). Consequently, when $\\sigma_{\\min}(X)=0$, e.g. when number of features exceeds number of samples,  Thm. 3.1. on which all else is based doesn’t hold. This aspect needs to be checked and clarified. \n\n__(C) Notion of optimality:__ In the paper, starting from the title and abstract, the word _optimal_ is used. I believe that many of such uses are not entirely clear. To me it seams that sometimes it means just feasible k-sparse solution. Although upper bound and lower bound guarantees are shown, in the paper I do not find theoretical guarantee for the convergence of the proposed method to the optimal solution stated. Can the authors, please, elaborate on this? \n\n\nTo conclude, the current evaluation score I have given reflect the aforementioned issues. Depending on the authors reply, I am willing to reconsider it.\n\nQUESTIONS:\nPlease elaborate on the claim in the line 57/58. What is the meaning of outperforms? Seams that on the validation set it performs similarly as MIOSR.\n\nLIMITATIONS:\nApart of currently discussed limitations, if the title is kept as is, the authors need more discussion as noted above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new framework for computing the k-sparse ridge regression. Specifically, the framework considers the Lagrangian dual and relaxes the lower bound to an ADMM problem with an isotonic proximal operator, which has a linear time solution. Combined with beam search and branch-and-bound, the authors demonstrate the framework's effectiveness in the experiment. Further, they claim to prove a tighter approximation bound based on the beam-search result.\n\nSTRENGTHS:\nThe paper's main contribution is the relation between the k-sparse problem with an isotonic proximal operator, which can be computed in linear time ($O(n log k)$ for sorting the coefficients.) I believe this technique is new and may have a significant impact on other works with k-sparse constraints. Actually, the framework can be generalized to any strongly convex and Lipschitz C^1 optimization with k-sparse constraints. The proposed Lagrangian dual lower-bound and isolation of the binary variables only depends on the k-sparse constraints, not the ridge regression itself. In addition, the proofs are clearly written with annotations on every inequality, which helps a lot in verification.\n\nWEAKNESSES:\nDespite the proofs being clearly annotated, there are still major bugs in the main theorems. However, I think these bugs may be corrected by weakening the theoretical results, and these errors may not impact the practical value of the work. Specifically, the $L_{\\text{ridge-}\\lambda}^\\text{saddle}$ problem should be a lower bound of equation (12), not an equivalent problem. To prove equivalence, we need to prove that the set of optimality conditions are the same, and the error can be easily checked by taking stationary condition on c in eq (10), which leads to the missing slackness condition $2\\beta_j=c_jz_j$ which doesn't appear in the stationary condition of the new formulation in eq (12) unless other assumptions are made. And I think the subtle error in the derivation is that the proof of Theorem 3.1 is not rigorous, i.e., not every step in the L569 derivation is an equivalence relation. The equation (40) may not be one-to-one, and the step of taking $\\beta=\\gamma$ with additional constraint eq (40), reparameterization of $c$, and moving $\\gamma$ from the minimum operator to the maximum operator may not be rigorous. There may be a way to prove it more rigorously, and in the worst case, the derivation becomes a relaxation, which is still helpful. However, we will need to see the more formal derivation first.\n\nSecond, there are also bugs in the authors' proof for the beam search in Theorem 3.4. In Section 3.2, the authors select the new support by the Gauss-Southwell rule of largest improvement. However, when combined with beam search, it is required to select the candidates with the lowest objective value, which is a conflict. As we see in Algorithm 1 and Algorithm 7 in the Appendix, the actual implementation generates the next step using the B largest improvements for each candidate and chooses the B best candidates with the best objective value, which differs from the proof in Theorem 3.4. On the other hand, the theorem and proofs on the beam search solution are independent to the number of candidates $B$ in the authors' proof, so there must be some error or hidden assumption (e.g., $B\\geq p^k$) here.\n\nHowever, I still consider this draft as a good paper and recommend it for acceptance. Just that these bugs need to be addressed, or the Theorems need to be weakened. However, please address these main issues in the rebuttal.\n\nQUESTIONS:\n1. The proposed method is an algorithm for general sparse ridge regression with experiments on the dynamic system. There is no other utilization of the property of the dynamic system here other than the data itself. So I don't think the title is appropriate.\n2. Appendix L562: missing z.\n3. Appendix L567: The page overflows, so part of the equations are hidden.\n4. Appendix L588: The conclusion (Therefore $\\gamma=$) is your assumption.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}