Title: The Reliability of OKRidge Method in Solving Sparse Ridge Regression Problems
Venue: NeurIPS 2024
Accepted: yes
Keywords: regression
Abstract: Sparse ridge regression problems play a significant role across various domains. To solve sparse ridge regression, Liu et al. (2023) recently propose an advanced algorithm, Scalable Optimal $K$-Sparse Ridge Regression (OKRidge), which is both faster and more accurate than existing approaches. However, the absence of theoretical analysis on the error of OKRidge impedes its large-scale applications. In this paper, we reframe the estimation error of OKRidge as a Primary Optimization ($\textbf{PO}$) problem and employ the Convex Gaussian min-max theorem (CGMT) to simplify the $\textbf{PO}$ problem into an Auxiliary Optimization ($\textbf{AO}$) problem. Subsequently, we provide a theoretical error analysis for OKRidge based on the $\textbf{AO}$ problem. This error analysis improves the theoretical reliability of OKRidge. We also conduct experiments to verify our theorems and the results are in excellent agreement with our theoretical findings.

The Reliability of OKRidge Method in Solving Sparse Ridge Regression Problems

Abstract

Sparse ridge regression problems play a significant role across various domains. To solve sparse ridge regression, [1] recently proposes an advanced algorithm, Scalable Optimal K -Sparse Ridge Regression (OKRidge), which is both faster and more accurate than existing approaches. However, the absence of theoretical analysis on the error of OKRidge impedes its large-scale applications. In this paper, we reframe the estimation error of OKRidge as a Primary Optimization ( PO ) problem and employ the Convex Gaussian min-max theorem (CGMT) to simplify the PO problem into an Auxiliary Optimization ( AO ) problem. Subsequently, we provide a theoretical error analysis for OKRidge based on the AO problem. This error analysis improves the theoretical reliability of OKRidge. We also conduct experiments to verify our theorems and the results are in excellent agreement with our theoretical findings.

1 Introduction

Sparse Ridge Regression (SRR) has achieved notable success across various machine learning applications, including statistics [2], signal processing [3], dynamical systems [4, 5], and others. In this paper, we are interested in addressing the following k -sparse linear regression problem with additive noise:

where β ∗ ∈ R d represents the 'true" weight parameter, X = ( x 1 , x 2 , · · · , x n ) ⊤ ∈ R n × d is the input measurement matrix, y = ( y 1 , y 2 , · · · , y n ) ⊤ ∈ R n is the real output responses, ϵ = ( ϵ 1 , ϵ 2 , · · · , ϵ n ) ⊤ ∈ R n is the noise vector, k ∈ Z + specifies the maximum number of nonzero elements for the model, ∥ · ∥ 0 denotes the number of nonzero elements of the given vector. Moreover, the entries of X are drawn i.i.d. from N (0 , 1) ; the entries of ϵ are drawn i.i.d. from N (0 , σ 2 ) ; and we assume k d is a constant and lim d →∞ n ( d ) d = δ ∈ (0 , 1) .

The formulation (1) represents a black box model where β ∗ is fixed. Given X and y , to determine the target vector β ∗ , the most basic method is solving the following k -Sparse Ridge Regression Optimization ( k -SRO), as outlined by [1, 6, 7]:

where λ > 0 is a regularizer parameter, and ∥ · ∥ 2 denotes the Euclidean norm. Our paper focuses on the worst-case scenario ∥ β ∗ ∥ 0 = k . This k -SRO is different from the traditional ridge regression

due to the constraint of k -sparse structure for β . The k -SRO problem (2) is NP-hard, and is more challenging in the presence of highly correlated features [8].

Two main types of algorithms are commonly employed for solving k -SRO problem (2): heuristic algorithms [9, 10] and optimal algorithms [11]. However, heuristic algorithms lack the ability to assess the solution quality, while the optimal algorithms are slow. In order to rapidly solve k -SRO problem (2) while ensuring solution optimality, [1] introduces a highly efficient method called OKRidge. Therefore, a complete algorithm of OKRidge, including how to choose hyper-parameters can be seen in the original paper [1]. OKRidge substitutes k -SRO problem (2) with an unconstrained optimization on a novel tight lower bound. The experiment results in [1] show that OKRidge is superior to heuristic algorithms, optimal algorithms, and existing mixed-integer programming (MIP) formulations solved by the commercial solver Gurobi. Nevertheless, the absence of theoretical error analysis for OKRidge impedes its scalability in practical applications.

In this paper, we provide theoretical error analysis for OKRidge utilizing the framework of the CGMT [12]. Specifically, we propose another novel tight lower bound L OKRidge ( β ) to replace k -SRO problem (2):

where ⊙ denotes Hadamard product, and SumTop k ( · ) represents the summation of the largest k elements of a given vector. The tight lower bound (3) is equivalent to that proposed by [1]. Thus, L OKRidge ( β ) can replace the objective function of OKRidge. It is noteworthy that our proposed regularizer, defined as γ ( β ) = SumTop k ( β ⊙ β ) , differs from any previously proposed instances by [12]. Then, the optimal solution obtained by OKRidge is

[1] utilizes ˆ β as the estimate of β ∗ in problem (1). By combining formulations (1) and (3), the estimation error of OKRidge can be obtained through the following normalized optimization problem:

where w := β -β ∗ is a random variable with randomness from the random variables X and ϵ , and the estimation error can be measured by ∥ w ∥ 2 . Subsequently, we transform the optimization (5) into a PO problem about the error of OKRidge, using the Fenchel-Moreau theorem [13]. Then, we employ the CGMT framework to substitute the complex PO problem with a simplified AO problem. Finally, we present the theoretical error analysis of OKRidge based on the AO problem. Our theoretical results focus on the Normalized Squared Error (NSE) of OKRidge and can be summarized as:

where NSE := ∥ ˆ β -β ∗ ∥ 2 2 / σ 2 , and ∆( ˆ λ ) is a function of λ . These theoretical results indicate that if the regularizer parameter λ used in OKRidge is constant, the NSE limit of OKRidge is also fixed. Moreover, ˆ β learned by OKRidge is reliable to estimate β ∗ , due to lim d →∞ lim σ → 0 ∥ ˆ β -β ∗ ∥ 2 P -→ 0 . The comprehensive experiments of OKRidge on real-world examples were conducted by the NeurIPS 2023 paper [1] (see Figure 3 and Appendix H in [1]), which demonstrates that the error of OKRidge tends to zero. Our analysis explains the experimental phenomenon observed in [1], strengthens the theoretical underpinnings of OKRidge, and provides theoretical reliability for its broad application.

We also conduct numerical experiments to validate our theorems. The findings demonstrate that the NSE converges to a fixed constant determined by λ , aligning excellently with our theoretical predictions.

1.1 Outline

The structure of the remaining sections in this paper is as follows. Section 2 provides a review of related work. Section 3 offers background information on OKRidge, CGMT, and basic concepts. Section 4 introduces an alternative tight lower bound for the objective function of OKRidge. In Section 5, we convert the estimation error of OKRidge into a PO problem and simplify it into an AO problem using CGMT. Subsequently, an estimation error analysis of OKRidge based on the AO problem is conducted. Section 6 presents the experimental results. Finally, we conclude with a summary in Section 7. Additionally, the limitaion and impact of our work are detailed in Appendix A

2 Related work

2.1 Heuristic and Optimal Methods

Heuristic methods approximate solutions to optimization problems based on practical experience [14], including ensemble methods [15], swapping features [16], greedy methods [17], etc. While heuristic methods are fast, they often become trapped in local minima, and their solution quality cannot be assessed due to the absence of a lower bound on performance. Optimal methods aim to precisely solve sparse regression problems, such as the big-M method [18], the conditional-valueat-risk (CVaR) approach [19], big-M free mixed integer second order conic (MISOC) method [19], and so on. However, exact optimal methods are slow, particularly for large instances, to achieve near-optimality [20, 21]. To address the limitations of heuristic and optimal methods, [1] proposes an efficient approximation algorithm, OKRidge. Experimental results in [1] demonstrate that OKRidge outperforms heuristic algorithms, optimal algorithms, and existing MIP formulations solved by the commercial solver Gurobi.

2.2 Lower Bound Methods

Lower bound methods are capable of solving the NP-hard k -SRO problems. Several algorithms utilize the lower bound method, such as SOS1 formulation [18], big-M formulation [18], Subset Selection CIO method [22], and others. However, the SOS1 formulation lacks scalability in high dimensions, the big-M formulation is sensitive to hyperparameters, and the Subset Selection CIO method runs slowly. Recently, the perspective formulation [6, 23, 24] has been employed to induce a convex relaxed lower bound that is easier to solve. Building upon the perspective formulation, [1] proposes a novel lower bound used as the objective function for the OKRidge method.

2.3 Normalized Squared Error

NSE, defined as ∥ ˆ β -β ∗ ∥ 2 2 / σ 2 , serves as a natural measure of the estimation error. NSE is an important indicator in signal-to-noise ratio scenes [25, 26]. Bounds on NSE have been derived by [27, 28]. Additionally, [29] is the first to precisely formulate the limiting behavior of NSE. These studies primarily consider a Gaussian sensing matrix X and utilize the Approximate Message Passing (AMP) framework for analysis [30, 31]. These achievements motivate us to utilize NSE for evaluating the estimation error of the OKRidge method.

3 Preliminary

3.1 Relaxed Transformation of k -SRO

According to [1], the k -SRO problem (2) can be reformulated as the following optimization problem:

where z = ( z 1 , z 2 , · · · , z n ) ⊤ ∈ R d , and L ridge ( β ) := ∥ y -Xβ ∥ 2 2 + λ ∑ d j =1 β 2 j . This problem (7) remains NP-hard under the sparsity constraint [32]. Existing methods such as SOS1, big-M, or the perspective formulation do not leverage the k -sparse structure of the problem. [1] develops a novel method, OKRidge, to preserve the special structure through the following relaxed transformation.

By employing the perspective formulation [33, 34] and the Fenchel conjugate [35], [1] transforms the problem (7) to a new perspective optimization problem:

where c = ( c 1 , c 2 , · · · , c n ) ∈ R d , and L Fenchel ridge ( β,z,c ) := ∥ y -Xβ ∥ 2 2 + λ ∑ d j =1 ( β j c j -c 2 j 4 z j ) . This transformation does not change the optimal solution of the problem (7) [1, 35], indicating that problem (7) can be replaced by problem (8). To efficiently solve problem (8), [1] further relaxs

the binary constraint { 0 , 1 } to the interval [0 , 1] , ultimately yielding the following relaxed convex optimization problem:

According to problem (7) , to preserve the special sparse structure of β , we always have β j = 0 if z j = 0 . Directly solving the min-max problem (9) is computationally challenging. [1] utilizes the relaxed problem (9) to obtain a tight lower bound for the problem (8), where the lower bound corresponds to the objective function of OKRidge.

3.2 The Convex Gaussian Min-max Theorem

The CGMT framework, introduced by [28], has been utilized to analyze the performance of solutions to non-smooth regularized convex optimization problems. It has achieved significant success in various practical applications, including regularized logistic regression [36], max-margin classifiers [37], adversarial training [38, 39] and others. These achievements inspire us to apply the CGMT framework to analyze the NSE of the OKRidge method.

CGMT originates from Gordon's Gaussian Min-max Theorem (GMT) [40], which provides probabilistic bounds on the optimal cost of PO problem via a simpler AO problem. CGMT further tightens the bounds under convexity assumptions. According to GMT, [41] introduces the following asymptotic sequence and notation.

Definition 3.1 (GMT admissible sequence) . The sequence { G ( d ) , g ( d ) , h ( d ) , S ( d ) w , S ( d ) u , ψ ( d ) } d ∈ N indexed by d , with G ( d ) ∈ R n × d , g ( d ) ∈ R n , h ( d ) ∈ R d , S ( d ) w ⊂ R d , S ( d ) u ⊂ R n , ψ ( d ) : S ( d ) w × S ( d ) u → R and n = n ( d ) , is said to be admissible if, for each d ∈ N , S ( d ) w and S ( d ) u are compact sets and ψ ( d ) is continuous on its domain. Onwards, we will drop the superscript ( d ) from G ( d ) , g ( d ) , h ( d ) .

A sequence { G ( d ) , g ( d ) , h ( d ) , S ( d ) w , S ( d ) u , ψ ( d ) } d ∈ N defines a sequence of min-max problems

Importantly, the formulation (10) is called Primary Optimization (PO) and the formulation (11) is called Auxiliary Optimization (AO) . Additionally, let w ( d ) Φ ( G ) denote the optimal minimizer of PO problem (10), and w ( d ) ϕ ( g,h ) denote the optimal minimizer of AO problem (11) . Define υ ( d ) : S ( d ) w → R as follows,

Clearly, ϕ ( d ) ( g,h ) := min w ∈S ( d ) w υ ( d ) ( w ; g,h ) . For a sequence of random variables {X ( d ) } d ∈ N and a constant c ∈ R , X ( d ) P -→ c denotes convergence in probability, i.e. ∀ ϵ > 0 , lim d →∞ P ( |X ( d ) -c | > ϵ ) = 0 . Based on the GMT admissible sequence and the notation introduced above, we present the CGMT below.

Theorem 3.2 (CGMT [12]) . Let { G ( d ) , g ( d ) , h ( d ) , S ( d ) w , S ( d ) u , ψ ( d ) } d ∈ N be a GMT admissible sequence as in Definition 3.1, for which additionally the entries of G , g , h are drawn i.i.d. from N (0 , 1) . Let Φ ( d ) ( G ) , ϕ ( d ) ( g,h ) be the optimal costs, and, w ( d ) Φ ( G ) , w ( d ) ϕ ( g,h ) the corresponding optimal minimizers of the PO and AO problems in (10) and (11). The following three statements hold

(i) For any d ∈ N and c ∈ R ,

(ii) For any d ∈ N . If S ( d ) w , S ( d ) u are convex, and, ψ ( d ) ( · , · ) is convex-concave on S ( d ) w ×S ( d ) u , then, for any µ ∈ R and t > 0 ,

(iii) Assume the conditions of (ii) hold for all d ∈ N . Let ∥ · ∥ denote some norm in R d and recall (12). If, there exist constants (independent of d ) κ ∗ , α ∗ and τ > 0 such that

(b) ∥ w ( d ) ϕ ( g,h ) ∥ P -→ α ∗ ,

(c) with probability one in the limit d →∞

then,

Theorem 3.2 indicates that, if the optimal cost ϕ ( g,h ) of (11) concentrates to some value µ , the same holds true for Φ( G ) of (10). Furthermore, under appropriate additional assumptions, the optimal solutions of the AO and PO problems are also closely related by ∥ w Φ ( G ) ∥ = ∥ w ϕ ( g,h ) ∥ , as n →∞ . This suggests that, within the CGMT framework, a challenging PO problem can be replaced with a simplified AO problem, from which the optimal solution of the PO problem can be accurately inferred [12]. Subsequently, we rewrite the lower bound of problem (9) in the form of PO problem (10) and analyze the minimizer of the simplified AO problem instead.

3.3 Basic Concept

Suppose f : R d → R and u,v ∈ R d , the Fenchel conjugate of f is defined as f ∗ ( u ) = sup v v ⊤ u -f ( v ) . Additionally, f ∗ is always convex and lower semi-continuous. According to the FenchelMoreau theorem [13], if f is convex and continuous, we have f ( v ) = sup u u ⊤ v -f ∗ ( u ) . In this paper, we utilize the following conjugate pairs

If γ ( · ) : R d → R is a convex function of β , the subdifferential of γ ( · ) at β ∗ is the set of vectors: ∂γ ( β ∗ ) = { s ∈ R d ∣ ∣ γ ( β ∗ + u ) ≥ γ ( β ∗ ) + s ⊤ u } . According to [13], ∂γ ( β ∗ ) is nonempty, convex and compact. Given h ∈ R d , we define dist ( h, ∂γ ( β ∗ )) = min s ∈ ∂γ ( β ∗ ) ∥ h -s ∥ 2 . Then, the Gaussian squared distance corresponding to the scaled subdifferential is defined as D ( τ ) := D ∂γ ( β ∗ ) ( τ ) := E h [ dist 2 ( h, τ∂γ ( β ∗ )) ] , where τ > 0 . Suppose C ( τ ) = -τ 2 ∂D ( τ ) ∂τ , lim d →∞ n d → δ ∈ (0 , 1) , lim d →∞ D ( τ ) n → ¯ D ( τ ) ∈ (0 , 1) , lim d →∞ C ( τ ) n → ¯ C ( τ ) . Based on the Gaussian squared distance, we define a map function:

We denote λ map as the solution of map ( τ ) -λ/ 2 = 0 . Since map ( τ ) depends on γ ( · ) and β ∗ , when the form of γ ( · ) and the value of β ∗ are determined, the λ map is fixed.

4 Tight Lower Bound in OKRidge

In this section, we utilize problem (9) to derive another novel lower bound for problem (8), serving as used as the objective function of OKRidge. Our lower bound is equivalent to the tight lower bound provided by [1]. Specifically, [1] eliminates the parameter c in problem (9) by setting the gradient of β to 0 , while we eliminate the parameter c by setting the gradient of c to 0 . These two methods

are equivalent due to the independence and convexity of c and β . Given any β and z , the optimality condition for c in problem (9) is taking ∂ L Fenchel ridge ( β,z,c ) /∂c =0 . Therefore, we have

̸

where diag ( z ) is a diagonal matrix with z on the diagonal. Inspired by this optimality condition, we present the following theorem.

Theorem 4.1. If we define the parameter c as (16), the problem (9) is equivalent to the following optimization problem:

̸

The proof of Theorem 4.1 follows Theorem 3.1 of [1] and is included in Appendix B for completeness. Following the approach by [1], we can approximately solve the problem (18) while still obtaining a feasible lower bound. We define a new function L ( β ) as:

For any β , L ( β ) serves as a valid lower bound for problem (7). We should choose z such that this lower bound L ( β ) is tight.

Theorem 4.2. The function L ( β ) defined in Equation (20) is lower bounded by

where ⊙ is Hadamard product, and SumTop k ( · ) denotes the summation of the largest k elements of a given vector.

The proof of Theorem 4.2 can be seen in Appendix C. Based on (9), (18) and (20), the tight lower bound (21) is equivalent to the one provided by [1], as both are derived through equivalent processes. OKRidge solves the original k -sparse problem (7) using this tight lower bound (21) as its objective function. If we define

OKRidge solves k -SRO problem (2) with

where we obtain L OKRidge of formulation (3). So far, we transform the constrained k -SRO problem (2) into the unconstrained optimization problem (22). Let

OKRidge regards ˆ β as the estimation of β ∗ in problem (1). Next, we apply CGMT to analyze the error ∥ ˆ β -β ∗ ∥ 2 2 for OKRidge.

where

5 The Error Analysis for OKRidge

5.1 From PO to AO

As discussed in Section 4, the estimation error of OKRidge is characterized by ∥ ˆ β -β ∗ ∥ 2 2 . Taking formulation (1) into the properly normalized objective (22), OKRidge (22) can be equivalently transformed to the following optimization:

The crucial step is to convert (23) into a PO problem within the framework of CGMT. We introduce the new variable w := β -β ∗ and apply the Fenchel-Moreau theorem (14) to formulation (23),

where w ∈ R d , u ∈ R n . Based on (10) and (24), the PO problem corresponding to the estimation error of OKRidge is

where

Since the entries of X are drawn i.i.d. from N (0 , 1) , to replace the challenging PO problem (25) with a simplified AO problem through CGMT, ψ ( w,u ) should be a convex-concave function. The following Lemma illustrates that the ψ ( w,u ) satisfies the conditions of Theorem 3.2.

Lemma 5.1. Suppose ψ ( w,u ) is defined as in formulation (26). Then, ψ ( w,u ) is convex-concave function.

Because the PO problem (25) satisfies the assumptions of CGMT, we transform it to the following AO problem:

where the entries g , h are drawn i.i.d. from N (0 , 1) , due to the property of X . Suppose w Φ OKRidge is the of optimal solutions of the PO problem (25), and w ϕ OKRidge is the optimal solutions of the AO problem (27). According to Theorem 3.2, if ∥ w ϕ OKRidge ∥ 2 P -→ α ∗ , we have ∥ w Φ OKRidge ∥ 2 P -→ α ∗ . Thus, we can analyze the minimizer of AO problem (27) instead of PO problem (25).

5.2 Simplification for AO

In this chapter, we simplify the AO problem (27) into ones involving only scalar quantities. Since γ ( β ) is a convex (see Lemma 5.1), ∂γ ( β ∗ ) is nonempty, convex and compact. According to [13, Theorem 23.4], we have γ ( β ∗ + w ) = γ ( β ∗ ) + max s ∈ ∂γ ( β ∗ ) s ⊤ w + O ( ∥ w ∥ 2 2 ) . The first-order approximation of γ ( β ) around the vector of interest β ∗ is

where β = β ∗ + w . Then, following the approach from [12], the AO problem (27) can be simplified by the first-order approximation (28):

Suppose f ( β ) and ˆ f ( β ) denote the objective functions of the original and the approximated AO problems (27) and (29), respectively,

Then, based on (28), we have

Compared with AO problem (27), the approximated AO problem (29) is tight when ∥ β -β ∗ ∥ 2 → 0 , and we later demonstrate that this condition is satisfied as σ 2 → 0 , independent of the original AO problem (27). This fact allows us to translate the analysis on the optimal solution w ˆ ϕ OKRidge of the approximated AO problem (29) to the analysis on the optimal solution w ϕ OKRidge of the corresponding original AO problem (27). Because γ ( β ∗ ) is a constant, the approximated AO problem (29) is equivalent to the following optimization problem:

where we have approximated γ in the first order. Since ϵ ∼ N (0 , σ 2 I ) , the term ∥ w ∥ 2 g -ϵ above is statistically identical to a random vector with entries drawn i.i.d. from N (0 , ∥ w ∥ 2 2 + σ 2 ) , where I is the unit matrix. Following the method used by [42], we substitute the first term in the objective (31) with √ ∥ w ∥ 2 2 + σ 2 g ⊤ u . Then, we obtain:

Let η = ∥ u ∥ 2 . Since max ∥ u ∥ 2 = η g ⊤ u = ∥ g ∥ 2 · ∥ u ∥ 2 and h ∼ N (0 , I ) , in term of mathematical expectation, the optimization (32) can be equivalently expressed as:

The objective (33) is strongly convex in w and (jointly) concave in η , s , and the constraint sets are bounded. Therefore, we can reverse the order of min-max in problem (33) based on [13, Corollary 37.3.2]. Let α = ∥ w ∥ 2 . Since min ∥ w ∥ 2 = α ( -h + λ η s ) ⊤ w = -α ∥ h -λ η s ∥ 2 , the optimization (33) can be equivalently reformulated as:

Next, we further reverse the order of min-max, as the objective (34) exhibits the desired concaveconvex structure. Then, we proceed to maximize over s ∈ ∂γ ( β ∗ ) . Since min s ∈ ∂γ ( β ∗ ) ∥ h -λ η s ∥ 2 = dist ( h, λ η ∂γ ( β ∗ ) ) , the optimization problem (34) can alternatively be formulated as:

Because both the random components ∥ g ∥ 2 and dist ( h, λ η s ) are Lipschitz, ∥ g ∥ 2 concentrates around √ n and dist ( h, λ η s ) around √ D ( λ η ) [43, Lemma B.2]. Suppose, as d →∞ , D ( τ ) n → ¯ D ( τ ) ∈ (0 , 1) , C ( τ ) n → ¯ C ( τ ) , and Γ( η ) := lim n →∞ η 2 4 √ n . Then, the optimal minimizer of (35) converges to the optimal minimizer of the following deterministic optimization in probability [44]:

Here, we complete the simplifications by reducing the AO problem (27) to an equivalent optimization (36) that now only involves two scalar variables: α and η .

5.3 Error Analysis

Based on the analysis above, if the optimal solution of optimization (36) is α = α ∗ , we have ∥ w ˆ ϕ OKRidge ∥ 2 P -→ α ∗ for approximated AO problem (29). If α ∗ further tends to 0 , according to formulation (30) and CGMT, ∥ w Φ OKRidge ∥ 2 P -→ α ∗ holds for PO problem (25). Then, for the estimation error of OKRidge produced by (22), we have ∥ ˆ β -β ∗ ∥ 2 P -→ α ∗ . Therefore, it only remains to obtain the optimal value of α in optimization (36) that plays the role of ∥ w ∥ 2 . Following [45], we conclude the estimation error of OKRidge with Theorem 5.2 below.

Theorem 5.2. Suppose β ∗ is the true weight parameter of the problem (1), ˆ β is the optimal solution to the objective function (22) of OKRidge, D ( τ ) n → ¯ D ( τ ) ∈ (0 , 1) , aNSE := lim σ 2 → 0 NSE = lim σ 2 → 0 ∥ ˆ β -β ∗ ∥ 2 2 / σ 2 . Define λ map is the solution of map ( τ ) = 0 for τ > 0 , then, the estimation error of OKRidge is given by the following probability limit:

The proof of Theorem 5.2 can be seen in Appendix E.

Remark 5.3 . In the objective (24) concerning estimation error of OKRidge, γ ( β ) = SumTop k ( β ⊙ β ) and the value of β ∗ is assumed to be known. Then, the analysis on map ( · ) in Section 3.3 reveals that the form of map ( · ) and the value of λ map are fixed. Thus, ∆( ˆ λ ) is a function of λ . In other words, if the regularizer parameter λ of OKRidge is fixed, the NSE limit of OKRidge ∆( ˆ λ ) is also fixed. Additionally, Theorem 5.2 also indicates that lim d →∞ lim σ → 0 ∥ ˆ β -β ∗ ∥ 2 P -→ 0 , which guarantees the effectiveness of ˆ β learned by OKRidge in accurately estimating β ∗ . These results substantiate the theoretical reliability of OKRidge and promote its broad application in the real world.

6 Numerical Experiments

In this section, we conduct experiments to verify Theorem 5.2. The experiments contain two aspects: (i) When λ is fixed, NSE tends to a fixed constant as σ → 0 . (ii) When σ → 0 , NSE is determined by the weight λ of the regularizer. In other words, NSE is a function of λ .

In our experiments, β ∗ is randomly generated with ∥ β ∗ ∥ 0 ≤ k . For i ∈ { 1 , 2 , · · · , n } , x i is drawn i.i.d. from N (0 , I ) , and ϵ i is drawn i.i.d. from N (0 , σ 2 ) . According to the k -sparse linear regression (1), y i = x ⊤ i β ∗ + ϵ i , we get dataset ( x i , y i ) with i = 1 , 2 , · · · , n . Then, we appy OKRidge to get the estimator ˆ β and calculate the NSE by ∥ ˆ β -β ∗ ∥ 2 2 / σ 2 . The NSE is averaged over 10 trials to evaluate the effectiveness of the OKRidge algorithm. In the main paper, we set n d = 0 . 5 , k d = 0 . 1 , d = 100 . The computer resources are detailed in Appendix F.1. More experiments with various settings about n d and k d can be seen in Appendix F.2.

2

4

6

8

10

1/

0.5

1.0

1.5

2.0

2.5

3.0

3.5

NSE

=1

=5

=10

=

best

=20

=25

=30

=35

=40

=45

=50

6.1 The Change of NSE with σ

We investigate the change of NSE with 1 /σ under λ = 1 , 5 , 10 , λ best , · · · . The results are illustrated in Figure 1. As depicted in Figure 1, when λ is constant, NSE converges to a fixed value as σ → 0 . This observation validates aspect (i) of Theorem 5.2.

6.2 The Change of NSE with λ

We analyze the change of NSE with λ under σ 2 = 1 , 0 . 1 , 0 . 01 , 0 . 001 . The outcomes are depicted in Figure 2. As shown in Figure 2, the curves converge towards the real blue curve as σ → 0 , where the blue curve relies on λ . This observation confirms aspect (ii) of Theorem 5.2.

7 Conclusion

In this paper, we present a theoretical high-dimensional error analysis of the OKRidge algorithm in idealized settings using the CGMT framework. Specifically, when OKRidge tackles a k -sparse linear model with x ∼ N (0 , I ) , ϵ ∼ N (0 , σ 2 ) , and lim d →∞ n d = δ ∈ (0 , 1) , we have

where ∆( ˆ λ ) depends on λ . This indicates that ( i ) the NSE limit of OKRidge remains constant when λ is fixed; ( ii ) ˆ β learned by OKRidge is effective in estimating β ∗ . Our experimental findings support these theoretical assertions. This theoretical error analysis substantiates the reliability of OKRidge and provides guidelines on the error analysis of other algorithms.

Acknowledgments and Disclosure of Funding

This work is supported by the Key R&D Program of Hubei Province under Grant 2024BAB038, National Key R&D Program of China under Grant 2023YFC3604702, and the Fundamental Research Fund Program of LIESMARS.

0

20

40

60

80

100

0

2

4

6

8

10

NSE

best

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

References

Jiachang Liu, Sam Rosen, Chudi Zhong, and Cynthia Rudin. Okridge: Scalable optimal k-sparse ridge regression for learning dynamical systems. In NeurIPS , 2023.

Emmanuel Candes and Terence Tao. The dantzig selector: Statistical estimation when p is much larger than n. The Annals of Statistics , page 2313-2351, 2007.

Alexandre Belloni, Victor Chernozhukov, and Lie Wang. Square-root lasso: pivotal recovery of sparse signals via conic programming. Biometrika , 98(4):791-806, 2011.

Xiyuan Li, Xin Zou, and Weiwei Liu. Residual network with self-adaptive time step size. Pattern Recognition , 158:111008, 2025.

Xiyuan Li, Zou Xin, and Weiwei Liu. Defending against adversarial attacks via neural dynamic system. In NeurIPS , 2022.

Weijun Xie and Xinwei Deng. Scalable algorithms for the sparse ridge regression. SIAM Journal on Optimization , 30(4):3359-3386, 2020.

Mert Pilanci, Martin J. Wainwright, and Laurent El Ghaoui. Sparse learning via boolean relaxations. Mathematical Programming , 151(1):63-87, 2015.

B. K. Natarajan. Sparse approximate solutions to linear systems. SIAM Journal on Scientific Computing , 24(2):227-234, 1995.

Kathleen P. Champion, Peng Zheng, Aleksandr Y. Aravkin, Steven L. Brunton, and J. Nathan Kutz. A unified sparse optimization framework to learn parsimonious physics-informed models from data. IEEE Access , 8:169259-169271, 2020.

Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Data-driven discovery of partial differential equations. Science advances , 3(4):e1602614, 2017.

Stephen Boyd and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, 2009.

Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. Precise error analysis of regularized m-estimators in high dimensions. IEEE Transactions on Information Theory , 64(8):5592-5628, 2018.

R. Tyrrell Rockafellar. Convex Analysis , volume 28. Princeton University Press, 1997.

Joel A. Tropp. Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory , 50(10):2231-2242, 2004.

Urban Fasel, J Nathan Kutz, Bingni W Brunton, and Steven L Brunton. Ensemble-sindy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. Proceedings of the Royal Society A , 478(2260):20210904, 2022.

Amir Beck and Yonina C. Eldar. Sparsity constrained nonlinear optimization: Optimality conditions and algorithms. SIAM Journal on Optimization , 23(3):1480-1509, 2013.

T. Tony Cai and Lie Wang. Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Transactions on Information Theory , 57(7):4680-4688, 2011.

Dimitris Bertsimas, Angela King, and Rahul Mazumder. Best subset selection via a modern optimization lens. The Annals of Statistics , 44(2):813-852, 2016.

Arkadi Nemirovski and Alexander Shapiro. Convex approximations of chance constrained programs. SIAM Journal on Optimization , 17(4):969-996, 2006.

Dimitris Bertsimas and Wes Gurnee. Learning sparse nonlinear dynamics via mixed-integer optimization. Nonlinear Dynamics , 111(7):6585-6604, 2023.

Weiwei Liu and Ivor W. Tsang. Making decision trees feasible in ultrahigh feature and label dimensions. Journal Of Machine Learning Research , 18:81:1-81:36, 2017.

Dimitris Bertsimas, Jean Pauphilet, and Bart Van Parys. Sparse regression: Scalable algorithms and empirical performance. Statistical Science , 35(4):555-578, 2020.

Alper Atamtürk, Andrés Gómez, and Shaoning Han. Sparse and smooth signal estimation: Convexification of l0-formulations. Journal of Machine Learning Research , 22:52:1-52:43, 2021.

Oktay Günlük and Jeff Linderoth. Perspective reformulations of mixed integer nonlinear programs with indicator variables. Mathematical programming , 124(1):183-205, 2010.

Chenglin Yu, Xinsong Ma, and Weiwei Liu. Delving into noisy label detection with clean data. In ICML , volume 202, pages 40290-40305.

Lianghe Shi and Weiwei Liu. Adversarial self-training improves robustness and generalization for gradual domain adaptation. In NeurIPS , 2023.

Peter J Bickel, Ya'acov Ritov, and Alexandre B Tsybakov. Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics , 37(4):pp. 1705-1732, 2009.

Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. Regularized linear regression: A precise analysis of the estimation error. In COLT , pages 1683-1709, 2015.

David L. Donoho, Arian Maleki, and Andrea Montanari. The noise-sensitivity phase transition in compressed sensing. IEEE Transactions on Information Theory , 57(10):6920-6941, 2011.

Arian Maleki, Laura Anitori, Zai Yang, and Richard G. Baraniuk. Asymptotic analysis of complex LASSO via complex approximate message passing (CAMP). IEEE Transactions on Information Theory , 59(7):4290-4308, 2013.

Christopher A. Metzler, Arian Maleki, and Richard G. Baraniuk. From denoising to compressed sensing. IEEE Transactions on Information Theory , 62(9):5117-5144, 2016.

Ryuhei Miyashiro and Yuichi Takano. Subset selection by mallows' c p : A mixed integer programming approach. Expert Systems with Applications , 42(1):325-331, 2015.

Antonio Frangioni and Claudio Gentile. Perspective cuts for a class of convex 0-1 mixed integer programs. Mathematical Programming , 106(2):225-236, 2006.

Patrick L Combettes. Perspective functions: Properties, constructions, and examples. Set-Valued and Variational Analysis , 26(2):247-264, 2018.

Alper Atamturk and Andrés Gómez. Safe screening rules for l0-regression from perspective relaxations. In ICML , pages 421-430, 2020.

Fariborz Salehi, Ehsan Abbasi, and Babak Hassibi. The impact of regularization on highdimensional logistic regression. In NeurIPS , volume 32, 2019.

Fariborz Salehi, Ehsan Abbasi, and Babak Hassibi. The performance analysis of generalized margin maximizers on separable data. In ICML , pages 8417-8426, 2020.

Yanbo Chen and Weiwei Liu. A theory of transfer-based black-box attacks: Explanation and implications. In NeurIPS , 2023.

Jingyuan Xu and Weiwei Liu. Characterization of overfitting in robust multiclass classification. In NeurIPS , 2023.

Yehoram Gordon. On milman's inequality and random subspaces which escape through a mesh in R n . In Geometric Aspects of Functional Analysis , 1988.

Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. The gaussian min-max theorem in the presence of convexity. arXiv preprint arXiv:1408.4837 , 2014.

Christos Thrampoulidis, Ashkan Panahi, and Babak Hassibi. Asymptotically exact error analysis for the generalized equation-lasso. In ISIT , pages 2021-2025, 2015.

Samet Oymak, Christos Thrampoulidis, and Babak Hassibi. The squared-error of generalized LASSO: A precise analysis. In Allerton , pages 1002-1009, 2013.

Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Handbook of econometrics , 4:2111-2245, 1994.

Christos Thrampoulidis, Ashkan Panahi, Daniel Guo, and Babak Hassibi. Precise error analysis of the LASSO. In ICASSP , pages 3467-3471, 2015.

The Reliability of OKRidge (Appendix)

A Limitaion and Impact

Limitaion : Our results rely on Gaussian input settings. For non-Gaussian settings, we can utilize Fisher transformation, Box-Cox transformation, or inversion sampling to transform non-Gaussian distribution to Gaussian distribution. In our future work, we will discuss the potential extensions of our findings to non-Gaussian input settings, providing insights into the universality of the results.

Impact : Our work provides theoretical support for the broad application of OKRidge, which does not require proprietary software or expensive licenses [1], unlike its main competitor. This can significantly impact various regression applications.

B Proof of Theorem 4.1

Theorem 4.1 . If we define the parameter c as (16), the problem (9) is equivalent to the following saddle point optimization problem:

where

Proof. The problem (9) is where

The parameter c in (16) is

̸

̸

̸

Substitute parameter c of (16) into (9),

̸

̸

Define

Then, we have

̸

Then, the problem (9) can be equivalently written as

which is our saddle point optimization (38).

C Proof of Theorem 4.2

Theorem 4.2 . The function L ( β ) defined in Equation (20) is lower bounded by

where ⊙ is Hadamard product, and SumTop k ( · ) denotes the summation of the largest k elements of a given vector.

Proof. The Equation (20) is

According to (20) and (39),

̸

Following the method of [1], if we minimize z in the optimization (41) under the constraints ∑ p j =1 z j ≤ k and z j ∈ [0 , 1] for ∀ j , we have z j = 1 for the top k terms of β 2 j and z j = 0 otherwise. Then, we have

̸

D Proof of Lemma 5.1

Lemma 5.1 . Suppose ψ ( w,u ) is defined as formulation (26). Then, ψ ( w,u ) is convex-concave on R d × R n .

Proof. Obviously, ψ ( w,u ) is concave about u . Next, we indicate that ψ ( w,u ) is convex about w . For any ¯ w, ˜ w ∈ R d and ∀ θ ∈ (0 , 1) ,

SumTop k ( · ) is a convex operator. Therefore, ψ ( w,u ) is concave about u and convex about w .

E Proof of Theorem 5.2

Theorem 5.2 . Suppose β ∗ is the true weight parameter of the problem (1), ˆ β is the optimal solution to the objective function (22) of OKRidge, D ( τ ) n → ¯ D ( τ ) ∈ (0 , 1) ,

Define λ map is the solution of map ( τ ) = 0 for τ > 0 , then, the estimation error of OKRidge is given by the following probability limit:

where ∆( ˆ λ ) = ¯ D ( ˆ λ ) 1 -¯ D ( ˆ λ ) , and ˆ λ = λ map .

Proof. Starting from the simplified AO problem (36), let's define

Given that κ ( α, η ) is strongly convex in α and concave in η , we denote ( α ∗ , η ∗ ) as the Nash equilibrium of κ ( α, η ) . Then, α ∗ is unique and we use duality to compute α ∗ . For any η ,

which is well-defined, due to 1 -¯ D ( λ η ) > 0 . Substituting α ∗ ( η ) back into (43) gives:

Differentiating κ ( α ∗ , η ) with respect to η , we have

Here, map ( τ ) = 0 when τ = λ map . Hence, η ∗ = λ λ map is the optimal solution, due to

Because the form of γ ( · ) and the value of β ∗ are determined, the form of map ( · ) and the value of λ map are fixed. In other words, λ map is a function about λ . Therefore, we can take η ∗ = λ λ map to formulation (44),

Moreover κ ( α ∗ , η ∗ ) = λ λ map σ √ 1 -¯ D ( λ map ) . Denote ˆ λ = λ map

Based on the analysis above, if the optimal solution of optimization (36) is α = α ∗ , we have ∥ w ˆ ϕ OKRidge ∥ 2 P -→ α ∗ for approximated AO problem (29). Furthermore, according to formulation (46), lim σ → 0 α ∗ = 0 occurs for the approximated AO problem (29) and is independent of the original AO problem (27). Thus, formulation (30) holds:

In the case n →∞ and σ → 0 , formulation (47) allows us to translate the optimal error α ∗ of the approximated AO problem (29) to the optimal error of the original AO problem (27). Combining formulations (46), (47), and CGMT, ∥ w Φ OKRidge ∥ 2 P -→ α ∗ holds for PO problem (25). Then, for the estimation error of OKRidge produced by (22), we have

Therefore, combining PO problem (25), AO problem (27), the relation (47), and the formulations (23) and (46), according to CGMT, we obtain::

To sum up,

F Experiments Appendix

F.1 Computing Platform

All experiments were run on the 10x TensorEX TS2-673917-DPN Intel Xeon Gold 6226 Processor, 2.7Ghz. We set the memory limit to be 100GB.

F.2 More Experiments

Figures F1 ∼ F7 (a) demonstrate that when λ is fixed, NSE converges to a constant as σ → 0 . In Figures F1 ∼ F7 (b), the curves converge towards the real blue curve as σ → 0 , with the real blue curve representing ∆( ˆ λ ) . These observation validate Theorem 5.2.

2

4

6

8

10

1/

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

NSE

=1

=5

=10

=

best

=15

=20

=25

=30

=35

=40

=45

=50

0

best

20

40

60

80

100

1

2

3

4

5

6

7

8

NSE

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

2

4

6

8

10

1/

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

NSE

=1

=5

=10

=15

=20

=25

=

best

=35

=40

=45

=50

0

20

best

40

60

80

100

0.5

1.0

1.5

2.0

2.5

3.0

3.5

NSE

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

0

5

10

15

20

25

30

1/

1

2

3

4

5

6

NSE

=1

=5

=10

=

best

=20

=25

=30

=35

=40

=45

=50

0

20

40

60

80

100

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

NSE

best

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

2

4

6

8

10

1/

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

NSE

=1

=5

=10

=15

=

best

=25

=30

=35

=40

=45

=50

0

40

60

80

100

1

2

3

4

NSE

best

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

2

4

6

8

10

1/

0.5

1.0

1.5

2.0

2.5

3.0

3.5

NSE

=1

=5

=10

=

best

=20

=25

=30

=35

=40

=45

=50

0

20

40

60

80

100

0

2

4

6

8

10

NSE

best

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

0

5

10

15

20

25

30

1/

1

2

3

4

5

NSE

=1

=5

=

best

=20

=25

=30

=35

=40

=45

=50

0

40

60

80

100

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

NSE

best

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

0

5

10

15

20

25

30

1/

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

NSE

=1

=5

=

best

=20

=25

=30

=35

=40

=45

=50

0

40

60

80

100

0

10

20

30

40

50

60

70

80

NSE

best

2 =1.0

2 =0.1

2 =0.01

2 =0.001

real

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We highly summarize what we do in the abstract. The introduction clearly introduces the concepts and issues that are related to our main results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our paper explicitly acknowledges the dependency on Gaussian inputs as a limitation, prompting further investigation into its applicability to non-Gaussian contexts. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the assumptions of the theorems and all our theorems are followed by their proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper meticulously details all methodologies, parameters, and experimental conditions necessary for replication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We focus on the OKRidge algorithm in our paper. The codes of the OKRidge algorithm are from [1], which is open.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our paper comprehensively details all training and testing protocols, including data management, hyperparameter selection, and optimization techniques.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our results are based on averaging the results from 10 trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our paper specifies the computing platform, processor details, and memory constraints, ensuring accurate replication of the experimental setup and results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research strictly adheres to the NeurIPS Code of Ethics, focusing on theoretical advancements without ethical concerns in implementation or application.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our paper assesses the broad accessibility and potential application impacts of OKRidge, highlighting its advantages over costlier alternatives.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper focuses on theoretical aspects of the OKRidge algorithm and uses synthetic data, mitigating risks associated with real-world data misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Our paper properly credits the original creators of the OKRidge algorithm and adheres to the licensing terms of the cited open resources required by NeurIPS paper [1].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We don't introduce new assets in our paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] ,

Justification: Our paper is theoretical and includes numerical verification experiments. It does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper is theoretical and includes numerical verification experiments. It does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the estimation error of the Scalable Optimal K-Sparse Ridge Regression (OKRidge) method proposed by [1]. Specifically, they reframe the estimation error of OKRidge as a Primary Optimization (PO) problem and use the Convex Gaussian Min-Max Theorem (CGMT) to simplify the PO problem into an Auxiliary Optimization (AO) problem. Subsequently, this paper provides a theoretical error analysis for OKRidge based on the AO problem.\n\nSTRENGTHS:\n__Originality:__ The absence of theoretical analysis on the error of OKRidge impedes its large-scale applications. This paper analyzes the estimation error of OKRidge to fill this gap.\n\n__Clarity:__ This paper demonstrates a high level of clarity in its presentation, which significantly enhances its readability. All symbols are clearly defined at the outset and the proofs provided are rigorous and detailed. \n\n__Quality and Significance:__ This paper applies CGMT technology to transform the estimation error of OKRidge into a simplified AO problem and offer a theoretical error analysis for OKRidge. This theoretical error analysis substantiates the reliability of OKRidge and provides guidelines for the error analysis of other algorithms.\n\nWEAKNESSES:\n1. Many formulas lack punctuation marks after them, such as formulations (10), (11), (22), etc. The author should review the entire text and pay attention to these details. \n\n2. Theorem 5.2 demonstrates that $\\hat{\\lambda}=\\lambda_{map}$. It would be more concise to substitute $\\Delta(\\hat{\\lambda})$ with $\\Delta(\\lambda_{map})$.\n\n3. The colors of the curves for $\\lambda=1$ and $\\lambda=50$ in Figure 1 and F1-F4 (a) are the same. This paper should ensure that the curves are distinguishable by using different colors.\n\nQUESTIONS:\nThe theoretical results are based on CGMT technology, which requires the input to be Gaussian. Could you generalize the input to non-Gaussian distributions?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAn error analysis of a lower bound technique for solving sparse ridge regression problems is presented. Sparse ridge regression is ridge regression with the constraint that the parameter vector has at most k non-zero entries, where k is a parameter. It has been proposed to solve the sparse ridge regression problem by solving a \"tight lower\" bound problem. The normalized deviation of the solution of the lower bound problem to the \"true\" model parameter vector is analyzed in the high-dimensional setting, that is, the setting where the number of features (and data points) go to infinity.\n\nSTRENGTHS:\nThe problem that is addressed by the paper is interesting (for an expert community) and technically challenging.\n\nWEAKNESSES:\nI am not able to thoroughly review the paper in the time available for a NeurIPS review. However, since the contributions of the paper are mostly theoretical, a careful review by someone closer to the topic would be necessary to make sure that the paper is sound. Probably a journal is a better venue for this type paper.  Nonetheless, I would have tried to check technical details of the paper if it had been more accessible. The presentation is fairly poor, which makes it difficult (and time-consuming) for a non-expert like me to thoroughly review it. \n\nSome areas where the presentation can be improved:\n- Abstracts should not include references.\n- The introduction immediately becomes technical, which makes it difficult to understand that problem that you are addressing. Make clear that you provide a theoretical high-dimensional analysis of an algorithm in an idealized setting. You are analyzing the case, where the entries of data matrix are chosen i.i.d from standard normal distributions. Another assumption is that the number n(d) of data points grows with d such that n(d)/d = \\delta \\in (0,1), where \\delta is a constant. All this information is there in the paper, but it could be put better into perspective if your goals had been clearly stated at this point. \n- Make clear that the estimation error in Equation 5 is a random variable, because X and \\epsilon are random.\n- Equation 5 is missing a \\odot\n- The main paper provides many technical definitions/details but no proofs. The details are relevant when checking the proofs, but do not help with the intuition.\n\nQUESTIONS:\n1. When you solve Problem 4 with the objective function from Equation 3, do you set the n-k smallest entries in \\hat\\beta to zero?\n2. In Line 56, why is it \\Delta (\\hat\\lambda) and not \\Delta (\\lambda)?\n\nLIMITATIONS:\nNone. This is a theory paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

comments: SUMMARY:
The authors provide a theoretical error analysis for the OKRidge method, which is both faster and more accurate than existing approaches for solving sparse ridge regression. The experimental results are in excellent agreement with the theoretical findings.

STRENGTHS:
1. The authors analyze the estimation error of OKRidge using Convex Gaussian min-max theorem, and the proof appears to be correct.

2. The theoretical and experimental results validate the reliability of the OKRidge method.

3. This theoretical error analysis provides support for the broad application of OKRidge.

WEAKNESSES:
1. Because the OKRidge method is used to solve a k-sparse linear regression problem and the authors make the Gaussian assumption, the following related literatures are recommended to be cited:

> * 1.1 Mohsen Bayati and Andrea Montanari. The lasso risk for Gaussian matrices. Information Theory, IEEE Transactions on, 58(4):1997–2017, 2012

> * 1.2 D. Bertsimas, J. Pauphilet, and B. Van Parys. Sparse regression: Scalable algorithms and empirical performance. Statistical Science, 35(4):555–578, 2020

> *1.3 Peter J Bickel, Yaacov Ritov, and Alexandre B Tsybakov. Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics, 37(4):1705–1732, 2009.

2. The authors provide numerical experiments with $n/d=0.4, 0.5 and 0.6$, but the sparsity is just $k/d=0.1$. I want to see more experiments with various sparsity.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
The authors adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOKRidge proposed in [1] shows promising results and becomes the SOTA sparse ridge regression solvers. This paper conducts the first error analysis for OKRidge. Convex Gaussian min-max theorem (CGMT) is well introduced in this paper. Based on CGMT, they develop the asymptotic theory for OKRidge. The experiments are also conducted to support the theory. This is a solid and up-to-date work.\n\nSTRENGTHS:\nThe paper first uses CGMT to conduct the rigorous analysis for OKRidge, which is an up-to-date research. It is novel to reformulate the estimation error of OKRidge as a PO problem and further reduce the PO problem to an AO problem. By analyzing the AO problem, they show that the NSE of OKRidge tends to a constant. Then the estimators of OKRidge is close to the groundtruth. The empirical results also support the theories presented in this paper. The theoretical results in this paper are original which has never been explored before. This paper bridges the gap between theory and practice.\n\nWEAKNESSES:\nThe paper presents both GMT and CGMT. But the difference between these two concepts and the advantages of CGMT over GMT are not well clarified. \n\nIn Line 208, the LHS of (23) misses the \\min_{\\beta}.\n\nIt is interesting to reduce the AO problem (26) to an equivalent optimization (35) that only involves two scalar variables. But more descriptions about why can we do this transformation are needed.\n\nIn Line 300, it is improper for the formulation to use “=”, as it should indicate an approximation in probability according to Theorem 5.2.\n\nQUESTIONS:\n1, What is the difference between GMT and CGMT and what are the advantages of CGMT over GMT?\n2, Why can we reduce the problem (26) to (35) that only involves two scalar variables?\n\nLIMITATIONS:\nThe author has presented the limitations in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper improves the theoretical reliability of OKRidge method for the sparse ridge regression by introducing a theoretical error analysis. OKRidge is reframed in this paper as a Primary Optimization problem. Then this paper use the Convex Gaussian Min-max Theorem (CGMT) to simplify it to an Auxiliary Optimization problem. This paper provide a theoretical error analysis for OKRidge based on the  problem.\n\nSTRENGTHS:\n- The paper provides a novel theoretical error analysis for the OKRidge method, leveraging the CGMT framework, which is a contribution to the field of sparse regression.\n- The paper is well-structured, with a clear outline of the problem, methodology, and results.\n\nWEAKNESSES:\n- While the theoretical contributions are clear, the practical motivation for why this particular error analysis is crucial could be better articulated. More emphasis on how this analysis directly impacts real-world applications would strengthen the paper.\n- The experiments, although validating the theoretical claims, are limited in scope. They primarily focus on synthetic data with Gaussian noise. Including more diverse datasets, especially real-world examples, would enhance the validity of the results.\n- The reliance on Gaussian input settings is a significant limitation. The paper acknowledges this but does not provide a clear path for extending the results to non-Gaussian settings. This could limit the applicability of the findings.\n\nQUESTIONS:\n- Can the authors provide more insight into the practical applications where this theoretical error analysis will be most impactful?\n- Are there any specific strategies proposed for extending the theoretical findings to non-Gaussian input settings?\n- How robust are the experimental findings to variations in the underlying assumptions, such as the sparsity level and noise distribution?\n\nLIMITATIONS:\nThe authors have addressed the limitations related to Gaussian input settings and indicated future work on extending the results to non-Gaussian settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}