Title: Feature Adaptation for Sparse Linear Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: theory

sparse linear regression

feature adaptation

lasso
Abstract: Sparse linear regression is a central problem in high-dimensional statistics. We study the correlated random design setting, where the covariates are drawn from a multivariate Gaussian $N(0,\Sigma)$, and we seek an estimator with small excess risk. 

If the true signal is $t$-sparse, information-theoretically, it is possible to achieve strong recovery guarantees with only $O(t\log n)$ samples. However, computationally efficient algorithms have sample complexity linear in (some variant of) the *condition number* of $\Sigma$. Classical algorithms such as the Lasso can require significantly more samples than necessary even if there is only a single sparse approximate dependency among the covariates.

We provide a polynomial-time algorithm that, given $\Sigma$, automatically adapts the Lasso to tolerate a small number of approximate dependencies. In particular, we achieve near-optimal sample complexity for constant sparsity and if $\Sigma$ has few ``outlier'' eigenvalues.
Our algorithm fits into a broader framework of *feature adaptation* for sparse linear regression with ill-conditioned covariates. With this framework, we additionally provide the first polynomial-factor improvement over brute-force search for constant sparsity $t$ and arbitrary covariance $\Sigma$.

Feature Adaptation for Sparse Linear Regression

Abstract

Sparse linear regression is a central problem in high-dimensional statistics. We study the correlated random design setting, where the covariates are drawn from a multivariate Gaussian N (0 , Σ) , and we seek an estimator with small excess risk. If the true signal is t -sparse, information-theoretically, it is possible to achieve strong recovery guarantees with only O ( t log n ) samples. However, computationally efficient algorithms have sample complexity linear in (some variant of) the condition number of Σ . Classical algorithms such as the Lasso can require significantly more samples than necessary even if there is only a single sparse approximate dependency among the covariates.

We provide a polynomial-time algorithm that, given Σ , automatically adapts the Lasso to tolerate a small number of approximate dependencies. In particular, we achieve near-optimal sample complexity for constant sparsity and if Σ has few 'outlier' eigenvalues. Our algorithm fits into a broader framework of feature adaptation for sparse linear regression with ill-conditioned covariates. With this framework, we additionally provide the first polynomial-factor improvement over brute-force search for constant sparsity t and arbitrary covariance Σ .

1 Introduction

Sparse linear regression is a fundamental problem in high-dimensional statistics. In a natural random design formulation of this problem, we are given m independent and identically distributed samples ( X i , y i ) i m =1 where each sample's covariates are drawn from an n -dimensional Gaussian random vector X i ∼ N (0 , Σ) , and each response is y i = ⟨ X i , v ∗ ⟩ + ξ i for independent noise ξ i ∼ N (0 , σ 2 ) and a t -sparse ground truth regressor v ∗ ∈ R n , where t is much smaller than n . The goal 5 is to output a vector ˆ v ∈ R n for which the excess risk

is as small as possible, where ( X 0 , y 0 ) is an independent sample from the same model.

Without the sparsity assumption, the number of samples needed to achieve small excess risk (say, O ( σ 2 ) ) is linear in the dimension; with O ( n ) samples, simple and computationally efficient algorithms such as ordinary least squares achieve the statistically optimal excess risk O ( σ 2 n m ) . Sparsity

Dhruv Rohatgi § MIT

allows for a significant statistical improvement: ignoring computational efficiency, it is well known that there is an estimator ˆ v with excess risk O ( σ 2 t log n m ) as long as m = Ω( t log n ) (see e.g. [13, 33]; Theorem 4.1 in [23]).

The catch is that computing this estimator involves a brute-force search over ( n t ) possibilities (i.e., the possible supports for v ∗ ). At first glance, this combinatorial search may seem unavoidable if we wish to take advantage of sparsity. Indeed, similar problems are notoriously difficult: the only non-trivial algorithms for e.g., learning t -sparse parities with noise still require n Ω( t ) time [29, 37]. However, it is a celebrated fact that for sparse linear regression, computationally efficient methods such as Lasso and Orthogonal Matching Pursuit can avoid this combinatorial search and still achieve very strong theoretical guarantees under conditions such as the Restricted Isometry Property (see e.g. [7, 10, 5, 4, 3, 1]). In the random design setting we consider, the Lasso is known to achieve optimal statistical rates (up to constants) when the covariance matrix Σ is well-conditioned [32, 46].

What about when Σ is ill-conditioned? In contrast with the statistically optimal estimator, Lasso and its cousins provably require sample complexity scaling with (some variant of) the condition number of Σ (see e.g. Theorem 14 in [38] or Theorem 6.5 in [23]). And with a few exceptions (e.g., in some settings with special graphical structure [23]) there has been little progress on designing new efficient algorithms for sparse linear regression with ill-conditioned Σ (see Section 4 for further discussion). For a general covariance Σ , no algorithm is even known that can achieve sample complexity f ( t ) · n 1 -ϵ (for an arbitrary function f ) without brute-force search.

A computationally efficient algorithm that approaches the optimal statistical rate for arbitrary Σ might be too much to hope for. While no computational lower bounds are known, even in restricted computational models such as the Statistical Query model, 6 the related worst-case problem of finding a t -sparse solution to a system of linear equations requires n Ω( t ) time under standard complexity assumptions [15]. So it is plausible, though not certain, that some assumptions on Σ are necessary. In this work - inspired by a long tradition (in random matrix theory, statistics, graph theory, and other areas) of studying matrices with a spectrum that is split between a large 'bulk' and a small number of outlier 'spike' eigenvalues [28, 39, 43] - we identify a broad generalization of the standard well-conditionedness assumption, under which brute-force search can still be avoided.

1.1 Beyond well-conditioned Σ

Say that Σ has eigenvalues λ 1 ≤ · · · ≤ λ n , and that the sparsity t is a constant. 7 Then standard bounds for Lasso require sample complexity ( λ n /λ 1 ) · O (log n ) . But if the covariates contain even a single approximate linear dependency, then λ n /λ 1 may be arbitrarily large. Moreover, if the dependency is sparse (e.g. two covariates are highly correlated), then there is a natural choice of v ∗ for which Lasso provably fails (see Theorem 6.5 of [23]). Indeed, this phenomenon is not just a limitation of the analysis; Lasso fails empirically as well, even for very small t (see Figure 2 in Appendix H for a simple example with t = 3 ).

Such dependencies arise in applications ranging from finance (e.g., where some pairs of stocks or ETFs may be highly correlated, and an investor may be interested in the differences) to genomic data (where functionally related genes may have highly correlated expression patterns). Two-sparse dependencies can be directly identified by looking at the covariance matrix; see Section 4 for some discussion of previous research in this direction. But as t increases, naive methods for identifying t -sparse dependencies quickly become computationally intractable. With domain knowledge, it may be possible to manually identify and correct such dependencies, but this process would also be time-consuming. Thus, we ask the following question: instead of assuming that λ n /λ 1 is bounded, suppose that there are constants d ℓ and d h so that λ n -d h /λ d ℓ +1 is bounded, i.e. the spectrum of Σ has only d ℓ outliers at the low end, and only d h outliers at the high end. Can we still design an algorithm that achieves sample complexity O (log n ) without resorting to brute-force search?

Main result. We give a positive answer: an algorithm for sparse linear regression that is both computationally and statistically efficient for covariance matrices with a small number of 'outlier'

eigenvalues. In particular, this means we can handle a few approximate dependencies among the covariates (quantified by the number of eigenvalues below a threshold). In comparison, Lasso and other classical algorithms cannot tolerate even a single sparse approximate dependency. Our main algorithmic result is the following:

Theorem 1.1. Let n, t, d ℓ , d h , L ∈ N and σ, δ > 0 . Let Σ ∈ R n × n be a positive semi-definite matrix with (non-negative) eigenvalues λ 1 ≤ · · · ≤ λ n . Let v ∗ ∈ R n be any t -sparse vector. Let ( X i , y i ) i m =1 be independent with X i ∼ N (0 , Σ) and y i = ⟨ X i , v ∗ ⟩ + ξ i , where ξ i ∼ N (0 , σ 2 ) .

Let n eff := t ( λ n -d h /λ d ℓ +1 ) log( nL/δ ) + t O ( t ) d l + d h . Given Σ , t , d ℓ , δ , and ( X i , y i ) i m =1 , there is an estimator ˆ v ∈ R n that has excess risk

with probability at least 1 -δ , so long as m ≥ Ω( n eff L ) . Moreover, ˆ v can be computed in time poly( n ) .

Specifically, taking L ∼ log( m ∥ v ∗ ∥ 2 Σ /σ 2 ) , the time complexity is dominated by L eigendecompositions and L calls to a Lasso program, for overall runtime ˜ O ( n 3 ) (see Algorithm 2). This is substantially faster than the brute-force method (which takes O ( n t ) time) even for small values of t .

The excess risk decays at rate ˜ O ( σ 2 n eff /m ) (hiding the logarithmic factor), which is near the statistically optimal rate of ˜ O ( σ 2 t/m ) so long as n eff is small, i.e. t is small and only a few eigenvalues lie outside a constant-factor range. In our analysis, we prove that the standard Lasso estimator can already tolerate a few large eigenvalues - the main algorithmic innovation is needed to tolerate a few small eigenvalues, which turns out to be much trickier. Notice that when d ℓ = d h = 0 we recover standard Lasso guarantees up to the factor of L ; thus, Theorem 1.1 morally represents a generalization of classical results.

We also show how to achieve a different trade-off between time and samples, eliminating the dependence on d ℓ in sample complexity at the cost of larger runtime:

Theorem 1.2. In the setting of Theorem 1.1, let n ′ eff := t ( λ n -d h /λ d ℓ +1 ) log( nL/δ )+ t 2 log( t )+ d h . Given Σ , t , d ℓ , δ , and ( X i , y i ) i m =1 , there is an estimator ˆ v ∈ R n that has excess risk

with probability at least 1 -δ , so long as m ≥ Ω( n ′ eff L ) . Moreover, ˆ v can be computed in time poly( n, m, d t ℓ , t t 2 ) .

Discussion & limitations. We discuss two limitations of the above results. First, both results incur exponential dependence on the sparsity t (in the sample complexity for Theorem 1.1, and the runtime for Theorem 1.2), which may be suboptimal. For Theorem 1.1, we remark that in practice the algorithm may not suffer this dependence (see e.g. Figure 1), and it is possible that the analysis can be tightened. For Theorem 1.2, we emphasize that the runtime is still fundamentally different than brute-force search: in particular, it's fixed-parameter tractable in t and d ℓ .

Second, both results require that Σ is known. Thus, they are only applicable in settings where we either have a priori knowledge, or can estimate Σ accurately because a large amount of unlabelled data is available. At a high level, this limitation is due to the need to compute the eigendecomposition of Σ , which cannot be approximated from the empirical covariance of a small number of samples.

For simplicity, we have stated our results in terms of Gaussian covariates and noise, but this is not a fundamental limitation. We expect it is possible to prove similar results in the sub-Gaussian case at the cost of making the proof longer - for instance, by building upon the techniques from [25] and related works.

Pseudocode & simulation. See Algorithm 1 for complete pseudocode of AdaptedBP() , a simplification of the method for the noiseless setting σ = 0 . In Figure 1 we show that AdaptedBP() significantly outperforms standard Basis Pursuit (i.e. Lasso for noiseless data [7]) on a simple example with n = 1000 variables, d ℓ = 10 sparse approximate dependencies, and a ground truth

2.00

BP

Adapted BP

1.75

1.50

Excess risk

1.25

1.00

0.75

0.50

0.25

0.00

0

100

200

300

400

500

Number of samples

regressor with sparsity t = 13 . The covariates X 1:1000 are all independent N (0 , 1) except for 10 disjoint triplets { ( X i , X i +1 , X i +2 ) : i = 1 , 4 , . . . , 28 } , each of which has joint distribution

where Z i , Z i +1 , Z i +2 ∼ N (0 , 1) are independent. The (noiseless) responses are y = 6 . 25( X 1 -X 2 ) + 2 . 5 X 3 + 1 √ 10 ∑ 1000 i =991 X i . See Appendix I for implementation details.

1.2 Organization

In Section 2 we give an overview of the proofs of Theorem 1.1 and 1.2 (the complete proofs and full algorithm pseudocode are given in Appendix C). In Section 3 we discuss our other results obtained via feature adaptation. Section 4 covers related work.

2 Proof techniques

We obtain Theorems 1.1 and 1.2 as outcomes of a flexible algorithmic approach for tackling sparse linear regression with ill-conditioned covariates: feature adaptation . As a pre-processing step, adapt

or augment the covariates with additional features (i.e. well-chosen linear combinations of the covariates). Then, to predict the responses, apply ℓ 1 -regularized regression (Lasso) over the new set of features rather than the original covariates. In other words, we algorithmically change the dictionary (set of features) used in the Lasso regression. See Section 4 for a comparison to past approaches.

We start by explaining the goals of feature adaptation for general Σ , and then show how we achieve those desiderata when Σ has few outlier eigenvalues. More precisely, the main technical difficulty is in dealing with the small eigenvalues, so in this proof overview we focus on the case where the only outliers are small eigenvalues. Complete proofs of Theorems 1.1 and 1.2 are in Appendix C.

2.1 What makes a good dictionary: the view from weak learning

Obviously, the feature adaptation approach generalizes Lasso. Surprisingly, even though the sample complexity of the standard Lasso estimator is thoroughly understood, the basic question of whether for every covariate distribution (i.e. every Σ ) there exists a good dictionary remains wide-open. To crystallize the power of feature adaptation, we introduce the following notion of a 'good' dictionary. Wesuggest considering the simplified setting of α -weak learning , where the goal is just to find some ˆ v so that the predictions ⟨ X, ˆ v ⟩ are α -correlated with the ground truth ⟨ X,v ∗ ⟩ when X ∼ N (0 , Σ) . Moreover, we focus first on the existential question (rather than the algorithmic question of finding the dictionary). We will return to the setting of Theorems 1.1 and 1.2 later. For now, in the weak learning setting, a good dictionary (when the covariate distribution is N (0 , Σ) ) is one that satisfies the following covering property, but is not too large:

Definition 2.1. Let Σ ∈ R n × n be a positive semi-definite matrix and let t, α > 0 . A set { D 1 , . . . , D N } ⊆ R n is a ( t, α ) -dictionary for Σ if for every t -sparse v ∈ R n , there is some i ∈ [ N ] with

where we define ⟨ x, y ⟩ Σ := x ⊤ Σ y and ∥ x ∥ 2 Σ := x ⊤ Σ x for any x, y ∈ R n . Let N t,α (Σ) be the size of the smallest ( t, α ) -dictionary.

The relevance of the covering number N t,α (Σ) is quite simple: given a ( t, α ) -dictionary D for Σ , and given samples ( X i , y i ) i m =1 , the weak learning algorithm can simply output the vector ˆ v ∈ D that maximizes the empirical correlation between the predictions ⟨ X i , ˆ v ⟩ and the responses y i . So long as there are enough samples for empirical correlations to concentrate, Definition 2.1 guarantees success. Formally, allowing for preprocessing time to compute the dictionary, O ( α ) -weak learning is possible in time N t,α (Σ) · poly( n ) , with O ( α -2 log N t,α (Σ)) samples (Proposition A.5).

Hypothetically, bounding N t,α (Σ) may not be necessary to develop an efficient sparse linear regression algorithm. However, all assumptions on Σ that are currently known to enable efficient sparse linear regression also immediately imply bounds on N t,α (see Appendix G). For example, when Σ is well-conditioned, the standard basis is a good dictionary of size n (Fact A.4).

In contrast, the only known bounds for arbitrary Σ (until the present work) are N t, 1 / √ t (Σ) ≤ t · ( n t ) (the brute-force dictionary, which includes a Σ -orthonormal basis for every set of t covariates) and N t, 1 / √ n (Σ) ≤ n (a Σ -orthonormal basis for all n covariates, which doesn't take advantage of sparsity and corresponds to algorithms such as Ordinary Least Squares). Thus, the following basic question - when can we improve upon these trivial bounds - seems central to understanding when brute-force search can be avoided in sparse linear regression:

Question 2.2. How large is N t,α (Σ) for an arbitrary positive semi-definite Σ ∈ R n × n ? Are there natural families of ill-conditioned Σ (and functions f, g ) for which N t, 1 /f ( t ) (Σ) ≤ g ( t ) · poly( n ) ?

2.2 Constructing a good dictionary when Σ has few small eigenvalues

We now address Question 2.2 in the setting where Σ has a small number of eigenvalues that are much smaller than λ n . In this setting, the standard basis may not be a good dictionary. For example, if two covariates are highly correlated, their difference may not be correlated with any of them. Nonetheless, we can prove the following covering number bound:

Theorem 2.3. Let n, t, d ∈ N . Let Σ ∈ R n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ · · · ≤ λ n . Then N t,α (Σ) ≤ t (7 t ) 2 t 2 + t d t + n , where α = 1 7 √ t √ λ d +1 /λ n .

In particular, when t = O (1) and Σ is well-conditioned except for O (1) outliers λ 1 , . . . , λ d , we get a linear-size dictionary just as in the case where Σ is well-conditioned. In fact, the desired ( t, α ) -dictionary can be constructed efficiently. Our key lemma shows that when Σ has few small eigenvalues, there is a small subset of covariates that 'causes' all of the sparse approximate dependencies - in the sense that the ℓ 2 norm of any sparse vector, excluding the mass on the subset, can be upper bounded in terms of the Σ -norm of the vector. Moreover, there is an efficient algorithm that finds a superset of these covariates. Formally, we prove the following:

Lemma 2.4. Let n, t, d ∈ N . Let Σ ∈ R n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ · · · ≤ λ n . Given Σ , d , and t , there is a polynomial-time algorithm IterativePeeling() producing a set S ⊆ [ n ] with the following guarantees:

(a) For every t -sparse v ∈ R n , it holds that ∥ ∥ v [ n ] \ S ∥ ∥ 2 ≤ 3 λ -1 / 2 d +1 ∥ v ∥ Σ .

Once this set S has been found, the dictionary is simply the standard basis { e 1 , . . . , e n } , together with a Σ -orthonormal basis for every set of t covariates in S . By guarantee (a), we can prove that every t -sparse vector correlates with some element of this dictionary under the Σ -inner product. By guarantee (b), the dictionary is much smaller than the brute-force dictionary that contains a basis for all ( n t ) sets of t covariates. Together, this gives an algorithmic proof for Theorem 2.3.

Intuition for IterativePeeling() . We compute the set S via a new iterative method which leverages knowledge of the small eigenspaces of Σ . See Algorithm 1 for the pseudocode. To compute S , the algorithm IterativePeeling() first computes the orthogonal projection matrix P that projects onto the subspace spanned by the top n -d eigenvectors of Σ . Starting with the set of coordinates that correlate with ker( P ) , the procedure then iteratively grows S in such a way that at each step, a new participant of each approximate sparse dependency is discovered, but S does not become too much larger.

The intuition is as follows: as a preliminary attempt, we could identify all O ( d ) coordinates that correlate (with respect to the standard inner product) with the lowest d eigenspaces of Σ . If e.g. the covariates have a sparse dependency

then ker Σ contains the vector e 1 + e 2 , so the coordinates { e 1 , e 2 } will be correctly discovered. Unfortunately, if Σ contains a more complex sparse dependency such as

where ϵ > 0 is very small, then this heuristic will discover { e 1 , e 2 } but miss { e 3 , e 4 } . For this example, the solution is to notice that e 3 and e 4 do correlate with the subspace spanned by ker(Σ) ∪ { e 1 , e 2 } (which contains e 3 + e 4 ). In general, if S is the set of coordinates discovered thus far, then by finding basis vectors that correlate with an appropriate subspace (of dimension at most | S | ), we can efficiently augment S with at least one new coordinate from each t -sparse approximate dependency, without making S bigger by more than a factor of O ( t ) . Iterating this augmentation t times therefore provably identifies all problematic coordinates.

To formalize this intuition, the following lemma will be needed to bound how much S grows at each iteration; it shows that the number of coordinates that correlate with a low-dimensional subspace is not too large (proof deferred to Appendix B):

Lemma 2.5. Let V ⊆ R n be a subspace with d := dim V . For some α > 0 define

Then | S | ≤ d/α 2 . Moreover, given a set of vectors that span V , we can compute S in time poly( n ) .

We also define the set of vectors v that have unusually large norm outside a set S , compared to √ v ⊤ Pv , which is the distance from v to the subspace spanned by the bottom d eigenvectors of Σ : Definition 2.6. For any matrix P ∈ R n × n and subset S ⊆ [ n ] , define W P,S := { v ∈ R n : ∥ v S c ∥ 2 > 3 √ v ⊤ Pv } .

We then formalize the guarantee of each iteration of IterativePeeling() as follows:

Lemma 2.7. Let n, t ∈ N and let P : n × n be an orthogonal projection matrix. Suppose τ ≥ 1 and K ⊆ [ n ] satisfy

Then there exists a set I P ( K ) with |I P ( K ) | ≤ 36 t 2 | K | such that

for all v ∈ B 0 ( t ) ∩ W P,K . Moreover, given P , K , and t , we can compute I P ( K ) in time poly( n ) .

Proof sketch. We define the set

It is clear from Lemma B.2 (applied with parameters V := span { Pe i : i ∈ K } and α := 1 / (6 t ) ) that |I P ( K ) | ≤ 36 t 2 | K | , and that I P ( K ) can be computed in time poly( n ) . It remains to show that |G P ( v ) \ ( I P ( K ) ∪ K ) | ≤ τ -1 for all v ∈ B 0 ( t ) .

Consider any v ∈ B 0 ( t ) ∩ W P,K . Then ∥ v K c ∥ 2 > 3 ∥ Pv ∥ 2 . It's sufficient to show that I P ( K ) contains some j ∈ supp( v ) \ K , i.e. that there is some j ∈ supp( v ) \ K such that e j correlates with span { P i : i ∈ K } . We accomplish this by showing that v K c correlates with Pv K = ∑ i ∈ K v i P i .

At a high level, the reason for this is that v K c is close to Pv K c (since P i ≈ e i for i ∈ K c ), and Pv = Pv K + Pv K c is much smaller than Pv K c ≈ v K c , so Pv K and Pv K c must be highly correlated. See Appendix B for the full proof. ■

We can now complete the proof of Lemma 2.4 by repeatedly invoking Lemma B.4.

Proof of Lemma 2.4. Let Σ = ∑ n i =1 λ i u i u ⊤ i be the eigendecomposition of Σ , and let P := ∑ n i = d +1 u i u ⊤ i be the projection onto the top n -d eigenspaces of Σ . Set K t = { i ∈ [ n ] : P ii < 1 -1 / (9 t 2 ) } . Because tr( P ) = n -d and P ii ≤ 1 for all i ∈ [ n ] , it must be that | K t | ≤ 9 t 2 d . Also, for any v ∈ B 0 ( t ) ∩ W P,K t we have trivially by t -sparsity that | supp( v ) \ K t | ≤ t.

Define K t -1 to be K t ∪I P ( K t ) where I P ( K t ) is as defined in Lemma B.4; we have the guarantees that | K t -1 | ≤ (1+36 t 2 ) | K t | and |G P ( v ) \ K t | ≤ t -1 for all v ∈ B 0 ( t ) ∩W P,K t . Since K t -1 ⊇ K t , it holds that W P,K t -1 ⊆ W P,K t , and thus |G P ( v ) \ K t | ≤ t -1 for all v ∈ B 0 ( t ) ∩ W P,K t -1 . Moreover, since K t -1 ⊇ K t , it obviously holds that P ii ≥ 1 -1 / (9 t 2 ) for all i ̸∈ K t -1 . This means we can apply Lemma B.4 with τ := t -1 and K := K t -1 and so iteratively define sets K t -2 ⊆ · · · ⊆ K 1 ⊆ K 0 ⊆ [ n ] in the same way. In the end, we obtain the set K 0 ⊆ [ n ] with | K 0 | ≤ 9 t 2 d (1 + 36 t 2 ) t and supp( v ) ⊆ K 0 for all v ∈ B 0 ( t ) ∩W P,K 0 . The latter guarantee means that in fact B 0 ( t ) ∩ W P,K 0 = ∅ . So for any t -sparse v ∈ R n it holds that

where the last inequality holds since λ d +1 P ⪯ Σ .

2.3 Beyond weak learning

So far, we have sketched a proof that if Σ has few outlier eigenvalues, then there is an efficient algorithm to compute a good dictionary (as in Theorem 2.3). This gives an efficient α -weak learning algorithm (via Proposition A.5). However, our ultimate goal is to find a regressor ˆ v with prediction error going to 0 as the number of samples increases. Definition 2.1 is not strong enough to ensure this. 8 However, it turns out that the dictionary constructed in Theorem 2.3 in fact satisfies a stronger guarantee 9 that is sufficient to achieve vanishing prediction error:

■

Definition 2.8. Let Σ ∈ R n × n be a positive semi-definite matrix and let t, B > 0 . A set { D 1 , . . . , D N } ⊆ R n is a ( t, B ) -ℓ 1 -representation for Σ if for any t -sparse v ∈ R n there is some α ∈ R N with v = ∑ N i =1 α i D i and ∑ N i =1 | α i | · ∥ D i ∥ Σ ≤ B · ∥ v ∥ Σ .

With this definition in hand, we can actually prove the following strengthening of Theorem 2.3:

Lemma 2.9. Let n, t, d ∈ N . Let Σ ∈ R n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ ·· · ≤ λ n . Then Σ has a ( t, 7 √ t √ λ n /λ d +1 ) -ℓ 1 -representation D of size at most n + t (7 t ) 2 t 2 + t d t . Moreover, D can be computed in time t O ( t 2 ) d t poly( n ) .

Proof sketch. Let S be the output of IterativePeeling( Σ , d, t ) . The dictionary D consists of the standard basis, together with a Σ -orthogonal basis for each set of t coordinates from S . The bound on |D| comes from the guarantee | S | ≤ (7 t ) 2 t +1 d . For any t -sparse vector v ∈ R n , we know that v S c is efficiently represented by the standard basis (because Theorem B.1 guarantees that ∥ v S c ∥ 2 ≤ O ( λ -1 / 2 d +1 ∥ v ∥ Σ ) ), and v S is efficiently represented by one of the Σ -orthonormal bases. See Appendix B for the full proof. ■

Why is the above guarantee useful? If each D i is normalized to unit Σ -norm, then the condition of ( t, B ) -ℓ 1 -representability is equivalent to ∥ α ∥ 1 ≤ B · ∥ v ∥ Σ . That is, with respect to the new set of features, the regressor α has bounded ℓ 1 norm. Thus, if we apply the Lasso with a set of features that is a ( t, B ) -ℓ 1 -representation for Σ , then standard 'slow rate' guarantees hold (proof in Section A):

Proposition 2.10. Let n, m, N, t ∈ N and B > 0 . Let Σ ∈ R n × n be a positive semi-definite matrix and let D be a ( t, B ) -ℓ 1 -representation of size N for Σ , normalized so that ∥ v ∥ Σ = 1 for all v ∈ D . Fix a t -sparse vector v ∗ ∈ R n , let X 1 , . . . , X m ∼ N (0 , Σ) be independent and let y i = ⟨ X i , v ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . For any R > 0 , define

where D ∈ R n × N is the matrix with columns comprising the elements of D , and X ∈ R m × n is the matrix with rows X 1 , . . . , X m . So long as m = Ω(log( n/δ )) and ∥ w ∗ ∥ Σ ∈ [ R/ 2 , R ] , it holds with probability at least 1 -δ that

Combining Proposition 2.10 with Lemma 2.9 shows that there is an algorithm with time complexity t O ( t 2 ) d t poly( n ) and sample complexity O (poly( t )( λ n /λ d +1 ) log( n ) log( d )) for finding a regressor with squared prediction error o ( σ 2 + ∥ v ∗ ∥ 2 Σ ) . This is a simplified version of Theorem 1.2. The full proof involves additional technical details (e.g. more careful analysis to take care of large eigenvalues, and to avoid needing an estimate R for ∥ w ∗ ∥ Σ ) but the above exposition contains the central ideas. Theorem 1.1 similarly computes the set S from Lemma 2.4 but uses it to construct a different dictionary: the standard basis, plus a Σ -orthonormal basis for S . 10 See Appendix C for the full proofs and pseudocode.

3 Additional Results

We now return to Question 2.2 and ask whether there are other families of ill-conditioned Σ for which we can prove non-trivial bounds on N t,α (Σ) .

First, we ask what can be shown for arbitrary covariance matrices. We prove that every covariance matrix Σ satisfies a non-trivial bound N t, 1 /O ( t 3 / 2 log n ) (Σ) ≤ O ( n t -1 / 2 ) . In fact, building on tools from computational geometry, we show the stronger result that Σ has a ( t, O ( t 3 / 2 log n )) -ℓ 1 -representation that of size O ( n t -1 / 2 ) , that is computable from samples in time ˜ O ( n t -Ω(1 /t ) ) for

any constant t > 1 (Theorem D.5). As a corollary, we provide the first sparse linear regression algorithm with time complexity that is a polynomial-factor better than brute force, and with near-optimal sample complexity, for any constant t and arbitrary Σ (proof in Section D):

Theorem 3.1. Let n, m, t, B ∈ N and σ > 0 , and let Σ ∈ R n × n be a positive-definite matrix. Let w ∗ ∈ R n be t -sparse, and suppose ∥ w ∗ ∥ Σ ∈ [ B/ 2 , B ] . Suppose m ≥ Ω( t log n ) . Let ( X i , y i ) i m =1 be independent samples where X i ∼ N (0 , Σ) and y i = ⟨ X i , w ∗ ⟩ + N (0 , σ 2 ) . Then there is an O ( m 2 n t -1 / 2 + n t -Ω(1 /t ) log O ( t ) n ) -time algorithm that, given ( X i , y i ) i m =1 , B , and σ 2 , produces an estimate ˆ w ∈ R n satisfying, with probability 1 -o (1) ,

Second, one goal is to improve 'sample complexity' (i.e. obtain α without dependence on condition number) without paying too much in 'time complexity' (i.e. retain bounds on N t,α that are better than n t ). To this end, we prove that the dependence on κ in the correlation level (see Fact A.4) can actually be replaced by dependence on κ in the dictionary size (proof in Appendix E):

Theorem 3.2. Let n, t ∈ N . Let Σ ∈ R n × n be a positive-definite matrix with condition number κ . Then N t, 1 / 3 t +1 (Σ) ≤ 2 O ( t 2 ) κ 2 t +1 · n.

In particular, for any constant t = 1 /ϵ , our result shows that there is a nearly-linear size dictionary with constant correlations even for covariance matrices with polynomially-large condition number κ ≤ n ϵ/ 100 . While we are not currently aware of an efficient algorithm for computing the dictionary, the above bound nonetheless raises the interesting possibility that there may be a sample-efficient and computationally-efficient weak learning algorithm under a super-constant bound on κ .

4 Related work

Dealing with correlated covariates. There is considerable work on improving the performance of Lasso in situations where some clusters of covariates are highly correlated [47, 19, 2, 42, 21, 12, 27]. These methods can work well for two-sparse dependencies, but generally do not work as well for higher-order dependencies - hence they cannot be used to prove our main result. The approach of [2] is perhaps the closest in spirit to ours. They perform agglomerative clustering of correlated covariates, orthonormalize the clusters with respect to Σ , and apply Lasso (or solve an essentially equivalent group Lasso problem). This method fails, for example, when there is a single threesparse dependency, and the remaining covariates have some mild correlations. Depending on the correlation threshold, their method will either aggressively merge all covariates into a single cluster, or fail to merge the dependent covariates.

Feature adaptation and preconditioning. Generalizations of Lasso via a preliminary change-ofbasis (or explicitly altering the regularization term) have been studied in the past, but largely not to solve sparse linear regression per se; instead the goal has been using ℓ 1 regularization to encourage other structural properties such as piecewise continuity (e.g. in the 'fused lasso', see [35, 36, 20, 8] for some more examples). An exception is recent work showing that a 'sparse preconditioning' step can enable Lasso to be statistically efficient for sparse linear regression when the covariates have a certain Markovian structure [23]. Our notion of feature adaptation via dictionaries generalizes sparse preconditioning, which corresponds to choosing a non-standard basis in which Σ becomes well-conditioned and the sparsity of the signal is preserved.

Statistical query (SQ) model; sparse halfspaces. From the complexity standpoint, N t,α (Σ) is a covering number and therefore closely corresponds to a packing number P t,α (Σ) (see Section A.1 for the definition). This packing number is essentially the (correlational) statistical dimension , which governs the complexity of sparse linear regression with covariates from N (0 , Σ) in the (correlational) SQ model (see e.g. [14] for exposition of this model). Whereas strong n Ω( t ) SQ lower bounds are known for related problems such as sparse parities with noise [29], no non-trivial (i.e. super-linear) lower bounds are known for sparse linear regression. Relatedly, in a COLT open problem, Feldman asked whether any non-trivial bounds can be shown for the complexity of weak learning sparse halfspaces in the SQ model [11]. Our results also yield improved bounds for weakly SQ-learning sparse halfspaces over certain families of multivariate Gaussian distributions.

Improving brute-force for arbitrary Σ . Several prior works have suggested improvements on brute-force search for variants of t -sparse linear regression [18, 16, 31, 6]. However, all of these have limitations preventing their application to the general setting we address in Theorem 3.1. Specifically, [18] requires Ω( n t ) preprocessing time on the covariates; [16, 31] require noiseless responses; and [6] has time complexity scaling with log m n (since our random-design setting necessitates m ≥ Ω( t log n ) , their algorithm has time complexity much larger than n t ).

References

Peter J Bickel, Ya'acov Ritov, Alexandre B Tsybakov, et al. Simultaneous analysis of lasso and dantzig selector. The Annals of statistics , 37(4):1705-1732, 2009.

Peter B¨ uhlmann, Philipp R¨ utimann, Sara Van De Geer, and Cun-Hui Zhang. Correlated variables in regression: clustering and sparse estimation. Journal of Statistical Planning and Inference , 143(11):1835-1858, 2013.

Emmanuel Candes, Terence Tao, et al. The dantzig selector: Statistical estimation when p is much larger than n. Annals of statistics , 35(6):2313-2351, 2007.

Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 59(8):1207-1223, 2006.

Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE transactions on information theory , 51(12):4203-4215, 2005.

Jean Cardinal and Aur´ elien Ooms. Algorithms for approximate sparse regression and nearest induced hulls. Journal of Computational Geometry , 13(1):377-398, 2022.

Shaobing Chen and David Donoho. Basis pursuit. In Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers , volume 1, pages 41-44. IEEE, 1994.

Arnak S Dalalyan, Mohamed Hebiri, Johannes Lederer, et al. On the prediction performance of the lasso. Bernoulli , 23(1):552-581, 2017.

Abhimanyu Das and David Kempe. Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection. arXiv preprint arXiv:1102.3975 , 2011.

David L Donoho and Philip B Stark. Uncertainty principles and signal recovery. SIAM Journal on Applied Mathematics , 49(3):906-931, 1989.

Vitaly Feldman. Open problem: The statistical query complexity of learning sparse halfspaces. In Conference on Learning Theory , pages 1283-1289. PMLR, 2014.

Mario Figueiredo and Robert Nowak. Ordered weighted l1 regularized regression with strongly correlated covariates: Theoretical aspects. In Artificial Intelligence and Statistics , pages 930938. PMLR, 2016.

Dean P Foster and Edward I George. The risk inflation criterion for multiple regression. The Annals of Statistics , pages 1947-1975, 1994.

Surbhi Goel, Aravind Gollakota, Zhihan Jin, Sushrut Karmalkar, and Adam Klivans. Superpolynomial lower bounds for learning one-layer neural networks using gradient descent. In International Conference on Machine Learning , pages 3587-3596. PMLR, 2020.

Aparna Gupte and Vinod Vaikuntanathan. The fine-grained hardness of sparse linear regression. arXiv preprint arXiv:2106.03131 , 2021.

Aparna Ajit Gupte and Kerri Lu. Fine-grained complexity of sparse linear regression. 2020.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.

Sariel Har-Peled, Piotr Indyk, and Sepideh Mahabadi. Approximate sparse linear regression. In 45th International Colloquium on Automata, Languages, and Programming (ICALP 2018) . Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2018.

Jim C Huang and Nebojsa Jojic. Variable selection through correlation sifting. In RECOMB , volume 6577, pages 106-123. Springer, 2011.

Jan-Christian H¨ utter and Philippe Rigollet. Optimal rates for total variation denoising. In Conference on Learning Theory , pages 1115-1146. PMLR, 2016.

Jinzhu Jia, Karl Rohe, et al. Preconditioning the lasso for sign consistency. Electronic Journal of Statistics , 9(1):1150-1172, 2015.

Jonathan Kelner, Frederic Koehler, Raghu Meka, and Ankur Moitra. Learning some popular gaussian graphical models without condition number bounds. In Proceedings of Neural Information Processing Systems (NeurIPS) , 2020.

Jonathan Kelner, Frederic Koehler, Raghu Meka, and Dhruv Rohatgi. On the power of preconditioning in sparse linear regression. 62nd Annual IEEE Symposium on Foundations of Computer Science , 2021.

Jonathan A Kelner, Frederic Koehler, Raghu Meka, and Dhruv Rohatgi. Distributional hardness against preconditioned lasso via erasure-robust designs. arXiv preprint arXiv:2203.02824 , 2022.

Guillaume Lecu´ e and Shahar Mendelson. Learning subgaussian classes: Upper and minimax bounds. arXiv preprint arXiv:1305.4825 , 2013.

Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 1049-1065. IEEE, 2015.

Yuan Li, Benjamin Mark, Garvesh Raskutti, and Rebecca Willett. Graph-based regularization for regression problems with highly-correlated designs. In 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP) , pages 740-742. IEEE, 2018.

Shahar Mendelson et al. On the performance of kernel classes. 2003.

Elchanan Mossel, Ryan O'Donnell, and Rocco P Servedio. Learning juntas. In Proceedings of the thirty-fifth annual ACM symposium on Theory of computing , pages 206-212, 2003.

Wolfgang Mulzer, Huy L Nguyˆ en, Paul Seiferth, and Yannik Stein. Approximate k-flat nearest neighbor search. In Proceedings of the forty-seventh annual ACM symposium on Theory of Computing , pages 783-792, 2015.

Eric Price, Sandeep Silwal, and Samson Zhou. Hardness and algorithms for robust and sparse optimization. In International Conference on Machine Learning , pages 17926-17944. PMLR, 2022.

Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Restricted eigenvalue properties for correlated gaussian designs. The Journal of Machine Learning Research , 11:2241-2259, 2010.

Phillippe Rigollet and Jan-Christian H¨ utter. High dimensional statistics. Lecture notes for course 18S997 , 813:814, 2015.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Optimistic rates for learning with a smooth loss. arXiv preprint arXiv:1009.3896 , 2010.

Robert Tibshirani, Michael Saunders, Saharon Rosset, Ji Zhu, and Keith Knight. Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 67(1):91-108, 2005.

Ryan J Tibshirani and Jonathan Taylor. The solution path of the generalized lasso. The annals of statistics , 39(3):1335-1371, 2011.

Gregory Valiant. Finding correlations in subquadratic time, with applications to learning parities and juntas. In 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science , pages 11-20. IEEE, 2012.

Sara Van De Geer. On tight bounds for the lasso. Journal of Machine Learning Research , 19:46, 2018.

Sara A Van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

Sara A Van De Geer, Peter B¨ uhlmann, et al. On the conditions used to prove oracle results for the lasso. Electronic Journal of Statistics , 3:1360-1392, 2009.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Fabian L Wauthier, Nebojsa Jojic, and Michael I Jordan. A comparative framework for preconditioned lasso algorithms. Advances in Neural Information Processing Systems , 26:1061-1069, 2013.

Tong Zhang. Learning bounds for kernel regression using effective data dimensionality. Neural Computation , 17(9):2077-2098, 2005.

Yuchen Zhang, Martin J Wainwright, Michael I Jordan, et al. Optimal prediction for sparse linear models? lower bounds for coordinate-separable m-estimators. Electronic Journal of Statistics , 11(1):752-799, 2017.

Lijia Zhou, Frederic Koehler, Danica J Sutherland, and Nathan Srebro. Optimistic rates: A unifying theory for interpolation learning and regularization in linear regression. arXiv preprint arXiv:2112.04470 , 2021.

Shuheng Zhou. Restricted eigenvalue conditions on subgaussian random matrices. arXiv preprint arXiv:0912.4045 , 2009.

Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology) , 67(2):301-320, 2005.

A Preliminaries

Throughout, we use the following standard notation. For positive integers n, m ∈ N , we write A : m × n to denote a matrix with m rows, n columns, and real-valued entries. The standard inner product on R n is denoted ⟨ u, v ⟩ := u ⊤ v . For a positive semi-definite matrix Σ : n × n we define the Σ -inner product on R n by ⟨ u, v ⟩ Σ := u ⊤ Σ v and the Σ -norm by ∥ u ∥ Σ = √ ⟨ u, u ⟩ Σ . For n ∈ N (made clear by context) we let e 1 , . . . , e n ∈ R n be the standard basis vectors e i ( j ) := ✶ [ j = i ] . For a vector v ∈ R n and set S ⊆ [ n ] we write v S to denote the restriction of v to coordinates in S . For symmetric matrices A,B : n × n we write A ⪯ B to denote that B -A is positive semi-definite.

A.1 Covering, packing, and ℓ 1 -representability

Wepreviously defined the covering number of t -sparse vectors with respect to a covariance matrix Σ . We next define the packing number (i.e. correlational statistical dimension) and ℓ 1 -representability, and discuss the connections between these quantities as well as their algorithmic implications.

Definition A.1. Let Σ : n × n be a positive semi-definite matrix and let t, α > 0 . A set { v 1 , . . . , v N } ⊆ R n is a ( t, α ) -packing for Σ if every v i is t -sparse, and

for all i, j ∈ [ N ] with i = j . The (correlational) statistical dimension of t -sparse vectors with maximum correlation α , under the Σ -inner product, is denoted P t,α (Σ) and defined as the size of the largest ( t, α ) -packing.

̸

We will make use of the following connections between packing, covering, and ℓ 1 -representability.

Lemma A.2 (Covering ⇔ packing) . For any positive semi-definite matrix Σ : n × n and t, α > 0 , it holds that ( α 2 / 3) P t,α 2 / 2 (Σ) ≤ N t,α (Σ) ≤ P t,α (Σ) .

̸

Proof. First inequality. Let { D 1 , . . . , D N } be any maximum-size ( t, α 2 / 2) -packing. Since the D i 's are all t -sparse, each must be correlated with some element of a ( t, α ) -dictionary. Thus, it suffices to show that for any v ∈ R n , the set S ( v ) := { i ∈ [ N ] : |⟨ D i , v ⟩ Σ | ≥ α ∥ D i ∥ Σ ∥ v ∥ Σ } has size | S ( v ) | ≤ 3 /α 2 . Indeed, for any i, j ∈ S ( v ) with i = j , we have by the definition of a packing that

For each i ∈ S ( v ) define R i = D i -⟨ D i , v ⟩ Σ v/ ∥ v ∥ 2 Σ . Then

̸

where the last inequality uses the bound ∥ R i ∥ Σ ≤ ∥ D i ∥ Σ . Rearranging gives | S ( v ) | ≤ 1 + (2 /α 2 ) . Second inequality. Let { D 1 , . . . , D N } be any maximal ( t, α ) -packing. Then for any t -sparse v ∈ R n , maximality implies that there must be some i ∈ [ N ] with |⟨ D i , v ⟩ Σ | ≥ α ∥ D i ∥ Σ ∥ v ∥ Σ . So { D 1 , . . . , D N } is also a ( t, α ) -dictionary.

Lemma A.3 ( ℓ 1 -representation = ⇒ covering) . Let Σ : n × n be a positive semi-definite matrix and let t, B > 0 . If { D 1 , . . . , D N } ⊆ R n is a ( t, B ) -ℓ 1 -representation for Σ , then it is also a ( t, 1 /B ) -dictionary for Σ .

Proof. Pick any t -sparse v ∈ R n . By ℓ 1 -representability, there is some α ∈ R N with v = ∑ N i =1 α i D i and ∑ N i =1 | α i | · ∥ D i ∥ Σ ≤ B · ∥ v ∥ Σ . Hence

and thus max j ∈ [ N ] |⟨ v,D j ⟩ Σ | ∥ v ∥ Σ ∥ D j ∥ Σ ≥ 1 /B .

We can now easily prove that the standard basis is a good dictionary for well-conditioned Σ .

Fact A.4. Let Σ be a positive definite matrix with condition number λ max (Σ) λ min (Σ) ≤ κ . Under the Σ -inner product, every t -sparse vector is at least 1 / ( √ κt ) -correlated with some standard basis vector.

Proof. By Lemma A.3, it suffices to show that the standard basis { e 1 , . . . , e n } is a ( t, √ κt ) -ℓ 1 -representation for Σ . Indeed, for any t -sparse v ∈ R n ,

as desired.

A.2 Algorithmic implications

An existential proof that N t,α (Σ) is small unfortunately does not in general give an efficient algorithm for constructing a concise dictionary. However, with the caveat that the dictionary must be given to the algorithm as advice, bounds on N t,α do imply weak learning algorithms with sample complexity O ( α -2 log( n )) :

Proposition A.5. Let Σ : n × n be a positive semi-definite matrix and let D be a ( t, α ) -dictionary for Σ , for some t ∈ N and α ∈ (0 , 1) . For m ∈ N and t -sparse v ∗ ∈ R n , let X 1 , . . . , X m ∼ N (0 , Σ) be independent and let y i = ⟨ X i , v ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . Define the estimator

where X : m × n is the matrix with rows X 1 , . . . , X m . For any δ > 0 , if m ≥ Cα -2 log(32 |D| /δ ) for a sufficiently large absolute constant C , then with probability at least 1 -δ ,

Proof. Since D is a ( t, α ) -dictionary, we know that there is some ˜ v ∈ D with |⟨ ˜ v, v ∗ ⟩ Σ | ≥ α ∥ ˜ v ∥ Σ ∥ v ∗ ∥ Σ . We then apply Lemma F.4.

The above guarantee is essentially of the form 'at least 1% of the signal variance can be explained'. Under the ℓ 1 -representability condition, something much stronger is true:

Proposition A.6. Let n, m, N, t ∈ N and B > 0 . Let Σ : n × n be a positive semi-definite matrix and let D be a ( t, B ) -ℓ 1 -representation of size N for Σ , normalized so that ∥ v ∥ Σ = 1 for all v ∈ D .

Fix a t -sparse vector v ∗ ∈ R n , let X 1 , . . . , X m ∼ N (0 , Σ) be independent and let y i = ⟨ X i , v ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . For any R > 0 , define

where D : n × N is the matrix with columns comprising the elements of D , and X : m × n is the matrix with rows X 1 , . . . , X m . So long as m = Ω(log( n/δ )) and R ≥ ∥ v ∗ ∥ Σ , it holds with probability at least 1 -δ that

Proof. By ℓ 1 -representability and normalization of D , there is some w ∗ ∈ R N such that v ∗ = Dw ∗ and ∥ w ∗ ∥ 1 ≤ B ∥ v ∗ ∥ Σ ≤ BR . Let Γ = D ⊤ Σ D . Also, by normalization, max i Γ ii = 1 . Thus, we can apply standard 'slow rate' Lasso guarantees to the samples ( D ⊤ X i , y i ) i m =1 to get the claimed bound (see e.g. Theorem 14 of [22]).

A.3 Optimizing the Lasso in near-linear time

Theorem A.7 (see e.g. Corollary 4 and Section 5.3 in [34]) . Let n, m, B, H, T ∈ N and σ > 0 . Fix X 1 , . . . , X m ∈ R n with ∥ X i ∥ ∞ ≤ H for all i , and fix w ∗ ∈ R n with ∥ w ∗ ∥ 1 ≤ B . For i ∈ [ m ] define y i = ⟨ X i , w ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) are independent random variables. Given ( X i , y i ) i m =1 as well as B , T , and σ 2 , there is an algorithm MirrorDescentLasso( ( X i , y i ) i m =1 , B , T , σ 2 ) , which optimizes the Lasso objective via T iterations of mirror descent, that produces an estimate ˆ w ∈ R n satisfying ∥ ˆ w ∥ 1 ≤ B and, with probability 1 -o (1) ,

Moreover, the time complexity of MirrorDescentLasso() is ˜ O ( nmT ) .

Theorem A.8. Let n, m, B, H ∈ N and σ > 0 . Let Σ : n × n be positive semi-definite with max j ∈ [ n ] Σ jj ≤ H 2 . Fix w ∗ ∈ R n with ∥ w ∗ ∥ 1 ≤ B . Let ( X i , y i ) i m =1 be independent draws where X i ∼ N (0 , Σ) and y i = ⟨ X i , w ∗ ⟩ + N (0 , σ 2 ) . Then MirrorDescentLasso( ( X i , y i ) i m =1 , B , m, σ 2 ) computes, in time ˜ O ( nm 2 ) , an estimate ˆ w satisfying, with probability 1 -o (1) ,

Proof. Since max j Σ jj ≤ H we have that max i ∥ X i ∥ ∞ ≤ O ( H log n ) with probability 1 -o (1) . Applying Theorem A.7 with this bound and with T = m , we obtain some ˆ w ∈ R n with ∥ ˆ w ∥ 1 ≤ B and, with probability 1 -o (1) ,

where ϵ = ˜ O ( H 2 B 2 /m ) + √ H 2 B 2 σ 2 /m ) . By χ 2 -concentration, we have 1 m ∥ Xw ∗ -y ∥ 2 2 ≤ σ 2 (1 + O (1 / √ m )) with probability 1 -o (1) . Thus,

and

Next, since sup w ∈ R n : ∥ w ∥ 1 ≤ B ⟨ w -w ∗ , x ⟩ ≤ 2 B ∥ x ∥ ∞ ≤ O ( HB log n ) with probability 1 -o (1) over x ∼ N (0 , Σ) , we can apply Theorem C.1 to get that with probability 1 -o (1) ,

Substituting the bounds on ∥ X ˆ w -y ∥ 2 and ∥ X ˆ w -y ∥ 2 2 gives

Substituting in the value of ϵ and simplifying, we get

as claimed.

B Iterative Peeling

In this section we give the complete proof of Lemma 2.4, restated below as Theorem B.1, which describes the guarantees of IterativePeeling() (see Algorithm 1). This is a key ingredient in the proofs of Theorems 1.1 and 1.2. We also use it to formally prove Theorem 2.3, as well as Lemma 2.9.

Theorem B.1. Let n, t, d ∈ N . Let Σ : n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ · · · ≤ λ n . Given Σ , d , and t , there is a polynomial-time algorithm IterativePeeling() producing a set S ⊆ [ n ] with the following guarantees:

For every t -sparse v ∈ R n , it holds that ∥ ∥ v [ n ] \ S ∥ ∥ 2 ≤ 3 λ -1 / 2 d +1 ∥ v ∥ Σ .

| S | ≤ (7 t ) 2 t +1 d.

Essentially, the set S contains every coordinate i ∈ [ n ] that 'participates' in an approximate sparse dependency, in the sense that there is some sparse linear combination of the covariates with small variance compared to the coefficient on i . To compute S , the algorithm IterativePeeling() first computes the orthogonal projection matrix P that projects onto the subspace spanned by the top n -d eigenvectors of Σ . Starting with the set of coordinates that correlate with ker( P ) , the procedure then iteratively grows S in such a way that at each step, a new participant of each approximate sparse dependency is discovered, but S does not become too much larger.

The following lemma will be needed to bound how much S grows at each iteration:

Lemma B.2. Let V ⊆ R n be a subspace with d := dim V . For some α > 0 define

Then | S | ≤ d/α 2 . Moreover, given a set of vectors that span V , we can compute S in time poly( n ) .

Proof. Let k := | S | and without loss of generality suppose S = { 1 , . . . , k } . Define a matrix A ∈ R n × n as follows. For 1 ≤ i ≤ k let row A i ∈ V be some vector such that ∥ A i ∥ 2 = 1 and A ii ≥ α . For k + 1 ≤ i ≤ n let A i = 0 . Then tr( A ) ≥ kα and ∥ A ∥ F = √ k . However, rank( A ) ≤ d , so the singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ n ≥ 0 of A satisfy σ d +1 = 0 . Thus,

where the second inequality is by e.g. Von Neumann's trace inequality, and the third inequality is by d -sparsity of the vector σ . It follows that k ≤ d/α 2 as claimed.

Let A be the matrix with columns consisting of the given spanning set for V . By Gram-Schmidt, we may transform the spanning set into an orthonormal basis for V , so that A has d columns, and A ⊤ A = I d . Fix i ∈ [ n ] . Then sup x ∈ V \{ 0 } x i / ∥ x ∥ 2 ≥ α if and only if ( Av ) 2 i -α 2 ∥ Av ∥ 2 2 ≥ 0 for some nonzero v ∈ R d . Equivalently, ( Av ) 2 i ≥ α 2 for some unit vector v . This is possible if and only if ∥ A i ∥ 2 ≥ α (where A i is the i -th row of A ), which can be checked in polynomial time.

For notational convenience, we also define the set W P,S of vectors v with unusually large norm outside the set S .

Definition B.3. For any matrix P : n × n and subset S ⊆ [ n ] , define W P,S := { v ∈ R n : ∥ v S c ∥ 2 > 3 √ v ⊤ Pv } .

We then formalize the guarantee of each iteration of IterativePeeling() as follows:

Lemma B.4. Let n, t ∈ N and let P : n × n be an orthogonal projection matrix. Suppose τ ≥ 1 and K ⊆ [ n ] satisfy

Then there exists a set I P ( K ) with |I P ( K ) | ≤ 36 t 2 | K | such that

for all v ∈ B 0 ( t ) ∩ W P,K . Moreover, given P , K , and t , we can compute I P ( K ) in time poly( n ) .

Proof. We define the set

It is clear from Lemma B.2 (applied with parameters V := span { Pe i : i ∈ K } and α := 1 / (6 t ) ) that |I P ( K ) | ≤ 36 t 2 | K | , and that I P ( K ) can be computed in time poly( n ) . It remains to show that | supp( v ) \ ( I P ( K ) ∪ K ) | ≤ τ -1 for all v ∈ B 0 ( t ) ∩ W P,K .

Consider any v ∈ B 0 ( t ) ∩ W P,K . Then ∥ v K c ∥ 2 > 3 √ v ⊤ Pv . We have

where the first equality uses the fact that P is a projection matrix. We also know that

by the triangle inequality, the bound ∥ P i -e i ∥ 2 2 = ( I -P ) ii = 1 -P ii ≤ 1 / (9 t ) (since i ̸∈ K ), and t -sparsity of v . Moreover, (2) implies that

Combining (1) and (3), the triangle inequality gives

Next, observe that

and hence

where the last inequality is by (4). On the other hand, observe that

Hence, there is some j ∈ supp( v ) \ K such that

So the vector x ( v ) := ∑ i ∈ K v i P i ∈ span { P i : i ∈ K } satisfies | x ( v ) j | > ∥ x ( v ) ∥ 2 / (5 √ t ) . Moreover, x ( v ) is nonzero since | x ( v ) j | > 0 . Thus, j ∈ I P ( K ) . Since we chose j to be in supp( v ) \ K , it follows that

where the last inequality is by assumption (b) in the lemma statement.

We can now complete the proof of Theorem B.1 by repeatedly invoking Lemma B.4 (this proof was given in Section 2.2 and is duplicated here for completeness).

Proof of Theorem B.1. Let Σ = ∑ n i =1 λ i u i u ⊤ i be the eigendecomposition of Σ , and let P := ∑ n i = d +1 u i u ⊤ i be the projection onto the top n -d eigenspaces of Σ . Set K t = { i ∈ [ n ] : P ii < 1 -1 / (9 t 2 ) } . Because tr( P ) = n -d and P ii ≤ 1 for all i ∈ [ n ] , it must be that | K t | ≤ 9 t 2 d . Also, for any v ∈ B 0 ( t ) ∩ W P,K t we have trivially by t -sparsity that | supp( v ) \ K t | ≤ t.

Define K t -1 to be K t ∪I P ( K t ) where I P ( K t ) is as defined in Lemma B.4; we have the guarantees that | K t -1 | ≤ (1+36 t 2 ) | K t | and |G P ( v ) \ K t | ≤ t -1 for all v ∈ B 0 ( t ) ∩W P,K t . Since K t -1 ⊇ K t , it holds that W P,K t -1 ⊆ W P,K t , and thus |G P ( v ) \ K t | ≤ t -1 for all v ∈ B 0 ( t ) ∩ W P,K t -1 . Moreover, since K t -1 ⊇ K t , it obviously holds that P ii ≥ 1 -1 / (9 t 2 ) for all i ̸∈ K t -1 . This means we can apply Lemma B.4 with τ := t -1 and K := K t -1 and so iteratively define sets K t -2 ⊆ · · · ⊆ K 1 ⊆ K 0 ⊆ [ n ] in the same way. In the end, we obtain the set K 0 ⊆ [ n ] with | K 0 | ≤ 9 t 2 d (1 + 36 t 2 ) t and supp( v ) ⊆ K 0 for all v ∈ B 0 ( t ) ∩W P,K 0 . The latter guarantee means that in fact B 0 ( t ) ∩ W P,K 0 = ∅ . So for any t -sparse v ∈ R n it holds that

where the last inequality holds since λ d +1 P ⪯ Σ .

■

̸

Proof of Lemma 2.9. By Theorem B.1, there is a polynomial-time computable set S ⊆ [ n ] such that ∥ v S c ∥ 2 ≤ 3 √ tλ -1 / 2 d +1 ∥ v ∥ Σ for all v ∈ B 0 ( t ) , and | S | ≤ (7 t ) 2 t +1 d . Let the dictionary D consist of the standard basis { e 1 , . . . , e n } together with a Σ -orthogonal basis for each subspace spanned by t vectors in { e i : i ∈ S } . Let v ∈ R n be t -sparse. Let v S denote the restriction of v to S , i.e. v S := v -∑ i ∈ [ n ] \ S v i e i . By construction of the dictionary, there is a Σ -orthogonal basis for { e i : i ∈ S ∩ supp( v ) } , so there are d 1 , . . . , d t ∈ D and coefficients b d 1 , . . . , b d t ∈ R with v S = ∑ t i =1 b d i d i and ⟨ d i , d j ⟩ Σ = 0 for all i, j ∈ [ t ] with i = j . Note that ∥ v S ∥ 2 Σ = ∑ t i =1 b 2 d i ∥ d i ∥ 2 Σ , so

Now, we claim that the desired coefficient vector { α d : d ∈ D} for v is defined by α d = b d + ∑ i ∈ [ n ] \ S v i ✶ [ d = e i ] . We can check that ∑ d ∈D α d d = ∑ t i =1 b d i + ∑ i ∈ [ n ] \ S v i e i = v. Also,

by the guarantee of set S .

It follows that

Thus,

which completes the proof.

Proof of Theorem 2.3. Immediate from Lemma 2.9 and Lemma A.3.

C An efficient algorithm for handling outlier eigenvalues

In this section we describe and provide error guarantees for a novel sparse linear regression algorithm BOAR-Lasso() (see Algorithm 2 for pseudocode), completing the proof of Theorem 1.1; in Section C.1 we then analyze a modified algorithm to prove Theorem 1.2.

The key subroutine of BOAR-Lasso() is the procedure AdaptivelyRegularizedLasso() , which (like the simplified procedure AdaptedBP() from Section 3) first invokes procedure IterativePeeling() to compute the set of coordinates that participate in sparse approximate dependencies, and second computes a modified Lasso estimate where those coordinates are not regularized.

We start with Theorem C.2, which shows that, in the setting where Σ has few outlier eigenvalues, the procedure AdaptivelyRegularizedLasso() estimates the sparse ground truth regressor at the 'slow rate' (e.g. in the noiseless setting, the excess risk is at most O ( ∥ v ∗ ∥ 2 Σ r eff /m ) ). Typical excess risk analyses for Lasso proceed by applying some general-purpose machinery for generalization bounds, such as the following result which only requires understanding ⟨ w -w ∗ , X ⟩ for X ∼ N (0 , Σ) .

■

■

Algorithm 2: Solve sparse linear regression when covariate eigenspectrum has few outliers

Data: Covariance matrix Σ : n × n , samples ( X i , y i ) i m =1 , sparsity t , small eigenvalue count d l , failure probability δ

Result: Estimate ˆ v of unknown sparse regressor, satisfying Theorem C.2

∑

⊤

n

i

=1

i

λ

S

i

i

eigendecomposition of

u

u

IterativePeeling(

Return

Procedure BOAR-Lasso( Σ , ( Y i , y i ) i m =1 , t , d l , L , δ )

Data: Covariance matrix Σ : n × n , samples ( X i , y i ) i m =1 , sparsity t , small eigenvalue count d l , repetition count L , failure probability δ

Result: Estimate ˆ v of unknown sparse regressor, satisfying Theorem C.3

Set

A

:=

(

j

)

ˆ

w

(

j

((

+1)

X

i

,

(

v

ˆ

j

+1)

ˆ

s

(

j

+1)

return

s

ˆ

(

L

mj

+1

, . . . , m

(

j

+1)

{

}

⟨

X

)

Theorem C.1 (Theorem 1 in [45]) . Let n, m ∈ N and ϵ, δ, σ > 0 . Let Σ : n × n be a positive semi-definite matrix and fix w ∗ ∈ R n . Let X : m × n have i.i.d. rows X 1 , . . . , X m ∼ N (0 , Σ) , and let y = Xw ∗ + ξ where ξ ∼ N (0 , σ 2 I m ) . Let F : R d → [0 , ∞ ] be a continuous function such that

If m ≥ 196 ϵ -2 log(12 /δ ) , then with probability at least 1 -4 δ it holds that for all w ∈ R d ,

In classical settings, e.g. (a) where ∥ v ∗ ∥ 1 is bounded and max i Σ ii ≤ 1 (see Proposition A.6) or (b) where Σ satisfies the compatibility condition (see Definition G.1), the above result can be applied together with the straightforward bound ⟨ v -v ∗ , X ⟩ ≤ ∥ v -v ∗ ∥ 1 ∥ X ∥ ∞ . To prove Theorem C.2 we follow the same general recipe as (a), with several modifications.

First, since max i Σ ii could be arbitrarily large, we need to treat the (few) large eigenspaces of Σ separately when bounding ⟨ v -v ∗ , X ⟩ . Similarly, since Theorem B.1 only gives bounds on v ∗ for coordinates outside S , we separately bound ⟨ ( v -v ∗ ) S , X ⟩ using that | S | is small. Second, to achieve the optimal rate of σ 2 n eff /m rather than σ 2 √ n eff /m , we do not directly apply Theorem C.1 to the noisy samples ( X i , y i ) ; instead, we derive a modification of that result (Lemma F.7) that only invokes Theorem C.1 on the noiseless samples ( X i , ⟨ X i , v ∗ ⟩ ) , and separately bounds the in-sample prediction error ∥ X (ˆ v -v ∗ ) ∥ 2 . A similar technique is used in [45] for constrained least-squares programs (see their Lemma 15); our Lemma F.7 applies to a broad family of additively regularized programs, which obviates the need to independently estimate ∥ v ∗ ∥ Σ but otherwise achieves comparable bounds.

Theorem C.2. Let n, t, d l , d h , m ∈ N and σ, δ > 0 . Let Σ : n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ · · · ≤ λ n . Let ( X i , y i ) i m =1 be independent samples where X i ∼ N (0 , Σ) and

i

Σ

-⟨

i

X

,

ˆ

s

⟩

(

j

)

,

ˆ

w

(

s

ˆ

(

j

j

)

⟩

)

+1)

[

ˆ

s

(

n

]

j

)

, y

i

+ ˆ

w

(

j

+1)

n

+ ˆ

v

(

j

+1

+1)

ˆ

s

(

j

)

+1

,

)

i

∈

A

(

j

)

,

t

Σ

, d

l

, t

)

Σ

(

j

)

d

l

,

+1

,

δ/L

/*

)

See

Algorithm

1

*/

y i = ⟨ X i , v ∗ ⟩ + ξ i , for ξ i ∼ N (0 , σ 2 ) and a fixed t -sparse vector v ∗ ∈ R n . Let ˆ v be the output of AdaptivelyRegularizedLasso( Σ , ( X i , y i ) i m =1 , t, d l , δ ) . Let n eff := (7 t ) 2 t +1 d l + d h +log(48 /δ ) and let r eff := t ( λ n -d h /λ d l +1 ) log(12 n/δ ) . There are absolute constants c, C > 0 so that the following holds. If m ≥ Cn eff, then with probability at least 1 -δ ,

Proof. Define projection matrix P := ∑ n -d h i =1 u i u ⊤ i , so that rank( P ⊥ ) = d h and λ max ( P Σ P ) ≤ λ n -d h . For any v ∈ R n and X ∼ N (0 , Σ) , we can bound

where PX ∼ N (0 , P Σ P ) , Z ∼ N (0 , Σ -1 / 2 ( P ⊥ Σ P ⊥ ) S c S c Σ -1 / 2 ) , and W ∼ N (0 , Σ -1 / 2 Σ SS Σ -1 / 2 ) . First, since max i ( P Σ P ) ii ≤ λ max ( P Σ P ) ≤ λ n -d h , we have the Gaussian tail bound

Second, since

we have that ∥ Z ∥ 2 2 is stochastically dominated by χ 2 d h , and thus

Third, similarly, since Σ -1 / 2 Σ SS Σ -1 / 2 ⪯ I (again by Cauchy Interlacing Theorem) and also rank(Σ -1 / 2 Σ SS Σ -1 / 2 ) ≤ | S | , we have that ∥ W ∥ 2 2 is stochastically dominated by χ 2 | S | , and thus

Combining the above bounds, we have that with probability at least 1 -δ/ 4 over X ∼ N (0 , Σ) , for all v ∈ R n ,

We can therefore apply Lemma F.7 with covariance Σ , seminorm Φ( v ) := 2 √ 2 λ n -d h log(12 n/δ ) ∥ v S c ∥ 1 , p := 4( d h + | S | ) , ground truth v ∗ , samples ( X i , y i ) i m =1 , and failure probability δ/ 4 . By the bound on | S | (Theorem B.1) we have | S | + d h ≤ (7 t ) 2 t +1 d l + d h ≤ n eff , so it holds that m ≥ 16 p +196log(48 /δ ) . Thus, with probability at least 1 -2 δ , we have

By the guarantee of S (Theorem B.1) and t -sparsity of v ∗ , we have ∥ v ∗ S c ∥ 2 ≤ 3 λ -1 / 2 d l +1 ∥ v ∗ ∥ Σ , and thus ∥ v ∗ S c ∥ 1 ≤ 3 √ tλ -1 / 2 d l +1 ∥ v ∗ ∥ Σ . Substituting into the previous bound, we get

as claimed.

The limitation of AdaptivelyRegularizedLasso() is that the excess risk bound depends on ∥ v ∗ ∥ 2 Σ rather than just σ 2 . We next show that by a boosting approach, we can exponentially attenuate that dependence, essentially achieving the near-optimal rate of σ 2 n eff /m . The key insight is that after producing an estimate ˆ v of v ∗ , we can augment the set of covariates with the feature ⟨ X , ˆ v ⟩ , and try to predict the response y -⟨ X , ˆ v ⟩ , which is now a ( t +1) -sparse combination of the features. In standard settings, this is typically a bad idea because it introduces a sparse linear dependence. However, by the Cauchy Interlacing Theorem it increases the number of outlier eigenvalues by at most one - so our algorithms still apply. Thus, if we have enough samples that the excess risk bound in Theorem C.2 is non-trivially smaller than ∥ v ∗ ∥ 2 Σ , then we can iteratively achieve better and better estimates up to the noise limit. This is precisely what BOAR-Lasso() does; the precise guarantees are stated in the following theorem, which completes the proof of Theorem 1.1.

Theorem C.3. Let n, t, d l , d h , m, L ∈ N and σ, δ > 0 . Let Σ : n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ ·· · ≤ λ n . Let ( X i , y i ) i m =1 be independent samples where X i ∼ N (0 , Σ) and y i = ⟨ X i , v ∗ ⟩ + ξ i , for ξ i ∼ N (0 , σ 2 ) and a fixed t -sparse vector v ∗ ∈ R n .

Then, given Σ , ( X i , y i ) i m =1 , t , d l , and δ , the algorithm BOAR-Lasso() outputs an estimator ˆ v with the following properties.

Let n eff := (7 t ) 2 t +1 d l + d h + log(48 /δ ) and let r eff := t ( λ n -d h /λ d l +1 ) log(12 n/δ ) . There are absolute constants c 0 , C 0 > 0 such that the following holds. If m ≥ C 0 L ( n eff + r eff ) , then with probability at least 1 -δ , it holds that

Moreover, BOAR-Lasso() has time complexity poly( n, m, t ) .

Proof. Let ( A 0 , . . . , A L -1 ) be an partition of [ m ] into L sets of size m/L . The idea of the algorithm is to compute vectors ˆ v (1) , . . . , ˆ v ( L ) where each v ( i ) is an estimate of v ∗ -∑ i -1 j =1 ˆ v ( j ) . Concretely, fix some 0 ≤ j ≤ L -1 and suppose that we have computed some vectors ˆ v (1) , . . . , ˆ v ( j ) . Set ˆ s ( j ) := ˆ v (1) + · · · + ˆ v ( j ) . Define a matrix Σ ( j ) : ( n +1) × ( n +1) by

Thus, for example, Σ (0) has zeroes in the last row and last column. Now for each i ∈ A j , define ( X ( j ) i , y ( j ) i ) by

By construction, the m/L samples ( X ( j ) i , y ( j ) i ) i ∈ A j are independent and distributed as X ( j ) i ∼ N (0 , Σ ( j ) ) and y ( j ) i = ⟨ X ( j ) i , ( v ∗ , -1) ⟩ + ξ i . Let λ ( j ) 1 ≤ · · · ≤ λ ( j ) n +1 be the eigenvalues of Σ ( j ) .

Now we apply Theorem C.2 with covariance Σ ( j ) , samples ( X ( j ) i , y ( j ) i ) i ∈ A j , sparsity t +1 , outlier counts d l +1 and d h +1 , and failure probability δ/L ; let n ( j ) eff and r ( j ) eff be the induced parameters defined in that theorem statement, and let c, C be the constants. By the Cauchy Interlacing Theorem, we have λ ( j ) d l +2 ≥ λ d l +1 and similarly λ ( j ) n +1 -( d h +1) ≤ λ n -d h . Thus r ( j ) eff ≤ 2 r eff. Also n ( j ) eff ≤ n eff .

Thus, if the constant C 0 is chosen appropriately large, then m/L ≥ 16 cr ( j ) eff and also m/L ≥ Cn ( j ) eff . Hence (by the error guarantee of Theorem C.2) with probability at least 1 -δ/L we obtain a vector ˆ w ( j +1) such that

where the second inequality uses AM-GM to bound the third term, and the third inequality is by choosing c 0 ≥ 4 c +8 c 2 .

But now define ˆ v ( j +1) := ˆ w ( j +1) [ n ] + ˆ w ( j +1) n +1 ˆ s ( j ) . Then we observe that ∥ ( v ∗ , -1) ∥ 2 Σ ( j ) = ∥ ∥ v ∗ -ˆ s ( j ) ∥ ∥ 2 Σ and ∥ ∥ ˆ w ( j +1) -( v ∗ , -1) ∥ ∥ 2 Σ ( j ) = ∥ ∥ ˆ v ( j +1) -( v ∗ -ˆ s ( j ) ) ∥ ∥ 2 Σ = ∥ ∥ v ∗ -ˆ s ( j +1) ∥ ∥ 2 Σ where ˆ s ( j +1) = ˆ v (1) + · · · + ˆ v ( j +1) . So (5) is equivalent to

Inductively, we conclude that

as desired. The time complexity (see Algorithm 2 for full pseudocode) is dominated by L eigendecompositions of n × n Hermitian matrices (each of which takes time O ( n 3 ) by e.g. the QR algorithm), as well as L convex optimizations (each of which takes time ˜ O ( n 3 ) to solve to inversepolynomial accuracy [26], which is sufficient for the correctness proof).

C.1 An alternative algorithm (proof of Theorem 1.2)

In this section we prove Theorem 1.2, which essentially states that the sample complexity dependence on d l in BOAR-Lasso() can be removed at the cost of a time complexity depending on d t l . See Algorithm 3 for the pseudocode of how we modify AdaptivelyRegularizedLasso() : essentially, we brute force search over all sizet subsets of the set S produced by IterativePeeling() , construct an appropriate dictionary for each of these ( | S | t ) subsets, and then perform a final model selection step (with fresh samples) to pick the best dictionary/estimator. The boosting step is exactly identical to that in BOAR-Lasso() .

Lemma C.4. Let n, t, d ∈ N . Let Σ : n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ · · · ≤ λ n . Then there is a family D ⊆ R n × ( n + t ) of size |D| ≤ (7 t ) 2 t 2 + t (2 d ) t , consisting entirely of n × ( n + t ) matrices with the form

with the following property. For any t -sparse v ∈ R n , there is some D ∈ D and w ∈ R n + k with v = Dw and

Proof. Let u 1 , . . . , u n ∈ R n be the eigenvectors of Σ corresponding to eigenvalues λ 1 , . . . , λ n respectively, so that Σ = ∑ n i =1 λ i u i u ⊤ i . Define Σ := λ -1 d +1 ∑ n i =1 min( λ i , λ d +1 ) u i u ⊤ i . Let S be the output of IterativePeeling( Σ , d l , t ) , and let D := { D ( T ) : T ∈ ( S t ) } , where for any T ∈ ( S t ) , we let { d 1 , . . . , d t } be a Σ -orthonormal basis for span { e i : i ∈ T } , and let D ( T ) be the n × ( n + t ) matrix with columns e 1 , . . . , e n , d 1 , . . . , d t . The bound on |D| follows from Theorem B.1.

For any t -sparse v ∈ R n , pick the matrix D ∈ D indexed by any T ∈ ( S t ) with S ∩ supp( v ) ⊆ T . Let d 1 , . . . , d t ∈ R n be the last t columns of D . Then there are coefficients b 1 , . . . , b t so that we can write v S = ∑ t i =1 b i d i . Since d ⊤ i Σ d i ′ = ✶ [ i = i ′ ] for all i, i ′ ∈ [ t ] , we have v ⊤ S Σ v S = ∑ t i =1 b 2 i . Hence, ∥ b ∥ 1 ≤ √ t √ v ⊤ S Σ v S . But we can bound

(by triangle inequality)

Algorithm 3: Alternative algorithm to solve sparse linear regression when covariate eigenspectrum has few outliers

Data: Covariance matrix Σ : n × n , samples ( X i , y i ) i m =1 , sparsity t , small eigenvalue count d l , failure probability δ

Result: Estimate ˆ v of unknown sparse regressor, satisfying Theorem C.5

∑

S

n

i

⊤

=1

λ

i

u

i

u

i

eigendecomposition of

IterativePeeling(

Σ

, d

l

, t

)

Select best hypothesis return D ( ˆ T ) ˆ w ( ˆ T )

We conclude that ∥ b ∥ 1 ≤ 4 √ tλ -1 / 2 d +1 √ v ⊤ Σ v . Thus, if we define

where here e 1 , . . . , e n + k refer to the standard basis vectors in R n + t , then we have Dw = v [ n ] \ S + ∑ t i =1 b i d i = v , and also

as desired.

Theorem C.5. Let n, t, d l , d h , m ∈ N and let Σ : n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ · · · ≤ λ n . Let ( X i , y i ) i m =1 be independent samples where X i ∼ N (0 , Σ) and y i = ⟨ X i , v ∗ ⟩ + ξ i , for ξ i ∼ N (0 , σ 2 ) and a fixed t -sparse vector v ∗ ∈ R n . Set k := t (7 t ) 2 t 2 + t d t l and let D be the family of matrices (of size at most k ) guaranteed by Lemma C.4.

Let δ > 0 . For every D ∈ D , define

Σ

/*

See

Algorithm

1

*/

where X (1) : ( m/ 2) × n is the matrix with rows X 1 , . . . , X m/ 2 , and define ˆ v = ˆ D ˆ w ( ˆ D ) where

where X (2) : ( m/ 2) × n is the matrix with rows X m/ 2+1 , . . . , X m .

Let n eff := t 2 log( t ) + t log( d l ) + d h +log(48 /δ ) and let r eff := t ( λ n -d h /λ d l +1 ) log(8 n/δ ) . There are absolute constants c, C > 0 so that the following holds. If m ≥ Cn eff, then with probability at least 1 -3 δ it holds that

Let D ∗ ∈ D and w ∗ ∈ R n + t be the matrix and vector guaranteed by Lemma C.4 for the t -sparse vector v ∗ . Let Γ = ( D ∗ ) ⊤ Σ D ∗ with eigenvalues γ 1 ≤ · · · ≤ γ n + t . We make the following claim:

Claim C.6. With probability at least 1 -δ/ 4 over G ∼ N (0 , Γ) , it holds uniformly in w ∈ R n + t that

Proof. Since Σ is a principal submatrix of Γ , we have γ n -d h ≤ λ n -d h (by the Cauchy Interlacing Theorem). Suppose that Γ has eigendecomposition Γ = ∑ n + t i =1 γ i g i g ⊤ i , and define projection matrix P : ( n + t ) × ( n + t ) by P := ∑ n -d h i =1 g i g ⊤ i , so that rank( P ⊥ ) = d h + t and λ max ( P Γ P ) ≤ γ n -d h ≤ λ n -d h . Then for any w ∈ R n + t and G ∼ N (0 , Γ) , we can bound

where Z ∼ N (0 , P ⊥ ) . The second equality above uses that Γ -1 / 2 and P ⊥ are simultaneously diagonalizable (and therefore commute). But now for any δ > 0 , we have the Gaussian tail bounds and

Thus, with probability at least 1 -δ/ 4 over G ∼ N (0 , Γ) , for any w ∈ R n + t , we have

which proves the claim.

We now proceed with proving the theorem.

Proof of Theorem C.5. Applying Claim C.6, we can now invoke Lemma F.7 with covariance matrix Γ , seminorm Φ( v ) := 2 √ 2 λ n -d h · log(8 n/δ ) ∥ v ∥ 1 , p := 2( d h + t ) , ground truth w ∗ , samples (( D ∗ ) ⊤ X i , y i ) m/ 2 i =1 , and failure probability δ/ 4 . Since we chose m sufficiently large that m/ 2 ≥ 16 p + 196log(12 /δ ) , we conclude that with probability at least 1 -2 δ over the randomness of ( X i , y i ) m/ 2 i =1 , it holds that

Since v ∗ = D ∗ w ∗ and ∥ w ∗ ∥ 1 ≤ 7 t 1 / 2 λ -1 / 2 d l +1 ∥ v ∗ ∥ Σ (the guarantees of Lemma C.4), it follows that

To complete the proof of the theorem, condition on any values of ( X i , y i ) m/ 2 i =1 for which the above bound holds. By applying Lemma F.2 with covariance matrix Σ , hypothesis set W := { D ˆ w ( D ) : D ∈ D} , and samples ( X i , y i ) m i = m/ 2+1 (which are independent of W ), since m/ 2 ≥ 32 log(2 |D| /δ ) , we have with probability at least 1 -2 δ over the samples ( X i , y i ) m i = m/ 2+1 that

Hence, with probability at least 1 -5 δ we have

which proves the theorem.

■

We can use the above theorem (together with the previously discussed boosting approach) to get the following result, which proves Theorem 1.2.

Theorem C.7. Let n, t, d l , d h , m, L ∈ N and σ, δ > 0 . Let Σ : n × n be a positive semi-definite matrix with eigenvalues λ 1 ≤ ·· · ≤ λ n . Let ( X i , y i ) i m =1 be independent samples where X i ∼ N (0 , Σ) and y i = ⟨ X i , v ∗ ⟩ + ξ i , for ξ i ∼ N (0 , σ 2 ) and a fixed t -sparse vector v ∗ ∈ R n .

Then, given Σ , ( X i , y i ) i m =1 , t , d l , and δ , there is an estimator ˆ v with the following properties.

Let n ′ eff := t 2 log( t ) + t log( d l ) + d h + log(48 L/δ ) and let r ′ eff := t ( λ n -d h /λ d l +1 ) log(8 nL/δ ) . There are absolute constants c 0 , C 0 > 0 such that the following holds. If m ≥ C 0 L ( n ′ eff + r ′ eff ) , then with probability at least 1 -δ , it holds that

Moreover, ˆ v is computable in time ( t +1) O ( t 2 ) ( d l +1) t +1 · poly( n ) .

Proof. Identical to that of Theorem C.3, except using Theorem C.5 instead of Theorem C.2.

D Faster sparse linear regression for arbitrary Σ

In this section we prove Theorem 3.1. The approach is via feature adaptation: in Theorem D.5, we show that any covariance matrix Σ has a ( t, O ( t 3 / 2 log n ) -ℓ 1 -representation of size O ( n t -1 / 2 ) that is computable in time n t -Ω(1 /t ) log O ( t ) n , using O ( t log n ) samples from N (0 , Σ) . The algorithm for computing this representation is described in Algorithm 4. One of the key tools is the following result from computational geometry:

Theorem D.1 ([30]) . Let n, d, k ∈ N and δ > 0 . Given points p 1 , . . . , p n ∈ R d , query dimension k , and failure probability δ , there an algorithm DS (( p 1 , . . . , p n ) , k, δ ) with time complexity n k +1 (log n ) O ( k ) poly( d ) log(1 /δ ) , that constructs a data structure N that answers queries of the following form. Given a k -dimensional subspace F ⊆ R d , the output N ( F ) is some i ∗ ∈ [ n ] . With probability at least 1 -δ , the query time complexity is n 1 -1 / (2 k ) poly( d ) log(1 /δ ) , and it holds that

How do we use the above theorem to efficiently construct the ℓ 1 -representation? The intuition is as follows. Let X be the m × n matrix where each row is a sample from N (0 , Σ) . Then each column is a vector p i representing a particular covariate. To find the ℓ 1 -representation, it essentially suffices

Algorithm 4: ℓ -representation for arbitrary Σ

̸

to find a dictionary D of O ( n t -1 / 2 ) sparse combinations of { p 1 , . . . , p n } so that every t -sparse combination of { p 1 , . . . , p n } can be written in terms of the chosen combinations, with a coefficient vector that has bounded ℓ 1 norm.

For notational ease, we define C ( x ) to be the 'cost' of a particular linear combination x ∈ R n with respect to the set D of chosen combinations:

Definition D.2. For a subset D ⊆ R n , define C D : R n → [0 , ∞ ] by

̸

With this notation, we want to construct a set D of size O ( n t -1 / 2 ) , consisting of t -sparse vectors, such that for all t -sparse x ∈ R n .

The construction is quite simple: divide the set { p 1 , . . . , p n } into √ n equal-sized groups. For each set T of t -1 vectors and each of the √ n groups, find the closest vector in the group to the subspace spanned by T (using Theorem D.1 to achieve sublinear time complexity). Then add the difference between the vector and its projection (onto the subspace) to the dictionary. Finally, for each set of t vectors where two of the vectors lie in the same group, add an orthonormal basis for those vectors to the dictionary. See the procedure RepresentVectors() in Algorithm 5 for pseudocode.

By construction, the dictionary clearly has size O ( n t -1 / 2 ) . At a high level, the reason it satisfies the representational property is the following. Consider some t -sparse combination, such as p 1 + · · · + p t . If -p t is not very close to p 1 + · · · + p t -1 , then we can bound C ( p 1 + · · · + p t ) by C ( p 1 + · · · + p t -1 ) and C ( p t ) , which are O ( √ t ∥ p 1 + · · · + p t -1 ∥ 2 ) and O ( √ t ∥ p t ∥ 2 ) respectively, since the dictionary contains an orthonormal basis for both terms. The only case where these bounds are not good enough is when ∥ p 1 + · · · + p t ∥ 2 is much smaller than ∥ p 1 + · · · + p t -1 ∥ 2 and ∥ p t ∥ 2 . In this case, p t is very close to span { p 1 , . . . , p t -1 } . However, in the construction we found some (potentially different) p j which is just as close to span { p 1 , . . . , p t -1 } , and moreover is in the same group as p t . Letting q be the projection of p j onto span { p 1 , . . . , p t -1 } , we have the crucial fact that ∥ p j -q ∥ 2 is as small as ∥ p 1 + · · · + p t ∥ 2 .

Now, bounding C ( p 1 + · · · + p t ) proceeds as follows. We can subtract some appropriate (bounded) multiple of p j -q from p 1 + · · · + p t to zero out at least one of the coefficients. This residual then is a t -sparse combination of { p 1 , . . . , p t , p j } where two of the vectors { p t , p j } are in the same group; thus it has small cost with respect to D . Moreover, p j -q is contained in D and thus has small cost (specifically, not much more than ∥ p j -q ∥ 2 , which crucially is not much more than ∥ p 1 + · · · + p t ∥ 2 ). It follows that p 1 + · · · + p t has small cost.

Formalizing this argument, we start by proving one of the facts that we freely used above: that the cost function C satisfies the triangle inequality.

Fact D.3. For any D ⊆ R n and x, y ∈ R n , it holds that C ( x + y ) ≤ C ( x ) + C ( y ) .

Proof. For any α, β ∈ R D with ∑ d α d d = x and ∑ d β d d = y , the vector α + β satisfies ∑ d ( α + β ) d d = x + y . Applying the triangle inequality to ∑ d | ( α + β ) d | · ∥ ∑ i d i p i ∥ 2 completes the proof.

We now prove the key lemma, formalizing the above intuition.

Lemma D.4. Let n, m, t ∈ N , with t ≥ 2 , and δ > 0 . Fix p 1 , . . . , p n ∈ R m with ∥ p i ∥ 2 = 1 for all i ∈ [ n ] . Let D be the output of RepresentVectors( { p 1 , . . . , p n } , t, δ ) . Then |D| = O ( n t -1 / 2 ) , and every element of D is t -sparse. Also, with probability at least 1 -δ , the following guarantees hold. The time complexity of computing D is O ( n t -Ω(1 /t ) (log n ) O ( t ) m O (1) log(1 /δ )) . Moreover, for every t -sparse x ∈ R n it holds that

Proof. Since the algorithm RepresentVectors() makes less than n t queries to the data structures N j , and each query has failure probability at most δ ′ = δ/n t , the probability that any query fails is at most 1 -δ . We henceforth assume that all queries succeed, i.e. satisfy the correctness guarantee and time complexity bound stated in Theorem D.1.

Time complexity. We start by analyzing the time complexity of RepresentVectors( { p 1 , . . . , p n } , t, δ ) . For any fixed j ∈ [ √ n ] , the construction time of N j (with | I j | = O ( √ n ) points in R m , query dimension t -1 , and failure probability δ/n t ) is O ( n t/ 2 (log n ) O ( t ) m O (1) log(1 /δ )) . We make ( n t -1 ) + | I j | 2 ( n t -2 ) = O ( n t -1 ) queries to N j , each with time complexity n 1 / 2 -1 / (4( t -1)) m O (1) log(1 /δ ) . Each projection step and each orthonormalization step has time complexity poly( t, m ) . Thus, since t ≥ 2 , the time complexity for any

fixed j is bounded by n t -1 / 2 -1 / (8 t ) (log n ) O ( t ) m O (1) log(1 /δ ) . Summing over j , the overall time complexity to compute D is at most n t -1 / (8 t ) (log n ) O ( t ) m O (1) log(1 /δ ) as claimed.

̸

Correctness. The bound on |D| and the fact that all elements of D are t -sparse are immediate from the algorithm definition. It remains to bound C D ( x ) for t -sparse vectors x . First, note that for any ( t -1) -sparse y ∈ R n , because of step (4), the dictionary contains vectors γ 1 , . . . , γ t -1 that span supp( y ) and satisfy ⟨ ∑ n i =1 γ k i p i , ∑ n i =1 γ ℓ i p i ⟩ = 0 for all k = ℓ . Thus, letting α 1 , . . . , α t -1 ∈ R be such that y = α 1 γ 1 + · · · + α t -1 γ t -1 , we get

Now fix any nonzero t -sparse x ∈ R n . Fix any a ∈ arg max i ∈ [ n ] | x i | , and let j ∈ [ √ n ] be such that a ∈ I j . Let T = supp( x ) \ { a } . Let q := Proj span { p i : i ∈ T } p h ( T,j ) . Then by the correctness guarantee of N j on query span { p i : i ∈ T } ,

̸

Case I. Suppose that ∥ ∥ p h ( T,j ) -q ∥ ∥ 2 ≥ 1 / 2 . Then by (8), we have | x a | ≤ O (log n ) · ∥ ∑ i x i p i ∥ 2 . Thus, by the triangle inequality,

̸

It follows from Fact D.3 and (7) that

as desired.

Case II. It remains to consider the case that ∥ ∥ p h ( T,j ) -q ∥ ∥ 2 ≤ 1 / 2 . In this case we have ∥ q ∥ 2 ≥ ∥ ∥ p h ( T,j ) ∥ ∥ 2 -1 / 2 ≥ 1 / 2 . By step (3) of the algorithm, the dictionary contains some vector γ -e h ( T,j ) such that supp( γ ) ⊆ T and q = ∑ i ∈ T γ i p i . Fix any b ∈ arg max i | γ i | . Since q = ∑ γ i p i we get | γ b | ≥ ∥ q ∥ 2 t ≥ 1 / (2 t ) . Now, by Fact D.3,

By construction, e h ( T,j ) -γ is an element of the dictionary, so we can bound the first term as

where the equality uses that q = ∑ n i =1 γ i p i , the second inequality uses that | x b | ≤ | x a | and | γ b | ≥ 1 / (2 t ) , and the final inequality uses (8).

̸

Finally, observe that

̸

since the coefficients on e b cancel out. Thus, z is a linear combination of two elements of { p i : i ∈ I j } together with t -2 elements of { p i : i ∈ [ n ] } . Because of step (4) of the algorithm, the dictionary contains vectors γ 1 , . . . , γ t that span supp( z ) and satisfy ⟨ ∑ n i =1 γ k i p i , ∑ n i =1 γ ℓ i p i ⟩ = 0 for all k = ℓ . The same argument as for (7) gives that

where the second inequality uses the triangle inequality and (8), and the final inequality uses that | x b | ≤ | x a | and | γ b | ≥ 1 / (2 t ) . Putting everything together, we conclude that

as claimed.

We now show that RepresentVectors() can be applied to the columns of the sample matrix to obtain a ℓ 1 -representation for Σ (procedure ComputeL1Representation() in Algorithm 5). Up to an appropriate rescaling of the covariates, Lemma D.4 immediately implies that D gives a ℓ 1 -representation for the empirical covariance ˆ Σ . The main result then follows from concentration of ˆ Σ and sparsity of the elements of the dictionary.

Theorem D.5. Let n, m, t ∈ N and let Σ : n × n be a positive-definite matrix. Suppose m ≥ Ct log n for a sufficiently large constant C . Let X 1 , . . . , X m ∼ N (0 , Σ) be independent samples, and let D be the output of ComputeL1Representation( { X 1 , . . . , X m } , t ) . Then |D| ≤ O ( n t -1 / 2 ) , and every element of D is t -sparse. Also, with probability at least 1 -e -Ω( m ) , the time complexity of the algorithm is O ( n t -Ω(1 /t ) (log n ) O ( t ) m O (1) ) , and D is a ( t, C l1rep t 3 / 2 log n ) -ℓ 1 -representation for Σ , for some universal constant C l1rep .

Proof. Let ˆ Σ = 1 m X ⊤ X . Let ˜ D denote the intermediary dictionary constructed by the algorithm using RepresentVectors() . With probability at least 1 -e -m , the successful event of Lemma D.4 holds. By standard concentration bounds (see e.g. Exercise 4.7.3 in [41]), it holds that 1 2 ∥ x ∥ Σ ≤ ∥ x ∥ ˆ Σ ≤ 2 ∥ x ∥ Σ for all t -sparse x ∈ R n , with probability at least 1 -e -Ω( m ) . Henceforth assume that both of these events hold.

Time complexity. The time complexity of the algorithm is dominated by the call to RepresentVectors() . By the guarantee of Lemma D.4, this takes time O ( n t -Ω(1 /t ) (log n ) O ( t ) m O (1) ) .

Correctness. The bounds on |D| and sparsity of elements of D follow from identical bounds for ˜ D (see Lemma D.4), and the fact that every element of D is obtained by rescaling the coordinates of some element of ˜ D . It remains to show that D is a ( t, O ( t 3 / 2 log n )) -ℓ 1 representation for Σ .

Fix any t -sparse v ∈ R n , and define ˜ v = ˆ Dv . By the guarantee of Lemma D.4, since ˜ v is also t -sparse, there is some α ∈ R ˜ D such that ˜ v = ∑ ˜ d ∈ ˜ D α ˜ d ˜ d and

But note that ˜ v i = ˆ D ii v i = ∥ q i ∥ 2 v i for all i . Similarly, every ˜ d ∈ ˜ D corresponds to some d ∈ D with ˜ d i = ∥ q i ∥ 2 d i for all i . Thus, reindexing α according to D in the natural way, we have that v = ∑ d ∈D α d d and

But now let ˆ Σ = 1 m X ⊤ X . For any i, j ∈ [ n ] we have ⟨ q i , q j ⟩ = m ˆ Σ ii . Hence,

and similarly for ∥ ∑ n i =1 d i q i ∥ 2 2 . Thus, we get

But as shown above, we know that 1 2 ∥ x ∥ Σ ≤ ∥ x ∥ ˆ Σ ≤ 2 ∥ x ∥ Σ for all t -sparse x ∈ R n . Since v and all d ∈ D are t -sparse, we conclude that

as desired.

We finally restate and prove Theorem 3.1, as a corollary of Theorem D.5 and the well-known fact that standard 'slow rate' guarantees for Lasso (i.e. based on the ℓ 1 norm of the regressor) can be achieved in near-linear time (Theorem A.8). The pseudocode for the main algorithm is given in Algorithm 5.

Corollary D.6. Let n, m, t, B ∈ N and σ > 0 , and let Σ : n × n be a positive-definite matrix. Let w ∗ ∈ R n be t -sparse with ∥ w ∗ ∥ Σ ≤ B . Suppose m ≥ Ct log n for a sufficiently large constant C . Let ( X i , y i ) i m =1 be independent samples where X i ∼ N (0 , Σ) and y i = ⟨ X i , w ∗ ⟩ + N (0 , σ 2 ) . Then there is an O ( m 2 n t -1 / 2 + n t -Ω(1 /t ) log O ( t ) n ) -time algorithm (Algorithm 5) that, given ( X i , y i ) i m =1 , t , B , σ 2 , produces an estimate ˆ w ∈ R n satisfying, with probability 1 -o (1) ,

Proof. By Theorem D.5 it holds with probability 1 -n -100 t that D is a ( t, C l1rep t 3 / 2 log n ) -ℓ 1 -representation for Σ . Also, by standard concentration bounds (e.g. Exercise 4.7.3 in [41]), we have

1 2 ∥ x ∥ Σ ≤ ∥ x ∥ ˆ Σ ≤ 2 ∥ x ∥ Σ for all t -sparse x ∈ R n (where ˆ Σ = 2 m ∑ m/ 2 i =1 X i X ⊤ i ) with probability at least 1 -exp( -Ω( m )) . Suppose that both of these events occur.

For each of the remaining m/ 2 samples X j , compute ˜ X j ∈ R D where the entry ˜ X j,d corresponding to d ∈ D is ⟨ X j , d/ ∥ d ∥ ˆ Σ ⟩ (where ˆ Σ = 2 m ∑ m/ 2 i =1 X i X ⊤ i is not explicitly computed; since d is sparse, both ⟨ X j , d ⟩ and ∥ d ∥ ˆ Σ can be computed in poly( t, m ) time). Let N (0 , Γ) denote the distribution of each ˜ X j . For each d ∈ D , since d is t -sparse, we have that E x ∼ N (0 , Σ) ⟨ x, d/ ∥ d ∥ ˆ Σ ⟩ 2 = ∥ d ∥ 2 Σ / ∥ d ∥ 2 ˆ Σ ≤ 4 . Thus, Γ dd ≤ 4 for all d .

Moreover, since w ∗ is t -sparse, there is some α ∈ R D with w ∗ = ∑ d α d d and ∑ d | α d | ∥ d ∥ Σ ≤ C l1rep t 3 / 2 log( n ) · ∥ w ∗ ∥ Σ . Define β ∈ R d by β d = α d ∥ d ∥ ˆ Σ . Then w ∗ = ∑ d β d d/ ∥ d ∥ ˆ Σ and

But now for any of the remaining m/ 2 samples, we have that

and thus y -⟨ ˜ X j , β ⟩ ∼ N (0 , σ 2 ) . So we can apply Theorem A.8 to samples ( ˜ X j , y j ) m j = m/ 2+1 to compute an estimator ˆ β satisfying

using that ∥ β ∥ 1 ≤ 2 C l1rep t 3 / 2 log( n ) · ∥ w ∗ ∥ Σ ≤ 2 C l1rep t 3 / 2 B log( n ) , and using the bound max d Γ dd ≤ 4 . The time complexity of this step is ˜ O ( |D| m 2 ) = ˜ O ( m 2 n t -1 / 2 ) . Finally, compute ˆ w := ∑ d ˆ β d d/ ∥ d ∥ ˆ Σ . We have that ∥ ˆ w -w ∗ ∥ Σ = ∥ ∥ ∥ ˆ β -β ∥ ∥ ∥ Γ , which completes the proof.

E Fixed-parameter tractability in κ and t

In this section we prove Theorem 3.2, which shows we can achieve upper bounds on N t,α (Σ) for α independent of κ and n , if we are willing to incur dependence on κ in the resulting bound. In fact, we actually prove an upper bound on the packing number P t,α (Σ) .

̸

To achieve this, the first key idea is to consider the dual certificates for a packing. Suppose that v 1 , . . . , v N are unit vectors (in the Σ -norm) with |⟨ v i , v j ⟩ Σ | ≤ α for all i = j . Then |⟨ v i , Σ v i ⟩| ≥ α -1 max j = i |⟨ v j , Σ v i ⟩| , so Σ v i certifies that any linear combination v i = ∑ j = i x j v j must have the property that ∥ x ∥ 1 ≥ α -1 . Thus, to show that there cannot be a large packing of sparse vectors in the Σ -norm, it would suffice to prove that any large set of sparse vectors must have one vector that can be written as a linear combination of the remaining vectors, where the coefficient vector has small ℓ 1 norm. In fact, this would give an upper bound on N t,α (Σ) for all Σ .

̸

̸

We do not know if such a statement is true. However, we can prove an approximate analogue. The following lemma shows that under a condition number bound on Σ , the dual certificate argument can be generalized to require only a weaker property: that any large set of sparse vectors must have one vector that can be approximately written as a linear combination of the remaining vectors, with low ℓ 1 cost. The approximation error determines how small the condition number must be:

Lemma E.1. Let n, N, t, T ∈ N and let δ > 0 . Suppose that for all t -sparse vectors v 1 , . . . , v N ∈ R n , there exists some i ∈ [ N ] and x ∈ R N such that ∥ x ∥ 1 ≤ T and

̸

Then for every positive-definite matrix Σ : n × n with κ (Σ) < 1 / (4 δ 2 ) it holds that P t, 1 / (3 T ) (Σ) ≤ N log 2 κ (Σ) .

Proof. Fix a positive-definite matrix Σ : n × n and suppose that K := P t, 1 / (3 T ) (Σ) > N log 2 κ (Σ) . By definition, there are nonzero t -sparse vectors v 1 , . . . , v K ∈ R N such that

for all i = j . Without loss of generality, assume that ∥ v i ∥ 2 = 1 for all i ∈ [ K ] , so that

So we can partition [ K ] into log 2 κ (Σ) buckets such that max i ∈ B ∥ v i ∥ 2 Σ / min i ∈ B ∥ v i ∥ 2 Σ ≤ 2 for each bucket B ⊆ [ K ] . There must be some bucket B with | B | ≥ N . By assumption, there is some i ∈ B and x ∈ R N such that ∥ x ∥ 1 ≤ T and

̸

̸

̸

Now

̸

Simplifying, we get ∥ v i ∥ Σ ≤ 2 δ √ λ max (Σ) . Since also ∥ v i ∥ Σ ≥ √ λ min (Σ) , it follows that κ (Σ) = λ max (Σ) /λ min (Σ) ≥ 1 / (4 δ 2 ) .

It remains to show that the precondition of Lemma E.1 can be satisfied for sub-constant δ without requiring N to scale with n t . We start by proving the desired property when the vectors are all t -sparse and binary , i.e. v 1 , . . . , v N ∈ { 0 , 1 } n , and afterwards we will black-box extend the result to the real-valued setting. Concretely, given sparse binary vectors v 1 , . . . , v N ∈ { 0 , 1 } n (with N ≫ n ), we want to find one that can be 'efficiently' approximated (in ℓ 2 norm) by the rest, where 'efficient' means that the coefficients have small absolute sum. Thinking of each vector as the indicator vector of a subset of [ n ] , a first step towards an efficient approximation for v i = ✶ [ · ∈ S i ] may be constructing an efficient approximation for a standard basis vector e j for some j ∈ S i .

For notational convenience, for vectors x, y ∈ { 0 , 1 } n we say that x ⪯ y if x i ≤ y i for all i ∈ [ n ] .

Indeed, there is some j ∈ [ n ] such that S j := { i : v ij = 1 } is large, i.e. |S j | ≥ N/n . If the vectors ( v i ) i ∈S j were in some sense random, then the average 1 |S j | ∑ i ∈S j v i would be a good approximation for e j . It is also efficient, in that the absolute sum of coefficients is 1 . But of course the vectors are not random; it could be that many vectors in S j also contain some other coordinate j ′ . In this case we restrict to the set of vectors containing both j and j ′ . Now we may hope to approximate the vector ✶ [ · ∈ { j, j ′ } ] . Completing this argument, we get the following lemma which states that there exists a subset of [ n ] that is contained in many of the vectors, and that is well-approximated by the average of those vectors.

Lemma E.2. Let n, N, t, s ∈ N with sn ≤ N , and let v 1 , . . . , v N ∈ { 0 , 1 } n be nonzero t -sparse binary vectors. Then there is some set S ⊆ [ N ] of size | S | ≥ s and some nonzero vector u ∈ { 0 , 1 } n such that u ⪯ v i for all i ∈ S , and

̸

̸

̸

̸

Proof. For each J ⊆ [ n ] , define S J := { i ∈ [ N ] : v ij = 1 ∀ j ∈ J } . Since all v i are nonzero, there is some j ∗ ∈ [ n ] with |S { j ∗ } | ≥ N/n . We iteratively construct a set J ⊆ [ n ] as follows. Initially, set J = { j ∗ } . While there exists some a ∈ [ n ] \ J such that |S J ∪{ a } | > ( sn/N ) 1 /t |S J | , update J to J ∪ { a } (if there are multiple such a , pick any one of them arbitrarily). At termination of this process, we have |S J | > 0 . Since every v i is t -sparse, it must be that | J | ≤ t . Thus, |S J | ≥ ( N/n ) · ( sn/N ) ( t -1) /t ≥ s . Set S := S J and u := ✶ J ∈ { 0 , 1 } n . By definition of S J , we have that u ⪯ v i for all i ∈ S . For any j ∈ J , we have u j = 1 = 1 | S | ∑ i ∈ S v ij . For any j ̸∈ J , we have u j = 0 and

by construction of J . Thus,

Additionally,

By the inequality ∥ x ∥ 2 2 ≤ ∥ x ∥ 1 ∥ x ∥ ∞ , we conclude that

as claimed.

We now use Lemma E.2 to show that if N is sufficiently large, then at least one of the vectors v i can be efficiently approximated by the rest. The proof is by induction on t . As a first attempt, one might use Lemma E.2 to find some u ∈ { 0 , 1 } n and some large set S ⊆ [ N ] such that u ⪯ v i for all i ∈ S , and the average of the v i 's approximates u . Then, restrict to the vectors in S , and induct on the ( t -1) -sparse residual vectors { v i -u : i ∈ S } . If one of the v i -u 's can be efficiently approximated by the other residuals, then since u can also be efficiently approximated, we can derive an efficient approximation of v i by the remaining v j 's.

This doesn't quite work, since at each step of the induction the set of vectors will become smaller by a factor of roughly n . However, instead of throwing away the vectors outside S =: S (1) we can iteratively re-apply Lemma E.2 to get disjoint sets S (1) , S (2) , . . . , S ( m ) , where each S ( a ) has the same property as S (for some potentially different vector u ( a ) ). We can then induct on the residual vectors ∪ a { v i -u ( a ) : i ∈ S ( a ) } . This suffices to efficiently approximate some v i . Since we throw away fewer vectors at each step of the induction, we do not need the initial number of vectors N to be as large.

We formalize the above ideas in the following theorem.

Theorem E.3. Let n, N, t ∈ N and let v 1 , . . . , v N ∈ { 0 , 1 } n be t -sparse binary vectors. Then there is some i ∈ [ N ] and x ∈ R N such that ∥ x ∥ 1 ≤ 3 t and

̸

Proof. We induct on t , observing that the case t = 0 is immediate. Fix t > 0 and t -sparse vectors { v 1 , . . . , v N } ∈ { 0 , 1 } n , and suppose that the theorem statement holds for t -1 . If any v i is identically zero, then the claim is trivially true with x = 0 . If N ≤ t 3 t +1 n then the RHS of the desired norm bound exceeds 4 t √ t , so the claim is trivially true with x = 0 and any i ∈ [ N ] . Thus, we may assume that all v i are nonzero, and N ≥ t 3 t +1 n . Applying the previous lemma with

s := 3 t +1 ≤ N/n gives some S (1) ⊆ [ N ] and nonzero u (1) ∈ { 0 , 1 } n such that | S (1) | ≥ 3 t +1 and u (1) ⪯ v i for all i ∈ S (1) , and

If | N | - | S (1) | ≥ N/t ≥ 3 t +1 n then we can reapply the lemma with vectors ( v i ) i ∈ [ N ] \ S (1) and s := 3 t +1 to get some S (2) ⊆ [ N ] \ S (1) and u (2) ∈ { 0 , 1 } n . Continuing this process so long as there are at least N/t ≥ 3 t +1 n remaining vectors, we can generate disjoint sets S (1) , . . . , S ( m ) ⊆ [ N ] and vectors u (1) , . . . , u ( m ) ∈ { 0 , 1 } n with the following properties:

(i) | S (1) ∪ · · · ∪ S ( m ) | > N -N/t

(ii) | S ( a ) | ≥ 3 t +1 for every a ∈ [ m ]

(iii) For every a ∈ [ m ] , it holds that u ( a ) is nonzero and u ( a ) ⪯ v i for all i ∈ S ( a )

(iv) For every a ∈ [ m ] ,

For each a ∈ [ m ] and i ∈ S ( a ) , define v ′ i := v i -u ( a ) . By Property (iii) we have that v ′ i ∈ { 0 , 1 } N and v ′ i is ( t -1) -sparse. By the inductive hypothesis applied to vectors ( v ′ i ) i ∈ S (1) ∪···∪ S ( m ) , there is some i ∈ S (1) ∪ · · · ∪ S ( m ) and x ′ ∈ R N (supported on S (1) ∪ · · · ∪ S ( m ) ) such that ∥ x ′ ∥ 1 ≤ 3 t -1 and

̸

where the last inequality uses Property (i) and the bound N ≥ tn . Of course, without loss of generality x ′ i = 0 . Let a ∈ [ m ] be the unique index such that i ∈ S ( a ) . We define x ∈ { 0 , 1 } N (supported on S (1) ∪ · · · ∪ S ( m ) ) as follows. For each b ∈ [ m ] and each r ∈ S ( b ) , set

Since ∥ x ′ ∥ 1 ≤ 3 t -1 , we can see that

Next, we use x to approximate v i . The following bound is almost what we want:

Proof of claim. We have

where the first and third inequalities use that v r = v ′ r + u ( b ) for all r ∈ S ( b ) , and throughout we use that x r = x ′ r = 0 for r ̸∈ S (1) ∪ · · · ∪ S ( m ) . Applying Property (iv) , equation (9), and the bound ∥ x ′ ∥ 1 ≤ 3 t -1 , we get

as claimed.

However, we wanted a bound on v i -∑ j = i x j v j , and unfortunately x i = 0 . Fortunately, it is enough that x i is bounded away from 1 . Since x ′ i = 0 , we have

̸

̸

̸

Finally, we have ∥ x/ (1 -x i ) ∥ 1 ≤ (9 / 8)(2 · 3 t -1 +1) ≤ 3 t , so x/ (1 -x i ) satisfies all the desired conditions. This completes the induction.

Finally, we extend Theorem E.3 to real-valued sparse vectors via a discretization argument.

Lemma E.5. Let n, N, t ∈ N and let v 1 , . . . , v N ∈ R n be t -sparse vectors. Then there is some i ∈ [ N ] and x ∈ R n such that ∥ x ∥ 1 ≤ 3 t and

̸

Proof. Without loss of generality assume that max j ∈ [ N ] ∥ v j ∥ ∞ = 1 . Let k ∈ N be fixed later. Define a map φ : [ -1 , 1] →{ 0 , 1 } 2 k +1 by

Also let Φ : R 2 k +1 → R be the linear map that sends Φ e i ↦→ ( i -k -1) /k for each i ∈ [2 k +1] . Note that | Φ φ ( c ) -c | ≤ 1 /k for all c ∈ [ -1 , 1] and Φ φ (0) = 0 . Define φ ⊕ n : [ -1 , 1] n → { 0 , 1 } (2 k +1) n by φ ( c 1 , . . . , c n ) = ( φ ( c 1 ) , . . . , φ ( c n )) , and define Φ ⊕ n : { 0 , 1 } (2 k +1) n → R n by Φ ⊕ n ( x 1 , . . . , x n ) = (Φ( x 1 ) , . . . , Φ( x n )) . For any i ∈ [ N ] , the vector φ ⊕ n ( v i ) is t -sparse and lies in { 0 , 1 } (2 k +1) n . Thus, applying Theorem E.3 gives some i ∈ [ N ] and x ∈ R N with ∥ x ∥ 1 ≤ 3 t and

̸

Since Φ ⊕ n is a linear map and ∥ Φ ⊕ n ∥ 2 = ∥ Φ ∥ 2 ≤ √ 2 k +1 , we then get

̸

But now for every j ∈ [ N ] , we know that

We conclude that

̸

̸

Taking k = ( N/n ) 1 / (4 t ) gives the claimed bound.

Combining Lemma E.5 with Lemma E.1 lets us prove Theorem 3.2.

Proof of Theorem 3.2. Set δ := √ 1 / (4 κ ) and N = 4 4 t ( t +3) t 2 t κ 2 t n . By Lemma E.5, for any t -sparse vectors v 1 , . . . , v N ∈ R n with ∥ v i ∥ 2 ≤ 1 for all i ∈ [ N ] , there is some i ∈ [ N ] and x ∈ R n such that ∥ x ∥ 1 ≤ 3 t and

̸

It follows from Lemma E.1 that P t, 1 / 3 t +1 (Σ) ≤ N log 2 κ . Finally, by Lemma A.2, we conclude that N t, 1 / 3 t +1 (Σ) ≤ N log 2 κ . ■

F Generalization bounds

F.1 Finite-class model selection

Lemma F.1. Let n, m, n eff ∈ N and let Σ be a positive semi-definite matrix. Fix a vector w ∗ ∈ R n and a closed set W ⊆ R n and let ( X i , y i ) i m =1 be independent draws X i ∼ N (0 , Σ) and y i = ⟨ X i , w ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . Pick

where X : m × n is the matrix with rows X 1 , . . . , X m . For any ϵ, δ ∈ (0 , 1) , suppose that with probability at least 1 -δ , the following bounds hold uniformly over w ∈ W :

̸

Then with probability at least 1 -δ it also holds that

Proof. Consider the event in which both bounds hold. Let w opt ∈ argmin w ∈W ∥ w -w ∗ ∥ 2 Σ . Then

Subtracting ∥ X ( w opt -w ∗ ) ∥ 2 2 from both sides and dividing by ∥ X ( ˆ w -w ∗ ) ∥ 2 + ∥ X ( w opt -w ∗ ) ∥ 2 , we get that √

It follows that as desired.

Lemma F.2. Let n, m ∈ N and let Σ be a positive semi-definite matrix. Fix a vector w ∗ ∈ R n and a finite set W ⊆ R n and let ( X i , y i ) i m =1 be independent draws X i ∼ N (0 , Σ) and y i = ⟨ X i , w ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . Pick

For any ϵ, δ ∈ (0 , 1) , if m ≥ 8 ϵ -2 log(2 |W| /δ ) , then with probability at least 1 -2 δ , we have

Proof. For any fixed w ∈ W , the random variables ⟨ X i , w -w ∗ ⟩ ∼ N (0 , ∥ w -w ∗ ∥ 2 Σ ) are independent, and therefore ∥ X ( w -w ∗ ) ∥ 2 2 ∼ ∥ w -w ∗ ∥ 2 Σ χ 2 m . It follows that for any ϵ > 0 ,

By the union bound, if m ≥ 8 ϵ -2 log(2 |W| /δ ) , then with probability at least 1 -δ it holds that for all w ∈ W ,

Also, for any fixed w ∈ W , conditioned on X , the random variable ⟨ ξ, X ( w -w ∗ ) ∥ X ( w -w ∗ ) ∥ 2 ⟩ has distribution N (0 , σ 2 ) . Thus, by a Gaussian tail bound and the union bound, we have for any t > 0 that

In particular, with probability at least 1 -δ it holds that

Using (10) and (11) we apply Lemma F.1 which gives the desired bound.

F.2 Weak learning

Lemma F.3. Let n, m ∈ N and ϵ, δ > 0 . Let Σ : n × n be a positive semi-definite matrix and let X : m × n have independent rows X 1 , . . . , X m ∼ N (0 , Σ) . For any fixed u, v ∈ R n , if m ≥ 8 ϵ -2 log(8 /δ ) , then it holds with probability at least 1 -δ that

Proof. Decompose u = av + w where ⟨ v, w ⟩ Σ = 0 , so that a = ⟨ u, v ⟩ Σ / ∥ v ∥ 2 Σ . Since ∥ X v ∥ 2 2 ∼ ∥ v ∥ 2 Σ χ 2 m and m ≥ 8 ϵ -2 log(4 /δ ) it holds with probability at least 1 -δ/ 2 that

Next,

where we define independent random vectors Z 1 , . . . , Z m ∼ N (0 , I n ) so that X i = Σ 1 / 2 Z i . Since m ≥ 8 log(2 /δ ) , with probability at least 1 -δ/ 4 we have ∑ m i =1 ⟨ Z i , Σ 1 / 2 v ⟩ 2 ≤ 2 m ∥ v ∥ 2 Σ . Condition on the value of this sum, and note that since Σ 1 / 2 v ⊥ Σ 1 / 2 w , the random variables ⟨ Z i , Σ 1 / 2 w ⟩ are still (independent and) distributed as N (0 , ∥ w ∥ 2 Σ ) . Thus

When the variance is at most 2 ∥ w ∥ 2 Σ ∥ v ∥ 2 Σ /m , we have with probability at least 1 -δ/ 4 that the sum is at most 2 ∥ w ∥ Σ ∥ v ∥ Σ √ 2 log(8 /δ ) /m in magnitude. So, using m ≥ 8 ϵ -2 log(8 /δ ) it holds unconditionally with probability at least 1 -δ/ 2 that

In all, we have that

using that | a | ≤ ∥ u ∥ Σ / ∥ v ∥ Σ and ∥ w ∥ Σ ≤ ∥ u ∥ Σ .

Lemma F.4. Let n, m ∈ N and let Σ be a positive semi-definite matrix. Fix a vector w ∗ ∈ R n and a finite set W ⊆ R n and let ( X i , y i ) i m =1 be independent draws X i ∼ N (0 , Σ) and y i = ⟨ X i , w ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . Pick

Suppose α := max w ∈W ⟨ w,w ∗ ⟩ Σ ∥ w ∥ Σ ∥ w ∗ ∥ Σ > 0 . For any δ > 0 , if m ≥ Cα -2 log(32 |W| /δ ) for a sufficiently large absolute constant C , then with probability at least 1 -δ ,

Proof. For any vectors u, v ∈ R n , define ∆( u, v ) = u ⊤ ( 1 m X ⊤ X -Σ ) v .

Claim F.5. With probability at least 1 -δ , the following bounds hold uniformly over w ∈ W and β ∈ R :

Proof of claim. For item (1), fix w ∈ W . Let Φ ( w ) : 2 × m be a matrix whose rows form an orthonormal basis for span { X w, X w ∗ } ⊆ R m . Then (denoting the unit Euclidean ball in R 2 by B 2 ) we have for all β ∈ R that

Since ⟨ Φ ( w ) i , ξ ⟩ ∼ N (0 , σ 2 ) , we have Pr[ |⟨ Φ ( w ) i , ξ ⟩| > σ √ 2 log(4 |W| /δ )] ≤ δ/ (4 |W| ) . A union bound over i ∈ [2] and w ∈ W gives that condition (2) in Lemma F.1 is satisfied with probability at least 1 -δ/ 2 .

For items (2) and (3), note that ∆ is bilinear, so it suffices to take β = 1 . Applying Lemma F.3 and the union bound, so long as m ≥ Cα -2 log(32 |W| /δ ) for a sufficiently large constant C , items (2) and (3) hold simultaneously with probability at least 1 -δ/ 2 .

Henceforth we assume that all of the events in the above claim hold. Let w 0 ∈ W be such that |⟨ w 0 , w ∗ ⟩ Σ | = α ∥ w 0 ∥ Σ ∥ w ∗ ∥ Σ . Let β 0 = ⟨ w 0 , w ∗ ⟩ Σ / ∥ w 0 ∥ 2 Σ . Then

Claim F.6. The excess empirical risk can be bounded as

Proof of claim. We have

where the last bound is by item (1) of Claim F.5. Simplifying, we get the claimed bound.

Now we have

where the first inequality is by Claim F.6, the second inequality is by AM-GM, and the final inequality is expanding out the terms ∆( ˆ β ˆ w -w ∗ , ˆ β ˆ w -w ∗ ) and ∆( β 0 w 0 -w ∗ , β 0 w 0 -w ∗ ) (via bilinearity) and cancelling out the common term ∆( w ∗ , w ∗ ) . Finally applying items (2) and (3) of Claim F.5, we get

where the second inequality uses the bounds ∥ β 0 w 0 -w ∗ ∥ 2 Σ = (1 -α 2 ) ∥ w ∗ ∥ 2 Σ and

But now on the other hand,

Comparing with (12) gives

and therefore

Substituting into (12) we finally get

as desired.

F.3 Excess risk at optima of additively-regularized programs

Lemma F.7. Let n ∈ N , and let Σ : n × n be a positive semi-definite matrix. For some seminorm Φ : R n → [0 , ∞ ) and some p, δ > 0 , assume that with probability at least 1 -δ over G ∼ N (0 , Σ) it holds uniformly over v ∈ R n that

Fix a vector v ∗ ∈ R n . For any m ∈ N and σ > 0 let ( X i , y i ) i m =1 be independent samples distributed as X i ∼ N (0 , Σ) and y i = ⟨ X i , v ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . Define

where X : m × n is the matrix with rows X 1 , . . . , X m . Then with probability at least 1 -7 δ over ( X i , y i ) i m =1 , so long as m ≥ 16 p +196log(12 /δ )) , it holds that

Proof. For notational convenience, define F ( v ) := (1 / 2)Φ( v -v ∗ ) + √ p ∥ v -v ∗ ∥ Σ . We apply the lemma's assumption twice:

For any fixed ξ , the random variable X ξ has distribution N (0 , ∥ ξ ∥ 2 2 Σ) . By the above claim, with probability at least 1 -δ over X , we have ⟨ ξ, X ( v -v ∗ ) ⟩ ≤ ∥ ξ ∥ 2 F ( v ) uniformly in v ∈ R n .

Since ∥ ξ ∥ 2 2 ∼ σ 2 χ 2 m and m ≥ 8 log(2 /δ ) , it holds with probability at least 1 -δ that 1 √ m ∥ ξ ∥ 2 ≤ √ 2 σ . Thus, with probability at least 1 -2 δ , we have

uniformly in v ∈ R n .

The assumption means that we can apply Theorem C.1 with (noiseless) samples ( X i , ⟨ X i , v ∗ ⟩ ) i m =1 to get the following: since m ≥ 196 log(12 /δ ) , it holds with probability at least 1 -4 δ over the randomness of X that for all v ∈ R n ,

Wealso observe that the entries of y are independent and identically distributed as N (0 , ∥ v ∗ ∥ 2 Σ + σ 2 ) , so by a χ 2 tail bound, since m ≥ 32 log(2 /δ ) , it holds with probability at least 1 -δ that

We now condition on the event (which occurs with probability at least 1 -7 δ ) that the bounds (13), (14), and (15) all hold. Specifying (14) to v := ˆ v , we get that

where the first inequality is by (14), the second inequality is by optimality of ˆ v , and the third inequality is by (13). We now expand F (ˆ v ) in the above expression. If √ 2 mpσ ∥ ˆ v -v ∗ ∥ Σ exceeds m 8 ∥ ˆ v -v ∗ ∥ 2 Σ then the lemma immediately holds since

So we may assume that in fact √ 2 mpσ ∥ ˆ v -v ∗ ∥ Σ ≤ m 8 ∥ ˆ v -v ∗ ∥ 2 Σ . By the lemma assumptions, we also know that m ≥ 16 p . Thus, expanding F (ˆ v ) and applying these bounds,

as claimed.

Simplifying, applying the triangle inequality Φ(ˆ v -v ∗ ) ≤ Φ(ˆ v ) + Φ( v ∗ ) , and grouping terms, we get

where the last inequality uses both sides of the bound (15).

G Covering bounds from classical assumptions

In this section, we further motivate the definition of our covering number N t,α (Σ) by showing that in all settings where efficient SLR algorithms are known, there is a straightforward linear upper bound on the covering number. This lends weight to the need for stronger upper bounds on N t,α as a stepping stone towards more efficient algorithms for sparse linear regression.

G.1 Compatibility condition

Definition G.1 (Compatibility Condition, see e.g. [40]) . For a positive semidefinite matrix Σ : n × n , L ≥ 1 , and set S ⊂ [ n ] , we say Σ has S -restricted ℓ 1 -eigenvalue

where the cone C ( S ) is defined as

̸

For t ∈ N , the t -restricted ℓ 1 -eigenvalue ϕ 2 (Σ , t ) is the minimum over all S of size at most t .

It is well-known that an upper bound on max i Σ ii ϕ 2 (Σ ,t ) is sufficient for the success of Lasso (as well as nearly necessary; see e.g. the Weak Compatibility Condition defined in [23]):

Theorem G.2 (see e.g. Corollary 5 in [45]) . Fix n, m, t ∈ N , σ, δ > 0 , and a positive semi-definite matrix Σ : n × n with max i Σ ii ≤ 1 . Fix a t -sparse vector v ∗ ∈ R n and let ( X i , y i ) i m =1 be independent samples distributed as X i ∼ N (0 , Σ) and y i = ⟨ X i , v ∗ ⟩ + ξ i where ξ i ∼ N (0 , σ 2 ) . Define

where X : m × n is the matrix with rows X 1 , . . . , X m . If m ≥ 4 ϕ 2 (Σ , t ) · t log(16 n/δ ) , then with probability at least 1 -δ , it holds that

Fact G.3. Let n, t ∈ N . For any positive semi-definite Σ : n × n with ϕ 2 := ϕ 2 (Σ , t ) and max i Σ ii ≤ 1 , it holds that N t,ϕ/ √ t (Σ) ≤ n .

Proof. The proof is essentially the same as that of Fact A.4. By Lemma A.3, it suffices to show that the standard basis is a ( t, √ t/ϕ ) -ℓ 1 -representation for Σ . Indeed, for any t -sparse v ∈ R n , we have

G.2 Submodularity ratio

Definition G.4 (see e.g. [9]) . For a positive semi-definite matrix Σ : n × n and a set L ⊆ [ n ] define the normalized residual covariance matrx Σ ( L ) : n × n by

where D := diag ( Σ -Σ ⊤ L Σ † LL Σ L ) .

Definition G.5. Fix a positive semi-definite matrix Σ : n × n , a positive integer t ∈ N , and any v ∗ ∈ R n . Define the t -submodularity ratio of Σ with respect to v ∗ by

In any t -sparse linear regression model with true regressor v ∗ , when the above quantity γ := γ t (Σ , v ∗ ) is bounded away from zero, it can be shown that the standard Forward Regression algorithm finds some t -sparse estimate ˆ v ∈ R n such that ∥ ˆ v -v ∗ ∥ 2 Σ ≤ e -γ ∥ v ∗ ∥ 2 Σ (see e.g. Theorem 3.2 in [9]; that result is for the model where the algorithm is given exact access to ⟨ v, v ∗ ⟩ Σ for any t -sparse v ∈ R n , but analogous finite-sample bounds can be obtained with O ( γ -O (1) t log( n )) samples by applying the theorem to the empirical covariance matrix and using concentration of t × t submatrices). A similar guarantee is also known for Orthogonal Matching Pursuit (Theorem 3.7 in [9]).

Once again, it is simple to show that the standard basis is a good dictionary for matrices with a large submodularity ratio.

Fact G.6. Let n, t ∈ N . For any positive semi-definite Σ : n × n with γ := min v ∗ ∈ R n ∩ B 0 ( t ) γ t (Σ , v ∗ ) , it holds that N t, √ γ/t (Σ) ≤ n .

Proof. We show that the standard basis is a ( t, γ/t ) -dictionary for Σ . Without loss of generality assume that Σ ii = 1 for all i ∈ [ n ] . Then Σ ( ∅ ) = Σ . Fix any t -sparse v ∗ ∈ R n . Setting S := supp( v ∗ ) , we have that

where the inequality is by definition of γ , and the final equality uses that Σ S v ∗ = Σ SS ( v ∗ ) S (since v ∗ is supported on S ). It follows that max i ∈ S ⟨ e i , v ∗ ⟩ 2 Σ ≥ ( γ/t ) ∥ v ∗ ∥ 2 Σ . Since ∥ e i ∥ Σ = 1 for all i , we conclude that as claimed.

G.3 Sparse preconditioning

Recent work [23] showed that if Σ : n × n is a positive definite matrix and the support of Θ := Σ -1 is the adjacency matrix of a graph with low treewidth , then there is a polynomial-time, sampleefficient algorithm for sparse linear regression with covariates drawn from N (0 , Σ) . The key to this result was a proof that such covariance matrices are sparsely preconditionable : i.e., there is a matrix S : n × n such that Σ = SS ⊤ and S has sparse rows. We claim that this property also immediately enables succinct dictionaries.

Concretely, suppose that S has s -sparse rows. By a change-of-basis argument, any t -sparse vector in the standard basis is st -sparse in the basis { ( S ⊤ ) -1 1 , . . . , ( S ⊤ ) -1 n } . Moreover these vectors are orthonormal under Σ . Thus, by the same argument as for Fact A.4, it's easy to see that { ( S ⊤ ) -1 1 , . . . , ( S ⊤ ) -1 n } is a ( t, 1 / √ st ) -dictionary for Σ .

H Supplementary figure

BP

1.0

0.8

Excessrisk

0.6

0.4

0.2

0.0

0

100

200

300

400

500

Numberofsamples

I Experimental details

The simulations were done using Python 3.9 and the Gurobi library [17]. Each figure took several minutes to generate using a standard laptop. See the file auglasso.py for code and execution instructions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of sparse linear regression under the statistical model where the examples are drawn as zero-mean Gaussians with covariance matrix $\\Sigma$ and each response is a t-sparse linear combination of the examples plus i.i.d. Gaussian noise. While classical results establish that the LASSO can computationally efficiently recover a good solution v* with a nearly optimal number of samples when the covariance matrix $\\Sigma$ is well-conditioned, guarantees beyond this simple setting are substantially lacking, with a few notable exceptions. In general, not much is known beyond a brute-force approach of trying all $\\binom{n}{t}$ sparsity patterns, which requires $n^t$ time. This work shows that even if $\\Sigma$ is ill-conditioned, if Sigma is well-conditioned after removing the top or bottom O(1) eigenvalues (i.e. a notion of “robust” well-condtionedness), then this running time can be improved to roughly $f(t) \\cdot n^3$. \n\nWhen there are a small number of tiny eigenvalues, the main problem is the existence of sparse linear dependencies among the variables. The algorithmic approach to address this problem is then to iteratively peel off a small number of these variables at a time (IterativePeeling()). This procedure gives a construction of a small dictionary of vectors such that any t-sparse vector can be written as a linear combination of this dictionary with coefficients bounded in L1 (Lemma 2.9), which in turn implies that the LASSO identifies a good solution v in the sense of bounded excess risk. The authors interpret the technique of applying LASSO to an augmented dictionary as “feature adaptation”.\n\nSTRENGTHS:\n* Sparse linear regression is a widely studied yet notoriously difficult problem, and this work identifies a very natural class of inputs for which positive results can be obtained. \n* I find the discussion in Section 2.1 on the proposal of (t, alpha) dictionaries as a canonical abstraction of all existing approaches to sparse linear regression to be very interesting and valuable. It helps concentrate research efforts on this problem to this more structured approach, and I believe it may prove to be influential in following works on sparse linear regression.\n\nWEAKNESSES:\n* It seems like like there are no bounds on the sparsity of the approximate solution v that is outputted by this algorithm, probably due to the fact that the feature adaptation step augments the feature set with dense linear combinations of the existing variables (please let me know if I have misunderstood). Thus, the measure of the “sparsity” of this algorithm lies in the small sample complexity, rather than its ability to output a sparse solution. Thus, the guarantees of this algorithm are tightly linked to the statistical setting of sparse linear regression, and it may be difficult to adapt these techniques to the problem of outputting a good sparse solution to linear regression. However, I do find this limitation very interesting, and I wonder if there are gaps between the performance of these algorithms if one is required to output a sparse solution. Such separations are known in certain sparse recovery settings (see, e.g., https://arxiv.org/abs/1110.4414).\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nThe authors have provided adequate discussion of limitations", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an algorithm to solve sparse regression when the covariates are generated from a normal distribution with ill-conditioned covariance matrix, i.e. outlayer eigenvalues. The algorithm is based on feature augmentation where, meaning that the covariates are completed with well-chosen vectors, and is designed to provably achieve near optimal sample complexity in the studied framework.\n\nSTRENGTHS:\nRegarding presentation, the paper is clearly written and presented. Each theoretical result is explained and justified, which makes the paper easy to follow. Regarding the content, the idea of augmenting the features by taking the data distribution into account before solving the Lasso seems new. The paper provides interesting theoretical insights on this method, while maintaining a computational perspective, for instance when explaining the design of the dictionary.\n\nWEAKNESSES:\nWhile the main contributions of the paper are theoretical, it would have been interesting to get more experimental details on the algorithm. In particular, numerical illustrations of the behavior with respect to the conditioning of the covariance matrix (number of outlayers, gap between largest and smallest eigenvalues, ...), and of the robustness with respect to standard algorithms like Basis Pursuit (for instance in \"phase transitions\" between well and ill-conditioned scenarios), may broaden the audience and the impact of the contributions.\n\nQUESTIONS:\nThe ideas of experiments below may help provide more information on the practical behavior of the algorithm and put it into perspective with the theoretical results.\n\n* Could the authors provide the number of samples necessary to reach low risk with respect to the ratio $\\frac{\\lambda_{n - d_h}}{\\lambda_{d_l+1}}$ and with respect to the number of outlayer eigenvalues when using their algorithm and when using BP ?\n\n* Could the authors provide the computation time with respect to the number of samples in addition to the risk ?\n\n* Could the authors provide the size of the set $S$ obtained in practice with Iterative Peeling, and compare it to the bounds given in Lemma 2.4 and 2.5, with respect to the sparsity $t$ ? \n\n* Could the authors provide a comparison of the performance with and without the knowledge of $\\Sigma$ (estimated from samples v. known distribution) ?\n\nLIMITATIONS:\nThe limitations are discussed by the authors.\n\n-----------\n\nAfter reading the other reviews and the rebuttal, I increased my rating.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an innovative polynomial-time algorithm for sparse linear regression in the correlated random design setting. The algorithm adapts the Lasso technique to effectively tolerate a limited number of approximate dependencies, resulting in both computational and statistical efficiency for covariance matrices with a few \"outlier\" eigenvalues. The proposed method is part of a more extensive framework of feature adaptation for sparse linear regression with ill-conditioned covariates and offers the first polynomial-factor improvement over brute-force search for constant sparsity and arbitrary covariance.\n\nSTRENGTHS:\n1. The paper contributes a novel algorithm for sparse linear regression, adeptly adapting the Lasso to accommodate a small number of approximate dependencies.\n\n2. The proposed algorithm exhibits both computational and statistical efficiency for covariance matrices with a few \"outlier\" eigenvalues, providing a substantial advancement over existing methods in this context.\n\nWEAKNESSES:\n1. The paper assumes constant sparsity for simplicity, but it is essential to explore the impact of this assumption on the main results.\n\n2. To more convincingly demonstrate the algorithm's computational and statistical efficiency, the paper would benefit from the inclusion of supplementary numerical simulations.\n\nQUESTIONS:\n1. The paper assumes constant sparsity $t$ for simplicity. It is unclear whether this assumption is necessary to establish the main results of the paper? In particular, it would be interesting to examine the case where the sparsity $t$ takes the order of $\\log n$.\n\n2. It is important to consider the more general case where $t$ is treated as a variable instead of a constant. I wonder whether the main results in the paper remain valid in this case.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the correlated random design setting, where the covariates are drawn from a multivariate Gaussian, and seeks an estimator with small excess risk. This work provides a polynomial-time algorithm that, given Σ, automatically adapts the Lasso to tolerate a small number of approximate dependencies, and achieves near-optimal sample complexity for constant sparsity and if Σ has few “outlier” eigenvalues.\n\nSTRENGTHS:\nSparse linear regression is a fundamental problem in high-dimensional statistics.\nThis paper studies a polynomial-time algorithm that automatically adapts the Lasso to tolerate a small number of approximate dependencies. In theoretical analysis, this work achieves near-optimal sample complexity for constant sparsity. The proposed algorithm fits into a broader framework of feature adaptation for sparse linear regression with ill-conditioned covariates.\n\nWEAKNESSES:\n1.If this work can provide experimental verification of the superiority of the proposed algorithm, it will be more convincing. For example, compare with the related work to verify the performance of the proposed algorithm in terms of time complexity and accuracy.\n\n2.In figure 1, when the number of samples is greater than 100, is the standard deviation of adapted BP algorithm zero?\n\n3.The presentation of references is not standardized, such as:\n\n[38] Sara Van De Geer. On tight bounds for the lasso. Journal of Machine Learning Research, 19:46, 2018.\n\n[22] Jonathan Kelner, Frederic Koehler, Raghu Meka, and Ankur Moitra. Learning some popular gaussian graphical models without condition number bounds. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020. \n\n4.The organization and presentation of this paper can be further improved.\n\nQUESTIONS:\nSee “Weaknesses”.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}