Title: Robust Sparse Regression with Non-Isotropic Designs
Venue: NeurIPS 2024
Accepted: yes
Keywords: linear regression

sparse regression

robust regression

sum-of-squares
Abstract: We develop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive.
Consider the model $y^*=X^*\beta^*+ \eta$ where $X^*$ is an $n\times d$ random design matrix, $\beta^*\in \mathbb{R}^d$ is a $k$-sparse vector, and the noise $\eta$ is independent of $X^*$ and chosen by the \emph{oblivious adversary}. 
Apart from the independence of $X^*$, we only require a small fraction entries of $\eta$ to have magnitude at most $1$. 
The \emph{adaptive adversary} is allowed to arbitrarily corrupt an $\varepsilon$-fraction of the samples $(X_1^*, y_1^*),\ldots, (X_n^*, y_n^*)$.
Given the $\varepsilon$-corrupted samples $(X_1, y_1),\ldots, (X_n, y_n)$, the goal is to estimate $\beta^*$.  
We assume that the rows of $X^*$ are iid samples from some $d$-dimensional distribution $\mathcal{D}$  with zero mean and (unknown) covariance matrix $\Sigma$ with bounded condition number.

We design several robust algorithms that outperform the state of the art even in the special case of Gaussian noise $\eta \sim N(0,1)^n$. 
In particular, we provide a polynomial-time algorithm that with high probability recovers $\beta^*$ up to error $O(\sqrt{\varepsilon})$  as long as  $n \ge \tilde{O}(k^2/\varepsilon)$, only assuming some bounds on the third and the fourth moments of $\mathcal{D}$. 
In addition, prior to this work, even in the special case of Gaussian design $\mathcal{D} = N(0,\Sigma)$ and noise $\eta \sim N(0,1)$, no polynomial time algorithm was known to achieve error $o(\sqrt{\varepsilon})$ in the sparse setting $n < d^2$.  
We show that under some assumptions on the fourth and the eighth moments of $\mathcal{D}$, there is a polynomial-time algorithm that achieves error $o(\sqrt{\varepsilon})$ as long as $n \ge \tilde{O}(k^4 / \varepsilon^3)$. 
For Gaussian distribution $\mathcal{D} = N(0,\Sigma)$, this algorithm achieves error $O(\varepsilon^{3/4})$. 
Moreover, our algorithm achieves error $o(\sqrt{\varepsilon})$ for all log-concave distributions if $\varepsilon \le 1/\text{polylog(d)}$. 

Our algorithms are based on the filtering  of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with $\ell_1$ regularizer.  We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.

Robust Sparse Regression with Non-Isotropic Designs

Abstract

Wedevelop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive. Consider the model 𝑦 ∗ = 𝑋 ∗ 𝛽 ∗ + 𝜂 where 𝑋 ∗ is an 𝑛 × 𝑑 random design matrix, 𝛽 ∗ ∈ R 𝑑 is a 𝑘 -sparse vector, and the noise 𝜂 is independent of 𝑋 ∗ and chosen by the oblivious adversary . Apart from the independence of 𝑋 ∗ , we only require a small fraction entries of 𝜂 to have magnitude at most 1. The adaptive adversary is allowed to arbitrarily corrupt an 𝜀 -fraction of the samples ( 𝑋 ∗ 1 , 𝑦 ∗ 1 ) , . . . , ( 𝑋 ∗ 𝑛 , 𝑦 ∗ 𝑛 ) . Given the 𝜀 -corrupted samples ( 𝑋 1 , 𝑦 1 ) , . . . , ( 𝑋𝑛, 𝑦 𝑛 ) , the goal is to estimate 𝛽 ∗ . We assume that the rows of 𝑋 ∗ are iid samples from some 𝑑 -dimensional distribution 𝒟 with zero mean and (unknown) covariance matrix Σ with bounded condition number. We design several robust algorithms that outperform the state of the art even in the special case of Gaussian noise 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 . In particular, we provide a polynomial-time algorithm that with high probability recovers 𝛽 ∗ up to error 𝑂 ( √ 𝜀 ) as long as 𝑛 ⩾ ˜ 𝑂 ( 𝑘 2 / 𝜀 ) , only assuming some bounds on the third and the fourth moments of 𝒟 . In addition, prior to this work, even in the special case of Gaussian design 𝒟 = 𝑁 ( 0 , Σ ) and noise 𝜂 ∼ 𝑁 ( 0 , 1 ) , no polynomial time algorithm was known to achieve error 𝑜 ( √ 𝜀 ) in the sparse setting 𝑛 < 𝑑 2 . We show that under some assumptions on the fourth and the eighth moments of 𝒟 , there is a polynomial-time algorithm that achieves error 𝑜 ( √ 𝜀 ) as long as 𝑛 ⩾ ˜ 𝑂 ( 𝑘 4 / 𝜀 3 ) . For Gaussian distribution 𝒟 = 𝑁 ( 0 , Σ ) , this algorithm achieves error 𝑂 ( 𝜀 3 / 4 ) . Moreover, our algorithm achieves error 𝑜 ( √ 𝜀 ) for all log-concave distributions if 𝜀 ⩽ 1 / polylog(d).

Our algorithms are based on the filtering of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with ℓ 1 regularizer. We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.

1 Introduction

Linear regression is the fundamental task in statistics, with many applications in data science and machine learning. In ordinary (non-sparse) linear regression, we are given observations 𝑦 ∗ 1 , . . . , 𝑦 ∗ 𝑛 and 𝑋 ∗ 1 , . . . , 𝑋 ∗ 𝑛 ∈ R 𝑑 such that 𝑦 ∗ = GLYPH<10> 𝑋 ∗ , 𝛽 ∗ GLYPH<11> + 𝜂 𝑖 for some 𝛽 ∗ ∈ R 𝑑 and some noise 𝜂 ∈ R 𝑛 , and the goal

𝑖

𝑖

is to estimate 𝛽 ∗ . If 𝜂 is independent of 𝑋 ∗ and has iid Gaussian entries 𝜂 𝑖 ∼ 𝑁 ( 0 , 1 ) , the classical least squares estimator ˆ 𝛽 with high probability achieves the prediction error 1 √ 𝑛 ∥ 𝑋 ∗ ( ˆ 𝛽 -𝛽 ∗ )∥ ⩽ 𝑂 GLYPH<16>p 𝑑 / 𝑛 GLYPH<17> . Note that if 𝑑 / 𝑛 → 0, the error is vanishing.

Despite the huge dimensions of modern data, many practical applications only depend on a small part of the dimensions of data, thus motivating sparse regression, where only 𝑘 ≪ 𝑑 explanatory variables are actually important (i.e., 𝛽 ∗ is 𝑘 -sparse). In this case we want the error to be small even if we only have 𝑛 ≪ 𝑑 samples. In this case, there exists an estimator that achieves prediction error 𝑂 GLYPH<16>p 𝑘 log ( 𝑑 )/ 𝑛 GLYPH<17> (for 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 ). However, this estimator requires exponential computation time. Moreover, under a standard assumption from computational complexity theory ( NP ⊄ P/poly ), estimators that can be computed in polynomial time require an assumption on 𝑋 ∗ called a restricted eigenvalue condition in order to achieve error 𝑂 GLYPH<16>p 𝑘 log ( 𝑑 )/ 𝑛 GLYPH<17> (see [ZWJ14] for more details). One efficiently computable estimator that achieves error 𝑂 GLYPH<16>p 𝑘 log ( 𝑑 )/ 𝑛 GLYPH<17> under the restricted eigenvalue condition is Lasso, that is, a minimizer of the quadratic loss with ℓ 1 regularizer. In particular, the restricted eigenvalue condition is satisfied for 𝑋 ∗ with rows 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Σ ) , where Σ has condition number 𝑂 ( 1 ) , as long as 𝑛 ≳ 𝑘 log 𝑑 (with high probability).

Further we assume that the designs have iid random rows, and the condition number of the covariance matrix is bounded by some constant. In addition, for random designs, we use the standard error ∥ Σ 1 / 2 ( ˆ 𝛽 -𝛽 ∗ )∥ . Note that when the number of samples is large enough, this error is very close to 1 √ 𝑛 ∥ 𝑋 ∗ ( ˆ 𝛽 -𝛽 ∗ )∥ .

Recently, there was an extensive interest in the linear regression with the presence of adversarially chosen outliers. Under the assumption 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Σ ) , the line of works [TJSO14, BJKK17, SBRJ19, dNS21, dLN + 21] studied the case when the noise 𝜂 is unbounded and chosen by an oblivious adversary, i.e., when 𝜂 is an arbitrary vector independent of 𝑋 ∗ . As was shown in [dLN + 21], in this case, it is possible to achieve the same error (up to a constant factor) as for 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 if we only assume that Ω ( 1 ) fraction of the entries of 𝜂 have magnitude at most 1. They analyzed the Huber loss estimator with ℓ 1 regularizer.

Another line of works [BJK15, DT19, MNW22, Tho23] assumed that 𝜂 has iid random entries that satisfy some assumptions on the moments, but an adversarially chosen 𝜀 -fraction of 𝑦 ∗ 1 , . . . , 𝑦 ∗ 𝑛 is replaced by arbitrary values by an adaptive adversary that can observe 𝑋 ∗ , 𝛽 ∗ and 𝜂 (so the corruptions can depend on them). [Tho23] showed that for 𝑋 ∗ with iid sub-Gaussian rows and 𝜂 with iid sub-Gaussian entries with unit variance, Huber loss estimator with ℓ 1 regularizer achieves an error of 𝑂 GLYPH<16>p 𝑘 log ( 𝑑 )/ 𝑛 + 𝜀 log ( 1 / 𝜀 ) GLYPH<17> with high probability. Note that the second term depends on 𝜀 , but not on 𝑛 ; hence, even if we take more samples, this term does not decrease (if 𝜀 remains the same). It is inherent: in the presence of the adaptive adversarial outliers, even for 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Id ) and 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 , the information theoretically optimal error is Ω GLYPH<16>p 𝑘 log ( 𝑑 )/ 𝑛 + 𝜀 GLYPH<17> , so independently of the number of samples, it is Ω ( 𝜀 ) . In the algorithmic high-dimensional robust statistics, we are interested in estimators that are computable in time poly ( 𝑑 ) . There is evidence that it is unlikely that poly ( 𝑑 ) -time computable estimators can achieve error 𝑂 ( 𝜀 ) [DKS17]. Furthermore, for other design distributions the optimal error can be different.

Hence the natural questions to ask are : Given an error bound 𝑓 ( 𝜀 ) , does there exist a poly ( 𝑑 ) -time computable estimator that achieves error at most 𝑓 ( 𝜀 ) with high probability? If possible, what is the smallest number of samples 𝑛 that is enough to achieve error 𝑓 ( 𝜀 ) in time poly ( 𝑑 ) ? In the rest of this section, we write error bounds in terms of 𝜀 and mention the number of samples that is required to achieve this error. In addition, we focus on the results for the high dimensional regime, where 𝑓 ( 𝜀 ) does not depend polynomially on 𝑘 or 𝑑 .

Another line of works [BDLS17, LSLC20, PJL20, Sas22, SF23] considered the case when the adaptive adversary is allowed to corrupt 𝜀 -fraction of all observed data, i.e. not only 𝑦 ∗ 1 , . . . , 𝑦 ∗ 𝑛 , but also 𝑋 ∗ 1 , . . . , 𝑋 ∗ 𝑛 , while the noise 𝜂 is assumed to have iid random entries that satisfy some concentration assumptions. For simplicity, to fix the scale of the noise, we formulate their results

assuming that 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 . In non-sparse settings, [PJL20] showed that in the case of identity covariance sub-Gaussian designs, Huber loss minimization after a proper filtering of 𝑋 ∗ achieves error ˜ 𝑂 ( 𝜀 ) with 𝑛 ≳ 𝑑 / 𝜀 2 samples. Informally speaking, filtering removes the samples 𝑋 ∗ 𝑖 that look corrupted, and if the distribution of the design is nice enough, then after filtering we can work with ( 𝑋 ∗ , 𝑦 ∗ ) just like in the case when only 𝑦 ∗ is corrupted. For unknown covariance they showed a bound 𝑂 GLYPH<0> √ 𝜀 GLYPH<1> for a large class of distributions of the design. If 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Σ ) for unknown Σ , one can use 𝑛 ⩾ ˜ 𝑂 GLYPH<0> 𝑑 2 / 𝜀 2 GLYPH<1> samples to robustly estimate the covariance, and achieve nearly optimal error ˜ 𝑂 ( 𝜀 ) in the case (see [DKS19] for more details).

In the sparse setting, there is likely an information-computation gap for the sample complexity of this problem, even in the case of the isotropic Gaussian design 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Id ) . While it is informationtheoretically possible to achieve optimal error 𝑂 ( 𝜀 ) with 𝑛 ⩾ ˜ 𝑂 ( 𝑘 / 𝜀 2 ) samples, achieving any error 𝑜 ( 1 ) is likely to be not possible for poly ( 𝑑 ) -time computable estimators if 𝑛 ≪ 𝑘 2 . Formal evidence for this conjecture include reductions from some version of the Planted Clique problem [BB20], as well as a Statistical Query lower bound (Proposition 1.10). For 𝑛 ⩾ ˜ 𝑂 ( 𝑘 2 / 𝜀 2 ) , several algorithmic results are known to achieve error ˜ 𝑂 ( 𝜀 ) , in particular, [BDLS17, LSLC20], and [SF23] for more general isotropic sub-Gaussian designs. Similarly to the approach of [PJL20], [SF23] used ( ℓ 1-penalized) Huber minimization after filtering 𝑋 ∗ .

The non-isotropic case (when Σ ≠ Id is unknown) is more challenging. [SF23] showed that for sub-Gaussian designs it is possible to achieve error 𝑂 GLYPH<0> √ 𝜀 GLYPH<1> with 𝑛 ⩾ ˜ 𝑂 ( 𝑘 2 ) samples. [Sas22] showed that 𝑂 GLYPH<0> √ 𝜀 GLYPH<1> error with 𝑛 ⩾ ˜ 𝑂 ( 𝑘 2 + ∥ 𝛽 ∗ ∥ 4 1 / 𝑘 2 ) samples can be achieved under some assumptions on the fourth and the eighth moments of the design distribution. While this result works for a large class of designs, the clear disadvantage is that the sample complexity depends polynomially on the norm of 𝛽 ∗ . For example, if all nonzero entries of 𝛽 ∗ have the same magnitude and ∥ 𝛽 ∗ ∥ = √ 𝑑 , then the sample complexity is 𝑛 > 𝑑 2 , which is not suitable in the sparse regime.

Prior to this work, no poly ( 𝑑 ) -time computable estimator that could achieve error 𝑜 GLYPH<0> 𝜀 GLYPH<1> with unknown Σ was known, even in the case of Gaussian designs 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Σ ) and the Gaussian noise 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 (apart from the non-sparse setting, where such estimators require 𝑛 > 𝑑 2 ).

1.1 Results

We present two main results, both of them follow from a more general statement; see Theorem B.3. Before formally stating the results, we define the model as follows.

Definition 1.1 (Robust Sparse Regression with 2 Adversaries) . Let 𝑛, 𝑑, 𝑘 ∈ N such that 𝑘 ⩽ 𝑑 , 𝜎 > 0, and 𝜀 ∈ ( 0 , 1 ) is smaller than some sufficiently small absolute constant. Let 𝒟 be a probability distribution in R 𝑑 with mean 0 and covariance Σ . Let 𝑦 ∗ = 𝑋 ∗ 𝛽 ∗ + 𝜂 , where 𝑋 ∗ is an 𝑛 × 𝑑 random matrix with rows 𝑋 ∗ 𝑖 iid ∼ 𝒟 , 𝛽 ∗ ∈ R 𝑑 is 𝑘 -sparse, 𝜂 ∈ R 𝑛 is independent of 𝑋 ∗ and has at least 0 . 01 · 𝑛 entries bounded by 𝜎 in absolute value 1 . We denote by 𝜅 ( Σ ) the condition number of Σ .

An instance of our model is a pair ( 𝑋, 𝑦 ) , where 𝑋 ∈ R 𝑛 × 𝑑 is a matrix and 𝑦 ∈ R 𝑛 is a vector such that there exists a set 𝑆 good ⊆ [ 𝑛 ] of size at least ( 1 -𝜀 ) 𝑛 such that for all 𝑖 ∈ 𝑆 good, 𝑋𝑖 = 𝑋 ∗ 𝑖 and 𝑦𝑖 = 𝑦 ∗ 𝑖 .

Note that random noise models studied in prior works are captured by our model in Definition 1.1. For example, if 𝜂 has iid entries that satisfy E | 𝜂 𝑖 | ⩽ 𝜎 / 2, by Markov's inequality, | 𝜂 𝑖 | ⩽ 𝜎 with probability at least 1 / 2, and with overwhelming probability, at least 0 . 01 · 𝑛 entries of 𝜂 are bounded by 𝜎 in absolute value. In addition, Cauchy noise (that does not have the first moment) with location parameter 0 and scale 𝜎 also satisfies these assumptions, as well as other heavy-tailed distributions studied in literature (with appropriate scale parameter 𝜎 ).

√

We formulate our results assuming that the condition number of the covariance is bounded by some constant: 𝜅 ( Σ ) ⩽ 𝑂 ( 1 ) . In the most general formulation (Theorem B.3), we show the dependence 2 of the number of samples and the error on 𝜅 ( Σ ) .

1.1.1 Robust regression with heavy-tailed designs

We use the following notion of boundness of the moments of 𝒟 :

Definition 1.2. Let 𝑀 > 0, 𝑡 ⩾ 2 and 𝑑 ∈ N . We say that a probability distribution 𝒟 in R 𝑑 with zero mean and covariance Σ has 𝑀 -bounded 𝑡 -th moment , if for all 𝑢 ∈ R 𝑑

Note that an arbitrary linear transformation of an isotropic distribution with 𝑀 -bounded 𝑡 -th moment also has 𝑀 -bounded 𝑡 -th moment. Also note that if 𝑡 ′ ⩽ 𝑡 and a distribution 𝒟 has 𝑀 -bounded 𝑡 -th moment, then the 𝑡 ′ -th moment of 𝒟 is also 𝑀 -bounded. In particular, 𝑀 cannot be smaller than 1, since the second moment cannot be 𝑀 -bounded for 𝑀 < 1. In addition, we will need the following (weaker) notion of the boundness of moments:

Definition 1.3. Let 𝜈 > 0, 𝑡 ⩾ 2 and 𝑑 ∈ N . We say that a probability distribution 𝒟 in R 𝑑 with zero mean and covariance Σ has entrywise 𝜈 -bounded 𝑡 -th moment , if

If a distribution has 𝑀 -bounded 𝑡 -th moment, then it also has entrywise 𝑀 -bounded 𝑡 -th moment, but the converse might not be true for some distributions. Now we are ready to state our first result.

Theorem 1.4. Let 𝑛, 𝑑, 𝑘, 𝑋, 𝑦, 𝜀 , 𝒟 , Σ , 𝜎 , 𝛽 ∗ be as in Definition 1.1. Suppose that 𝜅 ( Σ ) ⩽ 𝑂 ( 1 ) and that for some 1 ⩽ 𝑀 ⩽ 𝑂 ( 1 ) and 1 ⩽ 𝜈 ⩽ 𝑂 ( 1 ) , 𝒟 has 𝑀 -bounded 3 -rd moment and entrywise 𝜈 -bounded 4 -th moment. There exists an algorithm that, given 𝑋, 𝑦, 𝑘, 𝜀 , 𝜎 , in time ( 𝑛 + 𝑑 ) 𝑂 ( 1 ) outputs ˆ 𝛽 ∈ R 𝑑 such that if 𝑛 ≳ 𝑘 2 log ( 𝑑 )/ 𝜀 , then with probability at least 1 -𝑑 -10 ,

Let us compare Theorem 1.4 with the state of the art. For heavy-tailed designs, prior to this work, the best estimator was [Sas22]. That estimator also achieves error 𝑂 ( 𝜎 √ 𝜀 ) , but its sample complexity depends polynomially on the norm of 𝛽 ∗ , while our sample complexity does not depend on it. In addition, they require the distribution to have bounded 4-th moment (as opposed to our 3-rd moment assumption), and bounded entrywise 8-th moment (as opposed to our entrywise 4-th moment assumption). Finally, our noise assumption is weaker than theirs since they required the entries of 𝜂 to be iid random variables such that E | 𝜂 𝑖 | ⩽ 𝜎 ′ for some 𝜎 ′ > 0 known to the algorithm designer; as we mentioned after Definition 1.1, it is a special case of the oblivious noise with 𝜎 = 2 𝜎 ′ .

Let us also discuss our assumptions and possibilities of an improvement of our result. The third moment assumption can be relaxed, more precisely, it is enough to require the 𝑡 -th moment to be bounded, where 𝑡 is an arbitrary constant greater than 2, and in this case the sample complexity is increased by a constant factor 3 ; see Theorem B.3 for more details. The entrywise fourth moment assumption is not improvable with our techniques, that is, we get worse dependence on 𝑘 if we relax it to, say, the third moment assumption.

The dependence of 𝑛 on 𝜀 is not improvable with our techniques 4 . The dependence of the error on 𝜎 is optimal. The dependence of 𝑛 on 𝑘 and the error on √ 𝜀 is likely to be (nearly) optimal: Statistical Query lower bounds (Proposition 1.10 and Proposition 1.11) provide evidence that for 𝜎 = Θ ( 1 ) , it is unlikely that polynomial-time algorithms can achieve error 𝑜 ( 1 ) if 𝑛 ≪ 𝑘 2 , or error 𝑜 ( √ 𝜀 ) if 𝑛 ≪ 𝑘 4 . Remark 1.5 . Our results also imply bounds on other types of error studied in literature. In particular, observe that ∥ ˆ 𝛽 -𝛽 ∗ ∥ ⩽ ∥ Σ 1 / 2 ( ˆ 𝛽 -𝛽 ∗ )∥/ p 𝜆 min ( Σ ) , where 𝜆 min ( Σ ) is the minimal eigenvalue of Σ .

In addition, our estimator also satisfies ∥ ˆ 𝛽 -𝛽 ∗ ∥ 1 ⩽ 𝑂 (∥ Σ 1 / 2 ( ˆ 𝛽 -𝛽 ∗ )∥ · p 𝑘 / 𝜆 min ( Σ )) . The same is also true for our estimator from Theorem 1.7 below. These relations between different types of errors are standard for sparse regression, and they are not improvable.

√

1.1.2 Beyond 𝜀 error

Prior to this work, no polynomial-time algorithm for (non-isotropic) robust sparse regression was known to achieve error 𝑜 ( 𝜎 √ 𝜀 ) , even for Gaussian designs 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Σ ) and Gaussian 𝜂 ∼ 𝑁 ( 0 , 𝜎 ) 𝑛 . In this section we show that for a large class of designs, it is possible to achieve error 𝑜 ( 𝜎 √ 𝜀 ) in polynomial time, even when 𝜂 is chosen by an oblivious adversary. For our second result, we require not only some bounds on the moments of 𝒟 , but also their certifiability in the sum-of-squares proof system :

Definition 1.6. Let 𝑀 > 0 and let ℓ ⩾ 4 be an even number. We say that a probability distribution 𝒟 in R 𝑑 with zero mean and covariance Σ has ℓ -certifiably 𝑀 -bounded 4 -th moment , if there exist polynomials ℎ 1 , . . . , ℎ𝑚 ∈ R [ 𝑢 1 , . . . , 𝑢 𝑑 ] of degree at most ℓ / 2 such that

Definition 1.6 with arbitrary ℓ implies Definition 1.2 (with the same 𝑀 ). Under standard complexitytheoretic assumptions, there exist distributions with bounded moments that are not ℓ -certifiably bounded even for very large ℓ [HL19]. Note that similarly to Definition 1.2, an arbitrary linear transformation of an isotropic distribution with ℓ -certifiably 𝑀 -bounded 4-th moment also has ℓ -certifiably 𝑀 -bounded 4-th moment.

Distributions with certifiably bounded moments are very important in algorithmic robust statistics. They were extensively studied in literature, e.g. [KS17a, KS17b, HL18, HL19, DKK + 22].

Now we can state our second result.

Theorem 1.7. Let 𝑛, 𝑑, 𝑘, 𝑋, 𝑦, 𝜀 , 𝒟 , Σ , 𝜎 , 𝛽 ∗ be as in Definition 1.1. Suppose that 𝜅 ( Σ ) ⩽ 𝑂 ( 1 ) , and that for some 𝑀 ⩾ 1 , some even number ℓ ⩾ 4 , and 1 ⩽ 𝜈 ⩽ 𝑂 ( 1 ) , 𝒟 has ℓ -certifiably 𝑀 -bounded 4 -th moment and entrywise 𝜈 -bounded 8 -th moment. There exists an algorithm that, given 𝑋, 𝑦, 𝑘, 𝜀 , 𝜎 , 𝑀, ℓ , in time ( 𝑛 + 𝑑 ) 𝑂 ( ℓ ) outputs ˆ 𝛽 ∈ R 𝑑 such that if 𝑛 ≳ 𝑀 4 · 𝑘 4 log ( 𝑑 )/ 𝜀 3 , then with probability at least 1 -𝑑 -10 ,

In particular, in the regime 𝑀 ⩽ 𝑂 ( 1 ) , as long as 𝑛 ⩾ ˜ 𝑂 ( 𝑘 4 / 𝜀 3 ) , the algorithm recovers 𝛽 ∗ from ( 𝑋, 𝑦 ) up to error 𝑂 ( 𝜎𝜀 3 / 4 ) (with high probability). If ℓ ⩽ 𝑂 ( 1 ) , the algorithm runs in polynomial time. Note that in this theorem we do not assume that 𝑀 is constant as opposed to Theorem 1.4 since for some natural classes of distributions, only some bounds on 𝑀 that depend on 𝑑 are known.

The natural question is what distributions have certifiably bounded fourth moment with ℓ ⩽ 𝑂 ( 1 ) . First, these are products of one-dimensional distributions with 𝑀 -bounded fourth moment, and their linear transformations (with ℓ = 4). Hence, linear transformations of products of one-dimensional distributions with 𝑂 ( 1 ) -bounded 8-th moment satisfy the assumptions of the theorem with 𝑀 ⩽ 𝑂 ( 1 ) and ℓ = 4. Note that such distributions might not even have a 9-th moment. This class also includes Gaussian distributions (since they are linear transformations of the 𝑁 ( 0 , 1 ) 𝑑 and 𝑁 ( 0 , 1 ) has 𝑂 ( 1 ) -bounded 8-th moment).

Another important class is the distributions that satisfy Poincaré inequality . Concretely, these distributions, for some 𝐶𝑃 ⩾ 1, satisfy Var 𝑥 ∼𝒟 𝑔 ( 𝑥 ) ⩽ 𝐶 2 𝑃 · ∥ Σ ∥ · E 𝑥 ∼𝒟 ∥∇ 𝑔 ( 𝑥 )∥ 2 2 for all continuously differentiable functions 𝑔 : R 𝑑 → R . [KS17a] showed that such distributions have 4-certifiably 𝑂 ( 𝐶𝑃 ) -bounded fourth moment. We will not further discuss Poincaré inequality, and focus on the known results on the classes of distributions satisfy this inequality.

The Kannan-Lovász-Simonovits (KLS) conjecture from convex geometry says that 𝐶𝑃 is bounded by some universal constant for all log-concave distributions. Recall that a distribution 𝒟 is called log-concave if for some convex function 𝑉 : R 𝑑 → R , the density of 𝒟 is proportional to 𝑒 -𝑉 ( 𝑥 ) .

Apart from the Gaussian distribution, examples include uniform distributions over convex bodies, the Wishart distribution and the Dirichlet distribution ([Pré71], see also [KBJ00] for further examples). In recent years there has been a big progrees towards the proof of the KLS conjecture. [Che21] showed that 𝐶𝑃 ⩽ 𝑑 𝑜 ( 1 ) , and since then, the upper bound has been further significantly improved. The best current bound is 𝐶𝑃 ⩽ 𝑂 ( p log 𝑑 ) obtained by [Kla23]. This bound implies that for all log-concave distributions whose covariance has bounded condition number, the error of our estimator is 𝑂 ( 𝜎 p log 𝑑 · 𝜀 3 / 4 ) . Hence for 𝜀 ⩽ 𝑜 ( 1 / log 2 ( 𝑑 )) and 𝜎 ⩽ 𝑂 ( 1 ) , the error is 𝑜 ( √ 𝜀 ) . Note that if the KLS conjecture is true, the error of our estimator is 𝑂 ( 𝜎𝜀 3 / 4 ) for all log-concave distributions with 𝜅 ( Σ ) ⩽ 𝑂 ( 1 ) , without any restrictions on 𝜀 (except the standard 𝜀 ≲ 1).

Remark 1.8 . Theorem 1.7 can be generalized as follows: If the ( 2 𝑡 ) -th moment of 𝒟 is 𝑀 -bounded for a constant 𝑡 ∈ N ⩾ 2, if this bound can be certified by a constant degree sum-of-squares proof 5 , and if 𝒟 has entrywise ( 4 𝑡 ) -th 𝑂 ( 1 ) -bounded moment, then with high probability, there is a poly ( 𝑑 ) -time computable estimator that achieves error 𝑂 ( 𝑀 𝜎𝜀 1 -1 /( 2 𝑡 ) ) as long as 𝑛 ≳ 𝑀 4 𝑘 2 𝑡 log ( 𝑑 )/ 𝜀 2 𝑡 -1 . See Theorem B.3 for more details.

Remark 1.9 . The dependence of 𝑛 on 𝜀 can be improved under the assumption that 𝒟 is a subexponential distribution. In particular, all log-concave distributions are sub-exponential. Under this additional assumption, in order to achieve the error 𝑂 ( 𝜎 √ 𝜀 ) , it is enough to take 𝑛 ≳ 𝑘 2 polylog ( 𝑑 ) + 𝑘 log ( 𝑑 )/ 𝜀 , and to achieve error 𝑂 ( 𝑀 𝜎𝜀 3 / 4 ) , it is enough to take 𝑛 ≳ 𝑘 4 polylog ( 𝑑 ) + 𝑘 log ( 𝑑 )/ 𝜀 3 / 2 samples (assuming, as in Theorem 1.7, that the fourth moment is 𝑀 -certifiably bounded).

1.1.3 Lower bounds

We provide Statistical Query (SQ) lower bounds by which our estimators likely have optimal sample complexities needed to achieve the errors 𝑂 ( √ 𝜀 ) and 𝑜 ( √ 𝜀 ) , even when the design and the noise are Gaussian. SQ lower bounds are usually interpreted as a tradeoff between the time complexity and sample complexity of estimators; see Appendix G and [DKS17] for more details. Our proofs are very similar to prior works [DKS17, DKS19, DKK + 22] since as was observed in [DKS19], lower bounds for mean estimation can be used to prove lower bounds for linear regression, and we use the lower bounds for sparse mean estimation from [DKS17, DKK + 22].

Let us fix the scale of the noise 𝜎 = 1. The first proposition shows that already for Σ = Id, 𝑘 2 samples are likely to be necessary to achieve error 𝑜 ( 1 ) :

Proposition 1.10 (Informal, see Proposition G.9) . Let 𝑛, 𝑑, 𝑘, 𝑋, 𝑦, 𝜀 , 𝒟 , Σ , 𝜎 , 𝛽 ∗ be as in Definition 1.1. Suppose that 𝒟 = 𝑁 ( 0 , Id ) and 𝜂 ∼ 𝑁 ( 0 , ˜ 𝜎 2 ) 𝑛 , where 0 . 99 ⩽ ˜ 𝜎 ⩽ 1 . Suppose that 𝑑 0 . 01 ⩽ 𝑘 ⩽ √ 𝑑 , 𝜀 ≳ 1 √ log 𝑑 , and 𝑛 ⩽ 𝑘 1 . 99 . Then for each SQ algorithm 𝐴 that finds ˆ 𝛽 such that

∥ 𝛽 ∗ - ˆ 𝛽 ∥ ⩽ 10 -5 , the simulation of 𝐴 with 𝑛 samples has to simulate super-polynomial ( exp ( 𝑑 Ω ( 1 ) ) ) number of queries.

Note that under assumptions of Proposition 1.10, Theorem 1.4 implies that if we take 𝑛 ⩾ 𝑘 2 polylog ( 𝑑 ) samples, the estimator achieves error 𝑂 ( √ 𝜀 ) that is 𝑜 ( 1 ) if 𝜀 → 0 as 𝑑 →∞ .

The second proposition shows that for 1 2 ⪯ Σ ⪯ Id, 𝑘 4 samples are likely to be necessary to achieve error 𝑜 ( √ 𝜀 ) :

Proposition 1.11 (Informal, see Proposition G.10) . Let 𝑛, 𝑑, 𝑘, 𝑋, 𝑦, 𝜀 , 𝒟 , Σ , 𝜎 , 𝛽 ∗ be as in Definition 1.1. Suppose that 𝒟 = 𝑁 ( 0 , Σ ) for some Σ such that 1 2 ⪯ Σ ⪯ Id , and 𝜂 ∼ 𝑁 ( 0 , ˜ 𝜎 2 ) 𝑛 , where 0 . 99 ⩽ ˜ 𝜎 ⩽ 1 . Suppose that 𝑑 0 . 01 ⩽ 𝑘 ⩽ √ 𝑑 , 𝜀 ≳ 1 log 𝑑 , and 𝑛 ⩽ 𝑘 3 . 99 . Then for each SQ algorithm 𝐴 that finds ˆ 𝛽 such that ∥ 𝛽 ∗ - ˆ 𝛽 ∥ ⩽ 10 -5 √ 𝜀 , the simulation of 𝐴 with 𝑛 samples has to simulate super-polynomial ( exp ( 𝑑 Ω ( 1 ) ) ) number of queries.

Note that under assumptions of Proposition 1.11, Theorem 1.7 implies that if we take 𝑛 ⩾ 𝑘 4 polylog ( 𝑑 ) samples, the estimator achieves error 𝑂 ( 𝜀 3 / 4 ) that is 𝑜 ( √ 𝜀 ) if 𝜀 → 0 as 𝑑 →∞ .

2 Techniques

Since the problem has multiple aspects, we first illustrate our approach on the simplest example 𝑋 ∗ 𝑖 iid ∼ 𝑁 ( 0 , Σ ) under the assumption that 0 . 1 · Id ⪯ Σ ⪯ 10 · Id. Note that already in this case, even for 𝜂 ∼ 𝑁 ( 0 , 1 ) 𝑛 , our estimator from Theorem 1.7 outperforms the state of the art. In addition, we assume that 𝜎 = 1.

Our estimators are based on preprocessing 𝑋 , and then minimizing ℓ 1 -penalized Huber loss . In the Gaussian case, the preprocessing step consists only of filtering , while for heavy-tailed designs, an additional truncation step is required. The idea of using filtering before minimizing the Huber loss first appeared in [PJL20] for the dense settings, and was applied to sparse settings in [Sas22, SF23]. We will not discuss the filtering method in detail, and rather focus on its outcome: It is a set ˆ 𝑆 ⊆ [ 𝑛 ] of size at least ( 1 -𝑂 ( 𝜀 )) 𝑛 that satisfies some nice properties 6 . Further, we will see what properties we need from ˆ 𝑆 , and now let us define the Huber loss estimator.

Definition 2.1. For 𝑆 ⊆ [ 𝑛 ] , the Huber loss function restricted to 𝑆 is defined as

For a penalty parameter 𝜆 , the ℓ 1-penalized Huber loss restricted to 𝑆 is defined as 𝐿𝑆 ( 𝛽 ) : = 𝐻𝑆 ( 𝛽 ) + 𝜆 · ∥ 𝛽 ∥ 1 . We use the notation 𝜙 ( 𝑥 ) for the derivative of ℎ ( 𝑥 ) . Note that for all 𝑥 , | 𝜙 ( 𝑥 )| ⩽ 2.

Our estimator is the minimizer ˆ 𝛽 ˆ 𝑆 of 𝐿 ˆ 𝑆 ( 𝛽 ) , where ˆ 𝑆 is the set returned by the filtering algorithm. To investigate the properties of this estimator, it is convenient to work with elastic balls . The 𝑘 -elastic ball of radius 𝑟 is the following set: ℰ 𝑘 ( 𝑟 ) : = { 𝑢 ∈ R 𝑑 | ∥ 𝑢 ∥ ⩽ 𝑟 , ∥ 𝑢 ∥ 1 ⩽ √ 𝑘 · 𝑟 } . Note that this ball contains all 𝑘 -sparse vectors with Euclidean norm at most 𝑟 (as well as some other vectors). Elastic balls are very useful for sparse regression since if the following two properties hold,

Gradient bound: For all 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) , | ⟨∇ 𝐻 ˆ 𝑆 , 𝑢 ⟩| ≲ 𝑟 √ 𝑘 ∥ 𝑢 ∥ 1 + 𝑟 ∥ 𝑢 ∥ ,

Strong convexity on the boundary: For all 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) such that ∥ 𝑢 ∥ = 𝑟 ,

then for an appropriate choice of the penalty parameter 𝜆 , then GLYPH<13> GLYPH<13> 𝛽 ∗ - ˆ 𝛽 ˆ 𝑆 GLYPH<13> GLYPH<13> < 𝑟 . 7

Hence it is enough to show these two properties. In the Gaussian case, the strong convexity property can be proved in exactly the same way as it is done in [dLN + 21] for the case of the oblivious adversary, while for heavy-tailed designs it is significantly more challenging. Since we now discuss the Gaussian case, let us focus on the gradient bound. Denote 𝐻 ∗ 𝑆 ( 𝛽 ) = 1 𝑛 ˝ 𝑖 ∈ 𝑆 ℎ GLYPH<0> ⟨ 𝑋 ∗ 𝑖 , 𝛽 ⟩ -𝑦 ∗ 𝑖 GLYPH<1> . By triangle inequality,

√

Since the first term can be bounded by ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ · ∥ 𝑢 ∥ 1, it is enough to show that ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ ≲ 𝑟 / 𝑘 , where 𝑟 is the error we aim to achieve. Note that ∇ 𝐻 ∗ [ 𝑛 ] = 1 𝑛 ˝ 𝑛 𝑖 = 1 𝜙 ( 𝜂 𝑖 )⟨ 𝑋 ∗ 𝑖 , 𝑢 ⟩ does not depend on the outliers created by the adaptive adversary. The sharp bound on ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ can be derived in exactly the same way as in [dLN + 21] (or other prior works): Since 𝜂 and 𝑋 ∗ are independent and | 𝜙 ( 𝜂 )| ⩽ 2, ∇ 𝐻 ∗ [ 𝑛 ] is a Gaussian vector whose entries have variance ( 1 / 𝑛 ) . By standard properties of Gaussian vectors, ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ ⩽ 𝑂 ( p log ( 𝑑 )/ 𝑛 ) with high probability.

√

To bound the second and the third term, we can use Cauchy-Schwarz inequality and get 𝑂 ( 𝜀 ) dependence on the error (like it is done in prior works on robust sparse regression, for example, [Sas22] or [SF23]), or use Hölder's inequality and get better dependence, but also more challenges since we have to work with higher (empirical) moments of 𝑋 ∗ and 𝑋 . Let us use Hölder'sinequality and illustrate how we work with higher moments. Note that both sets [ 𝑛 ] \ ( 𝑆 good ∩ ˆ 𝑆 ) and 𝑆 bad ∩ ˆ 𝑆 have size at most 𝑂 ( 𝜀 𝑛 ) . Hence the second term can be bounded by

while the third term is bounded by

A careful probabilistic analysis shows that with high probability, for all 𝑟 ⩾ 0 and all 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) , ˝ 𝑖 ∈[ 𝑛 ] 1 𝑛 ⟨ 𝑋 ∗ 𝑖 , 𝑢 ⟩ 4 ⩽ 𝑂 GLYPH<0> ∥ 𝑢 ∥ 4 GLYPH<1> . Hence, our requirement on ˆ 𝑆 is that ˝ 𝑖 ∈ ˆ 𝑆 1 𝑛 ⟨ 𝑋𝑖 , 𝑢 ⟩ 4 ⩽ 𝑂 ( 1 ) for all 𝑢 ∈ ℰ 𝑘 ( 1 ) (by scaling argument, it is enough to consider 𝑟 = 1). If we find such a set ˆ 𝑆 , we get the desired bound. Indeed, if 𝑛 ≳ 𝑘 log ( 𝑑 )/ 𝜀 3 / 2 , ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ ⩽ 𝑂 ( 𝜀 3 / 4 / √ 𝑘 ) , and the other terms are bounded by 𝑂 ( 𝜀 3 / 4 ) , implying that GLYPH<13> GLYPH<13> ˆ 𝛽 - ˆ 𝛽 ˆ 𝑆 GLYPH<13> GLYPH<13> < 𝑟 = 𝑂 ( 𝜀 3 / 4 ) .

Note that such sets of size ( 1 -𝑂 ( 𝜀 )) 𝑛 exist since 𝑆 good satisfies this property. It is clear how to find such a set inefficiently: we just need to check all candidate sets 𝑆 and maximize the quartic function ˝ 𝑖 ∈ 𝑆 ⟨ 𝑋𝑖 , 𝑢 ⟩ 4 over 𝑢 ∈ ℰ 𝑘 ( 1 ) . Furthermore, the by-now standard filtering method allows to avoid checking all the sets: If we can maximize ˝ 𝑖 ∈ 𝑆 ⟨ 𝑋𝑖 , 𝑢 ⟩ 4 over 𝑢 ∈ ℰ 𝑘 ( 1 ) efficiently, we can also find the desired set efficiently.

Before explaining how we maximize this function, let us see how prior works [BDLS17, SF23], optimized a simpler quadratic function ˝ 𝑖 ∈ 𝑆 ⟨ 𝑋𝑖 , 𝑢 ⟩ 2 over 𝑢 ∈ ℰ 𝑘 ( 1 ) . They use the basic SDP relaxation for sparse PCA, that is, they optimize the linear function ˝ 𝑖 ∈ 𝑆 ⟨ 𝑋𝑖 𝑋 ⊤ 𝑖 , 𝑈 ⟩ over ℬ 𝑘 : = { 𝑈 ∈ R 𝑑 × 𝑑 | 𝑈 ⪰ 0 , Tr ( 𝑈 ) ⩽ 1 , ∥ 𝑈 ∥ 1 ⩽ 𝑘 } . This set has been used in literature for numerous sparse problems since it is a nice (perhaps the best) convex relaxation of the set 𝒮 𝑘 = { 𝑢𝑢 ⊤ | 𝑢 ∈ R 𝑑 , ∥ 𝑢 ∥ ⩽ 1 , ∥ 𝑢 ∥ 0 ⩽ 𝑘 } . Moreover, crucially for sparse regression, it is easy to see that ℬ 𝑘 also contains all matrices 𝑢𝑢 ⊤ such that 𝑢 ∈ ℰ 𝑘 ( 1 ) . Hence, one may try to optimize quartic functions by using relaxations of 𝒮 𝑘 = { 𝑢 ⊗ 4 | 𝑢 ∈ R 𝑑 , ∥ 𝑢 ∥ ⩽ 1 , ∥ 𝑢 ∥ 0 ⩽ 𝑘 } . A natural relaxation is the sumof-squares with sparsity constraints . [DKK + 22] used these relaxations for sparse mean estimation 8 . They showed that these relaxations provide nice guarantees for distributions with certifiably bounded 4-th moment, assuming that the distribution has sub-exponential tails. Since we now discuss the Gaussian case, the assumption on the tails is satisfied. However, there is no guarantee that these relaxations capture 𝑢 ⊗ 4 for all 𝑢 ∈ ℰ 𝑘 ( 1 ) . So, for sparse regression, we need another relaxation.

We use the sum-of-squares relaxations with elastic constraints . These constraints ensure that the set of relaxations 𝒫 𝑘 ⊂ R 𝑑 4 is guaranteed to contain 𝑢 ⊗ 4 for all 𝑢 ∈ ℰ 𝑘 ( 1 ) . We show that if 𝑛 ≳ ˜ 𝑂 ( 𝑘 4 ) , there is a degree𝑂 ( 1 ) sum-of-squares proof from the elastic constraints of the fact that 1 𝑛 ˝ 𝑖 ∈[ 𝑛 ] ⟨ 𝑋𝑖 , 𝑢 ⟩ 4 ⩽ 𝑂 ( 1 ) . It implies that the relaxation is nice: If 1 𝑛 ˝ 𝑖 ∈ 𝑆 ⟨ 𝑋𝑖 , 𝑢 ⟩ 4 ⩽ 𝑂 ( 1 ) for all 𝑢 ∈ ℰ 𝑘 ( 1 ) , then 1 𝑛 ˝ 𝑖 ∈ 𝑆 ⟨ 𝑋 ⊗ 4 𝑖 , 𝑈 ⟩ ⩽ 𝑂 ( 1 ) for all 𝑈 ∈ 𝒫 𝑘 . Since we can efficiently optimize over 𝒫 𝑘 , we get an efficiently computable estimator with error 𝑂 ( 𝜀 3 / 4 ) for Gaussian distributions. Furthermore, if we first use a proper thresholding (that we discuss below), our sum-of-squares proof also works for heavy-tailed distributions, that, apart from the certifiably bounded 4-th moment (that we cannot avoid with the sum-of-squares approach), are only required to have entrywise bounded 8-th moment.

Robust sparse regression with heavy-tailed designs is much more challenging. Again, for simplicity assume that 0 . 1 · Id ⪯ Σ ⪯ 10 · Id and 𝜎 = 1. First, there is an issue even without the adversarial noise: ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ can be very large. Even under bounded fourth moment assumption, it can have magnitude ˜ 𝑂 ( 𝑑 1 / 4 / 𝑛 ) , which is too large in the sparse setting. Hence we have to perform an additional thresholding step and remove large entries of 𝑋 . Usually thresholding of the design matrix should be

done very carefully since it breaks the relation between 𝑋 and 𝑦 . [Sas22] required the thresholding parameter 𝜏 to be large enough and depend polynomially on ∥ 𝛽 ∗ ∥ so that this dependence does not break significantly. Since ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ can be as large as ˜ 𝑂 ( 𝜏 / 𝑛 ) , the sample complexity of their estimator also depends polynomially on ∥ 𝛽 ∗ ∥ .

Our idea of thresholding is very different, and it plays a significant role in our analysis, especially in the proof of strong convexity. Since we already have to work with outliers chosen by the adaptive adversary, we know that for an 𝜀 -fraction of samples, the dependence of 𝑦 on 𝑋 can already be broken. So, if we choose the thresholding parameter 𝜏 to be large enough so that with high probability it only affects an 𝜀 -fraction of samples, we can simply treat the samples affected by such thresholding as additional adversarial outliers, and assume that the adaptive adversary corrupted 2 𝜀 𝑛 samples. Note that since 𝒟 is heavy-tailed, each sample 𝑋 ∗ 𝑖 might have entries of magnitude 𝑑 Ω ( 1 ) . However, 𝑦 depends only on the inner products ⟨ 𝑋 ∗ 𝑖 , 𝛽 ∗ ⟩ , and this inner product depends only on the entries of 𝑋 ∗ that correspond to the support of 𝛽 ∗ . Even though we don't know the support, we can guarantee that for 𝜏 ⩾ 20 p 𝑘 / 𝜀 , all entries of 𝑋𝑖 from the support of 𝛽 ∗ are bounded by 𝜏 with probability 1 -𝜀 / 2. Indeed, since the variance of each entry is bounded by 10, Chebyshev's inequality implies that this entry is smaller than 𝜏 with probability at least 1 -𝜀 /( 2 𝑘 ) , and by union bound, ⟨ 𝑋 ∗ 𝑖 , 𝛽 ∗ ⟩ is not affected by the thresholding with probability 1 -𝜀 / 2. Hence by Chernoff bound, with overwhelming probability, the number of samples affected by our thresholding is at most 𝜀 𝑛 .

Let us denote the distribution of the rows of 𝑋 ∗ after thresholding with parameter 𝜏 by 𝒟( 𝜏 ) . After the thresholding step, we can assume that 𝑋 ∗ 𝑖 iid ∼ 𝒟( 𝜏 ) . Note that thresholding can shift the mean, i.e. E 𝑋 ∗ 𝑖 can be nonzero. It is easy to see that ∥ E 𝑥 ∼𝒟( 𝜏 ) 𝑥 ∥ ∞ ⩽ 𝑂 ( 1 / 𝜏 ) . Hence by Bernstein's inequality, ∥∇ 𝐻 ∗ [ 𝑛 ] ∥ ∞ ⩽ ˜ 𝑂 GLYPH<16>p 1 / 𝑛 + 𝜏 / 𝑛 + 1 / 𝜏 GLYPH<17> with high probability 9 . In particular, in order to get the error bounded by 𝑂 ( 𝜀 3 / 4 ) , we need to take 𝜏 ≳ √ 𝑘 / 𝜀 3 / 4 , and it affects sample complexity. Furthermore, our sum-of-squares proof requires that GLYPH<13> GLYPH<13> GLYPH<13> 1 𝑛 ˝ 𝑛 𝑖 = 1 GLYPH<0> 𝑋 ∗ 𝑖 GLYPH<1> ⊗ 4 -E GLYPH<0> 𝑋 ∗ 1 GLYPH<1> ⊗ 4 GLYPH<13> GLYPH<13> GLYPH<13> ∞ is smaller that 1 / 𝑘 2 . It can be shown that this quantity is bounded by ˜ 𝑂 GLYPH<16>p 1 / 𝑛 + 𝜏 4 / 𝑛 + 1 / 𝜏 4 GLYPH<17> with high probability 10 . In particular, we need 𝑛 ⩾ ˜ 𝑂 GLYPH<0> 𝜏 4 𝑘 2 GLYPH<1> , so for 𝜏 ≳ √ 𝑘 / 𝜀 3 / 4 , we have to take 𝑛 ⩾ ˜ 𝑂 GLYPH<0> 𝑘 4 / 𝜀 3 GLYPH<1> . As was discussed in Remark 1.9, if 𝒟 has sub-exponential tails, we do not have to do the thresholding, and the bounds from [DKK + 22] allow to avoid this dependence of 𝑛 on 𝜀 . Note that due to the SQ lower 𝑘 4

bound (Proposition 1.11), sample complexity is likely to be necessary, even for Gaussian designs.

Finally, let us discuss the strong convexity property. Here, we do not assume any properties related to sum-of-squares, and focus on the weak assumptions of Theorem 1.4. First, assume that we need to show strong convexity only for sparse vectors, and not for all 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) . As was observed in prior works on regression with oblivious outliers, e.g. [dLN + 21], 𝜌 ( 𝑢 ) : = 𝐻 ˆ 𝑆 ( 𝛽 ∗ + 𝑢 ) -𝐻 ˆ 𝑆 ( 𝛽 ∗ ) - ⟨∇ 𝐻 ˆ 𝑆 , 𝑢 ⟩ can be lower bounded by 1 2 ˝ 𝑖 ∈ ˆ 𝑆 ⟨ 𝑋𝑖 , 𝑢 ⟩ 2 1 [|⟨ 𝑋𝑖 ,𝑢 ⟩-𝑦𝑖 | ⩽ 1 ] 1 [|⟨ 𝑋𝑖 ,𝑢 ⟩| ⩽ 1 ] . Let 𝐶 ( 𝑢 ) = 𝑆 good ∩ ˆ 𝑆 ∩ 𝐴 ∩ 𝐵 ( 𝑢 ) , where 𝐴 is the set of samples where | 𝜂 𝑖 | ⩽ 1 and 𝐵 ( 𝑢 ) = { 𝑖 ∈ [ 𝑛 ] | | ⟨ 𝑋𝑖 , 𝑢 ⟩| ⩽ 1 } . Then, 𝜌 ( 𝑢 ) ⩾ Ω ( ˝ 𝑖 ∈ 𝐶 ( 𝑢 ) ⟨ 𝑋 ∗ 𝑖 , 𝑢 ⟩ 2 ). It can be shown that for some suitable 𝑟 and for each 𝑘 -sparse 𝑢 of norm 𝑟 , 𝐶 ( 𝑢 ) is a large subset of the set 𝐴 (of size at least 0 . 99 | 𝐴 | ). Note that since 𝐴 is independent of 𝑋 ∗ , the rows of 𝑋 ∗ that correspond to indices from 𝐴 are just iid samples from 𝒟 . If 𝑋 ∗ 𝑖 were Gaussian, we could have applied concentration bounds and prove strong convexity via union bound argument over subsets of size 0 . 99 | 𝐴 | . In the heavy-tailed case, we need a different argument. For a fixed set 𝐶 of size 0 . 99 | 𝐴 | , we can use Bernstein's inequality 11 . We cannot use union bound argument over all subsets of size 0 . 99 | 𝐴 | (there are too many), but fortunately we do not need it since for each 𝑘 -sparse 𝑢 of norm 𝑟 , it is enough to show that ˝ 𝑖 ∈ 𝑇 ( 𝑢 ) ⟨ 𝑋 ∗ 𝑖 , 𝑢 ⟩ 2 ⩾ Ω ( 𝑟 2 ) , where 𝑇 ( 𝑢 ) ⊂ 𝐴 is the set of the smallest (in absolute value) 0 . 99 | 𝐴 | entries of the vector 𝑋 ∗ 𝐴 𝑢 ∈ R | 𝐴 | . Hence, we can use an epsilon-net argument for the set of 𝑘 -sparse vectors 𝑢 (of norm 𝑟 ). This set has very dense nets of

(relatively) small size, and this is enough to show the lower bound ˝ 𝑖 ∈ 𝐶 ( 𝑢 ) ⟨ 𝑋 ∗ 𝑖 , 𝑢 ⟩ 2 ⩾ Ω ( 𝑟 2 ) for all 𝑘 -sparse 𝑢 of norm 𝑟 with high probability, as long as 𝑛 ⩾ ˜ 𝑂 ( 𝑘 2 ) .

In order to show the same bound for all 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) of norm 𝑟 , we observe that 12 if a quadratic form is Θ ( 𝑟 2 ) on 𝐾 -sparse vectors of norm 𝑟 for some 𝐾 ≳ 𝑘 , then it is also Θ ( 𝑟 2 ) on all 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) , and applying the argument from the previous paragraph to 𝐾 -sparse vectors, we get the desired bound. We remark that directly proving it for 𝑢 ∈ ℰ 𝑘 ( 𝑟 ) is challenging, since we extensively used the properties of the set of sparse vectors that are not satisfied by ℰ 𝑘 ( 𝑟 ) , e.g. the existence of very dense epsilon-nets of small size.

3 Future Work

There is an interesting open problem in robust sparse regression that is not captured by our techniques. For sparse mean estimation, in the Gaussian case, there exists a polynomial time algorithm with nearly optimal guarantees: It achieves error 𝑂 ( ˜ 𝜀 ) with 𝑘 4 polylog ( 𝑑 )/ 𝜀 2 samples ([DKK + 22]). This algorithm uses a sophisticated sum-of-squares program 13 . It is reasonable to apply the techniques of [DKK + 22] to robust sparse regression in order to achieve nearly optimal error 𝑂 ( ˜ 𝜀 ) with poly ( 𝑘 ) samples. However, simple approaches (e.g. our approach with replacing the sparse constraints by the elastic constraints) fail in this case. Here we provide a high-level explanation of the issue. In order to combine the filtering algorithm with their techniques, we need to check whether the values of a certain quartic form are small on all sparse vectors. The analysis in [DKK + 22] shows that this form is indeed small for the uncorrupted sample with high probability (see their Lemma E.2.). Since we want the filtering algorithm to be efficient, we have to use a relaxation of sparse vectors. Hence we need to find a sum-of-squares (or some other nice relaxation) version of the proof from [DKK + 22]. However, in their proof they use a covering argument , and it is not clear how to avoid it. This argument fails for reasonable relaxations that we have thought about. Both potential outcomes (either an algorithm or a computational lower bound) are interesting: An algorithm would likely require new sophisticated ideas, and a lower bound would show a significant difference between robust sparse regression and robust mean estimation, while, so far, the complexity pictures of these problems have seemed to be quite similar.

√

Another interesting direction is to get error 𝑜 ( 𝜀 ) for distributions that do not necessarily have certifiably bounded moments. As was shown in [HL19], only moment assumptions (without certifiability) are not enough for efficient robust mean estimation, and the same should be true also for linear regression. However, other assumptions on distribution 𝒟 can make the problem solvable in polynomial time. For robust mean estimation, some symmetry assumptions are enough even for heavy-tailed distributions without the second moment 14 (see [NST23]). It is interesting to investigate what assumptions on the design distribution are sufficient for existence of efficiently computable estimators for robust sparse regression.

Acknowledgments and Disclosure of Funding

Chih-Hung Liu is supported by Ministry of Education, Taiwan under Yushan Fellow Program with the grant number MOE-111-YSFEE-0003-006-P1 and by National Science and Technology Council, Taiwan with the grant number 111-2222-E-002-017-MY2.

References

[BB20] Matthew S. Brennan and Guy Bresler, Reducibility and statistical-computational gaps from secret leakage , Proceedings of the 33rd Annual Conference on Learning Theory (COLT), vol. 125, 2020, pp. 648-847.

[BDLS17] Sivaraman Balakrishnan, Simon S. Du, Jerry Li, and Aarti Singh, Computationally efficient robust sparse estimation in high dimensions , Proceedings of the 30th Conference on Learning Theory COLT 2017, 2017, pp. 169-212.

[BJK15] Kush Bhatia, Prateek Jain, and Purushottam Kar, Robust regression via hard thresholding , NIPS, 2015, pp. 721-729.

[BJKK17] Kush Bhatia, Prateek Jain, Parameswaran Kamalaruban, and Purushottam Kar, Consistent robust regression , Advances in Neural Information Processing Systems (I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017.

[Che21] Yuansi Chen, An almost constant lower bound of the isoperimetric coefficient in the kls conjecture , 2021.

[DKK + 22] Ilias Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas, Robust sparse mean estimation via sum of squares , Proceedings of the 35th Annual Conference on Learning Theory (COLT22), Proceedings of Machine Learning Research, 2022, pp. 4703-4763.

[DKLP22] Ilias Diakonikolas, Daniel Kane, Jasper C. H. Lee, and Ankit Pensia, Outlier-robust sparse mean estimation for heavy-tailed distributions , Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 (Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, eds.), 2022.

[dKNS20] Tommaso d'Orsi, Pravesh K Kothari, Gleb Novikov, and David Steurer, Sparse pca: algorithms, adversarial perturbations and certificates , 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), IEEE, 2020, pp. 553-564.

[DKS17] Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart, Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures , Proceedings of the IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS17), 2017, pp. 73-84.

[DKS19] Ilias Diakonikolas, Weihao Kong, and Alistair Stewart, Efficient algorithms and lower bounds for robust linear regression , Proceedings of the 30th Annual Symposium on Discrete Algorithms (SODA19), 2019, pp. 2745-2754.

[dLN + 21] Tommaso d'Orsi, Chih-Hung Liu, Rajai Nasser, Gleb Novikov, David Steurer, and Stefan Tiegel, Consistent estimation for pca and sparse regression with oblivious outliers , Advances in Neural Information Processing Systems 34 (2021), 25427-25438.

[dNS21] Tommaso d'Orsi, Gleb Novikov, and David Steurer, Consistent regression when oblivious outliers overwhelm , Proceedings of the 38th International Conference on Machine Learning, (ICML 2021) (Marina Meila and Tong Zhang, eds.), 2021, pp. 2297-2306.

[DT19] Arnak S. Dalalyan and Philip Thompson, Outlier-robust estimation of a sparse linear model using ℓ 1 -penalized huber's 𝑀 -estimator , Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NeurIPS19), 2019, pp. 13188-13198.

[HL18] Samuel B. Hopkins and Jerry Li, Mixture models, robustness, and sum of squares proofs , Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018, 2018, pp. 1021-1034.

[HL19] , How hard is robust mean estimation? , Proceedings of the 32nd Annual Conference on Learning Theory (COLT19), 2019, pp. 1649-1682.

[KBJ00] Samuel Kotz, N. Balakrishnan, and Norman L. Johnson, Continuous multivariate distributions: Models and applications , Wiley, 2000.

[Kla23] Bo'az Klartag, Logarithmic bounds for isoperimetry and slices of convex sets , 2023.

[KMZ22] Pravesh K. Kothari, Peter Manohar, and Brian Hu Zhang, Polynomial-time sum-ofsquares can robustly estimate mean and covariance of gaussians optimally , Proceedings of The 33rd International Conference on Algorithmic Learning Theory (Sanjoy Dasgupta and Nika Haghtalab, eds.), Proceedings of Machine Learning Research, vol. 167, PMLR, 29 Mar-01 Apr 2022, pp. 638-667.

[KS17a] Pravesh K. Kothari and Jacob Steinhardt, Better agnostic clustering via relaxed tensor norms , CoRR abs/1711.07465 (2017).

[KS17b] Pravesh K. Kothari and David Steurer, Outlier-robust moment-estimation via sum-ofsquares , CoRR abs/1711.11581 (2017).

[Las01] Jean B. Lasserre, New positive semidefinite relaxations for nonconvex quadratic programs , Advances in convex analysis and global optimization (Pythagorion, 2000), Nonconvex Optim. Appl., vol. 54, Kluwer Acad. Publ., Dordrecht, 2001, pp. 319-331. MR 1846160

[LSLC20] Liu Liu, Yanyao Shen, Tianyang Li, and Constantine Caramanis, High dimensional robust sparse regression , Proceedings of the 23rd International Conference on Artificial Intelligence and stics AISTATS 2020, 2020, pp. 411-421.

[MNW22] Stanislav Minsker, Mohamed Ndaoud, and Lang Wang, Robust and tuning-free sparse linear regression via square-root slope , arXiv preprint arXiv:2210.16808 (2022).

[Nes00] Yurii Nesterov, Squared functional systems and optimization problems , High performance optimization, Appl. Optim., vol. 33, Kluwer Acad. Publ., Dordrecht, 2000, pp. 405-440. MR1748764

[NST23] Gleb Novikov, David Steurer, and Stefan Tiegel, Robust mean estimation without moments for symmetric distributions , Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 (Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, eds.), 2023.

[Par00] Pablo A Parrilo, Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization , Ph.D. thesis, California Institute of Technology, 2000.

[PJL20] Ankit Pensia, Varun S. Jog, and Po-Ling Loh, Robust regression with covariate filtering: Heavy tails and adversarial contamination , CoRR abs/2009.12976 (2020).

[Pré71] András Prékopa, Logarithmic concave measures with application to stochastic programming , Acta Scientiarum Mathematicarum (1971), 301-316.

[RH23] Philippe Rigollet and Jan-Christian Hütter, High-dimensional statistics , 2023.

[Sas22] Takeyuki Sasai, Robust and sparse estimation of linear regression coefficients with heavy-tailed noises and covariates , CoRR abs/2206.07594 (2022).

[SBRJ19] Arun Sai Suggala, Kush Bhatia, Pradeep Ravikumar, and Prateek Jain, Adaptive hard thresholding for near-optimal consistent robust regression , Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA (Alina Beygelzimer and Daniel Hsu, eds.), Proceedings of Machine Learning Research, vol. 99, PMLR, 2019, pp. 2892-2897.

[SF23] Takeyuki Sasai and Hironori Fujisawa, Outlier robust and sparse estimation of linear regression coefficients , CoRR abs/2208.11592 (2023).

[Sho87] N. Z. Shor, Quadratic optimization problems , Izv. Akad. Nauk SSSR Tekhn. Kibernet. (1987), no. 1, 128-139, 222. MR 939596

[Tho23] Philip Thompson, Outlier-robust sparse/low-rank least-squares regression and robust matrix completion , 2023.

[TJSO14] Efthymios Tsakonas, Joakim Jaldén, Nicholas D. Sidiropoulos, and Björn Ottersten, Convergence of the huber regression m-estimate in the presence of dense outliers , IEEE Signal Processing Letters 21 (2014), no. 10, 1211-1214.

[Tro15] Joel A. Tropp, An introduction to matrix concentration inequalities , Foundations and Trends in Machine Learning 8 (2015), no. 1-2, 1-230.

[ZWJ14] Yuchen Zhang, Martin J. Wainwright, and Michael I. Jordan, Lower bounds on the performance of polynomial-time algorithms for sparse linear regression , Proceedings of the 27th Annual Conference on Learning Theory (COLT14), 2014, pp. 921-948.

A Properties of the Huber loss minimizer

Definition A.1. For 𝑤 ∈ R 𝑛 ⩾ 0 , the weighted Huber loss function is defined as

For a penalty parameter 𝜆 , the ℓ 1-penalized Huber loss restricted to 𝑆 is defined as 𝐿𝑤 ( 𝛽 ) : = 𝐻𝑤 ( 𝛽 ) + 𝜆 · ∥ 𝛽 ∥ 1 .

Lemma A.2. Suppose that 𝑤 ∈ R 𝑛 ⩾ 0 , 𝑢 ∈ R 𝑑 , 𝛾 1 , 𝛾 2 , 𝜆 > 0 satisfy the following properties:

Then where 𝜎 min is the minimal eigenvalue of Σ .

Proof. Let 𝒦 = supp ( 𝛽 ∗ ) . Note that

By the convexity of 𝐻𝑤 ,

Hence

Therefore,

Theorem A.3. Let 𝜌 , 𝛾 1 , 𝛾 2 > 0 and

where 𝜎 min is the minimal eigenvalue of Σ . Let 𝑘 ′ ⩾ 100 𝑘 / 𝜎 min . Consider the 𝑘 ′ -elastic ellipsoid of radius 𝑟 : √

Suppose that the weights 𝑤 ∈ R 𝑛 are such that the following two properties hold:

Gradient bound: For all 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) ,

Let

Strong convexity on the boundary: For all 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) such that GLYPH<13> GLYPH<13> Σ 1 / 2 𝑢 GLYPH<13> GLYPH<13> = 𝑟 ,

Then the minimizer ˆ 𝛽 of the weighted penalized Huber loss with penalty 𝜆 and weights 𝑤 satisfies

Proof. Let ˆ 𝑢 = ˆ 𝛽 -𝛽 ∗ . If GLYPH<13> GLYPH<13> Σ 1 / 2 ˆ 𝑢 GLYPH<13> GLYPH<13> < 𝑟 , we get the desired bound. Otherwise, let 𝑢 be the (unique) point in the intersection of 𝜕 ℰ 𝑘 ′( 𝑟 ) and the segment [ 0 , ˆ 𝑢 ] ⊂ R 𝑑 . By convexity of the penalized loss,

Since 𝑢 ∈ 𝜕 ℰ 𝑘 ′( 𝑟 ) , either GLYPH<13> GLYPH<13> Σ 1 / 2 𝑢 GLYPH<13> GLYPH<13> = 𝑟 , or ∥ 𝑢 ∥ 1 = √ 𝑘 ′ · 𝑟 . Let us show that the latter is not possible. Since 𝜆 ⩾ 2 𝛾 1, we can apply Lemma A.2:

Cancelling 𝑟 and using the bound 𝜆 ⩾ 𝛾 2 · q 𝜎 min 𝑘 , we get a contradiction. Hence GLYPH<13> GLYPH<13> Σ 1 / 2 𝑢 GLYPH<13> GLYPH<13> = 𝑟 . By the strong convexity and the gradient bound,

Note that

By putting the above two inequality together and by Lemma A.2 , we have that

Dividing both sides by 𝜌 · 𝑟 , we get

a contradiction. Therefore, GLYPH<13> GLYPH<13> Σ 1 / 2 ˆ 𝑢 GLYPH<13> GLYPH<13> < 𝑟 .

B Heavy-tailed Designs

First, we define a bit more general model than Definition 1.1

Definition B.1 (Robust Sparse Regression with 2 Adversaries) . Let 𝑛, 𝑑, 𝑘 ∈ N such that 𝑘 ⩽ 𝑑 , 𝜎 > 0, 𝛼 ∈ ( 0 , 1 ] and 𝜀 ≲ 𝛼 . Let 𝒟 be a probability distribution in R 𝑑 with mean 0 and covariance

□

Σ . Let 𝑦 ∗ = 𝑋 ∗ 𝛽 ∗ + 𝜂 , where 𝑋 is an 𝑛 × 𝑑 random matrix with rows 𝑋 ∗ 𝑖 iid ∼ 𝒟 , 𝛽 ∗ ∈ R 𝑑 is 𝑘 -sparse, 𝜂 ∈ R 𝑛 is independent of 𝑋 ∗ and has at least 𝛼 · 𝑛 entries bounded by 𝜎 in absolute value 15 .

An instance of our model is a pair ( 𝑋, 𝑦 ) , where 𝑋 ∈ R 𝑛 × 𝑑 is a matrix and 𝑦 ∈ R 𝑛 is a vector such that there exists a set 𝑆 good ⊆ [ 𝑛 ] of size at least ( 1 -𝜀 ) 𝑛 such that for all 𝑖 ∈ 𝑆 good, 𝑋𝑖 = 𝑋 ∗ 𝑖 and 𝑦𝑖 = 𝑦 ∗ .

𝑖

Definition B.2. Let 𝑀 > 0, 𝑡 ∈ N , and let ℓ ⩾ 2 𝑡 be an even number. We say that a probability distribution 𝒟 in R 𝑑 with zero mean and covariance Σ has ℓ -certifiably 𝑀 -bounded ( 2 𝑡 ) -th moment , if there exist polynomials ℎ 1 , . . . , ℎ𝑚 ∈ R [ 𝑢 1 , . . . , 𝑢 𝑑 ] of degree at most ℓ / 2 such that

In this section we prove the following theorem

Theorem B.3 (Heavy-tailed designs, general formulation) . Let 𝑛, 𝑑, 𝑘, 𝑋, 𝑦, 𝜀 , 𝒟 , Σ , 𝜎 , 𝛼 be as in Definition B.1, and let 𝛿 ∈ ( 0 , 1 ) .

Suppose that for some 𝑠 > 2 , 𝑡 ∈ N , 𝑀𝑠, 𝑀 2 𝑡 ⩾ 1 , and even number ℓ ⩾ 2 𝑡 , 𝒟 has 𝑀𝑠 -bounded 𝑠 -th moment, and ℓ -certifiably 𝑀 2 𝑡 -bounded ( 2 𝑡 ) -th moment. In addition, 𝒟 has entrywise 𝜈 -bounded ( 4 𝑡 ) -th moment.

There exists an algorithm that, given 𝑋, 𝑦, 𝑘, 𝜀 , 𝜎 , 𝑀 2 𝑡 , ℓ , 𝑡 , 𝛿 and ˆ 𝜎 max such that ∥ Σ ∥ ⩽ ˆ 𝜎 max ⩽ 𝑂 (∥ Σ ∥) , in time ( 𝑛 + 𝑑 ) 𝑂 ( ℓ ) outputs 𝑋 ′ ∈ R 𝑛 × 𝑑 and weights 𝑤 = ( 𝑤 1 , . . . , 𝑤 𝑛 ) such that if

then with probability at least 1 -𝛿 , the weighted ℓ 1 -penalized Huber loss estimator ˆ 𝛽 𝑤 = ˆ 𝛽 𝑤 ( 𝑋 ′ , 𝑦 ) with weights 𝑤 (as in Definition 2.1) and parameter ℎ satisfies

Let us explain how this result implies Theorem 1.4 and Theorem 1.7.

Theorem 1.4 is a special case of Theorem B.3 with 𝑡 = 1, 𝑠 = 3, ℓ = 2, 𝑀 2 𝑡 = 1, 𝑀𝑠 = 𝑀 , 𝛼 = 0 . 01. Indeed, we only need to estimate ∥ Σ ∥ up to a constant factor. We can do it by estimating the variance of the first coordinate of 𝑥 ∼ 𝒟 . Applying median-of-means algorithm 16 to the first coordinate, we get an estimator ˜ 𝜎 2 that is 𝑂 ( 𝜈 2 ∥ Σ ∥ √ 𝜀 ) -close to the variance of the first coordinate 𝜎 2 1 . Note that ∥ Σ ∥/ 𝜅 ( Σ ) ⩽ 𝜎 2 1 ⩽ ∥ Σ ∥ . Since in Theorem 1.4 𝜅 ( Σ ) and 𝜈 are constants, and 𝜀 is sufficiently small, we get that 1 2 𝜅 ( Σ ) ∥ Σ ∥ ⩽ ˜ 𝜎 2 max ⩽ 2 ∥ Σ ∥ . Hence for a constant 𝐶 ⩾ 𝜅 ( Σ ) , ˆ 𝜎 max = 2 𝐶 ˜ 𝜎 2 is the desired estimator of ∥ Σ ∥ .

Similarly, Theorem 1.7 is a special case of Theorem B.3 with 𝑡 = 2, 𝑠 = 4, 𝑀 2 𝑡 = 𝑀𝑠 = 𝑀 , 𝛼 = 0 . 01. ∥ Σ ∥ can be estimated using the procedure described above.

Before proving the theorem, note that we can without loss of generality assume that 𝜎 = 1. Indeed, since 𝜎 is known, we can simply divide 𝑋 and 𝑦 by it before applying the algorithm.

B.1 Truncation

We cannot work with 𝑋 ∗ directly since it might have very large values, and Bernstein inequality that we use for random vectors concentration would give very bad bounds if we work with 𝑋 ∗ . Fortunately,

we can perform truncation. This technique was used in [DKLP22] for sparse mean estimation and in [Sas22] for sparse regression.

then the number of entries 𝑖 where GLYPH<10> 𝑋 ∗ 𝑖 , 𝛽 ∗ GLYPH<11> ≠ GLYPH<10> 𝑋 ′ 𝑖 ( 𝜏 ) , 𝛽 ∗ GLYPH<11> is at most 𝜀 𝑛 with probability at least 1 -2 -𝜀 𝑛 / 10 ⩾ 1 -𝛿 / 10. Hence in the algorithm we assume that the input is 𝑋 ′ ( 𝜏 ) instead of 𝑋 ∗ , and we treat the entries where GLYPH<10> 𝑋 ∗ , 𝛽 ∗ GLYPH<11> ≠ GLYPH<10> 𝑋 ′ ( 𝜏 ) , 𝛽 ∗ GLYPH<11> as corrupted by an adversary.

𝑖

𝑖

Concretely, further we assume that we are given GLYPH<8> GLYPH<0> 𝑋 ′′ 𝑖 ( 𝜏 ) , 𝑦 𝑖 , 𝑤 𝑖 GLYPH<1> GLYPH<9> 𝑛 𝑖 = 1 such that 𝑦 = 𝑋 ′ ( 𝜏 ) 𝛽 ∗ + 𝜂 + 𝜁 , where 𝑋 ′′ ( 𝜏 ) ∈ R 𝑛 × 𝑑 differs from 𝑋 ′ ( 𝜏 ) ∈ R 𝑛 × 𝑑 only in rows from the set 𝑆 bad ⊂ [ 𝑛 ] of size at most ˜ 𝜀 𝑛 (where 𝜀 ⩽ ˜ 𝜀 ⩽ 𝑂 ( 𝜀 ) ), 𝜁 ∈ R 𝑛 is an ˜ 𝜀 𝑛 -sparse vector such that supp ( 𝜁 ) ⊆ 𝑆 bad, 𝛽 ∗ ∈ R 𝑑 a 𝑘 -sparse vector, and 𝜂 ∈ R 𝑛 is oblivious noise such that at least 𝛼 𝑛 entries do not exceed 1 in absolute value.

In addition, we define

The weights for the Huber loss will be from 𝒲˜ 𝜀 .

Appendix E will discuss more properties of the truncation.

B.2 Gradient Bound

Lemma B.4. Let 𝑏, 𝛾 1 > 0 . Suppose that 𝑤 ∈ 𝒲˜ 𝜀 and 𝑢 ∈ R 𝑑 satisfy

and

Then

Proof. Denote 𝐹 ( 𝑤 ) = ˝ 𝑖 ∈[ 𝑛 ] ( 1 / 𝑛 -𝑤𝑖 ) 𝜙 ( 𝜂 𝑖 ) GLYPH<10> 𝑋 ′ 𝑖 ( 𝜏 ) , 𝑢 GLYPH<11> . It is a linear function of 𝑤 , so | 𝐹 ( 𝑤 )| is maximized in one of the vertices of the polytope 𝒲˜ 𝜀 . This vertex corresponds to set 𝑆𝑤 of size at least ( 1 - ˜ 𝜀 ) 𝑛 . That is, the weights of the entries from 𝑆𝑤 are 1 / 𝑛 , and outside of 𝑆𝑤 the weights are zero. It follows that

□

□

The following lemma provides a bound on 𝛾 1:

Lemma B.5. With probability at least 1 -𝛿 / 10 ,

Proof. It follows from Bernstein's inequality Fact I.1 and the fact that

where we used Corollary E.4.

B.2.1 Strong Convexity

Lemma B.6. Suppose that 𝛼 ⩾ 1000 ˜ 𝜀 , 𝜏 ≳ 1000 · 𝜈 2 · ∥ Σ ∥ √ 𝑘 ′′ / GLYPH<0> 𝑟 √ 𝜎 min GLYPH<1> and

where 𝑘 ′′ = 10 4 · 𝑘 ′ · p ∥ Σ ∥ . Then with probability 1 -𝛿 / 10 , for all 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) such that GLYPH<13> GLYPH<13> Σ 1 / 2 𝑢 GLYPH<13> GLYPH<13> = 𝑟 ,

Proof. Denote 𝐴 good = 𝑆 good ∩ 𝒜 , where 𝒜 is a set of entries 𝑖 such that | 𝜂 𝑖 | ⩽ 1. Note that 𝒜 is independent of 𝑋 ∗ . It follows that

where ˜ 𝑋𝑖 = 1 h ∥ 𝑋 ′ 𝑖 ( 𝜏 ) ∥ ⩽ 10 5 𝑠 /( 𝑠 -2 ) · 𝑀 𝑠 /( 𝑠 -2 ) 𝑠 √ ∥ Σ ∥· 𝑘 ′′ i 𝑋 ′ 𝑖 ( 𝜏 ) .

Denote 𝐹 ( 𝑤 ) = ˝ 𝑖 ∈ 𝐴 good 𝑤𝑖 GLYPH<10> ˜ 𝑋𝑖 , 𝑢 GLYPH<11> 2 1 [| ⟨ ˜ 𝑋𝑖 ,𝑢 ⟩| ⩽ 1 ] . It is a linear function of 𝑤 , so it is maximized in one of the vertices of the polytope 𝒲˜ 𝜀 . This vertex corresponds to set 𝑆𝑤 of size at least ( 1 - ˜ 𝜀 ) 𝑛 . That is, the weights of the entries from 𝑆𝑤 are 1 / 𝑛 , and outside of 𝑆𝑤 the weights are zero.

Hence we need a lower bound for ˝ 𝑖 ∈ 𝐴 ( 𝑢 ) GLYPH<10> ˜ 𝑋𝑖 , 𝑢 GLYPH<11> 2 , where

In order to bound ˝ 𝑖 ∈ 𝐴 ( 𝑢 ) GLYPH<10> ˜ 𝑋𝑖 , 𝑢 GLYPH<11> 2 for vectors 𝑢 from the elastic ball ℰ 𝑘 ′( 𝑟 ) , we first show that it is bounded for 𝑘 ′′ -sparse vectors 𝑢 ′ for some large enough 𝑘 ′′ . First we need to show that ˝ 𝑖 ∈ℐ GLYPH<10> ˜ 𝑋𝑖 , 𝑢 ′ GLYPH<11> 2 is well-concentrated for a fixed set ℐ . Concretely, we need the following lemma:

Lemma B.7. Suppose that 𝜏 ⩾ 1000 · 𝑀 2 𝑡 · 𝜈 2 · ∥ Σ ∥ √ 𝑘 ′′ / GLYPH<0> 𝑟 √ 𝜎 min GLYPH<1> for some 𝑘 ′′ ∈ N . Then for a fixed (independent of ˜ 𝑋 ) set ℐ of size

and for all 𝑘 ′′ -sparse vectors 𝑢 ′ ∈ R 𝑑 such that 𝑟 ⩽ ∥ Σ 1 / 2 𝑢 ′ ∥ ⩽ 2 𝑟 ,

with probability at least 1 -𝛿 .

Proof. First let us show that

Since for each set 𝒦 of size 𝑘 ′′ , E GLYPH<13> GLYPH<13> GLYPH<13> GLYPH<0> 𝑋 ′ 𝑖 ( 𝜏 ) GLYPH<1> 𝒦 GLYPH<13> GLYPH<13> GLYPH<13> 2 = ˝ 𝑗 ∈𝒦 E GLYPH<16> 𝑋 ′ 𝑖𝑗 ( 𝜏 ) GLYPH<17> 2 ⩽ 2 ∥ Σ ∥ 𝑘 ′′ , by Markov's inequality,

Denote 𝐵 = 10 5 𝑠 /( 𝑠 -2 ) · 𝑀 𝑠 /( 𝑠 -2 ) 𝑠 · 𝜅 ( Σ ) 𝑠 /( 2 𝑠 -4 ) p ∥ Σ ∥ · 𝑘 ′′ . By Hölder's inequality, for all vectors 𝑢 ′ ∈ R 𝑑 with support 𝒦 ,

where we used Lemma E.1 and the fact that ∥ 𝑢 ′ 𝑢 ′⊤ ∥ 1 ⩽ 𝑘 ′′ ∥ 𝑢 ′ 𝑢 ′⊤ ∥ ⩽ 𝑘 ′′ ∥ 𝑢 ∥ 2 . By Corollary E.5, E GLYPH<10> 𝑋 ∗ 𝑖 , 𝑢 ′ GLYPH<11> 2 -2 𝑟 2 / 10 6 ⩽ · E GLYPH<10> 𝑋 ′ 𝑖 ( 𝜏 ) , 𝑢 ′ GLYPH<11> 2 ⩽ E GLYPH<10> 𝑋 ∗ 𝑖 , 𝑢 ′ GLYPH<11> 2 + 2 𝑟 2 / 10 6 . Hence

For a fixed set 𝒦 of size 𝑘 ′′ and for all unit vectors 𝑢 ′ ∈ R 𝑑 with support 𝒦 , by Bernstein inequality for covariance Fact I.2, with probability 1 -𝛿 ,

In order to make this quantity smaller than 𝑟 2 / 1000, it is sufficient to take |ℐ| ≳ 10 10 𝑠 /( 𝑠 -2 ) 𝑀 2 𝑠 /( 𝑠 -2 ) 𝑠 𝜅 ( Σ ) 2 + 𝑠 /( 𝑠 -2 ) · 𝑘 ′′ log ( 𝑑 / 𝛿 ) .

By union bound over all subsets 𝒦 of [ 𝑑 ] of size 𝑘 ′′ , we get the desired bound. □

Let us bound the size of 𝐴 ( 𝑢 ) . 𝐴 good ∩ 𝑆𝑤 has size at least ( 𝛼 -3 ˜ 𝜀 ) 𝑛 ⩾ 0 . 997 𝛼 𝑛 . By Lemma B.7 and Lemma F.2, GLYPH<13> GLYPH<13> ˜ 𝑋𝑢 GLYPH<13> GLYPH<13> 2 ⩽ 1 . 1 · 𝛼 𝑛𝑟 2 , hence at most 3 𝛼 𝑛𝑟 2 / ℎ ⩽ 0 . 001 𝛼 𝑛 entries of ˜ 𝑋𝑢 can be greater than ℎ / 2. Therefore, | 𝐴 ( 𝑢 )| ⩾ 0 . 99 𝛼 𝑛 .

Let 𝑘 ′′ = min GLYPH<8> ⌈ 10 4 𝑘 ′ ∥ Σ ∥⌉ , 𝑑 GLYPH<9> . Recall that 𝒜 a set of entries 𝑖 such that | 𝜂 𝑖 | ⩽ 1, and 𝒜 is independent of 𝑋 ∗ . By union bound, the result of Lemma B.7 also holds for all sets ℐ that correspond to the bottom 0 . 99-fraction of entries of vectors GLYPH<0> ˜ 𝑋𝑢 ′′ GLYPH<1> 𝒜 , where 𝑢 ′′ are from an GLYPH<0> 1 / 𝑛 10 GLYPH<1> -net 𝒩 in the set of all 𝑘 ′′ -sparse vectors 𝑢 ′ such that GLYPH<13> GLYPH<13> Σ 1 / 2 𝑢 ′ GLYPH<13> GLYPH<13> = 1 . 01 𝑟 . Let 𝑢 ′ be an arbitrary 𝑘 ′′ -sparse vector such that GLYPH<13> GLYPH<13> Σ 1 / 2 𝑢 ′ GLYPH<13> GLYPH<13> = 1 . 01 𝑟 , and let 𝑢 ′′ = 𝑢 ′ + Δ 𝑢 be the closest vector in the net 𝒩 to 𝑢 ′ . It follows that

If 𝑘 ′′ = 𝑑 , we get the desired bound, since we can take 𝑢 ′ = 𝑢 . Otherwise, by Lemma B.7,

and we get the desired bound by Lemma F.2.

B.3 Putting everything together

First, we truncate the entries of 𝑋 and 𝑋 ∗ and obtain 𝑋 ′′ ( 𝜏 ) and 𝑋 ′ ( 𝜏 ) using some 𝜏 such that

where 𝑘 ′′ = 10 6 · 𝑘 · 𝜅 ( Σ ) . We discuss the choice of 𝜏 further in this subsection. Let us denote 𝜏 ′ = 𝜏 / p ∥ Σ ∥ .

Then we find the weights 𝑤 1 , . . . 𝑤 𝑛 using Algorithm C.1.

We will show all the conditions of Theorem A.3 are satisfied if

for some large enough absolute constant 𝐶 and

First let us show that the assumptions of Lemma B.4 are satisfied with 𝛾 1 ⩽ 100 · 𝑀 2 𝑡 p ∥ Σ ∥ · 𝜀 1 -1 /( 2 𝑡 ) / √ 𝑘 and 𝛾 2 ⩽ 10 𝑀 2 𝑡 p 𝜅 ( Σ ) .

First we bound 𝛾 2. Note that if 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) for 𝑘 ′ = 100 𝑘 / 𝜎 min, then ∥ 𝑢 ∥ 1 ⩽ 𝑘 ′′ ∥ 𝑢 ∥ . Hence if

□

then Lemma D.2 implies that for all 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) , with probability 1 -𝛿 / 10,

Lemma D.2 and Lemma C.2 imply that for all 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) , with probability 1 -𝛿 / 10,

Let us bound 𝛾 1. By Lemma B.5, if

then by with probability 1 -𝛿 / 10, 𝛾 1 ⩽ 100 · 𝑀 2 𝑡 √ 𝜅 ( Σ )· 𝜀 1 -1 /( 2 𝑡 ) √ 𝑘 / 𝜎 min .

The strong convexity holds by Lemma B.6 with probability 1 -𝛿 / 10 as long as

where we used the fact that 𝜀 ≲ 𝛼 and that 𝜏 ≳ p ∥ Σ ∥ · 𝜈 2 · √ 𝑘 ′′ / 𝜀 1 -1 2 𝑡 satisfies the assumption of that lemma.

Therefore, all the conditions of Theorem A.3 are satisfied and we attain the desired bound of 𝑂 GLYPH<18> 𝑀 2 𝑡 √ 𝜅 ( Σ ) 𝛼 · 𝜀 1 -1 2 𝑡 GLYPH<19> stated in Theorem B.3.

Now let us discuss the choice of 𝜏 . First we can find an estimator ˆ 𝜅 of 𝜅 ( Σ ) by plugging it into the formula

Then we can take 𝜏 ′ = 0 . 01 · GLYPH<16> 𝑛 ˆ 𝜅 𝑡 · 𝑘 𝑡 · 𝑡 log ( 𝑑 / 𝛿 ) GLYPH<17> 1 /( 2 𝑡 ) . Note that if we express 𝑛 in terms of ˆ 𝜅 and plug into the formula for 𝜏 ′ , we get that 𝜏 ′ is an increasing function of ˆ 𝜅 . Also note that ˆ 𝜅 ⩾ 𝜅 ( Σ ) . Hence both conditions are satisfied: 𝜏 : = √ ˆ 𝜎 max · 𝜏 ′ is larger than the required lower bound for it, and 𝑛 is larger than 10000 ( 𝜏 ′ ) 2 𝑡 · ( 𝑘 ′′ ) 𝑡 log ( 𝑑 / 𝛿 ) and 10000 𝜏 log ( 𝑑 / 𝛿 ) √ 𝑘 ′ / 𝜀 1 -1 /( 2 𝑡 ) as required.

C Filtering

We use the following system of elastic constraints with sparsity parameter 𝐾 ⩾ 1 and variables 𝑣 1 , . . . , 𝑣 𝑑 , 𝑠 1 , . . . , 𝑠 𝑑 :

  Note that the vectors from the elastic ball n 𝑣 ∈ R 𝑑 GLYPH<12> GLYPH<12> GLYPH<12> ∥ 𝑣 ∥ ⩽ 1 , ∥ 𝑣 ∥ 1 ⩽ √ 𝐾 o satisfy these constraints with 𝑠 𝑖 = sign ( 𝑣𝑖 ) . We will later discuss the corresponding sum-of-squares certificates in Appendix D.

Let 𝑎 > 0 be such that D 1 𝑛 ˝ 𝑛 𝑖 = 1 GLYPH<0> 𝑋 ∗ 𝑖 GLYPH<1> ⊗ 2 𝑡 , ˜ E 𝑣 ⊗ 2 𝑡 E ⩽ 𝑎 2 𝑡 .

Algorithm C.1 (Filtering algorithm) .

Assign weights 𝑤 1 = . . . 𝑤 𝑛 = 1 / 𝑛 .

Find a degree 2 ℓ pseudo-expectation ˜ E that satisfies 𝒜 𝐾 and maximizes GLYPH<10>˝ 𝑛 𝑖 = 1 𝑤𝑖 𝑋 ⊗ 𝑡 𝑖 , ˜ E 𝑣 ⊗ 𝑡 GLYPH<11> .

If GLYPH<10> 1 𝑛 ˝ 𝑛 𝑖 = 1 𝑋 ⊗ 2 𝑡 𝑖 , ˜ E 𝑣 ⊗ 2 𝑡 GLYPH<11> < 10 𝑡 𝑎 2 𝑡 , stop.

Compute 𝜏 𝑖 = GLYPH<10> 𝑋 ⊗ 2 𝑡 𝑖 , ˜ E 𝑣 ⊗ 2 𝑡 GLYPH<11> and reweight: 𝑤 ′ 𝑖 = ( 1 -𝜏 𝑖 ∥ 𝜏 ∥ ∞ ) · 𝑤𝑖 .

goto 2.

Lemma C.2. If at each step D 1 𝑛 ˝ 𝑛 𝑖 = 1 GLYPH<0> 𝑋 ∗ 𝑖 GLYPH<1> ⊗ 2 𝑡 , ˜ E 𝑣 ⊗ 2 𝑡 E ⩽ 𝑎 2 𝑡 , then the algorithm terminates in at most ⌈ 2 𝜀 𝑛 ⌉ steps, and the resulting weights satisfy ˝ 𝑛 𝑖 = 1 𝑤𝑖 ⩾ 1 -2 𝜀 .

To prove it, we will use the following lemma:

Lemma C.3. Assume that D 1 𝑛 ˝ 𝑛 𝑖 = 1 GLYPH<0> 𝑋 ∗ 𝑖 GLYPH<1> ⊗ 2 𝑡 , ˜ E 𝑣 ⊗ 2 𝑡 E ⩽ 𝑎 2 𝑡 , GLYPH<10> 1 𝑛 ˝ 𝑛 𝑖 = 1 𝑋 ⊗ 2 𝑡 𝑖 , ˜ E 𝑣 ⊗ 2 𝑡 GLYPH<11> ⩾ 10 𝑡 𝑎 2 𝑡 and

Proof of Lemma C.3. Note, that it is enough to show that

Further, recall that 𝑤 ′ 𝑖 = GLYPH<16> 1 -𝜏 𝑖 𝜏 max GLYPH<17> 𝑤𝑖 , so for all 𝑖 ∈ [ 𝑛 ] , 𝑤𝑖 -𝑤 ′ 𝑖 = 1 𝜏 max 𝜏 𝑖 𝑤 𝑖 . Hence is enough to show that

Since 𝑆 𝑔 and 𝑆𝑏 partition [ 𝑛 ] and

we can prove ˝ 𝑖 ∈ 𝑆 𝑔 𝜏 𝑖 𝑤 𝑖 < ˝ 𝑖 ∈ 𝑆𝑏 𝜏 𝑖 𝑤 𝑖 by showing that

Then

Note that

Proof of Lemma C.2. We will show that the algorithm terminates after at most ⌈ 2 𝜀 𝑛 ⌉ iterations. Assume that it does not terminate after 𝑇 = ⌈ 2 𝜀 𝑛 ⌉ iterations. Note that the number of entries of 𝑤 that are equal to 0 increases by at least 1 in every iteration. Hence, after 𝑇 iterations we have set

at least 𝜀 𝑛 entries of 𝑤 to zero whose index lies in 𝑆 𝑔 . By assumption that the algorithm did not terminate and Lemma C.3, it holds that

a contradiction.

Let 𝑇 be the index of the last iteration of the algorithm before termination. Then

D Sum-of-Squares Certificates

We use the standard sum-of-squares machinery, used in numerous prior works, e.g. [KS17a, KS17b, HL18, HL19, dKNS20, DKK + 22].

Let 𝑓 1 , 𝑓 2 , . . . , 𝑓 𝑟 and 𝑔 be multivariate polynomials in 𝑥 . A sum-of-squares proof that the constraints { 𝑓 1 ⩾ 0 , . . . , 𝑓 𝑚 ⩾ 0 } imply the constraint { 𝑔 ⩾ 0 } consists of sum-of-squares polynomials ( 𝑝𝑆 ) 𝑆 ⊆[ 𝑚 ] such that

We say that this proof has degree ℓ if for every set 𝑆 ⊆ [ 𝑚 ] , the polynomial 𝑝𝑆 Π 𝑖 ∈ 𝑆 𝑓 𝑖 has degree at most ℓ . If there is a degree ℓ SoS proof that { 𝑓 𝑖 ⩾ 0 | 𝑖 ⩽ 𝑟 } implies { 𝑔 ⩾ 0 } , we write:

ℓ

We provide degree 2 ℓ sum-of-squares proofs from the system 𝒜 𝐾 (see below) of ( 𝑛 + 𝑑 ) 𝑂 ( 1 ) constraints. The sum-of-squares algorithm (that appeared it [Sho87, Par00, Nes00, Las01]. See, e.g., Theorem 2.6. [DKK + 22] for the precise formulation) returns a linear functional ˜ E : R [ 𝑥 ] ⩽ 2 ℓ → R , that is called a degree 2 ℓ pseudo-expectation , that satisfies the constraints of 𝒜 𝐾 in time ( 𝑛 + 𝑑 ) 𝑂 ( ℓ ) . In particular, it means that once we prove in sum-of-squares of degree 2 ℓ that constraints 𝒜 𝐾 imply that some polynomial 𝑔 ( 𝑢 ) is non-negative, the value of the ˜ E returned by the algorithm on 𝑔 ( 𝑢 ) is also non-negative.

Recall the system 𝒜 𝐾 of elastic constraints in Equation (C.1) as follows:

Also recall that the vectors from the elastic ball n 𝑣 ∈ R 𝑑 GLYPH<12> GLYPH<12> GLYPH<12> ∥ 𝑣 ∥ ⩽ 1 , ∥ 𝑣 ∥ 1 ⩽ √ 𝐾 o satisfy these constraints with 𝑠 𝑖 = sign ( 𝑣𝑖 ) .





The following lemma is similar to Lemma 3.4 from [DKK + 22], but we prove it in using the elastic constraints. The derivation from the elastic constraints requires a bit more work.

Lemma D.1. For arbitrary polynomial 𝑝 ( 𝑣 ) = ˝ 1 ⩽ 𝑖 1 ,...,𝑖 𝑡 ⩽ 𝑑 𝑝 𝑖 1 ...𝑖 𝑡 · 𝑣𝑖 1 · · · 𝑣𝑖𝑡 of degree at most 𝑡 we have

Proof. Observe that 𝒜 𝐾 2 𝑠,𝑣 𝑠 𝑖 𝑣 𝑖 ⩾ 0, hence 𝒜 𝐾 4 𝑡 𝑠,𝑣 GLYPH<16> ˝ 𝑑 𝑖 = 1 𝑠 𝑖 𝑣 𝑖 GLYPH<17> 2 𝑡 ⩽ 𝐾 𝑡 . In addition, by note that, 𝑣𝑖 1 · · · 𝑣𝑖𝑡 ⩽ 𝑠 𝑖 1 𝑣𝑖 1 · · · 𝑠 𝑖 𝑡 𝑣𝑖𝑡 . It follows that

□

The following lemma shows that we can certify an upper bound on the value of the empirical moments (as polylinear functions) of truncated distribution 𝑍𝑖 ( 𝜏 ) on the vectors from the elastic ball.

Lemma D.2 (Certifiable bound on empirical moments) . Suppose that for some 𝑡, ℓ ∈ N and 𝑀 2 𝑡 ⩾ 1 ,

and for some 𝜈 ⩾ 1

then with probability at least 1 -𝛿 , for each degree 2 ℓ pseudo-expectation ˜ E that satisfies 𝒜 𝐾 ,

Proof. Consider the polynomial

By Lemma E.6 and the assumptions on 𝑛 and 𝜏 , its coefficients are bounded by

It follows that

Hence

Then

Proof.

Then

By Cauchy-Schwarz inequality for pseudo-expectations (see, for example, Fact A.2. from [DKK + 22]) we get the desired bound. □

E Properties of the truncation

As before, let 𝑋 ∗ 1 , . . . , 𝑋 ∗ 𝑛 be iid samples from 𝒟 .

For 𝜏 > 0, let 𝑋 ′ 𝑖𝑗 ( 𝜏 ) = 𝑋 ∗ 𝑖𝑗 1 h | 𝑋 ∗ 𝑖𝑗 | ⩽ 𝜏 i . In this section we prove some properties of 𝑋 ′ 𝑖𝑗 ( 𝜏 ) that we use in the paper. We start with the following lemma.

Lemma E.1. Suppose that for some 𝜈 ⩾ 1 ,

□

The following lemma shows that the moments of the truncated distribution are close to the moments of 𝑋 ∗ in ℓ ∞ -norm.

𝑖

Lemma E.2. Let 𝑡 ∈ N and suppose that for some 𝐵 > 0 and 𝑞 > 0 ,

It follows that

The following statement is a straightforward corollary of Lemma E.2 with 𝑞 = 𝑡 :

Corollary E.3. Let 𝑡 ∈ N and suppose that for some 𝐵 > 0 ,

Then

The following two statements are special cases of Corollary E.3 for 𝑡 = 1 and 𝑡 = 2.

Corollary E.4.

Then

Corollary E.5. Suppose that for some 𝜈 ⩾ 1 ,

The following lemma shows that the empirical mean of GLYPH<0> 𝑋 ′ 𝑖 ( 𝜏 ) GLYPH<1> ⊗ 𝑡 is close to E GLYPH<0> 𝑋 ∗ 1 GLYPH<1> ⊗ 𝑡 for an appropriate choice of 𝜏 and large enough 𝑛 .

Lemma E.6. Let 𝑡 ∈ N be and suppose that for some 𝜈 ⩾ 1

Then with probability 1 -𝛿 ,

Proof. It follows from Corollary E.3, Bernstein inequality Fact I.1, and a union bound over all 𝑑 𝑡 entries of E GLYPH<0> 𝑋 ∗ 1 GLYPH<1> ⊗ 𝑡 . □

□

F Properties of sparse vectors

Lemma F.1. Let Σ ∈ R 𝑑 × 𝑑 be a positive definite matrix, 𝑘 ′ , 𝑘 ′′ ∈ N , 𝑟, 𝛿 ⩾ 0 , and

If 𝑘 ′′ ⩾ 4 𝑘 ′ ∥ Σ ∥/ 𝛿 2 , then

Proof. Let us take some 𝑢 ∈ ℰ 𝑘 ′( 𝑟 ) . Without loss of generality assume that 𝑢 1 ⩾ 𝑢 2 ⩾ . . . ⩾ 𝑢𝑑 . Let's split indices { 1 , 2 , . . . , 𝑑 } into blocks 𝐵 1 . . . , 𝐵 ⌈ 𝑑 / 𝑘 ′′ ⌉ of size 𝑘 ′′ (the last block might be of smaller size). Let for each block 𝐵𝑖 , let

Since ˝ ⌈ 𝑑 / 𝑘 ′′ ⌉ 𝑖 𝑝𝑖 = 1 and 𝑢 = ˝ ⌈ 𝑑 / 𝑘 ′′ ⌉ 𝑖 𝑝𝑖 𝑢 𝐵 𝑖 / 𝑝𝑖 , it is sufficient to show that for all 𝑖 , ∥ Σ 1 / 2 𝑢𝐵𝑖 ∥/ 𝑝𝑖 ⩽ ( 1 + 𝛿 ) 𝑟 .

Note that for all 𝑗 ⩾ 2, since ∥ 𝑢𝐵𝑗 ∥ ⩽ √ 𝑘 ′′ ∥ 𝑢𝐵𝑗 ∥ ∞ and ∥ 𝑢𝐵𝑗 ∥ ∞ ⩽ 1 𝑘 ′′ ∥ 𝑢𝐵𝑗 -1 ∥ 1,

By the triangle inequality,

Hence

□

Lemma F.2. Let Σ ∈ R 𝑑 × 𝑑 be a positive definite matrix, and let 𝑋 ∈ R 𝑚 × 𝑑 be a matrix such that for some 𝑟 > 0 and 𝛿 ∈ ( 0 , 1 ) , for all 𝑘 ′′ -sparse vectors 𝑢 ′ such that 𝑟 ⩽ ∥ Σ 1 / 2 𝑢 ′ ∥ ⩽ 2 𝑟 ,

If 𝑘 ′′ ⩾ 4 𝑘 ′ ∥ Σ ∥/ 𝛿 2 , then for all 𝑢 such that ∥ Σ 1 / 2 𝑢 ∥ = 𝑟 and ∥ 𝑢 ∥ 1 ⩽ 𝑟 √ 𝑘 ′ ,

Proof. The inequlity 1 √ 𝑚 ∥ 𝑋𝑢 ∥ ⩽ ( 1 + 𝛿 ) 2 · 𝑟 ⩽ ( 1 + 4 𝛿 ) · 𝑟 follows from Jensen's inequality and Lemma F.1.

Let us show that ( 1 -4 𝛿 ) · 𝑟 ⩽ 1 √ 𝑚 ∥ 𝑋𝑢 ∥ . Let 𝐵 1 , . . . , 𝐵 ⌈ 𝑑 / 𝑘 ′′ ⌉ be blocks of indices as in the proof of Lemma F.1. It follows that

□

G Lower bounds

In this section we prove Statistical Query lower bounds. SQ lower bounds is a standard tool of showing computational lower bounds for statistical estimation and decision problems. SQ algorithms do not use samples, but have access to an oracle that can return the expectation of any bounded function (up to a desired additive error, called tolerance ). The SQ lower bounds formally show the trateoff between the number of queries to the oracle and the tolerance. The standard interpretation of SQ lower bounds relies on the fact that simulating a query with small tolerance using iid samples requires large number of samples. Hence these lower bounds are interpreted as a tradeoff between the time complexity (number of queries) and sample complexity (tolerance) of estimators. See [DKS17] for more details.

First we give necessary definitions. These definitions are standard and can be found in [DKS17].

Definition G.1 (STAT Oracle) . Let 𝒟 be a distribution over R 𝑑 . A statistical query is a function 𝑓 : R 𝑑 →[-1 , 1 ] . For 𝜏 > 0 the STAT ( 𝜏 ) oracle responds to the query 𝑓 with a value 𝑣 such that | 𝑣 -E 𝑋 ∼𝒟 𝑓 ( 𝑋 )| ⩽ 𝜏 . Parameter 𝜏 is called the tolerance of the statistical query.

Simulating a query STAT ( 𝜏 ) normally requires Ω ( 1 / 𝜏 2 ) iid samples from 𝒟 , hence SQ lower bounds provide a trade-off between the running time (number of queries) and the sample complexity ( Ω ( 1 / 𝜏 2 ) ).

Definition G.2 (Pairwise Correlation) . Let 𝒟 1 , 𝒟 2 , 𝒟 be absolutely continuous distributions over R 𝑑 , and suppose that supp (𝒟) = R 𝑑 . The pairwise correlation of 𝒟 1 and 𝒟 2 with respect to 𝒟 is defined as

where 𝑝 𝒟 1 ( 𝑥 ) , 𝑝 𝒟 2 ( 𝑥 ) , 𝑝 𝒟( 𝑥 ) are densities of 𝒟 1 , 𝒟 2 , 𝒟 respectively.

Definition G.3 (Chi-Squared Divergence) . Let 𝒟 ′ , 𝒟 be absolutely continuous distributions over R 𝑑 , and suppose that supp (𝒟) = R 𝑑 . The chi-squared divergence from 𝒟 ′ to 𝒟 is

Definition G.4 ( ( 𝛾 , 𝜌 ) -correlation) . Let 𝜌 , 𝛾 > 0, and let 𝒟 be a distribution over R 𝑑 . We say that a family of distributions ℱ over R 𝑑 is ( 𝛾 , 𝜌 ) -correlated relative to 𝒟 , if for all distinct 𝒟 ′ , 𝒟 ′′ ∈ ℱ , | 𝜒 𝐷 (𝒟 ′ , 𝒟 ′′ )| ⩽ 𝛾 and | 𝜒 𝐷 (𝒟 ′ , 𝒟 ′ )| ⩽ 𝜌 .

Fact G.5. Let 𝒟 be a distribution over R 𝑑 and ℱ be a family of distributions over R 𝑑 that does not contain 𝒟 , and consider a hypothesis testing problem of determining whether a given distribution 𝒟 ′ = 𝒟 or 𝒟 ′ ∈ ℱ .

Let 𝛾 , 𝜌 > 0 , 𝑠 ∈ N , and suppose that there exists a subfamily of ℱ of size 𝑠 that is ( 𝛾 , 𝜌 ) -correlated relative to 𝒟 . Then for all 𝛾 ′ > 0 , every SQ algorithm for the hypothesis testing problem requires queries of tolerance √ 𝛾 + 𝛾 ′ or makes at least 𝑠 𝛾 ′ /( 𝜌 -𝛾 ) queries.

We will also need the following facts:

√

Fact G.6 ([DKS17, Lemma 6.7]) . Let 𝑐 ∈ ( 0 , 1 ) and 𝑘, 𝑑 ∈ N be such that 𝑘 ⩽ 𝑑 . There exists a set 𝒱 ⊂ R 𝑑 of 𝑘 -sparse unit vectors of size 𝑑 𝑐𝑘 𝑐 / 8 such that for all distinct 𝑢, 𝑣 ∈ 𝒱 , ⟨ 𝑣, 𝑢 ⟩ ⩽ 2 𝑘 𝑐 -1 .

Fact G.7 ([DKS17, Lemma 3.4]) . Let 𝑚 ∈ N , and suppose that a distribution ℳ over R matches first 𝑚 moments of 𝑁 ( 0 , 1 ) . For a unit vector 𝑣 ∈ R 𝑑 let 𝒫 𝑣 be a distribution such that its projections onto the direction of 𝑣 has distribution ℳ , the projection onto the orthogonal complement is 𝑁 ( 0 , Id 𝑑 -1 ) , and these projections are independent. Then for all 𝑢, 𝑣 ∈ R 𝑑 ,

The following fact is a slight reformulation of Lemma E.4 from [DKS19]

Fact G.8 ([DKS19, Lemma E.4]) . Let 𝑦 ∼ 𝑁 ( 0 , 1 ) , 𝜇 0 > 0 , 𝑚 ∈ N , and 𝑔 : R → R . Let ℳ 𝜇 be a family of distributions over R satisfies the following properties:

ℳ = ( 1 -𝜀𝜇 ) 𝑁 ( 𝜇 , Θ ( 1 )) + 𝜀𝜇 ℬ 𝜇 for some 𝜀𝜇 and ℬ 𝜇 such that ℳ 𝜇 has the same first 𝑚 moments as 𝑁 ( 0 , 1 ) .

If | 𝜇 | ⩾ 10 𝜇 0 , then 𝜀𝜇 /( 1 -𝜀𝜇 ) ⩽ 𝑂 GLYPH<0> 𝜇 2 GLYPH<1> and 𝜒 2 (ℳ , 𝑁 ( 0 , 1 )) ⩽ 𝑒 𝑂 ( max { 1 / 𝜇 2 , 𝜇 2 }) .

If | 𝜇 | ⩽ 10 𝜇 0 , then 𝜀𝜇 = 𝜀 and 𝜒 2 (ℳ , 𝑁 ( 0 , 1 )) ⩽ 𝑔 ( 𝜀 ) .

For unit 𝑣 ∈ R 𝑑 let 𝒫 𝑣, 𝜇 be the same as 𝒫 𝑣 in Fact G.7 whose projection onto 𝑣 is ℳ 𝜇 . Let 𝒬 ′ 𝑣 be a distribution over R 𝑑 + 1 such that ( 𝑋, 𝑦 ) ∼ 𝒬 ′ 𝑣 satisfy the following properties: 𝑦 ∼ 𝑁 ( 0 , 1 ) , and 𝑋 | 𝑦 ∼ 𝒫 𝑣, 𝜇 0 · 𝑦 . Then for all unit 𝑢, 𝑣 ∈ R 𝑑 ,

where 𝒟 = 𝑁 ( 0 , Id 𝑑 + 1 ) .

√

Proposition G.9 (Formal version of Proposition 1.10) . Let 𝑘, 𝑑 ∈ N , 𝑘 ⩽ 𝑑 , 𝜀 ∈ ( 0 , 1 / 2 ) , 𝑐 ∈ ( 0 , 1 ) . For a vector 𝛽 ∗ ∈ R 𝑑 , and a number 𝜎 > 0 , consider the distribution 𝒢( 𝛽 ∗ , 𝜎 ) over R 𝑑 + 1 such that ( 𝑋, 𝑦 ) ∼ 𝒢( 𝛽 ∗ , 𝜎 ) satisfy 𝑋 ∼ 𝑁 ( 0 , Id ) and 𝑦 = ⟨ 𝑋, 𝛽 ∗ ⟩ + 𝜂 , where 𝜂 ∼ 𝑁 ( 0 , 𝜎 2 ) is independent of 𝑋 .

There exist a set ℬ ⊂ R 𝑑 of 𝑘 -sparse vectors, 0 . 99 ⩽ 𝜎 ⩽ 1 and a distribution 𝒬 over R 𝑑 + 1 , such that if an SQ algorithm 𝒜 given access to a mixture ( 1 -𝜀 )𝒢( 𝛽 ∗ , Σ , 𝜎 ) + 𝜀 𝒬 for 𝛽 ∗ ∈ ℬ , outputs ˆ 𝛽 ∗ such that ∥ 𝛽 ∗ - ˆ 𝛽 ∥ ⩽ 10 -5 , then 𝒜 either

makes 𝑑 𝑐𝑘 𝑐 / 8 · 𝑘 -2 + 2 𝑐 queries,

or makes at least one query with tolerance smaller than 𝑘 -1 + 𝑐 𝑒 𝑂 ( 1 / 𝜀 2 ) .

Proof. Note that ( 𝑋, 𝑦 ) ∼ 𝒢( 𝛽 ∗ , 𝜎 ) satisfy 𝑦 ∼ 𝑁 ( 0 , 𝜎 2 𝑦 ) , where 𝜎 2 𝑦 = ∥ 𝛽 ∗ ∥ 2 + 𝜎 2 and 𝑋 | 𝑦 ∼ 𝑁 GLYPH<16> 𝑦 𝜎 2 𝑦 𝛽 ∗ , Id -1 𝜎 2 𝑦 𝛽 ∗ 𝛽 ∗⊤ GLYPH<17> .

We will use vectors 𝛽 ∗ of norm 10 -5 . Denote 𝑣 = 𝛽 ∗ /∥ 𝛽 ∗ ∥ and let 𝜎 2 = 1 - ∥ 𝛽 ∗ ∥ 2 . Consider a distribution ℳ 𝜇 = ( 1 -𝜀 ) 𝑁 GLYPH<0> 𝜇 , 1 - ∥ 𝛽 ∗ ∥ 2 GLYPH<1> + 𝜀 𝑁 GLYPH<0> -1 -𝜀 𝜀 𝜇 , 1 GLYPH<1> . Note that 𝜒 2 GLYPH<0> ℳ 𝜇 , 𝑁 ( 0 , 1 ) GLYPH<1> ⩽ 𝑒 𝑂 ( max { 1 / 𝜇 2 , 𝜇 2 }) , and 𝜀 /( 1 -𝜀 ) ⩽ 𝑂 GLYPH<0> 𝜇 2 GLYPH<1> for 𝜇 ⩾ 10 -4 . Hence by Fact G.8,

for all unit 𝑣, 𝑢 ∈ R 𝑑 .

Using Fact G.6, we can apply Fact G.5 with 𝛾 = 𝑘 2 𝑐 -2 𝑒 𝑂 ( 1 / 𝜀 2 ) , 𝜌 = 𝑒 𝑂 ( 1 / 𝜀 2 ) , 𝛾 ′ = ( 𝜌 -𝛾 ) · 𝑘 -2 + 2 𝑐 , we get that 𝒜 requires at least 𝑑 𝑐𝑘 𝑐 / 8 𝑘 -2 + 2 𝑐 queries with tolerance greater than 𝑘 -1 + 𝑐 𝑒 𝑂 ( 1 / 𝜀 2 ) . □

for all unit 𝑣, 𝑢 ∈ R 𝑑 .

Using Fact G.6, we can apply Fact G.5 with 𝛾 = 𝑘 4 𝑐 -4 𝑒 𝑂 ( 1 / 𝜀 ) , 𝜌 = 𝑒 𝑂 ( 1 / 𝜀 ) , 𝛾 ′ = ( 𝜌 -𝛾 ) · 𝑘 -4 + 4 𝑐 , we get that 𝒜 requires at least 𝑑 𝑐𝑘 𝑐 / 8 𝑘 -4 + 4 𝑐 queries with tolerance smaller than 𝑘 -2 + 2 𝑐 𝑒 𝑂 ( 1 / 𝜀 ) . □

H Sub-exponential designs

Recall that a distribution 𝒟 in R 𝑑 is called 𝐿 -sub-exponential, if it has ( 𝐿𝑡 ) -bounded 𝑡 -th moment for each 𝑡 ∈ N . In particular, all log-concave distributions are 𝐿 -sub-exponential for some 𝐿 ⩽ 𝑂 ( 1 ) .

In this section we discuss how we can improve the dependence of the sample complexity on 𝜀 if (in addition to the assumptions of Theorem B.3) we assume that 𝒟 is 𝐿 -sub-exponential. For these designs we do not need a truncation.

First, let us show how the gradient bound Lemma B.5 modifies in this case. It can be obtained directly from Bernstein's inequality for sub-exponential distributions ([RH23, Theorem 1.13])

Lemma H.1. With probability at least 1 -𝛿 / 10 ,

The proof of strong convexity bound (Lemma B.6) is exactly the same, with 𝑋 ′ ( 𝜏 ) = 𝑋 ∗ .

Finally, we need to bound GLYPH<13> GLYPH<13> GLYPH<13> 1 𝑛 ˝ 𝑛 𝑖 = 1 GLYPH<0> 𝑋 ∗ 𝑖 GLYPH<1> ⊗ 2 𝑡 -E GLYPH<0> 𝑋 ∗ 1 GLYPH<1> ⊗ 2 𝑡 GLYPH<13> GLYPH<13> GLYPH<13> ∞ , since we need to prove Lemma D.2 for sub-exponential distributions. By Lemma C.1. from [DKK + 22], for all 𝐿 -sub-exponential distributions, with probability 1 -𝛿 ,

√

Proposition G.10 (Formal version of Proposition 1.11) . Let 𝑘, 𝑑 ∈ N , 𝑘 ⩽ 𝑑 , 𝜀 ∈ ( 0 , 1 / 2 ) , 𝑐 ∈ ( 0 , 1 ) . For a vector 𝛽 ∗ ∈ R 𝑑 , a positive definite matrix Σ and a number 𝜎 > 0 , consider the distribution 𝒢( 𝛽 ∗ , Σ , 𝜎 ) over R 𝑑 + 1 such that ( 𝑋, 𝑦 ) ∼ 𝒢( 𝛽 ∗ , Σ , 𝜎 ) satisfy 𝑋 ∼ 𝑁 ( 0 , Σ ) and 𝑦 = ⟨ 𝑋, 𝛽 ∗ ⟩ + 𝜂 , where 𝜂 ∼ 𝑁 ( 0 , 𝜎 2 ) is independent of 𝑋 .

There exist a set ℬ ⊂ R 𝑑 of 𝑘 -sparse vectors, 1 2 Id ⪯ Σ ⪯ Id , 0 . 99 ⩽ 𝜎 ⩽ 1 and a distribution 𝒬 over R 𝑑 + 1 , such that if an SQ algorithm 𝒜 given access to a mixture ( 1 -𝜀 )𝒢( 𝛽 ∗ , Σ , 𝜎 ) + 𝜀 𝒬 for 𝛽 ∗ ∈ ℬ , outputs ˆ 𝛽 such that ∥ 𝛽 ∗ - ˆ 𝛽 ∥ ⩽ 10 -5 √ 𝜀 , then 𝒜 either

makes 𝑑 𝑐𝑘 𝑐 / 8 · 𝑘 -4 + 4 𝑐 queries,

or makes at least one query with tolerance at least 𝑘 -2 + 2 𝑐 𝑒 𝑂 ( 1 / 𝜀 ) .

Proof. Note that ( 𝑋, 𝑦 ) ∼ 𝒢( 𝛽 ∗ , Σ , 𝜎 ) satisfy 𝑦 ∼ 𝑁 ( 0 , 𝜎 2 𝑦 ) , where 𝜎 2 𝑦 = 𝛽 ∗⊤ Σ 𝛽 ∗ + 𝜎 2 and 𝑋 | 𝑦 ∼ 𝑁 GLYPH<16> 𝑦 𝜎 2 𝑦 Σ 𝛽 ∗ , Σ -1 𝜎 2 𝑦 ( Σ 𝛽 ∗ )( Σ 𝛽 ∗ ) ⊤ GLYPH<17> .

Wewill use vectors 𝛽 ∗ of norm 10 -5 √ 𝜀 . Denote 𝑣 = 𝛽 ∗ /∥ 𝛽 ∗ ∥ and let 𝜎 2 = 1 -𝛽 ∗⊤ Σ 𝛽 ∗ , Σ = Id -𝑐 ′ 𝑣𝑣 ⊤ , where 𝑐 ′ is a constant such that

By [DKS19, Lemmas E.2], there exists a distribution ℳ that satisfies the assumption of Fact G.8 with 𝑚 = 3 and 𝑔 ( 𝜀 ) = 𝑒 𝑂 ( 1 / 𝜀 ) . Hence by Fact G.8,

Hence with 𝑛 ≳ 𝐾 2 𝑡 · GLYPH<0> 10 𝑡 2 log ( 𝑑 / 𝛿 ) GLYPH<1> 4 𝑡 + 1 , we get GLYPH<13> GLYPH<13> GLYPH<13> 1 𝑛 ˝ 𝑛 𝑖 = 1 GLYPH<0> 𝑋 ∗ 𝑖 GLYPH<1> ⊗ 2 𝑡 -E GLYPH<0> 𝑋 ∗ 1 GLYPH<1> ⊗ 2 𝑡 GLYPH<13> GLYPH<13> GLYPH<13> ∞ ⩽ 𝐿 2 𝑡 ∥ Σ ∥ 𝑡 / 𝐾 𝑡 , so we get the conclusion of Lemma D.2.

Putting everything together, the sample complexity is

Note that we can use 𝑠 = 4 and 𝑀𝑠 = 4 𝐿 .

Consider the case when 𝒟 is log-concave, so 𝐿 ⩽ 𝑂 ( 1 ) . For 𝜅 ( Σ ) ⩽ 𝑂 ( 1 ) , 𝛼 ⩾ Ω ( 1 ) , 𝑡 = 1, with high probability we get error 𝑂 ( 𝜎 √ 𝜀 ) as long as

Similarly, for 𝜅 ( Σ ) ⩽ 𝑂 ( 1 ) , 𝛼 ⩾ Ω ( 1 ) , 𝑡 = 2, with high probability we get error 𝑂 ( 𝑀 𝜎𝜀 3 / 4 ) (where 𝑀 ⩽ 𝑂 ( p log 𝑑 ) is the same as in Theorem 1.7) as long as

Note that for sub-Gaussian distributions one can use better tail bounds and the polylog ( 𝑑 ) factors should be better in this case.

I Concentration Bounds

Throughout the paper we use the following versions of versions of Bernstein's inequality. The proofs can be found in [Tro15].

Fact I.1 (Bernstein inequality) . Let 𝐿 > 0 and let 𝑥 ∈ R 𝑑 be a zero-mean random variable. Let 𝑥 1 , . . . , 𝑥 𝑛 be i.i.d. copies of 𝑥 . Suppose that | 𝑥 | ⩽ 𝐿 . Then the estimator ¯ 𝑥 = 1 𝑛 ˝ 𝑛 𝑖 = 1 𝑥𝑖 satisfies for all 𝑡 > 0

Fact I.2 (Bernstein inequality for covariance) . Let 𝐿 > 0 and let 𝑥 ∈ R 𝑑 be a 𝑑 -dimensional random vector. Let 𝑥 1 , . . . , 𝑥 𝑛 be i.i.d. copies of 𝑥 . Suppose that ∥ 𝑥 ∥ 2 ⩽ 𝐿 . Then the estimator ¯ Σ = 1 𝑛 ˝ 𝑛 𝑖 = 1 𝑥𝑖 𝑥 ⊤ 𝑖 satisfies for all 𝑡 > 0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract, Section 1, Section 1.1 and Section 2.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 1.1 and Section 2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Sections 1.1 and 2 and Appendices A to I

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theory paper, and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theory paper, and does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theory paper, and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theory paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theory paper, and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors provide a computationally efficient estimator for robust and sparse linear regression under non-isotropic covariance matrices. Their first result achieves $O(\sqrt{\epsilon})$ error with state-of-the-art sample complexity under a weaker noise assumption than prior work. Their second result is the first to achieve $o(\sqrt{\epsilon})$ error for robust and sparse linear regression under non-isotropic covariance matrices, under a sum-of-squares certificate for the $4^{th}$ moment. They further provide Statistica Query Lower bounds supporting their results.

STRENGTHS:
- They improve the state-of-the-art result for $O(\sqrt{\epsilon})$ error in terms of the sample complexity being independent of the $l-2$ norm of the true weight vector, $\beta_{*}$
- They provide the first result achieving $o(\sqrt{\epsilon})$ error, under a suitable sum-of-squares certificate for the $4^{th}$ moment.
- They provide SQ lower bounds which suggest that their results might be tight for polynomial time estimators

WEAKNESSES:
I don't see any weaknesses per se, was just curious to understand if the authors believe that the sum-of-squares assumption is necessary to achieve $o(\sqrt{\epsilon})$ error or a weaker assumption such as Definition 1.6 suffices.

QUESTIONS:
Small typo on Line 110 - ehtr

See weakness section.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies sparse linear regression $y^* = X^* \\beta^* + \\eta$ (where $\\beta^*$ is $k$-sparse) in the presence of two types of adversaries. \n\n- First, the noise vector $\\eta$ is sampled before $X^*$ in an (obliviously) adversarial way. Then $X^*$ is sampled independently of $\\eta$ with i.i.d. rows and $y^*$ is generated according to the model. \n\n- Second, an adaptive adversary takes $(X^*, y^*)$ and (adaptively) corrupts at most $\\epsilon$ fraction of $(X_i^*, y_i^*)$. \nThe resulting samples given to the statistician are denoted by $(X, y)$. \n\nThe results are two-fold. \n\n- First, a poly-time algorithm for estimating $\\beta^*$ up to $\\epsilon$ expected prediction error using $n \\approx k^2/\\epsilon^2$ samples. \n\n- Second, a poly-time algorithm for estimating $\\beta^*$ up to $o(\\epsilon)$ expected prediction error using $n \\approx k^4/\\epsilon^6$ samples. \nFor correlated Gaussian design, this can be improved to $n\\approx k^4/\\epsilon^4$.\n\nSTRENGTHS:\nNA\n\nWEAKNESSES:\nNA\n\nQUESTIONS:\nI'm not familiar with the SoS machinery, so the authors may find my comments shallow and unconstructive. \nThat said, I am familiar with ordinary (sparse) linear regression and I do think that the presentation of the paper should be improved. \n\nMajor comments: \n\n1. The sample complexity bounds are written in a wacky way. \nWhy don't we fix the goal to be achieving $\\epsilon$ error and write the sample complexity bounds accordingly?\n\n1. The two results in the paper (for achieving $O(\\sqrt{\\epsilon})$ and $o(\\sqrt{\\epsilon})$ errors) seem to have very discontinuous transitions as $\\epsilon$ varies. \nCan the authors comment on whether it's expected to be so or it's a proof artifact?\nIn fact, should I think of $\\epsilon$ as potentially depending on $n,k,d$ in an arbitrary way? \nThis is not clear since sometimes the authors view $\\epsilon$ as a constant in which case $O(\\epsilon), o(\\epsilon)$ don't make sense to me. \n\n1. I suggest the authors add a short section discussing related work at the level of techniques (e.g., filtering, SoS, etc.). \nCurrently, these techniques and the difference between the present paper and prior work are buried in Section 2 which itself is written in an obscure way. \n\nMinor comments:\n\n1. Line 48-51 seems out of place.\n\n1. Please unify the notation $N(0,\\mathrm{Id}), N(0,1)^n$. \n\n1. Multiple typos in line 110. \n\n1. Line 114, should $X$ be $X^*$?\n\n1. Typo in line 234. \n\n1. I don't understand line 243-244. \nPlease define \"simulation\". \nMy shallow understanding is that SQ lower bounds only apply to detection version of the problem. \nI'm not sure if this has the same spirit as \"simulation\". \n\n1. Section 2 (Techniques) is written in a very informal and dense way. \nAs someone who's not familiar with these techniques, I couldn't get much out of it. \n\n1. Typo in line 259, $\\eta\\sim N(0,\\mathrm{Id})$. \n\n1. Typo in line 264, it --> in. \n\n1. Line 291, please provide a reference for the sentence in the parentheses. \n\n1. Is line 298 the only property one needs from the output of filtering?\nIf so, why not say it upfront in line 266?\n\n1. Line 364, 365, there're two \"only\"'s in the sentence. \n\n1. I don't understand line 371, even grammatically. \n\n1. A very minor comment on the writing. I feel that the language used in this paper is quite informal and the writing was perhaps done in a hasty. \nFor instance, there're ~20 \"only\", ~20 \"even\", ~10 \"likely\"...\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
It developed efficient estimators for sparse linear regression in the presence of oblivious and adaptive adversaries. It presents a robust algorithm that outperforms sota under desired conditions on the moment of the distribution.

STRENGTHS:
- The paper introduces several robust algorithms that outperform the sota in sparse linear regression.

- It also gives analysis of weighted penalized Huber loss suitable for heavy-tailed designs.

- The paper includes theoretical results with proofs and assumptions, which are clearly stated.

WEAKNESSES:
- algorithms may require significant computational resources, which could limit practical applications.

- There is a lack of empirical evidence comparison with the sota even on a toy setup.

QUESTIONS:
NA.

LIMITATIONS:
The results depend on specific assumptions about the distribution and moments of the data, which may not always hold in real-world scenarios.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on robust estimation of sparse linear regression in the presence of both oblivious and adaptive adversaries. It claims to offer polynomial-time algorithms capable of recovering a sparse coefficient vector with high probability. The paper includes theoretical analyses using the standard techniques such as pre-filtering, truncation to ensure robustness.\n\nSTRENGTHS:\n- The authors claim to deal with two adversaries, which can be seen as a generalization of dealing with one adversary. \n\n- The theoretical analysis seems to be rigorous. \n\n- The analysis also covers heavy tails under the standard assumptions on the moments made in the existing literature. \n\n- The literature review is good.\n\nWEAKNESSES:\nAs per my understanding, the oblivious and adaptive adversary are dealt in literature separately. \n\n- For the oblivious adversary, the existing literature can handle constant outlier proportions and deliver consistent estimates. \n\n- For adaptive adversary, existing literature [PFL20] already has the results which are proposed in the submission. They use a trimmed MLE algorithm whose convergence rates are analyzed. Further, [Sas22] and [SF23] already extended these results for sparse settings (line 264). \n\nSo, as per my understanding, the authors claim bounds similar to those in the existing literature for a slightly more complex setting of two adversaries. \n\n- Why are few entries of $\\eta$ required to be close to $1$ in magnitude? Doesn't that make the oblivious adversary weak? \n\n- The authors use pre-filtering ideas, assuming a known covariance matrix and truncation to deal with heavy tails. They use standard techniques like sum-of-squares and weighted Huber loss minimization with $\\ell_1$ regularization. \n\nThese ideas are well explored in the literature. So, strictly speaking, there is a lack of major novelty in new ideas to deal with outliers and encourage robustness.  \n\n- More importantly, $O(\\sqrt{\\varepsilon})$ error is already proved to be minimax optimal in the existing literature even when $n \\to \\infty$ while using techniques like trimming, pre-filtering, or truncation. A more interesting question could be under what conditions on $\\varepsilon \\neq 0$ or adversary can we achieve zero error for at least $n \\to \\infty$.\n\nQUESTIONS:\n- Is the oblivious adversary allowed to change all the $n$ samples, unlike the adaptive adversary, which only changes $\\varepsilon*n$ samples? \n\n- If the oblivious adversary is also restricted to $\\varepsilon * n$ samples, it should be stated in line 6. Also, are the oblivious and adaptive adversaries changing the same $\\varepsilon * n$ samples? \n\n- The authors claim that their algorithm achieves $o(\\sqrt{\\varepsilon})$ error for all log-concave distributions if $\\varepsilon \\leq 1/\\text{polylog}(d)$. Hence, $\\varepsilon$ can be very small for large $d$. Can this upper bound on $\\varepsilon$ be improved? \n\n- The condition number of the covariance matrix is assumed to be bounded. Is that assumed in the existing literature or a new assumption in this submission? \n\n- Line 110 typo: 'ehrtr' and from 's' more \n\n- Line 234 typo: 'ghdsample' complexity\n\nLIMITATIONS:\n- In Definition 2.1, why was $|x_i| \\leq 2$ chosen as the threshold to define the quadratic and linear part of Huber loss? In general, it could be dependent on the data? For example, there could be some dataset where $|x_i| \\leq 3$ may be more appropriate. \n\n- The above could affect the upper bound on the gradient $|\\phi(\\eta)|$ taken as 2 in line 288. \n\n- A conclusion section and pointers to future work seem to be missing. \n\n- Towards the end of the main manuscript, the focus seems too much on the proof sketch. The writing or presentation can be improved. \n\n-  Some sort of empirical justification for the proposed results on synthetic or real datasets would have been really nice. This would have also helped to draw attention to the significance of the contributions of polynomial time algorithms. In Proposition 1.10 and 1.11, the number of queries is claimed as $\\exp(d^{\\Omega(1)})$, which does not seem to be very useful for practitioners.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies sparse linear regression with oblivious and adaptive adversaries. The design matrix is random, the noise is chosen by an adversary, and the goal is to find the optimal k-sparse weight vector. The results are as follows: to achieve $O(\\sqrt{\\varepsilon})$ error, a sample complexity of $\\widetilde{O}(k^2/\\varepsilon)}$ is needed, and to achieve $O(\\varepsilon^{3/4})$ error, a sample complexity of $\\widetilde{O}(k^4/\\varepsilon^3)$ is needed. Previous work did not achieve error better than $\\sqrt{\\varepsilon}$ with any sample complexity better than $d^2$, unless the rows of a design matrix are drawn from an isotropic Gaussian distribution.\n\nTo obtain their results, the authors make the following assumptions: (1) the noise vector has at least $0.01n$ entries that are bounded by $\\sigma$, where $n$ is the number of examples, (2) at most an $\\varepsilon$ fraction of rows of $X$/entries of $y$ are corrupted, and (3) the rows of $X$ have bounded 3rd moments and entrywise-bounded 4th moments, i.e. the rows of X can be heavy-tailed random vectors. \n\nIn this setting, they obtain two results:\n1. With $O(k^2/\\varepsilon)$ samples, they obtain $O(\\sqrt{\\varepsilon})$ error. Compared to the best prior work for heavy-tailed design matrices, the advantage of this result is that there is no dependence on the norm of the optimal k-sparse regression vector.\n2. With $k^4/\\varepsilon^3$ samples, they obtain $O(\\varepsilon^{3/4})$ error. This result additionally assumes that the distribution of the rows of the design matrix has a bounded 4th moment, with a constant-degree sum-of-squares proof of the bound. It also additionally assumes that the rows of the design matrix have entrywise-bounded 8th moment.\nAdditionally, they give matching statistical query lower bounds.\n\nThe algorithm consists of obtaining a filtered set of examples, then minimizing Huber loss with l1 regularization, on the remaining examples. In the analysis, the authors show that if the following two conditions:\n(1) A bound on the gradient of the loss function\n(2) A condition similar to strong convexity\nhold on the boundary of certain regions they call \"elastic balls\" (the elastic ball of radius $r$ has points with $\\ell_2$ norm at most $r$ and $\\ell_1$ norm at most $\\sqrt{k} \\cdot r$) then the weight vector they obtain will be similar to the true optimal weight vector. \n\nOne of the key steps in this analysis, compared to prior works is as follows: instead of using Cauchy-Schwarz to bound certain terms in the gradient bound, they use Holder's inequality. This leads to them having to bound the sum of $\\langle X_i, u \\rangle^4$, as $X_i$ ranges over the examples in the filtered set, and $u$ can be any vector in the elastic ball of radius 1. Thus, during the algorithm, they must select $\\widehat{S}$ so that it satisfies this property, and thus need an efficient algorithm to find such a set of size $(1 - O(\\varepsilon))n$. They do this using a sum-of-squares relaxation with elastic constraints, which makes it so the relaxation set is a superset of $u^{\\otimes 4}$ for $u$ in the elastic ball.\n\nFor the heavy-tailed case, it is also necessary to do some thresholding of X, which may have large entries even without adversarial noise. In previous works, the thresholding parameter needs to depend polynomially on the norm of the weight vector (to preserve the relationship between the un-corrupted X and y). This paper observes that in order to preserve the relationship between X and y, it is only necessary to analyze the effect of thresholding for the entries that are in the support of the optimal k-sparse regression vector $\\beta^*$. This allows the thresholding parameter to only depend on $k$ and $\\varepsilon$ and not the norm of $\\beta^*$.\n\nThe proof strategy for showing the strong convexity property for heavy-tailed design matrices is also interesting. They first show it for vectors $u$ in the elastic balls which are $k$-sparse, then extend to dense vectors $u$. They show the bound for all k-sparse vectors using Bernstein's inequality, and a small $\\epsilon$-net for the k-sparse vectors. To extend to dense vectors, they use the fact that if a quadratic form is $\\Theta(r^2)$ on $k$-sparse vectors of norm $r$, then the quadratic form is $\\Theta(r^2)$ on the elastic ball of radius $r$.\n\nSTRENGTHS:\n- The results are very strong compared to previous works.\n- The technical tools used are very interesting.\n\nWEAKNESSES:\n- While the writing is mostly good, it would be nice to organize the techniques section better. It would be useful for readers if you explicitly write the algorithm in a separate box, and divide the techniques section into some subsections.\n\nQUESTIONS:\n- On line 278, what is $\\hat{\\beta}$? I thought the final estimator you return is $\\hat{\\beta}_{\\hat{S}}$. Do you mean to refer to the ground-truth $k$-sparse regression vector?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-22"}