Title: SQ Lower Bounds for Non-Gaussian Component Analysis with Weaker Assumptions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Non-Gaussian Component Analysis
Abstract: We study the complexity of Non-Gaussian Component Analysis (NGCA) in the Statistical Query (SQ) model.
Prior work developed a methodology to prove SQ lower bounds for NGCA that have been applicable to a wide range of contexts.
In particular, it was known that for any univariate distribution $A$ satisfying certain conditions,
distinguishing between a standard multivariate Gaussian and a distribution that behaves like $A$ in a random hidden direction and like a standard Gaussian in the orthogonal complement, is SQ-hard.
The required conditions were that (1) $A$ matches many low-order moments with a standard Gaussian,
and (2) the chi-squared norm of $A$ with respect to the standard Gaussian is finite.
While the moment-matching condition is clearly necessary for hardness, the chi-squared condition was only required for technical reasons.
In this work, we establish that the latter condition is indeed not necessary.
In particular, we prove near-optimal SQ lower bounds for NGCA under the moment-matching condition only.

SQ Lower Bounds for Non-Gaussian Component Analysis with Weaker Assumptions

Abstract

We study the complexity of Non-Gaussian Component Analysis (NGCA) in the Statistical Query (SQ) model. Prior work developed a general methodology to prove SQ lower bounds for this task that have been applicable to a wide range of contexts. In particular, it was known that for any univariate distribution A satisfying certain conditions, distinguishing between a standard multivariate Gaussian and a distribution that behaves like A in a random hidden direction and like a standard Gaussian in the orthogonal complement, is SQ-hard. The required conditions were that (1) A matches many low-order moments with the standard univariate Gaussian, and (2) the chi-squared norm of A with respect to the standard Gaussian is finite. While the moment-matching condition is necessary for hardness, the chi-squared condition was only required for technical reasons. In this work, we establish that the latter condition is indeed not necessary. In particular, we prove near-optimal SQ lower bounds for NGCA under the moment-matching condition only. Our result naturally generalizes to the setting of a hidden subspace. Leveraging our general SQ lower bound, we obtain near-optimal SQ lower bounds for a range of concrete estimation tasks where existing techniques provide sub-optimal or even vacuous guarantees.

1 Introduction

Non-Gaussian Component Analysis (NGCA) is a statistical estimation task first considered in the signal processing literature [BKS + 06]. As the name suggests, the objective is to find a non-gaussian direction (or, more generally, low-dimensional subspace) in a high-dimensional dataset. Since its introduction, the NGCA problem has been studied in a range of works from an algorithmic standpoint; see [TV18, GS19] and references therein. Here we explore this problem from a hardness perspective in the Statistical Query (SQ) model. Before we motivate and state our results, we require basic background on the SQ model.

SQ Model SQ algorithms are a class of algorithms that are allowed to query expectations of bounded functions on the underlying distribution through the SQ oracle rather than directly access samples. The model was introduced by Kearns [Kea98] as a natural restriction of the PAC model [Val84] in the context of learning Boolean functions. Since then, the SQ model has been extensively studied in a range of settings, including unsupervised learning [Fel16]. The class of SQ algorithms is broad and captures a range of known algorithmic techniques in machine learning in-

cluding spectral techniques, moment and tensor methods, local search (e.g., EM), and many others (see, e.g., [FGR + 17, FGV17] and references therein).

Definition 1.1 (SQ Model) . Let D be a distribution on R n . A statistical query is a bounded function q : R n → [ -1 , 1] . We define STAT( τ ) to be the oracle that given any such query q , outputs a value v such that | v -E x ∼ D [ q ( x )] | ≤ τ , where τ > 0 is the tolerance parameter of the query. A statistical query (SQ) algorithm is an algorithm whose objective is to learn some information about an unknown distribution D by making adaptive calls to the corresponding STAT( τ ) oracle.

The following family of high-dimensional distributions forms the basis for the definition of the NGCA problem.

Definition 1.2 (Hidden-Subspace Distribution) . For a distribution A supported on R m and a matrix V ∈ R n × m with V ⊺ V = I m , we define the distribution P A V supported on R n such that it is distributed according to A in the subspace span( v 1 , . . . , v m ) and is an independent standard Gaussian in the orthogonal directions, where v 1 , . . . , v m denote the column vectors of V . In particular, if A is a continuous distribution with probability density function A ( y ) , then P A V is the distribution over R n with probability density function

That is, P A V is the product distribution whose orthogonal projection onto the subspace of V is A , and onto the subspace perpendicular to V is the standard ( n -m ) -dimensional normal distribution. An important special case of the above definition considers to m = 1 (i.e., the case when the hidden subspace is a hidden direction); for this setting, we will use the notation P A v for such a distribution, where A is a one-dimensional distribution and v ∈ R n is a unit vector.

Since we are focusing on establishing hardness, we will consider the following hypothesis testing version of NGCA (since the learning/search version typically reduces to the testing problem). We use N n to denote the standard n -dimensional Gaussian distribution N (0 , I n ) . We use U ( O n,m ) to denote the uniform distribution over the set of all orthogonal matrices V ∈ R n × m ; namely, this is the distribution obtained by taking RV ′ , where R ∈ R n × n is a random rotation matrix and V ′ ∈ R n × m is an arbitrary orthogonal matrix.

Definition 1.3 (Hypothesis Testing Version of NGCA) . Let n > m ≥ 1 be integers. For a distribution A supported on R m , one is given access to a distribution D such that either:

H 0 : D = N n ,

H 1 : D is given by P A V , where V ∼ U ( O n,m ) .

The goal is to distinguish between these two cases H 0 and H 1 .

For the special case that m = 1 (i.e., for a univariate distribution A ), prior work [DKS17] established SQ-hardness of NGCA 1 under the following condition:

Condition 1.4. Let d ∈ Z + . The distribution A on R is such that (i) the first d moments of A agree with the first d moments of N (0 , 1) , and (ii) the chi-squared distance χ 2 ( A, N ) is finite, where the chi-squared distance for two distributions (with probability density functions) P, Q : R n → R + is defined as χ 2 ( P, Q ) def = ∫ P ( x ) 2 /Q ( x ) d x -1 .

Specifically, the main result of [DKS17] shows that any SQ algorithm that solves the testing version of NGCA requires either 2 n Ω(1) many SQ queries or at least one query with accuracy

It is worth noting that subsequent works (see [DK22a] and [DKPZ21]) generalized this result so that it only requires that (i) A approximately matches moments with the standard Gaussian, and (ii) A is a low-dimensional distribution embedded in a hidden low-dimensional subspace, instead of a one-dimensional distribution.

The starting point of our investigation is a key technical limitation of this line of work. Specifically, if χ 2 ( A, N (0 , 1)) is very large (or infinite), e.g., if A has constant probability mass on a discrete set, the aforementioned SQ lower bound of [DKS17] can be very weak (or even vacuous). It is thus natural to ask if the finite chi-squared assumption is in fact necessary for the corresponding SQ lower bounds to hold.

A concrete motivation to answer this question comes from the applications of a generic SQ lower bound for NGCA to various learning problems. The SQ-hardness of NGCA can be used to obtain similar hardness for a number of well-studied learning problems that superficially appear very different. These include learning mixture models [DKS17, DKPZ23, DKS23], robust mean/covariance estimation [DKS17], robust linear regression [DKS19], learning halfspaces and other natural concepts with adversarial or semi-random label noise [DKZ20, GGK20, DK22a, DKPZ21, DKK + 22], list-decodable mean estimation and linear regression [DKS18, DKP + 21], learning simple neural networks [DKKZ20, GGJ + 20], and even learning simple generative models [CLL22]. In several of these applications, the requirement of bounded chi-squared distance is somewhat problematic, in some cases leading to quantitatively sub-optimal results. Moreover, in certain applications, this restriction leads to vacuous guarantees.

1.1 Our Results

1.1.1 Main Result

Our main result is a qualitatively near-optimal (up to the constant factor in the exponent) SQ lower bound for the NGCA problem, assuming only the moment-matching condition (i.e., without the chi-squared distance restriction). Informally, we essentially show that in order to solve the NGCA in n dimensions with an m -dimensional distribution A that (approximately) matches moments with the standard Gaussian up to degree d , any SQ algorithm will either require one query with accuracy O m,d ( n -Ω( d ) ) or exponential in n many queries.

Formally, we establish the following theorem.

Theorem 1.5 (Main SQ Lower Bound Result) . Let λ ∈ (0 , 1) and n, m, d ∈ N with d even and m,d ≤ n λ . Let ν ∈ R + and A be a distribution on R m such that for any polynomial f : R m → R of degree at most d and E x ∼N m [ f ( x ) 2 ] = 1 , the following holds: | E x ∼ A [ f ( x )] -E x ∼N m [ f ( x )] | ≤ ν . Let 0 < c < (1 -λ ) / 8 and n be at least a sufficiently large constant depending on c . Then any SQ algorithm solving the n -dimensional NGCA problem (as in Definition 1.3) with 2 / 3 success probability requires either (i) a query of tolerance O m,d ( n -((1 -λ ) / 8 -c ) d ) + (1 + o (1)) ν , or (ii) 2 n Ω( c ) many queries.

A few comments regarding the parameters are in order here. We first note that the constant in O m,d ( n -((1 -λ ) / 8 -c ) d ) is roughly the size of the Binomial coefficient ( ( d + m ) / 2 -1 m/ 2 -1 ) . Furthermore, we would like to point out that for most applications we will have m,d < 1 / poly( n ) . Therefore, the parameters λ and c can be taken to be arbitrarily close to 0 . So, informally, our lower bound on the query accuracy can be roughly thought of as ( ( d + m ) / 2 -1 m/ 2 -1 ) n -d/ 8 .

For all the applications given in this paper, the above theorem will be applied for the special case that m = 1 ; namely, the case that the hidden subspace is a hidden direction and A is a univariate distribution.

Relation to LLL-based Algorithms for NGCA Consider the special case of the NGCA problem corresponding to m = 1 where A is a discrete distribution that matches its first d moments with the standard Gaussian. Theorem 1.5 implies that any SQ algorithm for this version of the problem either uses a query with accuracy n -Ω( d ) or exponential many queries. On the other hand, recent works[DK22b, ZSWB22] gave polynomial-time algorithms for this problem with sample complexity O ( n ) , regardless of the degree d . It is worth noting that the existence of these algorithms does not contradict our SQ lower bound, as these algorithms are based on the LLL-method for lattice basis reduction that is not captured by the SQ framework. An an implication, it follows that LLLbased methods surpass any efficient SQ algorithm for these settings of NGCA. A similar observation was previously made in [DH23] for the special case that the discrete distribution A is supported on { 0 , ± 1 } ; consequently, [DH23] could only obtain a quadratic separation. Finally, we note that this

limitation of SQ algorithms is also shared by two other prominent restricted families of algorithms (namely, SoS algorithms and low-degree polynomial tests).

1.1.2 Applications

We believe that Theorem 1.5 is interesting in its own right, as it elucidates the SQ complexity of a natural and well-studied statistical problem. Here we discuss concrete applications to some natural statistical tasks. We note that the SQ lower bound in the prior work [DKS17] cannot give optimal (or even nontrivial) lower bounds for these applications.

List-decodable Gaussian Mean Estimation One can leverage our result to obtain a sharper SQ lower bound for the task of list-decodable Gaussian mean estimation. In this task, the algorithm is given as input points from R n where an α < 1 / 2 fraction of the points are drawn from an unknown mean and identity covariance Gaussian N ( µ , I ) , and the remaining points are arbitrary. The goal of the algorithm is to output a list of O (1 /α ) many hypothesis vectors at least one of which is close to µ in ℓ 2 -norm with probability at least 2 / 3 . [DKS18] established the following SQ lower bound for this problem (see also [DK23] for a different exposition).

Fact 1.6 ([DKS18]) . For each d ∈ Z + and c ∈ (0 , 1 / 2) , there exists c d > 0 such that for any α > 0 sufficiently small the following holds. Any SQ algorithm that is given access to a (1 -α ) -corrupted Gaussian N ( µ , I ) in n > d 3 /c dimensions and returns a list of hypotheses such that with probability at least 2 / 3 one of the hypotheses is within ℓ 2 -distance c d α -1 /d of the true mean µ , does one of the following: (i) Uses queries with error tolerance at most exp( O ( α -2 /d ))Ω( n ) -( d +1)(1 / 4 -c/ 2) . (ii) Uses at least exp(Ω( n c )) many queries. (iii) Returns a list of at least exp(Ω( n )) many hypotheses.

The above statement is obtained using the framework of [DKS17] by considering the distribution testing problem between N ( µ , I ) and P A v , where A is the one-dimensional moment-matching distribution of the following form.

Fact 1.7 ([DKS18], see Lemma 8.21 in [DK23]) . For d ∈ Z + , there exists a distribution A = α N ( µ, 1) + (1 -α ) E , for some distribution E and µ = 10 c d α -1 /d , such that the first d moments of A agree with those of N (0 , 1) . Furthermore, the probability density function of E can be taken to be pointwise at most twice the pdf of the standard Gaussian.

We note that χ 2 ( A, N ) = O (exp( µ 2 )) , and this results in the exp( O ( α -2 /d )) term in the error tolerance of Fact 1.6. Consequently, if α ≪ log( n ) -d/ 2 , the error tolerance is greater than one and Fact 1.6 fails to give a non-trivial bound. It is worth noting that the setting of 'small α ' (e.g., α is sub-constant in the dimension) is of significant interest in various applications, including in mean estimation. A concrete application is in the related crowd-sourcing setting of [MV18] dealing with this parameter regime.

We can circumvent this technical problem by combining our main result (Theorem 1.5) with Fact 1.7 to obtain the following sharper SQ lower bound (see Appendix C for the proof).

Theorem 1.8 (SQ Lower Bound for List-Decoding the Mean) . Let λ ∈ (0 , 1) and n, d ∈ N with d be even and d ≤ n λ . Let c > 0 and n be at least a sufficiently large constant depending on c . There exists c d > 0 such that for any α > 0 sufficiently small the following holds. Any SQ algorithm that is given access to a (1 -α ) -corrupted Gaussian N ( µ , I ) in n dimensions and returns a list of hypotheses such that with probability at least 2 / 3 one of the hypotheses is within ℓ 2 -distance c d α -1 /d of the true mean µ , does one of the following: (i) Uses queries with error tolerance at most O d ( n -((1 -λ ) / 8 -c ) d ) . (ii) Uses at least 2 n Ω( c ) many queries. (iii) Returns a list of at least exp(Ω( n )) many hypotheses.

Anti-concentration Detection Anti-concentration (AC) detection is the following hypothesis testing problem: given access to an unknown distribution D over R n , where the input distribution D is promised to satisfy either (i) D is the standard Gaussian; or (ii) D has at least α < 1 of its probability mass residing inside a dimension ( n -1) subspace V ⊂ R n . The goal is to distinguish between the two cases with success probability 2 / 3 .

In order to use our main result to derive an SQ lower bound for this task, we require the following lemma on univariate moment-matching (see Appendix C for the proof).

Lemma 1.9. Let D 0 denote the distribution that outputs 0 with probability 1 . For d ∈ Z + , there exists α d ∈ (0 , 1) and a univariate distribution A = α d D 0 +(1 -α d ) E for some distribution E , such that the first d moments of A agree with those of N (0 , 1) . Furthermore, the pdf of E can be taken to be pointwise at most twice the pdf of the standard Gaussian.

Using the above lemma and our main result, we can deduce the following SQ lower bound on the anti-concentration detection problem (see Appendix C for the proof).

Theorem 1.10 (SQ Lower Bound for AC Detection) . For any α ∈ (0 , 1 / 2) , any SQ algorithm that has access to a n -dimensional distribution that is either (i) the standard Gaussian; or (ii) a distribution that has at least α < 1 probability mass in a (n-1)-dimensional subspace V ⊂ R n , and distinguishes the two cases with success probability at least 2 / 3 , either requires a query with error at most n -ω α (1) , or uses at least 2 n Ω(1) many queries.

Learning Periodic Functions Another application is for the well-studied problem of learning periodic functions, see, e.g., [SVWX17] and [SZB21], which is closely related to the continuous Learning with Errors (cLWE) problem [BRST21]. In the task of learning periodic functions, the algorithm is given sample access to a distribution D of ( x , y ) over R n × R . The distribution D is such that x ∼ N (0 , I n ) and y = cos(2 π ( δ ⟨ w , x ⟩ + ζ )) with noise ζ ∼ N (0 , σ 2 ) . This implies that y is a periodic function of x along the direction of w with a small amount of noise. While the frequency and noise scale parameters δ, σ ∈ R + are known to the algorithm, the parameter w ∈ S n -1 is unknown; the goal of the algorithm is to output a hypothesis h : R n → R such that E x ∼ D [( h ( x ) -y ) 2 ] is minimized.

In order to use our main theorem to derive an SQ lower bound for this problem, we need to show that for any t ∈ [ -1 , 1] , D conditioned on y = t approximately matches moments with N (0 , I n ) . To this end, we introduce the following definition and fact of discrete Gaussian measure from [DK22a].

Definition 1.11. For s ∈ R + and θ ∈ R , let G s,θ denote the measure of the ' s -spaced discrete Gaussian distribution'. In particular, for each n ∈ Z , G s,θ assigns mass sg ( ns + θ ) to the point ns + θ , where g is the pdf function of N (0 , 1) .

Note that although G s,θ is not a probability measure (as the total measure is not one), it can be thought of as a probability distribution since the total measure is close to one for small σ . To see this, we introduce the following fact.

Fact 1.12 (Lemma 3.12 from [DK22a]) . For all k ∈ N , s > 0 and all θ ∈ R , we have that | E t ∼N (0 , 1) [ t k ] -E t ∼ G s,θ [ t k ] | = k ! O ( s ) k exp( -Ω(1 /s 2 )) .

Using the above fact, we are now ready to prove our SQ lower bound for learning periodic functions (see Appendix C for the proof).

Theorem 1.13 (SQ Lower Bound for Learning Periodic Functions) . Let c > 0 and n be at least a sufficiently large constant depending on c . Let D be the distribution of ( x , y ) over R n × R that is generated by x ∼ N (0 , I n ) and y = cos(2 π ( δ ⟨ w , x ⟩ + ζ )) with noise ζ ∼ N (0 , σ 2 ) . Let δ > n c , σ be known and w be unknown to the algorithm. Then any SQ algorithm that has access to the distribution D and returns a hypothesis h : R n → R such that E ( x ,y ) ∼ D [( h ( x ) -y ) 2 ] = o (1) , either requires a query with error at most exp( -n c ′ ) for c ′ < min(2 c, 1 / 10) , or uses at least 2 n Ω(1) many queries.

1.2 Technical Overview

We start by noting that we cannot use the standard SQ dimension argument [FGR + 17] to prove our result, due to the unbounded chi-squared norm. To handle this issue, we need to revisit the underlying ideas of that proof. In particular, we will show that for any bounded query function f : R n → [ -1 , 1] , with high probability over some V ∼ U ( O n,m ) , it holds that | E x ∼N n [ f ( x )] -E x ∼ P A V [ f ( x )] | will be small. This allows an adversarial oracle to return E x ∼N n [ f ( x )] to every query f regardless of which case we are in, unless the algorithm is lucky enough (or uses high accuracy) to find an f that causes | E x ∼N n [ f ( x )] -E x ∼ P A V [ f ( x )] | to be at least τ .

Our approach for calculating E x ∼ P A V [ f ( x )] will be via Fourier analysis. In particular, using the Fourier decomposition of f , we can write f ( x ) = ∑ ∞ k =0 ⟨ T k , H k ( x ) ⟩ , where H k ( x ) is the

properly normalized degreek Hermite polynomial tensor and T k is the degreek Fourier coefficients of f . Taking the inner product with the distribution P A V involves computing the expectation of E x ∼ P A V [ H k ( x )] , which can be seen to be ⟨ V ⊗ k A k , T k ⟩ , where A k = E x ∼ A [ H k ( x )] . Thus, we obtain (at least morally speaking) that

The k = 0 term of the above sum will be exactly T 0 = E x ∼N n [ f ( x )] . By the moment-matching condition, the k = 1 through k = d terms will be very small. Finally, we need to argue that the higher degree terms are small with high probability. To achieve this, we provide a non-trivial upper bound for E V ∼ U ( O n,m ) [ ∥ ( V ⊺ ) ⊗ k T k ∥ 2 2 ] . Combining with the fact that ∑ ∞ k =0 ∥ T k ∥ 2 2 = ∥ f ∥ 2 2 ≤ 1 , this allows us to prove high probability bounds on each term, and thus their sum.

Furthermore, if we want higher probability estimates of the terms with small k , we can instead bound E V ∼ U ( O n,m ) [ ∥ ( V ⊺ ) ⊗ k T k ∥ 2 a 2 ] for some integer a . Unfortunately, there are non-trivial technical issues with the above approach, arising from issues with (1). To begin with, as no assumptions other than moment-matching (for a few low-degree moments) were made on A , it is not guaranteed that A k is finite for larger values of k . To address this issue, we will truncate the distribution A . In particular, we pick a parameter B (which will be determined carefully), and define A ′ to be A conditioned on the value in the ball B m ( B ) . Due to the higher-moment bounds, we can show that A and A ′ are close in total variation distance, and thus that E x ∼ P A V [ f ( x )] is close to E x ∼ P A ′ V [ f ( x )] for any bounded f .

Furthermore, using the higher moment bounds, we can show that A ′ nearly matches the lowdegree moments of N m . The second issue arises with the interchange of summations used to derive (1). In particular, although f ( x ) = ∑ ∞ k =0 ⟨ T k , H k ( x ) ⟩ , it does not necessarily follow that we can interchange the infinite sum on the right-hand-side with taking the expectation over x ∼ P A V . To fix this issue, we split f into two parts f ≤ ℓ (consisting of its low-degree Fourier components) and f >ℓ . We note that (1) does hold for f ≤ ℓ , as the summation there will be finite, and we can use the above argument to bound E x ∼N n [ f ( x )] -E x ∼ P A V [ f ≤ ℓ ( x )] | with high probability. To bound | E x ∼ P A V [ f >ℓ ( x )] | , we note that by taking ℓ large, we can make ∥ f >ℓ ∥ 2 < δ for some exponentially small δ > 0 . We then bound E V ∼ U ( O n,m ) [ E x ∼ P A V [ | f >ℓ | ]] = E x ∼ Q [ | f >ℓ | ] , where Q is the average over V of P A V (note that everything here is non-negative, so there is no issue with the interchange of integrals). Thus, we can bound the desired quantity by noting that ∥ f >ℓ ∥ 2 is small and that the chi-squared norm of Q with respect to the standard Gaussian N n is bounded.

2 Preliminaries

We will use lowercase boldface letters for vectors and capitalized boldface letters for matrices and tensors. We use S n -1 = { x ∈ R n : ∥ x ∥ 2 = 1 } to denote the n -dimensional unit sphere. For vectors u , v ∈ R n , we use ⟨ u , v ⟩ to denote the standard inner product. For u ∈ R n , we use ∥ u ∥ k = ( ∑ n i =1 u k i ) 1 /k to denote the ℓ k -norm of u . For tensors, we will consider a k -tensor to be an element in ( R n ) ⊗ k ∼ = R n k . This can be thought of as a vector with n k coordinates. We will use A i 1 ,...,i k to denote the coordinate of a k -tensor A indexed by the k -tuple ( i 1 , . . . , i k ) . By abuse of notation, we will sometimes also use this to denote the entire tensor. The inner product and ℓ k -norm of k -tensor are defined by thinking of the tensor as vectors with n k coordinates and then use the definition of inner product and ℓ k -norm of vectors. For a vector v ∈ R n , we denote by v ⊗ k to be a vector (linear object) in R n k . For a matrix V ∈ R n × m , we denote by ∥ V ∥ 2 , ∥ V ∥ F to be the operator norm and Frobenius norm respectively. In addition, we denote by V ⊗ k to be a matrix (linear operator) mapping R n k to R m k . We use 1 to denote the indicator function of a set, specifically 1 ( t ∈ S ) = 1 if t ∈ S and 0 otherwise. We will use Γ : R → R to denote the gamma function Γ( z ) = ∫ ∞ 0 t z -1 e -t dt . We use B : R × R → R to denote the beta function B ( z 1 , z 2 ) = Γ( z 1 )Γ( z 2 ) / Γ( z 1 + z 2 ) . We use χ 2 k to denote the chi-squared distribution with k degrees of freedom. We use Beta( α, β ) to denote the Beta distribution with parameters α and β .

For a distribution D , we use Pr D [ S ] to denote the probability of an event S . For a continuous distribution D over R n , we sometimes use D for both the distribution itself and its probability density function. For two distributions D 1 , D 2 over a probability space Ω , let d TV ( D 1 , D 2 ) = sup S ⊆ Ω | Pr D 1 ( S ) -Pr D 2 ( S ) | denote the total variation distance between D 1 and D 2 . For two continuous distribution D 1 , D 2 over R n , we use χ 2 ( D 1 , D 2 ) = ∫ R n D 1 ( x ) 2 /D 2 ( x ) d x -1 to denote the chi-square norm of D 1 w.r.t. D 2 . For a subset S ⊆ R n with finite measure or finite surface measure, we use U ( S ) to denote the uniform distribution over S (w.r.t. Lebesgue measure for the volumn/surface area of S ).

Basics of Hermite Polynomials

Definition 2.1 (Normalized Hermite Polynomial) . For k ∈ N , we define the k -th probabilist's Hermite polynomials He k : R → R as He k ( t ) = ( -1) k e t 2 / 2 · d k dt k e -t 2 / 2 . We define the k -th normalized Hermite polynomial h k : R → R as h k ( t ) = He k ( t ) / √ k ! .

Furthermore, we will use multivariate Hermite polynomials in the form of Hermite tensors (as the entries in the Hermite tensors are rescaled multivariate Hermite polynomials). We define the Hermite tensor as follows.

Definition 2.2 (Hermite Tensor) . For k ∈ N and x ∈ R n , we define the k -th Hermite tensor as

We denote by L 2 ( R n , N n ) the function space of all functions f : R n → R such that E v ∼N n [ f 2 ( v )] < ∞ . For functions f, g ∈ L 2 ( R n , N n ) , we use ⟨ f, g ⟩ N n = E x ∼N n [ f ( x ) g ( x )] to denote their inner product. We use ∥ f ∥ 2 = √ ⟨ f, f ⟩ N n to denote its L 2 -norm. For a function f : R n → R and ℓ ∈ N , we use f ≤ ℓ to denote f ≤ ℓ ( x ) = ∑ ℓ k =0 ⟨ A k , H k ( x ) ⟩ , where A k = E x ∼N n [ f ( x ) H k ( x )] , which is the degreeℓ approximation of f . We use f >ℓ = f -f ≤ ℓ to denote its residue. We remark that normalized Hermite polynomials (resp. Hermite tensors) form a complete orthogonal system for the inner product space L 2 ( R , N ) (resp. L 2 ( R n , N n ) ). This implies that for f ∈ L 2 ( R n , N n ) , lim ℓ →∞ ∥ f >ℓ ∥ 2 = 0 . We also remark that both our definition of Hermite polynomial and Hermite tensor are 'normalized'. In the sense that for Hermite polynomials, ∥ h k ∥ 2 = 1 . For Hermite tensors, given any symmetric tensor A , we have ∥⟨ A , H k ( x ) ⟩∥ 2 2 = ⟨ A , A ⟩ . The following claim states that for any orthonormal transformation B , the Hermite tensor H k ( Bx ) can be written as applying the linear transformation B ⊗ k on the Hermite tensor H k ( x ) . The proof is deferred to Appendix A.

Claim 2.3. Let 1 ≤ m < n . Let B ∈ R m × n with BB ⊺ = I m . It holds that H k ( Bx ) = B ⊗ k H k ( x ) , x ∈ R n .

3 SQ-Hardness of NGCA: Proof of Theorem 1.5

The main idea of the proof is the following. Suppose that the algorithm only asks queries with tolerance τ , and let f be an arbitrary query function that the algorithm selects. The key ingredient is to show that | E x ∼ P A V [ f ( x )] -E x ∼N n [ f ( x )] | ≤ τ with high probability over V ∼ U ( O n,m ) . If this holds, then when the algorithm queries f , if the input is from the alternative hypothesis, with high probability, E x ∼N n [ f ( x )] is a valid answer for the query. Therefore, when the algorithm queries f , regardless of whether the input is from the alternative or null hypothesis, the oracle can just return E x ∼N n [ f ( x )] . Then the algorithm will not observe any difference between the two cases with any small number of queries. Thus, it is impossible to distinguish the two cases with high probability. To prove the desired bound, we introduce the following proposition.

Proposition 3.1. Let λ ∈ (0 , 1) and n, m, d ∈ N with d be even and m,d ≤ n λ . Let ν ∈ R + and A be a distribution on R m such that for any polynomial f : R m → R of degree at most d and E x ∼N m [ f ( x ) 2 ] = 1 ,

Let 0 < c < (1 -λ ) / 8 and n is at least a sufficiently large constant depending on c , then, for any function f : R n → [ -1 , 1] , it holds

Assuming Proposition 3.1, the proof of our main theorem is quite simple.

Proof for Theorem 1.5. Suppose there is an SQ algorithm A using q < 2 n Ω( c ) many queries of accuracy τ ≥ ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -((1 -λ ) / 8 -c ) d +(1+ o (1)) ν and succeeds with at least 2 / 3 probability. We prove by contradiction that such an A cannot exist. Suppose the input distribution is N n , and the SQ oracle always answers E x ∼N n [ f ( x )] for any query f . Then the assumption on A implies that it answers 'null hypothesis' with probability α > 2 / 3 . Now consider the case that the input distribution is P A V and V ∼ U ( O n,m ) . Suppose the SQ oracle still always answers E x ∼N n [ f ( x )] for any query f . Let f 1 , · · · , f q be the queries the algorithm asks, where q = 2 n Ω( c ) . By Proposition 3.1 and a union bound, we have

Therefore, with probability 1 -o (1) , the answers given by the oracle are valid. From our assumption on A , the algorithm needs to answer 'alternative hypothesis' with probability at least 2 3 (1 -o (1)) . But since the oracle always answers E x ∼N n [ f ( x )] (which is the same as the above discussed null hypothesis case), we know the algorithm will return 'null hypothesis' with probability α > 2 / 3 . This gives a contradiction and completes the proof.

The rest of this section is devoted to the proof of Proposition 3.1.

3.1 Fourier Analysis using Hermite Polynomials

The main idea of Proposition 3.1 is to analyze E x ∼ P A V [ f ( x )] through Fourier analysis using Hermite polynomials. Before we do the analysis, we will first truncate the distribution A inside B m ( B ) which is the ℓ 2 -norm unit ball in m -dimension with radius B for some B ∈ R + to be specified. Namely, we will consider the truncated distribution A ′ defined as the distribution of x ∼ A conditioned on x ∈ B m ( B ) . The following lemma shows that given any m -dimensional distribution A that approximately matches the first d moment tensor with the Gaussian, the truncated distribution A ′ is close to A in both the total variation distance and the first d moment tensors.

Lemma 3.2. Let n, m, d ∈ N with d be even. Let A be a distribution on R m such that for any polynomial f of degree at most d and E x ∼N m [ f ( x ) 2 ] = 1 ,

Let B ∈ R + such that B d ≥ c 1 ( 2 d/ 2 √ Γ( d + m/ 2) Γ( m/ 2) ) where c 1 is at least a sufficiently large universal constant and let A ′ be the truncated distribution defined as the distribution of x ∼ A conditioned on x ∈ B m ( B ) . Then d TV ( A,A ′ ) = O ( 2 d/ 2 √ Γ( d + m/ 2) Γ( m/ 2) ) B -d , Furthermore, any k ∈ N and k < d ,

The proof of Lemma 3.2 is deferred to Appendix B.

Since d TV ( A,A ′ ) ≤ O ( 2 d/ 2 √ Γ( d + m/ 2) Γ( m/ 2) ) B -d and f is bounded in [ -1 , 1] , it follows that

Therefore, if suffices for us to analyze E x ∼ P A ′ V [ f ( x )] instead of E x ∼ P A V [ f ( x )] . Furthermore, the property that A ′ is bounded inside B m ( B ) will be convenient in the Fourier analysis later. We introduce the following lemma which decomposes E x ∼ P A ′ V [ f ( x )] using Hermite analysis. The proof is deferred to Appendix B.

Lemma 3.3 (Fourier decomposition Lemma) . Let A ′ be any distribution supported on R m , V ∈ R n × m and V ⊺ V = I m . Then for any ℓ ∈ N , E x ∼ P A ′ V [ f ( x )] = ∑ ℓ k =0 ⟨ V ⊗ k A k , T k ⟩ + E x ∼ P A ′ V [ f >ℓ ( x )] , where A k = E x ∼ A ′ [ H k ( x )] and T k = E x ∼N n [ f ( x ) H k ( x )] .

Remark 3.4. Ideally, in Lemma 3.3, we would like to have E x ∼ P A ′ V [ f ( x )] = ∑ ∞ k =0 ⟨ V ⊗ k A k , T k ⟩ . However, since we do not assume that χ 2 ( A ′ , N m ) < ∞ , this convergence may not hold.

Recall that our goal is to show that | E x ∼ P A V [ f ( x )] -E x ∼N n [ f ( x )] | is small with high probability. Observe that E x ∼N n [ f ( x )] = T 0 which is the first term in the summation of ∑ ℓ k =0 ⟨ V ⊗ k A k , T k ⟩ (since A 0 = 1 ). Therefore, given Lemma 3.3, it suffices to show that | ∑ ℓ k =1 ⟨ V ⊗ k A k , T k ⟩| and | E x ∼ P A ′ V [ f ≥ ℓ ( x )] | are both small with high probability. We ignore the | E x ∼ P A ′ V [ f ≥ ℓ ( x )] | part for now as this is mostly a technical issue. To bound | ∑ ℓ k =1 ⟨ V ⊗ k A k , T k ⟩| , it suffices to analyze ∑ ℓ k =1 |⟨ V ⊗ k A k , T k ⟩| by looking at each term |⟨ V ⊗ k A k , T k ⟩| = |⟨ A k , ( V ⊺ ) ⊗ k T k ⟩| ≤ ∥ A k ∥ 2 ∥ ( V ⊺ ) ⊗ k T k ∥ 2 . To show that the summation is small, given we need to prove that (with high probability):

∥ A k ∥ 2 does not grow too fast w.r.t k ;

∥ ( V ⊺ ) ⊗ k T k ∥ 2 decays very fast w.r.t k (is small with high probability w.r.t the randomness of V ).

∥ A k ∥ 2 does not grow too fast: We will use slightly different arguments depending on the size of k . We consider three cases : k < m , m ≤ k ≤ n (1 -λ ) / 4 , and k ≥ n (1 -λ ) / 4 (the value in the exponent will deviate by a small quantity to make the proof go through). For k < m , ∥ A k ∥ 2 grows slowly by the approximate moment-matching property of A ′ . For m ≤ k ≤ n (1 -λ ) / 4 , we require the following fact:

Fact 3.5. Let H k be the k -th Hermite tensor for m -dimension. Suppose ∥ x ∥ 2 2 ≥ m , then ∥ H k ( x ) ∥ 2 = 2 O ( k ) max( ∥ x ∥ k 2 , 1) .

We provide the proof of Fact 3.5 in Appendix B. For k > n (1 -λ ) / 4 , we can show that ∥ A k ∥ 2 does not grow too fast by the following asymptotic bound on Hermite tensors.

Fact 3.6. Let H k be the k -th Hermite tensor for m -dimension, then ∥ H k ( x ) ∥ 2 ≤ 2 O ( m ) ( k + m -1 m -1 ) 1 / 2 exp( ∥ x ∥ 2 2 / 4) .

We provide the proof of Fact 3.6 in Appendix B.

∥ ( V ⊺ ) ⊗ k T k ∥ 2 decays very fast: We show that |⟨ V ⊗ k , T k ⟩| is small with high probability by bounding its a -th moment for some even a . Notice that since ∥ H k ∥ 2 ≤ ∥ f ∥ 2 ≤ 1 , we can then combine it with the following lemma. We defer its proof to Appendix B.

Lemma 3.7. Let k ∈ Z + , a ∈ Z + be even, T ∈ R n k and m ∈ Z + satisfy m<n . Then, we have

Roughly speaking, this is just the ak/ 2 -th moment of the correlation between a random subspace and a random direction, which can be upper bounded by the following lemma and corollary (see Appendix B for the proofs).

Corollary 3.9. Let c ∈ (0 , 1) and m ≤ n c . Let k ∈ N be even. We have that

To combine the above results and give the high probability upper bound on ∑ ℓ k =1 |⟨ A k , ( V ⊺ ) ⊗ k T k ⟩| , we require the following lemma. The proof is deferred to Appendix B.

Lemma 3.10. Under the conditions of Proposition 3.1, and further assume d, m ≤ n λ / log n , ν < 2 and ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -((1 -λ ) / 8 -c ) d < 2 . Then for any n that is at least a sufficiently small constant depending on c , there is an α ′ < (1 -λ ) / 8 such that for any B = n α where α ′ < α < (1 -λ ) / 8 the following holds. Let A ′ be the truncated distribution defined as the distribution of x ∼ A conditioned on x ∈ B m ( B ) . Then d TV ( A,A ′ ) ≤ ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -((1 -λ ) / 8 -c ) d . Furthermore for any ℓ ∈ N , except with probability at most 2 -n Ω( c ) w.r.t. V ∼ U ( O n,m ) , ∑ ℓ k =1 |⟨ A k , ( V ⊺ ) ⊗ k T k ⟩| = ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -((1 -λ ) / 8 -c ) d +(1 + o (1)) ν where A k = E x ∼ A ′ [ H k ( x )] and T k = E x ∼N n [ f ( x ) H k ( x )] .

3.2 Proof for Proposition 3.1

We are now ready to prove Proposition 3.1 which is the main technical ingredient of our main result. Proposition 3.1 states that | E x ∼ P A V [ f ( x )] -E x ∼N n [ f ( x )] | is small with high probability. The main idea of the proof is to use Fourier analysis on E x ∼ P A ′ V [ f ( x )] as we discussed in the last section, where A ′ is the the distribution obtained by truncating A inside B m ( B ) .

Proof for Proposition 3.1. For convenience, we let ζ = (1 -λ ) / 8 -c . We will analyze E x ∼ A [ H k ( x )] by truncating A . Therefore, we will apply Lemma 3.10 here. However, notice Lemma 3.10 additionally assumes d, m ≤ n λ / log n , ν < 2 and ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -ζd < 2 . We show that all these three conditions can be assumed true WLOG. If either the second or the third condition is not true, then our lower bound here is trivialized and is always true since f is bounded between [ -1 , +1] . For d, m ≤ n λ / log n , consider a λ ′ > λ such that (1 -λ ′ ) / 8 -ζ = (1 -λ ) / 8 -ζ 2 . Then it is easy to see for any sufficiently large n depending on (1 -λ ) / 8 -ζ , we have d, m ≤ n λ ′ / log n and ζ ≤ (1 -λ ) / 8 -ζ . Therefore, we can WLOG apply Lemma 3.10 for λ ′ .

Now let B = n α where α < (1 -λ ) / 8 is the constant in Lemma 3.10. Then we consider the truncated distribution A ′ defined as the distribution of x ∼ A conditioned on x ∈ B m ( B ) . By Lemma 3.10, we have d TV ( A,A ′ ) ≤ ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -ζd . Given that f is bounded between [ -1 , 1] , this implies | E x ∼ P A V [ f ( x )] -E x ∼ P A ′ V [ f ( x )] | ≤ 2 d TV ( P A V , P A ′ V ′ ) = 2 d TV ( A,A ′ ) ≤ 2 ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -ζd . Thus, it suffices for us to analyze E x ∼ P A ′ V [ f ( x )] instead of E x ∼ P A V [ f ( x )] .

Let ℓ = ℓ f ( n ) ∈ N be a function depending only on the query function f and the dimension n ( ℓ to be specified later). By Lemma 3.3, we have that E x ∼ P A ′ V [ f ( x )] = ∑ ℓ k =0 |⟨ A k , ( V ⊺ ) ⊗ k T k ⟩| + E x ∼ P A ′ V [ f >ℓ ( x )] . Recall that we want to bound | E x ∼ P A ′ V [ f ( x )] -E x ∼N n [ f ( x )] | with high probability, where we note that E x ∼N n [ f ( x )] = ⟨ A 0 , T 0 ⟩ . Therefore, we can write

For the first term, by Lemma 3.10, we have that | ∑ ℓ k =1 ⟨ A k , ( V ⊺ ) ⊗ k T k ⟩| = ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -ζd + (1 + o (1)) ν , except with probability 2 -n Ω( c ) .

It now remains for us to show that | E x ∼ P A ′ V [ f >ℓ ( x )] | is also small with high probability. Consider the distribution D = E v ∼ U ( O n,m ) [ P A ′ V ] . The following lemma shows that D is continuous

and χ 2 ( D, N n ) is at most a constant only depending on n (independent of the choice of the one dimensional distribution A ).

Lemma 3.11. Let A be any distribution supported on B m ( n ) for n ∈ N which is at least a sufficiently large universal constant. Let D = E V ∼ U ( O n,m ) [ P A ′ V ] . Then, D is a continuous distribution and χ 2 ( D, N n ) = O n (1) .

Roughly speaking, the proof of the lemma follows by noting that the average over V of P A V is spherically symmetric. We defer its proof to Appendix B.2. Using this lemma, we can write

where the first equality holds due to Fubini's theorem. Furthermore, since χ 2 ( D, N n ) = O n (1) , there is a function δ : R → R such that 1 + χ 2 ( D, N n ) ≤ δ ( n ) . Therefore, we have that

We can take ℓ = ℓ f ( n ) ( ℓ only depends on the query function f and dimension n ) to be a sufficiently large function such that ∥ f >ℓ ∥ 2 ≤ ( e -n δ ( n ) )( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -ζd . Then we get

This gives the tail bound Pr V ∼ U ( O n,m ) [∣ ∣ E x ∼ P A ′ V [ f >ℓ ( x )] ∣ ∣ ≥ ( Γ( d/ 2+ m/ 2) Γ( m/ 2) ) n -ζd ] ≤ e -n .

Using the above upper bounds, we have

except with probability 2 -n Ω(1) . As we have argued at the beginning of the proof,

Therefore,

except with probability 2 -n Ω(1) < 2 -n Ω( c ) given c = O (1) .

In the end, notice that the above argument is still true if we take ζ ′ > ζ such that (1 -λ ) / 8 -ζ ′ = (1 -λ ) / 8 -ζ 2 . Using the above argument for ζ ′ and given n is a sufficiently large constant depending (1 -λ ) / 8 -ζ = 2((1 -λ ) / 8 -ζ ′ ) , we get

except with probability 2 -n Ω((1 -λ ) / 8 -ζ ′ ) = 2 -n Ω( c ) . Replaceing ζ with (1 -λ ) / 8 -c completes the proof.

References

[BKS + 06] G. Blanchard, M. Kawanabe, M. Sugiyama, V. Spokoiny, and K.-R. M¨ uller. In search of non-gaussian components of a high-dimensional distribution. Journal of Machine Learning Research , 7(9):247-282, 2006.

[BRST21] J. Bruna, O. Regev, M. J. Song, and Y. Tang. Continuous LWE. In 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 694-707. ACM, 2021.

[CLL22] S. Chen, J. Li, and Y . Li. Learning (very) simple generative models is hard. In NeurIPS , 2022.

[DH23] J. Ding and Y. Hua. SQ lower bounds for random sparse planted vector problem. In International Conference on Algorithmic Learning Theory , volume 201 of Proceedings of Machine Learning Research , pages 558-596. PMLR, 2023.

[DK22a] I. Diakonikolas and D. Kane. Near-optimal statistical query hardness of learning halfspaces with massart noise. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4258-4282. PMLR, 2022. Full version available at https://arxiv.org/abs/2012.09720.

[DK22b] I. Diakonikolas and D. Kane. Non-gaussian component analysis via lattice basis reduction. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4535-4547. PMLR, 2022.

[DK23] I. Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics . Cambridge university press, 2023. Full version available at https://sites.google.com/view/ars-book.

[DKK + 22] I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Learning general halfspaces with general massart noise under the gaussian distribution. In STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 874-885, 2022. Full version available at https://arxiv.org/abs/2108.08767.

[DKKZ20] I. Diakonikolas, D. M. Kane, V. Kontonis, and N. Zarifis. Algorithms and SQ lower bounds for PAC learning one-hidden-layer relu networks. In Conference on Learning Theory, COLT 2020 , volume 125 of Proceedings of Machine Learning Research , pages 1514-1539. PMLR, 2020.

[DKP + 21] I. Diakonikolas, D. M. Kane, A. Pensia, T. Pittas, and A. Stewart. Statistical query lower bounds for list-decodable linear regression. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 3191-3204, 2021.

[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Conference on Learning Theory, COLT 2021 , volume 134 of Proceedings of Machine Learning Research , pages 1552-1584. PMLR, 2021.

[DKPZ23] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. SQ lower bounds for learning mixtures of separated and bounded covariance gaussians. In The Thirty Sixth Annual Conference on Learning Theory, COLT 2023 , volume 195 of Proceedings of Machine Learning Research , pages 2319-2349. PMLR, 2023.

[DKS17] I. Diakonikolas, D. M. Kane, and A. Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017 , pages 73-84, 2017. Full version at http://arxiv.org/abs/1611.03473.

[DKS18] I. Diakonikolas, D. M. Kane, and A. Stewart. List-decodable robust mean estimation and learning mixtures of spherical gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 1047-1060, 2018. Full version available at https://arxiv.org/abs/1711.07211.

[DKS19] I. Diakonikolas, W. Kong, and A. Stewart. Efficient algorithms and lower bounds for robust linear regression. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2019 , pages 2745-2754, 2019.

[DKS23] I. Diakonikolas, D. M. Kane, and Y. Sun. SQ lower bounds for learning mixtures of linear classifiers. CoRR , abs/2310.11876, 2023.

[DKZ20] I. Diakonikolas, D. M. Kane, and N. Zarifis. Near-optimal SQ lower bounds for agnostically learning halfspaces and relus under gaussian marginals. CoRR , abs/2006.16200, 2020.

[Fel16] V. Feldman. Statistical query learning. In Encyclopedia of Algorithms , pages 20902095. Springer New York, 2016.

[FGR + 17] V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. Statistical algorithms and a lower bound for detecting planted cliques. J. ACM , 64(2):8:1-8:37, 2017.

[FGV17] V. Feldman, C. Guzman, and S. S. Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the TwentyEighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017 , pages 1265-1277. SIAM, 2017.

[GGJ + 20] S. Goel, A. Gollakota, Z. Jin, S. Karmalkar, and A. R. Klivans. Superpolynomial lower bounds for learning one-layer neural networks using gradient descent. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , volume 119 of Proceedings of Machine Learning Research , pages 3587-3596. PMLR, 2020.

[GGK20] S. Goel, A. Gollakota, and A. R. Klivans. Statistical-query lower bounds via functional gradients. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020.

[GS19] N. Goyal and A. Shetty. Non-gaussian component analysis using entropy methods. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019 , pages 840-851. ACM, 2019.

[Kea98] M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM , 45(6):983-1006, 1998.

[Kra04] I. Krasikov. New bounds on the Hermite polynomials. arXiv preprint math/0401310 , 2004.

[MV18] M. Meister and G. Valiant. A data prism: Semi-verified learning in the small-alpha regime. In Conference On Learning Theory, COLT 2018 , volume 75 of Proceedings of Machine Learning Research , pages 1530-1546. PMLR, 2018.

[SVWX17] L. Song, S. S. Vempala, J. Wilmes, and B. Xie. On the complexity of learning neural networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NeurIPS 2017 , pages 55145522, 2017.

[SZB21] M. J. Song, I. Zadik, and J. Bruna. On the cryptographic hardness of learning single periodic neurons. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 29602-29615, 2021.

[TV18] Y. S. Tan and R. Vershynin. Polynomial time and sample complexity for non-gaussian component analysis: Spectral methods. In Conference On Learning Theory, COLT 2018 , volume 75 of Proceedings of Machine Learning Research , pages 498-534. PMLR, 2018.

[Val84] L. Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, 1984.

[ZSWB22] I. Zadik, M. J. Song, A. S. Wein, and J. Bruna. Lattice-based methods surpass sum-ofsquares in clustering. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1247-1248. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an SQ lower bound for the problem of distinguishing the standard Gaussian distribution from a distribution with a single non-Gaussian component, under relaxed assumptions compared to what was previously known. In particular, in prior work, similar SQ lower bounds were established in the case where the non-Gaussian component corresponded to any single-dimensional distribution $A$ whose low order moments match those of a standard Gaussian and the ($\\chi^2$) distance between $A$ and the standard Gaussian is bounded. In this work, the bounded ($\\chi^2$) distance assumption is removed.\n\nAs consequences of the main result, two additional SQ lower bounds are provided, one for List-Decodable Mean estimation and one for the problem of anti-concentration detection.\n\nThe approach is simple and consists of direct probabilistic bounds for the difference between the value of a statistical query under the Gaussian and the corresponding value under a distribution with a non-Gaussian component in a uniformly random direction. The bound is shown via Fourier analysis over the Hermite basis.\n\nSTRENGTHS:\nThe problem is clear and well motivated. The claimed result is a strict improvement over prior work with some further applications.\n\nWEAKNESSES:\nThe paper has several important weaknesses.\n\n1. The main result seems to contradict prior work (which is not cited in the paper), which raises serious doubts about its correctness. In particular, in Theorem 1.3 of [DK22] it is shown that non-Gaussian components that are far from the unit normal are detectable (even under the moment matching assumption). See also the discussion on page 2, paragraph \"Motivation of This Work\" of [DK22]. \n\n[DK22]:  Diakonikolas, Ilias, and Daniel Kane. \"Non-gaussian component analysis via lattice basis reduction.\" Conference on Learning Theory. PMLR, 2022.\n\n2. The presentation of related work contains inaccuracies. In particular, within the paragraph in lines 66-74, it is implied that all of the listed papers make use of results on the hardness of non-Gaussian component analysis (NGCA) to gain results for other problems. However, for example, [GGJ+20] does not even mention the problem of NGCA, but makes use of some appropriate combinatorial quantity (called statistical dimension) to arrive to their main result.\n\n3. The text in the introduction is sloppy and overly casual and section 3 is too technical. For example, lines 52-57 could use some polish and lines 148-153 give the impression that there are no further technical challenges, so the following paragraph becomes confusing. \n\n-- The authors addressed my main concern that their results seemed to contradict prior work.\n\nQUESTIONS:\n1. My main question concerns the correctness of the main result. How does this result compare to Theorem 1.3 of [DK22]? Such a discussion should, in any case, be a part of the paper.\n\n2. Section 3 is quite dense and cannot be easily parsed, so it might be beneficial to substitute the proofs with proof sketches that contain less quantitative information (the full proofs can be moved to the Appendix).\n\nLIMITATIONS:\nThe existence of an LLL-based algorithm for the problem in question [DK22] should be discussed thoroughly.\n\n[DK22]:  Diakonikolas, Ilias, and Daniel Kane. \"Non-gaussian component analysis via lattice basis reduction.\" Conference on Learning Theory. PMLR, 2022.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies non-Gaussian component (NGCA) analysis in the context of the statistical query model. In NGCA the task is to distinguish a standard multi-variate Gaussian distribution from a distribution that is standard Gaussian in all but a random direction $w$ and equal to a one-dimensional distribution $A$ along $w$.\n\nPreviously, it was known that this problem is hard in the SQ model when\n1. $A$ matches many moments with the (one-dimensional) standard Gaussian,\n2. the $\\chi^2$-divergence between $N(0,1)$ and $A$ is finite.\n\nAlso, the quality of the lower bound depends on the $\\chi^2$-divergence: If this is very large with respect to other problem parameters (e.g., the dimension and the number of moments matched), the lower bound can potentially be weaker. \n\nIn their work, the authors show that Assumption (2) above is not necessary for the hardness result to hold. They use this to obtain an improved result for list-decodable mean estimation (when the inliers are Gaussian with identity-covariance). They further show SQ hardness of detecting whether a distribution has a constant fraction of its probability mass in a lower-dimensional subspace.\n\nOn a technical level, they are able to remove the assumption on the $\\chi^2$-divergence since they do not show hardness based on the SQ dimension (a widely used notion which implies SQ hardness), but rather show directly that every statistical query on the two distributions must receive the same answer up to some small error. They show this by carefully analyzing the Fourier moments of both distributions.\n\nSTRENGTHS:\nIn my eyes, the main strength of the paper is that it allows for more flexibility when using the NGCA framework. Second, it also makes it easier to apply. Since this framework has found numerous applications, I believe that many researchers will appreciate this result. On a conceptual level, it is pleasing to see that the $\\chi^2$-divergence condition is indeed just a technicality (and the \"true hardness\" comes from the moment-matching condition).\n\nI believe the result can lead to:\n1. improved lower bounds (since the $\\chi^2$-divergence did affect the quality of the lower bound),\n2. and simple(r) constructions (since we now have more flexibility).\n\nThe authors gave one example for each of the two in their paper.\n\nThe paper is generally well-written. The authors give a nice and easy-to-follow overview of the main result in Section 1.2. Their solution is very natural. The actual proofs are more technical, but this is shared with other SQ lower bounds using the NGCA framework.\n\nWEAKNESSES:\nMinor: While the quantitative improvement for the list-decoding lower bound is non-trivial, it appears only in a very restricted regime of parameters, when the fraction of inliers goes to 0 when the dimension grows.\n\nQUESTIONS:\nIn some of the existing SQ lower bounds using the NGCA framework it was very useful if the moments of $A$ only need to *approximately* match those of a standard Gaussian (e.g., the lower bounds for learning halfspaces under Massart noise in the distribution-independent setting). Are your techniques able to capture this scenario as well? If this is the case, it would be very helpful to include it in the main theorems since it would make them even easier to apply.\n\nLIMITATIONS:\nThe authors give a fairly definitive answer to the question they studied.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the SQ-hardness of non-Gaussian component analysis. The main result of the paper is a statement of the hardness without an assumption required by results stated in previous works: finite chi-squared distance between the non-gaussian distribution and the standard normal. Two applications are discussed. The bulk of the paper is focused on proof techniques, which makes use of Fourier transforms and Hermite polynomials to bound expected values of query functions.\n\nSTRENGTHS:\nThe main result of the paper by itself is interesting enough to be the greatest strength of the paper. \n\nOriginality. While the topic of hypothesis testing of NGCA in the Statistical Query (SQ) model is not new, and it has been known that the problem itself is SQ hard, this paper removes one of the main assumptions required for results in previous works to hold. From this perspective, the paper provides original contributions to the field.\n\nQuality. The quality of the paper is quite nice. The paper does a good job in describing the problem formulation, downfalls of past approaches, and definitions of the various technical tools required to establish the proof of the main result. \n\nClarity. The paper is well written and evidently polished. The main contributions of the paper are well stated and clear. The proof sections in the last few pages of the paper may need a bit more revision to be better readable.\n\nSignificance. The main results of the paper are widely applicable to various areas of machine learning theory.\n\nWEAKNESSES:\nOverall, the paper is nicely written. Some (minor) comments on weaknesses of the paper are:\n\n1. It is stated in the paper that the main result is \"near-optimal\". Can some elaborations be made on what is the \"optimal\" result, and, what are the technical difficulties causing the discrepancy between the results in this paper and the \"optimal\" result?\n\n2. Discrepancy in order of Theorem 1.8: the results in the paper give a tolerance requirement of the order \"$Ω(d)^{−k/32}$\" on $d$ while Fact 1.6. has the order \"$Ω(d)^{-(k+1)(1/4-c/2)}$\". In cases where $c$ is closer to $1/2$ it appears that the previous result is tighter, while the opposite is true when $c$ is closer to $0$. What is causing the discrepancy and why is this not reflected in the original comparison in line 62, i.e., is the order of magnitude listed in the equation on line 62-63 inaccurate?\n\nQUESTIONS:\nSome additional comments and questions:\n\n1. Possible typo on Line 289: \"i-th\" might have been a typo of \"k-th\".\n\n2. Lemma 3.10 requires that $d$ \"is at least a sufficiently large universal constant.\" How large does $d$ need to be in order for the results in the paper to be applicable?\n\nLIMITATIONS:\nThere are limited discussions on limitations available in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper discusses SQ lower bounds for Non-Gaussian component analysis. A very influential result by [DKS17] has established an SQ-lower bound suggesting d^m time as long as the non-Gaussian component's distribution A, satisfies (a) that the first m moments of A match the m moments of N(0,1) and (b) the \chi^2 between A and N(0,1) is finite.
The authors prove that the same SQ lower bound holds under only condition (a) [i.e., without assuming condition (b)]

## Correctness

From my quick check, the argument appears correct and sound.

STRENGTHS:
I find the result very nice and the contribution a crucial addition to the literature. 

It is quite interesting that the authors follow a direct approach, not using the "standard" SQ dimension approach introduced in [Feldman et al '2017].

WEAKNESSES:
A weakness is perhaps that the SQ lower bound technique seems to be very tailored to NGCA, as opposed to the Feldman et al 2017 result.

QUESTIONS:
My understanding is that by removing the \chi^2 assumption, the authors result also shows that SQ algorithms fail even when the support of A is finite. This is a quite interesting case, as it includes examples of A supported on a lattice, where lattice-based methods work in polynomial time (e.g. see [Zadik et al' Lattice-Based Methods Surpass Sum-of-Squares in Clustering], , [Diakonikolas et al' Non-Gaussian Component Analysis via Lattice Basis Reduction]). Hence, this implies a separation between SQ methods and poly-time methods in NGCA. In case that is not already known, I encourage the authors to add this interesting implication of their result.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06