Title: SQ Lower Bounds for Learning Mixtures of Linear Classifiers
Venue: NeurIPS 2023
Accepted: yes
Keywords: mixtures models

linear classifier

Statistical Query model

spherical designs
Abstract: We study the problem of learning mixtures of linear classifiers under Gaussian covariates.
Given sample access to a mixture of $r$ distributions on $\mathbb{R}^n$ of the form $(\mathbf{x},y_{\ell})$, $\ell \in [r]$,
where $\mathbf{x}\sim\mathcal{N}(0,\mathbf{I}_n)$ and

$y_\ell=\mathrm{sign}(\langle\mathbf{v}_{\ell},\mathbf{x}\rangle)$

for an unknown unit vector $\mathbf{v}_{\ell}$,

the goal is to learn the underlying distribution in total variation distance. Our main result is a Statistical Query (SQ) lower bound suggesting that known algorithms for this problem are essentially best possible,
even for the special case of uniform mixtures.
In particular, we show that the complexity of any SQ algorithm for the problem is $n^{\mathrm{poly}(1/\Delta) \log(r)}$,
where $\Delta$ is a lower bound on the pairwise $\ell_2$-separation between the $\mathbf{v}_{\ell}$'s.
The key technical ingredient underlying our result is a new construction of spherical designs on the unit sphere that may be of independent interest.

SQ Lower Bounds for Learning Mixtures of Linear Classifiers

Abstract

We study the problem of learning mixtures of linear classifiers under Gaussian covariates. Given sample access to a mixture of r distributions on R n of the form ( x , y ℓ ) , ℓ ∈ [ r ] , where x ∼ N (0 , I n ) and y = sign( ⟨ v ℓ , x ⟩ )) for an unknown unit vector v ℓ , the goal is to learn the underlying distribution in total variation distance. Our main result is a Statistical Query (SQ) lower bound suggesting that known algorithms for this problem are essentially best possible, even for the special case of uniform mixtures. In particular, we show that the complexity of any SQalgorithm for the problem is n poly(1 / ∆)log( r ) , where ∆ is a lower bound on the pairwise ℓ 2 -separation between the v ℓ 's. The key technical ingredient underlying our result is a new construction of spherical designs that may be of independent interest.

1 Introduction

The motivation behind this work is to understand the computational complexity of learning highdimensional latent variable (aka mixture) models. The task of learning various mixture models has a long history in statistics with an early focus on sample efficiency, starting with the pioneering work of Karl Pearson [Pea94] on learning Gaussian mixtures. During the past decades, an extensive line of work in machine learning and theoretical computer science has made significant progress on the computational aspects of this general question for a range of mixture models, including mixtures of high-dimensional Gaussians [Das99, AK01, VW02, AM05, KSV08, BV08, MV10, RV17, HL18, DKS18], mixtures of linear regressions [SJA16, LL18, CLS20, DK20], and more generally mixtures of experts [JJ94, XA09, ME14, MVKO19, HYJ22].

In this paper, we focus on mixtures of linear classifiers , a classical supervised probabilistic model that has been intensely investigated from both statistical and algorithmic standpoints [SIM14, LLYH17, GMP20, CDV22]. A linear classifier (or halfspace) is any Boolean function h : R n → {± 1 } of the form h ( x ) = sign( ⟨ v , x ⟩ ) , where v ∈ R n is known as the weight vector and the univariate function sign is defined by sign( t ) = 1 for t ≥ 0 and sign( t ) = -1 otherwise. For an integer r ≥ 2 , we can now formally describe an r -mixture of linear classifiers. The parameters of the model contain r unknown positive weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 , and r unknown unit vectors v 1 , . . . , v r ∈ R n . A random sample is drawn from the underlying distribution D ( x , y ) as follows: the sample oracle selects the index ℓ ∈ [ r ] with probability w ℓ , and we then receive a random point ( x , y ) ∈ R n × {± 1 } where x ∼ N (0 , I n ) and y = sign( ⟨ v ℓ , x ⟩ )) . The goal is to approximately estimate the model either by learning the underlying distribution D ( x , y ) in total variation distance (density estimation), or by approximately recovering the hidden parameters, i.e.,

w ℓ and v ℓ , ℓ ∈ [ r ] (parameter estimation). An algorithm for parameter estimation can be used for density estimation (because closeness in parameters can be shown to imply closeness in total variation distance). In that sense, parameter estimation is a harder problem.

Before we proceed to describe prior algorithmic work, we provide basic background on the sample complexity of the problem. We start by noting that density estimation for r -mixtures of linear classifiers on R n is information-theoretically solvable using poly( n, r ) samples (with the optimal bound being ˜ Θ( nr/ϵ 2 ) for total variation error ϵ ) without any assumptions on the components. In contrast, for parameter estimation to be information-theoretically solvable with polynomial sample complexity, some further assumptions are needed. The typical assumption involves some kind of pairwise separation between the component vectors and a lower bound on the mixing weights. Let

̸

∆ def = min i = j {∥ v i -v j ∥ 2 , ∥ v i + v j ∥ 2 } > 0 be the pairwise separation between the components and w min be the minimum mixing weight. Under these assumptions, the parameter estimation problem is solvable using poly( n, r, 1 / ∆ , 1 /w min , 1 /ϵ ) samples to achieve parameter error of ϵ . In both cases, the term 'information-theoretically solvable' is used to mean that a sample-efficient algorithm exists, without any constraints on computational efficiency. The main question addressed in this paper is to what extent a computationally efficient learning algorithm exists.

On the algorithmic front, [CDV22] gave provable parameter estimation algorithms under a ∆ -separation assumption. Specifically, [CDV22] provided two algorithms with different complexity guarantees. Their first algorithm has sample and computational complexity poly( n O (log( r ) / ∆ 2 ) , 1 /w min , 1 /ϵ ) , while their second algorithm has complexity poly(( n/ ∆) r , 1 /w min , 1 /ϵ ) . Here we focus on settings where the number of components r is large and cannot be viewed as a constant. For the sake of intuition, it is instructive to simplify these upper bounds for the regime of uniform mixtures (corresponding to the special case that w ℓ = 1 /r for all ℓ ∈ [ r ] ) and ϵ is not too small. For this regime, the complexity upper bound achieved in [CDV22] is min { n O (log( r ) / ∆ 2 ) , ( n/ ∆) O ( r ) } . Concretely, if the separation ∆ is Ω(1) or even 1 / polylog( r ) , the first term yields a quasi-polynomial upper bound. On the other hand, for ∆ = O (1 /r c ) , for a constant c > 0 , the resulting upper bound is n poly( r ) . In both regimes, we observe a superpolynomial gap between the information-theoretic sample complexity - which is poly( n, r, ∆) -and the sample complexity of the [CDV22] algorithms. It is thus natural to ask if this gap is inherent.

What is the complexity of learning mixtures of linear classifiers? Is there an algorithm with significantly better sample-time tradeoff ?

We study these questions in a well-studied restricted model of computation, known as the Statistical Query (SQ) model [Kea98] (and, via [BBH + 20], also for low-degree polynomial tests). Our main result is that in both of these models the complexity of the above-mentioned algorithms is essentially best possible. Along the way, we establish new results on the existence of spherical designs on the unit sphere that may be of independent interest.

1.1 Our Results

Basics on SQ Model SQ algorithms are a class of algorithms that, instead of having direct access to samples, are allowed to query expectations of bounded functions of the distribution. Formally, an SQ algorithm has access to the following standard oracle.

Definition 1.1 ( STAT Oracle) . Let D be a distribution on R n . A Statistical Query is a bounded function f : R n → [ -1 , 1] . For τ > 0 , the STAT( τ ) oracle responds to the query f with a value v such that | v -E x ∼ D [ f ( x )] | ≤ τ . A Statistical Query (SQ) algorithm is an algorithm whose objective is to learn an unknown distribution D by making adaptive calls to the STAT( τ ) oracle.

The SQ model was introduced in [Kea98]. Subsequently, the model has been extensively studied in a range of contexts [Fel16]). The class of SQ algorithms is broad and captures a range of known supervised learning algorithms. More broadly, several known algorithmic techniques in machine learning are known to be implementable using SQs [FGR + 17, FGV17].

Our main result is a near-optimal SQ lower bound for mixtures of linear classifiers that applies even for the uniform case. Specifically, we establish the following:

Theorem 1.2 (Main Result: SQ Lower Bound for Uniform Mixtures) . Let ϵ ≤ c ∆ /r for some universal constant c > 0 sufficiently small. Then, any SQ algorithm that learns a uniform mixture of

̸

linear classifiers with directions v 1 , . . . , v r ∈ S n -1 satisfying min i = j {∥ v i -v j ∥ 2 , ∥ v i + v j ∥ 2 } ≥ Ω(∆) for some r -1 / 10 ≤ ∆ < 1 , within error ϵ in total variation distance must either use queries of tolerance n -poly(1 / ∆)log r , or make at least 2 n Ω(1) queries.

Informally speaking, Theorem 1.2 shows that no SQ algorithm can perform density estimation for uniform mixtures of linear classifiers to small accuracy with a sub-exponential in n Ω(1) many queries, unless using queries of very small tolerance - that would require at least n poly(1 / ∆)log r samples to simulate. This result can be viewed as a near-optimal information-computation tradeoff for the problem, within the class of SQ algorithms. In more detail, for ∆ = Ω(1) , we obtain a quasipolynomial SQ lower bound of n Ω(log r ) ; while for ∆ = 1 /r c , for some constant 0 < c < 1 / 2 , we obtain an SQ lower bound of n poly( r ) . In both cases, our SQ lower bounds qualitatively match the previously known algorithmic guarantees [CDV22] (that are easily implementable in the SQ model).

Aconceptual implication of Theorem 1.2 is that the uniform (i.e., equal mixing weights) case is essentially as hard as the general case for density estimation of these mixtures. In contrast, for related mixture models, specifically for mixtures of Gaussians, there is recent evidence that restricting the weights may make the problem computationally easier [BS21].

Remark 1.3. We note that the condition ∆ ≥ r -c , for some constant 0 < c < 1 , is necessary in the statement of Theorem 1.2 for the following reason: the algorithmic result of [CDV22] has sample and computational complexity min { n O (log( r ) / ∆ 2 ) , ( n/ ∆) O ( r ) } , which will be ( n/ ∆) O ( r ) << n poly(1 / ∆)log r if ∆ is sufficiently small by inverse polynomial in r .

Remark 1.4. Our SQ lower bound result has immediate implications to another well-studied restricted computational model - that of low-degree polynomial tests [HS17, HKP + 17, Hop18]. [BBH + 20] established that (under certain assumptions) an SQ lower bound also implies a qualitatively similar lower bound in the low-degree model. This connection can be used as a black-box to show a similar lower bound for low-degree polynomials.

The key technical ingredient required for our SQ lower bound is a theorem establishing the existence of spherical designs with appropriate properties. The definition of a spherical design follows.

Definition 1.5 (Spherical Design) . Let t be an odd integer. A set of points x 1 , . . . , x r ∈ S n -1 is called a spherical t -design if E [ p ( x )] = 1 r ∑ r i =1 p ( x i ) holds for every homogeneous n -variate polynomial p of degree t , where the expectation is taken over the uniform distribution on the unit sphere S n -1 .

We note that this definition differs slightly from the usual definition of spherical design, which requires that the equation in Definition 1.5 holds for all polynomials p of degree at most t . However, by multiplying by powers of ∥ x ∥ 2 2 , we note that our definition implies that the equation holds for every odd polynomial of degree at most t , and it is sufficient for the requirement of establishing our SQ lower bound.

Spherical designs have been extensively studied in combinatorial design theory and a number of efficient constructions are known, see, e.g., [Ban79, DGS91, GP11, BRV13, Kan15, Wom18]. However, none of the known constructions seem to be compatible with our separation assumptions. We establish the following result that may be of independent interest in this branch of mathematics.

Theorem 1.6 (Efficient Spherical Design) . Let t be an odd integer and r ≥ ( n +2 t -1 n -1 ) 5 . Let y 1 , . . . , y r be uniform random vectors over S n -1 . Then, with probability at least 99/100, there exist unit vectors z i ∈ S n -1 very close to y i such that ( z 1 , . . . , z r ) form a spherical t -design.

We mention here that the optimal sample complexity for the existence of spherical t -design (for even t ) is r = Θ (( n + t n )) and our result matches this within a polynomial factor. Theorem 1.6 is essential for our construction of moment-matching. Roughly speaking, the resulting mixture matches moments in the sense we require if and only if the weight vectors ( v 1 , . . . , v r ) form a spherical design . We mention that the closeness between z i and y i is necessary since this leads to the separation of the hidden weight vectors in the mixture of linear classifiers. In order to guarantee the pairwise separation of the weight vectors v i 's, we note that with high probability the y i 's are pairwise separated and therefore if z i is sufficiently close to y i they will be too.

1.2 Technical Overview

Our starting point is the SQ lower bound technique of [DKS17] and its generalization in [DKPZ21]. In particular, our overall strategy is to find a mixture of homogeneous halfspaces in some 'small' number m of dimensions that matches its first k moments with the distribution on ( x , y ) , where y is independent of x . By appropriately embedding this low-dimensional construction into R n via a random projection of R n → R m , the SQ lower bound machinery of [DKPZ21] can be shown to imply that learning the projection requires either 2 n Ω(1) queries or some query of accuracy n -Ω( k ) . This generic step reduces our problem to finding an appropriate m -dimensional construction. Such a construction is the main technical contribution of this work.

Note that specifying a mixture of halfspaces is equivalent to specifying a probability distribution with support consisting of at most r unit vectors and specifying the orthogonal vectors of the halfspaces. It is not hard to see that the resulting mixture matches moments in the sense we require if and only if these vectors form a spherical k -design [DGS91]: namely, that for any odd polynomial p of degree less than k that the average value of p over our distribution is the same as the average value of p over the unit sphere (namely, equal to zero). The bulk of our problem now reduces to finding constructions of weighted designs, where: (1) The support size of the design is relatively small. (2) The points in the design are pairwise separated. (3) For equally weighted mixtures, we require that the weight of each vector in the design is uniform.

In m = 2 two dimensions, there is a relatively simple explicit construction that can be given. In particular, if we take k evenly spaced points over the unit circle for some odd k , this matches the first k -1 moments and has separation Ω(1 /k ) . (This is similar to an SQ construction in [DKKZ20] for a different setting.) Unfortunately, we cannot obtain better separation in two dimensions, since any k unit vectors in two dimensions will necessarily have some pair separated by at most O (1 /k ) . Therefore, if we want constructions with greater separation, we will need to pick larger values of m . Indeed, we show (Proposition 3.2) that a random collection of approximately m O ( k ) points can form the support of an appropriate design. This can be proved by applying linear programming duality. Unfortunately, this argument does not allow us to control the mixing weights. Specifically, it may be the case that the minimum weight is exponentially small which seems unnatural in practice.

The case of equal weights requires a significantly more sophisticated argument. In this case, we need to find a spherical design with uniform weight. Indeed, merely selecting a random support will no longer work. Although there is an extensive literature on finding efficient spherical designs [Ban79, DGS91, GP11, BRV13, Kan15, Wom18], none of the known constructions seem to be compatible with our separation assumptions. In particular, [BRV13] proves that for each r ≥ c m k m , there exists a spherical k -design, where c m is a constant depending only on the dimension m . Although it is plausible that their construction can be adapted to satisfy our separation requirement, their sample complexity has a potentially bad dependence on the dimension m . On the other hand, [Kan15] achieves the optimal sample complexity r up to polynomials for all m and k . However, the construction in [Kan15] definitely does not satisfy our separation requirement. We note that the sample complexity of r = Θ (( m + k k )) should be optimal, and our results are polynomial in this bound. In conclusion, to establish our SQ lower bound, we need to prove a new technical result. Our starting point to that end will be the work of [BRV13]. The basic plan here will be to select random unit vectors y 1 , y 2 , . . . , y r ∈ R n , and then - applying topological techniques - to show that there is some small perturbation set x 1 , x 2 , . . . , x r ∈ R n that is a spherical design. In particular, we will find a continuous function F mapping degreek odd polynomials to sets of r unit vectors such that, for all unit-norm polynomials p , if F ( p ) = ( z 1 , . . . , z r ) then ∑ r i =1 p ( z i ) > 0 . Given this statement, a standard fixed point theorem [CC06] will imply that our design can be found as F ( q ) for some q with norm less than one. To construct the mapping F , we start with the points y 1 , . . . , y r and perturb each y i in the direction of ∇ o p ( y i ) in order to try to increase the average value of p , where ∇ o p ( y i ) is the component of ∇ p ( y i ) orthogonal to the direction y i . Intuitively, this construction should work because with high probability the average value of p ( y i ) is already small (since the empirical average should approximate the true average), the average gradient of p ( y i ) is not too small, and the contributions to p ( z i ) coming from higher-order terms will not be large as long as z i is sufficiently close to y i . These facts can all be made precise with some careful analysis of spherical harmonics (Lemma 4.5).

Finally, we need to show that the hardness of learning the appropriate projection implies hardness of learning the mixture. By standard facts, it suffices to show that mixtures of linear classifiers

are far from the distribution ( x , y ) , where the uniform label y is independent of x , in total variation distance. To show this, we prove that the total variation distance between any projection and the distribution where y is independent of x is at least Ω(∆ /r ) (Lemma 3.8).

2 Preliminaries

For n ∈ Z + , we denote [ n ] def = { 1 , . . . , n } . For two distributions p, q over a probability space Ω , let d TV ( p, q ) = sup S ⊆ Ω | p ( S ) -q ( S ) | denote the total variation distance between p and q . In this article, we typically use small letters to denote random variables and vectors. For a real random variable x , we use E [ x ] to denote the expectation. We use Pr [ E ] and I [ E ] for the probability and the indicator of event E . Let N n denote the standard n -dimensional Gaussian distribution and N denote the standard univariate Gaussian distribution. Let S n -1 = { x ∈ R n : ∥ x ∥ 2 = 1 } denote the n -dimensional unit sphere. For a subset S ⊆ R n , we will use U ( S ) to denote the uniform distribution over S . We will use small boldface letters for vectors and capital boldface letters for matrices. Let ∥ x ∥ 2 be the ℓ 2 -norm of the vector x ∈ R n . For vectors u , v ∈ R n , we use ⟨ u , v ⟩ to denote their inner product. We denote by L 2 ( R n , N n ) the function space of all functions f : R n → R such that E z ∈N n [ f 2 ( z )] < ∞ . The usual inner product for this space is E z ∈N n [ f ( z ) g ( z )] .

For a matrix P ∈ R m × n , we denote ∥ P ∥ 2 , ∥ P ∥ F to be its spectral norm and Frobenius norm respectively. For a tensor A ∈ ( R n ) ⊗ k , let ∥ A ∥ 2 denote its spectral norm and ∥ A ∥ F denote its Frobenius norm. We will use A i 1 ,...,i k to denote the coordinate of the k -tensor A indexed by the k -tuple ( i 1 , . . . , i k ) . The inner product between k -tensors is defined by thinking of tensors as vectors with n k coordinates. We use the framework of Statistical Query (SQ) algorithms for problems over distributions [FGR + 17] and require the following standard definition.

Definition 2.1 (Decision/Testing Problem over Distributions) . Let D be a distribution and D be a family of distributions over R n . We denote by B ( D , D ) the decision (or hypothesis testing) problem in which the input distribution D ′ is promised to satisfy either (a) D ′ = D or (b) D ′ ∈ D , and the goal of the algorithm is to distinguish between these two cases.

Basics on VC-Inequality We recall the definition of VC dimension for a set system.

Definition 2.2 (VC-Dimension) . For a class C of boolean functions on a set X , the VC-dimension of C is the largest integer d such that there exist d points x 1 , . . . , x d ∈ X such that for any boolean function g : { x 1 , . . . , x d } → {± 1 } , there exists an f ∈ C satisfying f ( x i ) = g ( x i ) , 1 ≤ i ≤ d .

We will use the following probabilistic inequality.

Lemma 2.3 (VC inequality, see, e.g., [Ver18]) . Let C be a class of boolean functions on X with VC-dimension d , and let X be a distribution on X . Let ϵ > 0 and let N be an integer at least a sufficiently large constant multiple of d/ϵ 2 . Then, if X 1 , X 2 , . . . , X N are i.i.d. samples from X , we have that:

To apply the VC inequality in our context, we additionally need the following fact which gives the VC-dimension of the class of bounded-degree polynomial threshold functions (PTFs).

Fact 2.4. Let C k denote the class of degreek polynomial threshold functions (PTFs) on R m , namely the collection of functions of the form f ( x ) = sign( p ( x )) for some degree at most k real polynomial p . Then, the VC-dimension of C k is VC( C k ) = ( m + k k ) .

3 Warmup: SQ Lower Bounds for General Weight Mixtures

SQ Lower Bound Machinery. We start by defining the family of distributions that we will use to prove our SQ hardness result.

Definition 3.1 ([DKPZ21]) . Given a function g : R m → [ -1 , +1] , we define D g to be the class of distributions over R n × {± 1 } of the form ( x , y ) such that x ∼ N n and E [ y | x = z ] = g ( Uz ) , where U ∈ R m × n with UU ⊺ = I m .

The following proposition states that if g has zero low-degree moments, then distinguishing D g from the distribution ( x , y ) with x ∼ N n , y ∼ U ( {± 1 } ) is hard in the SQ model.

Proposition 3.2 ([DKPZ21]) . Let g : R m → [ -1 , 1] be such that E x ∼N m [ g ( x ) p ( x )] = 0 , for every polynomial p : R m → R of degree less than k , and D g be the class of distributions from Definition 3.1. Then, if m ≤ n a , for some constant a < 1 / 2 , any SQ algorithm that solves the decision problem B ( D g , N n ×U ( {± 1 } )) must either use queries of tolerance n -Ω( k ) , or make at least 2 n Ω(1) queries.

We will apply Proposition 3.2 to establish our SQ lower bound for learning mixtures of linear classifiers. The main technical contribution required to achieve this is the construction of a class of distributions D g , where each element in D g represents a distribution of mixture of linear classifiers. In particular, we will carefully choose some appropriate unit vectors v 1 , . . . , v r ∈ R m and nonnegative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 , such that v ℓ are pairwise separated by ∆ > 0 .

Let g ( z ) = ∑ r ℓ =1 w ℓ sign( v ⊺ ℓ z ) , z ∈ R m . For an arbitrary matrix U ∈ R m × n with UU ⊺ = I m , we denote by D U the instance of mixture of linear classifiers with weight vectors U ⊺ v 1 , . . . , U ⊺ v r and weights w 1 , . . . , w r . In this way, we have that

3.1 Low-degree Moment Matching

The following proposition shows that there exist unit vectors v 1 , . . . , v r and non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that v ℓ are pairwise separated by some parameter ∆ > 0 and the low-degree moments of g vanish. Note that since g is odd (as the sign function is odd), we only require E z ∈N m [ g ( z ) p ( z )] = 0 for every odd polynomial p of degree less than k .

Proposition 3.3. Let m = c log r log(1 / ∆) , where r -1 / 10 ≤ ∆ < 1 and 1 . 99 ≤ c ≤ 2 is a universal constant. Let k be a positive integer such that r ≥ C ( m + k k ) for some constant C > 0 sufficiently large. There exist vectors v 1 , . . . , v r over the unit sphere S m -1 and non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 , such that

E z ∼N m [ g ( z ) p ( z )] = 0 holds for every odd polynomial p of degree less than k , where g ( z ) = ∑ r ℓ =1 w ℓ sign( v ⊺ ℓ z ) .

∥ v i + v j ∥ 2 , ∥ v i -v j ∥ 2 ≥ Ω(∆) , ∀ 1 ≤ i < j ≤ r .

The proof proceeds as follows: Weuniformly sample unit vectors v 1 , . . . , v r ∈ S m -1 . Wewill prove by the probabilistic method that both statements in Proposition 3.3 hold with high probability. We begin by showing that Pr [min 1 ≤ i<j ≤ r {∥ v i -v j ∥ 2 , ∥ v i + v j ∥ 2 } < O (∆)] is small. By symmetry, it suffices to bound Pr [min 1 ≤ i<j ≤ r ∥ v i -v j ∥ 2 < O (∆)] . Let Θ = min 1 ≤ i<j ≤ r arccos ⟨ v i , v j ⟩ . We require the following facts:

Applying Fact 3.4 by taking γ = ( 1 50 κ m -1 ) 1 m -1 and r = 1 √ 50 κ m -1 ∆ m -1 yields that Pr [Θ ≥ ∆] ≥ 99 / 100 . Given some fixed random vectors v 1 , . . . , v r ∈ S m -1 , we will prove that with high probability there exist non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that E z ∼N m [ g ( z ) p ( z )] = 0 holds for every odd polynomial p of degree less than k , where g ( z ) = ∑ r ℓ =1 w ℓ sign( v ⊺ ℓ z ) . Noting that E z ∼N m [ g ( z ) p ( z )] = ∑ r ℓ =1 w ℓ E z ∼N m [ p ( z )sign( v ⊺ ℓ z )] , it suffices to bound the probability that there exist non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that ∑ r ℓ =1 w ℓ E z ∼N m [ p ( z )sign( v ⊺ ℓ z )] = 0 holds for every odd polynomial p of degree less than k . Our main technical lemma is the following:

Lemma 3.5. Let p : R m → R be a polynomial of degree less than k and f ∈ L 2 ( R , N ) . Let v ∈ S m -1 be a unit vector. We have that E z ∼N m [ p ( z ) f ( v ⊺ z )] is a polynomial in v of degree less than k .

The proof proceeds by analyzing p and f as Hermite expansions. For completeness, we defer the proof of Lemma 3.5 to Appendix B. From Lemma 3.5, we can write ∑ r ℓ =1 w ℓ E z ∼N m [ p ( z )sign( v ⊺ ℓ z )] = ∑ r ℓ =1 w ℓ q ( v ℓ ) for some odd polynomial q of degree less than k (since sign is odd). In this way, it suffices to show that with high probability there exist nonnegative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that ∑ r ℓ =1 w ℓ q ( v ℓ ) = 0 for all odd polynomials q of degree less than k . To prove this, we leverage the following lemma:

Lemma 3.6. The following two statements are equivalent.

There exist non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that ∑ r ℓ =1 w ℓ q ( v ℓ ) = 0 for all odd polynomials q of degree less than k .

There does not exist any odd polynomial q of degree less than k such that q ( v ℓ ) > 0 , 1 ≤ ℓ ≤ r .

The proof of Lemma 3.6 follows via a careful application of LP duality; see Appendix B.

Proof of Proposition 3.3. Let Θ = min 1 ≤ i<j ≤ r arccos ⟨ v i , v j ⟩ . Applying Fact 3.4 by taking γ = ( 1 50 κ m -1 ) 1 m -1 yields Pr [Θ ≥ ∆] ≥ Pr [ Θ ≥ γr -2 m -1 ] ≥ 99 / 100 (since our choice of m will imply r = 1 / √ 50 κ m -1 ∆ m -1 ). This will imply that

We then show that there exist non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that E z ∼N m [ g ( z ) p ( z )] = 0 holds for every odd polynomial p with degree less than k , where g ( z ) = ∑ r ℓ =1 w ℓ sign( v ⊺ ℓ z ) . By Lemma 3.5, it suffices to show that there exist non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that ∑ r ℓ =1 w ℓ q ( v ℓ ) = 0 for all odd polynomials q of degree less than k . By Fact 2.4, we have that r ≥ C ( m + k k ) = C VC( C k ) for some sufficiently large constant C > 0 . Note that for any odd polynomial q in m variables of degree less than k , we have that E [sign( q ( v ))] = 0 . Therefore, by the VC-inequality (Lemma 2.3), we have that

Finally, applying Lemma 3.6 completes our proof.

3.2 Proof of SQ Lower Bound for General Weights

Let k be a positive integer such that r ≥ C ( m + k k ) for some constant C > 0 sufficiently large. Let m = c log r log(1 / ∆) , where r -1 / 10 ≤ ∆ < 1 and 1 . 99 ≤ c ≤ 2 . By Proposition 3.3, there exist unit vectors v 1 , . . . , v r ∈ R m such that

There exist non-negative weights w 1 , . . . , w r with ∑ r ℓ =1 w ℓ = 1 such that E z ∼N m [ g ( z ) p ( z )] = 0 holds for every odd polynomial p with degree less than k , where g ( z ) = ∑ r ℓ =1 w ℓ sign( v ⊺ ℓ z ) .

∥ v i + v j ∥ 2 , ∥ v i -v j ∥ 2 ≥ Ω(∆) , ∀ 1 ≤ i < j ≤ r for some ∆ > 0 .

For an arbitrary matrix U ∈ R m × n with UU ⊺ = I m , denote by D U the instance of mixture of linear classifiers with weight vectors U ⊺ v 1 , . . . , U ⊺ v r and weights w 1 , . . . , w r . Let D g = {D U | U ∈ R m × n , UU ⊺ = I m } . We have that ∥ U ⊺ v i ± U ⊺ v j ∥ 2 = ∥ v i ± v j ∥ 2 ≥ Ω(∆) , 1 ≤ i < j ≤ r , and E ( x ,y ) ∼ D U [ y | x = z ] = ∑ r ℓ =1 w ℓ sign(( U ⊺ v ℓ ) ⊺ z ) = ∑ r ℓ =1 w ℓ sign( v ⊺ ℓ Uz ) = g ( Uz ) , z ∈ R n . By Proposition 3.2, any SQ algorithm for the decision problem B ( D g , N n ×U ( {± 1 } )) must either use queries of tolerance n -Ω( k ) , or make at least 2 n Ω(1) queries. The last step is to reduce the decision problem B ( D g , N n ×U ( {± 1 } )) to the problem of learning mixture of linear classifiers.

Claim 3.7 (see, e.g., Lemma 8.5 in [DK23]) . Suppose there exists an SQ algorithm to learn an unknown distribution in a family D to total variation distance ϵ using at most N statistical queries

of tolerance τ . Suppose furthermore that for each D ′ ∈ D we have that d TV ( D,D ′ ) > 2( τ + ϵ ) . Then there exists an SQ algorithm that solves the testing problem B ( D , D ) using at most n + 1 queries of tolerance τ .

To apply Claim 3.7, we need to show that the distribution D U in the class D g is sufficiently far from the null hypothesis D 0 = N n ×U ( {± 1 } ) in total variation distance.

Lemma 3.8. Let U ∈ R m × n with UU ⊺ = I m . We have that d TV ( D U , D 0 ) ≥ Ω(∆ /r ) .

Webriefly sketch the proof idea and defer the proof details to Appendix B. We consider w ℓ sign( v ⊺ ℓ x ) a halfspace of heaviest weight in D U , and pick points z and z ′ close to the defining hyperplane that are mirrors of each other over it. We note that under D 0 the expectations of y conditioned on x being z or z ′ are both 0, whereas under D U they likely (in particular, unless they are also on opposite sides of another halfspace) differ by at least w ℓ .

4 SQ Lower Bound for Uniform Mixtures via Spherical Designs

In this section, we prove our SQ lower bound for mixture of linear classifiers with uniform weights, thereby establishing Theorem 1.2. Our basic lower bound technique is essentially the same as in the previous section, but we need to construct a spherical design with uniform weight .

Proposition 4.1. Let m = c log r log(1 / ∆) , where r -1 / 10 ≤ ∆ < 1 and 1 . 99 ≤ c ≤ 2 is a constant. Let k be an odd integer such that r ≥ ( m +2 k -1 2 k ) 5 . There exist vectors v 1 , . . . , v r on S m -1 such that:

E z ∼N m [ g ( z ) p ( z )] = 0 holds for every odd polynomial p with degree less than k , where g ( z ) = 1 r ∑ r ℓ =1 sign( v ⊺ ℓ z ) .

By Lemma 3.5, we can write ∑ r ℓ =1 E z ∼N m [ p ( z )sign( v ⊺ ℓ z )] = ∑ r ℓ =1 q ( v ℓ ) for some odd polynomial q of degree less than k . Therefore, it suffices to show that with high probability there exist unit vectors v 1 , . . . , v r such that ∑ r ℓ =1 q ( v ℓ ) = 0 holds for every odd polynomial q of degree less than k . To achieve this, we will leverage some techniques from [BRV13].

4.1 Spherical Design Construction

Notation. We start by introducing some additional notations we will use throughout this section. Let P d t denote the set of homogeneous polynomials in d variables of some odd degree t . For any p, q ∈ P d t , we consider the inner product ⟨ p, q ⟩ = E x ∼U ( S d -1 ) [ p ( x ) q ( x )] . For any p ∈ P d t , we define ∥ p ∥ 2 2 = ⟨ p, p ⟩ . We denote by N t,d = ( t + d -1 d -1 ) to be the dimension of P d t and Ω d t = { p ∈ P d t | ∥ p ∥ 2 ≤ 1 } . Let ∂ Ω d t denote the boundary of Ω d t , i.e., ∂ Ω d t = { p ∈ P d t | ∥ p ∥ 2 = 1 } . In the remaining part of this section, we will assume that the underlying distribution (over the expectation) is U ( S d -1 ) .

The following theorem - a more detailed version of Theorem 1.6 - establishes the existence of a spherical t -design of size poly( N 2 t,d ) with the desired pairwise separation properties.

Theorem 4.2 (Spherical Design Construction) . Let t be an odd integer and r ≥ N 5 2 t,d . Let y 1 , . . . , y r be uniform random vectors over S d -1 . Then, with probability at least 99/100, there exist unit vectors z 1 , . . . , z r ∈ S d -1 such that ∥ z i -y i ∥ 2 ≤ O (1 /N 2 t,d ) , i ∈ [ r ] , and ( z 1 , . . . , z r ) form a spherical t -design.

To prove Theorem 4.2, we will start from the following result.

Theorem 4.3 ([BRV13]) . If there exists a continuous mapping F : P d t → ( S d -1 ) r such that for all p ∈ ∂ Ω d t , ∑ r i =1 p ( x i ( p )) > 0 , where F ( p ) = ( x 1 ( p ) , . . . , x r ( p )) , then there exists a polynomial p ∗ ∈ Ω d t such that E [ p ( x )] = 1 r ∑ r i =1 p ( x i ( p ∗ )) holds for every polynomial p ∈ P d t .

To apply Theorem 4.3, we need to find a continuous function F mapping P d t to ( S d -1 ) r such that for any p ∈ ∂ Ω d t , ∑ r i =1 p ( z i ) > 0 , where F ( p ) = ( z 1 , . . . , z r ) . We will construct the mapping

F as follows: we sample y 1 , . . . , y r uniformly over the unit sphere S d -1 , and then try to make the value of p ( y i ) larger by moving each point y i in the direction of the gradient. In particular, we let z i = y i + δ ∇ o p ( y i ) ∥ y i + δ ∇ o p ( y i ) ∥ 2 for some δ > 0 sufficiently small, where ∇ o p ( y ) is the component of ∇ p ( y ) orthogonal to the direction y . We will prove that with high probability, for any p ∈ ∂ Ω d t , ∑ r i =1 p ( z i ) > 0 . Intuitively, this works because of two facts:

With high probability over the choice of y i , for all p ∈ Ω d t , the average value of p ( y ) is already close to zero.

Moving in the direction of ∇ o p ( y i ) increases p ( y i ) by a notable amount.

Lemma 4.4. Let Ω be a subspace of polynomials in d variables with mean zero. Let N be the dimension of Ω . Let x 1 , . . . , x r be i.i.d. random vectors over S d -1 . Then, with probability at least 1 -N rη 2 , we have that for any p ∈ Ω , ∣ ∣ 1 r ∑ r i =1 p ( x i ) ∣ ∣ ≤ η ∥ p ∥ 2 .

To prove Lemma 4.4, we take an orthonormal basis p 1 , . . . , p N ∈ Ω and define p ( x ) def = [ p 1 ( x ) , . . . , p N ( x )] ⊺ . Noting that p ( x ) is a random vector with mean zero and covariance identity, applying Markov's inequality will yield the result. We defer the proof details to Appendix C.

Lemma 4.5. Let y ∈ S d -1 and 0 < δ ≤ 1 /N 2 2 t,d . For any p ∈ ∂ Ω d t , let z = y + δ ∇ o p ( y ) ∥ y + δ ∇ o p ( y ) ∥ 2 . We have that p ( z ) -p ( y ) ≥ Cδ ∥∇ o p ( y ) ∥ 2 2 for some universal constant 0 < C < 1 .

The proof of Lemma 4.5 follows by Taylor expansion, where the contributions to p ( z ) coming from higher-order terms will not be large as long as z is sufficiently close to y . We defer the proof details to Appendix C. By applying the above two lemmas, we establish the following:

Theorem 4.6. Let y 1 , . . . , y r be i.i.d. random vectors over S d -1 . Let δ = 1 /N 2 2 t,d and r ≥ N 5 2 t,d . We consider the mapping F : P d t → ( S d -1 ) r as follows: for any p ∈ P d t , let

where ∇ o p ( y ) is the component of ∇ p ( y ) orthogonal to the direction y . Let F ( p ) := ( z 1 , . . . , z r ) . Then, with probability at least 99/100, we have that for any p ∈ ∂ Ω d t , ∑ r i =1 p ( z i ) > 0 .

To prove Theorem 4.6, we consider ˜ p ( y ) = p ( y ) + Cδ ( ∥∇ o p ( y ) ∥ 2 2 -E [ ∥∇ o p ( y ) ∥ 2 2 ]) , where C, δ comes from Lemma 4.5. Noting that E [ ˜ p ( y )] = 0 and ˜ p ( y ) is a polynomial containing only monomials of degree 2 t, 2 t -2 , t, 0 , we are able to apply Lemma 4.4 to obtain the desired result. See Appendix C for the proof.

We now prove our main technical result, Theorem 4.2.

Proof of Theorem 4.2. Let δ = 1 /N 2 2 t,d . We consider the mapping F : P d t → ( S d -1 ) r as follows: for any p ∈ P d t , let z i = y i + δ ∇ o p ( y i ) ∥ y i + δ ∇ o p ( y i ) ∥ 2 , where ∇ o p ( y ) is the component of ∇ p ( y ) orthogonal to the direction y . By Theorem 4.6, with probability at least 99/100, we have that for any p ∈ ∂ Ω d t , ∑ r i =1 p ( z i ) > 0 . Applying Theorem 4.3 yields that there exists some p ∗ ∈ Ω d t such that F ( p ∗ ) = ( z ∗ 1 , . . . , z ∗ r ) form a spherical t -design. Furthermore, we can show that ∥ z ∗ i -y i ∥ 2 ≤ O (1 /N 2 t,d ) by elementary calculation (see Appendix C).

4.2 Proof of Theorem 1.2

We first prove Proposition 4.1 based on our construction of spherical t -design. To achieve this, it suffices to show that with high probability there exist v 1 , . . . , v r ∈ S m -1 such that

v 1 , . . . , v r is a spherical k -design.

∥ v i + v j ∥ 2 , ∥ v i -v j ∥ 2 ≥ Ω(∆) , ∀ 1 ≤ i < j ≤ r for some ∆ > 0 .

since our choice of m will imply that r = 1 / √ 50 κ m -1 ∆ m -1 . By Theorem 4.2, with probability at least 99/100, there exist unit vectors z ∗ 1 , . . . , z ∗ r ∈ S m -1 such that ∥ z ∗ i -y i ∥ 2 ≤ O (1 /N 2 k,m ) , i ∈ [ r ] and ( z ∗ 1 , . . . , z ∗ r ) form a spherical k -design. Therefore, for any odd homogeneous polynomial p in m variables of odd degree t < k , we have that

which implies that ∑ r i =1 q ( z ∗ i ) = 0 holds for any polynomial q in m variables of degree less than k . In addition, for every 1 ≤ i < j ≤ r , we have that

Proof of Theorem 1.2. Let m = c log r log(1 / ∆) , where r -1 / 10 ≤ ∆ < 1 and 1 . 99 ≤ c ≤ 2 is a constant. Let c ′ = (1 /e )(1 / ∆) 1 / (5 c ) -1 2 and k = c ′ m = ( (1 /e )(1 / ∆) 1 / (5 c ) -1 ) c log r 2 log(1 / ∆) . In this way, we will have that r ≥ N 5 2 k,m (see Appendix C for calculation details). Therefore, by Proposition 4.1, there exist vectors v 1 , . . . , v r ∈ S m -1 such that

E z ∼N m [ g ( z ) p ( z )] = 0 holds for every odd polynomial p of degree less than k , where g ( z ) = 1 r ∑ r ℓ =1 sign( v ⊺ ℓ z ) .

For an arbitrary matrix U ∈ R m × n with UU ⊺ = I m , denote by D U the instance of mixture of linear classifiers with weight vectors U ⊺ v 1 , . . . , U ⊺ v r . Let D g = {D U | U ∈ R m × n , UU ⊺ = I m } . By definition, we have that E ( x ,y ) ∼ D U [ y | x = z ] = 1 r ∑ r ℓ =1 sign(( U ⊺ v ℓ ) ⊺ z ) = 1 r ∑ r ℓ =1 sign( v ⊺ ℓ Uz ) = g ( Uz ) , z ∈ R n , and ∥ U ⊺ v i ± U ⊺ v j ∥ 2 = ∥ v i ± v j ∥ 2 ≥ Ω(∆) , 1 ≤ i < j ≤ r , since N 2 k,m ≥ Ω((1 / ∆) 1 . 89 ) (see Appendix C for the calculation details). Therefore by Proposition 3.2, any SQ algorithm that solves the decision problem B ( D g , N n ×U ( {± 1 } )) must either use queries of tolerance n -Ω( k ) , or make at least 2 n Ω(1) queries. By Lemma 3.8, for any U ∈ R m × n with UU ⊺ = I m , we have that d TV ( D U , D 0 ) ≥ Ω(∆ /r ) ≥ 2( n -Ω( k ) + ϵ ) , where D 0 = N n ×U ( {± 1 } ) . Therefore, by Claim 3.7, any SQ algorithm that learns a distribution in D g within error ϵ in total variation distance must either use queries of tolerance n -Ω(log r/ (∆ 1 / (5 c ) log(1 / ∆))) , or make at least 2 n Ω(1) queries. This completes the proof.

5 Conclusion

This work establishes a near-optimal Statistical Query (SQ) lower bound for learning uniform mixtures of linear classifiers under the Gaussian distribution. Our lower bound nearly matches prior algorithmic work on the problem [CDV22]. Our result applies for the simplest (and well-studied) distributional setting where the covariates are drawn from the standard Gaussian distribution. This directly implies similar information-computation tradeoffs for the setting that the covariates are drawn from a more general distribution family (e.g., an unknown subgaussian or log-concave distribution) that includes the standard normal.

From a technical perspective, we believe that our new efficient construction of spherical designs is a mathematical contribution of independent interest that could be used to establish SQ lower bounds for other related latent variable models (e.g., various mixtures of experts). A natural direction is to establish information-computation tradeoffs for a fixed non-Gaussian distribution on covariates (e.g., the uniform distribution over the Boolean hypercube), for which a different hardness construction is needed.

References

[AK01] S. Arora and R. Kannan. Learning mixtures of arbitrary Gaussians. In Proceedings of the 33rd Symposium on Theory of Computing , pages 247-257, 2001.

[AM05] D. Achlioptas and F. McSherry. On spectral learning of mixtures of distributions. In Proceedings of the Eighteenth Annual Conference on Learning Theory (COLT) , pages 458-469, 2005.

[Ban79] E. Bannai. On tight spherical designs. Journal of Combinatorial Theory, Series A , 26(1):38-47, 1979.

[BBH + 20] M. Brennan, G. Bresler, S. B. Hopkins, J. Li, and T. Schramm. Statistical query algorithms and low-degree tests are almost equivalent. arXiv preprint arXiv:2009.06107 , 2020. In 34th Annual Conference on Learning Theory (COLT 2021) .

[BRS + 18] J. S. Brauchart, A. B. Reznikov, E. B. Saff, I. H. Sloan, Y. G. Wang, and R. S. Womersley. Random point sets on the sphere-hole radii, covering, and separation. Experimental Mathematics , 27(1):62-81, 2018.

[BRV13] A. Bondarenko, D. Radchenko, and M. Viazovska. Optimal asymptotic bounds for spherical designs. Annals of mathematics , pages 443-452, 2013.

[BS21] R. D. Buhai and D. Steurer. Beyond parallel pancakes: Quasi-polynomial time guarantees for non-spherical gaussian mixtures. CoRR , abs/2112.05445, 2021.

[BV08] S. C. Brubaker and S. Vempala. Isotropic PCA and Affine-Invariant Clustering. In Proc. 49th IEEE Symposium on Foundations of Computer Science , pages 551-560, 2008.

[CC06] Y. J. Cho and Y. Q. Chen. Topological degree theory and applications . CRC Press, 2006.

[CDV22] A. Chen, A. De, and A. Vijayaraghavan. Algorithms for learning a mixture of linear classifiers. In International Conference on Algorithmic Learning Theory , pages 205226. PMLR, 2022.

[CLS20] S. Chen, J. Li, and Z. Song. Learning mixtures of linear regressions in subexponential time via fourier moments. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 587-600, 2020.

[Das99] S. Dasgupta. Learning mixtures of Gaussians. In Proceedings of the 40th Annual Symposium on Foundations of Computer Science , pages 634-644, 1999.

[DGS91] P. Delsarte, J. Goethals, and J. J. Seidel. Spherical codes and designs. In Geometry and Combinatorics , pages 68-93. Elsevier, 1991.

[DK20] I. Diakonikolas and D. M. Kane. Small covers for near-zero sets of polynomials and learning latent variable models. In Proceedings of the 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS 2020) , 2020.

[DK23] I. Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics . Cambridge university press, 2023.

[DKKZ20] I. Diakonikolas, D. M. Kane, V. Kontonis, and N. Zarifis. Algorithms and SQ lower bounds for PAC learning one-hidden-layer relu networks. In Conference on Learning Theory, COLT 2020 , volume 125 of Proceedings of Machine Learning Research , pages 1514-1539. PMLR, 2020.

[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Conference on Learning Theory, COLT 2021 , volume 134 of Proceedings of Machine Learning Research , pages 1552-1584. PMLR, 2021.

[DKS17] I. Diakonikolas, D. M. Kane, and A. Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017 , pages 73-84, 2017. Full version at http://arxiv.org/abs/1611.03473.

[DKS18] I. Diakonikolas, D. M. Kane, and A. Stewart. List-decodable robust mean estimation and learning mixtures of spherical gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 1047-1060, 2018. Full version available at https://arxiv.org/abs/1711.07211.

[Fel16] V. Feldman. Statistical query learning. In Encyclopedia of Algorithms , pages 20902095. Springer New York, 2016.

[FGR + 17] V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. Statistical algorithms and a lower bound for detecting planted cliques. J. ACM , 64(2):8:1-8:37, 2017.

[FGV17] V. Feldman, C. Guzman, and S. S. Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the TwentyEighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017 , pages 1265-1277. SIAM, 2017.

[GMP20] V. Gandikota, A. Mazumdar, and S. Pal. Recovery of sparse linear classifiers from mixture of responses. Advances in Neural Information Processing Systems , 33:1468814698, 2020.

[GP11] M. Gr¨ af and D. Potts. On the computation of spherical designs by a new optimization approach based on fast spherical fourier transforms. Numerische Mathematik , 119(4):699-724, 2011.

[HKP + 17] S. B. Hopkins, P. K. Kothari, A. Potechin, P. Raghavendra, T. Schramm, and D. Steurer. The power of sum-of-squares for detecting hidden structures. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017 , pages 720-731. IEEE Computer Society, 2017.

[HL18] S. B. Hopkins and J. Li. Mixture models, robustness, and sum of squares proofs. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 1021-1034, 2018.

[Hop18] S. B. Hopkins. Statistical inference and the sum of squares method . PhD thesis, Cornell University, 2018.

[HS17] S. B. Hopkins and D. Steurer. Efficient bayesian estimation from few samples: Community detection and related problems. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017 , pages 379-390. IEEE Computer Society, 2017.

[HYJ22] N. Ho, C. Yang, and M. I. Jordan. Convergence rates for gaussian mixtures of experts. The Journal of Machine Learning Research , 23(1):14523-14603, 2022.

[JJ94] M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation , 6(2):181-214, 1994.

[Kan15] D. M. Kane. Small designs for path-connected spaces and path-connected homogeneous spaces. Transactions of the American Mathematical Society , 367(9):6387-6414, 2015.

[Kea98] M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM , 45(6):983-1006, 1998.

[KSV08] R. Kannan, H. Salmasian, and S. Vempala. The spectral method for general mixture models. SIAM J. Comput. , 38(3):1141-1156, 2008.

[LL18] Y. Li and Y. Liang. Learning mixtures of linear regressions with nearly optimal complexity. In Conference On Learning Theory , pages 1125-1144. PMLR, 2018.

[LLYH17] W. Li, P. Liang, X. Yuan, and J. Hu. Non-local information for a mixture of multiple linear classifiers. In 2017 International Joint Conference on Neural Networks (IJCNN) , pages 3741-3746. IEEE, 2017.

[ME14] S. Masoudnia and R. Ebrahimpour. Mixture of experts: a literature survey. The Artificial Intelligence Review , 42(2):275, 2014.

[MV10] A. Moitra and G. Valiant. Settling the polynomial learnability of mixtures of Gaussians. In FOCS , pages 93-102, 2010.

[MVKO19] A. Makkuva, P. Viswanath, S. Kannan, and S. Oh. Breaking the gridlock in mixture-ofexperts: Consistent and efficient algorithms. In International Conference on Machine Learning , pages 4304-4313. PMLR, 2019.

[Pea94] K. Pearson. Contribution to the mathematical theory of evolution. Phil. Trans. Roy. Soc. A , 185:71-110, 1894.

[RV17] O. Regev and A. Vijayaraghavan. On learning mixtures of well-separated gaussians. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017 , pages 85-96, 2017.

[SIM14] Y. Sun, S. Ioannidis, and A. Montanari. Learning mixtures of linear classifiers. In International Conference on Machine Learning , pages 721-729. PMLR, 2014.

[SJA16] H. Sedghi, M. Janzamin, and A. Anandkumar. Provable tensor methods for learning mixtures of generalized linear models. In Artificial Intelligence and Statistics , pages 1223-1231. PMLR, 2016.

[Ver18] R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

[VW02] S. Vempala and G. Wang. A spectral algorithm for learning mixtures of distributions. In Proceedings of the 43rd Annual Symposium on Foundations of Computer Science , pages 113-122, 2002.

[Wom18] R. S. Womersley. Efficient spherical designs with good geometric properties. Contemporary computational mathematics-A celebration of the 80th birthday of Ian Sloan , pages 1243-1285, 2018.

[XA09] L. Xu and S. Amari. Combining classifiers and learning mixture-of-experts. In Encyclopedia of artificial intelligence , pages 318-326. IGI Global, 2009.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of learning mixtures of linear classifiers under Gaussian sampling. The paper provides a stastical query lower bound which demonstrates that known algorithms for the problem in the literature are essentially best possible, even for the special case of learning uniform mixtures. It further establishes the complexity of any SQ algorithm.

STRENGTHS:
+) Statistical query complexity of learning mixtures of linear classifiers
+) Efficient spherical designs to fullfill the required separation assumptions for the results to hold

WEAKNESSES:
This paper is primarily a theoretical work. The assumptions, problem setup, and results are only of theoretical interest. I am not sure if the results and technical tools are interesting to the broad machine learning systems society.

QUESTIONS:
The paper is well written and easy to follow. There are some notation issues. For example, in the abstract, "y=sign(<v_\ell,x>))" where y shall be y_\ell right? This also appeared in several places in the paper. Plus, one right parenthesis ) shall be removed.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA statistical query (SQ) algorithm is an algorithm that attempts to learn the data distribution $D$ by querying $f$ to the oracle who responses with $v$ such that $\\lvert v - E_{x\\sim D} f(x) \\rvert$ is small. In this paper, the authors study the problem of finding an SQ lower bound for learning a mixture of linear classifiers. The main ingredients are \n\n1. a result in [DKPZ21], which establishes an SQ lower bound on testing a distribution of $(x,y)\\in \\mathbb{R}^n\\times \\{-1,1\\}$ where $x$ is normal, $E[y \\vert x = z]=g(Uz)$ for some function $g$ with zero low-degree moments and some matrix $U$.\n2. Authors' construction of well-separated _spherical design_, that is, unit vectors $v_1,\\ldots,v_r$ such that $g(z) = \\frac1{r}\\sum_{\\ell=1}^r\\text{sign} (v_{\\ell}^Tz)$ has zero low-degree moments, as required in 1.\n\nAnd the last step is to turn the testing problem into the problem of learning the mixture of linear classifiers. The proof of the main theorem is presented in the main paper, whereas those of several lemmas are postponed to the appendix.\n\nSTRENGTHS:\n- The authors have established a new SQ lower bound which matches the algorithmic guarantees in some cases [CDV22].\n- The idea of designing and using spherical designs to obtain the lower bound is interesting and can be applied to other learning problems.\n- The writing is well-organized and it is easy to see the high-level idea of the proof.\n\nWEAKNESSES:\nMy only minor concern is that there is a lack of discussion of the main result. See Questions below for some questions that I have in mind.\n\nQUESTIONS:\n1. the authors have mentioned that the main result provides \"a near-optimal information-computation tradeoff for the problem\". Does this mean that the problem of finding an optimal tradeoff is still open.\n2. Related to 1., are there any open problems introduced, or related to this work?\n3. The idea of using spherical designs could be used to find SQ lower bound of other learning problems in which $g$ is an odd function. Are there any other learning problems that the authors have in mind?\n4. Can the result be extended to non-normal data?\n\nLIMITATIONS:\nThe authors should include a section on discussion, or limitations of the main results. For example, I have already mentioned the case where the data is non-normal. \n\nAlso, am I correct in assuming that the result only applies when $r$ is already known. Is there anything we can do when $r$ is not known?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides statistical query lower bounds for learning mixture of linear classifier.\n\nIn the problem of learning mixture of linear classifier, there are $r$ linear classifier $v_1, \\ldots ,v_r \\in \\mathbb{R}^{n}$. The input feature $x\\in \\mathbb{R}^{n}$ is draw from gaussian, the label $y = \\mathsf{sign}(v_\\ell^\\top x)$, where the index $\\ell$ is chosen with probability $w_{\\ell}$ (one has $w_1 +\\cdots +w_{r} = 1$). Previous work [CDV'22] give an algorithm of sample complexity $n^{\\log(r)/\\Delta^2}$ where $\\delta$ is the minimum separation between classifiers. The major contribution of this paper is to give an almost matching lower bound of $n^{\\log (r) \\cdot \\mathsf{poly}(\\Delta^{-1})}$, under the SQ model.\n\nFrom a high level, the paper follows the framework of [DKS'17] and reduces the problem to sphere design. The major technical contribution of this paper is to provide a spherical design and find a set of vector from unit ball that satisfies (1) none trivial pairwise distance and (2) the correlation with any low degree polynomial equal $0$. They obtain the spherical design using a topological argument inspired from [BRV'13].\n\n\n----------------------\nI have read the rebuttal and I would keep my positive evaluation of the paper.\n\nSTRENGTHS:\nThe paper gives an almost matching SQ lower bound for learning mixture linear classifier, the technique is novel and could have broad applications.\n\nWEAKNESSES:\nThere is no major weakness, though I have a few minor questions (to be specified later)\n\nQUESTIONS:\n(1) I have some concern with Lemma 3.5. In the statement, it is only required that $f \\in L^2(R, N)$, but it seems not true for every such function, right? If I understand correctly, the proof only requires $f = sign$ (i.e., only need to work for sign function), but even for sign function, the claim is sloppy, because $E_{z\\sim N_m} [p(z)f(v^\\top z)]$ scaling invariant with $v$, which seems not true for polynomial?\nPlease clarify this point.\n\n\n(2) Is the algorithm of [CDV'22] falls into the SQ framework?\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors prove a lower bound for the number of queries needed in the statistical query model for learning a mixture of linear classifiers. The statistical query model essentially makes oracle queries with a polynomial f(x) and the oracle responds with a value v such that: |v-E[f(x)]\le t, for some threshold t (the accuracy of the oracle). The main result in the paper is Theorem 1.2 which roughly states that to learn a n-dimensional mixture of linear classifiers within TV distance  \eps any algorithm must use queries with accuracy 1/poly(n) or must make 2^poly(n) statistical queries.

STRENGTHS:
- The lower bounds are claimed to be tight. 
- The problem of learning a mixture of linear classifiers is simple and elegant and thus important from a learning theory viewpoint.
- The technical results in the paper leading to Thm 4.2 synthesize ideas from topology and analysis. From a technical perspective the proof is indeed interesting and informative.

WEAKNESSES:
- The authors claim the lower bound is "qualitatively match" previous results by Chen et al. 2022. It would be good to reduce the Che et al result to the SQ model or vice versa. Otherwise the lower bound loses some of its significance.

QUESTIONS:
In Theorem 1.2, is it possible to have \Delta not depend upon r i.e., can the dependence \Delta > 1/r^10 be eliminated?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of learning mixture of linear classifiers with Gaussian covariates. Their primary result is a near-optimal SQ lower bound which applies even for the uniform mixture case. Moreover, as a purely mathematical result, they construct an efficient spherical design (under a stronger definition of the structure) with sample complexity within a polylogarithmic factor of the optimum.\n\nSTRENGTHS:\nThe paper is well motivated and introduced succinctly. The strongest portion of the submission are the first two sections which nicely introduce the problem, its inherent difficulties, and a discussion of the papers approach. I thought the technical overview was especially helpful to highlight the problem and challenges in proving the major results.\n\nWEAKNESSES:\nThe paper predominantly lacks in presentation of the mathematical results. Namely the major theoretical results presented in the main text are somewhat obtuse and hard to parse / verify the theoretical claims. This may be due my unfamiliarity with the surrounding literature, however the second half of the paper is difficult to follow in the logic of the proofs. While the technical overview helps to introduce the complexities of the analysis, these final sections are entirely non-intuitive. I would suggest to the authors to reduce the number of theorem / lemma statements in the main text and instead more carefully chose results (in line with the major points of Section 1.2). A more expansion on these results with more fleshed out proofs (or proof sketches) would make the paper considerably more readable within the short page limit of the conference. Additionally, it is somewhat hard to tell where the current result fits in with the prior works on the topic.\n\nQUESTIONS:\nWhat is the relation of the given problem to that of sparse recovery?\n\nDoes the spherical design technique used to prove your main result extend well beyond simple linear classifiers?\n\nHow can the novel techniques discussed here be applied more broadly outside of the presented problem instance?\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}