SlowGuess/NeurIPS_2023_OCR huggingface dataset:
University of Wisconsin-Madison  
ilias@cs.wisc.edu
Yuxin Sun
yxsun@cs.wisc.edu
We study the problem of learning mixtures of linear classifiers under Gaussian covariates. Given sample access to a mixture of  $r$  distributions on  $\mathbb{R}^n$  of the form  $(\mathbf{x}, y_{\ell})$ ,  $\ell \in [r]$ , where  $\mathbf{x} \sim \mathcal{N}(0, \mathbf{I}_n)$  and  $y = \mathrm{sign}(\langle \mathbf{v}_{\ell}, \mathbf{x} \rangle)$  for an unknown unit vector  $\mathbf{v}_{\ell}$ , the goal is to learn the underlying distribution in total variation distance. Our main result is a Statistical Query (SQ) lower bound suggesting that known algorithms for this problem are essentially best possible, even for the special case of uniform mixtures. In particular, we show that the complexity of any SQ algorithm for the problem is  $n^{\mathrm{poly}(1 / \Delta)\log(r)}$ , where  $\Delta$  is a lower bound on the pairwise  $\ell_2$ -separation between the  $\mathbf{v}_{\ell}$ 's. The key technical ingredient underlying our result is a new construction of spherical designs that may be of independent interest.
The motivation behind this work is to understand the computational complexity of learning high-dimensional latent variable (aka mixture) models. The task of learning various mixture models has a long history in statistics with an early focus on sample efficiency, starting with the pioneering work of Karl Pearson [Pea94] on learning Gaussian mixtures. During the past decades, an extensive line of work in machine learning and theoretical computer science has made significant progress on the computational aspects of this general question for a range of mixture models, including mixtures of high-dimensional Gaussians [Das99, AK01, VW02, AM05, KSV08, BV08, MV10, RV17, HL18, DKS18], mixtures of linear regressions [SJA16, LL18, CLS20, DK20], and more generally mixtures of experts [JJ94, XA09, ME14, MVKO19, HYJ22].
In this paper, we focus on mixtures of linear classifiers, a classical supervised probabilistic model that has been intensely investigated from both statistical and algorithmic standpoints [SIM14, LLYH17, GMP20, CDV22]. A linear classifier (or halfspace) is any Boolean function  $h: \mathbb{R}^n \to \{\pm 1\}$  of the form  $h(\mathbf{x}) = \mathrm{sign}(\langle \mathbf{v}, \mathbf{x} \rangle)$ , where  $\mathbf{v} \in \mathbb{R}^n$  is known as the weight vector and the univariate function sign is defined by  $\mathrm{sign}(t) = 1$  for  $t \geq 0$  and  $\mathrm{sign}(t) = -1$  otherwise. For an integer  $r \geq 2$ , we can now formally describe an  $r$ -mixture of linear classifiers. The parameters of the model contain  $r$  unknown positive weights  $w_1, \ldots, w_r$  with  $\sum_{\ell=1}^{r} w_\ell = 1$ , and  $r$  unknown unit vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r \in \mathbb{R}^n$ . A random sample is drawn from the underlying distribution  $D(\mathbf{x}, y)$  as follows: the sample oracle selects the index  $\ell \in [r]$  with probability  $w_\ell$ , and we then receive a random point  $(\mathbf{x}, y) \in \mathbb{R}^n \times \{\pm 1\}$  where  $\mathbf{x} \sim \mathcal{N}(0, \mathbf{I}_n)$  and  $y = \mathrm{sign}(\langle \mathbf{v}_\ell, \mathbf{x} \rangle)$ . The goal is to approximately estimate the model either by learning the underlying distribution  $D(\mathbf{x}, y)$  in total variation distance (density estimation), or by approximately recovering the hidden parameters, i.e.,
$w_{\ell}$  and  $\mathbf{v}_{\ell}$ ,  $\ell \in [r]$  (parameter estimation). An algorithm for parameter estimation can be used for density estimation (because closeness in parameters can be shown to imply closeness in total variation distance). In that sense, parameter estimation is a harder problem.
Before we proceed to describe prior algorithmic work, we provide basic background on the sample complexity of the problem. We start by noting that density estimation for  $r$ -mixtures of linear classifiers on  $\mathbb{R}^n$  is information-theoretically solvable using  $\mathrm{poly}(n,r)$  samples (with the optimal bound being  $\tilde{\Theta}(nr/\epsilon^2)$  for total variation error  $\epsilon$ ) without any assumptions on the components. In contrast, for parameter estimation to be information-theoretically solvable with polynomial sample complexity, some further assumptions are needed. The typical assumption involves some kind of pairwise separation between the component vectors and a lower bound on the mixing weights. Let  $\Delta \stackrel{\mathrm{def}}{=} \min_{i \neq j} \{ \| \mathbf{v}_i - \mathbf{v}_j \|_2, \| \mathbf{v}_i + \mathbf{v}_j \|_2 \} > 0$  be the pairwise separation between the components and  $w_{\mathrm{min}}$  be the minimum mixing weight. Under these assumptions, the parameter estimation problem is solvable using  $\mathrm{poly}(n,r,1/\Delta,1/w_{\mathrm{min}},1/\epsilon)$  samples to achieve parameter error of  $\epsilon$ . In both cases, the term "information-theoretically solvable" is used to mean that a sample-efficient algorithm exists, without any constraints on computational efficiency. The main question addressed in this paper is to what extent a computationally efficient learning algorithm exists.
On the algorithmic front, [CDV22] gave provable parameter estimation algorithms under a  $\Delta$ -separation assumption. Specifically, [CDV22] provided two algorithms with different complexity guarantees. Their first algorithm has sample and computational complexity  $\mathrm{poly}(n^{O(\log(r) / \Delta^2)}, 1 / w_{\min}, 1 / \epsilon)$ , while their second algorithm has complexity  $\mathrm{poly}((n / \Delta)^r, 1 / w_{\min}, 1 / \epsilon)$ . Here we focus on settings where the number of components  $r$  is large and cannot be viewed as a constant. For the sake of intuition, it is instructive to simplify these upper bounds for the regime of uniform mixtures (corresponding to the special case that  $w_\ell = 1 / r$  for all  $\ell \in [r]$ ) and  $\epsilon$  is not too small. For this regime, the complexity upper bound achieved in [CDV22] is  $\min\{n^{O(\log(r) / \Delta^2)}, (n / \Delta)^{O(r)}\}$ . Concretely, if the separation  $\Delta$  is  $\Omega(1)$  or even  $1 / \mathrm{poly} \log(r)$ , the first term yields a quasi-polynomial upper bound. On the other hand, for  $\Delta = O(1 / r^c)$ , for a constant  $c > 0$ , the resulting upper bound is  $n^{\mathrm{poly}(r)}$ . In both regimes, we observe a superpolynomial gap between the information-theoretic sample complexity — which is poly  $(n, r, \Delta)$  — and the sample complexity of the [CDV22] algorithms. It is thus natural to ask if this gap is inherent.
What is the complexity of learning mixtures of linear classifiers?
Is there an algorithm with significantly better sample-time tradeoff?
We study these questions in a well-studied restricted model of computation, known as the Statistical Query (SQ) model [Kea98] (and, via  $\mathrm{BBH}^{+}20$ , also for low-degree polynomial tests). Our main result is that in both of these models the complexity of the above-mentioned algorithms is essentially best possible. Along the way, we establish new results on the existence of spherical designs on the unit sphere that may be of independent interest.
Basics on SQ Model SQ algorithms are a class of algorithms that, instead of having direct access to samples, are allowed to query expectations of bounded functions of the distribution. Formally, an SQ algorithm has access to the following standard oracle.
Definition 1.1 (STAT Oracle). Let  $D$  be a distribution on  $\mathbb{R}^n$ . A Statistical Query is a bounded function  $f: \mathbb{R}^n \to [-1,1]$ . For  $\tau > 0$ , the STAT(τ) oracle responds to the query  $f$  with a value  $v$  such that  $|v - \mathbf{E}_{\mathbf{x} \sim D}[f(\mathbf{x})]| \leq \tau$ . A Statistical Query (SQ) algorithm is an algorithm whose objective is to learn an unknown distribution  $D$  by making adaptive calls to the STAT(τ) oracle.
The SQ model was introduced in [Kea98]. Subsequently, the model has been extensively studied in a range of contexts [Fel16]). The class of SQ algorithms is broad and captures a range of known supervised learning algorithms. More broadly, several known algorithmic techniques in machine learning are known to be implementable using SQs  $\mathrm{[FGR^{+}17}$  , FGV17].
Our main result is a near-optimal SQ lower bound for mixtures of linear classifiers that applies even for the uniform case. Specifically, we establish the following:
Theorem 1.2 (Main Result: SQ Lower Bound for Uniform Mixtures). Let  $\epsilon \leq c\Delta / r$  for some universal constant  $c > 0$  sufficiently small. Then, any SQ algorithm that learns a uniform mixture of
linear classifiers with directions  $\mathbf{v}_1, \ldots, \mathbf{v}_r \in \mathbb{S}^{n-1}$  satisfying  $\min_{i \neq j} \{\|\mathbf{v}_i - \mathbf{v}_j\|_2, \|\mathbf{v}_i + \mathbf{v}_j\|_2\} \geq \Omega(\Delta)$  for some  $r^{-1/10} \leq \Delta < 1$ , within error  $\epsilon$  in total variation distance must either use queries of tolerance  $n^{-\mathrm{poly}(1/\Delta)\log r}$ , or make at least  $2^{n^{\Omega(1)}}$  queries.
Informally speaking, Theorem 1.2 shows that no SQ algorithm can perform density estimation for uniform mixtures of linear classifiers to small accuracy with a sub-exponential in  $n^{\Omega(1)}$  many queries, unless using queries of very small tolerance – that would require at least  $n^{\mathrm{poly}(1/\Delta)\log r}$  samples to simulate. This result can be viewed as a near-optimal information-computation tradeoff for the problem, within the class of SQ algorithms. In more detail, for  $\Delta = \Omega(1)$ , we obtain a quasi-polynomial SQ lower bound of  $n^{\Omega(\log r)}$ ; while for  $\Delta = 1/r^{c}$ , for some constant  $0 < c < 1/2$ , we obtain an SQ lower bound of  $n^{\mathrm{poly}(r)}$ . In both cases, our SQ lower bounds qualitatively match the previously known algorithmic guarantees [CDV22] (that are easily implementable in the SQ model).
A conceptual implication of Theorem 1.2 is that the uniform (i.e., equal mixing weights) case is essentially as hard as the general case for density estimation of these mixtures. In contrast, for related mixture models, specifically for mixtures of Gaussians, there is recent evidence that restricting the weights may make the problem computationally easier [BS21].
Remark 1.3. We note that the condition  $\Delta \geq r^{-c}$ , for some constant  $0 < c < 1$ , is necessary in the statement of Theorem 1.2 for the following reason: the algorithmic result of [CDV22] has sample and computational complexity  $\min \{n^{O(\log (r) / \Delta^2)},(n / \Delta)^{O(r)}\}$ , which will be  $(n / \Delta)^{O(r)} << n^{\mathrm{poly}(1 / \Delta)\log r}$  if  $\Delta$  is sufficiently small by inverse polynomial in  $r$ .
Remark 1.4. Our SQ lower bound result has immediate implications to another well-studied restricted computational model — that of low-degree polynomial tests [HS17,  $\mathrm{HKP}^{+}17$ , Hop18].  $\left[\mathrm{BBH}^{+}20\right]$  established that (under certain assumptions) an SQ lower bound also implies a qualitatively similar lower bound in the low-degree model. This connection can be used as a black-box to show a similar lower bound for low-degree polynomials.
The key technical ingredient required for our SQ lower bound is a theorem establishing the existence of spherical designs with appropriate properties. The definition of a spherical design follows.
Definition 1.5 (Spherical Design). Let  $t$  be an odd integer. A set of points  $\mathbf{x}_1, \ldots, \mathbf{x}_r \in \mathbb{S}^{n-1}$  is called a spherical  $t$ -design if  $\mathbf{E}[p(\mathbf{x})] = \frac{1}{r} \sum_{i=1}^{r} p(\mathbf{x}_i)$  holds for every homogeneous  $n$ -variate polynomial  $p$  of degree  $t$ , where the expectation is taken over the uniform distribution on the unit sphere  $\mathbb{S}^{n-1}$ .
We note that this definition differs slightly from the usual definition of spherical design, which requires that the equation in Definition 1.5 holds for all polynomials  $p$  of degree at most  $t$ . However, by multiplying by powers of  $\| \mathbf{x} \|_2^2$ , we note that our definition implies that the equation holds for every odd polynomial of degree at most  $t$ , and it is sufficient for the requirement of establishing our SQ lower bound.
Spherical designs have been extensively studied in combinatorial design theory and a number of efficient constructions are known, see, e.g., [Ban79, DGS91, GP11, BRV13, Kan15, Wom18]. However, none of the known constructions seem to be compatible with our separation assumptions. We establish the following result that may be of independent interest in this branch of mathematics.
Theorem 1.6 (Efficient Spherical Design). Let  $t$  be an odd integer and  $r \geq \binom{n+2t-1}{n-1}^5$ . Let  $\mathbf{y}_1, \ldots, \mathbf{y}_r$  be uniform random vectors over  $\mathbb{S}^{n-1}$ . Then, with probability at least 99/100, there exist unit vectors  $\mathbf{z}_i \in \mathbb{S}^{n-1}$  very close to  $\mathbf{y}_i$  such that  $(\mathbf{z}_1, \ldots, \mathbf{z}_r)$  form a spherical  $t$ -design.
We mention here that the optimal sample complexity for the existence of spherical  $t$ -design (for even  $t$ ) is  $r = \Theta\left(\binom{n+t}{n}\right)$  and our result matches this within a polynomial factor. Theorem 1.6 is essential for our construction of moment-matching. Roughly speaking, the resulting mixture matches moments in the sense we require if and only if the weight vectors  $(\mathbf{v}_1,\dots,\mathbf{v}_r)$  form a spherical design. We mention that the closeness between  $\mathbf{z}_i$  and  $\mathbf{y}_i$  is necessary since this leads to the separation of the hidden weight vectors in the mixture of linear classifiers. In order to guarantee the pairwise separation of the weight vectors  $\mathbf{v}_i$ 's, we note that with high probability the  $\mathbf{y}_i$ 's are pairwise separated and therefore if  $\mathbf{z}_i$  is sufficiently close to  $\mathbf{y}_i$  they will be too.
Our starting point is the SQ lower bound technique of [DKS17] and its generalization in [DKPZ21]. In particular, our overall strategy is to find a mixture of homogeneous halfspaces in some "small" number  $m$  of dimensions that matches its first  $k$  moments with the distribution on  $(\mathbf{x},y)$ , where  $y$  is independent of  $\mathbf{x}$ . By appropriately embedding this low-dimensional construction into  $\mathbb{R}^n$  via a random projection of  $\mathbb{R}^n \to \mathbb{R}^m$ , the SQ lower bound machinery of [DKPZ21] can be shown to imply that learning the projection requires either  $2^{n^{\Omega(1)}}$  queries or some query of accuracy  $n^{-\Omega(k)}$ . This generic step reduces our problem to finding an appropriate  $m$ -dimensional construction. Such a construction is the main technical contribution of this work.
Note that specifying a mixture of halfspaces is equivalent to specifying a probability distribution with support consisting of at most  $r$  unit vectors and specifying the orthogonal vectors of the halfspaces. It is not hard to see that the resulting mixture matches moments in the sense we require if and only if these vectors form a spherical  $k$ -design [DGS91]: namely, that for any odd polynomial  $p$  of degree less than  $k$  that the average value of  $p$  over our distribution is the same as the average value of  $p$  over the unit sphere (namely, equal to zero). The bulk of our problem now reduces to finding constructions of weighted designs, where: (1) The support size of the design is relatively small. (2) The points in the design are pairwise separated. (3) For equally weighted mixtures, we require that the weight of each vector in the design is uniform.
In  $m = 2$  two dimensions, there is a relatively simple explicit construction that can be given. In particular, if we take  $k$  evenly spaced points over the unit circle for some odd  $k$ , this matches the first  $k - 1$  moments and has separation  $\Omega(1 / k)$ . (This is similar to an SQ construction in [DKKZ20] for a different setting.) Unfortunately, we cannot obtain better separation in two dimensions, since any  $k$  unit vectors in two dimensions will necessarily have some pair separated by at most  $O(1 / k)$ . Therefore, if we want constructions with greater separation, we will need to pick larger values of  $m$ . Indeed, we show (Proposition 3.2) that a random collection of approximately  $m^{O(k)}$  points can form the support of an appropriate design. This can be proved by applying linear programming duality. Unfortunately, this argument does not allow us to control the mixing weights. Specifically, it may be the case that the minimum weight is exponentially small which seems unnatural in practice.
The case of equal weights requires a significantly more sophisticated argument. In this case, we need to find a spherical design with uniform weight. Indeed, merely selecting a random support will no longer work. Although there is an extensive literature on finding efficient spherical designs [Ban79, DGS91, GP11, BRV13, Kan15, Wom18], none of the known constructions seem to be compatible with our separation assumptions. In particular, [BRV13] proves that for each  $r \geq c_m k^m$ , there exists a spherical  $k$ -design, where  $c_m$  is a constant depending only on the dimension  $m$ . Although it is plausible that their construction can be adapted to satisfy our separation requirement, their sample complexity has a potentially bad dependence on the dimension  $m$ . On the other hand, [Kan15] achieves the optimal sample complexity  $r$  up to polynomials for all  $m$  and  $k$ . However, the construction in [Kan15] definitely does not satisfy our separation requirement. We note that the sample complexity of  $r = \Theta\left(\binom{m+k}{k}\right)$  should be optimal, and our results are polynomial in this bound. In conclusion, to establish our SQ lower bound, we need to prove a new technical result. Our starting point to that end will be the work of [BRV13]. The basic plan here will be to select random unit vectors  $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_r \in \mathbb{R}^n$ , and then — applying topological techniques — to show that there is some small perturbation set  $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_r \in \mathbb{R}^n$  that is a spherical design. In particular, we will find a continuous function  $F$  mapping degree-  $k$  odd polynomials to sets of  $r$  unit vectors such that, for all unit-norm polynomials  $p$ , if  $F(p) = (\mathbf{z}_1, \ldots, \mathbf{z}_r)$  then  $\sum_{i=1}^{r} p(\mathbf{z}_i) > 0$ . Given this statement, a standard fixed point theorem [CC06] will imply that our design can be found as  $F(q)$  for some  $q$  with norm less than one. To construct the mapping  $F$ , we start with the points  $\mathbf{y}_1, \ldots, \mathbf{y}_r$  and perturb each  $\mathbf{y}_i$  in the direction of  $\nabla_o p(\mathbf{y}_i)$  in order to try to increase the average value of  $p$ , where  $\nabla_o p(\mathbf{y}_i)$  is the component of  $\nabla p(\mathbf{y}_i)$  orthogonal to the direction  $\mathbf{y}_i$ . Intuitively, this construction should work because with high probability the average value of  $p(\mathbf{y}_i)$  is already small (since the empirical average should approximate the true average), the average gradient of  $p(\mathbf{y}_i)$  is not too small, and the contributions to  $p(\mathbf{z}_i)$  coming from higher-order terms will not be large as long as  $\mathbf{z}_i$  is sufficiently close to  $\mathbf{y}_i$ . These facts can all be made precise with some careful analysis of spherical harmonics (Lemma 4.5).
Finally, we need to show that the hardness of learning the appropriate projection implies hardness of learning the mixture. By standard facts, it suffices to show that mixtures of linear classifiers
are far from the distribution  $(\mathbf{x},y)$ , where the uniform label  $y$  is independent of  $\mathbf{x}$ , in total variation distance. To show this, we prove that the total variation distance between any projection and the distribution where  $y$  is independent of  $\mathbf{x}$  is at least  $\Omega (\Delta /r)$  (Lemma 3.8).
For  $n \in \mathbb{Z}_+$ , we denote  $[n] \stackrel{\mathrm{def}}{=} \{1, \ldots, n\}$ . For two distributions  $p, q$  over a probability space  $\Omega$ , let  $d_{\mathrm{TV}}(p, q) = \sup_{S \subseteq \Omega} |p(S) - q(S)|$  denote the total variation distance between  $p$  and  $q$ . In this article, we typically use small letters to denote random variables and vectors. For a real random variable  $x$ , we use  $\mathbf{E}[x]$  to denote the expectation. We use  $\mathbf{Pr}[\mathcal{E}]$  and  $\mathbb{I}[\mathcal{E}]$  for the probability and the indicator of event  $\mathcal{E}$ . Let  $\mathcal{N}_n$  denote the standard  $n$ -dimensional Gaussian distribution and  $\mathcal{N}$  denote the standard univariate Gaussian distribution. Let  $\mathbb{S}^{n-1} = \{\mathbf{x} \in \mathbb{R}^n : \| \mathbf{x} \|_2 = 1\}$  denote the  $n$ -dimensional unit sphere. For a subset  $S \subseteq \mathbb{R}^n$ , we will use  $\mathcal{U}(S)$  to denote the uniform distribution over  $S$ . We will use small boldface letters for vectors and capital boldface letters for matrices. Let  $\| \mathbf{x} \|_2$  be the  $\ell^2$ -norm of the vector  $\mathbf{x} \in \mathbb{R}^n$ . For vectors  $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$ , we use  $\langle \mathbf{u}, \mathbf{v} \rangle$  to denote their inner product. We denote by  $\mathcal{L}^2(\mathbb{R}^n, \mathcal{N}_n)$  the function space of all functions  $f : \mathbb{R}^n \to \mathbb{R}$  such that  $\mathbf{E}_{\mathbf{z} \in \mathcal{N}_n}[f^2(\mathbf{z})] < \infty$ . The usual inner product for this space is  $\mathbf{E}_{\mathbf{z} \in \mathcal{N}_n}[f(\mathbf{z})g(\mathbf{z})]$ .
For a matrix  $\mathbf{P} \in \mathbb{R}^{m \times n}$ , we denote  $\| \mathbf{P} \|_2$ ,  $\| \mathbf{P} \|_F$  to be its spectral norm and Frobenius norm respectively. For a tensor  $A \in (\mathbb{R}^n)^{\otimes k}$ , let  $\| A \|_2$  denote its spectral norm and  $\| A \|_F$  denote its Frobenius norm. We will use  $A_{i_1, \ldots, i_k}$  to denote the coordinate of the  $k$ -tensor  $A$  indexed by the  $k$ -tuple  $(i_1, \ldots, i_k)$ . The inner product between  $k$ -tensors is defined by thinking of tensors as vectors with  $n^k$  coordinates. We use the framework of Statistical Query (SQ) algorithms for problems over distributions  $[\mathrm{FGR}^+ 17]$  and require the following standard definition.
Definition 2.1 (Decision/Testing Problem over Distributions). Let  $D$  be a distribution and  $\mathcal{D}$  be a family of distributions over  $\mathbb{R}^n$ . We denote by  $\mathcal{B}(\mathcal{D},D)$  the decision (or hypothesis testing) problem in which the input distribution  $D^{\prime}$  is promised to satisfy either (a)  $D^{\prime} = D$  or (b)  $D^{\prime}\in \mathcal{D}$ , and the goal of the algorithm is to distinguish between these two cases.
Basics on VC-Inequality We recall the definition of VC dimension for a set system.
Definition 2.2 (VC-Dimension). For a class  $\mathcal{C}$  of boolean functions on a set  $\mathcal{X}$ , the VC-dimension of  $\mathcal{C}$  is the largest integer  $d$  such that there exist  $d$  points  $x_{1},\ldots ,x_{d}\in \mathcal{X}$  such that for any boolean function  $g:\{x_1,\dots,x_d\} \to \{\pm 1\}$ , there exists an  $f\in \mathcal{C}$  satisfying  $f(x_{i}) = g(x_{i}),1\leq i\leq d$
We will use the following probabilistic inequality.
Lemma 2.3 (VC inequality, see, e.g., [Ver18]). Let  $\mathcal{C}$  be a class of boolean functions on  $\mathcal{X}$  with VC-dimension  $d$ , and let  $X$  be a distribution on  $\mathcal{X}$ . Let  $\epsilon > 0$  and let  $N$  be an integer at least a sufficiently large constant multiple of  $d / \epsilon^2$ . Then, if  $X_1, X_2, \ldots, X_N$  are i.i.d. samples from  $X$ , we have that:
$$
\mathbf {P r} \left[ \sup _ {f \in \mathcal {C}} \left| \frac {\sum_ {j = 1} ^ {N} f (X _ {j})}{N} - \mathbf {E} [ f (X) ] \right| > \epsilon \right] = \exp (- \Omega (N \epsilon^ {2})).
$$
To apply the VC inequality in our context, we additionally need the following fact which gives the VC-dimension of the class of bounded-degree polynomial threshold functions (PTFs).