SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fact 2.4. Let  $\mathcal{C}_k$  denote the class of degree- $k$  polynomial threshold functions (PTFs) on  $\mathbb{R}^m$ , namely the collection of functions of the form  $f(\mathbf{x}) = \mathrm{sign}(p(\mathbf{x}))$  for some degree at most  $k$  real polynomial  $p$ . Then, the VC-dimension of  $\mathcal{C}_k$  is  $\mathrm{VC}(\mathcal{C}_k) = \binom{m+k}{k}$ .
3 Warmup: SQ Lower Bounds for General Weight Mixtures
SQ Lower Bound Machinery. We start by defining the family of distributions that we will use to prove our SQ hardness result.
Definition 3.1 ([DKPZ21]). Given a function  $g: \mathbb{R}^m \to [-1, +1]$ , we define  $\mathcal{D}_g$  to be the class of distributions over  $\mathbb{R}^n \times \{\pm 1\}$  of the form  $(\mathbf{x}, y)$  such that  $\mathbf{x} \sim \mathcal{N}_n$  and  $\mathbf{E}[y \mid \mathbf{x} = \mathbf{z}] = g(\mathbf{U}\mathbf{z})$  where  $\mathbf{U} \in \mathbb{R}^{m \times n}$  with  $\mathbf{UU}^\intercal = \mathbf{I}_m$ .
The following proposition states that if  $g$  has zero low-degree moments, then distinguishing  $\mathcal{D}_g$  from the distribution  $(\mathbf{x}, y)$  with  $\mathbf{x} \sim \mathcal{N}_n, y \sim \mathcal{U}(\{\pm 1\})$  is hard in the SQ model.
Proposition 3.2 ([DKPZ21]). Let  $g: \mathbb{R}^m \to [-1,1]$  be such that  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_m}[g(\mathbf{x})p(\mathbf{x})] = 0$ , for every polynomial  $p: \mathbb{R}^m \to \mathbb{R}$  of degree less than  $k$ , and  $\mathcal{D}_g$  be the class of distributions from Definition 3.1. Then, if  $m \leq n^a$ , for some constant  $a < 1/2$ , any SQ algorithm that solves the decision problem  $\mathcal{B}(\mathcal{D}_g, \mathcal{N}_n \times \mathcal{U}(\{\pm 1\}))$  must either use queries of tolerance  $n^{-\Omega(k)}$ , or make at least  $2^{n^{\Omega(1)}}$  queries.
We will apply Proposition 3.2 to establish our SQ lower bound for learning mixtures of linear classifiers. The main technical contribution required to achieve this is the construction of a class of distributions  $\mathcal{D}_g$ , where each element in  $\mathcal{D}_g$  represents a distribution of mixture of linear classifiers. In particular, we will carefully choose some appropriate unit vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r \in \mathbb{R}^m$  and nonnegative weights  $w_1, \ldots, w_r$  with  $\sum_{\ell=1}^{r} w_\ell = 1$ , such that  $\mathbf{v}_\ell$  are pairwise separated by  $\Delta > 0$ .
Let  $g(\mathbf{z}) = \sum_{\ell=1}^{r} w_{\ell} \mathrm{sign}(\mathbf{v}_{\ell}^{\intercal} \mathbf{z}), \mathbf{z} \in \mathbb{R}^{m}$ . For an arbitrary matrix  $\mathbf{U} \in \mathbb{R}^{m \times n}$  with  $\mathbf{U} \mathbf{U}^{\intercal} = \mathbf{I}_{m}$ , we denote by  $D_{\mathbf{U}}$  the instance of a mixture of linear classifiers with weight vectors  $\mathbf{U}^{\intercal} \mathbf{v}_{1}, \ldots, \mathbf{U}^{\intercal} \mathbf{v}_{r}$  and weights  $w_{1}, \ldots, w_{r}$ . In this way, we have that
$$
\mathbf {E} _ {(\mathbf {x}, y) \sim D _ {\mathbf {U}}} [ y \mid \mathbf {x} = \mathbf {z} ] = \sum_ {\ell = 1} ^ {r} w _ {\ell} \mathrm {s i g n} ((\mathbf {U} ^ {\intercal} \mathbf {v} _ {\ell}) ^ {\intercal} \mathbf {z}) = \sum_ {\ell = 1} ^ {r} w _ {\ell} \mathrm {s i g n} (\mathbf {v} _ {\ell} ^ {\intercal} \mathbf {U} \mathbf {z}) = g (\mathbf {U} \mathbf {z}), \mathbf {z} \in \mathbb {R} ^ {n}.
$$
3.1 Low-degree Moment Matching
The following proposition shows that there exist unit vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r$  and non-negative weights  $w_1, \ldots, w_r$  with  $\sum_{\ell=1}^{r} w_\ell = 1$  such that  $\mathbf{v}_\ell$  are pairwise separated by some parameter  $\Delta > 0$  and the low-degree moments of  $g$  vanish. Note that since  $g$  is odd (as the sign function is odd), we only require  $\mathbf{E}_{\mathbf{z} \in \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = 0$  for every odd polynomial  $p$  of degree less than  $k$ .
Proposition 3.3. Let  $m = \frac{c\log r}{\log(1 / \Delta)}$ , where  $r^{-1 / 10} \leq \Delta < 1$  and  $1.99 \leq c \leq 2$  is a universal constant. Let  $k$  be a positive integer such that  $r \geq C\binom{m+k}{k}$  for some constant  $C > 0$  sufficiently large. There exist vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r$  over the unit sphere  $\mathbb{S}^{m-1}$  and non-negative weights  $w_1, \ldots, w_r$  with  $\sum_{\ell=1}^{r} w_\ell = 1$ , such that
-  $\mathbf{E}_{\mathbf{z} \sim \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = 0$  holds for every odd polynomial  $p$  of degree less than  $k$ , where  $g(\mathbf{z}) = \sum_{\ell=1}^{r} w_\ell \mathrm{sign}(\mathbf{v}_\ell^\intercal \mathbf{z})$ .
-  $\| \mathbf{v}_i + \mathbf{v}_j\| _2,\| \mathbf{v}_i - \mathbf{v}_j\| _2\geq \Omega (\Delta),\forall 1\leq i < j\leq r.$
The proof proceeds as follows: We uniformly sample unit vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r \in \mathbb{S}^{m-1}$ . We will prove by the probabilistic method that both statements in Proposition 3.3 hold with high probability. We begin by showing that  $\mathbf{Pr}[\min_{1 \leq i < j \leq r} \{\|\mathbf{v}_i - \mathbf{v}_j\|_2, \|\mathbf{v}_i + \mathbf{v}_j\|_2\} < O(\Delta)]$  is small. By symmetry, it suffices to bound  $\mathbf{Pr}[\min_{1 \leq i < j \leq r} \|\mathbf{v}_i - \mathbf{v}_j\|_2 < O(\Delta)]$ . Let  $\Theta = \min_{1 \leq i < j \leq r} \arccos \langle \mathbf{v}_i, \mathbf{v}_j \rangle$ . We require the following facts:
Fact 3.4 (Proposition 3.5 in [BRS+18]).  $\mathbf{Pr}\left[\Theta \geq \gamma r^{-\frac{2}{m - 1}}\right] \geq 1 - \frac{\kappa_{m - 1}}{2}\gamma^{m - 1}$ , where  $\kappa_{m} = \frac{\Gamma((m + 1) / 2)}{m\sqrt{\pi}\Gamma(m / 2)} \in \left[\frac{1}{m}\sqrt{\frac{m - 1}{2\pi}}, \frac{1}{m}\sqrt{\frac{m + 1}{2\pi}}\right]$ .
Applying Fact 3.4 by taking  $\gamma = \left(\frac{1}{50\kappa_{m - 1}}\right)^{\frac{1}{m - 1}}$  and  $r = \frac{1}{\sqrt{50\kappa_{m - 1}\Delta^{m - 1}}}$  yields that  $\mathbf{Pr}[\Theta \geq \Delta ]\geq 99 / 100$ . Given some fixed random vectors  $\mathbf{v}_1,\ldots ,\mathbf{v}_r\in \mathbb{S}^{m - 1}$ , we will prove that with high probability there exist non-negative weights  $w_{1},\dots,w_{r}$  with  $\sum_{\ell = 1}^{r}w_{\ell} = 1$  such that  $\mathbf{E}_{\mathbf{z}\sim \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = 0$  holds for every odd polynomial  $p$  of degree less than  $k$ , where  $g(\mathbf{z}) = \sum_{\ell = 1}^{r}w_{\ell}\mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{z})$ . Noting that  $\mathbf{E}_{\mathbf{z}\sim \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = \sum_{\ell = 1}^{r}w_{\ell}\mathbf{E}_{\mathbf{z}\sim \mathcal{N}_m}[p(\mathbf{z})\mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{z})]$ , it suffices to bound the probability that there exist non-negative weights  $w_{1},\ldots ,w_{r}$  with  $\sum_{\ell = 1}^{r}w_{\ell} = 1$  such that  $\sum_{\ell = 1}^{r}w_{\ell}\mathbf{E}_{\mathbf{z}\sim \mathcal{N}_m}[p(\mathbf{z})\mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{z})] = 0$  holds for every odd polynomial  $p$  of degree less than  $k$ . Our main technical lemma is the following:
Lemma 3.5. Let  $p: \mathbb{R}^m \to \mathbb{R}$  be a polynomial of degree less than  $k$  and  $f \in \mathcal{L}^2(\mathbb{R},\mathcal{N})$ . Let  $\mathbf{v} \in \mathbb{S}^{m-1}$  be a unit vector. We have that  $\mathbf{E}_{\mathbf{z} \sim \mathcal{N}_m}[p(\mathbf{z})f(\mathbf{v}^\top \mathbf{z})]$  is a polynomial in  $\mathbf{v}$  of degree less than  $k$ .
The proof proceeds by analyzing  $p$  and  $f$  as Hermite expansions. For completeness, we defer the proof of Lemma 3.5 to Appendix B. From Lemma 3.5, we can write  $\sum_{\ell=1}^{r} w_{\ell} \mathbf{E}_{\mathbf{z} \sim \mathcal{N}_m} [p(\mathbf{z}) \mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}} \mathbf{z})] = \sum_{\ell=1}^{r} w_{\ell} q(\mathbf{v}_{\ell})$  for some odd polynomial  $q$  of degree less than  $k$  (since sign is odd). In this way, it suffices to show that with high probability there exist nonnegative weights  $w_1, \ldots, w_r$  with  $\sum_{\ell=1}^{r} w_{\ell} = 1$  such that  $\sum_{\ell=1}^{r} w_{\ell} q(\mathbf{v}_{\ell}) = 0$  for all odd polynomials  $q$  of degree less than  $k$ . To prove this, we leverage the following lemma:
Lemma 3.6. The following two statements are equivalent.
1. There exist non-negative weights  $w_{1},\ldots ,w_{r}$  with  $\sum_{\ell = 1}^{r}w_{\ell} = 1$  such that  $\sum_{\ell = 1}^{r}w_{\ell}q(\mathbf{v}_{\ell}) = 0$  for all odd polynomials  $q$  of degree less than  $k$ .
2. There does not exist any odd polynomial  $q$  of degree less than  $k$  such that  $q(\mathbf{v}_{\ell}) > 0, 1 \leq \ell \leq r$ .
The proof of Lemma 3.6 follows via a careful application of LP duality; see Appendix B.
Proof of Proposition 3.3. Let  $\Theta = \min_{1\leq i < j\leq r}\arccos \langle \mathbf{v}_i,\mathbf{v}_j\rangle$ . Applying Fact 3.4 by taking  $\gamma = \left(\frac{1}{50\kappa_{m - 1}}\right)^{\frac{1}{m - 1}}$  yields  $\mathbf{Pr}[\Theta \geq \Delta ]\geq \mathbf{Pr}\left[\Theta \geq \gamma r^{-\frac{2}{m - 1}}\right]\geq 99 / 100$  (since our choice of  $m$  will imply  $r = 1 / \sqrt{50\kappa_{m - 1}\Delta^{m - 1}}$ ). This will imply that
$$
\mathbf {P r} \left[ \min  _ {1 \leq i <   j \leq r} \left\{\| \mathbf {v} _ {i} - \mathbf {v} _ {j} \| _ {2}, \| \mathbf {v} _ {i} + \mathbf {v} _ {j} \| _ {2} \right\} \geq \Omega (\Delta) \right] \geq 9 9 / 1 0 0.
$$
We then show that there exist non-negative weights  $w_{1},\ldots ,w_{r}$  with  $\sum_{\ell = 1}^{r}w_{\ell} = 1$  such that  $\mathbf{E}_{\mathbf{z}}\sim \mathcal{N}_m[g(\mathbf{z})p(\mathbf{z})] = 0$  holds for every odd polynomial  $p$  with degree less than  $k$ , where  $g(\mathbf{z}) = \sum_{\ell = 1}^{r}w_{\ell}\mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{z})$ . By Lemma 3.5, it suffices to show that there exist non-negative weights  $w_{1},\ldots ,w_{r}$  with  $\sum_{\ell = 1}^{r}w_{\ell} = 1$  such that  $\sum_{\ell = 1}^{r}w_{\ell}q(\mathbf{v}_{\ell}) = 0$  for all odd polynomials  $q$  of degree less than  $k$ . By Fact 2.4, we have that  $r\geq C\left({}^{\underline{m + k}}_k\right) = C\mathrm{VC}(\mathcal{C}_k)$  for some sufficiently large constant  $C > 0$ . Note that for any odd polynomial  $q$  in  $m$  variables of degree less than  $k$ , we have that  $\mathbf{E}[\mathrm{sign}(q(\mathbf{v}))] = 0$ . Therefore, by the VC-inequality (Lemma 2.3), we have that
$\mathbf{Pr}[\exists$  odd polynomial  $q$  of degree less than  $k$  such that  $q(\mathbf{v}_{\ell}) > 0,1\leq \ell \leq r]$
$$
\begin{array}{l} \leq \mathbf {P r} \left[ \sup  _ {f \in \mathcal {C} _ {k - 1}} \frac {\sum_ {\ell = 1} ^ {r} f (\mathbf {v} _ {\ell})}{r} = 1 \right] \leq \mathbf {P r} \left[ \sup  _ {f \in \mathcal {C} _ {k}} \left| \frac {\sum_ {\ell = 1} ^ {r} f (\mathbf {v} _ {\ell})}{r} - \mathbf {E} [ f (\mathbf {v}) ] \right| \geq 1 \right] \\ \leq \exp (- \Omega (r)) \leq 1 / 1 0 0. \\ \end{array}
$$
Finally, applying Lemma 3.6 completes our proof.
3.2 Proof of SQ Lower Bound for General Weights
Let  $k$  be a positive integer such that  $r \geq C\binom{m+k}{k}$  for some constant  $C > 0$  sufficiently large. Let  $m = \frac{c \log r}{\log(1 / \Delta)}$ , where  $r^{-1 / 10} \leq \Delta < 1$  and  $1.99 \leq c \leq 2$ . By Proposition 3.3, there exist unit vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r \in \mathbb{R}^m$  such that
- There exist non-negative weights  $w_{1},\ldots ,w_{r}$  with  $\sum_{\ell = 1}^{r}w_{\ell} = 1$  such that  $\mathbf{E}_{\mathbf{z}\sim \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = 0$  holds for every odd polynomial  $p$  with degree less than  $k$ , where  $g(\mathbf{z}) = \sum_{\ell = 1}^{r}w_{\ell}\mathrm{sign}(\mathbf{v}_{\ell}^{\intercal}\mathbf{z})$ .
-  $\| \mathbf{v}_i + \mathbf{v}_j\| _2,\| \mathbf{v}_i - \mathbf{v}_j\| _2\geq \Omega (\Delta),\forall 1\leq i < j\leq r$  for some  $\Delta >0$
For an arbitrary matrix  $\mathbf{U} \in \mathbb{R}^{m \times n}$  with  $\mathbf{U}\mathbf{U}^{\intercal} = \mathbf{I}_m$ , denote by  $D_{\mathbf{U}}$  the instance of mixture of linear classifiers with weight vectors  $\mathbf{U}^{\intercal}\mathbf{v}_1, \ldots, \mathbf{U}^{\intercal}\mathbf{v}_r$  and weights  $w_1, \ldots, w_r$ . Let  $\mathcal{D}_g = \{\mathcal{D}_{\mathbf{U}} \mid \mathbf{U} \in \mathbb{R}^{m \times n}, \mathbf{U}\mathbf{U}^{\intercal} = \mathbf{I}_m\}$ . We have that  $\|\mathbf{U}^{\intercal}\mathbf{v}_i \pm \mathbf{U}^{\intercal}\mathbf{v}_j\|_2 = \|\mathbf{v}_i \pm \mathbf{v}_j\|_2 \geq \Omega(\Delta), 1 \leq i < j \leq r$ , and  $\mathbf{E}_{(\mathbf{x}, y) \sim D_{\mathbf{U}}} [y \mid \mathbf{x} = \mathbf{z}] = \sum_{\ell=1}^{r} w_\ell \mathrm{sign}((\mathbf{U}^{\intercal}\mathbf{v}_\ell)^{\intercal}\mathbf{z}) = \sum_{\ell=1}^{r} w_\ell \mathrm{sign}(\mathbf{v}_\ell^{\intercal}\mathbf{U}\mathbf{z}) = g(\mathbf{U}\mathbf{z}), \mathbf{z} \in \mathbb{R}^n$ . By Proposition 3.2, any SQ algorithm for the decision problem  $\mathcal{B}(\mathcal{D}_g, \mathcal{N}_n \times \mathcal{U}(\{\pm 1\}))$  must either use queries of tolerance  $n^{-\Omega(k)}$ , or make at least  $2^{n^{\Omega(1)}}$  queries. The last step is to reduce the decision problem  $\mathcal{B}(\mathcal{D}_g, \mathcal{N}_n \times \mathcal{U}(\{\pm 1\}))$  to the problem of learning mixture of linear classifiers.
Claim 3.7 (see, e.g., Lemma 8.5 in [DK23]). Suppose there exists an  $SQ$  algorithm to learn an unknown distribution in a family  $\mathcal{D}$  to total variation distance  $\epsilon$  using at most  $N$  statistical queries
of tolerance  $\tau$ . Suppose furthermore that for each  $D' \in \mathcal{D}$  we have that  $d_{\mathrm{TV}}(D, D') > 2(\tau + \epsilon)$ . Then there exists an SQ algorithm that solves the testing problem  $\mathcal{B}(\mathcal{D}, D)$  using at most  $n + 1$  queries of tolerance  $\tau$ .
To apply Claim 3.7, we need to show that the distribution  $D_{\mathbf{U}}$  in the class  $\mathcal{D}_g$  is sufficiently far from the null hypothesis  $D_0 = \mathcal{N}_n\times \mathcal{U}(\{\pm 1\})$  in total variation distance.
Lemma 3.8. Let  $\mathbf{U} \in \mathbb{R}^{m \times n}$  with  $\mathbf{UU}^{\intercal} = \mathbf{I}_m$ . We have that  $d_{\mathrm{TV}}(D_{\mathbf{U}}, D_0) \geq \Omega(\Delta / r)$ .
We briefly sketch the proof idea and defer the proof details to Appendix B. We consider  $w_{\ell} \mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{x})$  a halfspace of heaviest weight in  $D_{\mathbf{U}}$ , and pick points  $\mathbf{z}$  and  $\mathbf{z}'$  close to the defining hyperplane that are mirrors of each other over it. We note that under  $D_{0}$  the expectations of  $y$  conditioned on  $\mathbf{x}$  being  $\mathbf{z}$  or  $\mathbf{z}'$  are both 0, whereas under  $D_{\mathbf{U}}$  they likely (in particular, unless they are also on opposite sides of another halfspace) differ by at least  $w_{\ell}$ .
4 SQ Lower Bound for Uniform Mixtures via Spherical Designs
In this section, we prove our SQ lower bound for mixture of linear classifiers with uniform weights, thereby establishing Theorem 1.2. Our basic lower bound technique is essentially the same as in the previous section, but we need to construct a spherical design with uniform weight.
Proposition 4.1. Let  $m = \frac{c\log r}{\log(1 / \Delta)}$ , where  $r^{-1 / 10} \leq \Delta < 1$  and  $1.99 \leq c \leq 2$  is a constant. Let  $k$  be an odd integer such that  $r \geq \binom{m+2k-1}{2k}^5$ . There exist vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r$  on  $\mathbb{S}^{m-1}$  such that:
-  $\mathbf{E}_{\mathbf{z} \sim \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = 0$  holds for every odd polynomial  $p$  with degree less than  $k$ , where  $g(\mathbf{z}) = \frac{1}{r}\sum_{\ell=1}^{r}\mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{z})$ .
$\| \mathbf{v}_i + \mathbf{v}_j\| _2,\| \mathbf{v}_i - \mathbf{v}_j\| _2\geq \Omega (\Delta) - O\big(1 / \binom {m + 2k - 1}{2k}\big),\forall 1\leq i <   j\leq r.$
By Lemma 3.5, we can write  $\sum_{\ell = 1}^{r}\mathbf{E}_{\mathbf{z}\sim \mathcal{N}_m}[p(\mathbf{z})\mathrm{sign}(\mathbf{v}_\ell^\top \mathbf{z})] = \sum_{\ell = 1}^{r}q(\mathbf{v}_\ell)$  for some odd polynomial  $q$  of degree less than  $k$ . Therefore, it suffices to show that with high probability there exist unit vectors  $\mathbf{v}_1,\ldots ,\mathbf{v}_r$  such that  $\sum_{\ell = 1}^{r}q(\mathbf{v}_{\ell}) = 0$  holds for every odd polynomial  $q$  of degree less than  $k$ . To achieve this, we will leverage some techniques from [BRV13].
4.1 Spherical Design Construction
Notation. We start by introducing some additional notations we will use throughout this section. Let  $\mathcal{P}_t^d$  denote the set of homogeneous polynomials in  $d$  variables of some odd degree  $t$ . For any  $p, q \in \mathcal{P}_t^d$ , we consider the inner product  $\langle p, q \rangle = \mathbf{E}_{\mathbf{x} \sim \mathcal{U}(\mathbb{S}^{d-1})}[p(\mathbf{x})q(\mathbf{x})]$ . For any  $p \in \mathcal{P}_t^d$ , we define  $\|p\|_2^2 = \langle p, p \rangle$ . We denote by  $N_{t,d} = \binom{t+d-1}{d-1}$  to be the dimension of  $\mathcal{P}_t^d$  and  $\Omega_t^d = \{p \in \mathcal{P}_t^d \mid \|p\|_2 \leq 1\}$ . Let  $\partial \Omega_t^d$  denote the boundary of  $\Omega_t^d$ , i.e.,  $\partial \Omega_t^d = \{p \in \mathcal{P}_t^d \mid \|p\|_2 = 1\}$ . In the remaining part of this section, we will assume that the underlying distribution (over the expectation) is  $\mathcal{U}(\mathbb{S}^{d-1})$ .
The following theorem — a more detailed version of Theorem 1.6 — establishes the existence of a spherical  $t$ -design of size  $\mathrm{poly}(N_{2t,d})$  with the desired pairwise separation properties.
Theorem 4.2 (Spherical Design Construction). Let  $t$  be an odd integer and  $r \geq N_{2t,d}^5$ . Let  $\mathbf{y}_1, \ldots, \mathbf{y}_r$  be uniform random vectors over  $\mathbb{S}^{d-1}$ . Then, with probability at least 99/100, there exist unit vectors  $\mathbf{z}_1, \ldots, \mathbf{z}_r \in \mathbb{S}^{d-1}$  such that  $\| \mathbf{z}_i - \mathbf{y}_i \|_2 \leq O(1/N_{2t,d})$ ,  $i \in [r]$ , and  $(\mathbf{z}_1, \ldots, \mathbf{z}_r)$  form a spherical  $t$ -design.
To prove Theorem 4.2, we will start from the following result.
Theorem 4.3 ([BRV13]). If there exists a continuous mapping  $F: \mathcal{P}_t^d \to (\mathbb{S}^{d-1})^r$  such that for all  $p \in \partial \Omega_t^d$ ,  $\sum_{i=1}^r p(\mathbf{x}_i(p)) > 0$ , where  $F(p) = (\mathbf{x}_1(p), \ldots, \mathbf{x}_r(p))$ , then there exists a polynomial  $p^* \in \Omega_t^d$  such that  $\mathbf{E}[p(\mathbf{x})] = \frac{1}{r} \sum_{i=1}^r p(\mathbf{x}_i(p^*))$  holds for every polynomial  $p \in \mathcal{P}_t^d$ .
To apply Theorem 4.3, we need to find a continuous function  $F$  mapping  $\mathcal{P}_t^d$  to  $(\mathbb{S}^{d - 1})^r$  such that for any  $p\in \partial \Omega_t^d$ ,  $\sum_{i = 1}^{r}p(\mathbf{z}_i) > 0$ , where  $F(p) = (\mathbf{z}_1,\dots ,\mathbf{z}_r)$ . We will construct the mapping
$F$  as follows: we sample  $\mathbf{y}_1, \ldots, \mathbf{y}_r$  uniformly over the unit sphere  $\mathbb{S}^{d-1}$ , and then try to make the value of  $p(\mathbf{y}_i)$  larger by moving each point  $\mathbf{y}_i$  in the direction of the gradient. In particular, we let  $\mathbf{z}_i = \frac{\mathbf{y}_i + \delta \nabla_o p(\mathbf{y}_i)}{\|\mathbf{y}_i + \delta \nabla_o p(\mathbf{y}_i)\|_2}$  for some  $\delta > 0$  sufficiently small, where  $\nabla_o p(\mathbf{y})$  is the component of  $\nabla p(\mathbf{y})$  orthogonal to the direction  $\mathbf{y}$ . We will prove that with high probability, for any  $p \in \partial \Omega_t^d$ ,  $\sum_{i=1}^{r} p(\mathbf{z}_i) > 0$ . Intuitively, this works because of two facts:
1. With high probability over the choice of  $\mathbf{y}_i$ , for all  $p \in \Omega_t^d$ , the average value of  $p(\mathbf{y})$  is already close to zero.
2. Moving in the direction of  $\nabla_{o}p(\mathbf{y}_{i})$  increases  $p(\mathbf{y}_i)$  by a notable amount.
Lemma 4.4. Let  $\Omega$  be a subspace of polynomials in  $d$  variables with mean zero. Let  $N$  be the dimension of  $\Omega$ . Let  $\mathbf{x}_1, \ldots, \mathbf{x}_r$  be i.i.d. random vectors over  $\mathbb{S}^{d-1}$ . Then, with probability at least  $1 - \frac{N}{r\eta^2}$ , we have that for any  $p \in \Omega$ ,  $\left|\frac{1}{r}\sum_{i=1}^{r}p(\mathbf{x}_i)\right| \leq \eta \| p\|_2$ .
To prove Lemma 4.4, we take an orthonormal basis  $p_1,\ldots ,p_N\in \Omega$  and define  $\mathbf{p}(\mathbf{x})\stackrel {\mathrm{def}}{=}[p_1(\mathbf{x}),\dots ,p_N(\mathbf{x})]^{\intercal}$ . Noting that  $\mathbf{p}(\mathbf{x})$  is a random vector with mean zero and covariance identity, applying Markov's inequality will yield the result. We defer the proof details to Appendix C.
Lemma 4.5. Let  $\mathbf{y} \in \mathbb{S}^{d-1}$  and  $0 < \delta \leq 1/N_{2t,d}^2$ . For any  $p \in \partial \Omega_t^d$ , let  $\mathbf{z} = \frac{\mathbf{y} + \delta\nabla_o p(\mathbf{y})}{\|\mathbf{y} + \delta\nabla_o p(\mathbf{y})\|_2}$ . We have that  $p(\mathbf{z}) - p(\mathbf{y}) \geq C\delta\|\nabla_o p(\mathbf{y})\|_2^2$  for some universal constant  $0 < C < 1$ .
The proof of Lemma 4.5 follows by Taylor expansion, where the contributions to  $p(\mathbf{z})$  coming from higher-order terms will not be large as long as  $\mathbf{z}$  is sufficiently close to  $\mathbf{y}$ . We defer the proof details to Appendix C. By applying the above two lemmas, we establish the following:
Theorem 4.6. Let  $\mathbf{y}_1, \ldots, \mathbf{y}_r$  be i.i.d. random vectors over  $\mathbb{S}^{d-1}$ . Let  $\delta = 1/N_{2t,d}^2$  and  $r \geq N_{2t,d}^5$ . We consider the mapping  $F: \mathcal{P}_t^d \to (\mathbb{S}^{d-1})^r$  as follows: for any  $p \in \mathcal{P}_t^d$ , let
$$
\mathbf {z} _ {i} = \frac {\mathbf {y} _ {i} + \delta \nabla_ {o} p (\mathbf {y} _ {i})}{\| \mathbf {y} _ {i} + \delta \nabla_ {o} p (\mathbf {y} _ {i}) \| _ {2}},
$$
where  $\nabla_{o}p(\mathbf{y})$  is the component of  $\nabla p(\mathbf{y})$  orthogonal to the direction  $\mathbf{y}$ . Let  $F(p) \coloneqq (\mathbf{z}_1, \dots, \mathbf{z}_r)$ . Then, with probability at least 99/100, we have that for any  $p \in \partial \Omega_t^d$ ,  $\sum_{i=1}^{r} p(\mathbf{z}_i) > 0$ .
To prove Theorem 4.6, we consider  $\widetilde{p} (\mathbf{y}) = p(\mathbf{y}) + C\delta (\| \nabla_{o}p(\mathbf{y})\| _2^2 -\mathbf{E}[\| \nabla_{o}p(\mathbf{y})\| _2^2])$  , where  $C,\delta$  comes from Lemma 4.5. Noting that  $\mathbf{E}[\widetilde{p} (\mathbf{y})] = 0$  and  $\widetilde{p} (\mathbf{y})$  is a polynomial containing only monomials of degree  $2t,2t - 2,t,0$  , we are able to apply Lemma 4.4 to obtain the desired result. See Appendix C for the proof.
We now prove our main technical result, Theorem 4.2.
Proof of Theorem 4.2. Let  $\delta = 1 / N_{2t,d}^2$ . We consider the mapping  $F:\mathcal{P}_t^d\to (\mathbb{S}^{d - 1})^r$  as follows: for any  $p\in \mathcal{P}_t^d$ , let  $\mathbf{z}_i = \frac{\mathbf{y}_i + \delta\nabla_o p(\mathbf{y}_i)}{\|\mathbf{y}_i + \delta\nabla_o p(\mathbf{y}_i)\|_2}$ , where  $\nabla_{o}p(\mathbf{y})$  is the component of  $\nabla p(\mathbf{y})$  orthogonal to the direction  $\mathbf{y}$ . By Theorem 4.6, with probability at least 99/100, we have that for any  $p\in \partial \Omega_t^d$ ,  $\sum_{i = 1}^{r}p(\mathbf{z}_i) > 0$ . Applying Theorem 4.3 yields that there exists some  $p^*\in \Omega_t^d$  such that  $F(p^{*}) = (\mathbf{z}_{1}^{*},\dots,\mathbf{z}_{r}^{*})$  form a spherical  $t$ -design. Furthermore, we can show that  $\| \mathbf{z}_i^* -\mathbf{y}_i\| _2\leq O(1 / N_{2t,d})$  by elementary calculation (see Appendix C).
4.2 Proof of Theorem 1.2
We first prove Proposition 4.1 based on our construction of spherical  $t$ -design. To achieve this, it suffices to show that with high probability there exist  $\mathbf{v}_1,\ldots ,\mathbf{v}_r\in \mathbb{S}^{m - 1}$  such that
-  $\mathbf{v}_1, \ldots, \mathbf{v}_r$  is a spherical  $k$ -design.
-  $\| \mathbf{v}_i + \mathbf{v}_j\| _2,\| \mathbf{v}_i - \mathbf{v}_j\| _2\geq \Omega (\Delta),\forall 1\leq i < j\leq r$  for some  $\Delta >0$
Proof of Proposition 4.1. Let  $\Theta = \min_{1\leq i < j\leq r}\arccos \langle \mathbf{y}_i,\mathbf{y}_j\rangle$ . Applying Fact 3.4 by taking  $\gamma = \left(\frac{1}{50\kappa_{m - 1}}\right)^{\frac{1}{m - 1}}$  yields  $\mathbf{Pr}[\Theta \geq \Delta ]\geq \mathbf{Pr}\left[\Theta \geq \gamma r^{-\frac{2}{m - 1}}\right]\geq 99 / 100$ . This will give