SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Pr \left[ \min  _ {1 \leq i <   j \leq r} \left\{\| \mathbf {y} _ {i} - \mathbf {y} _ {j} \| _ {2}, \| \mathbf {y} _ {i} + \mathbf {y} _ {j} \| _ {2} \right\} \geq \Omega (\Delta) \right] \geq 9 9 / 1 0 0,
$$
since our choice of  $m$  will imply that  $r = 1 / \sqrt{50\kappa_{m - 1}\Delta^{m - 1}}$ . By Theorem 4.2, with probability at least 99/100, there exist unit vectors  $\mathbf{z}_1^*,\ldots ,\mathbf{z}_r^*\in \mathbb{S}^{m - 1}$  such that  $\| \mathbf{z}_i^* -\mathbf{y}_i\| _2\leq O(1 / N_{2k,m})$ ,  $i\in [r]$  and  $(\mathbf{z}_1^*,\dots ,\mathbf{z}_r^*)$  form a spherical  $k$ -design. Therefore, for any odd homogeneous polynomial  $p$  in  $m$  variables of odd degree  $t < k$ , we have that
$$
\frac {1}{r} \sum_ {i = 1} ^ {r} p (\mathbf {z} _ {i} ^ {*}) = \frac {1}{r} \sum_ {i = 1} ^ {r} \left(\| \mathbf {z} _ {i} ^ {*} \| _ {2} ^ {2}\right) ^ {\frac {k - t}{2}} p (\mathbf {z} _ {i} ^ {*}) = \mathbf {E} \left[ \left(\| \mathbf {z} \| _ {2} ^ {2}\right) ^ {\frac {k - t}{2}} p (\mathbf {z}) \right] = 0,
$$
which implies that  $\sum_{i=1}^{r} q(\mathbf{z}_i^*) = 0$  holds for any polynomial  $q$  in  $m$  variables of degree less than  $k$ . In addition, for every  $1 \leq i < j \leq r$ , we have that
$$
\begin{array}{l} \left\| \mathbf {z} _ {i} ^ {*} \pm \mathbf {z} _ {j} ^ {*} \right\| _ {2} = \| (\mathbf {y} _ {i} \pm \mathbf {y} _ {j}) + (\mathbf {z} _ {i} ^ {*} - \mathbf {y} _ {i}) \pm (\mathbf {z} _ {j} ^ {*} - \mathbf {y} _ {j}) \| _ {2} \geq \| \mathbf {y} _ {i} \pm \mathbf {y} _ {j} \| - \| \mathbf {z} _ {i} ^ {*} - \mathbf {y} _ {i} \| _ {2} - \| \mathbf {z} _ {j} ^ {*} - \mathbf {y} _ {j} \| _ {2} \\ \geq \left\| \mathbf {y} _ {i} \pm \mathbf {y} _ {j} \right\| _ {2} - O \left(1 / N _ {2 k, m}\right). \\ \end{array}
$$
Proof of Theorem 1.2. Let  $m = \frac{c\log r}{\log(1 / \Delta)}$ , where  $r^{-1 / 10} \leq \Delta < 1$  and  $1.99 \leq c \leq 2$  is a constant. Let  $c' = \frac{(1 / e)(1 / \Delta)^{1 / (5c)} - 1}{2}$  and  $k = c'm = \frac{\left((1 / e)(1 / \Delta)^{1 / (5c)} - 1\right)c\log r}{2\log(1 / \Delta)}$ . In this way, we will have that  $r \geq N_{2k,m}^5$  (see Appendix C for calculation details). Therefore, by Proposition 4.1, there exist vectors  $\mathbf{v}_1, \ldots, \mathbf{v}_r \in \mathbb{S}^{m-1}$  such that
-  $\mathbf{E}_{\mathbf{z} \sim \mathcal{N}_m}[g(\mathbf{z})p(\mathbf{z})] = 0$  holds for every odd polynomial  $p$  of degree less than  $k$ , where  $g(\mathbf{z}) = \frac{1}{r}\sum_{\ell=1}^{r}\mathrm{sign}(\mathbf{v}_{\ell}^{\mathsf{T}}\mathbf{z})$ .
-  $\| \mathbf{v}_i + \mathbf{v}_j\| _2,\| \mathbf{v}_i - \mathbf{v}_j\| _2\geq \Omega (\Delta) - O(1 / N_{2k,m}),\forall 1\leq i < j\leq r.$
For an arbitrary matrix  $\mathbf{U} \in \mathbb{R}^{m \times n}$  with  $\mathbf{U}\mathbf{U}^{\intercal} = \mathbf{I}_m$ , denote by  $D_{\mathbf{U}}$  the instance of mixture of linear classifiers with weight vectors  $\mathbf{U}^{\intercal}\mathbf{v}_1, \ldots, \mathbf{U}^{\intercal}\mathbf{v}_r$ . Let  $\mathcal{D}_g = \{\mathcal{D}_{\mathbf{U}} \mid \mathbf{U} \in \mathbb{R}^{m \times n}, \mathbf{U}\mathbf{U}^{\intercal} = \mathbf{I}_m\}$ . By definition, we have that  $\mathbf{E}_{(\mathbf{x}, y) \sim D_{\mathbf{U}}}[y \mid \mathbf{x} = \mathbf{z}] = \frac{1}{r} \sum_{\ell=1}^{r} \mathrm{sign}((\mathbf{U}^{\intercal}\mathbf{v}_{\ell})^{\intercal}\mathbf{z}) = \frac{1}{r} \sum_{\ell=1}^{r} \mathrm{sign}(\mathbf{v}_{\ell}^{\intercal}\mathbf{U}\mathbf{z}) = g(\mathbf{U}\mathbf{z}), \mathbf{z} \in \mathbb{R}^n$ , and  $\|\mathbf{U}^{\intercal}\mathbf{v}_i \pm \mathbf{U}^{\intercal}\mathbf{v}_j\|_2 = \| \mathbf{v}_i \pm \mathbf{v}_j\|_2 \geq \Omega(\Delta)$ ,  $1 \leq i < j \leq r$ , since  $N_{2k,m} \geq \Omega((1/\Delta)^{1.89})$  (see Appendix C for the calculation details). Therefore by Proposition 3.2, any SQ algorithm that solves the decision problem  $\mathcal{B}(\mathcal{D}_g, \mathcal{N}_n \times \mathcal{U}(\{\pm 1\}))$  must either use queries of tolerance  $n^{-\Omega(k)}$ , or make at least  $2^{n^{\Omega(1)}}$  queries. By Lemma 3.8, for any  $\mathbf{U} \in \mathbb{R}^{m \times n}$  with  $\mathbf{U}\mathbf{U}^{\intercal} = \mathbf{I}_m$ , we have that  $d_{\mathrm{TV}}(D_{\mathbf{U}}, D_0) \geq \Omega(\Delta/r) \geq 2(n^{-\Omega(k)} + \epsilon)$ , where  $D_0 = \mathcal{N}_n \times \mathcal{U}(\{\pm 1\})$ . Therefore, by Claim 3.7, any SQ algorithm that learns a distribution in  $\mathcal{D}_g$  within error  $\epsilon$  in total variation distance must either use queries of tolerance  $n^{-\Omega(\log r / (\Delta^{1/(5c)}\log(1/\Delta)))}$ , or make at least  $2^{n^{\Omega(1)}}$  queries. This completes the proof.
This work establishes a near-optimal Statistical Query (SQ) lower bound for learning uniform mixtures of linear classifiers under the Gaussian distribution. Our lower bound nearly matches prior algorithmic work on the problem [CDV22]. Our result applies for the simplest (and well-studied) distributional setting where the covariates are drawn from the standard Gaussian distribution. This directly implies similar information-computation tradeoffs for the setting that the covariates are drawn from a more general distribution family (e.g., an unknown subgaussian or log-concave distribution) that includes the standard normal.
From a technical perspective, we believe that our new efficient construction of spherical designs is a mathematical contribution of independent interest that could be used to establish SQ lower bounds for other related latent variable models (e.g., various mixtures of experts). A natural direction is to establish information-computation tradeoffs for a fixed non-Gaussian distribution on covariates (e.g., the uniform distribution over the Boolean hypercube), for which a different hardness construction is needed.
[AK01] S. Arora and R. Kannan. Learning mixtures of arbitrary Gaussians. In Proceedings of the 33rd Symposium on Theory of Computing, pages 247-257, 2001.
[AM05] D. Achlioptas and F. McSherry. On spectral learning of mixtures of distributions. In Proceedings of the Eighteenth Annual Conference on Learning Theory (COLT), pages 458-469, 2005.
[Ban79] E. Bannai. On tight spherical designs. Journal of Combinatorial Theory, Series A, 26(1):38-47, 1979.
$\left[\mathrm{BBH}^{+}20\right]$  M. Brennan, G. Bresler, S. B. Hopkins, J. Li, and T. Schramm. Statistical query algorithms and low-degree tests are almost equivalent. arXiv preprint arXiv:2009.06107, 2020. In 34th Annual Conference on Learning Theory (COLT 2021).
[BRS+18] J. S. Brauchart, A. B. Reznikov, E. B. Saff, I. H. Sloan, Y. G. Wang, and R. S. Womersley. Random point sets on the sphere—hole radii, covering, and separation. Experimental Mathematics, 27(1):62–81, 2018.
[BRV13] A. Bondarenko, D. Radchenko, and M. Viazovska. Optimal asymptotic bounds for spherical designs. Annals of mathematics, pages 443-452, 2013.
[BS21] R. D. Buhai and D. Steurer. Beyond parallel pancakes: Quasi-polynomial time guarantees for non-spherical gaussian mixtures. CoRR, abs/2112.05445, 2021.
[BV08] S. C. Brubaker and S. Vempala. Isotropic PCA and Affine-Invariant Clustering. In Proc. 49th IEEE Symposium on Foundations of Computer Science, pages 551-560, 2008.
[CC06] Y. J. Cho and Y. Q. Chen. Topological degree theory and applications. CRC Press, 2006.
[CDV22] A. Chen, A. De, and A. Vijayaraghavan. Algorithms for learning a mixture of linear classifiers. In International Conference on Algorithmic Learning Theory, pages 205-226. PMLR, 2022.
[CLS20] S. Chen, J. Li, and Z. Song. Learning mixtures of linear regressions in subexponential time via fourier moments. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 587-600, 2020.
[Das99] S. Dasgupta. Learning mixtures of Gaussians. In Proceedings of the 40th Annual Symposium on Foundations of Computer Science, pages 634-644, 1999.
[DGS91] P. Delsarte, J. Goethals, and J. J. Seidel. Spherical codes and designs. In Geometry and Combinatorics, pages 68-93. Elsevier, 1991.
[DK20] I. Diakonikolas and D. M. Kane. Small covers for near-zero sets of polynomials and learning latent variable models. In Proceedings of the 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS 2020), 2020.
[DK23] I. Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics. Cambridge university press, 2023.
[DKKZ20] I. Diakonikolas, D. M. Kane, V. Kontonis, and N. Zarifis. Algorithms and SQ lower bounds for PAC learning one-hidden-layer relu networks. In Conference on Learning Theory, COLT 2020, volume 125 of Proceedings of Machine Learning Research, pages 1514-1539. PMLR, 2020.
[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Conference on Learning Theory, COLT 2021, volume 134 of Proceedings of Machine Learning Research, pages 1552–1584. PMLR, 2021.
[LLYH17] W. Li, P. Liang, X. Yuan, and J. Hu. Non-local information for a mixture of multiple linear classifiers. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 3741-3746. IEEE, 2017.
[ME14] S. Masoudnia and R. Ebrahimpour. Mixture of experts: a literature survey. The Artificial Intelligence Review, 42(2):275, 2014.
[MMV10] A. Moitra and G. Valiant. Settling the polynomial learnability of mixtures of Gaussians. In FOCS, pages 93-102, 2010.
[MMVKO19] A. Makkuva, P. Viswanath, S. Kannan, and S. Oh. Breaking the gridlock in mixture-of-experts: Consistent and efficient algorithms. In International Conference on Machine Learning, pages 4304-4313. PMLR, 2019.
[Pea94] K. Pearson. Contribution to the mathematical theory of evolution. Phil. Trans. Roy. Soc. A, 185:71-110, 1894.
[RV17] O. Regev and A. Vijayaraghavan. On learning mixtures of well-separated gaussians. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, pages 85-96, 2017.
[SIM14] Y. Sun, S. Ioannidis, and A. Montanari. Learning mixtures of linear classifiers. In International Conference on Machine Learning, pages 721-729. PMLR, 2014.
[SJA16] H. Sedghi, M. Janzamin, and A. Anandkumar. Provable tensor methods for learning mixtures of generalized linear models. In Artificial Intelligence and Statistics, pages 1223-1231. PMLR, 2016.
[Ver18] R. Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[VW02] S. Vempala and G. Wang. A spectral algorithm for learning mixtures of distributions. In Proceedings of the 43rd Annual Symposium on Foundations of Computer Science, pages 113-122, 2002.
[Wom18] R. S. Womersley. Efficient spherical designs with good geometric properties. Contemporary computational mathematics-A celebration of the 80th birthday of Ian Sloan, pages 1243-1285, 2018.
[XA09] L. Xu and S. Amari. Combining classifiers and learning mixture-of-experts. In Encyclopedia of artificial intelligence, pages 318-326. IGI Global, 2009.