SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We define the standard  $L^p$  norms with respect to the Gaussian measure, i.e.,  $\| g\|_{L^p} = (\mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[|g(\mathbf{x})|^p)^{1 / p}$ . We denote by  $L^2 (\mathcal{N})$  the vector space of all functions  $f:\mathbb{R}^d\to \mathbb{R}$  such that  $\mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[f^{2}(x)] < \infty$ . The usual inner product for this space is  $\mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[f(\mathbf{x})g(\mathbf{x})]$ . While, usually one considers the probabilist's or physicist's Hermite polynomials, in this work we define the normalized Hermite polynomial of degree  $i$  to be  $\mathrm{He}_0(x) = 1$ ,  $\mathrm{He}_1(x) = x$ ,  $\mathrm{He}_2(x) = \frac{x^2 - 1}{\sqrt{2}}$ , ...,  $\mathrm{He}_i(x) = \frac{\widehat{\mathrm{He}}_i(x)}{\sqrt{i!}}$ , ... where by  $\widehat{\mathrm{He}}_i(x)$  we denote the probabilist's Hermite polynomial of degree  $i$ . The unnormalized Hermite polynomials are defined as  $\widehat{\mathrm{He}}_i(z)\exp (-z^2 /2) = (-1)^i\frac{\mathrm{d}^i\exp(-z^2 / 2)}{\mathrm{d}z^i}$ . The normalized Hermite polynomials  $\mathrm{He}_1,\mathrm{He}_2,\ldots ,\mathrm{He}_i,\ldots$  form a complete orthonormal basis for the single dimensional version of the inner product space defined above. To get an orthonormal basis for  $L^2 (\mathcal{N})$ , we use a multi-index  $V\in \mathbb{N}^d$  to define the  $d$ -variate normalized Hermite polynomial as  $\mathrm{He}_V(\mathbf{x}) = \prod_{i = 1}^d\mathrm{He}_{v_i}(x_i)$ . The total degree of  $\mathrm{He}_V$  is  $|V| = \sum_{v_i\in V}v_i$ . Given a function  $f\in L^2$ , we compute its Hermite coefficients as  $\hat{f} (V) = \mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[f(\mathbf{x})\mathrm{He}_V(\mathbf{x})]$  and express it uniquely as  $\sum_{V\in \mathbb{N}^d}\hat{f} (V)\mathrm{He}_V(\mathbf{x})$ .
C.2 Additional Background on the SQ Model
To define the SQ dimension, we need the following definition.
Definition C.1 (Pairwise Correlation). The pairwise correlation of two distributions with probability density functions (pdfs)  $D_{1}, D_{2} : \mathcal{X} \to \mathbb{R}_{+}$  with respect to a distribution with pdf  $D : \mathcal{X} \to \mathbb{R}_{+}$ , where the support of  $D$  contains the supports of  $D_{1}$  and  $D_{2}$ , is defined as  $\chi_{D}(D_{1}, D_{2}) + 1 \coloneqq \int_{x \in \mathcal{X}} D_{1}(x) D_{2}(x) / D(x) \, \mathrm{d}x$ . We say that a collection of  $s$  distributions  $\mathfrak{D} = \{D_{1}, \ldots, D_{s}\}$  over  $\mathcal{X}$  is  $(\gamma, \beta)$ -correlated relative to a distribution  $D$  if  $|\chi_{D}(D_{i}, D_{j})| \leq \gamma$  for all  $i \neq j$ , and  $|\chi_{D}(D_{i}, D_{j})| \leq \beta$  for  $i = j$ .
The following notion of dimension effectively characterizes the difficulty of the decision problem.
Definition C.2 (SQ Dimension). For  $\gamma, \beta > 0$ , a decision problem  $\mathcal{B}(\mathfrak{D}, D)$ , where  $D$  is fixed and  $\mathfrak{D}$  is a family of distributions over  $\mathcal{X}$ , let  $s$  be the maximum integer such that there exists  $\mathfrak{D}_D \subseteq \mathfrak{D}$  such that  $\mathfrak{D}_D$  is  $(\gamma, \beta)$ -correlated relative to  $D$  and  $|\mathfrak{D}_D| \geq s$ . We define the Statistical Query dimension with pairwise correlations  $(\gamma, \beta)$  of  $\mathcal{B}$  to be  $s$  and denote it by  $\mathrm{SD}(\mathcal{B}, \gamma, \beta)$ .
The connection between SQ dimension and lower bounds is captured by the following lemma.
Lemma C.3 ([FGR⁺ 17]). Let  $\mathcal{B}(\mathfrak{D}, D)$  be a decision problem, where  $D$  is the reference distribution and  $\mathfrak{D}$  is a class of distributions over  $\mathcal{X}$ . For  $\gamma, \beta > 0$ , let  $s = \mathrm{SD}(\mathcal{B}, \gamma, \beta)$ . Any SQ algorithm that solves  $\mathcal{B}$  with probability at least  $2/3$  requires at least  $s \cdot \gamma/\beta$  queries to the VSTAT(1/ $\gamma$ ) oracle.
In order to construct a large set of nearly uncorrelated hypotheses, we need the following fact:
Fact C.4 (see, e.g., [DKS17]). Let  $d \in \mathbb{Z}_+$ . Let  $0 < c < 1/2$ . There exists a collection  $\mathcal{S}$  of  $2^{\Omega(d^c)}$  unit vectors in  $\mathbb{R}^d$ , such that any pair  $\mathbf{v}, \mathbf{u} \in \mathcal{S}$ , with  $\mathbf{v} \neq \mathbf{u}$ , satisfies  $|\mathbf{v} \cdot \mathbf{u}| < d^{-1/2 + c}$ .
C.3 Omitted Details from the Proof of Theorem 3.2
Claim C.5. Let  $f_{\mathbf{v}}(\mathbf{x}) = \mathrm{sign}(\mathbf{v}\cdot \mathbf{x} - t)$  and  $f_{\mathbf{u}}(\mathbf{x}) = \mathrm{sign}(\mathbf{u}\cdot \mathbf{x} - t)$  and let  $c_{i}$  be the Hermite coefficient of  $\mathrm{He}_i$ . Then, it holds  $\mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x})f_{\mathbf{u}}(\mathbf{x})] = \sum_{i = 0}^{\infty}\cos^i\theta c_i^2$ , where  $\theta$  is the angle between  $\mathbf{v}$  and  $\mathbf{u}$ .
Proof. We first need the following standard fact about the rotations of Hermite polynomials (see, e.g., Fact D.1 in [DKS17]):
Fact C.6. For  $\theta \in \mathbb{R}$ , it holds  $\mathrm{He}_i(x\cos (\theta) + y\sin \theta) = \sum_{j = 0}^i\binom {i}{j}\cos^j\theta \sin^{i - j}\theta \mathrm{He}_j(x)\mathrm{He}_{i - j}(y)$ .
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {N}} {\mathbf {E}} [ f _ {\mathbf {v}} (\mathbf {x}) f _ {\mathbf {u}} (\mathbf {x}) ] = \underset {\mathbf {x} _ {1}, \mathbf {x} _ {2} \sim \mathcal {N}} {\mathbf {E}} [ \operatorname {s i g n} (\mathbf {x} _ {1} - t) \operatorname {s i g n} (\cos \theta \mathbf {x} _ {1} + \sin \theta \mathbf {x} _ {2} - t) ] \\ = \mathbf {E} _ {\mathbf {x} _ {1}, \mathbf {x} _ {2} \sim \mathcal {N}} \left[ \left(\sum_ {i = 0} ^ {\infty} c _ {i} \mathrm {H e} _ {i} (\mathbf {x} _ {1})\right) \left(\sum_ {i = 0} ^ {\infty} c _ {i} \mathrm {H e} _ {i} (\cos \theta \mathbf {x} _ {1} + \sin \theta \mathbf {x} _ {2})\right) \right] \\ = \mathbf {E} _ {\mathbf {x} _ {1} \sim \mathcal {N}} \left[ \left(\sum_ {i = 0} ^ {\infty} c _ {i} \operatorname {H e} _ {i} (\mathbf {x} _ {1})\right) _ {\mathbf {x} _ {2} \sim \mathcal {N}} \mathbf {E} \left[ \left(\sum_ {i = 0} ^ {\infty} c _ {i} \operatorname {H e} _ {i} (\cos \theta \mathbf {x} _ {1} + \sin \theta \mathbf {x} _ {2})\right) \right] \right] \\ = \mathbf {E} _ {\mathbf {x} _ {1} \sim \mathcal {N}} \left[ \left(\sum_ {i = 0} ^ {\infty} c _ {i} \mathrm {H e} _ {i} (\mathbf {x} _ {1})\right) \left(\sum_ {i = 0} ^ {\infty} \cos^ {i} \theta c _ {i} \mathrm {H e} _ {i} (\mathbf {x} _ {1})\right) \right] = \sum_ {i = 0} ^ {\infty} \cos^ {i} \theta c _ {i} ^ {2}, \\ \end{array}
$$
where in the third equality, we used Fact C.6 and the orthogonality of the Hermite polynomials with respect to the Gaussian.
We prove the following.
Claim C.7. It holds that  $\mathbf{E}_{z\sim \mathcal{N}}[\mathrm{sign}(z - t)\mathrm{He}_i(z)] = 2(i)^{-1 / 2}\mathrm{He}_{i - 1}(t)\exp (-t^2 /2)$
Proof. Denote as  $\widehat{\mathrm{He}}_i(z)$  the non-normalized Hermite polynomial of order  $d$ . The Hermite polynomials are defined as follows:
$$
\widehat {\mathrm {H e}} _ {i} (z) \exp (- z ^ {2} / 2) = (- 1) ^ {i} \frac {\mathrm {d} ^ {i} \exp (- z ^ {2} / 2)}{\mathrm {d} z ^ {i}}.
$$
By taking the derivative over  $z$  (which exists as  $\mathrm{He}_i$  is a polynomial and  $\exp(-z^2/2)$  is differentiable), we have that  $\int (\widehat{\mathrm{He}}_i(z) \exp(-z^2/2)) = -\widehat{\mathrm{He}}_{i-1}(z)$ . Therefore, we have that
$$
\begin{array}{l} \mathbf {E} _ {z \sim \mathcal {N}} [ \operatorname {s i g n} (z - t) \widehat {\mathrm {H e}} _ {i} (z) ] = \int_ {z \in \mathbb {R}} \operatorname {s i g n} (z - t) \widehat {\mathrm {H e}} _ {i} (z) G (z) \mathrm {d} z \\ = 2 \int_ {t} ^ {\infty} \widehat {\mathrm {H e}} _ {i} (z) G (z) \mathrm {d} z, \\ \end{array}
$$
where we used that  $\int_{z\in \mathbb{R}}\widehat{\mathrm{He}}_i(z)G(z)\mathrm{d}z = 0$  by the orthogonality of the Hermite Polynomials with respect to the Gaussian measure. Furthermore, using that  $\int (\widehat{\mathrm{He}}_i(z)\exp (-z^2 /2)) = -\widehat{\mathrm{He}}_{i - 1}(z)$  we get that
$$
\begin{array}{l} \mathbf {E} _ {z \sim \mathcal {N}} [ \mathrm {s i g n} (z - t) \widehat {\mathrm {H e}} _ {i} (z) ] = 2 \int_ {t} ^ {\infty} (- \widehat {\mathrm {H e}} _ {i - 1} (z) G (z)) ^ {\prime} \mathrm {d} z \\ = 2 \widehat {\mathrm {H e}} _ {i - 1} (t) G (t). \\ \end{array}
$$
C.4 Proof of Lemma 3.6
We restate and prove the following lemma.
Lemma C.8. Let  $D_0$  be a product distribution over  $\mathcal{N} \times \{\pm 1\}$ , where  $\mathbf{Pr}_{(\mathbf{x},y) \sim D_0}[y = 1] = \mathbf{Pr}_{(\mathbf{x},y) \sim D_{\mathbf{v}}} [y = 1] = p$ . We have  $\chi_{D_0}(D_{\mathbf{v}}, D_{\mathbf{u}}) \leq 2(1 - 2\eta)(\mathbf{E}[f_{\mathbf{v}}(\mathbf{x}) f_{\mathbf{u}}(\mathbf{x})] - \mathbf{E}[f_{\mathbf{v}}(\mathbf{x})]\mathbf{E}[f_{\mathbf{u}}(\mathbf{x}))$  and  $\chi^2(D_{\mathbf{v}}, D_0) \leq (1 - 2\eta)(\mathbf{E}[f_{\mathbf{v}}(\mathbf{x})] - \mathbf{E}[f_{\mathbf{v}}(\mathbf{x})]^2)$ .
$$
\begin{array}{l} \chi_ {D _ {0}} \left(D _ {\mathbf {v}}, D _ {\mathbf {u}}\right) = \underset {(\mathbf {x}, y) \sim D _ {\mathbf {v}}} {\mathbf {P r}} [ y = 1 ] \chi_ {\mathcal {N}} \left(A _ {\mathbf {v}}, A _ {\mathbf {u}}\right) + \underset {(\mathbf {x}, y) \sim D _ {\mathbf {v}}} {\mathbf {P r}} [ y = 0 ] \chi_ {\mathcal {N}} \left(B _ {\mathbf {v}}, B _ {\mathbf {u}}\right) \\ = (1 / p) \chi_ {\mathcal {N}} \left(A _ {\mathbf {v}}, A _ {\mathbf {u}}\right) + (1 / (1 - p)) \chi_ {\mathcal {N}} \left(B _ {\mathbf {v}}, B _ {\mathbf {u}}\right). \\ \end{array}
$$
We bound each term. Note that by construction  $A_{\mathbf{v}}(\mathbf{x}) = G(\mathbf{x})(\eta + (1 - 2\eta)\mathbb{1}\{f_{\mathbf{v}}(\mathbf{x}) > 0\}) / (\eta + (1 - 2\eta)\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[\mathbb{1}\{f_{\mathbf{v}}(\mathbf{x}) > 0\}])$ . Note that  $\mathbb{1}\{f_{\mathbf{v}}(\mathbf{x}) > 0\} = (f(\mathbf{x}) + 1) / 2$ , therefore  $A_{\mathbf{v}}(\mathbf{x}) = G(\mathbf{x})(1 + (1 - 2\eta)f_{\mathbf{v}}(\mathbf{x})) / (1 + (1 - 2\eta)\mathbf{E}_{\mathbf{x} \sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x})])$ . Therefore,
$$
\frac {A _ {\mathbf {v}} (\mathbf {x})}{G (\mathbf {x})} - 1 = \frac {(1 - 2 \eta)}{p} \left(f _ {\mathbf {v}} (\mathbf {x}) - \underset {\mathbf {x} \sim \mathcal {N}} {\mathbf {E}} [ f _ {\mathbf {v}} (\mathbf {x}) ]\right) .
$$
Using the above, we get that  $\chi_{\mathcal{N}}(A_{\mathbf{v}},A_{\mathbf{u}}) = (1 - 2\eta) / p(\mathbf{E}[f_{\mathbf{v}}(\mathbf{x})f_{\mathbf{u}}(\mathbf{x})] - \mathbf{E}[f_{\mathbf{v}}(\mathbf{x})]\mathbf{E}[f_{\mathbf{u}}(\mathbf{x})])$  Similarly, we get that  $\chi_{\mathcal{N}}(B_{\mathbf{v}},B_{\mathbf{u}}) = (1 - 2\eta) / (1 - p)(\mathbf{E}[f_{\mathbf{v}}(\mathbf{x})f_{\mathbf{u}}(\mathbf{x})] - \mathbf{E}[f_{\mathbf{v}}(\mathbf{x})]\mathbf{E}[f_{\mathbf{u}}(\mathbf{x})])$  It remains to bound  $\chi^2 (D_{\mathbf{v}},D_0)$ . Note that  $\chi^2 (D_{\mathbf{v}},D_0) = \chi_{D_0}(D_{\mathbf{v}},D_{\mathbf{v}})$ , hence,  $\chi^2 (D_{\mathbf{v}},D_0)\leq (1 - 2\eta)\mathbf{E}_{\mathbf{x}\sim \mathcal{N}}([f_{\mathbf{v}}(\mathbf{x})] - \mathbf{E}_{\mathbf{x}\sim \mathcal{N}}[f_{\mathbf{v}}(\mathbf{x})]^2)$ .
C.5 Reduction of Testing to Learning
Lemma C.9 (Reduction of Testing to Learning). Any algorithm that learns halfspaces with  $\eta = 1 / 3$  RCN noise can be used to solve the decision problem of Theorem 3.2.
Proof. Assume that there is an algorithm  $\mathcal{A}$  which given  $\epsilon > 0$  and distribution  $D$  with Gaussian  $\mathbf{x}$ -marginals and corrupted with  $\eta = 1/3$  random classification noise, outputs a hypothesis  $h$  with  $\mathbf{Pr}_{(\mathbf{x},y)\sim D}[h(\mathbf{x})\neq y]\leq \eta +\epsilon$ . We can use  $\mathcal{A}$  to solve the decision problem  $\mathcal{B}(D_0,\mathfrak{D})$ . Note that if the distribution were  $D_0$ , then any hypothesis would get error at least  $\eta +(1 - 2\eta)p$  (as  $y$  is independent of  $\mathbf{x}$ ). If the distribution were one in the set  $\mathfrak{D}$ , then the algorithm for  $\epsilon = (1 - 2\eta)p / 2$  would give a hypothesis such that  $\mathbf{Pr}_{(\mathbf{x},y)\sim D}[h(\mathbf{x})\neq y]\leq \eta +(1 - 2\eta)p / 2$ . So making one additional query of tolerance  $(1 - 2\eta)p$  would be able to solve the decision problem. This completes the proof.
C.6 Solving the Decision Problem Efficiently
In this section, we show that our SQ lower bound (Theorem 3.2) for the testing problem is, in fact, tight. We prove the following:
Theorem C.10 (Efficient Algorithm for Testing). Let  $d \in \mathbb{N}$  and  $\epsilon \in (0, 1)$  and let  $\mathcal{D}$  be a distribution supported on  $\mathbb{R}^d \times \{\pm 1\}$  such that  $\mathcal{D}_{\mathbf{x}}$  is the standard Gaussian on  $\mathbb{R}^d$ . There exists an algorithm that, given  $N = C\sqrt{d} / (\epsilon^2\log(1/\epsilon))$  samples from  $\mathcal{D}$ , where  $C > 0$  is a sufficiently large absolute constant, distinguishes between the following cases with probability of error at most  $1/3$ :
1.  $\mathbf{x}$  is independent of  $y$ , where  $(\mathbf{x}, y) \sim \mathcal{D}$ .
2.  $y$  is  $f(\mathbf{x})$  corrupted with RCN for  $\eta = 1 / 3$ , where  $f$  is an LTF with  $\mathbf{Pr}[f(\mathbf{x}) = 1] = \epsilon$ .
Proof. Let  $\mathbf{Z} = y\mathbf{x}$  be the random variable where  $(\mathbf{x},y)\sim \mathcal{D}$  and let  $\mathbf{Z}_N = (1 / N)\sum_{i = 1}^N\mathbf{x}^{(i)}y^{(i)}$  be the random variables where  $(\mathbf{x}^{(i)},y^{(i)})$ , for  $i\in [N]$  are samples drawn from  $\mathcal{D}$ . The tester works as follows: If  $\| \mathbf{Z}_N\| _2^2 >d / N + c\epsilon^2\log (1 / \epsilon)$ , where  $c > 0$  is a sufficiently small universal constant, we answer that we are in Case 2, otherwise that we are in the Case 1. We prove the correctness of the algorithm below.
By normalizing the Hermite polynomial, we complete the proof of Claim C.7.
We note that if we are in the Case 1, then  $Z$  follows the standard Gaussian distribution. To see that, we show the following simple claim:
Claim C.11. Let  $\mathbf{x}$  be distributed as standard normal and let  $y$  supported in  $\{\pm 1\}$  be a random variable independent of  $\mathbf{x}$ , then  $y\mathbf{x}$  is distributed as standard normal.
Proof. We assume that  $y = 1$  with probability  $1 - \eta$  for some  $\eta \in (0,1)$ . Let  $\phi_A(t) = \mathbf{E}[\exp (itA)]$  be the characteristic function of  $A$ . Then, we have that  $\phi_{y\mathbf{x}}(t) = \mathbf{E}[\exp (ity\mathbf{x})] = (1 - \eta)\mathbf{E}[\exp (it\mathbf{x})] + \eta \mathbf{E}[\exp (-it\mathbf{x})] = (1 - \eta)\phi_{\mathbf{x}}(t) + \eta \phi_{-\mathbf{x}}(t) = \phi_{\mathbf{x}}(t)$ , where in the last equality we used that  $\phi_{-\mathbf{x}}(t) = \phi_{\mathbf{x}}(t)$  as the standard normal distribution is symmetric. Therefore,  $\phi_{y\mathbf{x}}(t) = \phi_{\mathbf{x}}(t)$ , hence the distribution of  $y\mathbf{x}$  and  $\mathbf{x}$  is the same.
From Claim C.11, we have that  $\mathbf{Z}$  follows standard normal distribution if we are in Case 1 and therefore  $\mathbf{Z}_N$  follows  $\mathcal{N}(\mathbf{0},\mathbf{I} / N)$ . Hence,  $\| \mathbf{Z}_N\| _2^2$  has mean  $d / N$  and standard deviation  $O(\sqrt{d} /N)$ . By Chebyshev's inequality, we answer correctly in this case with a probability of at least  $2 / 3$ .
We next analyze the case where  $Z$  is in Case 2. Let  $f(\mathbf{x}) = \mathrm{sign}(\mathbf{v} \cdot \mathbf{x} + t)$  be the defining halfspace. Then for any  $\mathbf{u}$  orthogonal to  $\mathbf{v}$ , we have that the random variables  $y$  and  $(\mathbf{x} \cdot \mathbf{u})$  are independent as  $y$  only depends on  $\mathbf{x} \cdot \mathbf{v}$ . Therefore, by Claim C.11, we have that  $y(\mathbf{x} \cdot \mathbf{u})$  is standard normal, therefore  $y\mathbf{x}^{\perp \mathbf{v}}$  is standard  $(d - 1)$ -dimensional normal. Furthermore, note that from Equation (1) that  $\mathbf{E}[y\mathbf{x} \cdot \mathbf{v}] = \Theta(\epsilon \sqrt{\log(1 / \epsilon)})$  and  $\mathrm{Var}[y\mathbf{x} \cdot \mathbf{v}] = \Theta(1)$ . Therefore, we can write  $\| \mathbf{Z}_N \|_2^2 = (1/N)(\sum_{i=1}^{d-1} g_i^2 + \mathbf{E}^2)$ , where  $g_i$  are distributed as standard normal and  $\mathbf{E}^2$  is the contribution due to the noise. We show that with probability at least  $2/3$ , it holds that  $\| \mathbf{Z}_N \|_2^2 \geq d/N + c\epsilon^2 \log(1/\epsilon)$ . Note that with probability at least  $9/10$ , we have that  $|\mathbf{E}| = \Theta(\epsilon \sqrt{\log(1 / \epsilon)})$  by Chebyshev's inequality. Due to the independence of the directions, the random variables  $g_i$  for  $i = 1, \ldots, d-1$  are independent of  $\mathbf{E}$ . Hence, conditioned on the event that  $|\mathbf{E}| = \Theta(\epsilon \sqrt{\log(1 / \epsilon)})$ , we have that  $\| \mathbf{Z}_n \|_2^2$  has mean  $(d-1)/N + \Theta(\epsilon^2 \log(1/\epsilon))$  and standard deviation  $O(\sqrt{d}/N)$ . Hence, again the tester would succeed with a probability of at least  $2/3$ .
C.7 SQ Algorithm for Learning Halfspaces with RCN with Exponential Number of Queries
Here we show that there exists a query-inefficient SQ algorithm that can be simulated with near-optimal sample complexity of  $\tilde{O}(d / \epsilon)$ .
Lemma C.12 (Inefficient SQ Algorithm). There is an  $SQ$  algorithm that makes  $2^{O(d)\mathrm{polylog}(1 / \epsilon)}$  queries to VSTAT(1/ε), and learns the class of halfspaces on  $\mathbb{R}^d$  in the presence of RCN with  $\eta = 1 / 3$  with error at most  $\eta +\epsilon$ .
Proof Sketch. We note that it is always possible to design an exponential query SQ algorithm that achieves the optimal sample complexity of  $\tilde{O}(d/\epsilon)$  (if we were to simulate it with samples). One approach is to generate an  $\epsilon$ -cover  $\mathcal{G}$  that encompasses all hypotheses in  $\mathbb{R}^d$  (with size roughly  $(1/\epsilon)^d$ ), and then utilize the query function  $f(\mathbf{x},y) = \mathbb{1}\{h_1(\mathbf{x}) \neq y\} - \mathbb{1}\{h_2(\mathbf{x}) \neq y\}$  for any  $h_1, h_2 \in \mathcal{G}$ . It can be readily observed that the variance of  $f$  is at most the probability that  $h_1(\mathbf{x}) \neq h_2(\mathbf{x})$ , which means that VSTAT  $(1/\epsilon)$  can distinguish which hypothesis,  $h_1$  or  $h_2$ , yields a smaller error. Note that if we were to simulate this SQ algorithm using samples, we would also need to do a union bound over the set of all hypotheses.
D Lower Bound for Low-Degree Polynomial Testing
D.1 Preliminaries: Low-Degree Method
We begin by recording the necessary notation, definitions, and facts. This section mostly follows  $\mathrm{[BBH^{+}20]}$ .
Low-Degree Polynomials A function  $f: \mathbb{R}^a \to \mathbb{R}^b$  is a polynomial of degree at most  $k$  if it can be written in the form
$$
f (x) = \left(f _ {1} (x), f _ {2} (x), \dots , f _ {b} (x)\right),
$$
where each  $f_{i}:\mathbb{R}^{a}\to \mathbb{R}$  is a polynomial of degree at most  $k$ . We allow polynomials to have random coefficients as long as they are independent of the input  $x$ . When considering list-decodable estimation problems, an algorithm in this model of computation is a polynomial  $f:\mathbb{R}^{d_1\times n}\to \mathbb{R}^{d_2\times \ell}$ , where  $d_{1}$  is the dimension of each sample,  $n$  is the number of samples,  $d_{2}$  is the dimension of the output hypotheses, and  $\ell$  is the number of hypotheses returned. On the other hand, [BBH+20] focuses on binary hypothesis testing problems defined in Definition D.2.
A degree- $k$  polynomial test for Definition D.2 is a degree- $k$  polynomial  $f: \mathbb{R}^{d \times n} \to \mathbb{R}$  and a threshold  $t \in \mathbb{R}$ . The corresponding algorithm consists of evaluating  $f$  on the input  $x_1, \ldots, x_n$  and returning  $H_0$  if and only if  $f(x_1, \ldots, x_n) > t$ .
Definition D.1 (n-sample  $\epsilon$ -good distinguisher). We say that the polynomial  $p: \mathbb{R}^{d \times n} \mapsto \mathbb{R}$  is an  $n$ -sample  $\epsilon$ -distinguisher for the hypothesis testing problem in Definition D.2 if
$$
\mid \underset {X \sim D _ {0} ^ {\otimes n}} {\mathbf {E}} [ p (X) ] - \underset {u \sim \mu} {\mathbf {E}} \underset {X \sim D _ {u} ^ {\otimes n}} {\mathbf {E}} [ p (X) ] \mid \geq \epsilon \sqrt {\operatorname {V a r} _ {X \sim D _ {0} ^ {\otimes n}} [ p (X) ]}.
$$
We call  $\epsilon$  the advantage of the distinguisher.
Let  $\mathcal{C}$  be the linear space of polynomials with a degree at most  $k$ . The best possible advantage is given by the low-degree likelihood ratio
$$
\max_{\substack{p\in \mathcal{C}\\ \mathbf{E}_{X\sim D_{0}^{\otimes n}}[p^{2}(X)]\leq 1}}\big|\mathbf{E}_{u\sim \mu}\mathbf{E}_{X\sim D_{u}^{\otimes n}}[p(X)] - \mathbf{E}_{X\sim D_{0}^{\otimes n}}[p(X)]\big| = \bigg\|\mathbf{E}_{u\sim \mu}\big[(\bar{D}_{u}^{\otimes n})^{\leq k}\big] - 1\bigg\|_{D_{0}^{\otimes n}},
$$
where we denote  $\bar{D}_u = D_u / D_0$  and the notation  $f^{\leq k}$  denotes the orthogonal projection of  $f$  to  $\mathcal{C}$ .
Another notation we will use regarding a finer notion of degrees is the following: We say that the polynomial  $f(x_{1},\ldots ,x_{n}):\mathbb{R}^{d\times n}\to \mathbb{R}$  has samplewise degree  $(r,k)$  if it is a polynomial, where each monomial uses at most  $k$  different samples from  $x_{1},\ldots ,x_{n}$  and uses degree at most  $r$  for each of them. In analogy to what was stated for the best degree-  $k$  distinguisher, the best distinguisher of samplewise degree  $(r,k)$ -achieves advantage  $\left\| \mathbf{E}_{u\sim \mu}[(\bar{D}_u^{\otimes n})^{\leq r,k}] - 1\right\|_{D_0^{\otimes n}}$  the notation  $f^{\leq r,k}$  now means the orthogonal projection of  $f$  to the space of all samplewise degree-  $(r,k)$  polynomials with unit norm.
We begin by formally defining a hypothesis problem.
Definition D.2 (Hypothesis testing). Let  $D_0$  be a distribution and  $\mathcal{S} = \{D_u\}_{u\in S}$  be a set of distributions on  $\mathcal{X}$ . Let  $\mu$  be a prior distribution on the indices  $S$  of that family. We are given access (via i.i.d. samples or oracle) to an underlying distribution where one of the two is true:
-  $H_0$ : The underlying distribution is  $D_0$ .
-  $H_{1}$ : First  $u$  is drawn from  $\mu$  and then the underlying distribution is set to be  $D_{u}$ .
We say that a (randomized) algorithm solves the hypothesis testing problem if it succeeds with non-trivial probability (i.e., greater than 0.9).
Definition D.3. Let  $D_0$  be the joint distribution over the pairs  $(\mathbf{x},y)\in \mathbb{R}^d\times \{\pm 1\}$  where  $\mathbf{x}\sim \mathcal{N}$  and  $y\sim D_0(y)$  independently of  $\mathbf{x}$ . Let  $D_{\mathbf{v}}$  be the joint distribution over pairs  $(\mathbf{x},y)\in \mathbb{R}^{d}\times \{\pm 1\}$  where the marginal on  $y$  is again  $D_0(y)$  but the conditional distribution  $E_{\mathbf{v}}(\mathbf{x}|1)$  is of the form  $A_{\mathbf{v}}$  (as in Theorem 3.2) and the conditional distribution  $E_{\mathbf{v}}(\mathbf{x}| - 1)$  is of the form  $B_{\mathbf{v}}$ . Define  $S = \{E_{\mathbf{v}}\}_{\mathbf{v}\in S}$  for  $S$  being the set of  $d$ -dimensional nearly orthogonal vectors from Fact C.4 and let the hypothesis testing problem be distinguishing between  $D_0$  vs.  $S$  with prior  $\mu$  being the uniform distribution on  $S$ .
In this section, we prove the following:
Theorem D.4. Let  $0 < c < 1/2$ . Consider the hypothesis testing problem of Definition D.3. For  $d \in \mathbb{Z}_+$  with  $d$  larger than an absolute constant, any  $n \leq \Omega(d)^{1/2 - c} / p^2$  and any even integer  $k < d^{c/4}$ , we have that
$$
\left\| _ {\mathbf {v} \sim \mu} \mathbf {E} \left[ (\bar {E} _ {\mathbf {v}} ^ {\otimes n}) ^ {\leq \infty , \Omega (k)} \right] - 1 \right\| _ {D _ {0} ^ {\otimes n}} ^ {2} \leq 1.
$$
We need the following variant of the statistical dimension from  $\mathrm{[BBH^{+}20]}$ , which is closely related to the hypothesis testing problems considered in this section. Since this is a slightly different definition from the statistical dimension (SD) used so far, we will assign the distinct notation (SDA) for it.
Notation For  $f: \mathbb{R} \to \mathbb{R}$ ,  $g: \mathbb{R} \to \mathbb{R}$  and a distribution  $D$ , we define the inner product  $\langle f, g \rangle_D = \mathbf{E}_{X \sim D}[f(X)g(X)]$  and the norm  $\| f \|_D = \sqrt{\langle f, f \rangle_D}$ .