SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {E} _ {\mathbf {x} \sim \mathbf {P} _ {\mathbf {V}} ^ {A}} [ f (\mathbf {x}) ] = \sum_ {k = 0} ^ {\infty} \left\langle \mathbf {V} ^ {\otimes k} \mathbf {A} _ {k}, \mathbf {T} _ {k} \right\rangle \leq \sum_ {k = 0} ^ {\infty} | \left\langle \mathbf {A} _ {k}, \left(\mathbf {V} ^ {\top}\right) ^ {\otimes k} \mathbf {T} _ {k} \right\rangle | \leq \sum_ {k = 0} ^ {\infty} \| \mathbf {A} _ {k} \| _ {2} \| \left(\mathbf {V} ^ {\top}\right) ^ {\otimes k} \mathbf {T} _ {k} \| _ {2}. \tag {1}
$$
The  $k = 0$  term of the above sum will be exactly  $\mathbf{T}_0 = \mathbf{E}_{\mathbf{x} \sim \mathcal{N}_n}[f(\mathbf{x})]$ . By the moment-matching condition, the  $k = 1$  through  $k = d$  terms will be very small. Finally, we need to argue that the higher degree terms are small with high probability. To achieve this, we provide a non-trivial upper bound for  $\mathbf{E}_{\mathbf{V} \sim U(\mathbf{O}_{n,m})}[\| (\mathbf{V}^{\mathsf{T}})^{\otimes k}\mathbf{T}_k\|_2^2]$ . Combining with the fact that  $\sum_{k=0}^{\infty} \| \mathbf{T}_k\|_2^2 = \| f\|_2^2 \leq 1$ , this allows us to prove high probability bounds on each term, and thus their sum.
Furthermore, if we want higher probability estimates of the terms with small  $k$ , we can instead bound  $\mathbf{E}_{\mathbf{V} \sim U(\mathbf{O}_{n,m})}[\| (\mathbf{V}^{\intercal})^{\otimes k}\mathbf{T}_k\|_2^{2a}]$  for some integer  $a$ . Unfortunately, there are non-trivial technical issues with the above approach, arising from issues with (1). To begin with, as no assumptions other than moment-matching (for a few low-degree moments) were made on  $A$ , it is not guaranteed that  $\mathbf{A}_k$  is finite for larger values of  $k$ . To address this issue, we will truncate the distribution  $A$ . In particular, we pick a parameter  $B$  (which will be determined carefully), and define  $A'$  to be  $A$  conditioned on the value in the ball  $\mathbb{B}^m(B)$ . Due to the higher-moment bounds, we can show that  $A$  and  $A'$  are close in total variation distance, and thus that  $\bar{\mathbf{E}}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^A}[f(\mathbf{x})]$  is close to  $\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A'}}[f(\mathbf{x})]$  for any bounded  $f$ .
Furthermore, using the higher moment bounds, we can show that  $A'$  nearly matches the low-degree moments of  $\mathcal{N}_m$ . The second issue arises with the interchange of summations used to derive (1). In particular, although  $f(\mathbf{x}) = \sum_{k=0}^{\infty} \langle \mathbf{T}_k, \mathbf{H}_k(\mathbf{x}) \rangle$ , it does not necessarily follow that we can interchange the infinite sum on the right-hand-side with taking the expectation over  $\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^A$ . To fix this issue, we split  $f$  into two parts  $f^{\leq \ell}$  (consisting of its low-degree Fourier components) and  $f^{> \ell}$ . We note that (1) does hold for  $f^{\leq \ell}$ , as the summation there will be finite, and we can use the above argument to bound  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_n}[f(\mathbf{x})] - \mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^A}[f^{\leq \ell}(\mathbf{x})]$  with high probability. To bound  $|\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^A}[f^{> \ell}(\mathbf{x})]|$ , we note that by taking  $\ell$  large, we can make  $\| f^{> \ell} \|_2 < \delta$  for some exponentially small  $\delta > 0$ . We then bound  $\mathbf{E}_{\mathbf{V} \sim U(\mathbf{O}_{n,m})}[\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^A}[|f^{> \ell}|]] = \mathbf{E}_{\mathbf{x} \sim \mathbf{Q}}[|f^{> \ell}|]$ , where  $\mathbf{Q}$  is the average over  $\mathbf{V}$  of  $\mathbf{P}_{\mathbf{V}}^A$  (note that everything here is non-negative, so there is no issue with the interchange of integrals). Thus, we can bound the desired quantity by noting that  $\| f^{> \ell} \|_2$  is small and that the chi-squared norm of  $\mathbf{Q}$  with respect to the standard Gaussian  $\mathcal{N}_n$  is bounded.
We will use lowercase boldface letters for vectors and capitalized boldface letters for matrices and tensors. We use  $\mathbb{S}^{n - 1} = \{\mathbf{x}\in \mathbb{R}^n:\| \mathbf{x}\| _2 = 1\}$  to denote the  $n$  -dimensional unit sphere. For vectors  $\mathbf{u},\mathbf{v}\in \mathbb{R}^n$ , we use  $\langle \mathbf{u},\mathbf{v}\rangle$  to denote the standard inner product. For  $\mathbf{u}\in \mathbb{R}^n$ , we use  $\| \mathbf{u}\| _k = \left(\sum_{i = 1}^n\mathbf{u}_i^k\right)^{1 / k}$  to denote the  $\ell^k$  -norm of  $\mathbf{u}$ . For tensors, we will consider a  $k$  -tensor to be an element in  $(\mathbb{R}^n)^{\otimes k}\cong \mathbb{R}^{n^k}$ . This can be thought of as a vector with  $n^k$  coordinates. We will use  $\mathbf{A}_{i_1,\dots ,i_k}$  to denote the coordinate of a  $k$  -tensor  $\mathbf{A}$  indexed by the  $k$  -tuple  $(i_1,\ldots ,i_k)$ . By abuse of notation, we will sometimes also use this to denote the entire tensor. The inner product and  $\ell^k$  -norm of  $k$  -tensor are defined by thinking of the tensor as vectors with  $n^k$  coordinates and then use the definition of inner product and  $\ell^k$  -norm of vectors. For a vector  $\mathbf{v}\in \mathbb{R}^n$ , we denote by  $\mathbf{v}^{\otimes k}$  to be a vector (linear object) in  $\mathbb{R}^{n^k}$ . For a matrix  $\mathbf{V}\in \mathbb{R}^{n\times m}$ , we denote by  $\| \mathbf{V}\| _2,\| \mathbf{V}\| _F$  to be the operator norm and Frobenius norm respectively. In addition, we denote by  $\mathbf{V}^{\otimes k}$  to be a matrix (linear operator) mapping  $\mathbb{R}^{n^k}$  to  $\mathbb{R}^{m^k}$ . We use  $\mathbb{1}$  to denote the indicator function of a set, specifically  $\mathbb{1}(t\in S) = 1$  if  $t\in S$  and 0 otherwise. We will use  $\Gamma :\mathbb{R}\to \mathbb{R}$  to denote the gamma function  $\Gamma (z) = \int_0^\infty t^{z - 1}e^{-t}dt$ . We use  $B:\mathbb{R}\times \mathbb{R}\rightarrow \mathbb{R}$  to denote the beta function  $B(z_{1},z_{2}) = \Gamma (z_{1})\Gamma (z_{2}) / \Gamma (z_{1} + z_{2})$ . We use  $\chi_k^2$  to denote the chi-squared distribution with  $k$  degrees of freedom. We use Beta(  $\alpha ,\beta$  ) to denote the Beta distribution with parameters  $\alpha$  and  $\beta$ .
For a distribution  $D$ , we use  $\mathbf{Pr}_D[S]$  to denote the probability of an event  $S$ . For a continuous distribution  $D$  over  $\mathbb{R}^n$ , we sometimes use  $D$  for both the distribution itself and its probability density function. For two distributions  $D_1, D_2$  over a probability space  $\Omega$ , let  $d_{\mathrm{TV}}(D_1, D_2) = \sup_{S \subseteq \Omega} |\mathbf{Pr}_{D_1}(S) - \mathbf{Pr}_{D_2}(S)|$  denote the total variation distance between  $D_1$  and  $D_2$ . For two continuous distributions  $D_1, D_2$  over  $\mathbb{R}^n$ , we use  $\chi^2(D_1, D_2) = \int_{\mathbb{R}^n} D_1(\mathbf{x})^2 / D_2(\mathbf{x}) d\mathbf{x} - 1$  to denote the chi-square norm of  $D_1$  w.r.t.  $D_2$ . For a subset  $S \subseteq \mathbb{R}^n$  with finite measure or finite surface measure, we use  $U(S)$  to denote the uniform distribution over  $S$  (w.r.t. Lebesgue measure for the column/surface area of  $S$ ).
Basics of Hermite Polynomials
Definition 2.1 (Normalized Hermite Polynomial). For  $k \in \mathbb{N}$ , we define the  $k$ -th probabilist's Hermite polynomials  $H_{k} : \mathbb{R} \to \mathbb{R}$  as  $H_{k}(t) = (-1)^{k} e^{t^{2}/2} \cdot \frac{d^{k}}{dt^{k}} e^{-t^{2}/2}$ . We define the  $k$ -th normalized Hermite polynomial  $h_{k} : \mathbb{R} \to \mathbb{R}$  as  $h_{k}(t) = H_{k}(t) / \sqrt{k!}$ .
Furthermore, we will use multivariate Hermite polynomials in the form of Hermite tensors (as the entries in the Hermite tensors are rescaled multivariate Hermite polynomials). We define the Hermite tensor as follows.
Definition 2.2 (Hermite Tensor). For  $k \in \mathbb{N}$  and  $\mathbf{x} \in \mathbb{R}^n$ , we define the  $k$ -th Hermite tensor as
$$
(\mathbf{H}_{k}(\mathbf{x}))_{i_{1},i_{2},\ldots ,i_{k}} = \frac{1}{\sqrt{k!}}\sum_{\substack{\text{Partitions $P$ of $[k]$}\\ \text{into sets of size 1 and 2}}}\bigotimes_{\{a,b\} \in P}(-\mathbf{I}_{i_{a},i_{b}})\bigotimes_{\{c\} \in P}\mathbf{x}_{i_{c}}.
$$
We denote by  $L^2(\mathbb{R}^n, \mathcal{N}_n)$  the function space of all functions  $f: \mathbb{R}^n \to \mathbb{R}$  such that  $\mathbf{E}_{\mathbf{v} \sim \mathcal{N}_n}[f^2(\mathbf{v})] < \infty$ . For functions  $f, g \in L^2(\mathbb{R}^n, \mathcal{N}_n)$ , we use  $\langle f, g \rangle_{\mathcal{N}_n} = \mathbf{E}_{\mathbf{x} \sim \mathcal{N}_n}[f(\mathbf{x})g(\mathbf{x})]$  to denote their inner product. We use  $\|f\|_2 = \sqrt{\langle f, f \rangle_{\mathcal{N}_n}}$  to denote its  $L^2$ -norm. For a function  $f: \mathbb{R}^n \to \mathbb{R}$  and  $\ell \in \mathbb{N}$ , we use  $f^{\leq \ell}$  to denote  $f^{\leq \ell}(\mathbf{x}) = \sum_{k=0}^{\ell} \langle \mathbf{A}_k, \mathbf{H}_k(\mathbf{x}) \rangle$ , where  $\mathbf{A}_k = \mathbf{E}_{\mathbf{x} \sim \mathcal{N}_n}[f(\mathbf{x})\mathbf{H}_k(\mathbf{x})]$ , which is the degree- $\ell$  approximation of  $f$ . We use  $f^{> \ell} = f - f^{\leq \ell}$  to denote its residue. We remark that normalized Hermite polynomials (resp. Hermite tensors) form a complete orthogonal system for the inner product space  $L^2(\mathbb{R}, \mathcal{N})$  (resp.  $L^2(\mathbb{R}^n, \mathcal{N}_n)$ ). This implies that for  $f \in L^2(\mathbb{R}^n, \mathcal{N}_n)$ ,  $\lim_{\ell \to \infty} \|f^{> \ell}\|_2 = 0$ . We also remark that both our definition of Hermite polynomial and Hermite tensor are "normalized". In the sense that for Hermite polynomials,  $\|h_k\|_2 = 1$ . For Hermite tensors, given any symmetric tensor  $A$ , we have  $\| \langle \mathbf{A}, \mathbf{H}_k(\mathbf{x}) \rangle \|_2^2 = \langle \mathbf{A}, \mathbf{A} \rangle$ . The following claim states that for any orthonormal transformation  $\mathbf{B}$ , the Hermite tensor  $\mathbf{H}_k(\mathbf{Bx})$  can be written as applying the linear transformation  $\mathbf{B}^{\otimes k}$  on the Hermite tensor  $\mathbf{H}_k(\mathbf{x})$ . The proof is deferred to Appendix A.
Claim 2.3. Let  $1 \leq m < n$ . Let  $\mathbf{B} \in \mathbb{R}^{m \times n}$  with  $\mathbf{BB}^{\intercal} = \mathbf{I}_m$ . It holds that  $\mathbf{H}_k(\mathbf{Bx}) = \mathbf{B}^{\otimes k}\mathbf{H}_k(\mathbf{x}), \mathbf{x} \in \mathbb{R}^n$ .
3 SQ-Hardness of NGCA: Proof of Theorem 1.5
The main idea of the proof is the following. Suppose that the algorithm only asks queries with tolerance  $\tau$ , and let  $f$  be an arbitrary query function that the algorithm selects. The key ingredient is to show that  $|\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A}}[f(\mathbf{x})] - \mathbf{E}_{\mathbf{x} \sim \mathcal{N}_{n}}[f(\mathbf{x})]| \leq \tau$  with high probability over  $\mathbf{V} \sim U(\mathbf{O}_{n,m})$ . If this holds, then when the algorithm queries  $f$ , if the input is from the alternative hypothesis, with high probability,  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_{n}}[f(\mathbf{x})]$  is a valid answer for the query. Therefore, when the algorithm queries  $f$ , regardless of whether the input is from the alternative or null hypothesis, the oracle can just return  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_{n}}[f(\mathbf{x})]$ . Then the algorithm will not observe any difference between the two cases with any small number of queries. Thus, it is impossible to distinguish the two cases with high probability. To prove the desired bound, we introduce the following proposition.
Proposition 3.1. Let  $\lambda \in (0,1)$  and  $n,m,d\in \mathbb{N}$  with  $d$  be even and  $m,d\leq n^{\lambda}$ . Let  $\nu \in \mathbb{R}_{+}$  and  $A$  be a distribution on  $\mathbb{R}^m$  such that for any polynomial  $f:\mathbb{R}^m\to \mathbb{R}$  of degree at most  $d$  and  $\mathbf{E}_{\mathbf{x}\sim \mathcal{N}_m}[f(\mathbf{x})^2 ] = 1$ ,
$$
\left| \mathbf {E} _ {\mathbf {x} \sim A} [ f (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} _ {m}} [ f (\mathbf {x}) ] \right| \leq \nu .
$$
Let  $0 < c < (1 - \lambda) / 8$  and  $n$  is at least a sufficiently large constant depending on  $c$ , then, for any function  $f: \mathbb{R}^n \to [-1,1]$ , it holds
$$
\mathbf{\Pr}_{\mathbf{V}\sim U(\mathbf{O}_{n,m})}\left[\left|\mathbf{E}_{\mathbf{x}\sim \mathbf{P}^{A}_{\mathbf{V}}} [f(\mathbf{x})] - \mathbf{E}_{\mathbf{x}\sim \mathcal{N}_{n}}[f(\mathbf{x})]\right|\geq \left(\frac{\Gamma(\frac{d + m}{2})}{\Gamma(\frac{m}{2})}\right)n^{-\left(\frac{1 - \lambda}{8} -c\right)d} + (1 + o(1))\nu \right]\leq 2^{-n^{\Omega (c)}}.
$$
Assuming Proposition 3.1, the proof of our main theorem is quite simple.
Proof for Theorem 1.5. Suppose there is an SQ algorithm  $\mathcal{A}$  using  $q < 2^{n^{\Omega (c)}}$  many queries of accuracy  $\tau \geq \left(\frac{\Gamma(d / 2 + m / 2)}{\Gamma(m / 2)}\right)n^{-((1 - \lambda) / 8 - c)d} + (1 + o(1))\nu$  and succeeds with at least  $2 / 3$  probability. We prove by contradiction that such an  $\mathcal{A}$  cannot exist. Suppose the input distribution is  $\mathcal{N}_n$ , and the SQ oracle always answers  $\mathbf{E}_{\mathbf{x}\sim \mathcal{N}_n}[f(\mathbf{x})]$  for any query  $f$ . Then the assumption on  $\mathcal{A}$  implies that it answers "null hypothesis" with probability  $\alpha >2 / 3$ . Now consider the case that the input distribution is  $\mathbf{P}_{\mathbf{V}}^{A}$  and  $\mathbf{V}\sim U(\mathbf{O}_{n,m})$ . Suppose the SQ oracle still always answers  $\mathbf{E}_{\mathbf{x}\sim \mathcal{N}_n}[f(\mathbf{x})]$  for any query  $f$ . Let  $f_{1},\dots ,f_{q}$  be the queries the algorithm asks, where  $q = 2^{n^{\Omega (c)}}$ . By Proposition 3.1 and a union bound, we have
$$
\mathbf {P r} _ {\mathbf {V} \sim U (\mathbf {O} _ {n, m})} [ \exists i \in [ q ], | \mathbf {E} _ {\mathbf {x} \sim \mathbf {P} _ {\mathbf {V}} ^ {A}} [ f _ {i} (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} _ {n}} [ f _ {i} (\mathbf {x}) ] | \geq \tau ] = o (1).
$$
Therefore, with probability  $1 - o(1)$ , the answers given by the oracle are valid. From our assumption on  $\mathcal{A}$ , the algorithm needs to answer "alternative hypothesis" with probability at least  $\frac{2}{3}(1 - o(1))$ . But since the oracle always answers  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_n}[f(\mathbf{x})]$  (which is the same as the above discussed null hypothesis case), we know the algorithm will return "null hypothesis" with probability  $\alpha > 2/3$ . This gives a contradiction and completes the proof.
The rest of this section is devoted to the proof of Proposition 3.1.
3.1 Fourier Analysis using Hermite Polynomials
The main idea of Proposition 3.1 is to analyze  $\mathbf{E}_{\mathbf{x}\sim \mathbf{P}_{\mathbf{V}}^{A}}[f(\mathbf{x})]$  through Fourier analysis using Hermite polynomials. Before we do the analysis, we will first truncate the distribution  $A$  inside  $\mathbb{B}^m (B)$  which is the  $\ell_2$ -norm unit ball in  $m$ -dimension with radius  $B$  for some  $B\in \mathbb{R}_+$  to be specified. Namely, we will consider the truncated distribution  $A^\prime$  defined as the distribution of  $\mathbf{x}\sim A$  conditioned on  $\mathbf{x}\in \mathbb{B}^m (B)$ . The following lemma shows that given any  $m$ -dimensional distribution  $A$  that approximately matches the first  $d$  moment tensor with the Gaussian, the truncated distribution  $A^\prime$  is close to  $A$  in both the total variation distance and the first  $d$  moment tensors.
Lemma 3.2. Let  $n, m, d \in \mathbb{N}$  with  $d$  be even. Let  $A$  be a distribution on  $\mathbb{R}^m$  such that for any polynomial  $f$  of degree at most  $d$  and  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_m}[f(\mathbf{x})^2] = 1$ ,
$$
\left| \mathbf {E} _ {\mathbf {x} \sim A} [ f (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} _ {m}} [ f (\mathbf {x}) ] \right| \leq \nu \leq 2.
$$
Let  $B \in \mathbb{R}_+$  such that  $B^d \geq c_1 \left(2^{d/2} \sqrt{\frac{\Gamma(d + m/2)}{\Gamma(m/2)}}\right)$  where  $c_1$  is at least a sufficiently large universal constant and let  $A'$  be the truncated distribution defined as the distribution of  $\mathbf{x} \sim A$  conditioned on  $\mathbf{x} \in \mathbb{B}^m(B)$ . Then  $d_{\mathrm{TV}}(A, A') = O\left(2^{d/2} \sqrt{\frac{\Gamma(d + m/2)}{\Gamma(m/2)}}\right) B^{-d}$ , Furthermore, any  $k \in \mathbb{N}$  and  $k < d$ ,
$$
\| \mathbf {E} _ {\mathbf {x} \sim A ^ {\prime}} [ \mathbf {H} _ {k} (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathcal {N} _ {m}} [ \mathbf {H} _ {k} (\mathbf {x}) ] \| _ {2} = 2 ^ {O (k)} \left(2 ^ {d / 2} \sqrt {\frac {\Gamma (d + m / 2)}{\Gamma (m / 2)}}\right) B ^ {- (d - k)} + \nu .
$$
The proof of Lemma 3.2 is deferred to Appendix B.
Since  $d_{\mathrm{TV}}(A,A^{\prime})\leq O\left(2^{d / 2}\sqrt{\frac{\Gamma(d + m / 2)}{\Gamma(m / 2)}}\right)B^{-d}$  and  $f$  is bounded in  $[-1,1]$ , it follows that
$$
\begin{array}{l} \left| \mathbf {E} _ {\mathbf {x} \sim \mathbf {P} _ {\mathbf {V}} ^ {A ^ {\prime}}} [ f (\mathbf {x}) ] - \mathbf {E} _ {\mathbf {x} \sim \mathbf {P} _ {\mathbf {V}} ^ {A}} [ f (\mathbf {x}) ] \right| \leq 2 d _ {\mathrm {T V}} \left(\mathbf {P} _ {\mathbf {V}} ^ {A ^ {\prime}}, \mathbf {P} _ {\mathbf {V}} ^ {A ^ {\prime}}\right) = 2 d _ {\mathrm {T V}} (A, A ^ {\prime}) \\ = O \left(\left(2 ^ {d / 2} \sqrt {\frac {\Gamma (d + m / 2)}{\Gamma (m / 2)}}\right) B ^ {- d}\right). \\ \end{array}
$$
Therefore, if suffices for us to analyze  $\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A'}}[f(\mathbf{x})]$  instead of  $\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A}}[f(\mathbf{x})]$ . Furthermore, the property that  $A'$  is bounded inside  $\mathbb{B}^m(B)$  will be convenient in the Fourier analysis later. We introduce the following lemma which decomposes  $\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A'}}[f(\mathbf{x})]$  using Hermite analysis. The proof is deferred to Appendix B.
Lemma 3.3 (Fourier decomposition Lemma). Let  $A'$  be any distribution supported on  $\mathbb{R}^m$ ,  $\mathbf{V} \in \mathbb{R}^{n \times m}$  and  $\mathbf{V}^\intercal \mathbf{V} = \mathbf{I}_m$ . Then for any  $\ell \in \mathbb{N}$ ,  $\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{\mathcal{A}'}}[f(\mathbf{x})] = \sum_{k=0}^{\ell} \langle \mathbf{V}^{\otimes k} \mathbf{A}_k, \mathbf{T}_k \rangle + \mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{\mathcal{A}'}}[f^{> \ell}(\mathbf{x})]$ , where  $\mathbf{A}_k = \mathbf{E}_{\mathbf{x} \sim A'}[\mathbf{H}_k(\mathbf{x})]$  and  $\mathbf{T}_k = \mathbf{E}_{\mathbf{x} \sim \mathcal{N}_n}[f(\mathbf{x}) \mathbf{H}_k(\mathbf{x})]$ .
Remark 3.4. Ideally, in Lemma 3.3, we would like to have  $\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A'}}[f(\mathbf{x})] = \sum_{k=0}^{\infty} \langle \mathbf{V}^{\otimes k} \mathbf{A}_k, \mathbf{T}_k \rangle$ . However, since we do not assume that  $\chi^2(A', \mathcal{N}_m) < \infty$ , this convergence may not hold.
Recall that our goal is to show that  $|\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A}}[f(\mathbf{x})] - \mathbf{E}_{\mathbf{x} \sim \mathcal{N}_{n}}[f(\mathbf{x})]|$  is small with high probability. Observe that  $\mathbf{E}_{\mathbf{x} \sim \mathcal{N}_{n}}[f(\mathbf{x})] = \mathbf{T}_{0}$  which is the first term in the summation of  $\sum_{k=0}^{\ell} \langle \mathbf{V}^{\otimes k} \mathbf{A}_{k}, \mathbf{T}_{k} \rangle$  (since  $\mathbf{A}_{0} = 1$ ). Therefore, given Lemma 3.3, it suffices to show that  $|\sum_{k=1}^{\ell} \langle \mathbf{V}^{\otimes k} \mathbf{A}_{k}, \mathbf{T}_{k} \rangle|$  and  $|\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A'}}[f^{\geq \ell}(\mathbf{x})]|$  are both small with high probability. We ignore the  $|\mathbf{E}_{\mathbf{x} \sim \mathbf{P}_{\mathbf{V}}^{A'}}[f^{\geq \ell}(\mathbf{x})]|$  part for now as this is mostly a technical issue. To bound  $|\sum_{k=1}^{\ell} \langle \mathbf{V}^{\otimes k} \mathbf{A}_{k}, \mathbf{T}_{k} \rangle|$ , it suffices to analyze  $\sum_{k=1}^{\ell} |\langle \mathbf{V}^{\otimes k} \mathbf{A}_{k}, \mathbf{T}_{k} \rangle|$  by looking at each term  $|\langle \mathbf{V}^{\otimes k} \mathbf{A}_{k}, \mathbf{T}_{k} \rangle| = |\langle \mathbf{A}_{k}, (\mathbf{V}^{\intercal})^{\otimes k} \mathbf{T}_{k} \rangle| \leq \| \mathbf{A}_{k}\|_{2}\| (\mathbf{V}^{\intercal})^{\otimes k} \mathbf{T}_{k}\|_{2}$ . To show that the summation is small, given we need to prove that (with high probability):
1.  $\| \mathbf{A}_k\| _2$  does not grow too fast w.r.t  $k$
2.  $\| (\mathbf{V}^{\intercal})^{\otimes k}\mathbf{T}_k\| _2$  decays very fast w.r.t  $k$  (is small with high probability w.r.t the randomness of  $\mathbf{V}$ ).
$\| \mathbf{A}_k\| _2$  does not grow too fast: We will use slightly different arguments depending on the size of  $k$ . We consider three cases:  $k < m$ ,  $m\leq k\leq n^{(1 - \lambda) / 4}$ , and  $k\geq n^{(1 - \lambda) / 4}$  (the value in the exponent will deviate by a small quantity to make the proof go through). For  $k < m$ ,  $\| \mathbf{A}_k\| _2$  grows slowly by the approximate moment-matching property of  $A^\prime$ . For  $m\leq k\leq n^{(1 - \lambda) / 4}$ , we require the following fact:
Fact 3.5. Let  $\mathbf{H}_k$  be the  $k$ -th Hermite tensor for  $m$ -dimension. Suppose  $\| \mathbf{x} \|_2^2 \geq m$ , then  $\| \mathbf{H}_k(\mathbf{x}) \|_2 = 2^{O(k)} \max (\| \mathbf{x} \|_2^k, 1)$ .
We provide the proof of Fact 3.5 in Appendix B. For  $k > n^{(1 - \lambda) / 4}$ , we can show that  $\| \mathbf{A}_k\| _2$  does not grow too fast by the following asymptotic bound on Hermite tensors.
Fact 3.6. Let  $\mathbf{H}_k$  be the  $k$ -th Hermite tensor for  $m$ -dimension, then  $\| \mathbf{H}_k(\mathbf{x}) \|_2 \leq 2^{O(m)} \binom{k+m-1}{m-1}^{1/2} \exp(\|\mathbf{x}\|_2^2/4)$ .
We provide the proof of Fact 3.6 in Appendix B.
$\| (\mathbf{V}^{\intercal})^{\otimes k}\mathbf{T}_k\| _2$  decays very fast: We show that  $|\langle \mathbf{V}^{\otimes k},\mathbf{T}_k\rangle |$  is small with high probability by bounding its  $a$  -th moment for some even  $a$  . Notice that since  $\| \mathbf{H}_k\| _2\leq \| f\| _2\leq 1$  , we can then combine it with the following lemma. We defer its proof to Appendix B.
Lemma 3.7. Let  $k \in \mathbb{Z}_{+}$ ,  $a \in \mathbb{Z}_{+}$  be even,  $\mathbf{T} \in \mathbb{R}^{n^k}$  and  $m \in \mathbb{Z}_{+}$  satisfy  $m < n$ . Then, we have
$$
\mathbf {E} _ {\mathbf {V} \sim U (\mathbf {O} _ {n, m})} [ \| (\mathbf {V} ^ {\intercal}) ^ {\otimes k} \mathbf {T} \| _ {2} ^ {a} ] \leq \mathbf {E} _ {\mathbf {V} \sim U (\mathbf {O} _ {n, m})} \left[ \| \mathbf {V} ^ {\intercal} \mathbf {u} \| _ {2} ^ {a k / 2} \right] \| \mathbf {T} \| _ {2} ^ {a}.
$$
Roughly speaking, this is just the  $ak / 2$ -th moment of the correlation between a random subspace and a random direction, which can be upper bounded by the following lemma and corollary (see Appendix B for the proofs).
Lemma 3.8. For any even  $k\in \mathbb{N}$ , and  $\mathbf{u}\in \mathbb{S}^{n - 1}$ ,  $\mathbf{E}_{\mathbf{V}^{\intercal}\sim U(\mathbf{O}_{n,m})}[\| \mathbf{V}\mathbf{u}\| _2^k ] = \Theta \left(\frac{\Gamma\left(\frac{k + m}{2}\right)\Gamma\left(\frac{n}{2}\right)}{\Gamma\left(\frac{k + n}{2}\right)\Gamma\left(\frac{m}{2}\right)}\right)$ .
Corollary 3.9. Let  $c \in (0,1)$  and  $m \leq n^c$ . Let  $k \in \mathbb{N}$  be even. We have that
$$
\mathbf{E}_{\mathbf{V}\sim U(\mathbf{o}_{n,m})}[\| \mathbf{V}^{\intercal}\mathbf{u}\|_{2}^{k}] = \left\{ \begin{array}{ll}O(2^{k / 2}n^{-(1 - c)k / 2}) & k\leq n^{c}  ,\\ \exp (-\Omega (n^{c}))O\left(\left(\frac{n^{c} + n}{k + n}\right)^{(n - m) / 2}\right) & k\geq n^{c}  . \end{array} \right.
$$
To combine the above results and give the high probability upper bound on  $\sum_{k=1}^{\ell} |\langle \mathbf{A}_k, (\mathbf{V}^\intercal)^{\otimes k} \mathbf{T}_k \rangle|$ , we require the following lemma. The proof is deferred to Appendix B.