SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \pi_ {A _ {0}} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \operatorname {p o l y} (A _ {0}) \left(\frac {\sigma^ {2}}{\sqrt {n}} + \frac {\log d}{\sqrt {n}} + B ^ {2} \sqrt {\frac {\log (1 / \delta)}{n}} + B ^ {2} \sqrt {\frac {\log d (\log (B) + \log (\gamma (U _ {\lambda})))}{n}}\right)
$$
$$
\log (\gamma (U _ {\lambda})) \lesssim \log (d) \log \left(\frac {1}{2 \lambda} \left(\sigma^ {2} + B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}}\right) + \log d\right).
$$
Simplify the inequality, we have
$$
\left\| \pi_ {A _ {0}} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \operatorname {p o l y} (A _ {0}, \sigma , \log (1 / \delta), \log \log n) \frac {\log n \log d \sqrt {\log n + \log \log d}}{\sqrt {n}}
$$
$$
n \geqslant \operatorname {p o l y} (\sigma , \log \frac {1}{\bar {\delta}}, \log \frac {1}{\epsilon}) \epsilon^ {- 2} \log^ {2} d (\log \log d) ^ {3}
$$
H LCNs vs. FCNs: Proofs of Section 5.3
H.1 Proof of Theorem 5.7
Under the assumptions in Theorem 5.7, Lemma 5.6 implies that  $\mathbb{A}_T^{\mathrm{fcn}}$  is  $G_{\mathrm{ort}}(4d)$ -equivalent. Then, by Lemma 5.1, we have
$$
\mathcal {C} \left(\mathbb {A} _ {T} ^ {\mathrm {f c n}}, \{\bar {h} ^ {*} \}, \epsilon\right) \geqslant \bar {\mathcal {C}} \left(\left\{\bar {h} ^ {*} \right\} \circ G _ {\text {o r t}} (4 d), \epsilon\right).
$$
We next use the Fano's method (Proposition B.20) to bound the right hand side, by which the key to obtaining a tight bound is to construct a packing of  $\{\bar{h}^*\} \circ G_{\mathrm{ort}}(4d)$  as large as possible. Specifically, we will prove the following lemma.
Lemma H.1. There exist absolute positive constants  $C, c_5, c_6 > 0$  such that if  $d, A_0 \geqslant C$ , there exists  $h_1, h_2, \ldots, h_M \in \{\bar{h}^*\} \circ O(4d)$  satisfying that
$$
\sup  _ {i, j} \| h _ {i} - h _ {j} \| _ {L ^ {2} (P)} ^ {2} \leqslant c _ {6}, \inf  _ {i \neq j} \| h _ {i} - h _ {j} \| _ {L ^ {2} (P)} ^ {2} \geqslant \frac {1}{4} c _ {5}, a n d M \geqslant 2 ^ {d (d - 1) / 2}.
$$
Assuming Lemma H.1 is right for now and applying the above packing to Fano's inequality (Proposition B.20), we have
$$
\begin{array}{l} \inf  _ {\mathbb {A} \in \mathcal {A}} \sup  _ {h ^ {*} \in \bar {h} ^ {*} \circ O (4 d)} \mathbb {E} \left[ \left\| h _ {\mathbb {A} (S _ {n})} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \right] \geqslant A \left(1 - \frac {n}{2 \sigma^ {2} M ^ {2} \log M} \sum_ {i, j = 1} ^ {M} \| h _ {i} - h _ {j} \| _ {L ^ {2} (P)} ^ {2} - \frac {\log 2}{\log M}\right) \\ \geqslant \frac {1}{4} c _ {5} \left(1 - \frac {c _ {6} n}{\sigma^ {2} d (d - 1) \log 2} - \frac {2}{d (d - 1)}\right) \\ \end{array}
$$
Taking  $n$  such that the right hand side to satisfy
$$
\frac {1}{4} c _ {5} \left(1 - \frac {c _ {6} n}{\sigma^ {2} d (d - 1) \log 2} - \frac {2}{d (d - 1)}\right) \leqslant \frac {c _ {5}}{8} =: \epsilon_ {0}
$$
gives  $n = \Omega(\sigma^2 d^2)$ . Thus, we can conclude that  $\bar{\mathcal{C}}(\{\bar{h}^*\} \circ G_{\mathrm{ort}}(4d), \epsilon_0) = \Omega(\sigma^2 d^2)$ . Otherwise, the minimax error becomes larger than  $\epsilon_0$ .
H.1.1 Proof of Lemma H.1
We now turn to construct a proper packing for  $\{\bar{h}^*\} \circ G_{\mathrm{ort}}(4d)$  to satisfy the condition in Lemma H.1. To simplify the statement, define  $q:\mathbb{R}^{2d}\mapsto \mathbb{R}$  and  $g_U:\mathbb{R}^{2d}\mapsto \mathbb{R}$  for  $U\in \mathbb{R}^{d\times d}$  by
$$
q (\mathbf {x}) = \sum_ {i = 1} ^ {d} \left(x _ {2 i - 1} ^ {2} - x _ {2 i} ^ {2}\right) \quad g _ {U} (\mathbf {x}) = \mathbf {x} _ {1: d} ^ {T} U \mathbf {x} _ {d + 1: 2 d}.
$$
Lemma H.2. (A restatement of (Li et al., 2020, Lemma D.2)) For any  $U \in G_{\text{ort}}(d)$ , there exist  $\tau_U \in G_{\text{ort}}(2d)$  such that
$$
g _ {U} (\mathbf {x}) = q \circ \tau_ {U} (\mathbf {x}).
$$
This lemma implies that for any  $U \in G_{\mathrm{ort}}(d)$ ,
$$
\begin{array}{l} \frac {1}{d} \left(\mathbf {x} _ {1: d} ^ {T} U \mathbf {x} _ {d + 1: 2 d}\right) \left(\sum_ {i = 1} ^ {d} \left(x _ {2 i + 2 d - 1} ^ {2} - x _ {2 i + 2 d} ^ {2}\right)\right) = d ^ {- 1} q \circ \tau_ {U} (\mathbf {x} _ {1: 2 d}) q (\mathbf {x} _ {2 d + 1: 4 d}) \\ = h ^ {*} \circ \left( \begin{array}{c c} \tau_ {U} & 0 \\ 0 & I _ {2 d} \end{array} \right) (\mathbf {x}) \in \{h ^ {*} \} \circ G _ {\text {o r t}} (4 d), \tag {113} \\ \end{array}
$$
where we slightly abuse the notation: using  $f(\mathbf{x})$  to denote the function  $f(\cdot)$ .
$$
\mathcal {U} _ {A _ {0}} = \left\{\pi_ {A _ {0}} \circ f _ {U}: f _ {U} (\mathbf {x}) = \frac {1}{d} \left(\mathbf {x} _ {1: d} ^ {T} U \mathbf {x} _ {d + 1: 2 d}\right) \left(\sum_ {i = 1} ^ {d} \left(x _ {2 i + 2 d - 1} ^ {2} - x _ {2 i + 2 d} ^ {2}\right)\right), U \in G _ {\text {o r t}} (d) \right\}. \tag {114}
$$
Note that equation (113) implies
$$
\mathcal {U} _ {A _ {0}} \subset \{\bar {h} ^ {*} \} \circ G _ {\mathrm {o r t}} (4 d).
$$
Next, the following lemma shows that when the target truncation threshold  $A_0$  is large enough, finding a packing of  $\left(\mathcal{U}_{A_0},\| \cdot \|_{L^2 (P)}\right)$ , thus a packing of  $\left(\{\bar{h}^{*}\} \circ G_{\mathrm{ort}}(4d),\| \cdot \|_{L^2 (P)}\right)$ , can be reduced to pack the equivariance group  $\left(G_{\mathrm{ort}}(d),\| \cdot \| _F / \sqrt{d}\right)$ .
Lemma H.3. There exist absolute constants  $C, c_3, c_4 > 0$  such that when  $d, A_0 \geqslant C$ , we have for any  $U, U' \in G_{\text{ort}}(d)$  that
$$
\frac {c _ {3}}{d} \left\| U - U ^ {\prime} \right\| _ {F} ^ {2} \leqslant \left\| \pi_ {A _ {0}} \circ f _ {U} - \pi_ {A _ {0}} \circ f _ {U ^ {\prime}} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \frac {c _ {4}}{d} \left\| U - U ^ {\prime} \right\| _ {F} ^ {2}
$$
Proof idea. The proof of this lemma is technically lengthy and deferred to Appendix I.2. Note that when  $A_0 = +\infty$ , i.e., there is no truncation in the target function, a simple calculation leads to
$$
\mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] = \frac {4 \left\| U - U ^ {\prime} \right\| _ {F} ^ {2}}{d}
$$
When  $A_0$  is finite but large enough, by concentration inequalities, we can show that the influence of truncation is negligible, although the proof is lengthy.
Lemma H.4. (Li et al., 2020, Lemma D.4.) For any positive integer  $d \geqslant 2$  and any positive  $\epsilon \leqslant c_2 / 2$ , we have
$$
\mathcal {P} (G _ {o r t} (d), c _ {1} \parallel \cdot \parallel_ {F} / \sqrt {d}, \epsilon) \geqslant \left(\frac {c _ {2}}{\epsilon}\right) ^ {\frac {d (d - 1)}{2}}
$$
where  $c_1$  and  $c_2$  are two positive absolute constants.
The above lemma provides a lower bound of the packing number of  $\left(G_{\mathrm{ort}}(d),\| \cdot \| _F / \sqrt{d}\right)$ . Intuitively speaking, this lemma is true due to the fact that  $G_{\mathrm{ort}}(d)$  can be viewed as a  $\frac{d(d - 1)}{2}$  dimensional compact manifold from a Lie group perspective.
Proof of Lemma H.1. First, for any  $U, U' \in G_{\mathrm{ort}}(d)$ , we have
$$
\left\| U - U ^ {\prime} \right\| _ {F} ^ {2} \leqslant 2 \left(\left\| U \right\| _ {F} ^ {2} + \left\| U ^ {\prime} \right\| _ {F} ^ {2}\right) = 4 d.
$$
By Lemma H.3, we have
$$
\left\| \pi_ {A _ {0}} \circ f _ {U} - \pi_ {A _ {0}} \circ f _ {U ^ {\prime}} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \frac {c _ {4}}{d} \| U - U ^ {\prime} \| _ {F} ^ {2} \leqslant 4 c _ {4}. \tag {115}
$$
By Lemma H.4 and taking  $\epsilon_0 = c_2 / 2$ , we there exist  $U_{1},\ldots ,U_{M}\in G_{\mathrm{ort}}(d)$  with  $M\geqslant 2^{\frac{d(d - 1)}{2}}$  such that  $\| U_i - U_j\| _F\geqslant \frac{c_2\sqrt{d}}{2c_1}$  for any  $i\neq j$ . Thus, by Lemma H.3, we have for any  $i\neq j$  that
$$
\left\| \pi_ {A _ {0}} \circ f _ {U _ {i}} - \pi_ {A _ {0}} \circ f _ {U _ {j}} \right\| _ {L ^ {2} (P)} ^ {2} \geqslant \frac {c _ {3}}{d} \frac {c _ {2} ^ {2} d}{4 c _ {1} ^ {2}} =: c _ {5}. \tag {116}
$$
Combining (115) and (116), we can conclude that  $\{\pi_{A_0} \circ f_{U_j}\}_{j=1}^m$  are a packing that satisfy the condition in Lemma H.1. Thus, we complete the proof.
H.2 Proof of Theorem 5.5
Let  $\sigma(z) \coloneqq \mathrm{ReLU}(z)$  in this subsection for notation simplicity. We first recall the architecture of the LCN model:  $L = \log_2(4d)$ , channel number  $C_1 = C_L = 4$ ,  $C_l = 2$  for  $l = 2, \ldots, L-1$ , and activation functions  $\sigma_1(z) = \sigma_L(z) = \sigma^2(z)$ ,  $\sigma_l(z) = \sigma(z)$  for  $l = 2, \ldots, L-1$ .
Step I: Representing the target using LCNs.
Lemma H.5. There is a parametrization  $\theta^{*}$  such that  $h_{\theta^{*}}^{\mathrm{len}} = h^{*}$  and  $\| \theta^{*}\|_{\mathcal{P}}\lesssim d$
Proof. With the same depth, activation functions, and channel numbers, a LCN can simulate its CNN counterpart. By the proof of Lemma G.7, the exact representation is obvious. The parameter norm bound is due to
$$
\begin{array}{l} \| \theta \| _ {\mathcal {P}} = \| W _ {o} \| + \sum_ {l = 1} ^ {L} (\| W ^ {(l)} \| _ {F} + \| \mathbf {b} ^ {(l)} \| _ {F}) \\ \lesssim 1 + \sum_ {l = 1} ^ {L} (D _ {l} + 0) = 1 + \sum_ {l = 1} ^ {\log d + 2} \frac {4 d}{2 ^ {l}} \lesssim d. \\ \end{array}
$$
Recall the input distribution  $P = \mathcal{N}(0, I_{4d})$ . Let  $\mathcal{H}_J^{\mathrm{LCN}} = \{h_\theta : \| \theta \| \leqslant J\}$ , where we recall  $\| \cdot \|$  is defined in Appendix C. By Proposition B.14, the covering number satisfies
$$
\mathcal {N} \left(\mathcal {H} _ {J} ^ {\mathrm {L C N}}, \hat {\rho} _ {n}, t\right) \leqslant \left(\frac {3 \hat {\gamma} _ {n} (J)}{t}\right) ^ {N _ {\mathrm {l c n}}} \tag {117}
$$
where  $\hat{\gamma}_n(J) = \hat{M}_nJ(1 + J)^L$ ,  $\hat{M}_n = \sqrt{\frac{1}{n}\sum_{i=1}^n\left(\bar{Q}_{\sigma}(\mathbf{x}_i)\right)^2\left(\|\mathbf{x}_i\|_2 + 1\right)^2}$  and  $N_{\mathrm{lcn}} = \mathcal{O}(d)$ . Here we use the fact that  $L = \log_2d + 2$ . Following the same argument as in Appendix G.2 we have  $\bar{Q}_{\sigma}(\mathbf{x}_i) \leqslant 2^{L+2}(1 + J)^{L+2}\left(\|\mathbf{x}_i\|_2 + 1\right)^3$ . Therefore, we will have
$$
\begin{array}{l} \gamma (J) = \mathbb {E} [ \hat {\gamma} _ {n} (J) ] \lesssim J (1 + J) ^ {L} \mathbb {E} [ \hat {M} _ {n} ] \\ \stackrel {(a)} {\lesssim} J (1 + J) ^ {2 L + 2} 2 ^ {L + 2} \sqrt {\mathbb {E} [ (\| \mathbf {x} \| _ {2} + 1) ^ {8} ]} \stackrel {(b)} {\lesssim} d ^ {2} J (1 + J) ^ {2 L + 2} 2 ^ {L + 2} \tag {118} \\ \end{array}
$$
where  $(a)$  uses the Jensen's inequality and  $(b)$  follows from the higher-order moment bound of sub-exponential random variables (Vershynin, 2018, Proposition 2.7.1).
Recalling we have  $\alpha_{\mathcal{H}} = 1$  from equation (61) and applying Proposition B.14, we have with probability at least  $1 - 2\delta$
$$
\left\| \pi_ {A} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} - \epsilon_ {*} \lesssim \frac {\sigma^ {3} B}{(B - 2 A) ^ {2}} e ^ {- \frac {(B - 2 A) ^ {2}}{2 \sigma^ {2}}} + \lambda M _ {*} + B ^ {2} \sqrt {\frac {\log (4 / \delta)}{n}} + B ^ {2} \sqrt {\frac {N _ {\mathrm {l c n}} \log (B \gamma (U _ {\lambda}))}{n}},
$$
$$
\epsilon_ {*} = 0, \quad M _ {*} \lesssim d
$$
$$
\left\| \pi_ {A _ {0}} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \operatorname {p o l y} (A _ {0}) \left(\frac {\sigma^ {2}}{\sqrt {n}} + \frac {\log d}{\sqrt {n}} + B ^ {2} \sqrt {\frac {\log (1 / \delta)}{n}} + B ^ {2} \sqrt {\frac {d (\log (B) + \log (\gamma (U _ {\lambda})))}{n}}\right)
$$
$$
\log (\gamma (U _ {\lambda})) \lesssim \log (d) \log \left(\frac {1}{2 \lambda} \left(\sigma^ {2} + B ^ {2} \sqrt {\frac {2 \log (2 / \delta)}{n}}\right) + d\right).
$$
$$
\left\| \pi_ {A _ {0}} \circ h _ {\hat {\theta} _ {n}} - h ^ {*} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \operatorname {p o l y} \left(A _ {0}, \sigma , \log (1 / \delta), \log \log n\right) \frac {\log n \sqrt {d \log d} \sqrt {\log n + \log d}}{\sqrt {n}}
$$
$$
n \geqslant \operatorname {p o l y} \left(A _ {0}, \sigma , \log \frac {1}{\delta}, \log \frac {1}{\epsilon}\right) \epsilon^ {- 2} d \log^ {4} d
$$
I Auxiliary Lemmas for Appendix G and H
Recall our input distribution  $P = \mathcal{N}(0, I_{4d})$ . Recall that target function  $\bar{h}^* = \pi_{A_0} \circ h^*$  where
$$
h ^ {*} (\mathbf {x}) = \frac {1}{d} \left(\sum_ {i = 1} ^ {d} (x _ {2 i - 1} ^ {2} - x _ {2 i} ^ {2})\right) \left(\sum_ {i = 1} ^ {d} (x _ {2 d + 2 i - 1} ^ {2} - x _ {2 d + 2 i} ^ {2})\right)
$$
In the following, we show that the truncation operator  $\pi_{A_0}$  on  $h^* \circ U$  for any  $U \in G_{\mathrm{ort}}(4d)$  will not matter a lot when  $A_0$  is moderately large.
Lemma I.1. Let  $X \sim P$ . For any  $U \in G_{\text{ort}}(4d)$ , there exists one absolute constant  $c_1 > 0$  such that for any  $\delta \in (0,1)$ , if  $A_0 \geqslant \max \left(\frac{\log(4 / \delta)}{c_1}, \frac{\log^2(4 / \delta)}{c_1^2 d}\right)$ , it holds that
$$
\mathbb {P} \left\{\pi_ {A _ {0}} \circ h ^ {*} \circ U (X) = h ^ {*} \circ U (X) \right\} \geqslant 1 - \delta
$$
Proof. Without loss of generality, we can set  $U = I_{4d}$  since  $P$  is invariant under  $G_{\mathrm{ort}}(4d)$ . For simplicity, let  $Y_{i} = X_{2i - 1}^{2} - X_{2i}^{2}$  for  $i \in [2d]$  in this proof. It is easy to check that  $Y_{i}$  are i.i.d. random variables with  $\| Y_i\|_{\psi_1} \lesssim 1$ . Then, using the Bernstein's inequality (Lemma B.3) gives
$$
\mathbb {P} \left\{\left| \sum_ {i = 1} ^ {d} Y _ {i} \right| \geqslant \sqrt {d} t \right\} = \mathbb {P} \left\{\left| \frac {1}{d} \sum_ {i = 1} ^ {d} Y _ {i} \right| \geqslant \frac {t}{\sqrt {d}} \right\} \leqslant 2 e ^ {- c _ {1} \min (\sqrt {d} t, t ^ {2})}
$$
where  $c_{1} > 0$  is one absolute constant. Setting the failure probability to be smaller than  $\delta / 2$ :  $2e^{-c_{1}\min(t^{2},\sqrt{d}t)} \leqslant \delta / 2$  gives  $t \geqslant \max\left(\sqrt{\frac{\log(4 / \delta)}{c_{1}}}, \frac{\log(4 / \delta)}{c_{1}\sqrt{d}}\right) =: C_{\delta}$ . This implies that it holds w.p. at least  $1 - \delta / 2$  that
$$
\left| \sum_ {i = 1} ^ {d} Y _ {i} \right| \leqslant C _ {\delta} \sqrt {d}.
$$
Similarly, it holds  $w.p.$  at least  $1 - \delta / 2$  that
$$
\left| \sum_ {i = d + 1} ^ {2 d} Y _ {i} \right| \leqslant C _ {\delta} \sqrt {d}.
$$
Combining them leads to
$$
\mathbb {P} \{\pi_ {A _ {0}} \circ h ^ {*} (X) = h ^ {*} (X) \} = \mathbb {P} \{h ^ {*} (X) \leqslant C _ {\delta} ^ {2} \} \geqslant 1 - \delta .
$$
I.1 Proof of Lemma G.2
For any  $\tau = \mathrm{diag}\left(U_1,U_2,\ldots ,U_{2d}\right),\tau ' = \mathrm{diag}\left(U_1,U_2',\ldots ,U_{2d}^{\prime}\right)\in G_{\mathrm{semiloc}}$  , let
$$
\mathcal {I} = \{i \in [ 2 d ]: U _ {i} \neq U _ {i} ^ {\prime} \}
$$
be the set of indices where the local permutations are different for  $\tau$  and  $\tau'$ . Let  $s = |\mathcal{I}|$ . By the definition of  $G_{\mathrm{semiloc}}$ , we must have  $j \leqslant d$  for any  $j \in \mathcal{I}$ . Thus, we have
$$
h ^ {*} \circ \tau (\mathbf {x}) - h ^ {*} \circ \tau^ {\prime} (\mathbf {x}) = \frac {2}{d} \left(\sum_ {j \in \mathcal {I}} (- 1) ^ {o _ {j}} \left(x _ {2 j - 1} ^ {2} - x _ {2 j} ^ {2}\right)\right) \left(\sum_ {i = 1} ^ {d} \left(x _ {2 d + 2 i - 1} ^ {2} - x _ {2 d + 2 i} ^ {2}\right)\right), \tag {119}
$$
where  $o_j \in \{+1, -1\}$  for  $j \in \mathcal{I}$ . Because of the symmetry of  $P$ , we can assume  $o_j = 1$  for all  $j \in \mathcal{I}$  without loss of generality.
Upper bound. First, recall that  $X \sim \mathcal{N}(0, I_{4d})$ , we have
$$
\begin{array}{l} \left\| \bar {h} ^ {*} \circ \tau - \bar {h} ^ {*} \circ \tau^ {\prime} \right\| _ {L ^ {2} (P)} ^ {2} = \mathbb {E} | \pi_ {A _ {0}} \circ h ^ {*} \circ \tau (X) - \pi_ {A _ {0}} \circ h ^ {*} \circ \tau^ {\prime} (X) | ^ {2} \\ \leqslant \mathbb {E} \left| h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X) \right| ^ {2} \\ = \mathbb {E} \left[ \frac {4}{d ^ {2}} \left(\sum_ {j \in \mathcal {I}} \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right)\right) ^ {2} \left(\sum_ {i = 1} ^ {d} \left(X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2}\right)\right) ^ {2} \right] \\ = \frac {4}{d ^ {2}} \mathbb {E} \left[ \left(\sum_ {j \in \mathcal {I}} \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right)\right) ^ {2} \right] \mathbb {E} \left[ \left(\sum_ {i = 1} ^ {d} \left(X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2}\right)\right) ^ {2} \right] \\ = \frac {4}{d ^ {2}} \operatorname {V a r} \left(\sum_ {j \in \mathcal {I}} \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right)\right) \operatorname {V a r} \left(\sum_ {i = 1} ^ {d} \left(X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2}\right)\right) \\ = \frac {4}{d ^ {2}} \cdot s d \cdot \left(\operatorname {V a r} \left(Y ^ {2} - Z ^ {2}\right)\right) ^ {2} \\ = \frac {6 4 s}{d} \tag {120} \\ \end{array}
$$
where  $Y, Z \sim \mathcal{N}(0,1)$  are independent random variables.
Lower Bound. Denote the events
$$
E _ {1} = \left\{\mathbf {x}: \pi_ {A _ {0}} \circ h ^ {*} \circ \tau (\mathbf {x}) = h ^ {*} \circ \tau (\mathbf {x}) \right\}
$$
$$
E _ {2} = \left\{\mathbf {x}: \pi_ {A _ {0}} \circ h ^ {*} \circ \tau^ {\prime} (\mathbf {x}) = h ^ {*} \circ \tau^ {\prime} (\mathbf {x}) \right\}
$$
for the lower bound, we have
$$
\begin{array}{l} \left\| \bar {h} ^ {*} \circ \tau - \bar {h} ^ {*} \circ \tau^ {\prime} \right\| _ {L ^ {2} (P)} ^ {2} \geqslant \mathbb {E} \left[ 1 _ {[ E _ {1} \cap E _ {2} ]} (X) \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {2} \right] - \mathbb {E} \left[ 1 _ {\left(E _ {1} \cap E _ {2}\right) ^ {c}} \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {2} \right] \\ \geqslant \mathbb {E} \left[ \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {2} \right] \\ - \sqrt {\left(1 - \mathbb {P} \left(E _ {1} \cap E _ {2}\right)\right) \mathbb {E} \left[ \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {4} \right]} \tag {121} \\ \end{array}
$$
where the last inequality follows from the Cauchy-Schwarz inequality:  $(\mathbb{E}YZ)^2\leqslant \mathbb{E}Y^2\mathbb{E}Z^2$
For the first term on the RHS of (121), we have from (120) that
$$
\mathbb {E} \left[ \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {2} \right] = \frac {6 4 s}{d} \tag {122}
$$
For the second term on the RHS of (121), we have
$$
\begin{array}{l} \mathbb {E} \left[ \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {4} \right] = \frac {1 6}{d ^ {4}} \mathbb {E} \left[ \left(\sum_ {j \in \mathcal {I}} \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right)\right) ^ {4} \left(\sum_ {i = 1} ^ {d} \left(X _ {2 i + 2 d - 1} ^ {2} - X _ {2 i + 2 d} ^ {2}\right)\right) ^ {4} \right] \\ = \frac {1 6}{d ^ {4}} \mathbb {E} \left[ \left(\sum_ {j \in \mathcal {I}} \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right)\right) ^ {4} \right] \mathbb {E} \left[ \left(\sum_ {i = 1} ^ {d} \left(X _ {2 i + 2 d - 1} ^ {2} - X _ {2 i + 2 d} ^ {2}\right)\right) ^ {4} \right] \tag {123} \\ \end{array}
$$
We first compute the first term by expanding it.
$$
\begin{array}{l} \mathbb {E} \left[ \left(\sum_ {j \in \mathcal {I}} \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right)\right) ^ {4} \right] = \sum_ {i, j \in \mathcal {I}} \mathbb {E} \left[ \left(X _ {2 j - 1} ^ {2} - X _ {2 j} ^ {2}\right) ^ {2} \left(X _ {2 j - 1} ^ {2} - X _ {2 i} ^ {2}\right) ^ {2} \right] \\ \lesssim s ^ {2}, \\ \end{array}
$$
where we use the fact:  $\mathbb{E}\left[(X_{2j - 1}^2 -X_{2j}^2)^2 (X_{2i - 1}^2 -X_{2i}^2)^2\right] = \mathcal{O}(1)$  for all  $i,j$ . By the same argument, we have
$$
\mathbb {E} \left[ \left(\sum_ {i = 1} ^ {d} \left(X _ {2 i + 2 d - 1} ^ {2} - X _ {2 i + 2 d} ^ {2}\right)\right) ^ {4} \right] \lesssim d ^ {2} \tag {124}
$$
That leads to the following estimate: there exists an absolute constant  $c_3 > 0$  such that
$$
\mathbb {E} \left[ \left(h ^ {*} \circ \tau (X) - h ^ {*} \circ \tau^ {\prime} (X)\right) ^ {4} \right] \leqslant c _ {3} \frac {s ^ {2}}{d ^ {2}} \tag {125}
$$
By Lemma I.1, there is an absolute constant  $C$  such that when  $A_0 \geqslant C$ , we have
$$
1 - \mathbb {P} \left(E _ {1} \cap E _ {2}\right) \leqslant \frac {1}{c _ {3}} \tag {126}
$$
Plugging (122), (125), and (126) into (121) gives
$$
\left\| \bar {h} ^ {*} \circ \tau - \bar {h} ^ {*} \circ \tau^ {\prime} \right\| _ {L ^ {2} (P)} ^ {2} \geqslant \frac {6 4 s}{d} - \frac {s}{d} = \frac {6 3 s}{d} \tag {127}
$$
Combining the upper bound and the lower bound, we complete the proof.
I.2 Proof of Lemma H.3
For any  $U\in O(d)$  , let  $h_U = \pi_{A_0}\circ f_U$  with
$$
f _ {U} (\mathbf {x}) = \frac {1}{d} \left(\mathbf {x} _ {1: d} ^ {\top} U \mathbf {x} _ {d + 1: 2 d}\right) \left(\sum_ {i = 1} ^ {d} \left(x _ {2 d + 2 i - 1} ^ {2} - x _ {2 d + 2 i} ^ {2}\right)\right).
$$
Thus, we have  $h_U \in \{\bar{h}^*\} \circ G_{\mathrm{ort}}(4d)$  due to Lemma H.2.
Upper Bound. We have
$$
\begin{array}{l} \left\| h _ {U} - h _ {U ^ {\prime}} \right\| _ {L ^ {2} (P)} ^ {2} = \mathbb {E} \left[ \left(\pi_ {A _ {0}} \circ f _ {U} (X) - \pi_ {A _ {0}} \circ f _ {U ^ {\prime}} (X)\right) ^ {2} \right] \\ \leqslant \mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] \\ = \frac {1}{d ^ {2}} \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {2} \left(\sum_ {i = 1} ^ {d} \left(X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2}\right)\right) ^ {2} \right] \\ = \frac {1}{d ^ {2}} \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} \left(U - U ^ {\prime}\right) X _ {d + 1: 2 d}\right) ^ {2} \right] \mathbb {E} \left[ \left(\sum_ {i = 1} ^ {d} \left(X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2}\right)\right) ^ {2} \right], \tag {128} \\ \end{array}
$$
where the last step follows from the independence between  $X_{1:2d}$  and  $X_{2d+1:4d}$ .
Note that by equation (120),
$$
\mathbb {E} \left[ \left(\sum_ {i = 1} ^ {d} \left(X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2}\right)\right) ^ {2} \right] = 4 d \tag {129}
$$
And, for the first term, we have
$$
\begin{array}{l} \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {2} \right] = \mathbb {E} \left[ \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {2} | X _ {d + 1: 2 d} \right] \right] \\ = \mathbb {E} \left[ \mathbb {E} \left[ X _ {d + 1: 2 d} ^ {\top} (U - U ^ {\prime}) ^ {\top} X _ {1: d} X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d} | X _ {d + 1: 2 d} \right] \right] \\ = \mathbb {E} \left[ X _ {d + 1: 2 d} ^ {\top} (U - U ^ {\prime}) ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d} \right] \\ = \operatorname {T r} \left(\left(U - U ^ {\prime}\right) ^ {\top} \left(U - U ^ {\prime}\right)\right) \\ = \| U - U ^ {\prime} \| _ {F} ^ {2} \tag {130} \\ \end{array}
$$
Plugging (129) and (130) into (128) gives
$$
\left\| h _ {U} - h _ {U ^ {\prime}} \right\| _ {L ^ {2} (P)} ^ {2} \leqslant \frac {4 \left\| U - U ^ {\prime} \right\| _ {F} ^ {2}}{d}
$$
Lower Bound. Define events
$$
E _ {1} = \left\{\mathbf {x}: \pi_ {A _ {0}} \circ f _ {U} (\mathbf {x}) = f _ {U} (\mathbf {x}) \right\}
$$
$$
E _ {2} = \left\{\mathbf {x}: \pi_ {A _ {0}} \circ f _ {U ^ {\prime}} (\mathbf {x}) = f _ {U ^ {\prime}} (\mathbf {x}) \right\}.
$$
$$
\begin{array}{l} \left\| h _ {U} - h _ {U ^ {\prime}} \right\| _ {L ^ {2} (P)} ^ {2} \geqslant \mathbb {E} \left[ 1 _ {[ E _ {1} \cap E _ {2} ]} \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] - \mathbb {E} \left[ 1 _ {\left(E _ {1} \cap E _ {2}\right) ^ {c}} \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] \\ \geqslant \mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] - \sqrt {\left(1 - \mathbb {P} \left(E _ {1} \cap E _ {2}\right)\right) \mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {4} \right]} \tag {131} \\ \end{array}
$$
where the last step uses the Cauchy-Schwartz inequality:  $(\mathbb{E}[YZ])^2 \leqslant \mathbb{E}Y^2\mathbb{E}Z^2$ . We next bound the two terms of the right hand side separately.
For the second term, we observe that