SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ (f _ {U} (X) - f _ {U ^ {\prime}} (X)) ^ {4} \right] = \frac {1}{d ^ {4}} \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {4} \left(\sum_ {i = 1} ^ {d} (X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d + 2 i} ^ {2})\right) ^ {4} \right] \\ = \frac {1}{d ^ {4}} \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {4} \right] \mathbb {E} \left[ \left(\sum_ {i = 1} ^ {d} (X _ {2 d + 2 i - 1} ^ {2} - X _ {2 d} ^ {2})\right) ^ {4} \right] \\ \lesssim \frac {1}{d ^ {2}} \mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {4} \right] \tag {132} \\ \end{array}
$$
where the last step follows from (124). Note that conditioned on  $X_{d + 1:2d}$
$$
X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d} \sim \mathcal {N} (0, \| (U - U ^ {\prime}) X _ {d + 1: 2 d} \| _ {2} ^ {2}).
$$
Combining with the fact that  $EY^4 = 3\sigma^2$  for  $Y \sim \mathcal{N}(0, \sigma^2)$ , we have
$$
\mathbb {E} \left[ \left(X _ {1: d} ^ {\top} (U - U ^ {\prime}) X _ {d + 1: 2 d}\right) ^ {4} \right] = 3 \mathbb {E} \left[ \| (U - U ^ {\prime}) X _ {d + 1: 2 d} \| _ {2} ^ {4} \right]
$$
Let  $B = U - U^{\prime}$ $Z\sim N(0,I_d)$  , and  $B^{\top}B = D$  . Then,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| (U - U ^ {\prime}) X _ {d + 1: 2 d} \right\| _ {2} ^ {4} \right] = \mathbb {E} \left[ (Z ^ {\top} D Z) ^ {2} \right] \\ = \mathbb {E} \left[ \left(\sum_ {i, j} D _ {i, j} Z _ {i} Z _ {j}\right) ^ {2} \right] \\ = \sum_ {i, j} D _ {i, j} ^ {2} \mathbb {E} \left[ Z _ {i} ^ {2} Z _ {j} ^ {2} \right] \\ \lesssim \| D \| _ {F} ^ {2} \leqslant \| B \| _ {F} ^ {4} = \| U - U ^ {\prime} \| _ {F} ^ {4} \tag {133} \\ \end{array}
$$
Plugging (133) into (132) gives
$$
\mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {4} \right] \leqslant c _ {4} \frac {\| U - U ^ {\prime} \| _ {F} ^ {4}}{d ^ {2}} \tag {134}
$$
for some absolute constant  $c_4 > 0$ . By Lemma I.1, there is an absolute constant  $C$  such that when  $A_0 \geqslant C$ , we have
$$
1 - \mathbb {P} \left(E _ {1} \cap E _ {2}\right) \leqslant \frac {1}{c _ {4}} \tag {135}
$$
Continuing the equation (131), we have
$$
\begin{array}{l} \left\| h _ {U} - h _ {U ^ {\prime}} \right\| _ {L ^ {2} (P)} ^ {2} \geqslant \mathbb {E} \left[ (f _ {U} (X) - f _ {U ^ {\prime}} (X)) ^ {2} \right] - \sqrt {(1 - \mathbb {P} (E _ {1} \cap E _ {2}))} \mathbb {E} \left[ (f _ {U} (X) - f _ {U ^ {\prime}} (X)) ^ {4} \right] \\ \geqslant \mathbb {E} \left[ (f _ {U} (X) - f _ {U ^ {\prime}} (X)) ^ {2} \right] - \sqrt {\frac {1}{c _ {4}} \cdot c _ {4} \frac {\| U - U ^ {\prime} \| _ {F} ^ {4}}{d ^ {2}}} \\ = \frac {4}{d} \| U - U ^ {\prime} \| _ {F} ^ {2} - \frac {\| U - U ^ {\prime} \| _ {F} ^ {2}}{d} \\ = \frac {3}{d} \| U - U ^ {\prime} \| _ {F} ^ {2}, \tag {136} \\ \end{array}
$$
where in the third step we use the result from the above upper bound:
$$
\mathbb {E} \left[ \left(f _ {U} (X) - f _ {U ^ {\prime}} (X)\right) ^ {2} \right] = \frac {4 \| U - U ^ {\prime} \| _ {F} ^ {2}}{d}.
$$