SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Phi := \min  _ {w _ {| d - k} \in \mathbb {R} ^ {d - k}:} \| w _ {| d - k} \| _ {2} = \min  _ {w _ {| d - k} \in \mathbb {R} ^ {d - k}:} \| w _ {| d - k} \| _ {2}. \\ \forall i, \langle w _ {| d - k}, x _ {i | d - k} \rangle = y _ {i} - \langle w _ {| k} ^ {*}, x _ {i | k} \rangle \\ \forall i, \langle w _ {| d - k}, \Sigma_ {| d - k} ^ {1 / 2} z _ {i} \rangle = \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i} | k \rangle}{h (x _ {i} | k)} \\ \end{array}
$$
By introducing the Lagrangian, we have
$$
\begin{array}{l} \Phi = \min _ {w _ {| d - k} \in \mathbb {R} ^ {d - k}} \max _ {\lambda \in \mathbb {R} ^ {n}} \sum_ {i = 1} ^ {n} \lambda_ {i} \left(\langle \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k}, z _ {i} \rangle - \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) + \| w _ {| d - k} \| _ {2} \\ = \min  _ {w _ {| d - k} \in \mathbb {R} ^ {d - k}} \max  _ {\lambda \in \mathbb {R} ^ {n}} \langle \lambda , Z \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k} \rangle - \sum_ {i = 1} ^ {n} \lambda_ {i} \left(\frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) + \| w _ {| d - k} \| _ {2}. \\ \end{array}
$$
Similarly, the above is only random in  $Z$  after conditioning on  $X_{|k}w_{|k}^{*}$  and  $\xi$  and the distribution of  $Z$  remains unchanged after conditioning because of the independence. By the same truncation argument as before and CGMT, it suffices to consider the auxiliary problem:
$$
\begin{array}{l} \min  _ {w _ {| d - k} \in \mathbb {R} ^ {d - k}} \max  _ {\lambda \in \mathbb {R} ^ {n}} \| \lambda \| _ {2} \langle H, \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k} \rangle + \sum_ {i = 1} ^ {n} \lambda_ {i} \left(\| \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k} \| _ {2} G _ {i} - \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) \\ + \| w _ {| d - k} \| _ {2} \\ = \min  _ {w _ {| d - k} \in \mathbb {R} ^ {d - k}} \max  _ {\lambda \in \mathbb {R} ^ {n}} \| \lambda \| _ {2} \left(\langle H, \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k} \rangle + \sqrt {\sum_ {i = 1} ^ {n} \left(\| \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k} \| _ {2} G _ {i} - \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) ^ {2}}\right) \\ + \| w _ {| d - k} \| _ {2} \\ \end{array}
$$
and so we can define
$$
\Psi := \quad \min  _ {d - k} \quad \| w _ {| d - k} \| _ {2}.
$$
$$
\begin{array}{c} w _ {| d - k} \in \mathbb {R} ^ {d - k}: \\ \sqrt {\sum_ {i = 1} ^ {n} \left(\| \Sigma_ {| d - k} ^ {1 / 2} w _ {| d - k} \| _ {2} G _ {i} - \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) ^ {2}} \leq \langle - \Sigma_ {| d - k} ^ {1 / 2} H, w _ {| d - k} \rangle \end{array}
$$
To upper bound  $\Psi$ , we consider  $w_{|d - k}$  of the form  $-\alpha \frac{\Sigma_{|d - k}^{1/2} H}{\|\Sigma_{|d - k}^{1/2} H\|_2}$ , then we just need
$$
\sum_ {i = 1} ^ {n} \left(\alpha \frac {\| \Sigma_ {| d - k} H \| _ {2}}{\| \Sigma_ {| d - k} ^ {1 / 2} H \| _ {2}} G _ {i} - \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) ^ {2} \leq \alpha^ {2} \| \Sigma_ {| d - k} ^ {1 / 2} H \| _ {2} ^ {2}.
$$
By a union bound, the following occurs together with probability at least  $1 - \delta / 2$  for some absolute constant  $C > 0$ :
1. Using the first part of Lemma 4, we have
$$
\left\| \Sigma_ {| d - k} ^ {1 / 2} H \right\| _ {2} ^ {2} \geq \operatorname {T r} \left(\Sigma_ {| d - k}\right) \left(1 - C \frac {\log (3 2 / \delta)}{\sqrt {R \left(\Sigma_ {| d - k}\right)}}\right)
$$
2. Using the last part of Lemma 4, requiring  $R(\Sigma_{|d - k}) \gtrsim \log (32 / \delta)^{2}$
$$
\frac {\| \Sigma_ {| d - k} H \| _ {2} ^ {2}}{\| \Sigma_ {| d - k} ^ {1 / 2} H \| _ {2} ^ {2}} \leq C \log (3 2 / \delta) \frac {\mathrm {T r} (\Sigma_ {| d - k} ^ {2})}{\mathrm {T r} (\Sigma_ {| d - k})}
$$
3. Using subexponential Bernstein's inequality (Theorem 2.8.1 of Vershynin (2018)), requiring  $n = \Omega (\log (1 / \delta))$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} G _ {i} ^ {2} \leq 2
$$
4. Using standard Gaussian tail bound  $\operatorname*{Pr}(|Z|\geq t)\leq 2e^{-t^2 /2}$ , we have
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {G _ {i} \left(y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle\right)}{h \left(x _ {i | k}\right)} \right| \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h \left(x _ {i | k}\right)}\right) ^ {2}} \sqrt {\frac {2 \log (3 2 / \delta)}{n}}
$$
5. By assumption, it holds that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) ^ {2} \leq (1 + \rho) \cdot \mathbb {E} \left[ \left(\frac {y - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {| k})}\right) ^ {2} \right].
$$
Then we use the above and the AM-GM inequality to show that
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\alpha \frac {\| \Sigma_ {| d - k} H \| _ {2}}{\| \Sigma_ {| d - k} ^ {1 / 2} H \| _ {2}} G _ {i} - \frac {y _ {i} - \langle w _ {| k} ^ {*} , x _ {i | k} \rangle}{h (x _ {i | k})}\right) ^ {2} \\ \leq 2 \alpha^ {2} \frac {\| \Sigma_ {| d - k} H \| _ {2} ^ {2}}{\| \Sigma_ {| d - k} ^ {1 / 2} H \| _ {2} ^ {2}} + (1 + \rho) \cdot \mathbb {E} \left[ \left(\frac {y - \langle w _ {| k} ^ {*} , x _ {| k} \rangle}{h (x _ {| k})}\right) ^ {2} \right] \\ + 2 \frac {\alpha \| \Sigma_ {| d - k} H \| _ {2}}{\| \Sigma_ {| d - k} ^ {1 / 2} H \| _ {2}} \sqrt {(1 + \rho) \cdot \mathbb {E} \left[ \left(\frac {y - \langle w _ {| k} ^ {*} , x _ {| k} \rangle}{h (x _ {| k})}\right) ^ {2} \right]} \sqrt {\frac {2 \log (3 2 / \delta)}{n}} \\ \leq C \log (3 2 / \delta) \left(2 + \sqrt {\frac {2 \log (3 2 / \delta)}{n}}\right) \alpha^ {2} \frac {\operatorname {T r} (\Sigma_ {| d - k} ^ {2})}{\operatorname {T r} (\Sigma_ {| d - k})} \\ + \left(1 + \sqrt {\frac {2 \log (3 2 / \delta)}{n}}\right) (1 + \rho) \cdot \mathbb {E} \left[ \left(\frac {y - \langle w _ {| k} ^ {*} , x _ {| k} \rangle}{h (x _ {| k})}\right) ^ {2} \right]. \\ \end{array}
$$
After some rearrangements, it is easy to see that we can choose
$$
\alpha^ {2} = \frac {\left(1 + \sqrt {\frac {2 \log (3 2 / \delta)}{n}}\right) (1 + \rho)}{1 - C \frac {\log (3 2 / \delta)}{\sqrt {R (\Sigma_ {| d - k})}} - C \log (3 2 / \delta) \left(2 + \sqrt {\frac {2 \log (3 2 / \delta)}{n}}\right) \frac {n}{R (\Sigma_ {| d - k})}} \frac {n \mathbb {E} \left[ \left(\frac {y - \langle w _ {| k} ^ {*} , x _ {| k} \rangle}{h (x _ {| k})}\right) ^ {2} \right]}{\operatorname {T r} (\Sigma_ {| d - k})}.
$$