SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof concludes by applying Lemma 6 and the tower law.
The following two simple lemmas will be useful to analyze the auxiliary problem.
Lemma 8. For  $a, b, H > 0$ , we have
$$
\sup_{\lambda \geq 0} - \lambda a + \frac{\lambda}{H + \lambda} b = (\sqrt{b} -\sqrt{Ha})_{+}^{2}.
$$
$$
\sup  _ {\lambda \geq 0} - \lambda a + \frac {\lambda}{H + \lambda} b = b - \inf  _ {\lambda \geq 0} \lambda a + \frac {H}{H + \lambda} b.
$$
Define  $f(\lambda) = \lambda a + \frac{H}{H + \lambda} b$ , then
$$
\begin{array}{l} f ^ {\prime} (\lambda) = a - \frac {H b}{(H + \lambda) ^ {2}} \leq 0 \Longleftrightarrow (H + \lambda) ^ {2} \leq \frac {H b}{a} \\ \Longleftrightarrow - \sqrt {\frac {H b}{a}} - H \leq \lambda \leq \sqrt {\frac {H b}{a}} - H \\ \end{array}
$$
Since we require  $\lambda \geq 0$ , we only need to consider whether  $\sqrt{\frac{Hb}{a}} - H \geq 0 \iff b \geq Ha$ . If  $b < Ha$ , the infimum is attained at  $\lambda = 0$ . Otherwise, the infimum is attained at  $\lambda^{*} = \sqrt{\frac{Hb}{a}} - H$ , at which point
$$
f \left(\lambda^ {*}\right) = 2 \sqrt {H b a} - H a.
$$
Plugging in, we see that the expression is equivalent to  $(\sqrt{b} - \sqrt{Ha})_{+}^{2}$  in both cases.
Lemma 9. For  $a, b \geq 0$ , we have
$$
\sup  _ {\lambda \geq 0} - \lambda a - \frac {b}{\lambda} = - \sqrt {4 a b}
$$
Proof. Define  $f(\lambda) = -\lambda a - \frac{b}{\lambda}$ , then
$$
f ^ {\prime} (\lambda) = - a + \frac {b}{\lambda^ {2}} \geq 0 \Longleftrightarrow \frac {b}{a} \geq \lambda^ {2}
$$
and so in the domain  $\lambda \geq 0$ , the optimum is attained at  $\lambda^{*} = \sqrt{b / a}$  at which point  $f(\lambda^{*}) = -2\sqrt{ab}$ .
We are now ready to analyze the auxiliary problem.
Lemma 10. In the same setting as in Lemma 6, assume that for every  $\delta > 0$
(A)  $C_{\delta}:\mathbb{R}^{d}\to [0,\infty ]$  is a continuous function such that with probability at least  $1 - \delta /4$  over  $H\sim \mathcal{N}(0,I_d)$ , uniformly over all  $w\in \mathbb{R}^d$ , we have that
$$
\langle \Sigma^ {1 / 2} P H, w \rangle \leq C _ {\delta} (w) \tag {63}
$$
(B)  $\epsilon_{\delta}$  is a positive real number such that with probability at least  $1 - \delta /4$  over  $\{(\tilde{x}_i,\tilde{y}_i)\}_{i = 1}^n$  drawn i.i.d. from  $\tilde{D}$ , it holds uniformly over all  $\theta \in \Theta$  that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} f (\langle \phi (w (\theta)), \tilde {x} _ {i} \rangle , \tilde {y} _ {i}, \theta) \geq \frac {1}{1 + \epsilon_ {\delta}} \mathbb {E} _ {(\tilde {x}, \tilde {y}) \sim \tilde {D}} [ f (\langle \phi (w (\theta)), \tilde {x} \rangle , \tilde {y}, \theta) ]. \tag {64}
$$
where the distribution  $\tilde{D}$  over  $(\tilde{x},\tilde{y})$  is given by
$$
\tilde {x} \sim \mathcal {N} (0, I _ {k + 1}), \quad \tilde {\xi} \sim \mathcal {D} _ {\xi}, \quad \tilde {y} = g (\tilde {x} _ {1}, \dots , \tilde {x} _ {k}, \tilde {\xi})
$$
and the mapping  $\phi :\mathbb{R}^d\to \mathbb{R}^{k + 1}$  is defined as
$$
\phi (w) = \left(\langle w, \Sigma w _ {1} ^ {*} \rangle , \dots , \langle w, \Sigma w _ {k} ^ {*} \rangle , \| P \Sigma^ {1 / 2} w \| _ {2}\right) ^ {T}.
$$
Then the following is true:
(i) suppose for some choice of  $M_{\theta}$  that is continuous in  $\theta$ , it holds for every  $y \in \mathcal{V}$  and  $\theta \in \Theta$ ,  $f$  is  $M_{\theta}$ -Lipschitz with respect to the first argument, then with probability at least  $1 - \delta$ , uniformly over all  $\theta \in \Theta$ , we have
$$
L (\theta) \leq \left(1 + \epsilon_ {\delta}\right) \left(\hat {L} (\theta) + M _ {\theta} \sqrt {\frac {C _ {\delta} (w (\theta)) ^ {2}}{n}}\right). \tag {65}
$$
(ii) suppose for some choice of  $H_{\theta}$  that is continuous in  $\theta$ , it holds for every  $y \in \mathcal{V}$  and  $\theta \in \Theta$ ,  $f$  is non-negative and  $\sqrt{f}$  is  $\sqrt{H_{\theta}}$ -Lipschitz with respect to the first argument, then with probability at least  $1 - \delta$ , uniformly over all  $\theta \in \Theta$ , we have
$$
L (\theta) \leq (1 + \epsilon_ {\delta}) \left(\sqrt {\hat {L} (\theta)} + \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}}\right) ^ {2}. \tag {66}
$$
Proof. First, let's simplify the auxiliary problem (59). Changing variables to subtract the quantity  $G_{i}\left\| P\Sigma^{1 / 2}w(\theta)\right\|_{2} + \sum_{l = 1}^{k}\langle w(\theta),\Sigma w_{l}^{*}\rangle \eta_{l,i}$  from each of the former  $u_{i}$ , we have that
$$
\Psi = \sup_{\substack{(u,\theta)\in \mathbb{R}^{n}\times \Theta \\ \| u\|_{2}\leq \langle H,P\Sigma^{1 / 2}w(\theta)\rangle}}F(\theta) - \frac{1}{n}\sum_{i = 1}^{n}f\left(u_{i} + G_{i}\left\| P\Sigma^{1 / 2}w(\theta)\right\|_{2} + \sum_{l = 1}^{k}\langle w(\theta),\Sigma w_{l}^{*}\rangle \eta_{l,i},y_{i},\theta\right)
$$
and separating the optimization problem in  $u$  and  $\theta$ , we obtain
$$
\begin{array}{l} \Psi = \sup_{\theta \in \Theta}F(\theta) \\ -\frac{1}{n}\inf_{\substack{u\in \mathbb{R}^{n}:\\ \| u\|_{2}\leq \langle H,P\Sigma^{1 / 2}w(\theta)\rangle}}\sum_{i = 1}^{n}f\left(u_{i} + G_{i}\left\| P\Sigma^{1 / 2}w(\theta)\right\|_{2} + \sum_{l = 1}^{k}\langle w(\theta),\Sigma w_{l}^{*}\rangle \eta_{l,i},y_{i},\theta\right). \\ \end{array}
$$
Next, we will lower bound the infimum term by weak duality to obtain upper bound on  $\Psi$ :
$$
\begin{array}{l} \inf_{\substack{u\in \mathbb{R}^{n}:\\ \| u\|_{2}\leq \langle H,P\Sigma^{1 / 2}w(\theta)\rangle}}\sum_{i = 1}^{n}f\left(u_{i} + G_{i}\left\| P\Sigma^{1 / 2}w(\theta)\right\|_{2} + \sum_{l = 1}^{k}\langle w(\theta),\Sigma w_{l}^{*}\rangle \eta_{l,i},y_{i},\theta\right) \\ = \inf  _ {u \in \mathbb {R} ^ {n}} \sup  _ {\lambda \geq 0} \lambda \left(\| u \| _ {2} ^ {2} - \langle \Sigma^ {1 / 2} P H, w (\theta) \rangle^ {2}\right) \\ + \sum_ {i = 1} ^ {n} f \left(u _ {i} + G _ {i} \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} + \sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i}, y _ {i}, \theta\right) \\ \end{array}
$$
$$
\begin{array}{l} \geq \sup_{\lambda \geq 0} - \lambda \langle \Sigma^{1 / 2}PH,w(\theta)\rangle^{2} \\ + \inf  _ {u \in \mathbb {R} ^ {n}} \sum_ {i = 1} ^ {n} f \left(u _ {i} + G _ {i} \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} + \sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i}, y _ {i}, \theta\right) + \lambda \| u \| _ {2} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} = \sup_{\lambda \geq 0} - \lambda \langle \Sigma^{1 / 2}PH,w(\theta)\rangle^{2} \\ + \sum_ {i = 1} ^ {n} \inf  _ {u _ {i} \in \mathbb {R}} f \left(u _ {i} + G _ {i} \| P \Sigma^ {1 / 2} w (\theta) \| _ {2} + \sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i}, y _ {i}, \theta\right) + \lambda u _ {i} ^ {2} \\ \end{array}
$$
$$
= \sup  _ {\lambda \geq 0} - \lambda \left\langle \Sigma^ {1 / 2} P H, w (\theta) \right\rangle^ {2} + \sum_ {i = 1} ^ {n} f _ {\lambda} \left(G _ {i} \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} + \sum_ {l = 1} ^ {k} \left\langle w (\theta), \Sigma w _ {l} ^ {*} \right\rangle \eta_ {l, i}, y _ {i}, \theta\right).
$$
Suppose that for every  $y \in \mathcal{V}$  and  $\theta \in \Theta$ ,  $f$  is  $M_{\theta}$ -Lipschitz with respect to the first argument, then by Proposition 2, the above can be further lower bounded by the following quantity:
$$
\sup _ {\lambda \geq 0} - \lambda \langle \Sigma^ {1 / 2} P H, w (\theta) \rangle^ {2} - \frac {n M _ {\theta} ^ {2}}{4 \lambda} + \sum_ {i = 1} ^ {n} f \left(\sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i} + \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} G _ {i}, y _ {i}, \theta\right).
$$
On the other hand, suppose that for every  $y \in \mathcal{V}$  and  $\theta \in \Theta$ ,  $f$  is non-negative and  $\sqrt{f}$  is  $\sqrt{H_{\theta}}$ . Lipschitz with respect to the first argument, then by Proposition 1, the above can be further lower bounded by:
$$
\sup _ {\lambda \geq 0} - \lambda \langle \Sigma^ {1 / 2} P H, w (\theta) \rangle^ {2} + \frac {\lambda}{H _ {\theta} + \lambda} \left[ \sum_ {i = 1} ^ {n} f \left(\sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i} + \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} G _ {i}, y _ {i}, \theta\right) \right].
$$
Notice that if we write  $\tilde{x}_i = (\eta_{1,i},\dots,\eta_{k,i},G_i)$ , then  $(\tilde{x}_i,y_i)$  are independent with distribution exactly equal to  $\tilde{\mathcal{D}}$ . Moreover, we have
$$
f \left(\sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i} + \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} G _ {i}, y _ {i}, \theta\right) = f (\langle \phi (w (\theta)), \tilde {x} _ {i} \rangle , y _ {i}, \theta)
$$
and it is easy to see that the joint distribution of  $(\langle \phi (w(\theta)),\tilde{x}\rangle ,y)$  with  $(\tilde{x},y)\sim \hat{\mathcal{D}}$  is exactly the same as  $(\langle w(\theta),x\rangle ,y)$  with  $(x,y)\sim \mathcal{D}$ . As a result, we have that
$$
\mathbb {E} _ {(\tilde {x}, y) \sim \tilde {D}} [ f (\langle \phi (w (\theta)), \tilde {x} \rangle , y, \theta) ] = L (\theta).
$$
By our assumption (63), (64) and a union bound, we have with probability at least  $1 - \delta /2$
$$
| \langle \Sigma^ {1 / 2} P H, w (\theta) \rangle | \leq C _ {\delta} (w (\theta))
$$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\sum_ {l = 1} ^ {k} \langle w (\theta), \Sigma w _ {l} ^ {*} \rangle \eta_ {l, i} + \left\| P \Sigma^ {1 / 2} w (\theta) \right\| _ {2} G _ {i}, y _ {i}, \theta\right) \geq \frac {1}{1 + \epsilon_ {\delta}} L (\theta).
$$
Therefore, if  $f$  is  $M_{\theta}$ -Lipschitz, then by Lemma 9, we have
$$
\begin{array}{l} \Psi \leq \sup  _ {\theta \in \Theta} F (\theta) - \sup  _ {\lambda \geq 0} - \lambda \frac {C _ {\delta} (w (\theta)) ^ {2}}{n} - \frac {M _ {\theta} ^ {2}}{4 \lambda} + \frac {1}{1 + \epsilon_ {\delta}} L (\theta) \\ = \sup _ {\theta \in \Theta} F (\theta) + \sqrt {M _ {\theta} ^ {2} \frac {C _ {\delta} (w (\theta)) ^ {2}}{n}} - \frac {1}{1 + \epsilon_ {\delta}} L (\theta) \\ \end{array}
$$
Consequently, by taking  $F(\theta) = \frac{1}{1 + \epsilon_{\delta}} L(\theta) - M_{\theta}\sqrt{\frac{C_{\delta}(w(\theta))^{2}}{n}}$  and Lemma 7, we have shown that with probability at least  $1 - \delta$ , we have
$$
\sup _ {\theta \in \mathcal {K}} F (\theta) - \hat {L} (\theta) \leq 0 \Rightarrow \frac {1}{1 + \epsilon_ {\delta}} L (\theta) \leq \hat {L} (\theta) + M _ {\theta} \sqrt {\frac {C _ {\delta} (w (\theta)) ^ {2}}{n}}.
$$
If  $\sqrt{f}$  is  $\sqrt{H_{\theta}}$ -Lipschitz, then by Lemma 8
$$
\begin{array}{l} \Psi \leq \sup  _ {\theta \in \mathcal {K}} F (\theta) - \sup  _ {\lambda \geq 0} - \lambda \frac {C _ {\delta} (w (\theta)) ^ {2}}{n} + \frac {\lambda}{H _ {\theta} + \lambda} \frac {1}{1 + \epsilon_ {\delta}} L (\theta) \\ = \sup  _ {\theta \in \mathcal {K}} F (\theta) - \left(\sqrt {\frac {L (\theta)}{1 + \epsilon_ {\delta}}} - \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}}\right) _ {+} ^ {2}. \\ \end{array}
$$
Consequently, by taking  $F(\theta) = \left(\sqrt{\frac{L(\theta)}{1 + \epsilon_{\delta}}} - \sqrt{\frac{H_{\theta}C_{\delta}(w(\theta))^{2}}{n}}\right)_{+}^{2}$  and Lemma 7, we have shown that with probability at least  $1 - \delta$ , we have
$$
\sup  _ {\theta \in \mathcal {K}} F (\theta) - \hat {L} (\theta) \leq 0.
$$
Rearranging, either we have
$$
\sqrt {\frac {L (\theta)}{1 + \epsilon_ {\delta}}} - \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}} <   0 \Rightarrow L (\theta) <   (1 + \epsilon_ {\delta}) \frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}
$$
or we have
$$
\begin{array}{l} \sqrt {\frac {L (\theta)}{1 + \epsilon_ {\delta}}} - \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}} \geq 0 \Rightarrow \left(\sqrt {\frac {L (\theta)}{1 + \epsilon_ {\delta}}} - \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}}\right) ^ {2} \leq \hat {L} (\theta) \\ \Rightarrow L (\theta) \leq (1 + \epsilon_ {\delta}) \left(\sqrt {\hat {L} (\theta)} + \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}}\right) ^ {2}. \\ \end{array}
$$
In either case, the desired bound holds.
Finally, we are ready to prove Theorem 6. In the version below, we also provide uniform convergence guarantee (with sharp constant) for Lipschitz loss.
Theorem 14. Suppose that assumptions (A), (B), (E) and (F) hold. For any  $\delta \in (0,1)$ , let  $C_{\delta}:\mathbb{R}^{d}\to [0,\infty ]$  be a continuous function such that with probability at least  $1 - \delta /4$  over  $x\sim \mathcal{N}(0,\Sigma)$  uniformly over all  $\theta \in \Theta$ ,
$$
\left\langle w (\theta), Q ^ {T} x \right\rangle \leq C _ {\delta} (w (\theta)). \tag {67}
$$
(i) if for each  $\theta \in \Theta$  and  $y \in \mathcal{V}$ ,  $f$  is  $M_{\theta}$ -Lipschitz with respect to the first argument and  $M_{\theta}$  is continuous in  $\theta$ , then with probability at least  $1 - \delta$ , it holds that uniformly over all  $\theta \in \Theta$ , we have
$$
(1 - \epsilon) L (\theta) \leq \hat {L} (\theta) + M _ {\theta} \sqrt {\frac {C _ {\delta} (w (\theta)) ^ {2}}{n}} \tag {68}
$$
(ii) if for each  $\theta \in \Theta$  and  $y \in \mathcal{Y}$ ,  $f$  is non-negative and  $\sqrt{f}$  is  $\sqrt{H_{\theta}}$ -Lipschitz with respect to the first argument, and  $H_{\theta}$  is continuous in  $\theta$ , then with probability at least  $1 - \delta$ , it holds that uniformly over all  $\theta \in \Theta$ , we have
$$
(1 - \epsilon) L (\theta) \leq \left(\sqrt {\hat {L} (\theta)} + \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}}\right) ^ {2} \tag {69}
$$
Proof. We apply the reduction argument at the beginning of the appendix. Given  $\mathcal{D}$  that satisfies assumptions (A) and (B), we define  $[\tilde{w}_1^*,\dots,\tilde{w}_k^*] = \tilde{W} = W(W^T\Sigma W)^{-1 / 2}$  and  $\tilde{f},\tilde{g}$  as in (54) and (55). For  $\{(x_i,y_i)\}_{i = 1}^n$  sampled independently from  $\mathcal{D}$ , we observe that the joint distribution of  $(x_{i} - \mu ,y_{i})$  can also be described by  $\mathcal{D}'$  as follows:
(A')  $x \sim \mathcal{N}(0, \Sigma)$
(B')  $y = \tilde{g}(\eta_1, \dots, \eta_k, \xi)$  where  $\eta_i = \langle x, \tilde{w}_i \rangle$ .
Indeed, we can check that
$$
\begin{array}{l} y = g \left(x ^ {T} W, \xi\right) \\ = g \left(\left(x - \mu\right) ^ {T} \tilde {W} \left(W ^ {T} \Sigma W\right) ^ {1 / 2} + \mu^ {T} W, \xi\right) \\ = \tilde {g} ((x - \mu) ^ {T} \bar {W}, \xi). \\ \end{array}
$$
Moreover, by construction, we have
$$
\hat {L} (\theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \tilde {f} (\langle w (\theta), x _ {i} - \mu \rangle , y _ {i}, \theta)
$$
$$
L (\theta) = \mathbb {E} _ {\mathcal {D} ^ {\prime}} \tilde {f} (\langle w (\theta), x _ {i} \rangle , y _ {i}, \theta)
$$
and  $\mathcal{D}'$  satisfies assumptions (A) and (B) with  $\mu = 0$  and orthonormal  $\Sigma^{1/2}\tilde{w}_1^*$ , ...,  $\Sigma^{1/2}\tilde{w}_1^*$  and falls into the setting in Lemma 6. We see that  $f$  being Lipschitz or square-root Lipschitz is equivalent to
$\tilde{f}$  being Lipschitz or square-root Lipschitz. It remains to check assumptions (63) and (64) and then apply Lemma 10. Observe that
$$
\begin{array}{l} \Sigma^ {- 1 / 2} P \Sigma^ {1 / 2} = \Sigma^ {- 1 / 2} \left(I _ {d} - \Sigma^ {1 / 2} \tilde {W} \tilde {W} ^ {T} \Sigma^ {1 / 2}\right) \Sigma^ {1 / 2} \\ = I _ {d} - \tilde {W} \tilde {W} ^ {T} \Sigma = I - W \left(W ^ {T} \Sigma W\right) ^ {- 1} W ^ {T} \Sigma \tag {70} \\ = Q \\ \end{array}
$$
and so  $\Sigma^{1 / 2}P = Q^T\Sigma^{1 / 2}$
To check that (63) holds, observe that  $\langle \Sigma^{1/2} PH, w \rangle$  has the same distribution as  $\langle Qw, x \rangle$ . To check that (64) holds, we will apply Theorem 13. Note that the joint distribution of  $(\langle \phi(w(\theta)), \tilde{x} \rangle, \tilde{y})$  with  $(\tilde{x}, \tilde{y}) \sim \tilde{\mathcal{D}}$  is exactly the same as  $(\langle w(\theta), x \rangle, y)$  with  $(x, y) \sim \mathcal{D}'$  and so
$$
\frac {\mathbb {E} _ {\tilde {\mathcal {D}}} [ \tilde {f} (\langle \phi (w (\theta)) , x \rangle , y , \theta) ^ {4} ] ^ {1 / 4}}{\mathbb {E} _ {\tilde {\mathcal {D}}} [ \tilde {f} (\langle \phi (w (\theta)) , x \rangle , y , \theta) ]} = \frac {\mathbb {E} _ {\mathcal {D} ^ {\prime}} [ \tilde {f} (\langle w (\theta) , x \rangle , y , \theta) ^ {4} ] ^ {1 / 4}}{\mathbb {E} _ {\mathcal {D} ^ {\prime}} [ \tilde {f} (\langle w (\theta) , x \rangle , y , \theta) ]} = \frac {\mathbb {E} _ {\mathcal {D}} [ f (\langle w (\theta) , x \rangle , y , \theta) ^ {4} ] ^ {1 / 4}}{\mathbb {E} _ {\mathcal {D}} [ f (\langle w (\theta) , x \rangle , y , \theta) ]}.
$$
Therefore, the assumption (E) is equivalent to the hypercontractivity condition in Theorem 13. Note that  $\{(x,y)\mapsto \mathbb{1}\{\tilde{f} (\langle \phi (w(\theta)),x\rangle ,y,\theta) > t\} :(\theta ,t)\in \Theta \times \mathbb{R}\}$  is a subclass of  $\{(x,y)\mapsto$ $\mathbb{1}\{f(\langle w,x\rangle +b,y,\theta) > t\} :(w,b,t,\theta)\in \mathbb{R}^{k + 1}\times \mathbb{R}\times \mathbb{R}\times \Theta \}$ . Therefore, by assumption (F), we can apply Theorem 13 and (64) holds.
D Norm Bounds
The following lemma is a version of Lemma 7 of Koehler et al. (2021) and follows straightforwardly from CGMT (Theorem 7), though it requires a slightly different truncation argument compared to the proof Theorem 6. For simplicity, we won't repeat the proof here and simply use it for our applications.
Lemma 11 (Koehler et al. 2021, Lemma 7). Let  $Z: n \times d$  be a matrix with i.i.d.  $\mathcal{N}(0,1)$  entries and suppose  $G \sim \mathcal{N}(0, I_n)$  and  $H \sim \mathcal{N}(0, I_d)$  are independent of  $Z$  and each other. Fix an arbitrary norm  $\| \cdot \|$ , any covariance matrix  $\Sigma$ , and any non-random vector  $\xi \in \mathbb{R}^n$ , consider the Primary Optimization (PO) problem:
$$
\Phi := \min  _ { \begin{array}{c} w \in \mathbb {R} ^ {d}: \\ Z \Sigma^ {1 / 2} w = \xi \end{array} } \| w \| \tag {71}
$$
and the Auxiliary Optimization (AO) problem:
$$
\Psi := \quad \min  _ {w \in \mathbb {R} ^ {d}:} \| w \|. \tag {72}
$$
$$
\left\| G \right\| \Sigma^ {1 / 2} w \| _ {2} - \xi \| _ {2} \leq \langle \Sigma^ {1 / 2} H, w \rangle
$$
Then for any  $t\in \mathbb{R}$  it holds that
$$
\Pr (\Phi > t) \leq 2 \Pr (\phi \geq t). \tag {73}
$$
The next lemma analyzes the AO in Lemma 11. Our proof closely follows Lemma 8 of Koehler et al. 2021, but we don't make assumptions on  $\xi$  yet to allow more applications.
Lemma 12. Let  $Z: n \times d$  be a matrix with i.i.d.  $\mathcal{N}(0,1)$  entries. Fix any  $\delta > 0$ , covariance matrix  $\Sigma$  and non-random vector  $\xi \in \mathbb{R}^n$ , then there exists  $\epsilon \lesssim \log(1/\delta) \left( \frac{1}{n} + \frac{1}{\sqrt{R(\Sigma)}} + \frac{n}{R(\Sigma)} \right)$  such that with probability at least  $1 - \delta$ , it holds that
$$
\min  _ { \begin{array}{c} w \in \mathbb {R} ^ {d}: \\ Z \Sigma^ {1 / 2} w = \xi \end{array} } \| w \| _ {2} ^ {2} \leq (1 + \epsilon) \frac {\| \xi \| _ {2} ^ {2}}{\operatorname {T r} (\Sigma)}. \tag {74}
$$
Proof. By a union bound, there exists a constant  $C > 0$  such that the following events occur together with probability at least  $1 - \delta / 2$ :
1. Since  $\langle G,\xi \rangle \sim \mathcal{N}(0,\| \xi \| _2^2)$ , by the standard Gaussian tail bound  $\operatorname *{Pr}(|Z|\geq t)\leq 2e^{-t^2 /2}$  we have
$$
| \langle G, \xi \rangle | \leq \| \xi \| _ {2} \sqrt {2 \log (3 2 / \delta)}
$$
2. Using subexponential Bernstein's inequality (Theorem 2.8.1 of Vershynin (2018)), requiring  $n = \Omega (\log (1 / \delta))$ , we have
$$
\| G \| _ {2} ^ {2} \leq 2 n
$$
3. Using the first part of Lemma 4, we have
$$
\| \Sigma^ {1 / 2} H \| _ {2} ^ {2} \geq \operatorname {T r} (\Sigma) \left(1 - C \frac {\log (3 2 / \delta)}{\sqrt {R (\Sigma)}}\right)
$$
4. Using the last part of Lemma 4, requiring  $R(\Sigma) \gtrsim \log(32 / \delta)^2$
$$
\frac {\| \Sigma H \| _ {2} ^ {2}}{\| \Sigma^ {1 / 2} H \| _ {2} ^ {2}} \leq C \log (3 2 / \delta) \frac {\operatorname {T r} (\Sigma^ {2})}{\operatorname {T r} (\Sigma)}
$$
Therefore, by the AM-GM inequality, it holds that
$$
\begin{array}{l} \| G \| \Sigma^ {1 / 2} w \| _ {2} - \xi \| _ {2} ^ {2} = \| G \| _ {2} ^ {2} \| \Sigma^ {1 / 2} w \| _ {2} ^ {2} + \| \xi \| _ {2} ^ {2} - 2 \langle G, \xi \rangle \| \Sigma^ {1 / 2} w \| _ {2} \\ \leq 2 n \| \Sigma^ {1 / 2} w \| _ {2} ^ {2} + \| \xi \| _ {2} ^ {2} + 2 \| \xi \| _ {2} \sqrt {2 \log (3 2 / \delta)} \| \Sigma^ {1 / 2} w \| _ {2} \\ \leq 3 n \| \Sigma^ {1 / 2} w \| _ {2} ^ {2} + \left(1 + \frac {2 \log (3 2 / \delta)}{n}\right) \| \xi \| _ {2} ^ {2}. \\ \end{array}
$$
To apply lemma 11, we will consider  $w$  of the form  $w = \alpha \frac{\Sigma^{1/2} H}{\|\Sigma^{1/2} H\|_2}$  for some  $\alpha > 0$ . Then we have
$$
\| G \| \Sigma^ {1 / 2} w \| _ {2} - \xi \| _ {2} ^ {2} \leq 3 n C \log (3 2 / \delta) \frac {\operatorname {T r} (\Sigma^ {2})}{\operatorname {T r} (\Sigma)} \alpha^ {2} + \left(1 + \frac {2 \log (3 2 / \delta)}{n}\right) \| \xi \| _ {2} ^ {2}
$$
$$
\langle \Sigma^ {1 / 2} H, w \rangle^ {2} = \alpha^ {2} \| \Sigma^ {1 / 2} H \| _ {2} ^ {2} \geq \alpha^ {2} \operatorname {T r} (\Sigma) \left(1 - C \frac {\log (3 2 / \delta)}{\sqrt {R (\Sigma)}}\right).
$$
So it suffices to choose  $\alpha$  such that
$$
\begin{array}{l} \alpha^ {2} \geq \frac {\left(1 + \frac {2 \log (3 2 / \delta)}{n}\right) \| \xi \| _ {2} ^ {2}}{\operatorname {T r} (\Sigma) \left(1 - C \frac {\log (3 2 / \delta)}{\sqrt {R (\Sigma)}}\right) - 3 n C \log (3 2 / \delta) \frac {\operatorname {T r} (\Sigma^ {2})}{\operatorname {T r} (\Sigma)}} \\ = \frac {1 + \frac {2 \log (3 2 / \delta)}{n}}{1 - C \log (3 2 / \delta) \left(\frac {1}{\sqrt {R (\Sigma)}} + 3 \frac {n}{R (\Sigma)}\right)} \frac {\| \xi \| _ {2} ^ {2}}{\operatorname {T r} (\Sigma)} \\ \end{array}
$$
A challenge for analyzing the minimal norm to interpolate is that the projection matrix  $Q$  is not necessarily an orthogonal projection. However, the following lemma suggests that if  $\Sigma^{\perp} = Q^{T}\Sigma Q$  has high effective rank, then we can let  $R$  be the orthogonal projection matrix onto the image of  $Q$  and  $R\Sigma R$  is approximately the same as  $\Sigma^{\perp}$  in terms of the quantities that are relevant to the norm analysis.
Lemma 13. Consider  $Q = I - \sum_{i=1}^{k} w_i^*(w_i^*)^T\Sigma$  where  $\Sigma^{1/2}w_1^*, \dots, \Sigma^{1/2}w_k^*$  are orthonormal and we let  $R$  be the orthogonal projection matrix onto the image of  $Q$ . Then it holds that  $\operatorname{rank}(R) = d - k$  and
$$
R \Sigma w _ {i} ^ {*} = 0 \quad f o r a n y i = 1, \dots , k.
$$
Moreover, we have  $QR = R$  and  $RQ = Q$ , and so
$$
\begin{array}{l} \frac {1}{\operatorname {T r} (R \Sigma R)} \leq \left(1 - \frac {k}{n} - \frac {n}{R \left(Q ^ {T} \Sigma Q\right)}\right) ^ {- 1} \frac {1}{\operatorname {T r} \left(Q ^ {T} \Sigma Q\right)} \\ \frac {n}{R (R \Sigma R)} \leq \left(1 - \frac {k}{n} - \frac {n}{R (Q ^ {T} \Sigma Q)}\right) ^ {- 2} \frac {n}{R (Q ^ {T} \Sigma Q)}. \\ \end{array}
$$
Proof. It is obvious that  $\mathrm{rank}(R) = \mathrm{rank}(Q)$  and by the rank-nullity theorem, it suffices to show the nullity of  $Q$  is  $k$ . To this end, we observe that
$$
\begin{array}{l} Q w = 0 \iff \Sigma^ {- 1 / 2} \left(I - \sum_ {i = 1} ^ {k} \left(\Sigma^ {1 / 2} w _ {i} ^ {*}\right) \left(\Sigma^ {1 / 2} w _ {i} ^ {*}\right) ^ {T}\right) \Sigma^ {1 / 2} w = 0 \\ \Longleftrightarrow \left(I - \sum_ {i = 1} ^ {k} \left(\Sigma^ {1 / 2} w _ {i} ^ {*}\right) \left(\Sigma^ {1 / 2} w _ {i} ^ {*}\right) ^ {T}\right) \Sigma^ {1 / 2} w = 0 \\ \Longleftrightarrow \Sigma^ {1 / 2} w \in \operatorname {s p a n} \left\{\Sigma^ {1 / 2} w _ {1} ^ {*}, \dots , \Sigma^ {1 / 2} w _ {k} ^ {*} \right\} \\ \Longleftrightarrow w \in \operatorname {s p a n} \left\{w _ {1} ^ {*}, \dots , w _ {k} ^ {*} \right\}. \\ \end{array}
$$
It is also straightforward to verify that  $Q^2 = Q$  and  $Q^T\Sigma w_i^* = 0$  for  $i = 1,\dots,k$ . For any  $v\in \mathbb{R}^d$ ,  $Rv$  lies in the image of  $Q$  and so there exists  $w$  such that  $Rv = Qw$ . Then we can check that
$$
\begin{array}{l} v ^ {T} R \Sigma w _ {i} ^ {*} = \langle R v, \Sigma w _ {i} ^ {*} \rangle \\ = \langle Q w, \Sigma w _ {i} ^ {*} \rangle = \langle w, Q ^ {T} \Sigma w _ {i} ^ {*} \rangle = 0 \\ \end{array}
$$