SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} = \left\| u ^ {(a)} - \frac {1}{\left| S ^ {(a)} \right|} \sum_ {r \in S ^ {(a)}} v _ {r} \right\| _ {2} + \left\| v _ {i} ^ {\prime} - \sum_ {r \in [ N ]} x _ {r} ^ {\prime} v _ {r} + \sum_ {b \in [ m ]} \sum_ {r \in S ^ {(b)}} \frac {1}{\left| S ^ {(b)} \right|} \sum_ {j \in S ^ {(b)}} x _ {j} ^ {\prime} v _ {r} \right\| _ {2} \\ \leq \left\| u ^ {(a)} - \frac {1}{| S ^ {(a)} |} \sum_ {r \in S ^ {(a)}} v _ {r} \right\| _ {2} + \left\| v _ {i} ^ {\prime} - \sum_ {r \in [ N ]} x _ {r} ^ {\prime} v _ {r} ^ {\prime} \right\| _ {2} \\ + \left\| - \sum_ {b \in [ m ]} \sum_ {r \in S ^ {(b)}} x _ {r} ^ {\prime} u ^ {(b)} + \sum_ {b \in [ m ]} \sum_ {r \in S ^ {(b)}} \frac {1}{| S ^ {(b)} |} \sum_ {j \in S ^ {(b)}} x _ {j} ^ {\prime} v _ {r} \right\| _ {2} \\ = \left\| u ^ {(a)} - \frac {1}{| S ^ {(a)} |} \sum_ {r \in S ^ {(a)}} v _ {r} \right\| _ {2} + \left\| v _ {i} ^ {\prime} - \sum_ {r \in [ N ]} x _ {r} ^ {\prime} v _ {r} ^ {\prime} \right\| _ {2} \\ + \left\| \sum_ {b \in [ m ]} \sum_ {j \in S ^ {(b)}} x _ {j} ^ {\prime} \left(u ^ {(b)} - \frac {1}{| S ^ {(b)} |} \sum_ {r \in S ^ {(b)}} v _ {r}\right) \right\| _ {2} \\ \end{array}
$$
where the first and third inequalities use that  $v_{r} = v_{r}^{\prime} + u^{(b)}$  for all  $r \in S^{(b)}$ , and throughout we use that  $x_{r} = x_{r}^{\prime} = 0$  for  $r \notin S^{(1)} \cup \dots \cup S^{(m)}$ . Applying Property (iv), equation (9), and the bound  $\| x^{\prime}\|_{1} \leq 3^{t - 1}$ , we get
$$
\begin{array}{l} \left\| v _ {i} - \sum_ {r \in [ N ]} x _ {r} v _ {r} \right\| _ {2} \leq \sqrt {9 t (t n / N) ^ {1 / t}} + 4 ^ {t - 1} \sqrt {9 t (t n / N) ^ {1 / t}} + 3 ^ {t - 1} \sqrt {9 t (t n / N) ^ {1 / t}} \\ \leq 3 \cdot 4 ^ {t - 1} \sqrt {9 t (t n / N) ^ {1 / t}} \\ \end{array}
$$
However, we wanted a bound on  $v_{i} - \sum_{j \neq i} x_{j} v_{j}$ , and unfortunately  $x_{i} \neq 0$ . Fortunately, it is enough that  $x_{i}$  is bounded away from 1. Since  $x_{i}' = 0$ , we have
$$
| x _ {i} | \leq \frac {1}{| S ^ {(a)} |} \sum_ {j \in S ^ {(a)}} | x _ {j} ^ {\prime} | + \frac {1}{| S ^ {(a)} |} \leq \frac {\left\| x ^ {\prime} \right\| _ {1} + 1}{| S ^ {(a)} |} \leq \frac {3 ^ {t - 1}}{3 ^ {t + 1}} = \frac {1}{9}.
$$
Thus, by Claim E.4,
$$
\left\| v _ {i} - \frac {1}{1 - x _ {i}} \sum_ {j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq \frac {1}{1 - x _ {i}} \cdot 3 \cdot 4 ^ {t - 1} \sqrt {9 t (t n / N) ^ {1 / t}} \leq 4 ^ {t} \sqrt {9 t (t n / N) ^ {1 / t}}.
$$
Finally, we have  $\| x / (1 - x_i)\| _1\leq (9 / 8)(2\cdot 3^{t - 1} + 1)\leq 3^t$  so  $x / (1 - x_{i})$  satisfies all the desired conditions. This completes the induction.
Finally, we extend Theorem E.3 to real-valued sparse vectors via a discretization argument.
Lemma E.5. Let  $n, N, t \in \mathbb{N}$  and let  $v_{1}, \ldots, v_{N} \in \mathbb{R}^{n}$  be  $t$ -sparse vectors. Then there is some  $i \in [N]$  and  $x \in \mathbb{R}^n$  such that  $\| x \|_1 \leq 3^t$  and
$$
\left\| v _ {i} - \sum_ {j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq 4 ^ {t + 2} \sqrt {t} (n / N) ^ {1 / (4 t)} \cdot \max  _ {j \in [ N ]} \| v _ {j} \| _ {\infty}.
$$
Proof. Without loss of generality assume that  $\max_{j\in [N]}\| v_j\|_{\infty} = 1$ . Let  $k\in \mathbb{N}$  be fixed later. Define a map  $\varphi :[-1,1]\to \{0,1\}^{2k + 1}$  by
$$
\varphi (c) = \left\{ \begin{array}{l l} e _ {k + 1 + \lfloor c k \rfloor} & \text {i f} c <   0 \\ e _ {k + 1} & \text {i f} c = 0  . \\ e _ {k + 1 + \lceil c k \rceil} & \text {i f} c > 0 \end{array} \right.
$$
Also let  $\Phi : \mathbb{R}^{2k + 1} \to \mathbb{R}$  be the linear map that sends  $\Phi e_i \mapsto (i - k - 1)/k$  for each  $i \in [2k + 1]$ . Note that  $|\Phi \varphi(c) - c| \leq 1/k$  for all  $c \in [-1,1]$  and  $\Phi \varphi(0) = 0$ . Define  $\varphi^{\oplus n}: [-1,1]^n \to \{0,1\}^{(2k + 1)n}$  by  $\varphi(c_1, \ldots, c_n) = (\varphi(c_1), \ldots, \varphi(c_n))$ , and define  $\Phi^{\oplus n}: \{0,1\}^{(2k + 1)n} \to \mathbb{R}^n$  by  $\Phi^{\oplus n}(x_1, \ldots, x_n) = (\Phi(x_1), \ldots, \Phi(x_n))$ . For any  $i \in [N]$ , the vector  $\varphi^{\oplus n}(v_i)$  is  $t$ -sparse and lies in  $\{0,1\}^{(2k + 1)n}$ . Thus, applying Theorem E.3 gives some  $i \in [N]$  and  $x \in \mathbb{R}^N$  with  $\|x\|_1 \leq 3^t$  and
$$
\left\| \varphi^ {\oplus n} (v _ {i}) - \sum_ {j \neq i} x _ {j} \varphi^ {\oplus n} (v _ {j}) \right\| _ {2} \leq 4 ^ {t} \sqrt {9 t (t n (2 k + 1) / N) ^ {1 / t}}.
$$
Since  $\Phi^{\oplus n}$  is a linear map and  $\| \Phi^{\oplus n}\| _2 = \| \Phi \| _2\leq \sqrt{2k + 1}$ , we then get
$$
\left\| \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {i}) - \sum_ {j \neq i} x _ {j} \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {j}) \right\| _ {2} \leq 4 ^ {t} \sqrt {9 t (2 k + 1) (t n (2 k + 1) / N) ^ {1 / t}}.
$$
But now for every  $j \in [N]$ , we know that
$$
\left\| v _ {j} - \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {j}) \right\| _ {2} ^ {2} = \sum_ {a \in \operatorname {s u p p} (v _ {j})} \left(v _ {j a} - \Phi \varphi (v _ {j a})\right) ^ {2} \leq \frac {t}{k ^ {2}}.
$$
$$
\begin{array}{l} \left\| v _ {i} - \sum_ {j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq \left\| \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {i}) - \sum_ {j \neq i} x _ {j} \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {j}) \right\| _ {2} + \left\| v _ {i} - \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {i}) \right\| _ {2} \\ + \sum_ {j \neq i} | x _ {j} | \cdot \left\| v _ {j} - \Phi^ {\oplus n} \varphi^ {\oplus n} (v _ {j}) \right\| _ {2} \\ \leq 4 ^ {t} \sqrt {9 t (2 k + 1) (t n (2 k + 1) / N) ^ {1 / t}} + (1 + 3 ^ {t}) \cdot \frac {\sqrt {t}}{k} \\ \leq (2 k + 1) \cdot 4 ^ {t + 1} \sqrt {t} (n / N) ^ {1 / (2 t)} + \frac {4 ^ {t} \sqrt {t}}{k}. \\ \end{array}
$$
Taking  $k = (N / n)^{1 / (4t)}$  gives the claimed bound.
Combining Lemma E.5 with Lemma E.1 lets us prove Theorem 3.2.
Proof of Theorem 3.2. Set  $\delta \coloneqq \sqrt{1 / (4\kappa)}$  and  $N = 4^{4t(t + 3)}t^{2t}\kappa^{2t}n$ . By Lemma E.5, for any  $t$ -sparse vectors  $v_{1},\ldots ,v_{N}\in \mathbb{R}^{n}$  with  $\| v_i\| _2\leq 1$  for all  $i\in [N]$ , there is some  $i\in [N]$  and  $x\in \mathbb{R}^n$  such that  $\| x\| _1\leq 3^t$  and
$$
\left\| v _ {i} - \sum_ {j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq 4 ^ {t + 2} \sqrt {t} (n / N) ^ {1 / (4 t)} \leq \frac {1}{4 \sqrt {\kappa}} <   \delta .
$$
It follows from Lemma E.1 that  $\mathcal{P}_{t,1/3^{t+1}}(\Sigma) \leq N\log_2\kappa$ . Finally, by Lemma A.2, we conclude that  $\mathcal{N}_{t,1/3^{t+1}}(\Sigma) \leq N\log_2\kappa$ .
F Generalization bounds
F.1 Finite-class model selection
Lemma F.1. Let  $n, m, n_{\text{eff}} \in \mathbb{N}$  and let  $\Sigma$  be a positive semi-definite matrix. Fix a vector  $w^{*} \in \mathbb{R}^{n}$  and a closed set  $\mathcal{W} \subseteq \mathbb{R}^{n}$  and let  $(X_{i}, y_{i})_{i=1}^{m}$  be independent draws  $X_{i} \sim N(0, \Sigma)$  and  $y_{i} = \langle X_{i}, w^{*} \rangle + \xi_{i}$  where  $\xi_{i} \sim N(0, \sigma^{2})$ . Pick
$$
\hat{w}\in \operatorname *{argmin}_{w\in \mathcal{W}}\| \mathbb{X}w - y\|_{2}^{2}
$$
where  $\mathbb{X}:m\times n$  is the matrix with rows  $X_{1},\ldots ,X_{m}$ . For any  $\epsilon, \delta \in (0,1)$ , suppose that with probability at least  $1 - \delta$ , the following bounds hold uniformly over  $w\in \mathcal{W}$ :
1.  $\left|\frac{1}{m}\| \mathbb{X}(w - w^{*})\|_{2}^{2} - \| w - w^{*}\|_{\Sigma}^{2}\right|\leq \epsilon \| w - w^{*}\|_{\Sigma}^{2}$
2.  $\left|\left\langle \xi ,\frac{\mathbb{X}(w - w^{*})}{\| \mathbb{X}(w - w^{*})\|_{2}}\right\rangle \right|\leq \sigma \sqrt{n_{eff}}.$
Then with probability at least  $1 - \delta$  it also holds that
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} \leq \sqrt {\frac {1 + \epsilon}{1 - \epsilon}} \inf  _ {w \in \mathcal {W}} \| w - w ^ {*} \| _ {\Sigma} + 2 \sigma \sqrt {\frac {2 n _ {\text {e f f}}}{m}}.
$$
Proof. Consider the event in which both bounds hold. Let  $w_{\mathrm{opt}} \in \operatorname{argmin}_{w \in \mathcal{W}} \| w - w^{*} \|_{\Sigma}^{2}$ . Then
$$
\begin{array}{l} \left\| \mathbb {X} (\hat {w} - w ^ {*}) \right\| _ {2} ^ {2} = \left\| \mathbb {X} \hat {w} - y \right\| _ {2} ^ {2} + 2 \langle \xi , \mathbb {X} (\hat {w} - w ^ {*}) \rangle - \left\| \xi \right\| _ {2} ^ {2} \\ \leq \| \mathbb {X} w _ {\text {o p t}} - y \| _ {2} ^ {2} + 2 \langle \xi , \mathbb {X} (\hat {w} - w ^ {*}) \rangle - \| \xi \| _ {2} ^ {2} \\ = \| \mathbb {X} \left(w _ {\text {o p t}} - w ^ {*}\right) \| _ {2} ^ {2} + 2 \langle \xi , \mathbb {X} (\hat {w} - w ^ {*}) \rangle - 2 \langle \xi , \mathbb {X} \left(w _ {\text {o p t}} - w ^ {*}\right) \rangle \\ \leq \left\| \mathbb {X} (w _ {\mathrm {o p t}} - w ^ {*}) \right\| _ {2} ^ {2} + 2 \left(\left\| \mathbb {X} (\hat {w} - w ^ {*}) \right\| _ {2} + \left\| \mathbb {X} (w _ {\mathrm {o p t}} - w ^ {*}) \right\| _ {2}\right) \sigma \sqrt {n _ {\mathrm {e f f}}}. \\ \end{array}
$$
Subtracting  $\| \mathbb{X}(w_{\mathrm{opt}} - w^{*})\|_{2}^{2}$  from both sides and dividing by  $\| \mathbb{X}(\hat{w} -w^{*})\|_{2} + \| \mathbb{X}(w_{\mathrm{opt}} - w^{*})\|_{2},$  we get that
$$
\left\| \mathbb {X} \left(\hat {w} - w ^ {*}\right) \right\| _ {2} - \left\| \mathbb {X} \left(w _ {\text {o p t}} - w ^ {*}\right) \right\| _ {2} \leq 2 \sigma \sqrt {n _ {\text {e f f}}}.
$$
$$
\begin{array}{l} \| \hat {w} - w ^ {*} \| _ {\Sigma} \leq \sqrt {\frac {1}{(1 - \epsilon) m}} \| \mathbb {X} (\hat {w} - w ^ {*}) \| _ {2} \\ \leq \sqrt {\frac {1}{(1 - \epsilon) m}} \left\| \mathbb {X} \left(w _ {\text {o p t}} - w ^ {*}\right) \right\| _ {2} + 2 \sigma \sqrt {\frac {(1 + \epsilon) n _ {\text {e f f}}}{m}} \\ \leq \sqrt {\frac {1 + \epsilon}{1 - \epsilon}} \| w _ {\text {o p t}} - w ^ {*} \| _ {\Sigma} + 2 \sigma \sqrt {\frac {2 n _ {\text {e f f}}}{m}} \\ \end{array}
$$
Lemma F.2. Let  $n, m \in \mathbb{N}$  and let  $\Sigma$  be a positive semi-definite matrix. Fix a vector  $w^{*} \in \mathbb{R}^{n}$  and a finite set  $\mathcal{W} \subseteq \mathbb{R}^n$  and let  $(X_{i},y_{i})_{i = 1}^{m}$  be independent draws  $X_{i} \sim N(0,\Sigma)$  and  $y_{i} = \langle X_{i},w^{*}\rangle +\xi_{i}$  where  $\xi_{i} \sim N(0,\sigma^{2})$ . Pick
$$
\hat{w}\in \operatorname *{argmin}_{w\in \mathcal{W}}\| \mathbb{X}w - y\|_{2}^{2}.
$$
For any  $\epsilon, \delta \in (0,1)$ , if  $m \geq 8\epsilon^{-2}\log(2|\mathcal{W}| / \delta)$ , then with probability at least  $1 - 2\delta$ , we have
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} \leq \sqrt {\frac {1 + \epsilon}{1 - \epsilon}} \inf  _ {w \in \mathcal {W}} \| w - w ^ {*} \| _ {\Sigma} + 4 \sigma \sqrt {\frac {\log (2 | \mathcal {W} | / \delta)}{m}}.
$$
Proof. For any fixed  $w \in \mathcal{W}$ , the random variables  $\langle X_i, w - w^* \rangle \sim N(0, \| w - w^* \|_{\Sigma}^2)$  are independent, and therefore  $\| \mathbb{X}(w - w^*) \|_2^2 \sim \| w - w^* \|_{\Sigma}^2 \chi_m^2$ . It follows that for any  $\epsilon > 0$ ,
$$
\operatorname * {P r} \left[ \left| \frac {1}{m} \| \mathbb {X} (w - w ^ {*}) \| _ {2} ^ {2} - \| w - w ^ {*} \| _ {\Sigma} ^ {2} \right| > \epsilon \| w - w ^ {*} \| _ {\Sigma} ^ {2} \right] \leq 2 e ^ {- m \epsilon^ {2} / 8}.
$$
By the union bound, if  $m \geq 8\epsilon^{-2}\log (2|\mathcal{W}| / \delta)$ , then with probability at least  $1 - \delta$  it holds that for all  $w \in \mathcal{W}$
$$
\left| \frac {1}{m} \| \mathbb {X} \left(w - w ^ {*}\right) \| _ {2} ^ {2} - \| w - w ^ {*} \| _ {\Sigma} ^ {2} \right| \leq \epsilon \| w - w ^ {*} \| _ {\Sigma} ^ {2}. \tag {10}
$$
Also, for any fixed  $w \in \mathcal{W}$ , conditioned on  $\mathbb{X}$ , the random variable  $\langle \xi, \frac{\mathbb{X}(w - w^{*})}{\|\mathbb{X}(w - w^{*})\|_{2}} \rangle$  has distribution  $N(0, \sigma^2)$ . Thus, by a Gaussian tail bound and the union bound, we have for any  $t > 0$  that
$$
\Pr \left[ \max  _ {w \in \mathcal {W}} \left| \left\langle \xi , \frac {\mathbb {X} (w - w ^ {*})}{\| \mathbb {X} (w - w ^ {*}) \|} \right\rangle \right| \geq \sigma t \right] \leq 2 | \mathcal {W} | \cdot e ^ {- t ^ {2} / 2}.
$$
In particular, with probability at least  $1 - \delta$  it holds that
$$
\left. \max  _ {w \in \mathcal {W}} \left| \left\langle \xi , \frac {\mathbb {X} (w - w ^ {*})}{\| \mathbb {X} (w - w ^ {*}) \|} \right\rangle \right| \leq \sigma \sqrt {2 \log (2 | \mathcal {W} | / \delta)}. \right. \tag {11}
$$
Using (10) and (11) we apply Lemma F.1 which gives the desired bound.
F.2 Weak learning
Lemma F.3. Let  $n, m \in \mathbb{N}$  and  $\epsilon, \delta > 0$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix and let  $\mathbb{X} : m \times n$  have independent rows  $X_1, \ldots, X_m \sim N(0, \Sigma)$ . For any fixed  $u, v \in \mathbb{R}^n$ , if  $m \geq 8\epsilon^{-2}\log (8 / \delta)$ , then it holds with probability at least  $1 - \delta$  that
$$
\left| u ^ {\top} \left(\frac {1}{m} \mathbb {X} ^ {\top} \mathbb {X} - \Sigma\right) v \right| \leq 2 \epsilon \| u \| _ {\Sigma} \| v \| _ {\Sigma}.
$$
Proof. Decompose  $u = av + w$  where  $\langle v, w \rangle_{\Sigma} = 0$ , so that  $a = \langle u, v \rangle_{\Sigma} / \| v \|_{\Sigma}^{2}$ . Since  $\| \mathbb{X}v \|_{2}^{2} \sim \| v \|_{\Sigma}^{2} \chi_{m}^{2}$  and  $m \geq 8\epsilon^{-2} \log(4/\delta)$  it holds with probability at least  $1 - \delta/2$  that
$$
\left| v ^ {\top} \left(\frac {1}{m} \mathbb {X} ^ {\top} \mathbb {X} - \Sigma\right) v \right| = \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \langle X _ {i}, v \rangle^ {2} - \| v \| _ {\Sigma} ^ {2} \right| \leq \epsilon \| v \| _ {\Sigma} ^ {2}.
$$
$$
\left| w ^ {\top} \left(\frac {1}{m} \mathbb {X} ^ {\top} \mathbb {X} - \Sigma\right) v \right| = \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \langle X _ {i}, w \rangle \langle X _ {i}, v \rangle \right| = \left| \frac {1}{m} \sum_ {i = 1} ^ {m} \langle Z _ {i}, \Sigma^ {1 / 2} w \rangle \langle Z _ {i}, \Sigma^ {1 / 2} v \rangle \right|
$$
where we define independent random vectors  $Z_{1},\ldots ,Z_{m}\sim N(0,I_{n})$  so that  $X_{i} = \Sigma^{1 / 2}Z_{i}$ . Since  $m\geq 8\log (2 / \delta)$ , with probability at least  $1 - \delta /4$  we have  $\sum_{i = 1}^{m}\langle Z_i,\Sigma^{1 / 2}v\rangle^2\leq 2m\| v\|_{\Sigma}^2$ . Condition on the value of this sum, and note that since  $\Sigma^{1 / 2}v\perp \Sigma^{1 / 2}w$ , the random variables  $\langle Z_i,\Sigma^{1 / 2}w\rangle$  are still (independent and) distributed as  $N(0,\| w\|_{\Sigma}^{2})$ . Thus
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \langle Z _ {i}, \Sigma^ {1 / 2} w \rangle \langle Z _ {i}, \Sigma^ {1 / 2} v \rangle \sim N \left(0, \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \| w \| _ {\Sigma} ^ {2} \langle Z _ {i}, \Sigma^ {1 / 2} v \rangle^ {2}\right).
$$
When the variance is at most  $2\| w\|_{\Sigma}^{2}\| v\|_{\Sigma}^{2} / m$ , we have with probability at least  $1 - \delta /4$  that the sum is at most  $2\| w\|_{\Sigma}\| v\|_{\Sigma}\sqrt{2\log(8 / \delta) / m}$  in magnitude. So, using  $m\geq 8\epsilon^{-2}\log (8 / \delta)$  it holds unconditionally with probability at least  $1 - \delta /2$  that
$$
\left| \frac {1}{m} \sum_ {i = 1} ^ {m} \langle Z _ {i}, \Sigma^ {1 / 2} w \rangle \langle Z _ {i}, \Sigma^ {1 / 2} v \rangle \right| \leq \epsilon \| w \| _ {\Sigma} \| v \| _ {\Sigma}.
$$
In all, we have that
$$
\left| u ^ {\top} \left(\frac {1}{m} \mathbb {X} ^ {\top} \mathbb {X} - \Sigma\right) v \right| \leq | a | \epsilon \| v \| _ {\Sigma} ^ {2} + \epsilon \| w \| _ {\Sigma} \| v \| _ {\Sigma} \leq 2 \epsilon \| u \| _ {\Sigma} \| v \| _ {\Sigma}
$$
using that  $|a| \leq \| u \|_{\Sigma} / \| v \|_{\Sigma}$  and  $\| w \|_{\Sigma} \leq \| u \|_{\Sigma}$ .
Lemma F.4. Let  $n, m \in \mathbb{N}$  and let  $\Sigma$  be a positive semi-definite matrix. Fix a vector  $w^{*} \in \mathbb{R}^{n}$  and a finite set  $\mathcal{W} \subseteq \mathbb{R}^n$  and let  $(X_{i},y_{i})_{i = 1}^{m}$  be independent draws  $X_{i} \sim N(0,\Sigma)$  and  $y_{i} = \langle X_{i},w^{*}\rangle +\xi_{i}$  where  $\xi_{i} \sim N(0,\sigma^{2})$ . Pick
$$
(\hat{w},\hat{\beta})\in \operatorname *{argmin}_{\substack{w\in \mathcal{W}\\ \beta \in \mathbb{R}}}\| \beta \mathbb{X}w - y\|_{2}^{2}.
$$
Suppose  $\alpha := \max_{w \in \mathcal{W}} \frac{\langle w, w^* \rangle_\Sigma}{\|w\|_\Sigma \|w^*\|_\Sigma} > 0$ . For any  $\delta > 0$ , if  $m \geq C\alpha^{-2}\log(32|\mathcal{W}| / \delta)$  for a sufficiently large absolute constant  $C$ , then with probability at least  $1 - \delta$ ,
$$
\left\| \hat {\beta} \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} \leq (1 - \alpha^ {2} / 4) \| w ^ {*} \| _ {\Sigma} ^ {2} + \frac {4 0 0 \sigma^ {2} \log (4 | \mathcal {W} | / \delta)}{\alpha^ {2} m}.
$$
Proof. For any vectors  $u, v \in \mathbb{R}^n$ , define  $\Delta(u, v) = u^\top \left( \frac{1}{m} \mathbb{X}^\top \mathbb{X} - \Sigma \right) v$ .
Claim F.5. With probability at least  $1 - \delta$ , the following bounds hold uniformly over  $w \in \mathcal{W}$  and  $\beta \in \mathbb{R}$ :
1.  $\left|\left\langle \xi ,\frac{\mathbb{X}(\beta w - w^{*})}{\left\| \mathbb{X}(\beta w - w^{*})\right\|_{2}}\right\rangle \right|\leq \sigma \sqrt{n_{eff}}$  where  $n_{eff}\coloneqq 2\log (32|\mathcal{W}| / \delta)$
2.  $|\Delta (\beta w,w^{*})|\leq \frac{\alpha}{100}\| \beta w\|_{\Sigma}\| w^{*}\|_{\Sigma}$
3.  $|\Delta (\beta w,\beta w)|\leq \frac{\alpha}{100}\| \beta w\|_{\Sigma}^{2}$
Proof of claim. For item (1), fix  $w \in \mathcal{W}$ . Let  $\Phi^{(w)}: 2 \times m$  be a matrix whose rows form an orthonormal basis for  $\operatorname{span}\{\mathbb{X}w, \mathbb{X}w^*\} \subseteq \mathbb{R}^m$ . Then (denoting the unit Euclidean ball in  $\mathbb{R}^2$  by  $B_2$ ) we have for all  $\beta \in \mathbb{R}$  that
$$
\left| \left\langle \xi , \frac {\mathbb {X} (\beta w - w ^ {*})}{\| \mathbb {X} (\beta w - w ^ {*}) \| _ {2}} \right\rangle \right| \leq \sup  _ {u \in B _ {2}} \left| \left\langle \xi , \left(\Phi^ {(w)}\right) ^ {\top} u \right\rangle \right| \leq \left\| \Phi^ {(w)} \xi \right\| _ {2} \leq \sqrt {2} \max  _ {i \in [ 2 ]} | \langle \Phi_ {i} ^ {(w)}, \xi \rangle |.
$$
Since  $\langle \Phi_i^{(w)},\xi \rangle \sim N(0,\sigma^2)$ , we have  $\operatorname*{Pr}[\lvert\langle\Phi_i^{(w)},\xi\rangle\rvert>\sigma\sqrt{2\log(4|\mathcal{W}|/\delta)}]\leq\delta/(4|\mathcal{W}|)$ . A union bound over  $i\in [2]$  and  $w\in \mathcal{W}$  gives that condition (2) in Lemma F.1 is satisfied with probability at least  $1-\delta/2$ .
For items (2) and (3), note that  $\Delta$  is bilinear, so it suffices to take  $\beta = 1$ . Applying Lemma F.3 and the union bound, so long as  $m \geq C\alpha^{-2}\log(32|\mathcal{W}| / \delta)$  for a sufficiently large constant  $C$ , items (2) and (3) hold simultaneously with probability at least  $1 - \delta / 2$ .
Henceforth we assume that all of the events in the above claim hold. Let  $w_0 \in \mathcal{W}$  be such that  $|\langle w_0, w^* \rangle_{\Sigma}| = \alpha \| w_0 \|_{\Sigma} \| w^* \|_{\Sigma}$ . Let  $\beta_0 = \langle w_0, w^* \rangle_{\Sigma} / \| w_0 \|_{\Sigma}^2$ . Then
$$
\left\| \beta_ {0} w _ {0} - w ^ {*} \right\| _ {\Sigma} ^ {2} = \left(1 - \alpha^ {2}\right) \left\| w ^ {*} \right\| _ {\Sigma} ^ {2}.
$$
Claim F.6. The excess empirical risk can be bounded as
$$
\left\| \mathbb {X} (\hat {\beta} \hat {w} - w ^ {*}) \right\| _ {2} \leq \| \mathbb {X} (w _ {0} - w ^ {*}) \| _ {2} + 2 \sigma \sqrt {n _ {e f f}}.
$$
$$
\begin{array}{l} \left\| \mathbb {X} (\hat {\beta} \hat {w} - w ^ {*}) \right\| _ {2} ^ {2} = \left\| \mathbb {X} \hat {\beta} \hat {w} - y \right\| _ {2} ^ {2} + 2 \langle \xi , \mathbb {X} (\hat {\beta} \hat {w} - w ^ {*}) \rangle - \| \xi \| _ {2} ^ {2} \\ \leq \left\| \mathbb {X} \beta_ {0} w _ {0} - y \right\| _ {2} ^ {2} + 2 \langle \xi , \mathbb {X} (\hat {\beta} \hat {w} - w ^ {*}) \rangle - \| \xi \| _ {2} ^ {2} \\ = \| \mathbb {X} (\beta_ {0} w _ {0} - w ^ {*}) \| _ {2} ^ {2} + 2 \langle \xi , \mathbb {X} (\hat {\beta} \hat {w} - w ^ {*}) \rangle - 2 \langle \xi , \mathbb {X} (\beta_ {0} w _ {0} - w ^ {*}) \rangle \\ \leq \left\| \mathbb {X} \left(\beta_ {0} w _ {0} - w ^ {*}\right) \right\| _ {2} ^ {2} + 2 \left(\left\| \mathbb {X} \left(\hat {\beta} \hat {w} - w ^ {*}\right) \right\| _ {2} + \left\| \mathbb {X} \left(\beta_ {0} w _ {0} - w ^ {*}\right) \right\| _ {2}\right) \sigma \sqrt {n _ {\mathrm {e f f}}} \\ \end{array}
$$
where the last bound is by item (1) of Claim F.5. Simplifying, we get the claimed bound.
$$
\begin{array}{l} \left\| \hat {\beta} \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} = \frac {1}{m} \left\| \mathbb {X} (\hat {\beta} \hat {w} - w ^ {*}) \right\| _ {2} ^ {2} - \Delta (\hat {\beta} \hat {w} - w ^ {*}, \hat {\beta} \hat {w} - w ^ {*}) \\ \leq \frac {1}{m} \left(\| \mathbb {X} \left(\beta_ {0} w _ {0} - w ^ {*}\right) \| _ {2} + 2 \sigma \sqrt {n _ {\mathrm {e f f}}}\right) ^ {2} - \Delta \left(\hat {\beta} \hat {w} - w ^ {*}, \hat {\beta} \hat {w} - w ^ {*}\right) \\ \leq \frac {1 + \alpha^ {2} / 1 0 0}{m} \| \mathbb {X} (\beta_ {0} w _ {0} - w ^ {*}) \| _ {2} ^ {2} + (1 + 1 0 0 \alpha^ {- 2}) \frac {\sigma^ {2} n _ {\mathrm {e f f}}}{m} - \Delta (\hat {\beta} \hat {w} - w ^ {*}, \hat {\beta} \hat {w} - w ^ {*}) \\ = (1 + \alpha^ {2} / 1 0 0) \| \beta_ {0} w _ {0} - w ^ {*} \| _ {\Sigma} ^ {2} + (1 + 1 0 0 \alpha^ {- 2}) \frac {\sigma^ {2} n _ {\mathrm {e f f}}}{m} \\ - \Delta (\hat {\beta} \hat {w} - w ^ {*}, \hat {\beta} \hat {w} - w ^ {*}) + \left(1 + \frac {\alpha^ {2}}{1 0 0}\right) \Delta (\beta_ {0} w _ {0} - w ^ {*}, \beta_ {0} w _ {0} - w ^ {*}) \\ \end{array}
$$
$$
\begin{array}{l} \leq \left(1 + \alpha^ {2} / 1 0 0\right) \| \beta_ {0} w _ {0} - w ^ {*} \| _ {\Sigma} ^ {2} + \left(1 + 1 0 0 \alpha^ {- 2}\right) \frac {\sigma^ {2} n _ {\mathrm {e f f}}}{m} \\ + \left| \Delta (\hat {\beta} \hat {w}, \hat {\beta} \hat {w}) \right| + 2 \left| \Delta (\hat {\beta} \hat {w}, w ^ {*}) \right| \\ + (1 + \alpha^ {2} / 1 0 0) | \Delta (\beta_ {0} w _ {0}, \beta_ {0} w _ {0}) | + 2 (1 + \alpha^ {2} / 1 0 0) | \Delta (\beta_ {0} w _ {0}, w ^ {*}) | \\ + \left(\alpha^ {2} / 1 0 0\right) | \Delta \left(w ^ {*}, w ^ {*}\right) |. \\ \end{array}
$$
where the first inequality is by Claim F.6, the second inequality is by AM-GM, and the final inequality is expanding out the terms  $\Delta (\hat{\beta}\hat{w} -w^{*},\hat{\beta}\hat{w} -w^{*})$  and  $\Delta (\beta_0w_0 - w^*,\beta_0w_0 - w^*)$  (via bilinearity) and cancelling out the common term  $\Delta (w^{*},w^{*})$ . Finally applying items (2) and (3) of Claim F.5, we get
$$
\begin{array}{l} \left\| \hat {\beta} \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} \leq (1 + \alpha^ {2} / 1 0 0) \| \beta_ {0} w _ {0} - w ^ {*} \| _ {\Sigma} ^ {2} + (1 + 1 0 0 \alpha^ {- 2}) \frac {\sigma^ {2} n _ {\mathrm {e f f}}}{m} \\ + \frac {\alpha}{1 0 0} \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} ^ {2} + \frac {\alpha}{5 0} \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} \| w ^ {*} \| _ {\Sigma} \\ + \frac {\alpha}{5 0} \| \beta_ {0} w _ {0} \| _ {\Sigma} ^ {2} + \frac {\alpha}{2 5} \| \beta_ {0} w _ {0} \| _ {\Sigma} \| w ^ {*} \| _ {\Sigma} + \frac {\alpha^ {3}}{1 0 0} \| w ^ {*} \| _ {\Sigma} ^ {2} \\ \leq \left(1 - 9 \alpha^ {2} / 1 0\right) \| w ^ {*} \| _ {\Sigma} ^ {2} + \frac {1 0 1 \sigma^ {2} n _ {\mathrm {e f f}}}{\alpha^ {2} m} \\ + \frac {\alpha}{1 0 0} \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} ^ {2} + \frac {\alpha}{5 0} \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} \| w ^ {*} \| _ {\Sigma} \tag {12} \\ \end{array}
$$
where the second inequality uses the bounds  $\| \beta_0w_0 - w^*\|_{\Sigma}^2 = (1 - \alpha^2)\| w^*\|_{\Sigma}^2$  and
$$
\left\| \beta_ {0} w _ {0} \right\| _ {\Sigma} = \frac {| \langle w _ {0} , w ^ {*} \rangle_ {\Sigma} |}{\| w _ {0} \| _ {\Sigma}} = \alpha \| w ^ {*} \| _ {\Sigma}.
$$
But now on the other hand,
$$
\left\| \hat {\beta} \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} = \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} ^ {2} + \| w ^ {*} \| _ {\Sigma} ^ {2} - 2 \langle \hat {\beta} \hat {w}, w ^ {*} \rangle_ {\Sigma} \geq \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} ^ {2} + \| w ^ {*} \| _ {\Sigma} ^ {2} + 2 \alpha \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} \| w ^ {*} \| _ {\Sigma}.
$$
Comparing with (12) gives
$$
\left(1 - \frac {\alpha}{1 0 0}\right) \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} ^ {2} \leq \frac {1 0 1 \sigma^ {2} n _ {\text {e f f}}}{\alpha^ {2} m} + 3 \alpha \left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} \| w ^ {*} \| _ {\Sigma}
$$