SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| v \| _ {\Sigma} ^ {2} = \sum_ {i = 1} ^ {N} \alpha_ {i} \langle v, D _ {i} \rangle_ {\Sigma} \\ \leq \sum_ {i = 1} ^ {N} | \alpha_ {i} | \| v \| _ {\Sigma} \| D _ {i} \| _ {\Sigma} \cdot \max  _ {j \in [ N ]} \frac {| \langle v , D _ {j} \rangle_ {\Sigma} |}{\| v \| _ {\Sigma} \| D _ {j} \| _ {\Sigma}} \\ \leq B \| v \| _ {\Sigma} ^ {2} \cdot \max  _ {j \in [ N ]} \frac {| \langle v , D _ {j} \rangle_ {\Sigma} |}{\| v \| _ {\Sigma} \| D _ {j} \| _ {\Sigma}} \\ \end{array}
$$
and thus  $\max_{j\in [N]}\frac{|\langle v,D_j\rangle_\Sigma|}{\|v\|_\Sigma\|D_j\|_\Sigma}\geq 1 / B.$
We can now easily prove that the standard basis is a good dictionary for well-conditioned  $\Sigma$ .
Fact A.4. Let  $\Sigma$  be a positive definite matrix with condition number  $\frac{\lambda_{max}(\Sigma)}{\lambda_{min}(\Sigma)} \leq \kappa$ . Under the  $\Sigma$ -inner product, every  $t$ -sparse vector is at least  $1 / (\sqrt{\kappa t})$ -correlated with some standard basis vector.
Proof. By Lemma A.3, it suffices to show that the standard basis  $\{e_1,\ldots ,e_n\}$  is a  $(t,\sqrt{\kappa t})$ - $\ell_1$ -representation for  $\Sigma$ . Indeed, for any  $t$ -sparse  $v\in \mathbb{R}^n$
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \left| v _ {i} \right| \cdot \left\| e _ {i} \right\| _ {\Sigma} \leq \sum_ {i = 1} ^ {n} \left| v _ {i} \right| \cdot \sqrt {\lambda_ {\max } (\Sigma)} \left\| e _ {i} \right\| _ {2} = \sqrt {\lambda_ {\max } (\Sigma)} \left\| v \right\| _ {1} \\ \leq \sqrt {\lambda_ {\max} (\Sigma)} \sqrt {t} \| v \| _ {2} \leq \sqrt {\frac {\lambda_ {\max} (\Sigma)}{\lambda_ {\min} (\Sigma)}} \sqrt {t} \| v \| _ {\Sigma} \\ \end{array}
$$
A.2 Algorithmic implications
An existential proof that  $\mathcal{N}_{t,\alpha}(\Sigma)$  is small unfortunately does not in general give an efficient algorithm for constructing a concise dictionary. However, with the caveat that the dictionary must be given to the algorithm as advice, bounds on  $\mathcal{N}_{t,\alpha}$  do imply weak learning algorithms with sample complexity  $O(\alpha^{-2}\log (n))$ :
Proposition A.5. Let  $\Sigma : n \times n$  be a positive semi-definite matrix and let  $\mathcal{D}$  be a  $(t, \alpha)$ -dictionary for  $\Sigma$ , for some  $t \in \mathbb{N}$  and  $\alpha \in (0, 1)$ . For  $m \in \mathbb{N}$  and  $t$ -sparse  $v^{*} \in \mathbb{R}^{n}$ , let  $X_{1}, \ldots, X_{m} \sim N(0, \Sigma)$  be independent and let  $y_{i} = \langle X_{i}, v^{*} \rangle + \xi_{i}$  where  $\xi_{i} \sim N(0, \sigma^{2})$ . Define the estimator
$$
\hat{v} = \operatorname *{argmin}_{\substack{v\in \mathcal{D}\\ \beta \in \mathbb{R}}}\| \beta \mathbb{X}v - y\|_{2}^{2}
$$
where  $\mathbb{X}:m\times n$  is the matrix with rows  $X_{1},\ldots ,X_{m}$ . For any  $\delta >0$ , if  $m\geq C\alpha^{-2}\log (32|\mathcal{D}| / \delta)$  for a sufficiently large absolute constant  $C$ , then with probability at least  $1 - \delta$
$$
\left\| \hat {\beta} \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} \leq (1 - \alpha^ {2} / 4) \| w ^ {*} \| _ {\Sigma} ^ {2} + \frac {4 0 0 \sigma^ {2} \log (4 | \mathcal {D} | / \delta)}{\alpha^ {2} m}.
$$
Proof. Since  $\mathcal{D}$  is a  $(t,\alpha)$ -dictionary, we know that there is some  $\tilde{v} \in \mathcal{D}$  with  $|\langle \tilde{v}, v^* \rangle_{\Sigma}| \geq \alpha \| \tilde{v} \|_{\Sigma} \| v^* \|_{\Sigma}$ . We then apply Lemma F.4.
The above guarantee is essentially of the form "at least  $1\%$  of the signal variance can be explained". Under the  $\ell_1$ -representability condition, something much stronger is true:
Proposition A.6. Let  $n, m, N, t \in \mathbb{N}$  and  $B > 0$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix and let  $\mathcal{D}$  be a  $(t, B)$ - $\ell_1$ -representation of size  $N$  for  $\Sigma$ , normalized so that  $\|v\|_{\Sigma} = 1$  for all  $v \in \mathcal{D}$ .
Fix a  $t$ -sparse vector  $v^{*} \in \mathbb{R}^{n}$ , let  $X_{1}, \ldots, X_{m} \sim N(0, \Sigma)$  be independent and let  $y_{i} = \langle X_{i}, v^{*} \rangle + \xi_{i}$  where  $\xi_{i} \sim N(0, \sigma^{2})$ . For any  $R > 0$ , define
$$
\hat{w}\in \operatorname *{argmin}_{w\in \mathbb{R}^{N}:||w||_{1}\leq BR}\| \mathbb{X}Dw - y\|_{2}^{2}
$$
where  $D: n \times N$  is the matrix with columns comprising the elements of  $\mathcal{D}$ , and  $\mathbb{X}: m \times n$  is the matrix with rows  $X_1, \ldots, X_m$ . So long as  $m = \Omega(\log(n / \delta))$  and  $R \geq \|v^*\|_{\Sigma}$ , it holds with probability at least  $1 - \delta$  that
$$
\| D \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} = O \left(B R \sigma \sqrt {\frac {\log (2 n / \delta)}{m}} + \frac {\sigma^ {2} \log (4 / \delta)}{m} + \frac {B ^ {2} R ^ {2} \log (n)}{m}\right).
$$
Proof. By  $\ell_1$ -representability and normalization of  $\mathcal{D}$ , there is some  $w^* \in \mathbb{R}^N$  such that  $v^* = Dw^*$  and  $\| w^* \|_1 \leq B \| v^* \|_\Sigma \leq BR$ . Let  $\Gamma = D^\top \Sigma D$ . Also, by normalization,  $\max_i \Gamma_{ii} = 1$ . Thus, we can apply standard "slow rate" Lasso guarantees to the samples  $(D^\top X_i, y_i)_{i=1}^m$  to get the claimed bound (see e.g. Theorem 14 of [22]).
A.3 Optimizing the Lasso in near-linear time
Theorem A.7 (see e.g. Corollary 4 and Section 5.3 in [34]). Let  $n, m, B, H, T \in \mathbb{N}$  and  $\sigma > 0$ . Fix  $X_1, \ldots, X_m \in \mathbb{R}^n$  with  $\| X_i \|_{\infty} \leq H$  for all  $i$ , and fix  $w^* \in \mathbb{R}^n$  with  $\| w^* \|_1 \leq B$ . For  $i \in [m]$  define  $y_i = \langle X_i, w^* \rangle + \xi_i$  where  $\xi_i \sim N(0, \sigma^2)$  are independent random variables. Given  $(X_i, y_i)_{i=1}^m$  as well as  $B, T$ , and  $\sigma^2$ , there is an algorithm MirrorDescentLasso  $((X_i, y_i)_{i=1}^m, B, T, \sigma^2)$ , which optimizes the Lasso objective via  $T$  iterations of mirror descent, that produces an estimate  $\hat{w} \in \mathbb{R}^n$  satisfying  $\| \hat{w} \|_1 \leq B$  and, with probability  $1 - o(1)$ ,
$$
\frac {1}{m} \left\| X \hat {w} - y \right\| _ {2} ^ {2} \leq \frac {1}{m} \left\| X w ^ {*} - y \right\| _ {2} ^ {2} + \tilde {O} \left(\frac {H ^ {2} B ^ {2}}{T} + \sqrt {\frac {H ^ {2} B ^ {2} \sigma^ {2}}{T}}\right).
$$
Moreover, the time complexity of MirrorDescentLasso() is  $\tilde{O}(nmT)$ .
Theorem A.8. Let  $n, m, B, H \in \mathbb{N}$  and  $\sigma > 0$ . Let  $\Sigma : n \times n$  be positive semi-definite with  $\max_{j \in [n]} \Sigma_{jj} \leq H^2$ . Fix  $w^* \in \mathbb{R}^n$  with  $\| w^* \|_1 \leq B$ . Let  $(X_i, y_i)_{i=1}^m$  be independent draws where  $X_i \sim N(0, \Sigma)$  and  $y_i = \langle X_i, w^* \rangle + N(0, \sigma^2)$ . Then MirrorDescentLasso  $((X_i, y_i)_{i=1}^m, B, m, \sigma^2)$  computes, in time  $\tilde{O}(nm^2)$ , an estimate  $\hat{w}$  satisfying, with probability  $1 - o(1)$ ,
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} \leq \tilde {O} \left(\frac {\sigma^ {2}}{\sqrt {m}} + \frac {\sigma H B}{\sqrt {m}} + \frac {H ^ {2} B ^ {2}}{m}\right)
$$
Proof. Since  $\max_j\Sigma_{jj}\leq H$  we have that  $\max_i\| X_i\|_\infty \leq O(H\log n)$  with probability  $1 - o(1)$ . Applying Theorem A.7 with this bound and with  $T = m$ , we obtain some  $\hat{w}\in \mathbb{R}^n$  with  $\| \hat{w}\| _1\leq B$  and, with probability  $1 - o(1)$ ,
$$
\frac {1}{m} \| X \hat {w} - y \| _ {2} ^ {2} \leq \frac {1}{m} \| X w ^ {*} - y \| _ {2} ^ {2} + \epsilon
$$
where  $\epsilon = \tilde{O}(H^2 B^2 / m) + \sqrt{H^2 B^2 \sigma^2 / m}$ . By  $\chi^2$ -concentration, we have  $\frac{1}{m} \| Xw^* - y\|_2^2 \leq \sigma^2 (1 + O(1 / \sqrt{m}))$  with probability  $1 - o(1)$ . Thus,
$$
\left\| X \hat {w} - y \right\| _ {2} \leq \left\| X w ^ {*} - y \right\| _ {2} + \sqrt {\epsilon m} \leq \sigma \sqrt {m} + O (\sigma m ^ {1 / 4}) + \sqrt {\epsilon m}
$$
$$
\| X \hat {w} - y \| _ {2} ^ {2} \leq \| X w ^ {*} - y \| _ {2} ^ {2} + m \epsilon \leq \sigma^ {2} m + O (\sigma^ {2} \sqrt {m}) + \epsilon m.
$$
Next, since  $\sup_{w\in \mathbb{R}^n:\| w\| _1\leq B}\langle w - w^*,x\rangle \leq 2B\| x\|_\infty \leq O(HB\log n)$  with probability  $1 - o(1)$  over  $x\sim N(0,\Sigma)$ , we can apply Theorem C.1 to get that with probability  $1 - o(1)$ ,
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} + \sigma^ {2} \leq \frac {1 + \tilde {O} (1 / \sqrt {m})}{m} (\| X \hat {w} - y \| _ {2} + \tilde {O} (H B)) ^ {2}.
$$
Substituting the bounds on  $\| X\hat{w} -y\| _2$  and  $\| X\hat{w} -y\| _2^2$  gives
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} + \sigma^ {2} \leq \sigma^ {2} + O (\sigma^ {2} m ^ {- 1 / 2} + \epsilon) + \tilde {O} (\sigma H B m ^ {- 1 / 2} + H B \sqrt {\epsilon / m}) + \tilde {O} (H ^ {2} B ^ {2} / m).
$$
Substituting in the value of  $\epsilon$  and simplifying, we get
$$
\left\| \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} \leq \tilde {O} \left(\frac {\sigma^ {2}}{\sqrt {m}} + \frac {\sigma H B}{\sqrt {m}} + \frac {H ^ {2} B ^ {2}}{m}\right)
$$
B Iterative Peeling
In this section we give the complete proof of Lemma 2.4, restated below as Theorem B.1, which describes the guarantees of IterativePeeling() (see Algorithm 1). This is a key ingredient in the proofs of Theorems 1.1 and 1.2. We also use it to formally prove Theorem 2.3, as well as Lemma 2.9.
Theorem B.1. Let  $n, t, d \in \mathbb{N}$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix with eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Given  $\Sigma, d,$  and  $t$ , there is a polynomial-time algorithm IterativePeeling() producing a set  $S \subseteq [n]$  with the following guarantees:
- For every  $t$ -sparse  $v \in \mathbb{R}^n$ , it holds that  $\left\| v_{[n] \setminus S} \right\|_2 \leq 3\lambda_{d+1}^{-1/2} \| v \|_\Sigma$ .
-  $|S| \leq (7t)^{2t + 1}d$ .
Essentially, the set  $S$  contains every coordinate  $i \in [n]$  that "participates" in an approximate sparse dependency, in the sense that there is some sparse linear combination of the covariates with small variance compared to the coefficient on  $i$ . To compute  $S$ , the algorithm IterativePeeling() first computes the orthogonal projection matrix  $P$  that projects onto the subspace spanned by the top  $n - d$  eigenvectors of  $\Sigma$ . Starting with the set of coordinates that correlate with  $\ker(P)$ , the procedure then iteratively grows  $S$  in such a way that at each step, a new participant of each approximate sparse dependency is discovered, but  $S$  does not become too much larger.
The following lemma will be needed to bound how much  $S$  grows at each iteration:
Lemma B.2. Let  $V \subseteq \mathbb{R}^n$  be a subspace with  $d \coloneqq \dim V$ . For some  $\alpha > 0$  define
$$
S = \left\{i \in [ n ]: \sup  _ {x \in V \backslash \{0 \}} \frac {x _ {i}}{\| x \| _ {2}} \geq \alpha \right\}.
$$
Then  $|S| \leq d / \alpha^2$ . Moreover, given a set of vectors that span  $V$ , we can compute  $S$  in time poly(  $n$  ).
Proof. Let  $k \coloneqq |S|$  and without loss of generality suppose  $S = \{1, \ldots, k\}$ . Define a matrix  $A \in \mathbb{R}^{n \times n}$  as follows. For  $1 \leq i \leq k$  let row  $A_i \in V$  be some vector such that  $\| A_i \|_2 = 1$  and  $A_{ii} \geq \alpha$ . For  $k + 1 \leq i \leq n$  let  $A_i = 0$ . Then  $\operatorname{tr}(A) \geq k\alpha$  and  $\| A \|_F = \sqrt{k}$ . However,  $\operatorname{rank}(A) \leq d$ , so the singular values  $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_n \geq 0$  of  $A$  satisfy  $\sigma_{d+1} = 0$ . Thus,
$$
k \alpha \leq \operatorname {t r} (A) \leq \sum_ {i = 1} ^ {n} \sigma_ {i} \leq \sqrt {d} \sqrt {\sum_ {i = 1} ^ {n} \sigma_ {i} ^ {2}} = \sqrt {d} \| A \| _ {F} = \sqrt {d k}
$$
where the second inequality is by e.g. Von Neumann's trace inequality, and the third inequality is by  $d$ -sparsity of the vector  $\sigma$ . It follows that  $k \leq d / \alpha^2$  as claimed.
Let  $A$  be the matrix with columns consisting of the given spanning set for  $V$ . By Gram-Schmidt, we may transform the spanning set into an orthonormal basis for  $V$ , so that  $A$  has  $d$  columns, and  $A^{\top}A = I_{d}$ . Fix  $i \in [n]$ . Then  $\sup_{x \in V \setminus \{0\}} x_i / \| x \|_2 \geq \alpha$  if and only if  $(Av)_i^2 - \alpha^2 \| Av \|_2^2 \geq 0$  for some nonzero  $v \in \mathbb{R}^d$ . Equivalently,  $(Av)_i^2 \geq \alpha^2$  for some unit vector  $v$ . This is possible if and only if  $\| A_i \|_2 \geq \alpha$  (where  $A_i$  is the  $i$ -th row of  $A$ ), which can be checked in polynomial time.
For notational convenience, we also define the set  $\mathcal{W}_{P,S}$  of vectors  $v$  with unusually large norm outside the set  $S$ .
Definition B.3. For any matrix  $P: n \times n$  and subset  $S \subseteq [n]$ , define  $\mathcal{W}_{P,S} := \{v \in \mathbb{R}^n : \| v_{S^c} \|_2 > 3\sqrt{v^\top Pv}\}$ .
Lemma B.4. Let  $n, t \in \mathbb{N}$  and let  $P: n \times n$  be an orthogonal projection matrix. Suppose  $\tau \geq 1$  and  $K \subseteq [n]$  satisfy
for all  $v \in B_0(t) \cap \mathcal{W}_{P,K}$ . Moreover, given  $P$ ,  $K$ , and  $t$ , we can compute  $\mathcal{I}_P(K)$  in time  $\mathrm{poly}(n)$ .
Proof. We define the set
It is clear from Lemma B.2 (applied with parameters  $V \coloneqq \operatorname{span}\{Pe_i : i \in K\}$  and  $\alpha \coloneqq 1/(6t)$ ) that  $|\mathcal{I}_P(K)| \leq 36t^2|K|$ , and that  $\mathcal{I}_P(K)$  can be computed in time  $\mathrm{poly}(n)$ . It remains to show that  $|\operatorname{supp}(v) \setminus (\mathcal{I}_P(K) \cup K)| \leq \tau - 1$  for all  $v \in B_0(t) \cap \mathcal{W}_{P,K}$ .
Consider any  $v \in B_0(t) \cap \mathcal{W}_{P,K}$ . Then  $\| v_{K^c}\| _2 > 3\sqrt{v^\top Pv}$ . We have
$$
\frac {\left\| v _ {K ^ {c}} \right\| _ {2} ^ {2}}{9} > v ^ {\top} P v = \| P v \| _ {2} ^ {2} = \left\| \sum_ {i = 1} ^ {n} v _ {i} P _ {i} \right\| _ {2} ^ {2} \tag {1}
$$
where the first equality uses the fact that  $P$  is a projection matrix. We also know that
$$
\left\| \sum_ {i \in [ n ] \backslash K} v _ {i} \left(P _ {i} - e _ {i}\right) \right\| _ {2} \leq \sum_ {i \in [ n ] \backslash K} | v _ {i} | \| P _ {i} - e _ {i} \| _ {2} \leq \frac {1}{3 \sqrt {t}} \| v _ {K ^ {c}} \| _ {1} \leq \frac {1}{3} \| v _ {K ^ {c}} \| _ {2} \tag {2}
$$
by the triangle inequality, the bound  $\| P_i - e_i\| _2^2 = (I - P)_{ii} = 1 - P_{ii}\leq 1 / (9t)$  (since  $i\notin K$ ) and  $t$ -sparsity of  $v$ . Moreover, (2) implies that
$$
\left\| \sum_ {i \in [ n ] \backslash K} v _ {i} P _ {i} \right\| _ {2} \leq \left\| \sum_ {i \in [ n ] \backslash K} v _ {i} \left(P _ {i} - e _ {i}\right) \right\| _ {2} + \| v _ {K ^ {c}} \| _ {2} \leq \frac {4}{3} \| v _ {K ^ {c}} \| _ {2}. \tag {3}
$$
Combining (1) and (3), the triangle inequality gives
$$
\left\| \sum_ {i \in K} v _ {i} P _ {i} \right\| _ {2} \leq \left\| \sum_ {i \in [ n ] \backslash K} v _ {i} P _ {i} \right\| _ {2} + \left\| \sum_ {i = 1} ^ {n} v _ {i} P _ {i} \right\| _ {2} \leq \frac {5}{3} \| v _ {K ^ {c}} \| _ {2}. \tag {4}
$$
$$
\begin{array}{l} \frac {\left\| v _ {K ^ {c}} \right\| _ {2} ^ {2}}{3} > \left\| \sum_ {i = 1} ^ {n} v _ {i} P _ {i} \right\| _ {2} \left\| v _ {K ^ {c}} \right\| _ {2} \tag {by(1)} \\ \geq \left| \left\langle \sum_ {i = 1} ^ {n} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right| \quad (\text {b y}) \\ \geq \left| \left\langle \sum_ {i \in [ n ] \backslash K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right| - \left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right| \quad (\text {b y}) \\ \end{array}
$$
$$
\geq \left| \left\langle \sum_ {i \in [ n ] \backslash K} v _ {i} e _ {i}, v _ {K ^ {c}} \right\rangle \right| - \left| \left\langle \sum_ {i \in [ n ] \backslash K} v _ {i} \left(P _ {i} - e _ {i}\right), v _ {K ^ {c}} \right\rangle \right| - \left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right|
$$
(by triangle inequality)
$$
\geq \| v _ {K ^ {c}} \| _ {2} ^ {2} - \left\| \sum_ {i \in [ n ] \backslash K} v _ {i} (P _ {i} - e _ {i}) \right\| _ {2} \| v _ {K ^ {c}} \| _ {2} - \left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right|
$$
(by Cauchy-Schwarz)
$$
\geq \left\| v _ {K ^ {c}} \right\| _ {2} ^ {2} - \frac {1}{3} \left\| v _ {K ^ {c}} \right\| _ {2} ^ {2} - \left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right| \tag {by(2)}
$$
$$
\left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right| > \frac {1}{3} \| v _ {K ^ {c}} \| _ {2} ^ {2} \geq \frac {1}{5} \| v _ {K ^ {c}} \| _ {2} \left\| \sum_ {i \in K} v _ {i} P _ {i} \right\| _ {2}
$$
where the last inequality is by (4). On the other hand, observe that
$$
\left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, v _ {K ^ {c}} \right\rangle \right| \leq \sum_ {j \in [ n ] \backslash K} | v _ {j} | \cdot \left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, e _ {j} \right\rangle \right| \leq \sqrt {t} \| v _ {K ^ {c}} \| _ {2} \max  _ {j \in \operatorname {s u p p} (v) \backslash K} \left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, e _ {j} \right\rangle \right|.
$$
Hence, there is some  $j \in \operatorname{supp}(v) \setminus K$  such that
$$
\left| \left\langle \sum_ {i \in K} v _ {i} P _ {i}, e _ {j} \right\rangle \right| > \frac {1}{5 \sqrt {t}} \left\| \sum_ {i \in K} v _ {i} P _ {i} \right\| _ {2}.
$$
So the vector  $x(v) \coloneqq \sum_{i \in K} v_i P_i \in \operatorname{span}\{P_i : i \in K\}$  satisfies  $|x(v)_j| > \| x(v)\|_2 / (5\sqrt{t})$ . Moreover,  $x(v)$  is nonzero since  $|x(v)_j| > 0$ . Thus,  $j \in \mathcal{I}_P(K)$ . Since we chose  $j$  to be in  $\operatorname{supp}(v) \setminus K$ , it follows that
$$
| \operatorname {s u p p} (v) \setminus (\mathcal {I} _ {P} (K) \cup K) | \leq | \operatorname {s u p p} (v) \setminus K | - 1 \leq \tau - 1
$$
where the last inequality is by assumption (b) in the lemma statement.
We can now complete the proof of Theorem B.1 by repeatedly invoking Lemma B.4 (this proof was given in Section 2.2 and is duplicated here for completeness).
Proof of Theorem B.1. Let  $\Sigma = \sum_{i=1}^{n} \lambda_i u_i u_i^\top$  be the eigendecomposition of  $\Sigma$ , and let  $P := \sum_{i=d+1}^{n} u_i u_i^\top$  be the projection onto the top  $n-d$  eigenspaces of  $\Sigma$ . Set  $K_t = \{i \in [n] : P_{ii} < 1 - 1/(9t^2)\}$ . Because  $\operatorname{tr}(P) = n-d$  and  $P_{ii} \leq 1$  for all  $i \in [n]$ , it must be that  $|K_t| \leq 9t^2d$ . Also, for any  $v \in B_0(t) \cap \mathcal{W}_{P,K_t}$  we have trivially by  $t$ -sparsity that  $|\operatorname{supp}(v) \setminus K_t| \leq t$ .
Proof of Lemma 2.9. By Theorem B.1, there is a polynomial-time computable set  $S \subseteq [n]$  such that  $\| v_{S^c} \|_2 \leq 3\sqrt{t}\lambda_{d+1}^{-1/2} \| v \|_{\Sigma}$  for all  $v \in B_0(t)$ , and  $|S| \leq (7t)^{2t+1} d$ . Let the dictionary  $\mathcal{D}$  consist of the standard basis  $\{e_1, \ldots, e_n\}$  together with a  $\Sigma$ -orthogonal basis for each subspace spanned by  $t$  vectors in  $\{e_i : i \in S\}$ . Let  $v \in \mathbb{R}^n$  be  $t$ -sparse. Let  $v_S$  denote the restriction of  $v$  to  $S$ , i.e.  $v_S := v - \sum_{i \in [n] \setminus S} v_i e_i$ . By construction of the dictionary, there is a  $\Sigma$ -orthogonal basis for  $\{e_i : i \in S \cap \operatorname{supp}(v)\}$ , so there are  $d_1, \ldots, d_t \in \mathcal{D}$  and coefficients  $b_{d_1}, \ldots, b_{d_t} \in \mathbb{R}$  with  $v_S = \sum_{i=1}^{t} b_{d_i} d_i$  and  $\langle d_i, d_j \rangle_{\Sigma} = 0$  for all  $i, j \in [t]$  with  $i \neq j$ . Note that  $\| v_S \|_{\Sigma}^2 = \sum_{i=1}^{t} b_{d_i}^2 \| d_i \|_{\Sigma}^2$ , so
$$
\sum_ {i = 1} ^ {t} \left| b _ {d _ {i}} \right| \| d _ {i} \| _ {\Sigma} \leq \sqrt {t} \sqrt {\sum_ {i = 1} ^ {t} b _ {d _ {i}} ^ {2} \| d _ {i} \| _ {\Sigma} ^ {2}} = \sqrt {t} \| v _ {S} \| _ {\Sigma}.
$$
Now, we claim that the desired coefficient vector  $\{\alpha_d:d\in \mathcal{D}\}$  for  $v$  is defined by  $\alpha_{d} = b_{d} + \sum_{i\in [n]\backslash S}v_{i}\mathbb{1}[d = e_{i}]$ . We can check that  $\sum_{d\in \mathcal{D}}\alpha_{d}d = \sum_{i = 1}^{t}b_{di} + \sum_{i\in [n]\backslash S}v_{i}e_{i} = v$ . Also,
$$
\begin{array}{l} \| v _ {S} \| _ {\Sigma} \leq \| v \| _ {\Sigma} + \| v _ {S ^ {c}} \| _ {\Sigma} \\ \leq \| v \| _ {\Sigma} + \sqrt {\lambda_ {n}} \| v _ {S ^ {c}} \| _ {2} \\ \leq \left(1 + 3 \sqrt {\lambda_ {n} / \lambda_ {d + 1}}\right) \| v \| _ {\Sigma} \\ \end{array}
$$
by the guarantee of set  $S$
$$
\sum_ {i = 1} ^ {t} \left| b _ {d _ {i}} \right| \| d _ {i} \| _ {\Sigma} \leq (1 + 3 \sqrt {\lambda_ {n} / \lambda_ {d + 1}}) \sqrt {t} \| v \| _ {\Sigma} \sqrt {\lambda_ {n} / \lambda_ {d + 1}}.
$$
$$
\begin{array}{l} \sum_ {d \in \mathcal {D}} | c _ {d} | \| d \| _ {\Sigma} \leq (1 + 3 \sqrt {\lambda_ {n} / \lambda_ {d + 1}}) \sqrt {t} \| v \| _ {\Sigma} + \sum_ {i \in [ n ] \backslash S} | v _ {i} | \| e _ {i} \| _ {\Sigma} \\ \leq \left(1 + 3 \sqrt {\lambda_ {n} / \lambda_ {d + 1}}\right) \sqrt {t} \| v \| _ {\Sigma} + \sqrt {t} \| v _ {S ^ {c}} \| _ {2} \sqrt {\lambda_ {n}} \\ \leq \left(1 + 3 \sqrt {\lambda_ {n} / \lambda_ {d + 1}}\right) \sqrt {t} \| v \| _ {\Sigma} + 3 \sqrt {t} \| v \| _ {\Sigma} \sqrt {\lambda_ {n} / \lambda_ {d + 1}} \\ \leq 7 \sqrt {t} \sqrt {\lambda_ {n} / \lambda_ {d + 1}} \| v \| _ {\Sigma} \\ \end{array}
$$
Proof of Theorem 2.3. Immediate from Lemma 2.9 and Lemma A.3.
C An efficient algorithm for handling outlier eigenvalues
In this section we describe and provide error guarantees for a novel sparse linear regression algorithm BOAR-Lasso() (see Algorithm 2 for pseudocode), completing the proof of Theorem 1.1; in Section C.1 we then analyze a modified algorithm to prove Theorem 1.2.
The key subroutine of BOAR-Lasso() is the procedure AdaptivelyRegularizedLasso(), which (like the simplified procedure AdaptedBP() from Section 3) first invokes procedure IterativePeeling() to compute the set of coordinates that participate in sparse approximate dependencies, and second computes a modified Lasso estimate where those coordinates are not regularized.
We start with Theorem C.2, which shows that, in the setting where  $\Sigma$  has few outlier eigenvalues, the procedure Adaptively Regularized Lasso() estimates the sparse ground truth regressor at the "slow rate" (e.g. in the noiseless setting, the excess risk is at most  $O(\| v^{*}\|_{\Sigma}^{2}r_{\mathrm{eff}} / m)$ ). Typical excess risk analyses for Lasso proceed by applying some general-purpose machinery for generalization bounds, such as the following result which only requires understanding  $\langle w - w^{*},X\rangle$  for  $X\sim N(0,\Sigma)$ .