SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2: Solve sparse linear regression when covariate eigenspectrum has few outliers
Procedure AdaptivelyRegularizedLasso  $(\Sigma ,(X_i,y_i)_{i = 1}^m,t,d_l,\delta)$    
Data: Covariance matrix  $\Sigma :n\times n$  samples  $(X_{i},y_{i})_{i = 1}^{m}$  sparsity  $t$  small eigenvalue count  $d_{l}$  , failure probability  $\delta$    
Result: Estimate  $\hat{v}$  of unknown sparse regressor, satisfying Theorem C.2  $\sum_{i = 1}^{n}\lambda_{i}u_{i}u_{i}^{\top}\gets$  eigendecomposition of  $\boldsymbol{\Sigma}$ $S\gets$  IterativePeeling(  $\Sigma ,d_l,t)$  /\* See Algorithm 1 \*/ Return   
 $\hat{v}\gets \underset {v\in \mathbb{R}^n}{\mathrm{argmin}}\sum_{i = 1}^{m}(\langle X_i,v\rangle -y)^2 +8\lambda_{n - d_h}\log (12n / \delta)\| v_{S^c}\| _1^2 +2\sqrt{2\lambda_{n - d_h}\log(12n / \delta)}\| v_{S^c}\| _1.$    
Procedure BOAR-Lasso  $(\Sigma ,(Y_i,y_i)_{i = 1}^m,t,d_l,L,\delta)$    
Data: Covariance matrix  $\Sigma :n\times n$  samples  $(X_{i},y_{i})_{i = 1}^{m}$  sparsity  $t$  small eigenvalue count  $d_{l}$  , repetition count  $L$  , failure probability  $\delta$    
Result: Estimate  $\hat{v}$  of unknown sparse regressor, satisfying Theorem C.3   
 $\hat{s}^{(0)}\gets 0\in \mathbb{R}^n$    
for  $0\leq j <   L$  do Set  $\begin{array}{r}\Sigma^{(j)}\leftarrow \left[ \begin{array}{cc}\Sigma & (\hat{s}^{(j)})^\top \Sigma \\ \Sigma \hat{s}^{(j)} & (\hat{s}^{(j)})^\top \Sigma \hat{s}^{(j)} \end{array} \right]. \end{array}$    
Set  $A^{(j)}\coloneqq \{mj + 1,\dots,m(j + 1)\}$ $\hat{w}^{(j + 1)}\gets$  AdaptivelyRegularizedLasso  $(\Sigma^{(j)}$  .  $((X_i,\langle X_i,\hat{s}^{(j)}\rangle),y_i - \langle X_i,\hat{s}^{(j)}\rangle)_{i\in A^{(j)}},t + 1,d_l + 1,\delta /L)$ $\hat{v}^{(j + 1)}\gets \hat{w}_{[n]}^{(j + 1)} + \hat{w}_{n + 1}^{(j + 1)}\hat{s}^{(j)}$ $\hat{s}^{(j + 1)}\gets \hat{s}^{(j)} + \hat{v}^{(j + 1)}$    
return  $\hat{s}^{(L)}$
Theorem C.1 (Theorem 1 in [45]). Let  $n, m \in \mathbb{N}$  and  $\epsilon, \delta, \sigma > 0$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix and fix  $w^{*} \in \mathbb{R}^{n}$ . Let  $X : m \times n$  have i.i.d. rows  $X_{1}, \ldots, X_{m} \sim N(0, \Sigma)$ , and let  $y = Xw^{*} + \xi$  where  $\xi \sim N(0, \sigma^{2}I_{m})$ . Let  $F: \mathbb{R}^{d} \to [0, \infty]$  be a continuous function such that
$$
\operatorname * {P r} _ {x \sim N (0, \Sigma)} \left[ \sup  _ {w \in \mathbb {R} ^ {n}} \langle w - w ^ {*}, x \rangle - F (w) > 0 \right] \leq \delta .
$$
If  $m \geq 196\epsilon^{-2}\log(12/\delta)$ , then with probability at least  $1 - 4\delta$  it holds that for all  $w \in \mathbb{R}^d$ ,
$$
\left\| w - w ^ {*} \right\| _ {\Sigma} ^ {2} + \sigma^ {2} \leq \frac {1 + \epsilon}{m} \left(\left\| X w - y \right\| _ {2} + F (w)\right) ^ {2}.
$$
In classical settings, e.g. (a) where  $\| v^{*}\|_{1}$  is bounded and  $\max_{i}\Sigma_{ii}\leq 1$  (see Proposition A.6) or (b) where  $\Sigma$  satisfies the compatibility condition (see Definition G.1), the above result can be applied together with the straightforward bound  $\langle v - v^{*},X\rangle \leq \| v - v^{*}\|_{1}\| X\|_{\infty}$ . To prove Theorem C.2 we follow the same general recipe as (a), with several modifications.
First, since  $\max_i\Sigma_{ii}$  could be arbitrarily large, we need to treat the (few) large eigenspaces of  $\Sigma$  separately when bounding  $\langle v - v^{*},X\rangle$ . Similarly, since Theorem B.1 only gives bounds on  $v^{*}$  for coordinates outside  $S$ , we separately bound  $\langle (v - v^{*})_S,X\rangle$  using that  $|S|$  is small. Second, to achieve the optimal rate of  $\sigma^2 n_{\mathrm{eff}} / m$  rather than  $\sigma^2\sqrt{n_{\mathrm{eff}} / m}$ , we do not directly apply Theorem C.1 to the noisy samples  $(X_{i},y_{i})$ ; instead, we derive a modification of that result (Lemma F.7) that only invokes Theorem C.1 on the noiseless samples  $(X_{i},\langle X_{i},v^{*}\rangle)$ , and separately bounds the in-sample prediction error  $\| \mathbb{X}(\hat{v} -v^{*})\| _2$ . A similar technique is used in [45] for constrained least-squares programs (see their Lemma 15); our Lemma F.7 applies to a broad family of additively regularized programs, which obviates the need to independently estimate  $\| v^{*}\|_{\Sigma}$  but otherwise achieves comparable bounds.
Theorem C.2. Let  $n, t, d_l, d_h, m \in \mathbb{N}$  and  $\sigma, \delta > 0$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix with eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Let  $(X_i, y_i)_{i=1}^m$  be independent samples where  $X_i \sim N(0, \Sigma)$  and
$y_{i} = \langle X_{i},v^{*}\rangle +\xi_{i}$  , for  $\xi_i\sim N(0,\sigma^2)$  and a fixed t-sparse vector  $v^{*}\in \mathbb{R}^{n}$  . Let  $\hat{v}$  be the output of. Adaptively RegularizedLasso  $(\Sigma ,(X_{i},y_{i})_{i = 1}^{m},t,d_{l},\delta)$  . Let  $n_{eff}\coloneqq (7t)^{2t + 1}d_l + d_h + \log (48 / \delta)$  and let  $r_{eff}\coloneqq t(\lambda_{n - d_h} / \lambda_{d_l + 1})\log (12n / \delta)$  . There are absolute constants  $c,C > 0$  so that the following holds. If  $m\geq Cn_{eff}$  , then with probability at least  $1 - \delta$
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq c \left(\frac {\sigma^ {2} n _ {e f f}}{m} + \frac {(\sigma + \| v ^ {*} \| _ {\Sigma}) \| v ^ {*} \| _ {\Sigma} \sqrt {r _ {e f f}}}{\sqrt {m}} + \frac {\| v ^ {*} \| _ {\Sigma} ^ {2} r _ {e f f}}{m}\right).
$$
Proof. Define projection matrix  $P := \sum_{i=1}^{n-d_h} u_i u_i^\top$ , so that  $\mathrm{rank}(P^\perp) = d_h$  and  $\lambda_{\max}(P\Sigma P) \leq \lambda_{n-d_h}$ . For any  $v \in \mathbb{R}^n$  and  $X \sim N(0, \Sigma)$ , we can bound
$$
\begin{array}{l} \langle v - v ^ {*}, X \rangle = \left\langle \left(v - v ^ {*}\right) _ {S ^ {c}}, P X \right\rangle + \left\langle \left(v - v ^ {*}\right) _ {S ^ {c}}, P ^ {\perp} X \right\rangle + \left\langle \left(v - v ^ {*}\right) _ {S}, X \right\rangle \\ = \left\langle \left(v - v ^ {*}\right) _ {S ^ {c}}, P X \right\rangle + \left\langle \Sigma^ {1 / 2} \left(v - v ^ {*}\right), \Sigma^ {- 1 / 2} \left(P ^ {\perp} X\right) _ {S ^ {c}} \right\rangle + \left\langle \Sigma^ {1 / 2} \left(v - v ^ {*}\right), \Sigma^ {- 1 / 2} X _ {S} \right\rangle \\ \leq \left\| (v - v ^ {*}) _ {S ^ {\epsilon}} \right\| _ {1} \left\| P X \right\| _ {\infty} + \left\| \Sigma^ {1 / 2} (v - v ^ {*}) \right\| _ {2} \left(\| Z \| _ {2} + \| W \| _ {2}\right) \\ \end{array}
$$
where  $PX \sim N(0, P\Sigma P)$ ,  $Z \sim N(0, \Sigma^{-1/2}(P^{\perp} \Sigma P^{\perp})_{S^{c} S^{c}} \Sigma^{-1/2})$ , and  $W \sim N(0, \Sigma^{-1/2} \Sigma_{SS} \Sigma^{-1/2})$ . First, since  $\max_i(P\Sigma P)_{ii} \leq \lambda_{\max}(P\Sigma P) \leq \lambda_{n - d_h}$ , we have the Gaussian tail bound
$$
\Pr \left[ \| P X \| _ {\infty} > \sqrt {\lambda_ {n - d _ {h}} \cdot 2 \log (1 2 n / \delta)} \right] \leq \delta / 1 2.
$$
Second, since
$$
\begin{array}{l} \Sigma^ {- 1 / 2} \left(P ^ {\perp} \Sigma P ^ {\perp}\right) _ {S ^ {c} S ^ {c}} \Sigma^ {- 1 / 2} \preceq \Sigma^ {- 1 / 2} \left(P ^ {\perp} \Sigma P ^ {\perp}\right) \Sigma^ {- 1 / 2} \\ = P ^ {\perp} \\ \end{array}
$$
(by Cauchy Interlacing Theorem)
(since  $P^{\perp}$  commutes with  $\Sigma$ )
we have that  $\| Z\| _2^2$  is stochastically dominated by  $\chi_{d_h}^2$ , and thus
$$
\Pr \left[ \| Z \| _ {2} > \sqrt {2 d _ {h}} \right] \leq e ^ {- m / 4} \leq \delta / 1 2.
$$
Third, similarly, since  $\Sigma^{-1/2}\Sigma_{SS}\Sigma^{-1/2} \preceq I$  (again by Cauchy Interlacing Theorem) and also  $\mathrm{rank}(\Sigma^{-1/2}\Sigma_{SS}\Sigma^{-1/2}) \leq |S|$ , we have that  $\|W\|_2^2$  is stochastically dominated by  $\chi_{|S|}^2$ , and thus
$$
\Pr \left[ \| W \| _ {2} ^ {2} > \sqrt {2 | S |} \right] \leq e ^ {- m / 4} \leq \delta / 1 2.
$$
Combining the above bounds, we have that with probability at least  $1 - \delta /4$  over  $X\sim N(0,\Sigma)$ , for all  $v\in \mathbb{R}^n$ ,
$$
\langle v - v ^ {*}, X \rangle \leq \| (v - v ^ {*}) _ {S ^ {c}} \| _ {1} \sqrt {\lambda_ {n - d _ {h}} \cdot 2 \log (1 2 n / \delta)} + \left\| \Sigma^ {1 / 2} (v - v ^ {*}) \right\| _ {2} (\sqrt {2 d _ {h}} + \sqrt {2 | S |}).
$$
We can therefore apply Lemma F.7 with covariance  $\Sigma$ , seminorm  $\Phi(v) := 2\sqrt{2\lambda_{n - d_h}\log(12n / \delta)}\|v_{S^c}\|_1$ ,  $p := 4(d_h + |S|)$ , ground truth  $v^*$ , samples  $(X_i, y_i)_{i=1}^m$ , and failure probability  $\delta / 4$ . By the bound on  $|S|$  (Theorem B.1) we have  $|S| + d_h \leq (7t)^{2t + 1}d_l + d_h \leq n_{\mathrm{eff}}$ , so it holds that  $m \geq 16p + 196\log(48 / \delta)$ . Thus, with probability at least  $1 - 2\delta$ , we have
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} n _ {\mathrm {e f f}}}{m} + \frac {(\sigma + \| v ^ {*} \| _ {\Sigma}) \| v _ {S ^ {c}} ^ {*} \| _ {1} \sqrt {\lambda_ {n - d _ {h}} \log (1 2 n / \delta)}}{\sqrt {m}} + \frac {\| v _ {S ^ {c}} ^ {*} \| _ {1} ^ {2} \lambda_ {n - d _ {h}} \log (1 2 n / \delta)}{m}\right).
$$
By the guarantee of  $S$  (Theorem B.1) and  $t$ -sparsity of  $v^{*}$ , we have  $\| v_{Sc}^{*}\|_{2} \leq 3\lambda_{d_{l} + 1}^{-1 / 2}\| v^{*}\|_{\Sigma}$ , and thus  $\| v_{Sc}^{*}\|_{1} \leq 3\sqrt{t}\lambda_{d_{l} + 1}^{-1 / 2}\| v^{*}\|_{\Sigma}$ . Substituting into the previous bound, we get
$$
\left\| \hat {v} - v ^ {*} \right\| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} n _ {\mathrm {e f f}}}{m} + \frac {\left(\sigma + \left\| v ^ {*} \right\| _ {\Sigma}\right) \left\| v ^ {*} \right\| _ {\Sigma} \sqrt {r _ {\mathrm {e f f}}}}{\sqrt {m}} + \frac {\left\| v ^ {*} \right\| _ {\Sigma} ^ {2} r _ {\mathrm {e f f}}}{m}\right)
$$
The limitation of Adaptively Regularized Lasso() is that the excess risk bound depends on  $\| v^{*}\|_{\Sigma}^{2}$  rather than just  $\sigma^2$ . We next show that by a boosting approach, we can exponentially attenuate that dependence, essentially achieving the near-optimal rate of  $\sigma^2 n_{\mathrm{eff}} / m$ . The key insight is that after producing an estimate  $\hat{v}$  of  $v^{*}$ , we can augment the set of covariates with the feature  $\langle \mathbb{X},\hat{v}\rangle$  and try to predict the response  $y - \langle \mathbb{X},\hat{v}\rangle$ , which is now a  $(t + 1)$ -sparse combination of the features. In standard settings, this is typically a bad idea because it introduces a sparse linear dependence. However, by the Cauchy Interlacing Theorem it increases the number of outlier eigenvalues by at most one – so our algorithms still apply. Thus, if we have enough samples that the excess risk bound in Theorem C.2 is non-trivially smaller than  $\| v^{*}\|_{\Sigma}^{2}$ , then we can iteratively achieve better and better estimates up to the noise limit. This is precisely what BOAR-Lasso() does; the precise guarantees are stated in the following theorem, which completes the proof of Theorem 1.1.
Theorem C.3. Let  $n, t, d_{l}, d_{h}, m, L \in \mathbb{N}$  and  $\sigma, \delta > 0$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix with eigenvalues  $\lambda_{1} \leq \dots \leq \lambda_{n}$ . Let  $(X_{i}, y_{i})_{i=1}^{m}$  be independent samples where  $X_{i} \sim N(0, \Sigma)$  and  $y_{i} = \langle X_{i}, v^{*} \rangle + \xi_{i}$ , for  $\xi_{i} \sim N(0, \sigma^{2})$  and a fixed  $t$ -sparse vector  $v^{*} \in \mathbb{R}^{n}$ .
Then, given  $\Sigma$ ,  $(X_{i},y_{i})_{i = 1}^{m}$ ,  $t$ ,  $d_{l}$ , and  $\delta$ , the algorithm BOAR-Lasso() outputs an estimator  $\hat{v}$  with the following properties.
Let  $n_{\text{eff}} \coloneqq (7t)^{2t + 1}d_l + d_h + \log(48 / \delta)$  and let  $r_{\text{eff}} \coloneqq t(\lambda_{n - d_h} / \lambda_{d_l + 1})\log(12n / \delta)$ . There are absolute constants  $c_0, C_0 > 0$  such that the following holds. If  $m \geq C_0L(n_{\text{eff}} + r_{\text{eff}})$ , then with probability at least  $1 - \delta$ , it holds that
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq c _ {0} \frac {\sigma^ {2} (n _ {\text {e f f}} + r _ {\text {e f f}})}{m / L} + 2 ^ {- L} \cdot \| v ^ {*} \| _ {\Sigma} ^ {2}.
$$
Moreover, BOAR-Lasso() has time complexity  $\mathrm{poly}(n,m,t)$ .
Proof. Let  $(A_0, \ldots, A_{L-1})$  be an partition of  $[m]$  into  $L$  sets of size  $m / L$ . The idea of the algorithm is to compute vectors  $\hat{v}^{(1)}, \ldots, \hat{v}^{(L)}$  where each  $v^{(i)}$  is an estimate of  $v^* - \sum_{j=1}^{i-1} \hat{v}^{(j)}$ . Concretely, fix some  $0 \leq j \leq L-1$  and suppose that we have computed some vectors  $\hat{v}^{(1)}, \ldots, \hat{v}^{(j)}$ . Set  $\hat{s}^{(j)} := \hat{v}^{(1)} + \dots + \hat{v}^{(j)}$ . Define a matrix  $\Sigma^{(j)}: (n+1) \times (n+1)$  by
$$
\Sigma^ {(j)} := \left[ \begin{array}{c c} \Sigma & (\hat {s} ^ {(j)}) ^ {\top} \Sigma \\ \Sigma \hat {s} ^ {(j)} & (\hat {s} ^ {(j)}) ^ {\top} \Sigma (\hat {s} ^ {(j)}) \end{array} \right].
$$
Thus, for example,  $\Sigma^{(0)}$  has zeroes in the last row and last column. Now for each  $i\in A_j$ , define  $(X_i^{(j)},y_i^{(j)})$  by
$$
X _ {i} ^ {(j)} := \left(X _ {i}, \langle X _ {i}, \hat {s} ^ {(j)} \rangle\right)
$$
$$
y _ {i} ^ {(j)} := y _ {i} - \langle X _ {i}, \hat {s} ^ {(j)} \rangle .
$$
By construction, the  $m / L$  samples  $(X_{i}^{(j)},y_{i}^{(j)})_{i\in A_{j}}$  are independent and distributed as  $X_{i}^{(j)}\sim N(0,\Sigma^{(j)})$  and  $y_{i}^{(j)} = \langle X_{i}^{(j)},(v^{*}, - 1)\rangle +\xi_{i}$ . Let  $\lambda_1^{(j)}\leq \dots \leq \lambda_{n + 1}^{(j)}$  be the eigenvalues of  $\Sigma^{(j)}$ .
Now we apply Theorem C.2 with covariance  $\Sigma^{(j)}$ , samples  $(X_{i}^{(j)},y_{i}^{(j)})_{i\in A_{j}}$ , sparsity  $t + 1$ , outlier counts  $d_{l} + 1$  and  $d_h + 1$ , and failure probability  $\delta /L$ ; let  $n_{\mathrm{eff}}^{(j)}$  and  $r_{\mathrm{eff}}^{(j)}$  be the induced parameters defined in that theorem statement, and let  $c,C$  be the constants. By the Cauchy Interlacing Theorem, we have  $\lambda_{d_l + 2}^{(j)}\geq \lambda_{d_l + 1}$  and similarly  $\lambda_{n + 1 - (d_h + 1)}^{(j)}\leq \lambda_{n - d_h}$ . Thus  $r_{\mathrm{eff}}^{(j)}\leq 2r_{\mathrm{eff}}$ . Also  $n_{\mathrm{eff}}^{(j)}\leq n_{\mathrm{eff}}$ . Thus, if the constant  $C_0$  is chosen appropriately large, then  $m / L\geq 16cr_{\mathrm{eff}}^{(j)}$  and also  $m / L\geq Cn_{\mathrm{eff}}^{(j)}$ . Hence (by the error guarantee of Theorem C.2) with probability at least  $1 - \delta /L$  we obtain a vector  $\hat{w}^{(j + 1)}$  such that
$$
\begin{array}{l} \left\| \hat {w} ^ {(j + 1)} - (v ^ {*}, - 1) \right\| _ {\Sigma^ {(j)}} ^ {2} \leq \frac {c \sigma^ {2} n _ {\mathrm {e f f}} ^ {(j)}}{m / L} + c \| (v ^ {*}, - 1) \| _ {\Sigma^ {(j)}} ^ {2} \sqrt {\frac {r _ {\mathrm {e f f}} ^ {(j)}}{m / L}} + c \sigma \| (v ^ {*}, - 1) \| _ {\Sigma^ {(j)}} \sqrt {\frac {r _ {\mathrm {e f f}} ^ {(j)}}{m / L}} \\ \leq \frac {2 c \sigma^ {2} n _ {\text {e f f}}}{m / L} + \frac {\| (v ^ {*} , - 1) \| _ {\Sigma (j)} ^ {2}}{4} + \left(\frac {\| (v ^ {*} , - 1) \| _ {\Sigma (j)} ^ {2}}{4} + \frac {4 c ^ {2} \sigma^ {2} r _ {\text {e f f}}}{m / L}\right) \\ \end{array}
$$
$$
\leq \frac {c _ {0}}{2} \frac {\sigma^ {2} \left(n _ {\text {e f f}} + r _ {\text {e f f}}\right)}{m / L} + \frac {\| (v ^ {*} , - 1) \| _ {\Sigma (j)} ^ {2}}{2} \tag {5}
$$
where the second inequality uses AM-GM to bound the third term, and the third inequality is by choosing  $c_0 \geq 4c + 8c^2$ .
But now define  $\hat{v}^{(j + 1)}\coloneqq \hat{w}_{[n]}^{(j + 1)} + \hat{w}_{n + 1}^{(j + 1)}\hat{s}^{(j)}$ . Then we observe that  $\| (v^{*}, - 1)\|_{\Sigma (j)}^{2}$  =  $\left\| v^{*} - \hat{s}^{(j)}\right\|_{\Sigma}^{2}$  and  $\left\| \hat{w}^{(j + 1)} - (v^{*}, - 1)\right\|_{\Sigma (j)}^{2} = \left\| \hat{v}^{(j + 1)} - (v^{*} - \hat{s}^{(j)})\right\|_{\Sigma}^{2} = \left\| v^{*} - \hat{s}^{(j + 1)}\right\|_{\Sigma}^{2}$  where  $\hat{s}^{(j + 1)} = \hat{v}^{(1)} + \dots +\hat{v}^{(j + 1)}$ . So (5) is equivalent to
$$
\left\| v ^ {*} - \hat {s} ^ {(j + 1)} \right\| _ {\Sigma} ^ {2} \leq \frac {c _ {0}}{2} \frac {\sigma^ {2} (n _ {\mathrm {e f f}} + r _ {\mathrm {e f f}})}{m / L} + \frac {1}{2} \left\| v ^ {*} - \hat {s} ^ {(j)} \right\| _ {\Sigma} ^ {2}.
$$
Inductively, we conclude that
$$
\left\| v ^ {*} - \hat {s} ^ {(L)} \right\| _ {\Sigma} ^ {2} \leq c _ {0} \frac {\sigma^ {2} (n _ {\mathrm {e f f}} + r _ {\mathrm {e f f}})}{m / L} + 2 ^ {- L} \| v ^ {*} \| _ {\Sigma} ^ {2}
$$
as desired. The time complexity (see Algorithm 2 for full pseudocode) is dominated by  $L$  eigendecompositions of  $n \times n$  Hermitian matrices (each of which takes time  $O(n^3)$  by e.g. the QR algorithm), as well as  $L$  convex optimizations (each of which takes time  $\tilde{O}(n^3)$  to solve to inverse-polynomial accuracy [26], which is sufficient for the correctness proof).
C.1 An alternative algorithm (proof of Theorem 1.2)
In this section we prove Theorem 1.2, which essentially states that the sample complexity dependence on  $d_{l}$  in BOAR-Lasso() can be removed at the cost of a time complexity depending on  $d_{l}^{t}$ . See Algorithm 3 for the pseudocode of how we modify AdaptivelyRegularizedLasso(): essentially, we brute force search over all size- $t$  subsets of the set  $S$  produced by IterativePeeling(), construct an appropriate dictionary for each of these  $\binom{|S|}{t}$  subsets, and then perform a final model selection step (with fresh samples) to pick the best dictionary/estimator. The boosting step is exactly identical to that in BOAR-Lasso().
Lemma C.4. Let  $n, t, d \in \mathbb{N}$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix with eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Then there is a family  $\mathcal{D} \subseteq \mathbb{R}^{n \times (n + t)}$  of size  $|\mathcal{D}| \leq (7t)^{2t^2 + t}(2d)^t$ , consisting entirely of  $n \times (n + t)$  matrices with the form
$$
D := \left[ \begin{array}{c c c c} I _ {n} & d _ {1} & \ldots & d _ {t} \end{array} \right],
$$
with the following property. For any  $t$ -sparse  $v \in \mathbb{R}^n$ , there is some  $D \in \mathcal{D}$  and  $w \in \mathbb{R}^{n + k}$  with  $v = Dw$  and
$$
\left\| w \right\| _ {1} \leq \frac {7 t ^ {1 / 2}}{\sqrt {\lambda_ {d + 1}}} \sqrt {v ^ {\top} \Sigma v}.
$$
Proof. Let  $u_1, \ldots, u_n \in \mathbb{R}^n$  be the eigenvectors of  $\Sigma$  corresponding to eigenvalues  $\lambda_1, \ldots, \lambda_n$  respectively, so that  $\Sigma = \sum_{i=1}^{n} \lambda_i u_i u_i^\top$ . Define  $\overline{\Sigma} := \lambda_{d+1}^{-1} \sum_{i=1}^{n} \min(\lambda_i, \lambda_{d+1}) u_i u_i^\top$ . Let  $S$  be the output of IterativePeeling  $(\Sigma, d_l, t)$ , and let  $D := \{D(T): T \in \binom{S}{t}\}$ , where for any  $T \in \binom{S}{t}$ , we let  $\{d_1, \ldots, d_t\}$  be a  $\overline{\Sigma}$ -orthonormal basis for span  $\{e_i: i \in T\}$ , and let  $D(T)$  be the  $n \times (n + t)$  matrix with columns  $e_1, \ldots, e_n, d_1, \ldots, d_t$ . The bound on  $|\mathcal{D}|$  follows from Theorem B.1.
For any  $t$ -sparse  $v \in \mathbb{R}^n$ , pick the matrix  $D \in \mathcal{D}$  indexed by any  $T \in \binom{S}{t}$  with  $S \cap \operatorname{supp}(v) \subseteq T$ . Let  $d_1, \ldots, d_t \in \mathbb{R}^n$  be the last  $t$  columns of  $D$ . Then there are coefficients  $b_1, \ldots, b_t$  so that we can write  $v_S = \sum_{i=1}^{t} b_i d_i$ . Since  $d_i^\top \overline{\Sigma} d_{i'} = \mathbb{1}[i = i']$  for all  $i, i' \in [t]$ , we have  $v_S^\top \overline{\Sigma} v_S = \sum_{i=1}^{t} b_i^2$ . Hence,  $\|b\|_1 \leq \sqrt{t} \sqrt{v_S^\top \overline{\Sigma} v_S}$ . But we can bound
$$
\begin{array}{l} \sqrt {v _ {S} ^ {\top} \overline {{\Sigma}} v _ {S}} = \left\| \overline {{\Sigma}} ^ {1 / 2} v _ {S} \right\| _ {2} \\ \leq \left\| \overline {{\Sigma}} ^ {1 / 2} v \right\| _ {2} + \left\| \overline {{\Sigma}} ^ {1 / 2} v _ {S ^ {c}} \right\| _ {2} \quad (\text {b y}) \\ \end{array}
$$
Algorithm 3: Alternative algorithm to solve sparse linear regression when covariate eigenspectrum has few outliers
Procedure AugmentedDictionaryLasso  $(\Sigma ,(X_i,y_i)_{i = 1}^m,t,d_l,\delta)$    
Data: Covariance matrix  $\Sigma :n\times n$  samples  $(X_{i},y_{i})_{i = 1}^{m}$  sparsity  $t$  , small eigenvalue count  $d_{l}$  , failure probability  $\delta$    
Result: Estimate  $\hat{v}$  of unknown sparse regressor, satisfying Theorem C.5   
 $\sum_{i = 1}^{n}\lambda_{i}u_{i}u_{i}^{\top}\gets$  eigendecomposition of  $\boldsymbol{\Sigma}$ $S\gets$  IterativePeeling(  $\Sigma ,d_l,t)$  /\* See Algorithm 1 \*/   
 $\overline{\Sigma}\gets \lambda_{d_l + 1}^{-1}\sum_{i = 1}^{n}\min (\lambda_{i},\lambda_{d_l + 1})u_{i}u_{i}^{\top}$    
for  $T\in \binom {S}{[t]}$  do   
 $d_1^{(T)},\ldots ,d_t^{(T)}\gets \overline{\Sigma}$  -orthogonal basis for span  $\{e_i:i\in T\}$ $D(T)\leftarrow \left[I_n d_1^{(T)}\dots d_t^{(T)}\right]$    
Compute   
 $\hat{w} (T)\gets \underset {w\in \mathbb{R}^{n + t}}{\mathrm{argmin}}\left[\sum_{i = 1}^{m / 2}\left(\langle X_i,D(T)w\rangle -y_{1:m / 2}\right)^2\right.$ $+8\lambda_{n - d}\log (8n / \delta)\| w\| _1^2 +2\sqrt{2\lambda_{n - d}\log(8n / \delta)}\| y_{1:m / 2}\| _2\| w\| _1$    
Select best hypothesis   
 $\hat{T}\gets \operatorname *{argmin}_{T\in \binom{S}{[t]}}\sum_{i = m / 2 + 1}^{m}(\langle X_i,D(T)\hat{w} (T)\rangle -y_i)^2$    
return  $D(\hat{T})\hat{w} (\hat{T})$
$$
\begin{array}{l} \leq \lambda_ {d + 1} ^ {- 1 / 2} \left\| \Sigma^ {1 / 2} v \right\| _ {2} + \| v _ {S ^ {c}} \| _ {2} \quad (\text {b y}) \\ \leq \lambda_ {d + 1} ^ {- 1 / 2} \left\| \Sigma^ {1 / 2} v \right\| _ {2} + 3 \lambda_ {d + 1} ^ {- 1 / 2} \sqrt {v ^ {\top} \Sigma v} \quad (\text {b y}) \\ \leq 4 \lambda_ {d + 1} ^ {- 1 / 2} \sqrt {v ^ {\top} \Sigma v}. \\ \end{array}
$$
We conclude that  $\| b\| _1\leq 4\sqrt{t}\lambda_{d + 1}^{-1 / 2}\sqrt{v^\top\Sigma v}$ . Thus, if we define
$$
w := \sum_ {i \in [ n ] \backslash S} v _ {i} e _ {i} + \sum_ {i = 1} ^ {t} b _ {i} e _ {n + i},
$$
where here  $e_1, \ldots, e_{n+k}$  refer to the standard basis vectors in  $\mathbb{R}^{n+t}$ , then we have  $Dw = v_{[n] \setminus S} + \sum_{i=1}^{t} b_i d_i = v$ , and also
$$
\| w \| _ {1} \leq \| b \| _ {1} + \sum_ {i \in [ n ] \backslash S} | v _ {i} | \leq 4 \sqrt {t} \lambda_ {d + 1} ^ {- 1 / 2} \sqrt {v ^ {\top} \Sigma v} + \sqrt {t} \| v _ {S ^ {c}} \| _ {2} \leq \frac {7 \sqrt {t}}{\lambda_ {d + 1} ^ {1 / 2}} \sqrt {v ^ {\top} \Sigma v}
$$
Theorem C.5. Let  $n, t, d_{l}, d_{h}, m \in \mathbb{N}$  and let  $\Sigma : n \times n$  be a positive semi-definite matrix with eigenvalues  $\lambda_{1} \leq \dots \leq \lambda_{n}$ . Let  $(X_{i}, y_{i})_{i=1}^{m}$  be independent samples where  $X_{i} \sim N(0, \Sigma)$  and  $y_{i} = \langle X_{i}, v^{*} \rangle + \xi_{i}$ , for  $\xi_{i} \sim N(0, \sigma^{2})$  and a fixed  $t$ -sparse vector  $v^{*} \in \mathbb{R}^{n}$ . Set  $k := t(7t)^{2t^{2} + t} d_{l}^{t}$  and let  $\mathcal{D}$  be the family of matrices (of size at most  $k$ ) guaranteed by Lemma C.4.
Let  $\delta > 0$ . For every  $D \in \mathcal{D}$ , define
$$
\hat {w} (D) \in \operatorname * {a r g m i n} _ {w \in \mathbb {R} ^ {n + t}} \left\| \mathbb {X} ^ {(1)} D w - y _ {1: m / 2} \right\| _ {2} ^ {2} + 8 \lambda_ {n - d} \log (8 n / \delta) \| w \| _ {1} ^ {2} + 2 \sqrt {2 \lambda_ {n - d} \log (8 n / \delta)} \left\| y _ {1: m / 2} \right\| _ {2} \| w \| _ {1} \tag {6}
$$
where  $\mathbb{X}^{(1)}:(m / 2)\times n$  is the matrix with rows  $X_{1},\ldots ,X_{m / 2}$  , and define  $\hat{v} = \hat{D}\hat{w} (\hat{D})$  where
$$
\hat {D} \in \operatorname * {a r g m i n} _ {D \in \mathcal {D}} \left\| \mathbb {X} ^ {(2)} D \hat {w} (D) - y _ {m / 2 + 1: m} \right\| _ {2} ^ {2}
$$
where  $\mathbb{X}^{(2)}:(m / 2)\times n$  is the matrix with rows  $X_{m / 2 + 1},\ldots ,X_{m}$
Let  $n_{\text{eff}} \coloneqq t^2 \log(t) + t \log(d_l) + d_h + \log(48/\delta)$  and let  $r_{\text{eff}} \coloneqq t(\lambda_{n - d_h} / \lambda_{d_l + 1}) \log(8n/\delta)$ . There are absolute constants  $c, C > 0$  so that the following holds. If  $m \geq C n_{\text{eff}}$ , then with probability at least  $1 - 3\delta$  it holds that
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq c \left(\frac {\sigma^ {2} n _ {e f f}}{m} + \| v ^ {*} \| _ {\Sigma} ^ {2} \left(\frac {r _ {e f f}}{m} + \sqrt {\frac {r _ {e f f}}{m}}\right) + \sigma \| v ^ {*} \| _ {\Sigma} \sqrt {\frac {r _ {e f f}}{m}}\right).
$$