SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jonathan A. Kelner*  
MIT
Frederic Koehler† Stanford
Raghu Meka\* UCLA
Dhruv Rohatgi\* MIT
Sparse linear regression is a central problem in high-dimensional statistics. We study the correlated random design setting, where the covariates are drawn from a multivariate Gaussian  $N(0,\Sigma)$ , and we seek an estimator with small excess risk.
If the true signal is  $t$ -sparse, information-theoretically, it is possible to achieve strong recovery guarantees with only  $O(t\log n)$  samples. However, computationally efficient algorithms have sample complexity linear in (some variant of) the condition number of  $\Sigma$ . Classical algorithms such as the Lasso can require significantly more samples than necessary even if there is only a single sparse approximate dependency among the covariates.
We provide a polynomial-time algorithm that, given  $\Sigma$ , automatically adapts the Lasso to tolerate a small number of approximate dependencies. In particular, we achieve near-optimal sample complexity for constant sparsity and if  $\Sigma$  has few "outlier" eigenvalues. Our algorithm fits into a broader framework of feature adaptation for sparse linear regression with ill-conditioned covariates. With this framework, we additionally provide the first polynomial-factor improvement over brute-force search for constant sparsity  $t$  and arbitrary covariance  $\Sigma$ .
Sparse linear regression is a fundamental problem in high-dimensional statistics. In a natural random design formulation of this problem, we are given  $m$  independent and identically distributed samples  $(X_{i},y_{i})_{i = 1}^{m}$  where each sample's covariates are drawn from an  $n$ -dimensional Gaussian random vector  $X_{i}\sim N(0,\Sigma)$ , and each response is  $y_{i} = \langle X_{i},v^{*}\rangle +\xi_{i}$  for independent noise  $\xi_{i}\sim N(0,\sigma^{2})$  and a  $t$ -sparse ground truth regressor  $v^{*}\in \mathbb{R}^{n}$ , where  $t$  is much smaller than  $n$ . The goal is to output a vector  $\hat{v}\in \mathbb{R}^n$  for which the excess risk
$$
\mathbb {E} \left(\left\langle X _ {0}, \hat {v} \right\rangle - y _ {0}\right) ^ {2} - \sigma^ {2} = \left(\hat {v} - v ^ {*}\right) ^ {\top} \Sigma \left(\hat {v} - v ^ {*}\right) =: \| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2}
$$
is as small as possible, where  $(X_0,y_0)$  is an independent sample from the same model.
Without the sparsity assumption, the number of samples needed to achieve small excess risk (say,  $O(\sigma^2)$ ) is linear in the dimension; with  $O(n)$  samples, simple and computationally efficient algorithms such as ordinary least squares achieve the statistically optimal excess risk  $O\left(\frac{\sigma^2n}{m}\right)$ . Sparsity
*kelner@mit.edu. This work was supported in part by NSF Large CCF-1565235, NSF Medium CCF-1955217, and NSF TRIPODS 1740751.
†fkoehler@stanford.edu. This work was supported in part by NSF award CCF-1704417, NSF award IIS-1908774, and N. Anari's Sloan Research Fellowship
$^{\ddagger}$ raghum@cs.ucla.edu. This work was supported in part by NSF CAREER Award CCF-1553605 and NSF Small CCF-2007682
$^{\S}$ drohatgi@mit.edu. This work was supported by a U.S. DoD NDSEG Fellowship.
5More generally, from a learning theory perspective, we could consider an arbitrary improper learner outputting a function  $\hat{f} (X_0)$  , rather than specifically learning a linear function  $\langle X_0,\hat{v}\rangle$  . At least when  $\Sigma$  is known, there is no advantage as we can always project  $\hat{f}$  onto the space of linear functions.
allows for a significant statistical improvement: ignoring computational efficiency, it is well known that there is an estimator  $\hat{v}$  with excess risk  $O\left(\frac{\sigma^2t\log n}{m}\right)$  as long as  $m = \Omega (t\log n)$  (see e.g. [13, 33]; Theorem 4.1 in [23]).
The catch is that computing this estimator involves a brute-force search over  $\binom{n}{t}$  possibilities (i.e., the possible supports for  $v^{*}$ ). At first glance, this combinatorial search may seem unavoidable if we wish to take advantage of sparsity. Indeed, similar problems are notoriously difficult: the only non-trivial algorithms for e.g., learning  $t$ -sparse parities with noise still require  $n^{\Omega(t)}$  time [29, 37]. However, it is a celebrated fact that for sparse linear regression, computationally efficient methods such as Lasso and Orthogonal Matching Pursuit can avoid this combinatorial search and still achieve very strong theoretical guarantees under conditions such as the Restricted Isometry Property (see e.g. [7, 10, 5, 4, 3, 1]). In the random design setting we consider, the Lasso is known to achieve optimal statistical rates (up to constants) when the covariance matrix  $\Sigma$  is well-conditioned [32, 46].
What about when  $\Sigma$  is ill-conditioned? In contrast with the statistically optimal estimator, Lasso and its cousins provably require sample complexity scaling with (some variant of) the condition number of  $\Sigma$  (see e.g. Theorem 14 in [38] or Theorem 6.5 in [23]). And with a few exceptions (e.g., in some settings with special graphical structure [23]) there has been little progress on designing new efficient algorithms for sparse linear regression with ill-conditioned  $\Sigma$  (see Section 4 for further discussion). For a general covariance  $\Sigma$ , no algorithm is even known that can achieve sample complexity  $f(t)\cdot n^{1 - \epsilon}$  (for an arbitrary function  $f$ ) without brute-force search.
A computationally efficient algorithm that approaches the optimal statistical rate for arbitrary  $\Sigma$  might be too much to hope for. While no computational lower bounds are known, even in restricted computational models such as the Statistical Query model,[6] the related worst-case problem of finding a  $t$ -sparse solution to a system of linear equations requires  $n^{\Omega(t)}$  time under standard complexity assumptions [15]. So it is plausible, though not certain, that some assumptions on  $\Sigma$  are necessary. In this work - inspired by a long tradition (in random matrix theory, statistics, graph theory, and other areas) of studying matrices with a spectrum that is split between a large "bulk" and a small number of outlier "spike" eigenvalues [28, 39, 43] - we identify a broad generalization of the standard well-conditionedness assumption, under which brute-force search can still be avoided.
1.1 Beyond well-conditioned  $\Sigma$
Say that  $\Sigma$  has eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ , and that the sparsity  $t$  is a constant. Then standard bounds for Lasso require sample complexity  $(\lambda_n / \lambda_1) \cdot O(\log n)$ . But if the covariates contain even a single approximate linear dependency, then  $\lambda_n / \lambda_1$  may be arbitrarily large. Moreover, if the dependency is sparse (e.g., two covariates are highly correlated), then there is a natural choice of  $v^*$  for which Lasso provably fails (see Theorem 6.5 of [23]). Indeed, this phenomenon is not just a limitation of the analysis; Lasso fails empirically as well, even for very small  $t$  (see Figure 2 in Appendix H for a simple example with  $t = 3$ ).
Such dependencies arise in applications ranging from finance (e.g., where some pairs of stocks or ETFs may be highly correlated, and an investor may be interested in the differences) to genomic data (where functionally related genes may have highly correlated expression patterns). Two-sparse dependencies can be directly identified by looking at the covariance matrix; see Section 4 for some discussion of previous research in this direction. But as  $t$  increases, naive methods for identifying  $t$ -sparse dependencies quickly become computationally intractable. With domain knowledge, it may be possible to manually identify and correct such dependencies, but this process would also be time-consuming. Thus, we ask the following question: instead of assuming that  $\lambda_{n} / \lambda_{1}$  is bounded, suppose that there are constants  $d_{\ell}$  and  $d_{h}$  so that  $\lambda_{n - d_h} / \lambda_{d_\ell +1}$  is bounded, i.e. the spectrum of  $\Sigma$  has only  $d_{\ell}$  outliers at the low end, and only  $d_{h}$  outliers at the high end. Can we still design an algorithm that achieves sample complexity  $O(\log n)$  without resorting to brute-force search?
Main result. We give a positive answer: an algorithm for sparse linear regression that is both computationally and statistically efficient for covariance matrices with a small number of "outlier"
There are lower bounds for a family of regression estimators with coordinate-separable regularization [44] and a family of "preconditioned-Lasso" estimators [23, 24].
Note that for moderate-sized datasets (e.g.  $n = 1000$ ), brute-force search is infeasible even for  $t$  as small as four or five.
eigenvalues. In particular, this means we can handle a few approximate dependencies among the covariates (quantified by the number of eigenvalues below a threshold). In comparison, Lasso and other classical algorithms cannot tolerate even a single sparse approximate dependency. Our main algorithmic result is the following:
Theorem 1.1. Let  $n, t, d_{\ell}, d_{h}, L \in \mathbb{N}$  and  $\sigma, \delta > 0$ . Let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive semi-definite matrix with (non-negative) eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Let  $v^* \in \mathbb{R}^n$  be any  $t$ -sparse vector. Let  $(X_i, y_i)_{i=1}^m$  be independent with  $X_i \sim N(0, \Sigma)$  and  $y_i = \langle X_i, v^* \rangle + \xi_i$ , where  $\xi_i \sim N(0, \sigma^2)$ .
Let  $n_{eff} \coloneqq t(\lambda_{n - d_h} / \lambda_{d_\ell + 1})\log (nL / \delta) + t^{O(t)}d_l + d_h$ . Given  $\Sigma, t, d_\ell, \delta$ , and  $(X_i, y_i)_{i = 1}^m$ , there is an estimator  $\hat{v} \in \mathbb{R}^n$  that has excess risk
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} n _ {e f f} L}{m}\right) + 2 ^ {- L} \cdot \| v ^ {*} \| _ {\Sigma} ^ {2}
$$
with probability at least  $1 - \delta$ , so long as  $m \geq \Omega(n_{\mathrm{eff}}L)$ . Moreover,  $\hat{v}$  can be computed in time  $\mathrm{poly}(n)$ .
Specifically, taking  $L \sim \log(m\|v^{*}\|_{\Sigma}^{2}/\sigma^{2})$ , the time complexity is dominated by  $L$  eigendecompositions and  $L$  calls to a Lasso program, for overall runtime  $\tilde{O}(n^{3})$  (see Algorithm 2). This is substantially faster than the brute-force method (which takes  $O(n^{t})$  time) even for small values of  $t$ .
The excess risk decays at rate  $\tilde{O}(\sigma^2 n_{\mathrm{eff}} / m)$  (hiding the logarithmic factor), which is near the statistically optimal rate of  $\tilde{O}(\sigma^2 t / m)$  so long as  $n_{\mathrm{eff}}$  is small, i.e.  $t$  is small and only a few eigenvalues lie outside a constant-factor range. In our analysis, we prove that the standard Lasso estimator can already tolerate a few large eigenvalues — the main algorithmic innovation is needed to tolerate a few small eigenvalues, which turns out to be much trickier. Notice that when  $d_\ell = d_h = 0$  we recover standard Lasso guarantees up to the factor of  $L$ ; thus, Theorem 1.1 morally represents a generalization of classical results.
We also show how to achieve a different trade-off between time and samples, eliminating the dependence on  $d_{\ell}$  in sample complexity at the cost of larger runtime:
Theorem 1.2. In the setting of Theorem 1.1, let  $n_{\text{eff}}' \coloneqq t(\lambda_{n - d_h} / \lambda_{d_\ell + 1}) \log (nL / \delta) + t^2 \log (t) + d_h$ . Given  $\Sigma, t, d_\ell, \delta$ , and  $(X_i, y_i)_{i=1}^m$ , there is an estimator  $\hat{v} \in \mathbb{R}^n$  that has excess risk
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} n _ {e f f} ^ {\prime} L}{m}\right) + 2 ^ {- L} \cdot \| v ^ {*} \| _ {\Sigma} ^ {2}
$$
with probability at least  $1 - \delta$ , so long as  $m \geq \Omega(n_{\text{eff}}' L)$ . Moreover,  $\hat{v}$  can be computed in time  $\mathrm{poly}(n, m, d_{\ell}^{t}, t^{t^{2}})$ .
Discussion & limitations. We discuss two limitations of the above results. First, both results incur exponential dependence on the sparsity  $t$  (in the sample complexity for Theorem 1.1, and the runtime for Theorem 1.2), which may be suboptimal. For Theorem 1.1, we remark that in practice the algorithm may not suffer this dependence (see e.g. Figure 1), and it is possible that the analysis can be tightened. For Theorem 1.2, we emphasize that the runtime is still fundamentally different than brute-force search: in particular, it's fixed-parameter tractable in  $t$  and  $d_{\ell}$ .
Second, both results require that  $\Sigma$  is known. Thus, they are only applicable in settings where we either have a priori knowledge, or can estimate  $\Sigma$  accurately because a large amount of unlabelled data is available. At a high level, this limitation is due to the need to compute the eigendecomposition of  $\Sigma$ , which cannot be approximated from the empirical covariance of a small number of samples.
For simplicity, we have stated our results in terms of Gaussian covariates and noise, but this is not a fundamental limitation. We expect it is possible to prove similar results in the sub-Gaussian case at the cost of making the proof longer — for instance, by building upon the techniques from [25] and related works.
Pseudocode & simulation. See Algorithm 1 for complete pseudocode of AdaptedBP(), a simplification of the method for the noiseless setting  $\sigma = 0$ . In Figure 1 we show that AdaptedBP() significantly outperforms standard Basis Pursuit (i.e. Lasso for noiseless data [7]) on a simple example with  $n = 1000$  variables,  $d_{\ell} = 10$  sparse approximate dependencies, and a ground truth
Algorithm 1: Adapted BP for sparse linear regression with few outlier eigenvalues
Procedure FindHeavyCoordinates  $\left(\left\{v_{1},\ldots ,v_{k}\right\} ,\alpha\right)$  /\*GRAM-SCHMIDT computes an orthonormalization of  $v_{1},\dots,v_{k}$  \*/  $a_1,\ldots ,a_k\gets$  GRAM-SCHMIDT(  $\{\{v_1,\dots ,v_k\})$  return  $\{i\in [n]:\sum_{j = 1}^{k}((a_j)_i)^2\geq \alpha^2\}$    
Procedure IterativePeeling  $(\Sigma ,d,t)$  Compute eigendecomposition  $\Sigma = \sum_{i = 1}^{n}\lambda_{i}u_{i}u_{i}^{\top}$ $P\gets \sum_{i = d + 1}^{n}u_{i}u_{i}^{\top}$ $K_{t}\gets \{i\in [n]:P_{ii} <   1 - 1 / (9t^{2})\}$  for  $j = t$  to 1 do  $\begin{array}{rlr}{\mathcal{I}_P(K_j)} & {\leftarrow} & {\mathrm{FindHeavyCoordinates}(\{P_i:i\in K_j\} ,1 / (6t))}\\ {K_{j - 1}} & {\leftarrow} & {K_j\cup \mathcal{I}_P(K_j)}\\ {\mathrm{return}} & {K_0} \end{array}$    
ProcedureAdaptedBP  $(\Sigma ,d,t,(X_i,y_i)_{i = 1}^m)$ $S\gets$  IterativePeeling  $(\Sigma ,d,t)$    
return  $\hat{v}\in \operatorname *{argmin}_{v\in \mathbb{R}^n:\mathbb{X}v = y}\sum_{i\notin S}|v_i|$
Figure 1: Basis Pursuit (BP) versus Adapted BP in a simple synthetic example with  $n = 1000$  covariates. The  $x$ -axis is the number of samples. The  $y$ -axis is the out-of-sample prediction error (averaged over 10 independent runs, and error bars indicate the standard deviation).
regressor with sparsity  $t = 13$ . The covariates  $X_{1:1000}$  are all independent  $N(0,1)$  except for 10 disjoint triplets  $\{(X_i, X_{i+1}, X_{i+2}) : i = 1, 4, \ldots, 28\}$ , each of which has joint distribution
$$
X _ {i} := Z _ {i}; \quad X _ {i + 1} = Z _ {i} + 0. 4 Z _ {i + 1}; \quad X _ {i + 2} = Z _ {i + 1} + 0. 4 Z _ {i + 2}
$$
where  $Z_{i}, Z_{i+1}, Z_{i+2} \sim N(0,1)$  are independent. The (noiseless) responses are  $y = 6.25(X_{1} - X_{2}) + 2.5X_{3} + \frac{1}{\sqrt{10}}\sum_{i=991}^{1000}X_{i}$ . See Appendix I for implementation details.
1.2 Organization
In Section 2 we give an overview of the proofs of Theorem 1.1 and 1.2 (the complete proofs and full algorithm pseudocode are given in Appendix C). In Section 3 we discuss our other results obtained via feature adaptation. Section 4 covers related work.
2 Proof techniques
We obtain Theorems 1.1 and 1.2 as outcomes of a flexible algorithmic approach for tackling sparse linear regression with ill-conditioned covariates: feature adaptation. As a pre-processing step, adapt
or augment the covariates with additional features (i.e. well-chosen linear combinations of the covariates). Then, to predict the responses, apply  $\ell_1$ -regularized regression (Lasso) over the new set of features rather than the original covariates. In other words, we algorithmically change the dictionary (set of features) used in the Lasso regression. See Section 4 for a comparison to past approaches.
We start by explaining the goals of feature adaptation for general  $\Sigma$ , and then show how we achieve those desiderata when  $\Sigma$  has few outlier eigenvalues. More precisely, the main technical difficulty is in dealing with the small eigenvalues, so in this proof overview we focus on the case where the only outliers are small eigenvalues. Complete proofs of Theorems 1.1 and 1.2 are in Appendix C.
2.1 What makes a good dictionary: the view from weak learning
Obviously, the feature adaptation approach generalizes Lasso. Surprisingly, even though the sample complexity of the standard Lasso estimator is thoroughly understood, the basic question of whether for every covariate distribution (i.e. every  $\Sigma$ ) there exists a good dictionary remains wide-open. To crystallize the power of feature adaptation, we introduce the following notion of a "good" dictionary. We suggest considering the simplified setting of  $\alpha$ -weak learning, where the goal is just to find some  $\hat{v}$  so that the predictions  $\langle X, \hat{v} \rangle$  are  $\alpha$ -correlated with the ground truth  $\langle X, v^* \rangle$  when  $X \sim N(0, \Sigma)$ . Moreover, we focus first on the existential question (rather than the algorithmic question of finding the dictionary). We will return to the setting of Theorems 1.1 and 1.2 later. For now, in the weak learning setting, a good dictionary (when the covariate distribution is  $N(0, \Sigma)$ ) is one that satisfies the following covering property, but is not too large:
Definition 2.1. Let  $\Sigma \in \mathbb{R}^{n\times n}$  be a positive semi-definite matrix and let  $t,\alpha >0$ . A set  $\{D_1,\ldots ,D_N\} \subseteq \mathbb{R}^n$  is a  $(t,\alpha)$ -dictionary for  $\Sigma$  if for every  $t$ -sparse  $v\in \mathbb{R}^n$ , there is some  $i\in [N]$  with
$$
| \langle v, D _ {i} \rangle_ {\Sigma} | \geq \alpha \| v \| _ {\Sigma} \| D _ {i} \| _ {\Sigma},
$$
where we define  $\langle x,y\rangle_{\Sigma}:= x^{\top}\Sigma y$  and  $\| x\|_{\Sigma}^{2}:= x^{\top}\Sigma x$  for any  $x,y\in \mathbb{R}^n$ . Let  $\mathcal{N}_{t,\alpha}(\Sigma)$  be the size of the smallest  $(t,\alpha)$ -dictionary.
The relevance of the covering number  $\mathcal{N}_{t,\alpha}(\Sigma)$  is quite simple: given a  $(t,\alpha)$ -dictionary  $\mathcal{D}$  for  $\Sigma$ , and given samples  $(X_{i},y_{i})_{i = 1}^{m}$ , the weak learning algorithm can simply output the vector  $\hat{v}\in \mathcal{D}$  that maximizes the empirical correlation between the predictions  $\langle X_i,\hat{v}\rangle$  and the responses  $y_{i}$ . So long as there are enough samples for empirical correlations to concentrate, Definition 2.1 guarantees success. Formally, allowing for preprocessing time to compute the dictionary,  $O(\alpha)$ -weak learning is possible in time  $\mathcal{N}_{t,\alpha}(\Sigma)\cdot \mathrm{poly}(n)$ , with  $O(\alpha^{-2}\log \mathcal{N}_{t,\alpha}(\Sigma))$  samples (Proposition A.5).
Hypothetically, bounding  $\mathcal{N}_{t,\alpha}(\Sigma)$  may not be necessary to develop an efficient sparse linear regression algorithm. However, all assumptions on  $\Sigma$  that are currently known to enable efficient sparse linear regression also immediately imply bounds on  $\mathcal{N}_{t,\alpha}$  (see Appendix G). For example, when  $\Sigma$  is well-conditioned, the standard basis is a good dictionary of size  $n$  (Fact A.4).
In contrast, the only known bounds for arbitrary  $\Sigma$  (until the present work) are  $\mathcal{N}_{t,1 / \sqrt{t}}(\Sigma)\leq t\cdot \binom{n}{t}$  (the brute-force dictionary, which includes a  $\Sigma$ -orthonormal basis for every set of  $t$  covariates) and  $\mathcal{N}_{t,1 / \sqrt{n}}(\Sigma)\leq n$  (a  $\Sigma$ -orthonormal basis for all  $n$  covariates, which doesn't take advantage of sparsity and corresponds to algorithms such as Ordinary Least Squares). Thus, the following basic question - when can we improve upon these trivial bounds - seems central to understanding when brute-force search can be avoided in sparse linear regression:
Question 2.2. How large is  $\mathcal{N}_{t,\alpha}(\Sigma)$  for an arbitrary positive semi-definite  $\Sigma \in \mathbb{R}^{n\times n}$ ? Are there natural families of ill-conditioned  $\Sigma$  (and functions  $f,g$ ) for which  $\mathcal{N}_{t,1 / f(t)}(\Sigma)\leq g(t)\cdot \mathrm{poly}(n)$ ?
2.2 Constructing a good dictionary when  $\Sigma$  has few small eigenvalues
We now address Question 2.2 in the setting where  $\Sigma$  has a small number of eigenvalues that are much smaller than  $\lambda_{n}$ . In this setting, the standard basis may not be a good dictionary. For example, if two covariates are highly correlated, their difference may not be correlated with any of them. Nonetheless, we can prove the following covering number bound:
Theorem 2.3. Let  $n, t, d \in \mathbb{N}$ . Let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive semi-definite matrix with eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Then  $\mathcal{N}_{t,\alpha}(\Sigma) \leq t(7t)^{2t^2 + t}d^t + n$ , where  $\alpha = \frac{1}{7\sqrt{t}}\sqrt{\lambda_{d+1} / \lambda_n}$ .
In particular, when  $t = O(1)$  and  $\Sigma$  is well-conditioned except for  $O(1)$  outliers  $\lambda_1, \ldots, \lambda_d$ , we get a linear-size dictionary just as in the case where  $\Sigma$  is well-conditioned. In fact, the desired  $(t, \alpha)$ -dictionary can be constructed efficiently. Our key lemma shows that when  $\Sigma$  has few small eigenvalues, there is a small subset of covariates that "causes" all of the sparse approximate dependencies – in the sense that the  $\ell_2$  norm of any sparse vector, excluding the mass on the subset, can be upper bounded in terms of the  $\Sigma$ -norm of the vector. Moreover, there is an efficient algorithm that finds a superset of these covariates. Formally, we prove the following:
Lemma 2.4. Let  $n, t, d \in \mathbb{N}$ . Let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive semi-definite matrix with eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Given  $\Sigma, d,$  and  $t$ , there is a polynomial-time algorithm IterativePeeling() producing a set  $S \subseteq [n]$  with the following guarantees:
(a) For every  $t$ -sparse  $v \in \mathbb{R}^n$ , it holds that  $\left\| v_{[n] \setminus S} \right\|_2 \leq 3\lambda_{d+1}^{-1/2} \| v \|_\Sigma$ .
(b)  $|S|\leq (7t)^{2t + 1}d.$
Once this set  $S$  has been found, the dictionary is simply the standard basis  $\{e_1, \ldots, e_n\}$ , together with a  $\Sigma$ -orthonormal basis for every set of  $t$  covariates in  $S$ . By guarantee (a), we can prove that every  $t$ -sparse vector correlates with some element of this dictionary under the  $\Sigma$ -inner product. By guarantee (b), the dictionary is much smaller than the brute-force dictionary that contains a basis for all  $\binom{n}{t}$  sets of  $t$  covariates. Together, this gives an algorithmic proof for Theorem 2.3.
Intuition for IterativePeeling(). We compute the set  $S$  via a new iterative method which leverages knowledge of the small eigenspaces of  $\Sigma$ . See Algorithm 1 for the pseudocode. To compute  $S$ , the algorithm IterativePeeling() first computes the orthogonal projection matrix  $P$  that projects onto the subspace spanned by the top  $n - d$  eigenvectors of  $\Sigma$ . Starting with the set of coordinates that correlate with  $\ker(P)$ , the procedure then iteratively grows  $S$  in such a way that at each step, a new participant of each approximate sparse dependency is discovered, but  $S$  does not become too much larger.