SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The intuition is as follows: as a preliminary attempt, we could identify all  $O(d)$  coordinates that correlate (with respect to the standard inner product) with the lowest  $d$  eigenspaces of  $\Sigma$ . If e.g. the covariates have a sparse dependency
$$
X _ {1} + X _ {2} = 0,
$$
then  $\ker \Sigma$  contains the vector  $e_1 + e_2$ , so the coordinates  $\{e_1, e_2\}$  will be correctly discovered. Unfortunately, if  $\Sigma$  contains a more complex sparse dependency such as
$$
\epsilon^ {- 1} \left(X _ {1} - X _ {2}\right) - X _ {3} - X _ {4} = 0
$$
where  $\epsilon > 0$  is very small, then this heuristic will discover  $\{e_1, e_2\}$  but miss  $\{e_3, e_4\}$ . For this example, the solution is to notice that  $e_3$  and  $e_4$  do correlate with the subspace spanned by  $\ker(\Sigma) \cup \{e_1, e_2\}$  (which contains  $e_3 + e_4$ ). In general, if  $S$  is the set of coordinates discovered thus far, then by finding basis vectors that correlate with an appropriate subspace (of dimension at most  $|S|$ ), we can efficiently augment  $S$  with at least one new coordinate from each  $t$ -sparse approximate dependency, without making  $S$  bigger by more than a factor of  $O(t)$ . Iterating this augmentation  $t$  times therefore provably identifies all problematic coordinates.
To formalize this intuition, the following lemma will be needed to bound how much  $S$  grows at each iteration; it shows that the number of coordinates that correlate with a low-dimensional subspace is not too large (proof deferred to Appendix B):
Lemma 2.5. Let  $V \subseteq \mathbb{R}^n$  be a subspace with  $d \coloneqq \dim V$ . For some  $\alpha > 0$  define
$$
S = \left\{i \in [ n ]: \sup  _ {x \in V \setminus \{0 \}} \frac {x _ {i}}{\| x \| _ {2}} \geq \alpha \right\}.
$$
Then  $|S| \leq d / \alpha^2$ . Moreover, given a set of vectors that span  $V$ , we can compute  $S$  in time poly(n).
We also define the set of vectors  $v$  that have unusually large norm outside a set  $S$ , compared to  $\sqrt{v^{\top}Pv}$ , which is the distance from  $v$  to the subspace spanned by the bottom  $d$  eigenvectors of  $\Sigma$ :
Definition 2.6. For any matrix  $P \in \mathbb{R}^{n \times n}$  and subset  $S \subseteq [n]$ , define  $\mathcal{W}_{P,S} := \{v \in \mathbb{R}^n : \| v_{S^c} \|_2 > 3\sqrt{v^\top Pv}\}$ .
We then formalize the guarantee of each iteration of IterativePeeling() as follows:
Lemma 2.7. Let  $n, t \in \mathbb{N}$  and let  $P: n \times n$  be an orthogonal projection matrix. Suppose  $\tau \geq 1$  and  $K \subseteq [n]$  satisfy
(a)  $P_{ii}\geq 1 - 1 / (9t^2)$  for all  $i\notin K$
(b)  $|\operatorname {supp}(v)\setminus K|\leq \tau$  for every  $v\in B_0(t)\cap \mathcal{W}_{P,K}$
Then there exists a set  $\mathcal{I}_P(K)$  with  $|\mathcal{I}_P(K)|\leq 36t^2 |K|$  such that
$$
| \operatorname {s u p p} (v) \setminus (\mathcal {I} _ {P} (K) \cup K) | \leq \tau - 1
$$
for all  $v \in B_0(t) \cap \mathcal{W}_{P,K}$ . Moreover, given  $P, K$ , and  $t$ , we can compute  $\mathcal{I}_P(K)$  in time poly(n).
Proof sketch. We define the set
$$
\mathcal{I}_{P}(K):= \left\{a\in [n]\setminus K:\sup_{x\in \operatorname {span}\{Pe_{i}:i\in K\} \setminus \{0\}}\frac{|x_{a}|}{\|x\|_{2}}\geq 1 / (6t)\right\} .
$$
It is clear from Lemma B.2 (applied with parameters  $V \coloneqq \operatorname{span}\{Pe_i : i \in K\}$  and  $\alpha \coloneqq 1/(6t)$ ) that  $|\mathcal{I}_P(K)| \leq 36t^2|K|$ , and that  $\mathcal{I}_P(K)$  can be computed in time poly(n). It remains to show that  $|\mathcal{G}_P(v) \setminus (\mathcal{I}_P(K) \cup K)| \leq \tau - 1$  for all  $v \in B_0(t)$ .
Consider any  $v \in B_0(t) \cap \mathcal{W}_{P,K}$ . Then  $\| v_{K^c}\| _2 > 3\| Pv\| _2$ . It's sufficient to show that  $\mathcal{I}_P(K)$  contains some  $j \in \mathrm{supp}(v) \setminus K$ , i.e. that there is some  $j \in \mathrm{supp}(v) \setminus K$  such that  $e_j$  correlates with  $\operatorname{span}\{P_i : i \in K\}$ . We accomplish this by showing that  $v_{K^c}$  correlates with  $Pv_K = \sum_{i \in K} v_i P_i$ .
At a high level, the reason for this is that  $v_{K^c}$  is close to  $Pv_{K^c}$  (since  $P_i \approx e_i$  for  $i \in K^c$ ), and  $Pv = Pv_K + Pv_{K^c}$  is much smaller than  $Pv_{K^c} \approx v_{K^c}$ , so  $Pv_K$  and  $Pv_{K^c}$  must be highly correlated. See Appendix B for the full proof.
We can now complete the proof of Lemma 2.4 by repeatedly invoking Lemma B.4.
Proof of Lemma 2.4. Let  $\Sigma = \sum_{i=1}^{n} \lambda_i u_i u_i^\top$  be the eigendecomposition of  $\Sigma$ , and let  $P := \sum_{i=d+1}^{n} u_i u_i^\top$  be the projection onto the top  $n-d$  eigenspaces of  $\Sigma$ . Set  $K_t = \{i \in [n] : P_{ii} < 1 - 1/(9t^2)\}$ . Because  $\operatorname{tr}(P) = n-d$  and  $P_{ii} \leq 1$  for all  $i \in [n]$ , it must be that  $|K_t| \leq 9t^2d$ . Also, for any  $v \in B_0(t) \cap \mathcal{W}_{P,K_t}$  we have trivially by  $t$ -sparsity that  $|\operatorname{supp}(v) \setminus K_t| \leq t$ .
Define  $K_{t-1}$  to be  $K_t \cup \mathcal{I}_P(K_t)$  where  $\mathcal{I}_P(K_t)$  is as defined in Lemma B.4; we have the guarantees that  $|K_{t-1}| \leq (1 + 36t^2)|K_t|$  and  $|\mathcal{G}_P(v) \setminus K_t| \leq t-1$  for all  $v \in B_0(t) \cap \mathcal{W}_{P,K_t}$ . Since  $K_{t-1} \supseteq K_t$ , it holds that  $\mathcal{W}_{P,K_{t-1}} \subseteq \mathcal{W}_{P,K_t}$ , and thus  $|\mathcal{G}_P(v) \setminus K_t| \leq t-1$  for all  $v \in B_0(t) \cap \mathcal{W}_{P,K_{t-1}}$ . Moreover, since  $K_{t-1} \supseteq K_t$ , it obviously holds that  $P_{ii} \geq 1 - 1/(9t^2)$  for all  $i \notin K_{t-1}$ . This means we can apply Lemma B.4 with  $\tau := t-1$  and  $K := K_{t-1}$  and so iteratively define sets  $K_{t-2} \subseteq \dots \subseteq K_1 \subseteq K_0 \subseteq [n]$  in the same way. In the end, we obtain the set  $K_0 \subseteq [n]$  with  $|K_0| \leq 9t^2 d(1 + 36t^2)^t$  and  $\operatorname{supp}(v) \subseteq K_0$  for all  $v \in B_0(t) \cap \mathcal{W}_{P,K_0}$ . The latter guarantee means that in fact  $B_0(t) \cap \mathcal{W}_{P,K_0} = \emptyset$ . So for any  $t$ -sparse  $v \in \mathbb{R}^n$  it holds that
$$
\left\| v _ {K _ {0} ^ {c}} \right\| _ {2} \leq 3 \sqrt {v ^ {\top} P v} \leq 3 \lambda_ {d + 1} ^ {- 1 / 2} \sqrt {v ^ {\top} \Sigma v}
$$
where the last inequality holds since  $\lambda_{d + 1}P\preceq \Sigma$
2.3 Beyond weak learning
So far, we have sketched a proof that if  $\Sigma$  has few outlier eigenvalues, then there is an efficient algorithm to compute a good dictionary (as in Theorem 2.3). This gives an efficient  $\alpha$ -weak learning algorithm (via Proposition A.5). However, our ultimate goal is to find a regressor  $\hat{v}$  with prediction error going to 0 as the number of samples increases. Definition 2.1 is not strong enough to ensure this.<sup>8</sup> However, it turns out that the dictionary constructed in Theorem 2.3 in fact satisfies a stronger guarantee<sup>9</sup> that is sufficient to achieve vanishing prediction error:
Moreover, standard notions of boosting weak learners (e.g. in distribution-free classification) do not apply in this setting.
See Lemma A.3 for a proof that the  $\ell_1$ -representation property implies the  $(t,\alpha)$ -dictionary property.
Definition 2.8. Let  $\Sigma \in \mathbb{R}^{n\times n}$  be a positive semi-definite matrix and let  $t,B > 0$ . A set  $\{D_1,\ldots ,D_N\} \subseteq \mathbb{R}^n$  is a  $(t,B)$ - $\ell_{1}$ -representation for  $\Sigma$  if for any  $t$ -sparse  $v\in \mathbb{R}^n$  there is some  $\alpha \in \mathbb{R}^N$  with  $v = \sum_{i = 1}^{N}\alpha_{i}D_{i}$  and  $\sum_{i = 1}^{N}|\alpha_{i}|\cdot \| D_{i}\|_{\Sigma}\leq B\cdot \| v\|_{\Sigma}$ .
With this definition in hand, we can actually prove the following strengthening of Theorem 2.3:
Lemma 2.9. Let  $n, t, d \in \mathbb{N}$ . Let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive semi-definite matrix with eigenvalues  $\lambda_1 \leq \dots \leq \lambda_n$ . Then  $\Sigma$  has a  $(t, 7\sqrt{t}\sqrt{\lambda_n / \lambda_{d + 1}})$ - $\ell_1$ -representation  $\mathcal{D}$  of size at most  $n + t(7t)^{2t^2 + t}d^t$ . Moreover,  $\mathcal{D}$  can be computed in time  $t^{O(t^2)}d^t$  poly(n).
Proof sketch. Let  $S$  be the output of IterativePeeling  $(\Sigma, d, t)$ . The dictionary  $\mathcal{D}$  consists of the standard basis, together with a  $\Sigma$ -orthogonal basis for each set of  $t$  coordinates from  $S$ . The bound on  $|\mathcal{D}|$  comes from the guarantee  $|S| \leq (7t)^{2t + 1}d$ . For any  $t$ -sparse vector  $v \in \mathbb{R}^n$ , we know that  $v_{S^c}$  is efficiently represented by the standard basis (because Theorem B.1 guarantees that  $\| v_{S^c} \|_2 \leq O(\lambda_{d + 1}^{-1/2} \| v \|_{\Sigma})$ ), and  $v_S$  is efficiently represented by one of the  $\Sigma$ -orthonormal bases. See Appendix B for the full proof.
Why is the above guarantee useful? If each  $D_{i}$  is normalized to unit  $\Sigma$ -norm, then the condition of  $(t,B)$ - $\ell_1$ -representability is equivalent to  $\| \alpha \| _1\leq B\cdot \| v\|_{\Sigma}$ . That is, with respect to the new set of features, the regressor  $\alpha$  has bounded  $\ell_1$  norm. Thus, if we apply the Lasso with a set of features that is a  $(t,B)$ - $\ell_1$ -representation for  $\Sigma$ , then standard "slow rate" guarantees hold (proof in Section A):
Proposition 2.10. Let  $n, m, N, t \in \mathbb{N}$  and  $B > 0$ . Let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive semi-definite matrix and let  $\mathcal{D}$  be a  $(t, B)$ - $\ell_1$ -representation of size  $N$  for  $\Sigma$ , normalized so that  $\|v\|_{\Sigma} = 1$  for all  $v \in \mathcal{D}$ . Fix a  $t$ -sparse vector  $v^* \in \mathbb{R}^n$ , let  $X_1, \ldots, X_m \sim N(0, \Sigma)$  be independent and let  $y_i = \langle X_i, v^*\rangle + \xi_i$  where  $\xi_i \sim N(0, \sigma^2)$ . For any  $R > 0$ , define
$$
\hat{w}\in \operatorname *{argmin}_{w\in \mathbb{R}^{N}:  \| w\|_{1}\leq BR}\| \mathbb{X}Dw - y\|_{2}^{2}
$$
where  $D \in \mathbb{R}^{n \times N}$  is the matrix with columns comprising the elements of  $\mathcal{D}$ , and  $\mathbb{X} \in \mathbb{R}^{m \times n}$  is the matrix with rows  $X_{1},\ldots,X_{m}$ . So long as  $m = \Omega(\log(n/\delta))$  and  $\|w^{*}\|_{\Sigma} \in [R/2,R]$ , it holds with probability at least  $1 - \delta$  that
$$
\| D \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} = O \left(B \| w ^ {*} \| _ {\Sigma} \sigma \sqrt {\frac {\log (2 n / \delta)}{m}} + \frac {\sigma^ {2} \log (4 / \delta)}{m} + \frac {B ^ {2} \| w ^ {*} \| _ {\Sigma} ^ {2} \log (n)}{m}\right).
$$
Combining Proposition 2.10 with Lemma 2.9 shows that there is an algorithm with time complexity  $t^{O(t^2)}d^t\mathrm{poly}(n)$  and sample complexity  $O(\mathrm{poly}(t)(\lambda_n / \lambda_{d + 1})\log (n)\log (d))$  for finding a regressor with squared prediction error  $o(\sigma^2 +\| v^*\|_\Sigma^2)$ . This is a simplified version of Theorem 1.2. The full proof involves additional technical details (e.g. more careful analysis to take care of large eigenvalues, and to avoid needing an estimate  $R$  for  $\| w^{*}\|_{\Sigma}$ ) but the above exposition contains the central ideas. Theorem 1.1 similarly computes the set  $S$  from Lemma 2.4 but uses it to construct a different dictionary: the standard basis, plus a  $\Sigma$ -orthonormal basis for  $S$ .<sup>10</sup> See Appendix C for the full proofs and pseudocode.
3 Additional Results
We now return to Question 2.2 and ask whether there are other families of ill-conditioned  $\Sigma$  for which we can prove non-trivial bounds on  $\mathcal{N}_{t,\alpha}(\Sigma)$ .
First, we ask what can be shown for arbitrary covariance matrices. We prove that every covariance matrix  $\Sigma$  satisfies a non-trivial bound  $\mathcal{N}_{t,1 / O(t^{3 / 2}\log n)}(\Sigma)\leq O(n^{t - 1 / 2})$ . In fact, building on tools from computational geometry, we show the stronger result that  $\Sigma$  has a  $(t,O(t^{3 / 2}\log n))$ - $\ell_1$ -representation that of size  $O(n^{t - 1 / 2})$ , that is computable from samples in time  $\tilde{O} (n^{t - \Omega (1 / t)})$  for
10More precisely, the algorithm just skips regularizing  $S$  , which is morally equivalent. As it is simpler to implement, that is shown in Algorithm 1, and analyzed for the proofs.
any constant  $t > 1$  (Theorem D.5). As a corollary, we provide the first sparse linear regression algorithm with time complexity that is a polynomial-factor better than brute force, and with near-optimal sample complexity, for any constant  $t$  and arbitrary  $\Sigma$  (proof in Section D):
Theorem 3.1. Let  $n, m, t, B \in \mathbb{N}$  and  $\sigma > 0$ , and let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive-definite matrix. Let  $w^* \in \mathbb{R}^n$  be  $t$ -sparse, and suppose  $\| w^* \|_{\Sigma} \in [B/2, B]$ . Suppose  $m \geq \Omega(t \log n)$ . Let  $(X_i, y_i)_{i=1}^m$  be independent samples where  $X_i \sim N(0, \Sigma)$  and  $y_i = \langle X_i, w^* \rangle + N(0, \sigma^2)$ . Then there is an  $O(m^2 n^{t-1/2} + n^{t-\Omega(1/t)} \log^{O(t)} n)$ -time algorithm that, given  $(X_i, y_i)_{i=1}^m$ ,  $B$ , and  $\sigma^2$ , produces an estimate  $\hat{w} \in \mathbb{R}^n$  satisfying, with probability  $1 - o(1)$ ,
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} \leq \tilde {O} \left(\frac {\sigma^ {2}}{\sqrt {m}} + \frac {\sigma \| w ^ {*} \| _ {\Sigma}}{\sqrt {m}} t ^ {3 / 2} + \frac {\| w ^ {*} \| _ {\Sigma} ^ {2} t ^ {3}}{m}\right).
$$
Second, one goal is to improve "sample complexity" (i.e. obtain  $\alpha$  without dependence on condition number) without paying too much in "time complexity" (i.e. retain bounds on  $\mathcal{N}_{t,\alpha}$  that are better than  $n^t$ ). To this end, we prove that the dependence on  $\kappa$  in the correlation level (see Fact A.4) can actually be replaced by dependence on  $\kappa$  in the dictionary size (proof in Appendix E):
Theorem 3.2. Let  $n, t \in \mathbb{N}$ . Let  $\Sigma \in \mathbb{R}^{n \times n}$  be a positive-definite matrix with condition number  $\kappa$ . Then  $\mathcal{N}_{t,1/3^{t+1}}(\Sigma) \leq 2^{O(t^2)}\kappa^{2t+1} \cdot n$ .
In particular, for any constant  $t = 1 / \epsilon$ , our result shows that there is a nearly-linear size dictionary with constant correlations even for covariance matrices with polynomially-large condition number  $\kappa \leq n^{\epsilon /100}$ . While we are not currently aware of an efficient algorithm for computing the dictionary, the above bound nonetheless raises the interesting possibility that there may be a sample-efficient and computationally-efficient weak learning algorithm under a super-constant bound on  $\kappa$ .
Dealing with correlated covariates. There is considerable work on improving the performance of Lasso in situations where some clusters of covariates are highly correlated [47, 19, 2, 42, 21, 12, 27]. These methods can work well for two-sparse dependencies, but generally do not work as well for higher-order dependencies — hence they cannot be used to prove our main result. The approach of [2] is perhaps the closest in spirit to ours. They perform agglomerative clustering of correlated covariates, orthonormalize the clusters with respect to  $\Sigma$ , and apply Lasso (or solve an essentially equivalent group Lasso problem). This method fails, for example, when there is a single three-sparse dependency, and the remaining covariates have some mild correlations. Depending on the correlation threshold, their method will either aggressively merge all covariates into a single cluster, or fail to merge the dependent covariates.
Feature adaptation and preconditioning. Generalizations of Lasso via a preliminary change-of-basis (or explicitly altering the regularization term) have been studied in the past, but largely not to solve sparse linear regression per se; instead the goal has been using  $\ell_1$  regularization to encourage other structural properties such as piecewise continuity (e.g. in the "fused lasso", see [35, 36, 20, 8] for some more examples). An exception is recent work showing that a "sparse preconditioning" step can enable Lasso to be statistically efficient for sparse linear regression when the covariates have a certain Markovian structure [23]. Our notion of feature adaptation via dictionaries generalizes sparse preconditioning, which corresponds to choosing a non-standard basis in which  $\Sigma$  becomes well-conditioned and the sparsity of the signal is preserved.
Statistical query (SQ) model; sparse halfspaces. From the complexity standpoint,  $\mathcal{N}_{t,\alpha}(\Sigma)$  is a covering number and therefore closely corresponds to a packing number  $\mathcal{P}_{t,\alpha}(\Sigma)$  (see Section A.1 for the definition). This packing number is essentially the (correlational) statistical dimension, which governs the complexity of sparse linear regression with covariates from  $N(0,\Sigma)$  in the (correlational) SQ model (see e.g. [14] for exposition of this model). Whereas strong  $n^{\Omega(t)}$  SQ lower bounds are known for related problems such as sparse parities with noise [29], no non-trivial (i.e. super-linear) lower bounds are known for sparse linear regression. Relatedly, in a COLT open problem, Feldman asked whether any non-trivial bounds can be shown for the complexity of weak learning sparse halfspaces in the SQ model [11]. Our results also yield improved bounds for weakly SQ-learning sparse halfspaces over certain families of multivariate Gaussian distributions.
Improving brute-force for arbitrary  $\Sigma$ . Several prior works have suggested improvements on brute-force search for variants of  $t$ -sparse linear regression [18, 16, 31, 6]. However, all of these have limitations preventing their application to the general setting we address in Theorem 3.1. Specifically, [18] requires  $\Omega(n^t)$  preprocessing time on the covariates; [16, 31] require noiseless responses; and [6] has time complexity scaling with  $\log^m n$  (since our random-design setting necessitates  $m \geq \Omega(t \log n)$ , their algorithm has time complexity much larger than  $n^t$ ).
[42] Fabian L Wauthier, Nebojsa Jojic, and Michael I Jordan. A comparative framework for preconditioned lasso algorithms. Advances in Neural Information Processing Systems, 26:1061-1069, 2013.
[43] Tong Zhang. Learning bounds for kernel regression using effective data dimensionality. Neural Computation, 17(9):2077-2098, 2005.
[44] Yuchen Zhang, Martin J Wainwright, Michael I Jordan, et al. Optimal prediction for sparse linear models? lower bounds for coordinate-separable m-estimators. Electronic Journal of Statistics, 11(1):752-799, 2017.
[45] Lijia Zhou, Frederic Koehler, Danica J Sutherland, and Nathan Srebro. Optimistic rates: A unifying theory for interpolation learning and regularization in linear regression. arXiv preprint arXiv:2112.04470, 2021.
[46] Shuheng Zhou. Restricted eigenvalue conditions on subgaussian random matrices. arXiv preprint arXiv:0912.4045, 2009.
[47] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2):301-320, 2005.
Throughout, we use the following standard notation. For positive integers  $n,m\in \mathbb{N}$ , we write  $A:m\times n$  to denote a matrix with  $m$  rows,  $n$  columns, and real-valued entries. The standard inner product on  $\mathbb{R}^n$  is denoted  $\langle u,v\rangle \coloneqq u^\top v$ . For a positive semi-definite matrix  $\Sigma :n\times n$  we define the  $\Sigma$ -inner product on  $\mathbb{R}^n$  by  $\langle u,v\rangle_{\Sigma}:= u^{\top}\Sigma v$  and the  $\Sigma$ -norm by  $\| u\|_{\Sigma} = \sqrt{\langle u,u\rangle_{\Sigma}}$ . For  $n\in \mathbb{N}$  (made clear by context) we let  $e_1,\ldots ,e_n\in \mathbb{R}^n$  be the standard basis vectors  $e_i(j)\coloneqq \mathbb{1}[j = i]$ . For a vector  $v\in \mathbb{R}^n$  and set  $S\subseteq [n]$  we write  $v_{S}$  to denote the restriction of  $v$  to coordinates in  $S$ . For symmetric matrices  $A,B:n\times n$  we write  $A\spreceq B$  to denote that  $B - A$  is positive semi-definite.
A.1 Covering, packing, and  $\ell_1$ -representability
We previously defined the covering number of  $t$ -sparse vectors with respect to a covariance matrix  $\Sigma$ . We next define the packing number (i.e. correlational statistical dimension) and  $\ell_1$ -representability, and discuss the connections between these quantities as well as their algorithmic implications.
Definition A.1. Let  $\Sigma : n \times n$  be a positive semi-definite matrix and let  $t, \alpha > 0$ . A set  $\{v_1, \ldots, v_N\} \subseteq \mathbb{R}^n$  is a  $(t, \alpha)$ -packing for  $\Sigma$  if every  $v_i$  is  $t$ -sparse, and
$$
| \langle v _ {i}, v _ {j} \rangle_ {\Sigma} | <   \alpha \| v _ {i} \| _ {\Sigma} \| v _ {j} \| _ {\Sigma}
$$
for all  $i,j\in [N]$  with  $i\neq j$ . The (correlational) statistical dimension of  $t$ -sparse vectors with maximum correlation  $\alpha$ , under the  $\Sigma$ -inner product, is denoted  $\mathcal{P}_{t,\alpha}(\Sigma)$  and defined as the size of the largest  $(t,\alpha)$ -packing.
We will make use of the following connections between packing, covering, and  $\ell_1$ -representability.
Lemma A.2 (Covering  $\Leftrightarrow$  packing). For any positive semi-definite matrix  $\Sigma : n \times n$  and  $t, \alpha > 0$ , it holds that  $(\alpha^2 / 3)\mathcal{P}_{t,\alpha^2 / 2}(\Sigma) \leq \mathcal{N}_{t,\alpha}(\Sigma) \leq \mathcal{P}_{t,\alpha}(\Sigma)$ .
Proof. First inequality. Let  $\{D_1, \ldots, D_N\}$  be any maximum-size  $(t, \alpha^2/2)$ -packing. Since the  $D_i$ 's are all  $t$ -sparse, each must be correlated with some element of a  $(t, \alpha)$ -dictionary. Thus, it suffices to show that for any  $v \in \mathbb{R}^n$ , the set  $S(v) := \{i \in [N] : |\langle D_i, v \rangle_{\Sigma}| \geq \alpha \| D_i \|_{\Sigma} \| v \|_{\Sigma}\}$  has size  $|S(v)| \leq 3/\alpha^2$ . Indeed, for any  $i, j \in S(v)$  with  $i \neq j$ , we have by the definition of a packing that
$$
\begin{array}{l} \left\langle D _ {i} - \frac {\langle D _ {i} , v \rangle_ {\Sigma}}{\| v \| _ {\Sigma} ^ {2}} v, D _ {j} - \frac {\langle D _ {j} , v \rangle_ {\Sigma}}{\| v \| _ {\Sigma} ^ {2}} v \right\rangle_ {\Sigma} = \langle D _ {i}, D _ {j} \rangle - \frac {\langle D _ {i} , v \rangle_ {\Sigma} \langle D _ {j} , v \rangle_ {\Sigma}}{\| v \| _ {\Sigma} ^ {2}} \\ \leq - \frac {\alpha^ {2}}{2} \left\| D _ {i} \right\| _ {\Sigma} \left\| D _ {j} \right\| _ {\Sigma}. \\ \end{array}
$$
For each  $i\in S(v)$  define  $R_{i} = D_{i} - \langle D_{i},v\rangle_{\Sigma}v / \| v\|_{\Sigma}^{2}$ . Then
$$
0 \leq \left\| \sum_ {i \in S (v)} \frac {R _ {i}}{\| R _ {i} \| _ {\Sigma}} \right\| _ {\Sigma} ^ {2} = | S (v) | + \sum_ {i, j \in S (v): i \neq j} \frac {\langle R _ {i} , R _ {j} \rangle_ {\Sigma}}{\| R _ {i} \| _ {\Sigma} \| R _ {j} \| _ {\Sigma}} \leq | S (v) | - | S (v) | (| S (v) | - 1) \cdot \frac {\alpha^ {2}}{2}
$$
where the last inequality uses the bound  $\| R_i\|_{\Sigma} \leq \| D_i\|_{\Sigma}$ . Rearranging gives  $|S(v)| \leq 1 + (2 / \alpha^2)$ .
Second inequality. Let  $\{D_1, \ldots, D_N\}$  be any maximal  $(t, \alpha)$ -packing. Then for any  $t$ -sparse  $v \in \mathbb{R}^n$ , maximality implies that there must be some  $i \in [N]$  with  $|\langle D_i, v \rangle_{\Sigma}| \geq \alpha \| D_i \|_{\Sigma} \| v \|_{\Sigma}$ . So  $\{D_1, \ldots, D_N\}$  is also a  $(t, \alpha)$ -dictionary.
Lemma A.3 ( $\ell_1$ -representation  $\Longrightarrow$  covering). Let  $\Sigma : n \times n$  be a positive semi-definite matrix and let  $t, B > 0$ . If  $\{D_1, \ldots, D_N\} \subseteq \mathbb{R}^n$  is a  $(t, B)$ - $\ell_1$ -representation for  $\Sigma$ , then it is also a  $(t, 1/B)$ -dictionary for  $\Sigma$ .
Proof. Pick any  $t$ -sparse  $v \in \mathbb{R}^n$ . By  $\ell_1$ -representability, there is some  $\alpha \in \mathbb{R}^N$  with  $v = \sum_{i=1}^{N} \alpha_i D_i$  and  $\sum_{i=1}^{N} |\alpha_i| \cdot \|D_i\|_{\Sigma} \leq B \cdot \|v\|_{\Sigma}$ . Hence