SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $D^{*} \in \mathcal{D}$  and  $w^{*} \in \mathbb{R}^{n + t}$  be the matrix and vector guaranteed by Lemma C.4 for the  $t$ -sparse vector  $v^{*}$ . Let  $\Gamma = (D^{*})^{\top}\Sigma D^{*}$  with eigenvalues  $\gamma_{1} \leq \dots \leq \gamma_{n + t}$ . We make the following claim:
Claim C.6. With probability at least  $1 - \delta / 4$  over  $G \sim N(0, \Gamma)$ , it holds uniformly in  $w \in \mathbb{R}^{n + t}$  that
$$
\langle w - w ^ {*}, G \rangle \leq \| w - w ^ {*} \| _ {1} \sqrt {\lambda_ {n - d _ {h}} \cdot 2 \log (8 n / \delta)} + \| w - w ^ {*} \| _ {\Gamma} \sqrt {2 (d _ {h} + t)}.
$$
Proof. Since  $\Sigma$  is a principal submatrix of  $\Gamma$ , we have  $\gamma_{n - d_h} \leq \lambda_{n - d_h}$  (by the Cauchy Interlacing Theorem). Suppose that  $\Gamma$  has eigendecomposition  $\Gamma = \sum_{i=1}^{n + t} \gamma_i g_i g_i^\top$ , and define projection matrix  $P: (n + t) \times (n + t)$  by  $P := \sum_{i=1}^{n - d_h} g_i g_i^\top$ , so that  $\mathrm{rank}(P^\perp) = d_h + t$  and  $\lambda_{\max}(P \Gamma P) \leq \gamma_{n - d_h} \leq \lambda_{n - d_h}$ . Then for any  $w \in \mathbb{R}^{n + t}$  and  $G \sim N(0, \Gamma)$ , we can bound
$$
\begin{array}{l} \langle w - w ^ {*}, G \rangle = \langle w - w ^ {*}, P G \rangle + \langle w - w ^ {*}, P ^ {\perp} G \rangle \\ \leq \| w - w ^ {*} \| _ {1} \| P G \| _ {\infty} + \left\langle \Gamma^ {1 / 2} (w - w ^ {*}), \Gamma^ {- 1 / 2} P ^ {\perp} G \right\rangle \\ = \| w - w ^ {*} \| _ {1} \| P G \| _ {\infty} + \left\langle \Gamma^ {1 / 2} (w - w ^ {*}), P ^ {\perp} \Gamma^ {- 1 / 2} G \right\rangle \\ \leq \left\| w - w ^ {*} \right\| _ {1} \left\| P G \right\| _ {\infty} + \left\| \Gamma^ {1 / 2} (w - w ^ {*}) \right\| _ {2} \left\| Z \right\| _ {2} \\ \end{array}
$$
where  $Z \sim N(0, P^{\perp})$ . The second equality above uses that  $\Gamma^{-1/2}$  and  $P^{\perp}$  are simultaneously diagonalizable (and therefore commute). But now for any  $\delta > 0$ , we have the Gaussian tail bounds
$$
\operatorname * {P r} \left[ \| P G \| _ {\infty} > \sqrt {\max _ {i} (P \Gamma P) _ {i i} \cdot 2 \log (8 n / \delta)} \right] \leq \delta / 8
$$
$$
\Pr \left[ \| Z \| _ {2} > \sqrt {2 \operatorname {r a n k} (P ^ {\perp})} \right] \leq e ^ {- m / 8} \leq \delta / 8.
$$
Thus, with probability at least  $1 - \delta /4$  over  $G\sim N(0,\Gamma)$  , for any  $w\in \mathbb{R}^{n + t}$  , we have
$$
\begin{array}{l} \langle w - w ^ {*}, G \rangle \leq \| w - w ^ {*} \| _ {1} \sqrt {\max  _ {i} (P \Gamma P) _ {i i} \cdot 2 \log (8 n / \delta)} + \left\| \Gamma^ {1 / 2} (w - w ^ {*}) \right\| _ {2} \sqrt {2 \operatorname {r a n k} (P ^ {\perp})} \\ \leq \| w - w ^ {*} \| _ {1} \sqrt {\lambda_ {n - d _ {h}} \cdot 2 \log (8 n / \delta)} + \left\| \Gamma^ {1 / 2} (w - w ^ {*}) \right\| _ {2} \sqrt {2 (d _ {h} + t)} = F (w) \\ \end{array}
$$
which proves the claim.
We now proceed with proving the theorem.
Proof of Theorem C.5. Applying Claim C.6, we can now invoke Lemma F.7 with covariance matrix  $\Gamma$ , seminorm  $\Phi(v) := 2\sqrt{2\lambda_{n - d_h} \cdot \log(8n / \delta)} \|v\|_1$ ,  $p := 2(d_h + t)$ , ground truth  $w^*$ , samples  $((D^*)^\top X_i, y_i)_{i=1}^{m/2}$ , and failure probability  $\delta/4$ . Since we chose  $m$  sufficiently large that  $m/2 \geq 16p + 196\log(12/\delta)$ , we conclude that with probability at least  $1 - 2\delta$  over the randomness of  $(X_i, y_i)_{i=1}^{m/2}$ , it holds that
$$
\| \hat {w} (D ^ {*}) - w ^ {*} \| _ {\Gamma} ^ {2} \leq O \left(\frac {\sigma^ {2} (d _ {h} + t)}{m} + \frac {(\sigma + \| w ^ {*} \| _ {\Gamma}) \| w ^ {*} \| _ {1} \sqrt {\lambda_ {n - d _ {h}} \cdot \log (8 n / \delta)}}{\sqrt {m}} + \frac {\| w ^ {*} \| _ {1} ^ {2} \lambda_ {n - d _ {h}} \log (8 n / \delta)}{m}\right).
$$
Since  $v^{*} = D^{*}w^{*}$  and  $\| w^{*}\|_{1}\leq 7t^{1 / 2}\lambda_{d_{l} + 1}^{-1 / 2}\| v^{*}\|_{\Sigma}$  (the guarantees of Lemma C.4), it follows that
$$
\| D ^ {*} \hat {w} (D ^ {*}) - v ^ {*} \| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} (d _ {h} + t)}{m} + \frac {(\sigma + \| v ^ {*} \| _ {\Sigma}) \| v ^ {*} \| _ {\Sigma} \sqrt {r _ {\mathrm {e f f}}}}{\sqrt {m}} + \frac {\| v ^ {*} \| _ {\Sigma} ^ {2} r _ {\mathrm {e f f}}}{m}\right).
$$
To complete the proof of the theorem, condition on any values of  $(X_{i},y_{i})_{i = 1}^{m / 2}$  for which the above bound holds. By applying Lemma F.2 with covariance matrix  $\Sigma$ , hypothesis set  $\mathcal{W} := \{D\hat{w}(D):D\in \mathcal{D}\}$ , and samples  $(X_{i},y_{i})_{i = m / 2 + 1}^{m}$  (which are independent of  $\mathcal{W}$ ), since  $m / 2\geq 32\log (2|\mathcal{D}| / \delta)$ , we have with probability at least  $1 - 2\delta$  over the samples  $(X_{i},y_{i})_{i = m / 2 + 1}^{m}$  that
$$
\left\| \hat {D} \hat {w} (\hat {D}) - v ^ {*} \right\| _ {\Sigma} ^ {2} \leq 6 \min _ {D \in \mathcal {D}} \| D \hat {w} (D) - v ^ {*} \| _ {\Sigma} ^ {2} + \frac {3 2 \sigma^ {2} \log (2 | \mathcal {D} | / \delta)}{m}.
$$
Hence, with probability at least  $1 - 5\delta$  we have
$$
\left\| \hat {D} \hat {w} (\hat {D}) - v ^ {*} \right\| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} (d _ {h} + t + \log (2 | \mathcal {D} | / \delta))}{m} + \frac {(\sigma + \| v ^ {*} \| _ {\Sigma}) \| v ^ {*} \| _ {\Sigma} \sqrt {r _ {\mathrm {e f f}}}}{\sqrt {m}} + \frac {\| v ^ {*} \| _ {\Sigma} ^ {2} r _ {\mathrm {e f f}}}{m}\right)
$$
which proves the theorem.
We can use the above theorem (together with the previously discussed boosting approach) to get the following result, which proves Theorem 1.2.
Theorem C.7. Let  $n, t, d_{l}, d_{h}, m, L \in \mathbb{N}$  and  $\sigma, \delta > 0$ . Let  $\Sigma : n \times n$  be a positive semi-definite matrix with eigenvalues  $\lambda_{1} \leq \dots \leq \lambda_{n}$ . Let  $(X_{i}, y_{i})_{i=1}^{m}$  be independent samples where  $X_{i} \sim N(0, \Sigma)$  and  $y_{i} = \langle X_{i}, v^{*} \rangle + \xi_{i}$ , for  $\xi_{i} \sim N(0, \sigma^{2})$  and a fixed  $t$ -sparse vector  $v^{*} \in \mathbb{R}^{n}$ .
Then, given  $\Sigma$ ,  $(X_{i},y_{i})_{i = 1}^{m}$ ,  $t$ ,  $d_{l}$ , and  $\delta$ , there is an estimator  $\hat{v}$  with the following properties.
Let  $n_{eff}^{\prime} \coloneqq t^{2}\log (t) + t\log (d_{l}) + d_{h} + \log (48L / \delta)$  and let  $r_{eff}^{\prime} \coloneqq t(\lambda_{n - d_h} / \lambda_{d_l + 1})\log (8nL / \delta)$ . There are absolute constants  $c_0, C_0 > 0$  such that the following holds. If  $m \geq C_0L(n_{eff}^{\prime} + r_{eff}^{\prime})$ , then with probability at least  $1 - \delta$ , it holds that
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq c _ {0} \frac {\sigma^ {2} (n _ {e f f} ^ {\prime} + r _ {e f f} ^ {\prime})}{m / L} + 2 ^ {- L} \cdot \| v ^ {*} \| _ {\Sigma} ^ {2}.
$$
Moreover,  $\hat{v}$  is computable in time  $(t + 1)^{O(t^2)}(d_l + 1)^{t + 1} \cdot \mathrm{poly}(n)$ .
Proof. Identical to that of Theorem C.3, except using Theorem C.5 instead of Theorem C.2.
D Faster sparse linear regression for arbitrary  $\Sigma$
In this section we prove Theorem 3.1. The approach is via feature adaptation: in Theorem D.5, we show that any covariance matrix  $\Sigma$  has a  $(t,O(t^{3 / 2}\log n) - \ell_1$ -representation of size  $O(n^{t - 1 / 2})$  that is computable in time  $n^{t - \Omega (1 / t)}\log^{O(t)}n$ , using  $O(t\log n)$  samples from  $N(0,\Sigma)$ . The algorithm for computing this representation is described in Algorithm 4. One of the key tools is the following result from computational geometry:
Theorem D.1 ([30]). Let  $n, d, k \in \mathbb{N}$  and  $\delta > 0$ . Given points  $p_1, \ldots, p_n \in \mathbb{R}^d$ , query dimension  $k$ , and failure probability  $\delta$ , there is an algorithm  $\mathrm{DS}((p_1, \ldots, p_n), k, \delta)$  with time complexity  $n^{k+1} (\log n)^{O(k)}$  poly(d) log(1/δ), that constructs a data structure  $\mathcal{N}$  that answers queries of the following form. Given a  $k$ -dimensional subspace  $F \subseteq \mathbb{R}^d$ , the output  $\mathcal{N}(F)$  is some  $i^* \in [n]$ . With probability at least  $1 - \delta$ , the query time complexity is  $n^{1 - 1/(2k)}$  poly(d) log(1/δ), and it holds that
$$
\min  _ {q \in F} \| p _ {i ^ {*}} - q \| _ {2} \leq O (\log n) \cdot \min  _ {i \in [ n ]} \min  _ {q \in F} \| p _ {i} - q \| _ {2}.
$$
How do we use the above theorem to efficiently construct the  $\ell_1$ -representation? The intuition is as follows. Let  $\mathbb{X}$  be the  $m \times n$  matrix where each row is a sample from  $N(0, \Sigma)$ . Then each column is a vector  $p_i$  representing a particular covariate. To find the  $\ell_1$ -representation, it essentially suffices
Algorithm 4:  $\ell_1$ -representation for arbitrary  $\Sigma$
Procedure FindOrthonormalization  $(p_1,\ldots ,p_t)$    
Data: Nonzero vectors  $p_1,\dots ,p_t\in \mathbb{R}^m$    
Result:  $\alpha^{(1)},\ldots ,\alpha^{(t)}\in \mathbb{R}^t$  such that span  $\{\alpha^{(1)},\ldots ,\alpha^{(t)}\} =$  span  $\{e_1,\ldots ,e_t\}$  and  $\langle \sum_{\ell}\alpha_{\ell}^{(i)}p_{\ell},\sum_{\ell}\alpha_{\ell}^{(j)}p_{\ell}\rangle = 0$  for all  $i\neq j$    
for  $i = 1,\dots ,t$  do  $\alpha^{(i)}\gets e_i / \| p_i\| _2\in \mathbb{R}^k$  for  $j = 1,\dots ,i - 1$  do if  $\sum_{\ell}\alpha_{\ell}^{(j)}p_{\ell}\neq 0$  then  $\alpha^{(i)}\gets \alpha^{(i)} - \frac{\langle\Sigma_{\ell}\alpha_{\ell}^{(i)}p_{\ell},\sum_{\ell}\alpha_{\ell}^{(j)}p_{\ell}\rangle}{\|\Sigma_{\ell}\alpha_{\ell}^{(j)}p_{\ell}\|_2}\alpha^{(j)}$  return  $\alpha^{(1)},\ldots ,\alpha^{(k)}$    
Procedure RepresentVectors  $(\{p_1,\ldots ,p_n\} ,t,\delta)$    
Data: Unit vectors  $p_1,\dots ,p_n\in \mathbb{R}^m$  sparsity parameter  $t$  , failure probability  $\delta$    
Result: Set  $\mathcal{D}\subseteq \mathbb{R}^n$  of size  $O(n^{t - 1 / 2})$  , where all elements  $d\in \mathcal{D}$  are  $t$  -sparse (and represented succinctly)   
Compute partition  $I_{1}\sqcup \dots \sqcup I_{\sqrt{n}} = [n]$  where  $|I_i|\leq |\sqrt{n} |$  for all i Initialize  $\mathcal{D}\gets \emptyset$    
for  $j = 1,\dots ,\sqrt{n}$  do Construct data structure  $\mathcal{N}^j\gets \mathrm{DS}((p_i:i\in I_j),t - 1,\delta /n^t)$  /\* Theorem D.1 \*/ for  $T\subseteq \binom {[n]}{t - 1}$  do  $h(T,j)\gets \mathcal{N}^j (\mathrm{span}\{p_i:i\in T\})$  /\* Theorem D.1 \*/ Find  $\gamma \in \mathbb{R}^T$  such that  $\sum_{i\in T}\gamma_{i}p_{i} = \mathrm{Proj}_{\mathrm{span}\{p_{i}:i\in T\}}p_{h(T,j)}$  Write  $\gamma$  as a sparse vector in  $\mathbb{R}^n$  (supported on  $T$  Add  $\gamma -e_{h(T,j)}$  to  $\mathcal{D}$    
for  $T\subseteq \binom {[n]}{t - 2}$  do for  $a,b\in I_j$  do  $\gamma^{(1)},\ldots ,\gamma^{(t)}\gets$  FindOrthonormalization((pi:i∈T∪{a,b})) Write  $\gamma^{(1)},\ldots ,\gamma^{(t)}$  as sparse vectors in  $\mathbb{R}^n$  (supported on  $T\cup \{a,b\})$  Add  $\gamma^{(1)},\ldots ,\gamma^{(t)}$  to  $\mathcal{D}$    
return D   
Procedure ComputeL1Representation(\{X1,...,Xm},t)   
Let  $\mathbb{X}:m\times n$  be the matrix with rows  $X_{1},\ldots ,X_{m}$    
Let  $q_{1},\ldots ,q_{n}$  be the columns of X, and let  $p_i:= q_i / \| q_i \|_2$  for  $i\in [n]$ $\tilde{\mathcal{D}}\gets$  RepresentVectors(\{p1,...,pn},t,e- m)   
 $\hat{D}\gets \mathrm{diag}(\| q_1\|_2,\dots,\| q_n\|_2)$    
Dd:d d d   
return D
to find a dictionary  $\mathcal{D}$  of  $O(n^{t - 1 / 2})$  sparse combinations of  $\{p_1,\ldots ,p_n\}$  so that every  $t$ -sparse combination of  $\{p_1,\dots ,p_n\}$  can be written in terms of the chosen combinations, with a coefficient vector that has bounded  $\ell_{1}$  norm.
For notational ease, we define  $C(x)$  to be the "cost" of a particular linear combination  $x \in \mathbb{R}^n$  with respect to the set  $\mathcal{D}$  of chosen combinations:
Definition D.2. For a subset  $\mathcal{D} \subseteq \mathbb{R}^n$ , define  $C_{\mathcal{D}}: \mathbb{R}^n \to [0, \infty]$  by
$$
C _ {\mathcal {D}} (x) := \min  _ {\alpha \in \mathbb {R} ^ {\mathcal {D}}: \sum_ {d \in \mathcal {D}} \alpha_ {d} d = x} \sum_ {d \in \mathcal {D}} | \alpha_ {d} | \cdot \left\| \sum_ {i = 1} ^ {n} d _ {i} p _ {i} \right\| _ {2}.
$$
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32
With this notation, we want to construct a set  $\mathcal{D}$  of size  $O(n^{t - 1 / 2})$ , consisting of  $t$ -sparse vectors, such that
$$
C _ {\mathcal {D}} (x) \leq \mathrm {p o l y} (t, \log n) \cdot \left\| \sum x _ {i} p _ {i} \right\| _ {2}
$$
for all  $t$ -sparse  $x \in \mathbb{R}^n$ .
The construction is quite simple: divide the set  $\{p_1, \ldots, p_n\}$  into  $\sqrt{n}$  equal-sized groups. For each set  $T$  of  $t - 1$  vectors and each of the  $\sqrt{n}$  groups, find the closest vector in the group to the subspace spanned by  $T$  (using Theorem D.1 to achieve sublinear time complexity). Then add the difference between the vector and its projection (onto the subspace) to the dictionary. Finally, for each set of  $t$  vectors where two of the vectors lie in the same group, add an orthonormal basis for those vectors to the dictionary. See the procedure RepresentVectors() in Algorithm 5 for pseudocode.
By construction, the dictionary clearly has size  $O(n^{t - 1 / 2})$ . At a high level, the reason it satisfies the representational property is the following. Consider some  $t$ -sparse combination, such as  $p_1 + \dots + p_t$ . If  $-p_t$  is not very close to  $p_1 + \dots + p_{t-1}$ , then we can bound  $C(p_1 + \dots + p_t)$  by  $C(p_1 + \dots + p_{t-1})$  and  $C(p_t)$ , which are  $O(\sqrt{t} \| p_1 + \dots + p_{t-1} \|_2)$  and  $O(\sqrt{t} \| p_t \|_2)$  respectively, since the dictionary contains an orthonormal basis for both terms. The only case where these bounds are not good enough is when  $\| p_1 + \dots + p_t \|_2$  is much smaller than  $\| p_1 + \dots + p_{t-1} \|_2$  and  $\| p_t \|_2$ . In this case,  $p_t$  is very close to span  $\{p_1, \ldots, p_{t-1}\}$ . However, in the construction we found some (potentially different)  $p_j$  which is just as close to span  $\{p_1, \ldots, p_{t-1}\}$ , and moreover is in the same group as  $p_t$ . Letting  $q$  be the projection of  $p_j$  onto span  $\{p_1, \ldots, p_{t-1}\}$ , we have the crucial fact that  $\| p_j - q \|_2$  is as small as  $\| p_1 + \dots + p_t \|_2$ .
Now, bounding  $C(p_1 + \dots + p_t)$  proceeds as follows. We can subtract some appropriate (bounded) multiple of  $p_j - q$  from  $p_1 + \dots + p_t$  to zero out at least one of the coefficients. This residual then is a  $t$ -sparse combination of  $\{p_1, \ldots, p_t, p_j\}$  where two of the vectors  $\{p_t, p_j\}$  are in the same group; thus it has small cost with respect to  $\mathcal{D}$ . Moreover,  $p_j - q$  is contained in  $\mathcal{D}$  and thus has small cost (specifically, not much more than  $\| p_j - q \|_2$ , which crucially is not much more than  $\| p_1 + \dots + p_t \|_2$ ). It follows that  $p_1 + \dots + p_t$  has small cost.
Formalizing this argument, we start by proving one of the facts that we freely used above: that the cost function  $C$  satisfies the triangle inequality.
Fact D.3. For any  $\mathcal{D} \subseteq \mathbb{R}^n$  and  $x, y \in \mathbb{R}^n$ , it holds that  $C(x + y) \leq C(x) + C(y)$ .
Proof. For any  $\alpha, \beta \in \mathbb{R}^{\mathcal{D}}$  with  $\sum_{d}\alpha_{d}d = x$  and  $\sum_{d}\beta_{d}d = y$ , the vector  $\alpha + \beta$  satisfies  $\sum_{d}(\alpha + \beta)_{d}d = x + y$ . Applying the triangle inequality to  $\sum_{d}|(\alpha + \beta)_{d}|\cdot\|\sum_{i}d_{i}p_{i}\|_{2}$  completes the proof.
We now prove the key lemma, formalizing the above intuition.
Lemma D.4. Let  $n, m, t \in \mathbb{N}$ , with  $t \geq 2$ , and  $\delta > 0$ . Fix  $p_1, \ldots, p_n \in \mathbb{R}^m$  with  $\| p_i \|_2 = 1$  for all  $i \in [n]$ . Let  $\mathcal{D}$  be the output of RepresentVectors  $(\{p_1, \ldots, p_n\}, t, \delta)$ . Then  $|\mathcal{D}| = O(n^{t - 1/2})$ , and every element of  $\mathcal{D}$  is  $t$ -sparse. Also, with probability at least  $1 - \delta$ , the following guarantees hold. The time complexity of computing  $\mathcal{D}$  is  $O(n^{t - \Omega(1/t)} (\log n)^{O(t)} m^{O(1)} \log (1/\delta))$ . Moreover, for every  $t$ -sparse  $x \in \mathbb{R}^n$  it holds that
$$
C _ {\mathcal {D}} (x) \leq O \left(t ^ {3 / 2} \log n\right) \cdot \left\| \sum x _ {i} p _ {i} \right\| _ {2}.
$$
Proof. Since the algorithm RepresentVectors() makes less than  $n^t$  queries to the data structures  $\mathcal{N}^j$ , and each query has failure probability at most  $\delta' = \delta / n^t$ , the probability that any query fails is at most  $1 - \delta$ . We henceforth assume that all queries succeed, i.e. satisfy the correctness guarantee and time complexity bound stated in Theorem D.1.
Time complexity. We start by analyzing the time complexity of RepresentVectors  $(\{p_1, \ldots, p_n\}, t, \delta)$ . For any fixed  $j \in [\sqrt{n}]$ , the construction time of  $\mathcal{N}^j$  (with  $|I_j| = O(\sqrt{n})$  points in  $\mathbb{R}^m$ , query dimension  $t - 1$ , and failure probability  $\delta / n^t$ ) is  $O(n^{t/2} (\log n)^{O(t)} m^{O(1)} \log (1/\delta))$ . We make  $\binom{n}{t-1} + |I_j|^2 \binom{n}{t-2} = O(n^{t-1})$  queries to  $\mathcal{N}^j$ , each with time complexity  $n^{1/2 - 1/(4(t-1))} m^{O(1)} \log (1/\delta)$ . Each projection step and each orthonormalization step has time complexity  $\text{poly}(t, m)$ . Thus, since  $t \geq 2$ , the time complexity for any
fixed  $j$  is bounded by  $n^{t - 1 / 2 - 1 / (8t)}(\log n)^{O(t)}m^{O(1)}\log (1 / \delta)$ . Summing over  $j$ , the overall time complexity to compute  $\mathcal{D}$  is at most  $n^{t - 1 / (8t)}(\log n)^{O(t)}m^{O(1)}\log (1 / \delta)$  as claimed.
Correctness. The bound on  $|\mathcal{D}|$  and the fact that all elements of  $\mathcal{D}$  are  $t$ -sparse are immediate from the algorithm definition. It remains to bound  $C_{\mathcal{D}}(x)$  for  $t$ -sparse vectors  $x$ . First, note that for any  $(t - 1)$ -sparse  $y \in \mathbb{R}^n$ , because of step (4), the dictionary contains vectors  $\gamma^1, \ldots, \gamma^{t-1}$  that span  $\operatorname{supp}(y)$  and satisfy  $\langle \sum_{i=1}^{n} \gamma_i^k p_i, \sum_{i=1}^{n} \gamma_i^\ell p_i \rangle = 0$  for all  $k \neq \ell$ . Thus, letting  $\alpha_1, \ldots, \alpha_{t-1} \in \mathbb{R}$  be such that  $y = \alpha_1 \gamma^1 + \dots + \alpha_{t-1} \gamma^{t-1}$ , we get
$$
C _ {\mathcal {D}} (y) \leq \sum_ {j = 1} ^ {t - 1} | \alpha_ {j} | \cdot \left\| \sum_ {i = 1} ^ {n} \gamma_ {i} ^ {j} p _ {i} \right\| _ {2} \leq \sqrt {t} \sqrt {\sum_ {j = 1} ^ {t - 1} \alpha_ {j} ^ {2} \left\| \sum_ {i = 1} ^ {n} \gamma_ {i} ^ {j} p _ {i} \right\| _ {2} ^ {2}} = \sqrt {t} \left\| \sum_ {i = 1} ^ {n} y _ {i} p _ {i} \right\| _ {2}. \tag {7}
$$
Now fix any nonzero  $t$ -sparse  $x \in \mathbb{R}^n$ . Fix any  $a \in \arg \max_{i \in [n]} |x_i|$ , and let  $j \in [\sqrt{n}]$  be such that  $a \in I_j$ . Let  $T = \operatorname{supp}(x) \setminus \{a\}$ . Let  $q \coloneqq \operatorname{Proj}_{\operatorname{span}\{p_i : i \in T\}} p_{h(T,j)}$ . Then by the correctness guarantee of  $\mathcal{N}^j$  on query span  $\{p_i : i \in T\}$ ,
$$
\left\| p _ {h (T, j)} - q \right\| _ {2} \leq O (\log n) \cdot \left\| p _ {a} + \sum_ {i \neq a} \frac {x _ {i}}{x _ {a}} p _ {i} \right\| _ {2} = O (\log n) \cdot \frac {\left\| \sum_ {i} x _ {i} p _ {i} \right\| _ {2}}{\left| x _ {a} \right|}. \tag {8}
$$
Case I. Suppose that  $\left\| p_{h(T,j)} - q\right\| _2\geq 1 / 2$ . Then by (8), we have  $|x_{a}|\leq O(\log n)\cdot \| \sum_{i}x_{i}p_{i}\|_{2}$ . Thus, by the triangle inequality,
$$
\left\| \sum_ {i \neq a} x _ {i} p _ {i} \right\| _ {2} \leq | x _ {a} | + \left\| \sum_ {i} x _ {i} p _ {i} \right\| _ {2} \leq O (\log n) \cdot \left\| \sum_ {i} x _ {i} p _ {i} \right\| _ {2}.
$$
It follows from Fact D.3 and (7) that
$$
C _ {\mathcal {D}} (x) \leq C _ {\mathcal {D}} (x _ {a} e _ {a}) + C _ {\mathcal {D}} (x - x _ {a} e _ {a}) \leq \sqrt {t} | x _ {a} | + \sqrt {t} \left\| \sum_ {i \neq a} x _ {i} p _ {i} \right\| _ {2} \leq O (\sqrt {t} \log n) \cdot \left\| \sum_ {i} x _ {i} p _ {i} \right\| _ {2}
$$
Case II. It remains to consider the case that  $\left\| p_{h(T,j)} - q\right\|_2 \leq 1/2$ . In this case we have  $\| q\|_2 \geq \left\| p_{h(T,j)}\right\|_2 - 1/2 \geq 1/2$ . By step (3) of the algorithm, the dictionary contains some vector  $\gamma - e_{h(T,j)}$  such that  $\operatorname{supp}(\gamma) \subseteq T$  and  $q = \sum_{i \in T} \gamma_i p_i$ . Fix any  $b \in \arg \max_i |\gamma_i|$ . Since  $q = \sum \gamma_i p_i$  we get  $|\gamma_b| \geq \frac{\|q\|_2}{t} \geq 1/(2t)$ . Now, by Fact D.3,
$$
C _ {\mathcal {D}} (x) \leq C _ {\mathcal {D}} \left(- \frac {x _ {b}}{\gamma_ {b}} \left(e _ {h (T, j)} - \gamma\right)\right) + C _ {\mathcal {D}} \left(x + \frac {x _ {b}}{\gamma_ {b}} \left(e _ {h (T, j)} - \gamma\right)\right).
$$
By construction,  $e_{h(T,j)} - \gamma$  is an element of the dictionary, so we can bound the first term as
$$
\begin{array}{l} C _ {\mathcal {D}} \left(- \frac {x _ {b}}{\gamma_ {b}} (e _ {h (T, j)} - \gamma)\right) \leq \frac {| x _ {b} |}{| \gamma_ {b} |} \left\| \sum_ {i = 1} ^ {n} (e _ {h (T, j)} - \gamma) _ {i} p _ {i} \right\| _ {2} \\ = \frac {\left| x _ {b} \right|}{\left| \gamma_ {b} \right|} \left\| p _ {h (T, j)} - q \right\| _ {2} \\ \leq 2 t \left| x _ {a} \right| \left\| p _ {h (T, j)} - q \right\| _ {2} \\ \leq O (t \log n) \left\| \sum_ {i = 1} ^ {n} x _ {i} p _ {i} \right\| _ {2} \\ \end{array}
$$
where the equality uses that  $q = \sum_{i=1}^{n} \gamma_i p_i$ , the second inequality uses that  $|x_b| \leq |x_a|$  and  $|\gamma_b| \geq 1/(2t)$ , and the final inequality uses (8).
Finally, observe that
$$
z := x + \frac {x _ {b}}{\gamma_ {b}} \left(e _ {h (T, j)} - \gamma\right) = x _ {a} e _ {a} + \frac {x _ {b}}{\gamma_ {b}} e _ {h (T, j)} + \sum_ {i \in T \backslash \{a, b \}} \left(x _ {i} - \frac {x _ {b} \gamma_ {i}}{\gamma_ {b}}\right) e _ {i}
$$
since the coefficients on  $e_b$  cancel out. Thus,  $z$  is a linear combination of two elements of  $\{p_i : i \in I_j\}$  together with  $t - 2$  elements of  $\{p_i : i \in [n]\}$ . Because of step (4) of the algorithm, the dictionary contains vectors  $\gamma^1, \ldots, \gamma^t$  that span  $\operatorname{supp}(z)$  and satisfy  $\langle \sum_{i=1}^{n} \gamma_i^k p_i, \sum_{i=1}^{n} \gamma_i^\ell p_i \rangle = 0$  for all  $k \neq \ell$ . The same argument as for (7) gives that
$$
\begin{array}{l} C _ {\mathcal {D}} \left(x + \frac {x _ {b}}{\gamma_ {b}} \left(e _ {h (T, j)} - \gamma\right)\right) \leq \sqrt {t} \left\| \sum_ {i = 1} ^ {n} x _ {i} p _ {i} + \frac {x _ {b}}{\gamma_ {b}} \left(p _ {h (T, j)} - q\right) \right\| _ {2} \\ \leq \sqrt {t} \left\| \sum_ {i = 1} ^ {n} x _ {i} p _ {i} \right\| _ {2} + O (\sqrt {t} \log n) \frac {\left| x _ {b} \right|}{\left| \gamma_ {b} \right| \left| x _ {a} \right|} \left\| \sum_ {i = 1} ^ {n} x _ {i} p _ {i} \right\| _ {2} \\ \leq O \left(t ^ {3 / 2} \log n\right) \left\| \sum_ {i = 1} ^ {n} x _ {i} p _ {i} \right\| _ {2} \\ \end{array}
$$
where the second inequality uses the triangle inequality and (8), and the final inequality uses that  $|x_{b}| \leq |x_{a}|$  and  $|\gamma_{b}| \geq 1 / (2t)$ . Putting everything together, we conclude that
$$
C _ {\mathcal {D}} (x) \leq O \left(t ^ {3 / 2} \log n\right) \left\| \sum_ {i = 1} ^ {n} x _ {i} p _ {i} \right\| _ {2}
$$
We now show that RepresentVectors() can be applied to the columns of the sample matrix to obtain a  $\ell_1$ -representation for  $\Sigma$  (procedure ComputeL1Representation() in Algorithm 5). Up to an appropriate rescaling of the covariates, Lemma D.4 immediately implies that  $\mathcal{D}$  gives a  $\ell_1$ -representation for the empirical covariance  $\hat{\Sigma}$ . The main result then follows from concentration of  $\hat{\Sigma}$  and sparsity of the elements of the dictionary.