SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem D.5. Let  $n, m, t \in \mathbb{N}$  and let  $\Sigma : n \times n$  be a positive-definite matrix. Suppose  $m \geq Ct\log n$  for a sufficiently large constant  $C$ . Let  $X_1, \ldots, X_m \sim N(0, \Sigma)$  be independent samples, and let  $\mathcal{D}$  be the output of ComputeL1Representation( $\{X_1, \ldots, X_m\}, t$ ). Then  $|\mathcal{D}| \leq O(n^{t - 1/2})$ , and every element of  $\mathcal{D}$  is  $t$ -sparse. Also, with probability at least  $1 - e^{-\Omega(m)}$ , the time complexity of the algorithm is  $O(n^{t - \Omega(1/t)}(\log n)^{O(t)}m^{O(1)})$ , and  $\mathcal{D}$  is a  $(t, C_{l1rep}t^{3/2}\log n) - \ell_1$ -representation for  $\Sigma$ , for some universal constant  $C_{l1rep}$ .
Proof. Let  $\hat{\Sigma} = \frac{1}{m}\mathbb{X}^{\top}\mathbb{X}$ . Let  $\tilde{\mathcal{D}}$  denote the intermediary dictionary constructed by the algorithm using RepresentVectors(). With probability at least  $1 - e^{-m}$ , the successful event of Lemma D.4 holds. By standard concentration bounds (see e.g. Exercise 4.7.3 in [41]), it holds that  $\frac{1}{2}\| x\|_{\Sigma} \leq \| x\|_{\hat{\Sigma}} \leq 2\| x\|_{\Sigma}$  for all  $t$ -sparse  $x \in \mathbb{R}^n$ , with probability at least  $1 - e^{-\Omega(m)}$ . Henceforth assume that both of these events hold.
Time complexity. The time complexity of the algorithm is dominated by the call to RepresentVectors(). By the guarantee of Lemma D.4, this takes time  $O(n^{t - \Omega(1/t)}(\log n)^{O(t)}m^{O(1)})$ .
Correctness. The bounds on  $|\mathcal{D}|$  and sparsity of elements of  $\mathcal{D}$  follow from identical bounds for  $\tilde{\mathcal{D}}$  (see Lemma D.4), and the fact that every element of  $\mathcal{D}$  is obtained by rescaling the coordinates of some element of  $\tilde{\mathcal{D}}$ . It remains to show that  $\mathcal{D}$  is a  $(t,O(t^{3 / 2}\log n)) - \ell_1$  representation for  $\Sigma$ .
Fix any  $t$ -sparse  $v \in \mathbb{R}^n$ , and define  $\tilde{v} = \hat{D} v$ . By the guarantee of Lemma D.4, since  $\tilde{v}$  is also  $t$ -sparse, there is some  $\alpha \in \mathbb{R}^{\tilde{D}}$  such that  $\tilde{v} = \sum_{\tilde{d} \in \tilde{D}} \alpha_{\tilde{d}} \tilde{d}$  and
$$
\sum_ {\tilde {d} \in \tilde {\mathcal {D}}} | \alpha_ {\tilde {d}} | \cdot \left\| \sum_ {i = 1} ^ {n} \tilde {d} _ {i} \frac {q _ {i}}{\| q _ {i} \| _ {2}} \right\| _ {2} \leq O (t ^ {3 / 2} \log n) \cdot \left\| \sum_ {i = 1} ^ {n} \tilde {v} _ {i} \frac {q _ {i}}{\| q _ {i} \| _ {2}} \right\| _ {2}.
$$
Algorithm 5: Sparse linear regression for arbitrary  $\Sigma$
1 Procedure SparseLinearRegression((Xi,yi)m i=1, t, B, σ2)  
2 D← ComputeL1Representation{X1,..., X100t log n}, t)  
3 for j = m/2 + 1, ..., m do  
4 for d ∈ D do  
5  $\tilde{X}_{j,d} \gets \left\langle X_j, d / \sqrt{(2/m) \sum_{i=1}^{m/2} \langle X_i, d \rangle^2} \right\rangle$   
/* See Theorem A.7 for definition of MirrorDescentLasso(), and Theorem D.5 for definition of C1rep  
6  $\hat{\beta} \gets \text{MirrorDescentLasso}((\tilde{X}_i, y_i)^m_{i=m/2+1}, 2C_{1rep} t^{3/2} B \log(n), m/2, \sigma^2)$   
7  $\hat{w} \gets \sum_{d \in \mathcal{D}} \hat{\beta}_d d / \sqrt{(2/m) \sum_{i=1}^{m/2} \langle X_i, d \rangle^2}$   
8 return  $\hat{w}$
But note that  $\tilde{v}_i = \hat{D}_{ii}v_i = \| q_i\| _2v_i$  for all  $i$ . Similarly, every  $\tilde{d}\in \tilde{\mathcal{D}}$  corresponds to some  $d\in \mathcal{D}$  with  $\tilde{d}_i = \| q_i\| _2d_i$  for all  $i$ . Thus, reindexing  $\alpha$  according to  $\mathcal{D}$  in the natural way, we have that  $v = \sum_{d\in \mathcal{D}}\alpha_d d$  and
$$
\sum_ {d \in \mathcal {D}} | \alpha_ {d} | \cdot \left\| \sum_ {i = 1} ^ {n} d _ {i} q _ {i} \right\| _ {2} \leq O (t ^ {3 / 2} \log n) \cdot \left\| \sum_ {i = 1} ^ {n} v _ {i} q _ {i} \right\| _ {2}.
$$
But now let  $\hat{\Sigma} = \frac{1}{m}\mathbb{X}^{\top}\mathbb{X}$ . For any  $i,j\in [n]$  we have  $\langle q_i,q_j\rangle = m\hat{\Sigma}_{ii}$ . Hence,
$$
\left\| \sum_ {i = 1} ^ {n} v _ {i} q _ {i} \right\| _ {2} ^ {2} = \sum_ {i, j \in [ n ]} v _ {i} v _ {j} \hat {\Sigma} _ {i j} = v ^ {\top} \hat {\Sigma} v
$$
and similarly for  $\| \sum_{i = 1}^{n}d_{i}q_{i}\|_{2}^{2}$ . Thus, we get
$$
\sum_ {d \in \mathcal {D}} | \alpha_ {d} | \cdot \| d \| _ {\hat {\Sigma}} \leq O (t ^ {3 / 2} \log n) \cdot \| v \| _ {\hat {\Sigma}}.
$$
But as shown above, we know that  $\frac{1}{2} \| x \|_{\Sigma} \leq \| x \|_{\hat{\Sigma}} \leq 2 \| x \|_{\Sigma}$  for all  $t$ -sparse  $x \in \mathbb{R}^n$ . Since  $v$  and all  $d \in \mathcal{D}$  are  $t$ -sparse, we conclude that
$$
\sum_ {d \in \mathcal {D}} | \alpha_ {d} | \cdot \| d \| _ {\Sigma} \leq O (t ^ {3 / 2} \log n) \cdot \| v \| _ {\Sigma}
$$
We finally restate and prove Theorem 3.1, as a corollary of Theorem D.5 and the well-known fact that standard "slow rate" guarantees for Lasso (i.e. based on the  $\ell_1$  norm of the regressor) can be achieved in near-linear time (Theorem A.8). The pseudocode for the main algorithm is given in Algorithm 5.
Corollary D.6. Let  $n, m, t, B \in \mathbb{N}$  and  $\sigma > 0$ , and let  $\Sigma : n \times n$  be a positive-definite matrix. Let  $w^{*} \in \mathbb{R}^{n}$  be  $t$ -sparse with  $\| w^{*} \|_{\Sigma} \leq B$ . Suppose  $m \geq Ct \log n$  for a sufficiently large constant  $C$ . Let  $(X_{i}, y_{i})_{i=1}^{m}$  be independent samples where  $X_{i} \sim N(0, \Sigma)$  and  $y_{i} = \langle X_{i}, w^{*} \rangle + N(0, \sigma^{2})$ . Then there is an  $O(m^{2} n^{t-1/2} + n^{t-\Omega(1/t)} \log^{O(t)} n)$ -time algorithm (Algorithm 5) that, given  $(X_{i}, y_{i})_{i=1}^{m}$ ,  $t, B, \sigma^{2}$ , produces an estimate  $\hat{w} \in \mathbb{R}^{n}$  satisfying, with probability  $1 - o(1)$ ,
$$
\| \hat {w} - w ^ {*} \| _ {\Sigma} ^ {2} \leq \tilde {O} \left(\frac {\sigma^ {2}}{\sqrt {m}} + \frac {\sigma B t ^ {3 / 2}}{\sqrt {m}} + \frac {B ^ {2} t ^ {3}}{m}\right).
$$
Proof. By Theorem D.5 it holds with probability  $1 - n^{-100t}$  that  $\mathcal{D}$  is a  $(t, C_{\mathrm{1rep}} t^{3/2} \log n) - \ell_1$ -representation for  $\Sigma$ . Also, by standard concentration bounds (e.g. Exercise 4.7.3 in [41]), we have
$\frac{1}{2}\|x\|_{\Sigma} \leq \|x\|_{\hat{\Sigma}} \leq 2\|x\|_{\Sigma}$  for all  $t$ -sparse  $x \in \mathbb{R}^n$  (where  $\hat{\Sigma} = \frac{2}{m}\sum_{i=1}^{m/2}X_iX_i^\top$ ) with probability at least  $1 - \exp(-\Omega(m))$ . Suppose that both of these events occur.
For each of the remaining  $m / 2$  samples  $X_{j}$ , compute  $\tilde{X}_j\in \mathbb{R}^{\mathcal{D}}$  where the entry  $\tilde{X}_{j,d}$  corresponding to  $d\in \mathcal{D}$  is  $\langle X_j,d / \| d\|_{\hat{\Sigma}}\rangle$  (where  $\hat{\Sigma} = \frac{2}{m}\sum_{i = 1}^{m / 2}X_iX_i^\top$  is not explicitly computed; since  $d$  is sparse, both  $\langle X_j,d\rangle$  and  $\| d\|_{\hat{\Sigma}}$  can be computed in poly  $(t,m)$  time). Let  $N(0,\Gamma)$  denote the distribution of each  $\tilde{X}_j$ . For each  $d\in \mathcal{D}$ , since  $d$  is  $t$ -sparse, we have that  $\mathbb{E}_{x\sim N(0,\Sigma)}\langle x,d / \| d\|_{\hat{\Sigma}}\rangle^2 = \| d\|_{\Sigma}^2 /\| d\|_{\hat{\Sigma}}^2\leq 4$ . Thus,  $\Gamma_{dd}\leq 4$  for all  $d$ .
Moreover, since  $w^{*}$  is  $t$ -sparse, there is some  $\alpha \in \mathbb{R}^{\mathcal{D}}$  with  $w^{*} = \sum_{d}\alpha_{d}d$  and  $\sum_{d}|\alpha_{d}|\| d\|_{\Sigma}\leq C_{1\mathrm{rep}}t^{3 / 2}\log (n)\cdot \| w^{*}\|_{\Sigma}$ . Define  $\beta \in \mathbb{R}^d$  by  $\beta_{d} = \alpha_{d}\| d\|_{\hat{\Sigma}}$ . Then  $w^{*} = \sum_{d}\beta_{d}d / \| d\|_{\hat{\Sigma}}$  and
$$
\sum_ {d} | \beta_ {d} | \leq 2 \cdot \sum_ {d} | \alpha_ {d} | \| d \| _ {\Sigma} \leq 2 C _ {1 1 \mathrm {r e p}} t ^ {3 / 2} \log (n) \cdot \| w ^ {*} \| _ {\Sigma}.
$$
But now for any of the remaining  $m / 2$  samples, we have that
$$
\langle \tilde {X} _ {j}, \beta \rangle = \sum_ {d} \langle X _ {j}, d / \| d \| _ {\hat {\Sigma}} \rangle \alpha_ {d} \| d \| _ {\hat {\Sigma}} = \langle X _ {j}, \sum_ {d} \alpha_ {d} d \rangle = \langle X _ {j}, w ^ {*} \rangle ,
$$
and thus  $y - \langle \tilde{X}_j,\beta \rangle \sim N(0,\sigma^2)$ . So we can apply Theorem A.8 to samples  $(\tilde{X}_j,y_j)_{j = m / 2 + 1}^m$  to compute an estimator  $\hat{\beta}$  satisfying
$$
\left\| \hat {\beta} - \beta \right\| _ {\Gamma} ^ {2} \leq \tilde {O} \left(\frac {\sigma^ {2}}{m} + \frac {\sigma B t ^ {3 / 2}}{\sqrt {m}} + \frac {B ^ {2} t ^ {3}}{m}\right)
$$
using that  $\| \beta \| _1\leq 2C_{\mathsf{I}\mathsf{I}\mathsf{rep}}t^{3 / 2}\log (n)\cdot \| w^*\|_{\Sigma}\leq 2C_{\mathsf{I}\mathsf{I}\mathsf{rep}}t^{3 / 2}B\log (n)$ , and using the bound  $\max_d\Gamma_{dd}\leq 4$ . The time complexity of this step is  $\tilde{O} (|\mathcal{D}|m^2) = \tilde{O} (m^2 n^{t - 1 / 2})$ . Finally, compute  $\hat{w} := \sum_d\hat{\beta}_d d / \| d\|_{\hat{\Sigma}}$ . We have that  $\| \hat{w} -w^{*}\|_{\Sigma} = \left\| \hat{\beta} -\beta \right\|_{\Gamma}$ , which completes the proof.
E Fixed-parameter tractability in  $\kappa$  and  $t$
In this section we prove Theorem 3.2, which shows we can achieve upper bounds on  $\mathcal{N}_{t,\alpha}(\Sigma)$  for  $\alpha$  independent of  $\kappa$  and  $n$ , if we are willing to incur dependence on  $\kappa$  in the resulting bound. In fact, we actually prove an upper bound on the packing number  $\mathcal{P}_{t,\alpha}(\Sigma)$ .
To achieve this, the first key idea is to consider the dual certificates for a packing. Suppose that  $v_{1}, \ldots, v_{N}$  are unit vectors (in the  $\Sigma$ -norm) with  $|\langle v_{i}, v_{j} \rangle_{\Sigma}| \leq \alpha$  for all  $i \neq j$ . Then  $|\langle v_{i}, \Sigma v_{i} \rangle| \geq \alpha^{-1} \max_{j \neq i} |\langle v_{j}, \Sigma v_{i} \rangle|$ , so  $\Sigma v_{i}$  certifies that any linear combination  $v_{i} = \sum_{j \neq i} x_{j} v_{j}$  must have the property that  $\| x \|_{1} \geq \alpha^{-1}$ . Thus, to show that there cannot be a large packing of sparse vectors in the  $\Sigma$ -norm, it would suffice to prove that any large set of sparse vectors must have one vector that can be written as a linear combination of the remaining vectors, where the coefficient vector has small  $\ell_{1}$  norm. In fact, this would give an upper bound on  $\mathcal{N}_{t,\alpha}(\Sigma)$  for all  $\Sigma$ .
We do not know if such a statement is true. However, we can prove an approximate analogue. The following lemma shows that under a condition number bound on  $\Sigma$ , the dual certificate argument can be generalized to require only a weaker property: that any large set of sparse vectors must have one vector that can be approximately written as a linear combination of the remaining vectors, with low  $\ell_1$  cost. The approximation error determines how small the condition number must be:
Lemma E.1. Let  $n, N, t, T \in \mathbb{N}$  and let  $\delta > 0$ . Suppose that for all  $t$ -sparse vectors  $v_{1}, \ldots, v_{N} \in \mathbb{R}^{n}$ , there exists some  $i \in [N]$  and  $x \in \mathbb{R}^{N}$  such that  $\| x \|_{1} \leq T$  and
$$
\left\| v _ {i} - \sum_ {j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq \delta \cdot \max  _ {j \in [ N ]} \| v _ {j} \| _ {2}.
$$
Then for every positive-definite matrix  $\Sigma : n \times n$  with  $\kappa(\Sigma) < 1/(4\delta^2)$  it holds that  $\mathcal{P}_{t,1/(3T)}(\Sigma) \leq N\log_2\kappa(\Sigma)$ .
Proof. Fix a positive-definite matrix  $\Sigma : n \times n$  and suppose that  $K \coloneqq \mathcal{P}_{t,1/(3T)}(\Sigma) > N \log_2 \kappa(\Sigma)$ . By definition, there are nonzero  $t$ -sparse vectors  $v_1, \ldots, v_K \in \mathbb{R}^N$  such that
$$
| \langle v _ {i}, v _ {j} \rangle_ {\Sigma} | \leq \frac {1}{3 T} \| v _ {i} \| _ {\Sigma} \| v _ {j} \| _ {\Sigma}
$$
for all  $i \neq j$ . Without loss of generality, assume that  $\| v_{i} \|_{2} = 1$  for all  $i \in [K]$ , so that
$$
\lambda_ {\min } (\Sigma) \leq \| v _ {i} \| _ {\Sigma} ^ {2} \leq \lambda_ {\max } (\Sigma).
$$
So we can partition  $[K]$  into  $\log_2\kappa (\Sigma)$  buckets such that  $\max_{i\in B}\| v_i\|_{\Sigma}^2 /\min_{i\in B}\| v_i\|_{\Sigma}^2\leq 2$  for each bucket  $B\subseteq [K]$ . There must be some bucket  $B$  with  $|B|\geq N$ . By assumption, there is some  $i\in B$  and  $x\in \mathbb{R}^N$  such that  $\| x\| _1\leq T$  and
$$
\left\| v _ {i} - \sum_ {j \in B: j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq \delta .
$$
$$
\begin{array}{l} \langle v _ {i}, v _ {i} \rangle_ {\Sigma} = \left\langle \Sigma v _ {i}, \sum_ {j \in B: j \neq i} x _ {j} v _ {j} \right\rangle + \left\langle \Sigma v _ {i}, v _ {i} - \sum_ {j \in B: j \neq i} x _ {j} v _ {j} \right\rangle \\ = \sum_ {j \in B: j \neq i} x _ {j} \left\langle v _ {i}, v _ {j} \right\rangle_ {\Sigma} + \left\langle \Sigma v _ {i}, v _ {i} - \sum_ {j \in B: j \neq i} x _ {j} v _ {j} \right\rangle \\ \leq \| x \| _ {1} \max _ {j \in B: j \neq i} | \langle v _ {i}, v _ {j} \rangle_ {\Sigma} | + \left\| v _ {i} ^ {\top} \Sigma \right\| _ {2} \cdot \delta \\ \leq \frac {\| x \| _ {1}}{3 T} \max  _ {j \in B: j \neq i} \| v _ {i} \| _ {\Sigma} \| v _ {j} \| _ {\Sigma} + \delta \sqrt {\lambda_ {\operatorname* {m a x}} (\Sigma) \cdot v _ {i} ^ {\top} \Sigma v _ {i}} \\ \leq \frac {\sqrt {2} \left\| v _ {i} \right\| _ {\Sigma} ^ {2}}{3} + \left\| v _ {i} \right\| _ {\Sigma} \delta \sqrt {\lambda_ {\max} (\Sigma)}. \\ \end{array}
$$
Simplifying, we get  $\| v_{i}\|_{\Sigma}\leq 2\delta \sqrt{\lambda_{\max}(\Sigma)}$  . Since also  $\| v_i\|_\Sigma \geq \sqrt{\lambda_{\min}(\Sigma)}$  , it follows that  $\kappa (\Sigma) =$ $\lambda_{\mathrm{max}}(\Sigma) / \lambda_{\mathrm{min}}(\Sigma)\geq 1 / (4\delta^2)$
It remains to show that the precondition of Lemma E.1 can be satisfied for sub-constant  $\delta$  without requiring  $N$  to scale with  $n^t$ . We start by proving the desired property when the vectors are all  $t$ -sparse and binary, i.e.  $v_1, \ldots, v_N \in \{0, 1\}^n$ , and afterwards we will black-box extend the result to the real-valued setting. Concretely, given sparse binary vectors  $v_1, \ldots, v_N \in \{0, 1\}^n$  (with  $N \gg n$ ), we want to find one that can be "efficiently" approximated (in  $\ell_2$  norm) by the rest, where "efficient" means that the coefficients have small absolute sum. Thinking of each vector as the indicator vector of a subset of  $[n]$ , a first step towards an efficient approximation for  $v_i = \mathbb{1}[\cdot \in S_i]$  may be constructing an efficient approximation for a standard basis vector  $e_j$  for some  $j \in S_i$ .
Indeed, there is some  $j \in [n]$  such that  $S^j \coloneqq \{i : v_{ij} = 1\}$  is large, i.e.  $|S^j| \geq N / n$ . If the vectors  $(v_i)_{i \in S^j}$  were in some sense random, then the average  $\frac{1}{|S^j|} \sum_{i \in S^j} v_i$  would be a good approximation for  $e_j$ . It is also efficient, in that the absolute sum of coefficients is 1. But of course the vectors are not random; it could be that many vectors in  $S^j$  also contain some other coordinate  $j'$ . In this case we restrict to the set of vectors containing both  $j$  and  $j'$ . Now we may hope to approximate the vector  $\mathbb{1}[\cdot \in \{j, j'\}]$ . Completing this argument, we get the following lemma which states that there exists a subset of  $[n]$  that is contained in many of the vectors, and that is well-approximated by the average of those vectors.
For notational convenience, for vectors  $x,y\in \{0,1\} ^n$  we say that  $x\spreceq y$  if  $x_{i}\leq y_{i}$  for all  $i\in [n]$ .
Lemma E.2. Let  $n, N, t, s \in \mathbb{N}$  with  $sn \leq N$ , and let  $v_{1}, \ldots, v_{N} \in \{0, 1\}^{n}$  be nonzero  $t$ -sparse binary vectors. Then there is some set  $S \subseteq [N]$  of size  $|S| \geq s$  and some nonzero vector  $u \in \{0, 1\}^{n}$  such that  $u \preceq v_{i}$  for all  $i \in S$ , and
$$
\left\| u - \frac {1}{| S |} \sum_ {i \in S} v _ {i} \right\| _ {2} \leq \sqrt {t (s n / N) ^ {1 / t}}.
$$
Proof. For each  $J \subseteq [n]$ , define  $\mathcal{S}^J \coloneqq \{i \in [N] : v_{ij} = 1 \quad \forall j \in J\}$ . Since all  $v_i$  are nonzero, there is some  $j^* \in [n]$  with  $|\mathcal{S}^{(j^*)}| \geq N / n$ . We iteratively construct a set  $J \subseteq [n]$  as follows. Initially, set  $J = \{j^*\}$ . While there exists some  $a \in [n] \setminus J$  such that  $|\mathcal{S}^{J \cup \{a\}}| > (sn / N)^{1 / t} |\mathcal{S}^J|$ , update  $J$  to  $J \cup \{a\}$  (if there are multiple such  $a$ , pick any one of them arbitrarily). At termination of this process, we have  $|\mathcal{S}^J| > 0$ . Since every  $v_i$  is  $t$ -sparse, it must be that  $|J| \leq t$ . Thus,  $|\mathcal{S}^J| \geq (N / n) \cdot (sn / N)^{(t - 1) / t} \geq s$ . Set  $S \coloneqq \mathcal{S}^J$  and  $u \coloneqq \mathbb{1}_J \in \{0, 1\}^n$ . By definition of  $\mathcal{S}^J$ , we have that  $u \preceq v_i$  for all  $i \in S$ .
For any  $j \in J$ , we have  $u_{j} = 1 = \frac{1}{|S|}\sum_{i \in S} v_{ij}$ . For any  $j \notin J$ , we have  $u_{j} = 0$  and
$$
\left| \frac {1}{| S |} \sum_ {i \in S} v _ {i j} \right| = \frac {| \{i \in S : v _ {i j} = 1 \} |}{| S |} = \frac {| \mathcal {S} ^ {J \cup \{j \}} |}{| \mathcal {S} ^ {J} |} \leq (s n / N) ^ {1 / t}
$$
by construction of  $J$ . Thus,
$$
\left\| u - \frac {1}{| S |} \sum_ {i \in S} v _ {i} \right\| _ {\infty} \leq (s n / N) ^ {1 / t}.
$$
$$
\left\| u - \frac {1}{| S |} \sum_ {i \in S} v _ {i} \right\| _ {1} \leq \left\| \frac {1}{| S |} \sum_ {i \in S} v _ {i} \right\| _ {1} \leq \frac {1}{| S |} \sum_ {i \in S} \| v _ {i} \| _ {1} \leq t.
$$
By the inequality  $\| x\| _2^2\leq \| x\| _1\| x\|_{\infty}$ , we conclude that
$$
\left\| u - \frac {1}{| S |} \sum_ {i \in S} v _ {i} \right\| _ {2} \leq \sqrt {t (s n / N) ^ {1 / t}}
$$
We now use Lemma E.2 to show that if  $N$  is sufficiently large, then at least one of the vectors  $v_{i}$  can be efficiently approximated by the rest. The proof is by induction on  $t$ . As a first attempt, one might use Lemma E.2 to find some  $u \in \{0,1\}^n$  and some large set  $S \subseteq [N]$  such that  $u \preceq v_{i}$  for all  $i \in S$ , and the average of the  $v_{i}$ 's approximates  $u$ . Then, restrict to the vectors in  $S$ , and induct on the  $(t - 1)$ -sparse residual vectors  $\{v_{i} - u : i \in S\}$ . If one of the  $v_{i} - u$ 's can be efficiently approximated by the other residuals, then since  $u$  can also be efficiently approximated, we can derive an efficient approximation of  $v_{i}$  by the remaining  $v_{j}$ 's.
This doesn't quite work, since at each step of the induction the set of vectors will become smaller by a factor of roughly  $n$ . However, instead of throwing away the vectors outside  $S \eqqcolon S^{(1)}$  we can iteratively re-apply Lemma E.2 to get disjoint sets  $S^{(1)}, S^{(2)}, \ldots, S^{(m)}$ , where each  $S^{(a)}$  has the same property as  $S$  (for some potentially different vector  $u^{(a)}$ ). We can then induct on the residual vectors  $\cup_{a} \{v_{i} - u^{(a)} : i \in S^{(a)}\}$ . This suffices to efficiently approximate some  $v_{i}$ . Since we throw away fewer vectors at each step of the induction, we do not need the initial number of vectors  $N$  to be as large.
We formalize the above ideas in the following theorem.
Theorem E.3. Let  $n, N, t \in \mathbb{N}$  and let  $v_{1}, \ldots, v_{N} \in \{0, 1\}^{n}$  be  $t$ -sparse binary vectors. Then there is some  $i \in [N]$  and  $x \in \mathbb{R}^{N}$  such that  $\| x \|_{1} \leq 3^{t}$  and
$$
\left\| v _ {i} - \sum_ {j \neq i} x _ {j} v _ {j} \right\| _ {2} \leq 4 ^ {t} \sqrt {9 t (t n / N) ^ {1 / t}}.
$$
Proof. We induct on  $t$ , observing that the case  $t = 0$  is immediate. Fix  $t > 0$  and  $t$ -sparse vectors  $\{v_{1},\ldots ,v_{N}\} \in \{0,1\}^{n}$ , and suppose that the theorem statement holds for  $t - 1$ . If any  $v_{i}$  is identically zero, then the claim is trivially true with  $x = 0$ . If  $N \leq t3^{t + 1}n$  then the RHS of the desired norm bound exceeds  $4^t\sqrt{t}$ , so the claim is trivially true with  $x = 0$  and any  $i \in [N]$ . Thus, we may assume that all  $v_{i}$  are nonzero, and  $N \geq t3^{t + 1}n$ . Applying the previous lemma with
$s \coloneqq 3^{t + 1} \leq N / n$  gives some  $S^{(1)} \subseteq [N]$  and nonzero  $u^{(1)} \in \{0,1\}^n$  such that  $|S^{(1)}| \geq 3^{t + 1}$  and  $u^{(1)} \preceq v_i$  for all  $i \in S^{(1)}$ , and
$$
\left\| u ^ {(1)} - \frac {1}{| S ^ {(1)} |} \sum_ {i \in S ^ {(1)}} v _ {i} \right\| _ {2} \leq \sqrt {9 t (n / N) ^ {1 / t}}.
$$
If  $|N| - |S^{(1)}| \geq N / t \geq 3^{t + 1}n$  then we can reapply the lemma with vectors  $(v_{i})_{i\in [N]\backslash S^{(1)}}$  and  $s := 3^{t + 1}$  to get some  $S^{(2)} \subseteq [N] \backslash S^{(1)}$  and  $u^{(2)} \in \{0,1\}^n$ . Continuing this process so long as there are at least  $N / t \geq 3^{t + 1}n$  remaining vectors, we can generate disjoint sets  $S^{(1)},\ldots ,S^{(m)} \subseteq [N]$  and vectors  $u^{(1)},\ldots ,u^{(m)} \in \{0,1\}^n$  with the following properties:
(i)  $|S^{(1)} \cup \dots \cup S^{(m)}| > N - N / t$
(ii)  $|S^{(a)}|\geq 3^{t + 1}$  for every  $a\in [m]$
(iii) For every  $a \in [m]$ , it holds that  $u^{(a)}$  is nonzero and  $u^{(a)} \preceq v_i$  for all  $i \in S^{(a)}$ .
(iv) For every  $a\in [m]$
$$
\left\| u ^ {(a)} - \frac {1}{| S ^ {(a)} |} \sum_ {i \in S ^ {(a)}} v _ {i} \right\| _ {2} \leq \sqrt {9 t (t n / N) ^ {1 / t}}.
$$
For each  $a \in [m]$  and  $i \in S^{(a)}$ , define  $v_i' := v_i - u^{(a)}$ . By Property (iii) we have that  $v_i' \in \{0,1\}^N$  and  $v_i'$  is  $(t-1)$ -sparse. By the inductive hypothesis applied to vectors  $(v_i')_{i \in S^{(1)} \cup \dots \cup S^{(m)}}$ , there is some  $i \in S^{(1)} \cup \dots \cup S^{(m)}$  and  $x' \in \mathbb{R}^N$  (supported on  $S^{(1)} \cup \dots \cup S^{(m)}$ ) such that  $\| x' \|_1 \leq 3^{t-1}$  and
$$
\begin{array}{l} \left\| v _ {i} ^ {\prime} - \sum_ {j \neq i} x _ {j} ^ {\prime} v _ {j} ^ {\prime} \right\| _ {2} \leq 4 ^ {t - 1} \sqrt {9 (t - 1) ((t - 1) n / | S ^ {(1)} \cup \cdots \cup S ^ {(m)} |) ^ {1 / (t - 1)}} \\ \leq 4 ^ {t - 1} \sqrt {9 t (t n / N) ^ {1 / t}} \tag {9} \\ \end{array}
$$
where the last inequality uses Property (i) and the bound  $N \geq tn$ . Of course, without loss of generality  $x_{i}^{\prime} = 0$ . Let  $a \in [m]$  be the unique index such that  $i \in S^{(a)}$ . We define  $x \in \{0,1\}^{N}$  (supported on  $S^{(1)} \cup \dots \cup S^{(m)}$ ) as follows. For each  $b \in [m]$  and each  $r \in S^{(b)}$ , set
$$
x _ {r} = x _ {r} ^ {\prime} - \frac {1}{| S ^ {(b)} |} \sum_ {j \in S ^ {(b)}} x _ {j} ^ {\prime} + \frac {\mathbb {1} [ b = a ]}{| S ^ {(b)} |}.
$$
Since  $\| x' \|_1 \leq 3^{t-1}$ , we can see that
$$
\begin{array}{l} \| x \| _ {1} \leq \| x ^ {\prime} \| _ {1} + \sum_ {b \in [ m ]} \sum_ {r \in S ^ {(b)}} \frac {1}{| S ^ {(b)} |} \sum_ {j \in S ^ {(b)}} | x _ {j} ^ {\prime} | + \sum_ {r \in S ^ {(a)}} \frac {1}{| S ^ {(a)} |} \\ \leq 2 \left\| x ^ {\prime} \right\| _ {1} + 1 \\ \leq 2 \cdot 3 ^ {t - 1} + 1. \\ \end{array}
$$
Next, we use  $x$  to approximate  $v_{i}$ . The following bound is almost what we want:
Claim E.4.  $\left\| v_{i} - \sum_{j\in [N]}x_{j}v_{j}\right\|_{2}\leq 3\cdot 4^{t - 1}\sqrt{9t(tn / N)^{1 / t}}$
Proof of claim. We have
$$
\left\| v _ {i} - \sum_ {r \in [ N ]} x _ {r} v _ {r} \right\| _ {2} \leq \left\| u ^ {(a)} - \frac {1}{| S ^ {(a)} |} \sum_ {r \in S ^ {(a)}} v _ {r} \right\| _ {2} + \left\| v _ {i} ^ {\prime} + \frac {1}{| S ^ {(a)} |} \sum_ {r \in S ^ {(a)}} v _ {r} - \sum_ {r \in [ N ]} x _ {r} v _ {r} \right\| _ {2}
$$