SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \hat {\beta} \hat {w} \right\| _ {\Sigma} \leq 4 \alpha \| w ^ {*} \| _ {\Sigma} + \sigma \sqrt {\frac {1 0 1 n _ {\mathrm {e f f}}}{\alpha^ {2} m}}.
$$
Substituting into (12) we finally get
$$
\left\| \hat {\beta} \hat {w} - w ^ {*} \right\| _ {\Sigma} ^ {2} \leq (1 - \alpha^ {2} / 2) \| w ^ {*} \| _ {\Sigma} ^ {2} + \frac {2 0 0 \sigma^ {2} n _ {\mathrm {e f f}}}{\alpha^ {2} m}
$$
F.3 Excess risk at optima of additively-regularized programs
Lemma F.7. Let  $n \in \mathbb{N}$ , and let  $\Sigma : n \times n$  be a positive semi-definite matrix. For some seminorm  $\Phi : \mathbb{R}^n \to [0, \infty)$  and some  $p, \delta > 0$ , assume that with probability at least  $1 - \delta$  over  $G \sim N(0, \Sigma)$  it holds uniformly over  $v \in \mathbb{R}^n$  that
$$
\langle v, G \rangle \leq \frac {1}{2} \Phi (v) + \sqrt {p} \| v \| _ {\Sigma}.
$$
Fix a vector  $v^{*} \in \mathbb{R}^{n}$ . For any  $m \in \mathbb{N}$  and  $\sigma > 0$  let  $(X_{i}, y_{i})_{i=1}^{m}$  be independent samples distributed as  $X_{i} \sim N(0, \Sigma)$  and  $y_{i} = \langle X_{i}, v^{*} \rangle + \xi_{i}$  where  $\xi_{i} \sim N(0, \sigma^{2})$ . Define
$$
\hat {v} \in \operatorname * {a r g m i n} _ {v \in \mathbb {R} ^ {n}} \| \mathbb {X} v - y \| _ {2} ^ {2} + \Phi (v) ^ {2} + \| y \| _ {2} \Phi (v)
$$
where  $\mathbb{X}:m\times n$  is the matrix with rows  $X_{1},\ldots ,X_{m}$ . Then with probability at least  $1 - 7\delta$  over  $(X_{i},y_{i})_{i = 1}^{m}$ , so long as  $m\geq 16p + 196\log (12 / \delta)$ , it holds that
$$
\| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq \frac {1 2 8 \sigma^ {2} p}{m} + \frac {8 (\sigma + \| v ^ {*} \| _ {\Sigma}) \Phi (v ^ {*})}{\sqrt {m}} + \frac {8 \Phi (w ^ {*}) ^ {2}}{m}.
$$
Proof. For notational convenience, define  $F(v) \coloneqq (1/2)\Phi(v - v^{*}) + \sqrt{p}\|v - v^{*}\|_{\Sigma}$ . We apply the lemma's assumption twice:
- For any fixed  $\xi$ , the random variable  $\mathbb{X}\xi$  has distribution  $N(0, \| \xi \|_2^2 \Sigma)$ . By the above claim, with probability at least  $1 - \delta$  over  $\mathbb{X}$ , we have  $\langle \xi, \mathbb{X}(v - v^*) \rangle \leq \| \xi \|_2 F(v)$  uniformly in  $v \in \mathbb{R}^n$ .
Since  $\| \xi \|_2^2 \sim \sigma^2 \chi_m^2$  and  $m \geq 8 \log(2/\delta)$ , it holds with probability at least  $1 - \delta$  that  $\frac{1}{\sqrt{m}} \| \xi \|_2 \leq \sqrt{2} \sigma$ . Thus, with probability at least  $1 - 2\delta$ , we have
$$
\langle \xi , \mathbb {X} (v - v ^ {*}) \rangle \leq \sqrt {2 m} \sigma F (v) \tag {13}
$$
uniformly in  $v\in \mathbb{R}^n$
- The assumption means that we can apply Theorem C.1 with (noiseless) samples  $(X_{i},\langle X_{i},v^{*}\rangle)_{i = 1}^{m}$  to get the following: since  $m\geq 196\log (12 / \delta)$ , it holds with probability at least  $1 - 4\delta$  over the randomness of  $\mathbb{X}$  that for all  $v\in \mathbb{R}^n$
$$
\left\| v - v ^ {*} \right\| _ {\Sigma} ^ {2} \leq \frac {2}{m} \left\| \mathbb {X} (v - v ^ {*}) \right\| _ {2} ^ {2} + \frac {2}{m} F (v) ^ {2}. \tag {14}
$$
We also observe that the entries of  $y$  are independent and identically distributed as  $N(0, \|v^*\|_{\Sigma}^2 + \sigma^2)$ , so by a  $\chi^2$  tail bound, since  $m \geq 32 \log(2/\delta)$ , it holds with probability at least  $1 - \delta$  that
$$
\frac {1}{m} \| y \| _ {2} ^ {2} \in \left[ \frac {1}{2} \left(\| v ^ {*} \| _ {\Sigma} ^ {2} + \sigma^ {2}\right), \frac {3}{2} \left(\| v ^ {*} \| _ {\Sigma} ^ {2} + \sigma^ {2}\right) \right]. \tag {15}
$$
We now condition on the event (which occurs with probability at least  $1 - 7\delta$ ) that the bounds (13), (14), and (15) all hold. Specifying (14) to  $v \coloneqq \hat{v}$ , we get that
$$
\begin{array}{l} \frac {m}{2} \left\| \hat {v} - v ^ {*} \right\| _ {\Sigma} ^ {2} \\ \leq \left\| \mathbb {X} \left(\hat {v} - v ^ {*}\right) \right\| _ {2} ^ {2} + F (\hat {v}) ^ {2} \\ \leq \left\| \mathbb {X} (\hat {v} - v ^ {*}) \right\| _ {2} ^ {2} - \left\| \mathbb {X} \hat {v} - y \right\| _ {2} ^ {2} - \Phi (\hat {v}) ^ {2} - \left\| y \right\| _ {2} \Phi (\hat {v}) \\ + \| \mathbb {X} v ^ {*} - y \| _ {2} ^ {2} + \Phi (v ^ {*}) ^ {2} + \| y \| _ {2} \Phi (v ^ {*}) + F (\hat {v}) ^ {2} \\ = 2 \langle \mathbb {X} v ^ {*} - y, \mathbb {X} (\dot {v} - v ^ {*}) \rangle \\ - \Phi (\hat {v}) ^ {2} - \| y \| _ {2} \Phi (\hat {v}) + \Phi (v ^ {*}) ^ {2} + \| y \| _ {2} \Phi (v ^ {*}) + F (\hat {v}) ^ {2} \\ \leq \sqrt {2 m} \sigma F (\hat {v}) - \Phi (\hat {v}) ^ {2} - \| y \| _ {2} \Phi (\hat {v}) + \Phi (v ^ {*}) ^ {2} + \| y \| _ {2} \Phi (v ^ {*}) + F (\hat {v}) ^ {2} \\ \end{array}
$$
where the first inequality is by (14), the second inequality is by optimality of  $\hat{v}$ , and the third inequality is by (13). We now expand  $F(\hat{v})$  in the above expression. If  $\sqrt{2mp}\sigma\|\hat{v} - v^{*}\|_{\Sigma}$  exceeds  $\frac{m}{8}\|\hat{v} - v^{*}\|_{\Sigma}^{2}$  then the lemma immediately holds since
$$
\left\| \hat {v} - v ^ {*} \right\| _ {\Sigma} ^ {2} \leq \frac {1 2 8 \sigma^ {2} p}{m}.
$$
So we may assume that in fact  $\sqrt{2mp}\sigma\|\hat{v} - v^{*}\|_{\Sigma} \leq \frac{m}{8}\|\hat{v} - v^{*}\|_{\Sigma}^{2}$ . By the lemma assumptions, we also know that  $m \geq 16p$ . Thus, expanding  $F(\hat{v})$  and applying these bounds,
$$
\begin{array}{l} \frac {m}{2} \| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \leq \sqrt {2 m} \sigma \left(\frac {1}{2} \Phi (\hat {v} - v ^ {*}) + \sqrt {p} \| \hat {v} - v ^ {*} \| _ {\Sigma}\right) \\ - \Phi (\hat {v}) ^ {2} - \| y \| _ {2} \Phi (\hat {v}) + \Phi (v ^ {*}) ^ {2} + \| y \| _ {2} \Phi (v ^ {*}) \\ + \frac {1}{2} \Phi (\hat {v} - v ^ {*}) ^ {2} + 2 p \| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \\ \leq \sqrt {\frac {m}{2}} \sigma \Phi (\hat {v} - v ^ {*}) + \frac {m}{8} \| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2} \\ - \Phi (\hat {v}) ^ {2} - \| y \| _ {2} \Phi (\hat {v}) + \Phi (v ^ {*}) ^ {2} + \| y \| _ {2} \Phi (v ^ {*}) \\ \end{array}
$$
$$
+ \frac {1}{2} \Phi (\hat {v} - v ^ {*}) ^ {2} + \frac {m}{8} \| \hat {v} - v ^ {*} \| _ {\Sigma} ^ {2}.
$$
Simplifying, applying the triangle inequality  $\Phi (\hat{v} - v^{*}) \leq \Phi (\hat{v}) + \Phi (v^{*})$ , and grouping terms, we get
$$
\begin{array}{l} \frac {m}{4} \left\| \hat {v} - v ^ {*} \right\| _ {\Gamma} ^ {2} \\ \leq \left(\sqrt {\frac {m}{2}} \sigma - \| y \| _ {2}\right) \Phi (\hat {v}) + \left(\sqrt {\frac {m}{2}} \sigma + \| y \| _ {2}\right) \Phi (v ^ {*}) + 2 \Phi (v ^ {*}) ^ {2} \\ \leq 2 (\sigma + \| v ^ {*} \| _ {\Sigma}) \sqrt {m} \Phi (v ^ {*}) + 2 \Phi (v ^ {*}) ^ {2} \\ \end{array}
$$
where the last inequality uses both sides of the bound (15).
G Covering bounds from classical assumptions
In this section, we further motivate the definition of our covering number  $\mathcal{N}_{t,\alpha}(\Sigma)$  by showing that in all settings where efficient SLR algorithms are known, there is a straightforward linear upper bound on the covering number. This lends weight to the need for stronger upper bounds on  $\mathcal{N}_{t,\alpha}$  as a stepping stone towards more efficient algorithms for sparse linear regression.
G.1 Compatibility condition
Definition G.1 (Compatibility Condition, see e.g. [40]). For a positive semidefinite matrix  $\Sigma : n \times n$ ,  $L \geq 1$ , and set  $S \subset [n]$ , we say  $\Sigma$  has  $S$ -restricted  $\ell_1$ -eigenvalue
$$
\phi^{2}(\Sigma ,S) = \min_{w\in \mathcal{C}(S)}\frac{|S|\cdot\langle w,\Sigma w\rangle}{\|w_{S}\|_{1}^{2}}
$$
where the cone  $\mathcal{C}(S)$  is defined as
$$
\mathcal {C} (S) = \{w \neq 0: \| w _ {S ^ {C}} \| _ {1} \leq L \| w _ {S} \| _ {1} \}.
$$
For  $t \in \mathbb{N}$ , the  $t$ -restricted  $\ell_1$ -eigenvalue  $\phi^2(\Sigma, t)$  is the minimum over all  $S$  of size at most  $t$ .
It is well-known that an upper bound on  $\frac{\max_i\Sigma_{ii}}{\phi^2(\Sigma,t)}$  is sufficient for the success of Lasso (as well as nearly necessary; see e.g. the Weak Compatibility Condition defined in [23]):
Theorem G.2 (see e.g. Corollary 5 in [45]). Fix  $n, m, t \in \mathbb{N}$ ,  $\sigma, \delta > 0$ , and a positive semi-definite matrix  $\Sigma: n \times n$  with  $\max_i \Sigma_{ii} \leq 1$ . Fix a  $t$ -sparse vector  $v^* \in \mathbb{R}^n$  and let  $(X_i, y_i)_{i=1}^m$  be independent samples distributed as  $X_i \sim N(0, \Sigma)$  and  $y_i = \langle X_i, v^* \rangle + \xi_i$  where  $\xi_i \sim N(0, \sigma^2)$ . Define
$$
\hat{v}\in \operatorname *{argmin}_{v\in \mathbb{R}^{n}:||v||_{1}\leq ||v^{*}||_{1}}\left\| \mathbb{X}v - y\right\|_{2}^{2}
$$
where  $\mathbb{X}:m\times n$  is the matrix with rows  $X_{1},\ldots ,X_{m}$ . If  $m\geq 4\phi^2 (\Sigma ,t)\cdot t\log (16n / \delta)$ , then with probability at least  $1 - \delta$ , it holds that
$$
\left\| \hat {v} - v ^ {*} \right\| _ {\Sigma} ^ {2} \leq O \left(\frac {\sigma^ {2} t \log (1 6 n / \delta)}{\phi^ {2} (\Sigma , t) m}\right).
$$
Fact G.3. Let  $n, t \in \mathbb{N}$ . For any positive semi-definite  $\Sigma : n \times n$  with  $\phi^2 \coloneqq \phi^2(\Sigma, t)$  and  $\max_i \Sigma_{ii} \leq 1$ , it holds that  $\mathcal{N}_{t,\phi/\sqrt{t}}(\Sigma) \leq n$ .
Proof. The proof is essentially the same as that of Fact A.4. By Lemma A.3, it suffices to show that the standard basis is a  $(t,\sqrt{t} /\phi)$ - $\ell_1$ -representation for  $\Sigma$ . Indeed, for any  $t$ -sparse  $v\in \mathbb{R}^n$ , we have
$$
\sum_ {i = 1} ^ {n} \left| v _ {i} \right| \cdot \left\| e _ {i} \right\| _ {\Sigma} \leq \left\| v \right\| _ {1} \cdot \max  _ {i} \sqrt {\Sigma_ {i i}} \leq \frac {\sqrt {t} \| v \| _ {\Sigma}}{\phi}
$$
G.2 Submodularity ratio
Definition G.4 (see e.g. [9]). For a positive semi-definite matrix  $\Sigma : n \times n$  and a set  $L \subseteq [n]$  define the normalized residual covariance matrix  $\Sigma^{(L)} : n \times n$  by
$$
\Sigma^ {(L)} := (D ^ {1 / 2}) ^ {\dagger} \left(\Sigma - \Sigma_ {L} ^ {\top} \Sigma_ {L L} ^ {\dagger} \Sigma_ {L}\right) (D ^ {1 / 2}) ^ {\dagger}
$$
where  $D\coloneqq \mathrm{diag}\left(\Sigma -\Sigma_L^\top \Sigma_{LL}^\dagger \Sigma_L\right)$
Definition G.5. Fix a positive semi-definite matrix  $\Sigma : n \times n$ , a positive integer  $t \in \mathbb{N}$ , and any  $v^{*} \in \mathbb{R}^{n}$ . Define the  $t$ -submodularity ratio of  $\Sigma$  with respect to  $v^{*}$  by
$$
\gamma_ {t} (\Sigma , v ^ {*}) := \min  _ {L, S \subseteq [ n ]: | L |, | S | \leq t, L \cap S = \emptyset} \frac {(v ^ {*}) ^ {\top} (\Sigma^ {(L)}) _ {S} ^ {\top} (\Sigma^ {(L)}) _ {S} v ^ {*}}{(v ^ {*}) ^ {\top} (\Sigma^ {(L)}) _ {S} ^ {\top} (\Sigma^ {(L)}) _ {S S} ^ {\dagger} (\Sigma^ {(L)}) _ {S} v ^ {*}}.
$$
In any  $t$ -sparse linear regression model with true regressor  $v^{*}$ , when the above quantity  $\gamma \coloneqq \gamma_{t}(\Sigma, v^{*})$  is bounded away from zero, it can be shown that the standard Forward Regression algorithm finds some  $t$ -sparse estimate  $\hat{v} \in \mathbb{R}^n$  such that  $\| \hat{v} - v^{*} \|_{\Sigma}^2 \leq e^{-\gamma} \| v^{*} \|_{\Sigma}^2$  (see e.g. Theorem 3.2 in [9]; that result is for the model where the algorithm is given exact access to  $\langle v, v^{*} \rangle_{\Sigma}$  for any  $t$ -sparse  $v \in \mathbb{R}^n$ , but analogous finite-sample bounds can be obtained with  $O(\gamma^{-O(1)} t \log(n))$  samples by applying the theorem to the empirical covariance matrix and using concentration of  $t \times t$  submatrices). A similar guarantee is also known for Orthogonal Matching Pursuit (Theorem 3.7 in [9]).
Once again, it is simple to show that the standard basis is a good dictionary for matrices with a large submodularity ratio.
Fact G.6. Let  $n, t \in \mathbb{N}$ . For any positive semi-definite  $\Sigma$ :  $n \times n$  with  $\gamma \coloneqq \min_{v^* \in \mathbb{R}^n \cap B_0(t)} \gamma_t(\Sigma, v^*)$ , it holds that  $\mathcal{N}_{t,\sqrt{\gamma / t}}(\Sigma) \leq n$ .
Proof. We show that the standard basis is a  $(t, \gamma / t)$ -dictionary for  $\Sigma$ . Without loss of generality assume that  $\Sigma_{ii} = 1$  for all  $i \in [n]$ . Then  $\Sigma^{(\emptyset)} = \Sigma$ . Fix any  $t$ -sparse  $v^* \in \mathbb{R}^n$ . Setting  $S := \operatorname{supp}(v^*)$ , we have that
$$
\sum_ {i \in S} \langle e _ {i}, v ^ {*} \rangle_ {\Sigma} ^ {2} = (v ^ {*}) ^ {\top} \Sigma_ {S} ^ {\top} \Sigma_ {S} v ^ {*} \geq \gamma (v ^ {*}) ^ {\top} \Sigma_ {S} ^ {\top} (\Sigma_ {S S}) ^ {\dagger} \Sigma_ {S} v ^ {*} = \gamma \| v ^ {*} \| _ {\Sigma} ^ {2}
$$
where the inequality is by definition of  $\gamma$ , and the final equality uses that  $\Sigma_S v^* = \Sigma_{SS}(v^*)_S$  (since  $v^*$  is supported on  $S$ ). It follows that  $\max_{i \in S} \langle e_i, v^* \rangle_{\Sigma}^2 \geq (\gamma / t) \| v^* \|_{\Sigma}^2$ . Since  $\| e_i \|_{\Sigma} = 1$  for all  $i$ , we conclude that
$$
\max _ {i \in [ n ]} \frac {| \langle e _ {i} , v ^ {*} \rangle_ {\Sigma} |}{\| e _ {i} \| _ {\Sigma} \| v ^ {*} \| _ {\Sigma}} \geq \sqrt {\frac {\gamma}{t}}
$$
G.3 Sparse preconditioning
Recent work [23] showed that if  $\Sigma : n \times n$  is a positive definite matrix and the support of  $\Theta := \Sigma^{-1}$  is the adjacency matrix of a graph with low treewidth, then there is a polynomial-time, sample-efficient algorithm for sparse linear regression with covariates drawn from  $N(0, \Sigma)$ . The key to this result was a proof that such covariance matrices are sparsely preconditionable: i.e., there is a matrix  $S : n \times n$  such that  $\Sigma = SS^{\top}$  and  $S$  has sparse rows. We claim that this property also immediately enables succinct dictionaries.
Concretely, suppose that  $S$  has  $s$ -sparse rows. By a change-of-basis argument, any  $t$ -sparse vector in the standard basis is  $st$ -sparse in the basis  $\{(S^{\top})_1^{-1}, \ldots, (S^{\top})_n^{-1}\}$ . Moreover these vectors are orthonormal under  $\Sigma$ . Thus, by the same argument as for Fact A.4, it's easy to see that  $\{(S^{\top})_1^{-1}, \ldots, (S^{\top})_n^{-1}\}$  is a  $(t, 1/\sqrt{st})$ -dictionary for  $\Sigma$ .
H Supplementary figure
Figure 2: Performance of Basis Pursuit in a synthetic example with  $n = 1000$  covariates. The covariates  $X_{1:1000}$  are all independent  $N(0,1)$  except for  $(X_0,X_1,X_2)$ , which have joint distribution  $X_0 = Z_0$ ,  $X_1 = Z_0 + 0.4Z_1$ , and  $X_2 = Z_1 + 0.4Z_2$  where  $Z_0,Z_1,Z_2\sim N(0,1)$  are independent. The noiseless responses are  $y = 6.25(X_1 - X_2) + 2.5X_3$ , i.e. the ground truth is 3-sparse. The  $x$ -axis is the number of samples. The  $y$ -axis is the out-of-sample prediction error (averaged over 10 independent runs, and error bars indicate the standard deviation).
I Experimental details
The simulations were done using Python 3.9 and the Gurobi library [17]. Each figure took several minutes to generate using a standard laptop. See the file auglasso.py for code and execution instructions.