SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption C1 imposes the standard restricted strong convexity (RSC) condition for the feature matrix  $\mathbf{X} \in \mathbb{R}^{N \times d}$ , and Assumption C2 asserts that the data is approximately generated from a sparse linear model, with a bound on the  $L_{\infty}$  norm of the error vector  $\mathbf{X}^{\top} \boldsymbol{\varepsilon}$ . Assumption C is entirely deterministic in nature, and suffices to imply the following convergence result. In the proof of Theorem 8, we show that Assumption C is satisfied with high probability when data is generated from the standard sparse linear model considered therein.
Theorem H.1 (Sharper convergence guarantee for Lasso). Under Assumption  $C$ , for the PGD iterates  $\{\mathbf{w}^t\}_{t\geq 0}$  on loss function  $\widehat{L}_{\mathrm{lasso}}$  with stepsize  $\eta = 1 / \beta$  and starting point  $\mathbf{w}^0 = \mathbf{0}$ , we have  $\widehat{L}_{\mathrm{lasso}}(\mathbf{w}^T) - \widehat{L}_{\mathrm{lasso}}(\mathbf{w}_{\mathrm{lasso}})\leq \varepsilon$  for all
$$
T \geq C \Bigg [ \frac {\beta (B _ {w} ^ {\star}) ^ {2}}{\nu} + \kappa \log \left(C \cdot \kappa \cdot \frac {\beta (B _ {w} ^ {\star}) ^ {2}}{\nu} \cdot \frac {\nu}{\varepsilon}\right) + \kappa \frac {\nu \omega_ {N} ^ {2}}{\varepsilon} \Bigg ],
$$
where  $C$  is a universal constant.
The proof can be found in Appendix H.4. Combining Theorem H.1 with the construction in Theorem 7, we directly obtain the following result as a corollary.
Theorem H.2 (In-context Lasso with transformers with sharper convergence). For any  $N, d, s \geq 1$ ,  $0 < \alpha \leq \beta$ ,  $\nu \geq 0$ ,  $\rho \geq 0$ , there exists a  $L$ -layer transformer  $\mathrm{TF}_{\theta}$  with
$$
L = \left\lceil C \big (\kappa_ {s} + \kappa \big (\log (C \kappa_ {s} / \varepsilon) + \nu \omega_ {N} ^ {2} / \varepsilon \big) \big) \right\rceil , \quad \max  _ {\ell \in [ L ]} M ^ {(\ell)} \leq 2, \quad \max  _ {\ell \in [ L ]} D ^ {(\ell)} \leq 2 d,
$$
$$
\| \boldsymbol {\theta} \| \leq 3 + R + (8 + 2 \lambda_ {N}) \beta^ {- 1},
$$
such that the following holds. On any input data  $(\mathcal{D},\mathbf{x}_{N + 1})$  such that the (ICLasso) problem satisfies Assumption  $C$  (which implies  $\| \mathbf{w}_{\mathrm{lasso}}\| _2\leq B_w / 2$  with  $B_{w} = 2B_{w}^{\star} + \sqrt{\nu / \alpha}$ ),  $\mathrm{TF}_{\pmb{\theta}}(\mathbf{H}^{(0)})$  approximately implements (ICLasso), in that it outputs  $\widehat{y}_{N + 1} = \mathrm{read}_y(\mathrm{TF}_{\pmb{\theta}}(\mathbf{H})) = \langle \mathbf{x}_{N + 1},\widehat{\mathbf{w}}\rangle$  with
$$
\widehat {L} _ {\mathrm {l a s s o}} (\widehat {\mathbf {w}}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\mathrm {l a s s o}}) \leq \varepsilon .
$$
H.3 Basic properties for Lasso
Lemma H.1 (Relaxed basic inequality). Suppose that Assumption C2 holds. Then it holds that
$$
\left\| \mathbf {w} - \mathbf {w} _ {\star} \right\| _ {1} \leq 4 \sqrt {s} \left\| \mathbf {w} - \mathbf {w} _ {\star} \right\| _ {2} + \frac {2}{\lambda_ {N}} \Big (\widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\star}) \Big), \quad \forall \mathbf {w} \in \mathbb {R} ^ {d}.
$$
As a corollary,  $\| \mathbf{w}_{\mathrm{lasso}} - \mathbf{w}_{\star}\| _1\leq 4\sqrt{s}\| \mathbf{w}_{\mathrm{lasso}} - \mathbf{w}_{\star}\| _2$
Proof. Let us first fix any  $\mathbf{w} \in \mathbb{R}^d$ . Denote  $\pmb{\Delta} = \mathbf{w} - \mathbf{w}_{\star}$ , and let  $S = \mathrm{supp}(\mathbf{w}_{\star})$  be the set of indexes of nonzero entries of  $\mathbf{w}_{\star}$ . Then by definition,  $\mathbf{y} = \mathbf{X}\mathbf{w}_{\star} + \varepsilon$  and  $|S| \leq s$ , and hence
$$
\begin{array}{l} \left\| \mathbf {X} \mathbf {w} - \mathbf {y} \right\| _ {2} ^ {2} - \left\| \mathbf {X} \mathbf {w} _ {\star} - \mathbf {y} \right\| _ {2} ^ {2} = \left\| \mathbf {X} \boldsymbol {\Delta} - \boldsymbol {\varepsilon} \right\| _ {2} ^ {2} - \left\| \boldsymbol {\varepsilon} \right\| _ {2} ^ {2} = \left\| \mathbf {X} \boldsymbol {\Delta} \right\| _ {2} ^ {2} - 2 \boldsymbol {\varepsilon} ^ {\top} \mathbf {X} \boldsymbol {\Delta}, \\ \left\| \mathbf {w} \right\| _ {1} - \left\| \mathbf {w} _ {\star} \right\| _ {1} = \sum_ {j \in S} \left(\left| \mathbf {w} [ j ] \right| - \left| \mathbf {w} _ {\star} [ j ] \right|\right) + \sum_ {j \not \in S} \left| \mathbf {w} [ j ] \right| \\ \geq - \sum_ {j \in S} | \mathbf {w} [ j ] - \mathbf {w} _ {\star} [ j ] | + \sum_ {j \not \in S} | \mathbf {w} [ j ] | = \| \boldsymbol {\Delta} _ {S ^ {c}} \| _ {1} - \| \boldsymbol {\Delta} _ {S} \| _ {1}. \\ \end{array}
$$
$$
\begin{array}{l} 0 \leq \frac {1}{2 N} \| \mathbf {X} \boldsymbol {\Delta} \| _ {2} ^ {2} \leq \frac {\varepsilon^ {\top} \mathbf {X} \boldsymbol {\Delta}}{N} + \lambda_ {N} \left(\| \boldsymbol {\Delta} _ {S} \| _ {1} - \| \boldsymbol {\Delta} _ {S ^ {c}} \| _ {1}\right) + \widehat {L} _ {\text {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\text {l a s s o}} (\mathbf {w} _ {\star}) \\ \leq \frac {\lambda_ {N}}{2} \| \boldsymbol {\Delta} \| _ {1} + \lambda_ {N} \left(\| \boldsymbol {\Delta} _ {S} \| _ {1} - \| \boldsymbol {\Delta} _ {S ^ {c}} \| _ {1}\right) + \widehat {L} _ {\text {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\text {l a s s o}} (\mathbf {w} _ {\star}) \tag {39} \\ = \frac {\lambda_ {N}}{2} (3 \| \pmb {\Delta} _ {S} \| _ {1} - \| \pmb {\Delta} _ {S ^ {c}} \| _ {1}) + \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\star}), \\ \end{array}
$$
where the second inequality follows from  $\frac{\varepsilon^{\top}\mathbf{X}\boldsymbol{\Delta}}{N} \leq \frac{\|\mathbf{X}^{\top}\boldsymbol{\varepsilon}\|_{\infty}}{N}\|\boldsymbol{\Delta}\|_{1}$  and our assumption that  $2\frac{\|\mathbf{X}^{\top}\boldsymbol{\varepsilon}\|_{\infty}}{N} \leq \lambda_{N}$ , and the last inequality is due to  $\|\boldsymbol{\Delta}\|_{1} = \|\boldsymbol{\Delta}_{S}\|_{1} + \|\boldsymbol{\Delta}_{Sc}\|_{1}$ . Therefore, we have
$$
\begin{array}{l} \| \boldsymbol {\Delta} \| _ {1} = \| \boldsymbol {\Delta} _ {S} \| _ {1} + \| \boldsymbol {\Delta} _ {S ^ {c}} \| _ {1} \leq 4 \| \boldsymbol {\Delta} _ {S} \| _ {1} + \frac {2}{\lambda_ {N}} \Big (\widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\star}) \Big) \\ \leq 4 \sqrt {s} \left\| \boldsymbol {\Delta} \right\| _ {2} + \frac {2}{\lambda_ {N}} \Big (\widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\star}) \Big), \\ \end{array}
$$
where the last inequality follows from  $\| \pmb{\Delta}_S\|_1 \leq \sqrt{s} \| \pmb{\Delta}_S\|_2 \leq \sqrt{s}\|\pmb{\Delta}\|_2$ . This completes the proof of our main inequality. As for the corollary, we only need to use the definition that  $\widehat{L}_{\mathrm{lasso}}(\mathbf{w}_{\mathrm{lasso}}) \leq \widehat{L}_{\mathrm{lasso}}(\mathbf{w}_{\star})$ .
Proposition H.1 (Gap to parameter estimation error). Suppose that Assumption  $C$  holds. Then for all  $\mathbf{w} \in \mathbb{R}^d$ ,
$$
\left\| \mathbf {w} - \mathbf {w} _ {\star} \right\| _ {2} ^ {2} \leq C \left[ \frac {s \lambda_ {N} ^ {2}}{\alpha^ {2}} + \nu^ {- 1} \operatorname {g a p} ^ {2} + \operatorname {g a p} \right],
$$
where we write  $\mathsf{gap} := \widehat{L}_{\mathrm{lasso}}(\mathbf{w}) - \widehat{L}_{\mathrm{lasso}}(\mathbf{w}_{\mathrm{lasso}})$ , and  $C = 120$  is a universal constant. In particular, we have  $\| \mathbf{w}_{\mathrm{lasso}} - \mathbf{w}_{\star}\|_2^2 \leq 10\frac{\rho\nu}{\alpha^2}\frac{s\log d}{N}$ .
Proof. We follow the notation in the proof of Lemma H.1. By (39), we have
$$
0 \leq \frac {1}{2 N} \| \mathbf {X} \boldsymbol {\Delta} \| _ {2} ^ {2} \leq \frac {\lambda_ {N}}{2} (3 \| \boldsymbol {\Delta} _ {S} \| _ {1} - \| \boldsymbol {\Delta} _ {S ^ {c}} \| _ {1}) + \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\star}),
$$
and hence  $\| \pmb{\Delta}\|_1\leq 4\sqrt{s}\|\pmb{\Delta}\|_2 + \frac{2\mathrm{gap}}{\lambda_N}$  due to  $\widehat{L}_{\mathrm{lasso}}(\mathbf{w}) - \widehat{L}_{\mathrm{lasso}}(\mathbf{w}_\star)\leq \mathrm{gap}$ . On the other hand, by the RSC condition (38), it holds that
$$
\frac {\left\| \mathbf {X} \boldsymbol {\Delta} \right\| _ {2} ^ {2}}{N} \geq \alpha \left\| \boldsymbol {\Delta} \right\| _ {2} ^ {2} - \rho \frac {\log d}{N} \left\| \boldsymbol {\Delta} \right\| _ {1} ^ {2}.
$$
$$
\begin{array}{l} \alpha \left\| \boldsymbol {\Delta} \right\| _ {2} ^ {2} \leq 3 \lambda_ {N} \sqrt {s} \left\| \boldsymbol {\Delta} \right\| _ {2} + \rho \frac {\log d}{N} \left\| \boldsymbol {\Delta} \right\| _ {1} ^ {2} + 2 \mathrm {g a p} \\ \leq 3 \lambda_ {N} \sqrt {s} \| \boldsymbol {\Delta} \| _ {2} + \rho \frac {\log d}{N} \left(4 \sqrt {s} \| \boldsymbol {\Delta} \| _ {2} + \frac {2 \mathrm {g a p}}{\lambda_ {N}}\right) ^ {2} + 2 \mathrm {g a p} \\ \leq \frac {5 s \lambda_ {N} ^ {2}}{\alpha} + \frac {\alpha}{6} \| \boldsymbol {\Delta} \| _ {2} ^ {2} + \rho \frac {2 0 s \log d}{\lambda_ {N} ^ {2} N} \| \boldsymbol {\Delta} \| _ {2} ^ {2} + \rho \frac {2 0 \log d}{N} \mathrm {g a p} ^ {2} + 2 \mathrm {g a p}, \\ \end{array}
$$
where the last inequality uses AM-GM inequality and Cauchy inequality. Notice that  $\rho \frac{20s\log d}{N}\leq \frac{2}{3}\alpha$  we now derive that
$$
\left\| \boldsymbol {\Delta} \right\| _ {2} ^ {2} \leq \frac {3 0 s \lambda_ {N} ^ {2}}{\alpha^ {2}} + \rho \frac {1 2 0 \log d}{\lambda_ {N} ^ {2} N} \text {g a p} ^ {2} + 1 2 \text {g a p}.
$$
Plugging in  $\lambda_{N} = \sqrt{\frac{\rho\nu\log d}{N}}$  completes the proof. The corollary follows immediately by letting  $\mathbf{w} = \mathbf{w}_{\mathrm{lasso}}$  in above proof (hence gap  $= 0$ ).
Lemma H.2 (Growth). It holds that
$$
\frac {1}{2 N} \left\| \mathbf {X} (\mathbf {w} - \mathbf {w} _ {\mathrm {l a s s o}}) \right\| _ {2} ^ {2} \leq \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w}) - \widehat {L} _ {\mathrm {l a s s o}} (\mathbf {w} _ {\mathrm {l a s s o}}), \quad \forall \mathbf {w}.
$$
Proof. For simplicity we denote  $\mathbf{w}_{\mathrm{lasso}} := \mathbf{w}_{\mathrm{lasso}}$ . By the first order optimality condition, it holds that
$$
0 \in \frac {1}{N} \mathbf {X} ^ {\top} (\mathbf {X} \mathbf {w} _ {\mathrm {l a s s o}} - \mathbf {y}) + \partial R (\mathbf {w} _ {\mathrm {l a s s o}}),
$$
where we write  $R(\mathbf{w})\coloneqq \lambda_N\| \mathbf{w}\| _1$  . Then by the convexity of  $R$  , we have
$$
\begin{array}{l} R (\mathbf {w}) - R (\mathbf {w} _ {\text {l a s s o}}) \geq \left\langle \partial R (\mathbf {w} _ {\text {l a s s o}}), \mathbf {w} - \mathbf {w} _ {\text {l a s s o}} \right\rangle = \left\langle - \frac {1}{N} \mathbf {X} ^ {\top} (\mathbf {X} \mathbf {w} _ {\text {l a s s o}} - \mathbf {y}), \mathbf {w} - \mathbf {w} _ {\text {l a s s o}} \right\rangle \\ = - \frac {1}{N} \left\langle \mathbf {X w} _ {\mathrm {l a s s o}} - \mathbf {y}, (\mathbf {X w} - \mathbf {y}) - (\mathbf {X w} _ {\mathrm {l a s s o}} - \mathbf {y}) \right\rangle \\ = - \frac {1}{2 N} \| \mathbf {X} \mathbf {w} - \mathbf {y} \| _ {2} ^ {2} + \frac {1}{2 N} \| \mathbf {X} \mathbf {w} _ {\mathrm {l a s s o}} - \mathbf {y} \| _ {2} ^ {2} + \frac {1}{2 N} \| \mathbf {X} (\mathbf {w} - \mathbf {w} _ {\mathrm {l a s s o}}) \| _ {2} ^ {2}. \\ \end{array}
$$
Rearranging completes the proof.
H.4 Proof of Theorem H.1
For the simplicity of presentation, we write  $\mathbf{w}_{\mathrm{lasso}} = \mathbf{w}_{\mathrm{lasso}}$  and we denote  $\mathsf{gap}^t \coloneqq \widehat{L}_{\mathrm{lasso}}(\mathbf{w}^t) - \widehat{L}_{\mathrm{lasso}}(\mathbf{w}_{\mathrm{lasso}})$ .
By Lemma H.1, we have  $\| \mathbf{w}^t -\mathbf{w}_\star \| _1\leq 4\sqrt{s}\| \mathbf{w}^t -\mathbf{w}_\star \| _2 + \frac{2\mathrm{gap}^t}{\lambda_N}$ , which implies
$$
\left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {1} \leq \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\star} \right\| _ {1} + \left\| \mathbf {w} _ {\mathrm {l a s s o}} - \mathbf {w} _ {\star} \right\| _ {1} \leq 4 \sqrt {s} \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} + 8 \sqrt {s} \left\| \mathbf {w} _ {\mathrm {l a s s o}} - \mathbf {w} _ {\star} \right\| _ {2} + \frac {2 g a p ^ {t}}{\lambda_ {N}}.
$$
We denote  $\mu_N = \rho^2\frac{\log d}{N}$ . Using the assumption that  $\mathbf{X}$  is  $(\alpha, \rho)$ -RSC, we obtain that
$$
\begin{array}{l} \frac {1}{N} \left\| X \left(\mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}}\right) \right\| _ {2} ^ {2} \geq \alpha \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} - \mu_ {N} \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {1} ^ {2} \\ \geq \alpha \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} - \mu_ {N} \left(2 0 s \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} + 6 4 0 s \| \mathbf {w} _ {\text {l a s s o}} - \mathbf {w} _ {\star} \| _ {2} ^ {2} + \frac {4 0}{\lambda_ {N} ^ {2}} (\mathsf {g a p} ^ {t}) ^ {2}\right). \\ \end{array}
$$
Thus, as long as  $N \geq \frac{30\rho^2s\log d}{\alpha}$ , we have
$$
\begin{array}{l} \frac {\alpha}{3} \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} \leq \frac {1}{N} \left\| \mathbf {X} \left(\mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}}\right) \right\| _ {2} ^ {2} + 6 4 0 s \mu_ {N} \left\| \mathbf {w} _ {\text {l a s s o}} - \mathbf {w} _ {\star} \right\| _ {2} ^ {2} + \frac {4 0 \mu_ {N}}{\lambda_ {N} ^ {2}} \left(\mathsf {g a p} ^ {t}\right) ^ {2} \\ \leq 2 \operatorname {g a p} ^ {t} + 4 0 \nu^ {- 1} (\operatorname {g a p} ^ {t}) ^ {2} + 6 4 0 s \mu_ {N} \| \mathbf {w} _ {\text {l a s s o}} - \mathbf {w} _ {\star} \| _ {2} ^ {2}, \\ \end{array}
$$
where the last inequality follows from Lemma H.2 and the definition of  $\lambda_N, \mu_N$ .
We define  $\varepsilon_{\mathrm{stat}}\coloneqq 640s\mu_N\| \mathbf{w}_{\mathrm{lasso}} - \mathbf{w}_\star \| _2^2$ $T_0\coloneqq 10\beta \nu^{-1}\| \mathbf{w}_{\mathrm{lasso}}\| _2^2$  By Proposition B.3(3), it holds that for  $t\geq T_0$
$$
\mathsf {g a p} ^ {t} \leq \frac {\beta}{2 t} \left\| \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} \leq \frac {\beta}{2 T _ {0}} \left\| \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} = \frac {\nu}{2 0}.
$$
Then for all  $t \geq T_0 - 1$ , we have (the second  $\leq$  below uses Proposition B.3(2))
$$
\begin{array}{l} \frac {\alpha}{3} \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} \leq 4 \mathrm {g a p} ^ {t + 1} + \varepsilon_ {\text {s t a t}} \leq 2 \beta \left(\left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} - \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2}\right) + \varepsilon_ {\text {s t a t}}, \\ \Rightarrow \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} ^ {2} - \frac {3 \varepsilon_ {\mathrm {s t a t}}}{\alpha} \leq \left(1 + \frac {\alpha}{6 \beta}\right) ^ {- 1} \left(\left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} ^ {2} - \frac {3 \varepsilon_ {\mathrm {s t a t}}}{\alpha}\right). \\ \end{array}
$$
Therefore, for  $t \geq T_0 - 1$ ,
$$
\begin{array}{l} \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} ^ {2} \leq \exp \left(- \frac {\alpha}{1 2 \beta} (t - \lceil T _ {0} \rceil + 1)\right) \left\| \mathbf {w} ^ {\lceil T _ {0} \rceil - 1} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} ^ {2} + \frac {3 \varepsilon_ {\mathrm {s t a t}}}{\alpha} \\ \leq \exp \left(- \frac {\alpha}{8 \beta} (t - T _ {0})\right) \| \mathbf {w} _ {\mathrm {l a s s o}} \| _ {2} ^ {2} + \frac {3 \varepsilon_ {\mathrm {s t a t}}}{\alpha}, \\ \end{array}
$$
where the last inequality follows from Proposition B.3(2). Further, by Proposition B.3(3), we have
$$
\mathsf {g a p} ^ {t + k} \leq \frac {\beta}{2 k} \left\| \mathbf {w} ^ {t} - \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} ^ {2} \leq \frac {\beta}{2 k} \left[ \exp \left(- \frac {\alpha}{8 \beta} (t - T _ {0})\right) \| \mathbf {w} _ {\mathrm {l a s s o}} \| _ {2} ^ {2} + \frac {3 \varepsilon_ {\mathrm {s t a t}}}{\alpha} \right], \quad \forall t \geq T _ {0} - 1, k \geq 0.
$$
Hence, we can conclude that  $\mathsf{gap}^T\leq \varepsilon$  for all  $T$  such that
$$
T \geq 1 0 \beta \nu^ {- 1} \| \mathbf {w} _ {\text {l a s s o}} \| _ {2} ^ {2} + 8 \kappa \log \left(\frac {\beta \| \mathbf {w} _ {\text {l a s s o}} \| _ {2} ^ {2}}{\varepsilon}\right) + \frac {3 \kappa \varepsilon_ {\text {s t a t}}}{\varepsilon} + 1.
$$
Now, by Proposition H.1, it holds that  $\| \mathbf{w}_{\mathrm{lasso}} - \mathbf{w}_{\star}\| _2^2\leq 10\frac{\rho\nu}{\alpha^2}\frac{s\log d}{N}$ , and hence
$$
\left\| \mathbf {w} _ {\text {l a s s o}} \right\| _ {2} ^ {2} \leq 2 \left\| \mathbf {w} _ {\star} \right\| _ {2} ^ {2} + 2 \left\| \mathbf {w} _ {\text {l a s s o}} - \mathbf {w} _ {\star} \right\| _ {2} ^ {2} \leq 2 \left(B _ {w} ^ {\star}\right) ^ {2} + \frac {2 0 \rho v s \log d}{\alpha^ {2} N}.
$$
Plugging in our definition of
$$
\mu_ {N} = \frac {\rho \log d}{N}, \qquad \varepsilon_ {\mathrm {s t a t}} := 4 0 0 s \mu_ {N} \left\| \mathbf {w} _ {\text {l a s s o}} - \mathbf {w} _ {\star} \right\| _ {2} ^ {2}, \qquad \omega_ {N} = \frac {\rho}{\alpha} \frac {s \log d}{N} \leq 1
$$
H.5 Proof of Theorem 8
In this section, we present the proof of Theorem 8 based on Theorem H.2. We begin by recalling the following RSC property of a Gaussian random matrix [87, Theorem 7.16], a classical result in the high-dimensional statistics literature.
Proposition H.2 (RSC for Gaussian random design). Suppose that  $\mathbf{X} = [\mathbf{x}_1;\dots ;\mathbf{x}_N]^\top \in \mathbb{R}^{N\times d}$  is a random matrix with each row  $\mathbf{x}_i$  being i.i.d. samples from  $\mathsf{N}(0,\Sigma)$ . Then there are universal constants  $c_{1} = \frac{1}{8}, c_{2} = 50$  such that with probability at least  $1 - \frac{e^{-N / 32}}{1 - e^{-N / 32}}$ ,
$$
\frac {\left\| \mathbf {X} \mathbf {w} \right\| _ {2} ^ {2}}{N} \geq c _ {1} \left\| \mathbf {w} \right\| _ {\boldsymbol {\Sigma}} ^ {2} - c _ {2} \rho (\boldsymbol {\Sigma}) \frac {\log d}{N} \left\| \mathbf {w} \right\| _ {1} ^ {2}, \quad \forall \mathbf {w} \in \mathbb {R} ^ {d}, \tag {40}
$$
where  $\rho (\pmb {\Sigma}) = \max_{i\in [d]}\Sigma_{ii}$  is the maximum of diagonal entries of  $\pmb{\Sigma}$ .
Fix a parameter  $\delta_1 \leq \delta$  (which we will specify in proof) and a large universal constant  $C_0$ . Let us set
$$
\alpha = c _ {1} = \Theta (1), \quad \beta = 8 (1 + (d / N)), \quad \rho = c _ {2} = \Theta (1),
$$
$$
B _ {x} = C _ {0} \sqrt {d \log (N / \delta_ {1})}, \qquad B _ {y} = C _ {0} (B _ {w} ^ {\star} + \sigma) \sqrt {\log (N / \delta_ {1})}.
$$
Similar to the proof of Corollary 6 (Appendix F.4), we consider the following good events (where  $\varepsilon = \mathbf{X}\mathbf{w}_{\star} - \mathbf{y}$ )
$$
\mathcal {E} _ {w} = \left\{\lambda_ {\max } \left(\mathbf {X} ^ {\top} \mathbf {X} / N\right) \leq \beta \text {a n d} \mathbf {X} \text {i s} (\alpha , \rho) \text {- R S C} \right\},
$$
$$
\mathcal {E} _ {r} = \left\{\left\| \mathbf {X} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {\infty} \geq 4 \sigma \sqrt {N \log (4 d / \delta)} \right\},
$$
and we define  $\mathcal{E} \coloneqq \mathcal{E}_w \cap \mathcal{E}_r \cap \mathcal{E}_b \cap \mathcal{E}_{b,N+1}$ .
Furthermore, we choose  $\nu > 0$  that correspond to the choice  $\lambda_{N} = 8\sigma \sqrt{\frac{\log(4d / \delta)}{N}}$ , and we also assume  $N \geq \frac{32c_2}{c_1} \cdot s\log d$ . Then, Assumption C holds on the event  $\mathcal{E}$ .
Therefore, we can apply Theorem H.2 with  $\varepsilon = \nu \omega_{N}$ , which implies that there exists a  $L$ -layer transformer  $\pmb{\theta}$  such that its prediction  $\widehat{y}_{N + 1} \coloneqq \widetilde{\mathrm{read}}_{\mathbf{y}}(\mathrm{TF}_{\pmb{\theta}}^{0}(\mathbf{H}))$ , so that under the good event  $\mathcal{E}$  we have  $\widehat{y}_{N + 1} = \mathrm{clip}_{B_y}(\langle \mathbf{x}_{N + 1}, \widehat{\mathbf{w}} \rangle)$ , where
$$
L _ {\text {l a s s o}} (\widehat {\mathbf {w}}) - L _ {\text {l a s s o}} \left(\mathbf {w} _ {\text {l a s s o}}\right) \leq \nu \omega_ {N}.
$$
In the following, we show that  $\theta$  is indeed the desired transformer (similarly to the proof in Appendix F.4). Consider the conditional prediction error
$$
\mathbb {E} \left[ \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right] = \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right] + \mathbb {E} \left[ 1 \{\mathcal {E} ^ {c} \} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right],
$$
and we analyze these two parts separately under the good event  $\mathcal{E}_0\coloneqq \mathcal{E}_w\cap \mathcal{E}_r\cap \mathcal{E}_b$  of  $\mathcal{D}$
Part I. We first note that
$$
\begin{array}{l} \mathbb {E} \left[ 1 \{\mathcal {E} \} (\widehat {y} _ {N + 1} - y _ {N + 1}) ^ {2} \mid \mathcal {D} \right] = \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\operatorname {c l i p} _ {B _ {y}} \left(\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} \rangle\right) - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right] \\ \leq \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\left\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} \right\rangle - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right], \\ \end{array}
$$
where the inequality is because  $y_{N + 1}\in [-B_y,B_y]$  under the good event  $\mathcal{E}$ . Notice that by our construction, under the good event  $\mathcal{E}$ ,  $\widehat{\mathbf{w}} = \widehat{\mathbf{w}} (\mathcal{D})$  depends only on the dataset  $\mathcal{D}$  (because it is the  $(L - 1)$ -th iterate of PGD on (ICLasso) problem). Applying Proposition H.1 to  $\widehat{\mathbf{w}} (\mathcal{D})$  and using the definition of  $\omega_{N}$  and our choice of  $\lambda_N$ , we obtain that (under  $\mathcal{E}_0$ )
$$
\left\| \widehat {\mathbf {w}} (\mathcal {D}) - \mathbf {w} _ {\star} \right\| _ {2} ^ {2} \leq C \cdot \left[ \frac {s \lambda_ {N} ^ {2}}{\alpha^ {2}} + \nu \omega_ {N} ^ {2} + \nu \omega_ {N} \right] = \mathcal {O} \left(\frac {\sigma^ {2} s \log (d / \delta)}{N}\right).
$$
Therefore, under  $\mathcal{E}_0$
$$
\begin{array}{l} \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} \rangle - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right] = \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} (\mathcal {D}) \rangle - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right] \\ \leq \mathbb {E} \left[ \left(\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} (\mathcal {D}) \rangle - y _ {N + 1}\right) ^ {2} \mid \mathcal {D} \right] \\ = \mathbb {E} \left[ \left(\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} (\mathcal {D}) \rangle - \langle \mathbf {x} _ {N + 1}, \mathbf {w} _ {\star} \rangle\right) ^ {2} \mid \mathcal {D} \right] + \sigma^ {2} \\ = \left\| \widehat {\mathbf {w}} (\mathcal {D}) - \mathbf {w} _ {\star} \right\| _ {2} ^ {2} + \sigma^ {2} \\ = \sigma^ {2} \left[ 1 + \mathcal {O} \left(\frac {s \log (d / \delta)}{N}\right) \right]. \\ \end{array}
$$
Part II. Notice that under good event  $\mathcal{E}_0$ , the bad event  $\mathcal{E}^c$  holds if and only if  $\mathcal{E}_{b,N+1}^c$  holds, and hence
$$
\begin{array}{l} \mathbb {E} \left[ 1 \{\mathcal {E} ^ {c} \} (\widehat {y} _ {N + 1} - y _ {N + 1}) ^ {2} \mid \mathcal {D} \right] = \mathbb {E} \left[ 1 \{\mathcal {E} _ {b, N + 1} ^ {c} \} (\widehat {y} _ {N + 1} - y _ {N + 1}) ^ {2} \mid \mathcal {D} \right] \\ \leq \sqrt {\mathbb {P} (\mathcal {E} _ {b , N + 1} ^ {c}) \mathbb {E} [ (\widehat {y} _ {N + 1} - y _ {N + 1}) ^ {4} ]}. \\ \end{array}
$$
With a large enough constant  $C_0$ , we clearly have  $\mathbb{P}(\mathcal{E}_{b,N + 1}^c)\leq (\delta_1 / N)^{10}$ . Further, a simple calculation yields
$$
\mathbb {E} (\widehat {y} _ {N + 1} - y _ {N + 1}) ^ {4} \leq 8 \mathbb {E} (\widehat {y} _ {N + 1} ^ {4} + y _ {N + 1} ^ {4}) \leq 8 B _ {y} ^ {4} + 8 \mathbb {E} y _ {N + 1} ^ {4} \leq 1 6 B _ {y} ^ {4},
$$