SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \eta^ {- 1} \mathbf {g} (\mathbf {w}) + \nabla \widehat {L} _ {N} (\mathbf {w}) \right\| _ {2} \leq \sqrt {K} B _ {x} \cdot [ \varepsilon_ {p} + B _ {u} L _ {r} (\varepsilon_ {\ell} + B _ {u} L _ {\ell} \varepsilon_ {r}) ] + 2 \sqrt {K} [ B _ {g} \varepsilon_ {r} + B _ {r} (\varepsilon_ {\ell} + B _ {u} L _ {\ell} \varepsilon_ {r}) ].
$$
Thus, to ensure  $\left\| \eta^{-1}\mathbf{g}(\mathbf{w}) + \nabla \widehat{L}_N(\mathbf{w})\right\| _2\leq \varepsilon$  , we only need to choose  $\varepsilon_{p},\varepsilon_{\ell},\varepsilon_{r}$  as
$$
\varepsilon_ {p} = \frac {\varepsilon}{3 \sqrt {K} B _ {x}}, \qquad \varepsilon_ {\ell} = \frac {\varepsilon}{9 \sqrt {K} \max  \left\{B _ {r} , L _ {r} B _ {x} B _ {u} \right\}}, \qquad \varepsilon_ {r} = \frac {\varepsilon}{1 5 \sqrt {K} \max  \left\{B _ {g} , L _ {\ell} B _ {r} B _ {u} , L _ {r} L _ {\ell} B _ {x} B _ {r} B _ {u} ^ {2} \right\}}.
$$
Thus, letting the attention layer  $\pmb{\theta}_{\mathrm{attn}}^{(2)} = \{(\mathbf{V}_{k,w,m}^{(2)},\mathbf{Q}_{k,w,m}^{(2)},\mathbf{K}_{k,w,m}^{(2)})\}_{(k,w,m)}$ , we have
$$
\operatorname {A t t n} _ {\boldsymbol {\theta} _ {\mathrm {a t t n}} ^ {(2)}}: \mathbf {h} _ {i} ^ {(1)} \mapsto \mathbf {h} _ {i} ^ {(1. 5)} = [ \mathbf {x} _ {i}; y _ {i} ^ {\prime}; \mathbf {w} + \eta \mathbf {g} (\mathbf {w}); \overline {{\operatorname {p r e d}}} (\mathbf {x} _ {i}; \mathbf {w}); g _ {i}; \mathbf {0}; 1; t _ {i} ].
$$
Step 4: construction of  $\theta_{\mathrm{mlp}}^{(2)}$ . We only need to pick  $\theta_{\mathrm{mlp}}^{(2)}$  so that it maps
$$
\mathbf {h} _ {i} ^ {(1. 5)} = [ \mathbf {x} _ {i}; y _ {i} ^ {\prime}; \mathbf {w} + \eta \mathbf {g} (\mathbf {w}); \overline {{\mathrm {p r e d}}} (\mathbf {x} _ {i}; \mathbf {w}); g _ {i}; \mathbf {0}; 1; t _ {i} ] \xrightarrow [ \underset {\mathrm {m l p}} {\mathrm {M L P}} ]{\mathrm {M L P} _ {\boldsymbol {\theta} _ {\mathrm {m l p}} ^ {(2)}}} \mathbf {h} _ {i} ^ {(2)} = [ \mathbf {x} _ {i}; y _ {i} ^ {\prime}; \operatorname {P r o j} _ {\mathcal {W}} (\mathbf {w} - \eta \mathbf {g} (\mathbf {w})); 0; 0; \mathbf {0}; 1; t _ {i} ].
$$
By our assumption on the map  $\mathrm{Proj}_{\mathcal{W}}$ , this is easy.
Combining the four steps above and taking  $\pmb{\theta} = (\pmb{\theta}_{\mathrm{attn}}^{(1)},\pmb{\theta}_{\mathrm{mlp}}^{(1)},\pmb{\theta}_{\mathrm{attn}}^{(2)},\pmb{\theta}_{\mathrm{mlp}}^{(2)})$  completes the proof.
E.6 Proof of Lemma D.3
For every  $\ell \geq 0$ , define the intermediate iterates (before projection)
$$
\widehat {\mathbf {w}} ^ {\ell + \frac {1}{2}} := \widehat {\mathbf {w}} ^ {\ell} - \eta \big (\nabla f \big (\widehat {\mathbf {w}} ^ {\ell} \big) + \boldsymbol {\varepsilon} ^ {\ell} \big), \quad \mathbf {w} _ {\mathrm {G D}} ^ {\ell + \frac {1}{2}} := \mathbf {w} _ {\mathrm {G D}} ^ {\ell} - \eta \nabla f \big (\mathbf {w} _ {\mathrm {G D}} ^ {\ell} \big),
$$
so that  $\widehat{\mathbf{w}}^{\ell +1} = \mathrm{Proj}_{\mathcal{W}}(\widehat{\mathbf{w}}^{\ell +\frac{1}{2}})$  and  $\mathbf{w}_{\mathrm{GD}}^{\ell +1} = \mathrm{Proj}_{\mathcal{W}}(\mathbf{w}_{\mathrm{GD}}^{\ell +\frac{1}{2}})$ .
We first prove part (a). We begin by deriving a relation between  $\left\| \widehat{\mathbf{w}}^{\ell +1} - \widehat{\mathbf{w}}^{\ell}\right\| _2^2$  and  $\left\| \eta G_{\mathcal{W},\eta}^{f}(\widehat{\mathbf{w}}^{\ell})\right\| _2^2$
Let  $\widetilde{\mathbf{w}}^{\ell +\frac{1}{2}}\coloneqq \widehat{\mathbf{w}}^{\ell} - \eta \nabla f(\widehat{\mathbf{w}}^{\ell})$  and  $\widetilde{\mathbf{w}}^{\ell +1}\coloneqq \mathrm{Proj}_{\mathcal{W}}(\widetilde{\mathbf{w}}^{\ell +\frac{1}{2}})$  denote the exact projected gradient iterate starting from  $\widehat{\mathbf{w}}^\ell$ . We have
$$
\begin{array}{l} \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} \overset {(i)} {\geq} \frac {1}{2} \left\| \widetilde {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} - \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widetilde {\mathbf {w}} ^ {\ell + 1} \right\| _ {2} ^ {2} \overset {(i i)} {\geq} \frac {1}{2} \left\| \widetilde {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} - \left\| \widehat {\mathbf {w}} ^ {\ell + \frac {1}{2}} - \widetilde {\mathbf {w}} ^ {\ell + \frac {1}{2}} \right\| _ {2} ^ {2} \\ \stackrel {(i i i)} {=} \frac {\eta^ {2}}{2} \left\| \mathbf {G} _ {\mathcal {W}, \eta} ^ {f} \left(\widehat {\mathbf {w}} ^ {\ell}\right) \right\| _ {2} ^ {2} - \| \eta \varepsilon \| _ {2} ^ {2} \geq \frac {\eta^ {2}}{2} \left\| \mathbf {G} _ {\mathcal {W}, \eta} ^ {f} \left(\widehat {\mathbf {w}} ^ {\ell}\right) \right\| _ {2} ^ {2} - \eta^ {2} \varepsilon^ {2}. \tag {19} \\ \end{array}
$$
Above, (i) uses the inequality  $\| a - b \|_2^2 \geq \frac{1}{2} \| a \|_2^2 - \| b \|_2^2$ ; (ii) uses the fact that projection to a convex set is a non-expansion; (iii) uses the definition of the gradient mapping.
By the  $L_{f}$ -smoothness of  $f$  within  $\mathcal{W}$ , we have
$$
f (\widehat {\mathbf {w}} ^ {\ell + 1}) - f (\widehat {\mathbf {w}} ^ {\ell}) \leq \left\langle \nabla f (\widehat {\mathbf {w}} ^ {\ell}), \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\rangle + \frac {L _ {f}}{2} \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2}
$$
$$
\begin{array}{l} = \left\langle \frac {\widehat {\mathbf {w}} ^ {\ell} - \widehat {\mathbf {w}} ^ {\ell + \frac {1}{2}}}{\eta} - \varepsilon^ {\ell}, \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\rangle + \frac {L _ {f}}{2} \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} \\ \stackrel {(i)} {\leq} \left\langle \frac {\widehat {\mathbf {w}} ^ {\ell} - \widehat {\mathbf {w}} ^ {\ell + 1}}{\eta}, \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\rangle + \left\langle \boldsymbol {\varepsilon} ^ {\ell}, \widehat {\mathbf {w}} ^ {\ell} - \widehat {\mathbf {w}} ^ {\ell + 1} \right\rangle + \frac {L _ {f}}{2} \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} \\ = \left(- \frac {1}{\eta} + \frac {L _ {f}}{2}\right) \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} + \frac {1}{4 \eta} \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} + \eta \left\| \boldsymbol {\varepsilon} ^ {\ell} \right\| _ {2} ^ {2} \\ \stackrel {(i i)} {\leq} - \frac {1}{4 \eta} \left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \widehat {\mathbf {w}} ^ {\ell} \right\| _ {2} ^ {2} + \eta \left\| \varepsilon^ {\ell} \right\| _ {2} ^ {2} \\ \stackrel {(i i i)} {\leq} - \frac {1}{4 \eta} \left(\frac {\eta^ {2}}{2} \left\| \mathsf {G} _ {\mathcal {W}, \eta} ^ {f} (\widehat {\mathbf {w}} ^ {\ell}) \right\| _ {2} ^ {2} - \left\| \eta \pmb {\varepsilon} ^ {\ell} \right\| _ {2} ^ {2}\right) + \eta \left\| \pmb {\varepsilon} ^ {\ell} \right\| _ {2} ^ {2} \\ \leq - \frac {\eta}{8} \left\| \mathsf {G} _ {\mathcal {W}, \eta} ^ {f} (\widehat {\mathbf {w}} ^ {\ell}) \right\| _ {2} ^ {2} + \frac {5 \eta}{4} \varepsilon^ {2}. \\ \end{array}
$$
Above, (i) uses the property  $\left\langle \widehat{\mathbf{w}}^{\ell +1} - \widehat{\mathbf{w}}^{\ell +\frac{1}{2}},\widehat{\mathbf{w}}^{\ell +1} - \widehat{\mathbf{w}}^{\ell}\right\rangle \leq 0$  of the projection  $\widehat{\mathbf{w}}^{\ell +1} = \mathrm{Proj}_{\mathcal{W}}(\widehat{\mathbf{w}}^{\ell +\frac{1}{2}})$  (using  $\widehat{\mathbf{w}}^{\ell}\in \mathcal{W}$ ); (ii) uses  $L_{f} / 2\leq 1 / (2\eta)$  by our choice of  $\eta \leq 1 / L_f$ ; (iii) uses (19).
Rearranging and summing the above over  $\ell = 0,\dots ,L - 1$  , we obtain
$$
\frac {\eta}{8} \sum_ {\ell = 0} ^ {L - 1} \left\| \mathsf {G} _ {\mathcal {W}, \eta} ^ {f} (\widehat {\mathbf {w}} ^ {\ell}) \right\| _ {2} ^ {2} \leq f (\mathbf {w} ^ {0}) - f (\widehat {\mathbf {w}} ^ {L}) + \frac {5 \eta L}{4} \varepsilon^ {2}.
$$
Dividing both sides by  $\eta L / 8$  yields part (a).
Next, we prove part (b). Let  $C\coloneqq 1 + \eta L_{f}$  . We prove by induction that
$$
\left\| \widehat {\mathbf {w}} ^ {\ell} - \mathbf {w} _ {\mathrm {G D}} ^ {\ell} \right\| _ {2} \leq \frac {C ^ {\ell} - 1}{C - 1} \cdot \eta \varepsilon \tag {20}
$$
for all  $\ell \geq 0$ . The base case of  $\ell = 0$  follows by definition that  $\widehat{\mathbf{w}}^0 = \mathbf{w}_{\mathrm{GD}}^0 = \mathbf{w}^0$ . Suppose the result holds for  $\ell$ . Then for  $\ell + 1$ , we have
$$
\left\| \widehat {\mathbf {w}} ^ {\ell + 1} - \mathbf {w} _ {\mathrm {G D}} ^ {\ell + 1} \right\| _ {2} \stackrel {(i)} {\leq} \left\| \widehat {\mathbf {w}} ^ {\ell + \frac {1}{2}} - \mathbf {w} _ {\mathrm {G D}} ^ {\ell + \frac {1}{2}} \right\| _ {2} = \left\| \widehat {\mathbf {w}} ^ {\ell} - \eta (\nabla f (\widehat {\mathbf {w}} ^ {\ell}) - \varepsilon^ {\ell}) - (\mathbf {w} _ {\mathrm {G D}} ^ {\ell} - \eta \nabla f (\mathbf {w} _ {\mathrm {G D}} ^ {\ell})) \right\| _ {2}
$$
$$
\stackrel {(i i)} {\leq} C \left\| \widehat {\mathbf {w}} ^ {\ell} - \mathbf {w} _ {\mathrm {G D}} ^ {\ell} \right\| _ {2} + \eta \varepsilon^ {(i i i)} \stackrel {(i i)} {\leq} C \cdot \frac {C ^ {\ell} - 1}{C - 1} \cdot \eta \varepsilon + \eta \varepsilon = \frac {C ^ {\ell + 1} - 1}{C - 1} \cdot \eta \varepsilon .
$$
Above, (i) uses again the non-expansiveness of the convex projection  $\mathrm{Proj}_{\mathcal{W}}$ ; (ii) uses the fact that the operator  $\mathbf{w} \mapsto \mathbf{w} - \eta \nabla f(\mathbf{w})$  is  $(1 + \eta L_{f}) = C$ -Lipschitz; and (iii) uses the inductive hypothesis. This proves the case for  $\ell + 1$  and thus finishes the induction. We can further relax (20) into
$$
\left\| \widehat {\mathbf {w}} ^ {\ell} - \mathbf {w} _ {\mathrm {G D}} ^ {\ell} \right\| _ {2} \leq \frac {C ^ {\ell}}{1 + \eta L _ {f} - 1} \cdot \eta \varepsilon = L _ {f} ^ {- 1} (1 + \eta L _ {f}) ^ {\ell} \varepsilon .
$$
This proves part (b).
F Proofs for Section 3.1
Fix  $\lambda \geq 0, 0 \leq \alpha \leq \beta$  with  $\kappa := \frac{\beta + \lambda}{\alpha + \lambda}$ , and  $B_w > 0$ , and consider any in-context data  $\mathcal{D}$  such that the precondition of Theorem 4 holds. Let
$$
L _ {\mathrm {r i d g e}} (\mathbf {w}) := \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(\langle \mathbf {w}, \mathbf {x} _ {i} \rangle - y _ {i}\right) ^ {2} + \frac {\lambda}{2} \| \mathbf {w} \| _ {2} ^ {2}
$$
denote the ridge regression loss in (ICRidge), so that  $\mathbf{w}_{\mathrm{ridge}}^{\lambda} = \arg \min_{\mathbf{w}\in \mathbb{R}^{d}}L_{\mathrm{ridge}}(\mathbf{w})$ . It is a standard result that  $\nabla^2 L_{\mathrm{ridge}}(\mathbf{w}) = \mathbf{X}^\top \mathbf{X} / N + \lambda \mathbf{I}_d$ , so that  $L_{\mathrm{ridge}}$  is  $(\alpha +\lambda)$ -strongly convex and  $(\beta +\lambda)$ -smooth over  $\mathbb{R}^d$ .
Consider the gradient descent algorithm on the ridge loss
$$
\mathbf {w} _ {\mathrm {G D}} ^ {t + 1} = \mathbf {w} _ {\mathrm {G D}} ^ {t} - \eta \nabla L _ {\text {r i d g e}} \left(\mathbf {w} _ {\mathrm {G D}} ^ {t}\right)
$$
with initialization, learning rate, and number of steps
$$
\mathbf {w} _ {\mathrm {G D}} ^ {0} := \mathbf {0} _ {d}, \quad \eta := \frac {1}{\beta + \lambda}, \quad T := \left\lceil 2 \kappa \log \left(\frac {B _ {x} B _ {w}}{2 \varepsilon}\right)\right\rceil .
$$
By standard convergence results for strongly convex and smooth functions (Proposition B.2), we have for all  $t \geq 1$  that
$$
\left\| \mathbf {w} _ {\mathrm {G D}} ^ {t} - \mathbf {w} _ {\mathrm {r i d g e}} ^ {\lambda} \right\| _ {2} ^ {2} \leq \exp \left(- \frac {t}{\kappa}\right) \left\| \mathbf {w} _ {\mathrm {G D}} ^ {0} - \mathbf {w} _ {\mathrm {r i d g e}} ^ {\lambda} \right\| _ {2} ^ {2} = \exp \left(- \frac {t}{\kappa}\right) \left\| \mathbf {w} _ {\mathrm {r i d g e}} ^ {\lambda} \right\| _ {2} ^ {2}.
$$
Further, we have
$$
\left\| \mathbf {w} _ {\mathrm {G D}} ^ {T} - \mathbf {w} _ {\text {r i d g e}} ^ {\lambda} \right\| _ {2} \leq \exp \left(- \frac {T}{2 \kappa}\right) \left\| \mathbf {w} _ {\text {r i d g e}} ^ {\lambda} \right\| _ {2} \leq \frac {2 \varepsilon}{B _ {x} B _ {w}} \cdot \frac {B _ {w}}{2} \leq \frac {\varepsilon}{B _ {x}}. \tag {21}
$$
It remains to construct a transformer to approximate  $\mathbf{w}_{\mathrm{GD}}^T$ . Notice that the problem (ICRidge) corresponds to an  $\ell_2$ -regularized ERM with the square loss  $\ell(s, t) := \frac{1}{2}(s - t)^2$ , whose partial derivative  $\partial_s \ell(s, t) = s - t$  is exactly a sum of two relus:
$$
\partial_ {s} \ell (s, t) = 2 \sigma ((s - t) / 2) - 2 \sigma (- (s - t) / 2).
$$
In particular, this shows that  $\partial_s\ell (s,t)$  is  $(0,R,2,4)$ -approximable for any  $R > 0$ , in particular for  $R = \max \{B_xB_w,B_y,1\}$ .
Therefore, we can apply Corollary E.1 with the square loss  $\ell$ , learning rate  $\eta$ , regularization strength  $\lambda$  and accuracy parameter  $\varepsilon = 0$  to obtain that there exists an attention-only transformer  $\mathrm{TF}_{\theta}^{0}$  with  $(T + 1) := L$  layers such that the final output  $\mathbf{h}_{N + 1}^{(L)} = [\mathbf{x}_{N + 1}; \widehat{y}_{N + 1}; *]$  with
$$
\left| \widehat {y} _ {N + 1} - \left\langle \mathbf {w} _ {\mathrm {G D}} ^ {T}, \mathbf {x} _ {N + 1} \right\rangle \right| = 0, \tag {22}
$$
and number of heads  $M^{(\ell)} = 3$  for all  $\ell \in [L - 1]$  (can be taken as 2 in the unregularized case  $\lambda = 0$  directly by Theorem D.1), and  $M^{(L)} = 2$ . Further,  $\pmb{\theta}$  admits norm bound  $\| \pmb{\theta} \| \leq 2 + R + \frac{8 + \lambda}{\beta + \lambda} \leq 3R + 8(\beta + \lambda)^{-1} + 1 \leq 4R + 8(\beta + \lambda)^{-1}$ .
$$
\left| \widehat {y} _ {N + 1} - \left\langle \mathbf {w} _ {\text {r i d g e}} ^ {\lambda}, \mathbf {x} _ {N + 1} \right\rangle \right| = \left| \left\langle \mathbf {w} _ {\text {G D}} ^ {T} - \mathbf {w} _ {\text {r i d g e}} ^ {\lambda}, \mathbf {x} _ {N + 1} \right\rangle \right| \leq (\varepsilon / B _ {x}) \cdot B _ {x} = \varepsilon .
$$
Further, we have  $\mathrm{read}_{\mathbf{w}}(\mathbf{h}_i^T) = \mathbf{w}_{\mathrm{GD}}^T$  for all  $i \in [N + 1]$ , where  $\mathrm{read}_{\mathbf{w}}(\mathbf{h}) \coloneqq \mathbf{h}_{(d + 2):(2d + 1)}$  (cf. Corollary E.1), so that  $\| \mathrm{read}_{\mathbf{w}}(\mathbf{h}_i^T) - \mathbf{w}_{\mathrm{ridge}}^\lambda \|_2 \leq \varepsilon / B_x$  as shown above. This finishes the proof.
F.2 Statistical analysis of in-context least squares
Consider the standard least-squares algorithm  $\mathcal{A}_{\mathrm{LS}}$  and least-squares estimator  $\widehat{\mathbf{w}}_{\mathrm{LS}} \in \mathbb{R}^d$  defined as
$$
\mathcal {A} _ {\mathrm {L S}} (\mathcal {D}) \left(\mathbf {x} _ {N + 1}\right) := \left\langle \widehat {\mathbf {w}} _ {\mathrm {L S}}, \mathbf {x} _ {N + 1} \right\rangle , \quad \widehat {\mathbf {w}} _ {\mathrm {L S}} = \left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \mathbf {X} ^ {\top} \mathbf {y} \in \mathbb {R} ^ {d}. \tag {ICLS}
$$
For any distribution  $\mathsf{P}$  over  $(\mathbf{x},y)\in \mathbb{R}^d\times \mathbb{R}$  and any estimator  $\mathbf{w}\in \mathbb{R}^d$  , let
$$
L _ {\mathrm {P}} (\mathbf {w}) := \mathbb {E} _ {(\mathbf {x} ^ {\prime}, y) \sim \mathrm {P}} \left[ \frac {1}{2} \left(\langle \mathbf {w}, \mathbf {x} ^ {\prime} \rangle - y ^ {\prime}\right) ^ {2} \right]
$$
denote the expected risk of  $\mathbf{w}$  over a new test example  $(\mathbf{x}', y') \sim \mathrm{P}$ .
Assumption A (Well-posedness for learning linear predictors). We say a distribution  $\mathsf{P}$  on  $\mathbb{R}^d\times \mathbb{R}$  is well-posed for learning linear predictors, if  $(\mathbf{x},y)\sim \mathsf{P}$  satisfies
(1)  $\| \mathbf{x} \|_2 \leq B_x$  and  $|y| \leq B_y$  almost surely;
(2) The covariance  $\pmb{\Sigma}_{\mathbb{P}} \coloneqq \mathbb{E}_{\mathbb{P}}[\mathbf{x}\mathbf{x}^{\top}]$  satisfies  $\lambda_{\min}\mathbf{I}_d \preceq \pmb{\Sigma}_{\mathbb{P}} \preceq \lambda_{\max}\mathbf{I}_d$ , with  $0 < \lambda_{\min} \leq \lambda_{\max}$ , and  $\kappa \coloneqq \lambda_{\max} / \lambda_{\min}$ .
(3) The whitened vector  $\pmb{\Sigma}_{\mathbb{P}}^{-1 / 2}\mathbf{x}$  is  $K^2$  -sub-Gaussian for some  $K\geq 1$
(4) The best linear predictor  $\mathbf{w}_{\mathsf{P}}^{\star} \coloneqq \mathbb{E}_{\mathsf{P}}[\mathbf{x}\mathbf{x}^{\top}]^{-1}\mathbb{E}_{\mathsf{P}}[\mathbf{x}y]$  satisfies  $\| \mathbf{w}_{\mathsf{P}}^{\star}\|_{2} \leq B_{w}^{\star}$ .
(5) We have  $\mathbb{E}[(y - \langle \mathbf{x},\mathbf{w}_{\mathsf{P}}^{\star}\rangle)^2 |\mathbf{x}]\leq \sigma^2$  with probability one (over  $\mathbf{x}$ ).
Further, we say  $\mathsf{P}$  is well-posed with canonical parameters if
$$
B _ {x} = \Theta (\sqrt {d}), \quad B _ {y} = \Theta (1), \quad B _ {w} ^ {\star} = \Theta (1), \quad \sigma \leq \mathcal {O} (1), \quad \lambda_ {\max } = \Theta (1), \quad K = \Theta (1), \tag {23}
$$
where  $\Theta (\cdot)$  and  $\mathcal{O}(\cdot)$  only hides absolute constants.
The following result bounds the excess risk of least squares under Assumption A with a clipping operation on the predictor; the clipping allows the result to only depend on the second moment of the noise (cf. Assumption A(5)) instead of e.g. its sub-Gaussianity, and also makes the result convenient to be directly translated to a result for transformers.
Proposition F.1 (Guarantees for in-context least squares). Suppose distribution  $\mathsf{P}$  satisfies Assumption A. Then as long as  $N \geq \mathcal{O}(dK^4\log (1 / \delta))$ , we have the following:
(a) The (clipped) least squares predictor achieves small expected excess risk (fast rate) over the best linear predictor: For any clipping radius  $R \geq B_y$ ,
$$
\mathbb {E} _ {\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1} \sim \mathsf {P}} \left[ \frac {1}{2} \left(\operatorname {c l i p} _ {R} \left(\left\langle \widehat {\mathbf {w}} _ {\mathrm {L S}}, \mathbf {x} _ {N + 1} \right\rangle\right) - y _ {N + 1}\right) ^ {2} \right] \leq \underbrace {\inf  _ {\mathbf {w} \in \mathbb {R} ^ {d}} L _ {\mathsf {P}} (\mathbf {w})} _ {L _ {\mathsf {P}} \left(\mathbf {w} _ {\mathsf {P}} ^ {\star}\right)} + \mathcal {O} \left(R ^ {2} \delta + \frac {d \sigma^ {2}}{N}\right). \tag {24}
$$
(b) We have  $\mathsf{P}(E_{\mathrm{cov}}\cap E_w)\geq 1 - \delta /10$  , where
$$
E _ {\mathrm {c o v}} = E _ {\mathrm {c o v}} (\mathcal {D}) := \left\{\frac {1}{2} \mathbf {I} _ {d} \preceq \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2} \widehat {\boldsymbol {\Sigma}} \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2} \preceq 2 \mathbf {I} _ {d} \right\}, \tag {25}
$$
$$
E _ {w} = E _ {w} (\mathcal {D}) := \left\{\| \widehat {\mathbf {w}} _ {\mathrm {L S}} \| _ {2} \leq B _ {w} ^ {\star} + \sqrt {\frac {8 0 d \sigma^ {2}}{\delta N \lambda_ {\operatorname* {m i n}}}} \right\}. \tag {26}
$$
Proof. We first show  $\mathsf{P}(E_{\mathrm{cov}}) \geq 1 - \delta / 20$ . Let  $\widehat{\pmb{\Sigma}} := \frac{1}{N} \sum_{i=1}^{N} \mathbf{x}_i \mathbf{x}_i^\top$ , and let the whitened covariance and noise variables be denoted as
$$
\widetilde {\mathbf {x}} _ {i} = \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2} \mathbf {x} _ {i}, \quad \widetilde {\boldsymbol {\Sigma}} := \frac {1}{N} \sum_ {i = 1} ^ {N} \widetilde {\mathbf {x}} _ {i} \widetilde {\mathbf {x}} _ {i} ^ {\top} = \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2} \widehat {\boldsymbol {\Sigma}} \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2}.
$$
Also let  $z_{i}\coloneqq y_{i} - \langle \mathbf{x}_{i},\mathbf{w}_{\mathsf{P}}^{\star}\rangle$  denote the "noise" variables. Note that
$$
E _ {\mathrm {c o v}} = \left\{\frac {1}{2} \mathbf {I} _ {d} \preceq \widetilde {\boldsymbol {\Sigma}} \preceq 2 \mathbf {I} _ {d} \right\}
$$
is exactly a covariance concentration of the whitened vectors  $\{\widetilde{\mathbf{x}}_i\}_{i\in [N]}$ . Recall that  $\mathbb{E}[\widetilde{\mathbf{x}}_i\widetilde{\mathbf{x}}_i^\top ] = \mathbf{I}_d$  and  $\widetilde{\mathbf{x}}_i$  are  $K^2$ -sub-Gaussian by assumption. Therefore, we can apply [85, Theorem 4.6.1], we have with probability at least  $1 - \delta /10$  that
$$
\left\| \widetilde {\boldsymbol {\Sigma}} - \mathbf {I} _ {d} \right\| _ {\mathrm {o p}} \leq \mathcal {O} \left(K ^ {2} \max  \left\{\sqrt {\frac {d + \log (1 / \delta)}{N}}, \frac {d + \log (1 / \delta)}{N} \right\}\right).
$$
Setting  $N \geq \mathcal{O}(K^4 (d + \log (1 / \delta)))$  ensures that the right-hand side above is at most  $1 / 2$ , on which event we have
$$
\frac {1}{2} \mathbf {I} _ {d} \preceq \widetilde {\boldsymbol {\Sigma}} \preceq \frac {3}{2} \mathbf {I} _ {d} \preceq 2 \mathbf {I} _ {d}, \tag {27}
$$
i.e.  $E_{\mathrm{cov}}$  holds. This shows that  $\mathsf{P}(E_{\mathrm{cov}}^c)\leq \delta /10$
Next, we show (24). Using  $E_{\mathrm{cov}}$ , we decompose the risk as
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{2} \left(\operatorname {c l i p} _ {R} \left(\langle \widehat {\mathbf {w}} _ {\mathrm {L S}}, \mathbf {x} _ {N + 1} \rangle\right) - y _ {N + 1}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \frac {1}{2} (\mathsf {c l i p} _ {R} (\langle \widehat {\mathbf {w}} _ {\mathrm {L S}}, \mathbf {x} _ {N + 1} \rangle) - y _ {N + 1}) ^ {2} 1 \{E _ {\mathrm {c o v}} \} \right] + \mathbb {E} \left[ \frac {1}{2} (\mathsf {c l i p} _ {R} (\langle \widehat {\mathbf {w}} _ {\mathrm {L S}}, \mathbf {x} _ {N + 1} \rangle) - y _ {N + 1}) ^ {2} 1 \{E _ {\mathrm {c o v}} ^ {c} \} \right] \\ \stackrel {(i)} {\leq} \mathbb {E} \left[ \frac {1}{2} \left(\langle \widehat {\mathbf {w}} _ {\mathrm {L S}}, \mathbf {x} _ {N + 1} \rangle - y _ {N + 1}\right) ^ {2} 1 \{E _ {\text {c o v}} \} \right] + 2 R ^ {2} \cdot (\delta / 2 0) \\ \stackrel {(i i)} {=} \mathbb {E} _ {\mathcal {D}, \mathbf {x} _ {N + 1}} \left[ \frac {1}{2} \big (\langle \widehat {\mathbf {w}} _ {\mathrm {L S}} - \mathbf {w} _ {\mathsf {P}} ^ {\star}, \mathbf {x} _ {N + 1} \rangle \big) ^ {2} 1 \{E _ {\mathrm {c o v}} \} \right] + \mathbb {E} _ {\mathbf {x} _ {N + 1}, y _ {N + 1}} \left[ \frac {1}{2} \big (\langle \mathbf {w} _ {\mathsf {P}} ^ {\star}, \mathbf {x} _ {N + 1} \rangle - y _ {N + 1} \big) ^ {2} 1 \{E _ {\mathrm {c o v}} \} \right] + \mathcal {O} (R ^ {2} \delta) \\ \leq \mathbb {E} _ {\mathcal {D}} \left[ \frac {1}{2} \| \widehat {\mathbf {w}} _ {\mathrm {L S}} - \mathbf {w} _ {\mathsf {P}} ^ {\star} \| _ {\boldsymbol {\Sigma} _ {\mathsf {P}}} ^ {2} 1 \{E _ {\text {c o v}} \} \right] + \underbrace {\mathbb {E} _ {\mathbf {x} _ {N + 1} , y _ {N + 1}} \left[ \frac {1}{2} \left(\left\langle \mathbf {w} _ {\mathsf {P}} ^ {\star}, \mathbf {x} _ {N + 1} \right\rangle - y _ {N + 1}\right) ^ {2} \right]} _ {L _ {\mathsf {P}} \left(\mathbf {w} _ {\mathsf {P}} ^ {\star}\right)} + \mathcal {O} \left(R ^ {2} \delta\right). \tag {28} \\ \end{array}
$$
Above, (i) follows by assumption that  $|y_{N + 1}| \leq B_y \leq R$  almost surely, so that removing the clipping can only potentially increase the distance in the first term, and the square loss is upper bounded by  $\frac{1}{2} \cdot (2R)^2$  almost surely in the second term; (ii) follows by the fact that  $\mathbb{E}_{\mathbf{x}_{N + 1}, y_{N + 1}}[\langle \widehat{\mathbf{w}}_{LS} - \mathbf{w}_P^*, \mathbf{x}_{N + 1} \rangle (\langle \mathbf{w}_P^*, \mathbf{x}_{N + 1} \rangle - y_{N + 1})] = 0$  by the definition of  $\mathbf{w}_P^*$ , as well as the fact that  $1\{E_{\mathrm{cov}}\}$  is independent of  $(\mathbf{x}_{N + 1}, y_{N + 1})$ .
It thus remains to bound  $\mathbb{E}_{\mathcal{D}}\left[\frac{1}{2}\|\widehat{\mathbf{w}}_{\mathrm{LS}} - \mathbf{w}_{\mathsf{P}}^{\star}\|_{\Sigma_{\mathsf{P}}}^{2}1\{E_{\mathrm{cov}}\}\right]$ . Note that on the event  $E_{\mathrm{cov}}$ , we have
$$
\boldsymbol {\Sigma} _ {\mathsf {P}} ^ {1 / 2} \widehat {\boldsymbol {\Sigma}} ^ {- 1} \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {1 / 2} = \left(\boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2} \widehat {\boldsymbol {\Sigma}} \boldsymbol {\Sigma} _ {\mathsf {P}} ^ {- 1 / 2}\right) ^ {- 1} \preceq 2 \mathbf {I} _ {d}.
$$