SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L = \lceil 2 \kappa \log (L _ {g} B _ {w} B _ {x} / \varepsilon) \rceil + 1, \quad \max  _ {\ell \in [ L ]} M ^ {(\ell)} \leq \widetilde {\mathcal {O}} \left(C _ {g} ^ {2} \kappa_ {w} ^ {2} \varepsilon^ {- 2}\right), \quad \| \boldsymbol {\theta} \| \leq \mathcal {O} \left(R + \beta^ {- 1} C _ {g}\right),
$$
(where  $L_g \coloneqq \sup_t |g'(t)|$ ,  $R \coloneqq \max \{B_x B_w, B_y, 1\}$ , and  $C_g > 0$  is a constant that depends only on  $R$  and the  $C^2$ -smoothness of  $g$  within  $[-R, R]$ ), such that the following holds. On any input data  $(\mathcal{D}, \mathbf{x}_{N+1})$  such that
$$
\alpha \leq \lambda_ {\min } \left(\nabla^ {2} \widehat {L} _ {N} (\mathbf {w})\right) \leq \lambda_ {\max } \left(\nabla^ {2} \widehat {L} _ {N} (\mathbf {w})\right) \leq \beta \text {f o r a l l} \mathbf {w} \in \mathrm {B} _ {2} \left(B _ {w}\right), \quad \| \mathbf {w} _ {\mathrm {G L M}} \| _ {2} \leq B _ {w} / 2, \tag {35}
$$
$\mathrm{TF}_{\pmb{\theta}}^{0}(\mathbf{H}^{(0)})$  approximately implements (ICGLM): We have  $\mathbf{h}_{N + 1}^{(L + 1)}\coloneqq [\mathbf{x}_{N + 1};\widehat{y}_{N + 1};\widehat{\mathbf{w}};1;1]$ , where
$$
\left| \hat {y} _ {N + 1} - g \left(\left\langle \mathbf {x} _ {N + 1}, \mathbf {w} _ {\mathrm {G L M}} \right\rangle\right) \right| \leq \varepsilon .
$$
In Theorem G.1, the number of heads scales as  $\widetilde{\mathcal{O}}(1/\varepsilon^2)$  as opposed to  $\Theta(1)$  as in ridge regression (Theorem 4), due to the fact that the gradient of the loss is in general a smooth function that can be only approximately expressed as a sum-of-relus (cf. Definition D.1 & Lemma B.5) rather than exactly expressed as in the case for the square loss.
In-context prediction power We next show that (proof in Appendix G.2) the transformer constructed in Theorem G.1 achieves desirable statistical power if the in-context data distribution satisfies standard statistical assumptions for learning GLMs. Let  $L_{\mathsf{P}}(\mathbf{w}) \coloneqq \mathbb{E}_{(\mathbf{x},y)\sim \mathsf{P}}[\ell (\langle \mathbf{w},\mathbf{x}\rangle ,y)]$  denote the corresponding population risk for any distribution  $\mathsf{P}$  of  $(\mathbf{x},y)$ . When  $\mathsf{P}$  is realizable by a generalized linear model of link function  $g$  and parameter  $\beta$  in the sense that  $\mathbb{E}_{\mathsf{P}}[y|\mathbf{x}] = g(\langle \beta ,\mathbf{x}\rangle)$ , it is a standard result that  $\beta$  is indeed a minimizer of  $L_{\mathsf{P}}$  [42] (see also [6, Appendix A.3]).
Theorem G.2 (Statistical guarantee for generalized linear models). For any fixed set of parameters defined in Assumption  $B$ , there exists a transformer  $\theta$  with  $L \leq \mathcal{O}(\log(N))$  layers and  $\max_{\ell \in [L]} M^{(\ell)} \leq \widetilde{\mathcal{O}}(d^3 N)$ , such that for any distribution  $\mathsf{P}$  satisfying Assumption  $B$  with those parameters, as long as  $N \geq \mathcal{O}(d)$ , that outputs  $\widehat{y}_{N+1} = \widetilde{\mathrm{read}}_y(\mathrm{TF}_\theta(\mathbf{H}))$  and  $\widehat{\mathbf{w}} = \widetilde{\mathrm{read}}_w(\mathrm{TF}_\theta(\mathbf{H})) \in \mathbb{R}^d$  (for another read-out function  $\widetilde{\mathrm{read}}_w$ ) satisfying the following.
(a)  $\widehat{\mathbf{w}}$  achieves small excess risk under the population loss, i.e. for the linear prediction  $\widehat{y}_{N + 1}^{\mathrm{lin}}\coloneqq \langle \mathbf{x}_{N + 1},\widehat{\mathbf{w}}\rangle$
$$
\mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right) \sim \mathcal {P}} \left[ \ell \left(\hat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] - \min  _ {\boldsymbol {\beta}} L _ {\mathcal {P}} (\boldsymbol {\beta}) \leq \mathcal {O} (d / N). \tag {36}
$$
(b) (Realizable setting) If there exists a  $\beta \in \mathbb{R}^d$  such that under  $\mathsf{P}$ ,  $\mathbb{E}[y|\mathbf{x}] = g(\langle \beta, \mathbf{x} \rangle)$  almost surely, then
$$
\mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right) \sim \mathbb {P}} \left[ \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \right] \leq \mathbb {E} _ {\left(\mathbf {x} _ {N + 1}, y _ {N + 1}\right) \sim \mathbb {P}} \left[ \left(g \left(\langle \boldsymbol {\beta}, \mathbf {x} _ {N + 1} \rangle\right) - y _ {N + 1}\right) ^ {2} \right] + \mathcal {O} (d / N), \tag {37}
$$
or equivalently,  $\mathbb{E}[(\widehat{y}_{N + 1} - \mathbb{E}[y_{N + 1}|\mathbf{x}_{N + 1}])^2 ]\leq \mathcal{O}(d / N)$
Above,  $\mathcal{O}(\cdot)$  hides constants that depend polynomially on the parameters in Assumption B. Similar as in Corollary 5, the  $\mathcal{O}(d / N)$  excess risk obtained here matches the optimal (fast) rate for typical learning problems with  $d$  parameters and  $N$  samples [87].
Assumption B (Well-posedness for learning GLMs). We assume that there is some  $B_{\mu} > 0$  such that for any  $t \in [-B_{\mu}, B_{\mu}]$ ,  $g'(t) \geq \mu_g > 0$ .
We also assume that for each  $i \in [N + 1]$ ,  $(\mathbf{x}_i, y_i)$  is independently sampled from  $\mathsf{P}$  such that the following holds.
(a) Under the law  $(\mathbf{x},y)\sim \mathsf{P}$ , We have  $\mathbf{x}\sim \mathrm{SG}(K_x)$ ,  $y\sim \mathrm{SG}(K_y)$  and  $g(\langle \mathbf{w},\mathbf{x}\rangle)\sim \mathrm{SG}(K_y)\forall \mathbf{w}\in \mathsf{B}_2(B_w)$ .
(b) For some  $\mu_x > 0$ , it holds that
$$
\mathbb {E} \left[ 1 \left\{\left| \mathbf {x} ^ {\top} \mathbf {w} \right| \leq B _ {\mu} / 2 \right\} \mathbf {x x} ^ {\top} \right] \succeq \mu_ {x} \mathbf {I} _ {d} \quad \forall \mathbf {w} \in \mathsf {B} _ {2} (B _ {w}).
$$
(c) For  $\beta^{\star} = \arg \min L_{\mathsf{P}}$  , it holds  $\| \beta^{\star}\|_{2}\leq B_{w} / 4.$
Applying Theorem G.2 to logistic regression, we have the following result as a direct corollary. Below, the Gaussian input assumption is for convenience only and can be generalized to e.g. sub-Gaussian input.
Corollary G.1 (In-context logistic regression). Consider any in-context data distribution  $\mathrm{P}$  satisfying
$$
\mathbf{x}\sim \mathsf{N}(0,\mathbf{I}_{d}),\qquad y\in \{0,1\} ,\qquad \operatorname *{arg  min}_{\boldsymbol {\beta}\in \mathbb{R}^{d}}L_{\mathsf{P}}(\boldsymbol {\beta})\in \mathsf{B}_{2}(B_{w}^{\star}).
$$
For the link function  $g = \sigma_{\log}$  and  $B_w^\star = \mathcal{O}(1)$ , we can choose  $B_w, B_\mu, \mu_g, L_g, \mu_x, K_x, K_y = \Theta(1)$  so that Assumption  $B$  holds. In that case, when  $N \geq \mathcal{O}(d)$ , there exists a transformer  $\theta$  with  $L = \mathcal{O}(\log(N))$  layers, such that for any  $\mathsf{P}$  considered above,
(a) The estimation  $\widehat{\mathbf{w}} = \widetilde{\mathrm{read}}_{\mathbf{w}}(\mathrm{TF}_{\boldsymbol{\theta}}(\mathbf{H}))$  outputted by  $\pmb{\theta}$  achieves excess risk bound (36).
(b) (Realizable setting) Consider the logistic in-context data distribution
$$
\mathsf {P} _ {\beta} ^ {\log}: \qquad \mathbf {x} \sim \mathsf {N} (0, \mathbf {I} _ {d}), \qquad y | \mathbf {x} \sim \text {B e r n o u l l i} (g (\langle \boldsymbol {\beta}, \mathbf {x} \rangle)).
$$
Then, for any distribution  $\mathsf{P} = \mathsf{P}_{\beta}^{\log}$  with  $\| \beta \|_2 \leq B_w^\star$ , the prediction  $\widehat{y}_{N+1} = \widetilde{\mathrm{read}}_{\mathsf{y}}(\mathrm{TF}_{\boldsymbol{\theta}}(\mathbf{H}))$  of  $\boldsymbol{\theta}$  additionally achieves the square loss excess risk (37).
G.1 Proof of Theorem G.1
Let us fix parameters  $\varepsilon_g > 0$  and  $T > 0$  (that we specify later in proof).
Define  $R = \max \{B_xB_w,B_y,1\}$  and
$$
C _ {g} := \max  _ {i = 0, 1, 2} \left(R ^ {i} \max  _ {s \in [ - B, B ]} \left| g ^ {(i)} (s) \right|\right).
$$
By Proposition B.1,  $g$  is  $(\varepsilon_g, M, R, C)$  with
$$
C \leq \mathcal {O} \left(C _ {g}\right), \qquad M \leq \mathcal {O} \left(C _ {g} ^ {2} \varepsilon_ {g} ^ {- 2} \log \left(1 + C _ {g} \varepsilon_ {g} ^ {- 1}\right)\right).
$$
Therefore, we can invoke Theorem D.1 to obtain that, as long as  $2T\varepsilon_{g} \leq B_{w}$ , there exists a  $T$ -layer attention-only transformer  $\pmb{\theta}^{(1:T)}$  with  $M$  heads per layer, such that for any input  $\mathbf{H}$  of format (3) and satisfies (35), its last layer outputs  $\mathbf{h}_i^{(T)} = [\mathbf{x}_i; y_i'; \widehat{\mathbf{w}}^T; \mathbf{0}_{D-2d-3}; 1; t_i]$ , such that
$$
\left\| \widehat {\mathbf {w}} ^ {T} - \mathbf {w} _ {\mathrm {G D}} ^ {T} \right\| _ {2} \leq \varepsilon_ {g} \cdot (L \beta^ {- 1} B _ {x}),
$$
where  $\{\mathbf{w}_{\mathrm{GD}}^{\ell}\}_{\ell \in [L]}$  is the sequence of gradient descent iterates with stepsize  $\beta^{-1}$  and initialization  $\mathbf{w}_{\mathrm{GD}}^{0} = \mathbf{0}$ . Notice that Proposition B.2 implies (with  $\kappa \coloneqq \beta /\alpha$ )
$$
\left\| \mathbf {w} _ {\mathrm {G D}} ^ {T} - \mathbf {w} _ {\mathrm {G L M}} \right\| _ {2} \leq \exp (- T / (2 \kappa)) \| \mathbf {w} _ {\mathrm {G L M}} \| _ {2} \leq \exp (- T / (2 \kappa)) \cdot \frac {B _ {w}}{2} := \varepsilon_ {o}.
$$
Furthermore, we can show that (similar to the proof of Theorem D.1 (b)), there exists a single attention layer  $\pmb{\theta}^{(T + 1)}$  with  $M$  heads such that it outputs  $\mathbf{h}_{N + 1}^{(T + 1)} = [\mathbf{x}_{N + 1};\widehat{y}_{N + 1};\widehat{\mathbf{w}}^T;\mathbf{0}_{D - 2d - 3};1;0]$ , where  $\left|\widehat{y}_{N + 1} - g(\langle \mathbf{x}_{N + 1},\widehat{\mathbf{w}}^T\rangle)\right| \leq \varepsilon_g$ .
In the following, we show that for suitably chosen  $(T,\varepsilon_g)$ ,  $\pmb{\theta} = (\pmb{\theta}^{(1:T)},\pmb{\theta}^{(T + 1)})$  is the desired transformer. First notice that its output  $\mathbf{h}_{N + 1}^{(T + 1)} = [\mathbf{x}_{N + 1};\widehat{y}_{N + 1};\widehat{\mathbf{w}}^T;\mathbf{0}_{D - 2d - 3};1;0]$  satisfies
$$
\begin{array}{l} {| \widehat {y} _ {N + 1} - g (\langle \mathbf {x} _ {N + 1}, \mathbf {w} _ {\mathrm {G L M}} \rangle) | \leq | \widehat {y} _ {N + 1} - g (\langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} ^ {T} \rangle) | + L _ {g} | \langle \mathbf {x} _ {N + 1}, \widehat {\mathbf {w}} ^ {T} \rangle - \langle \mathbf {x} _ {N + 1}, \mathbf {w} _ {\mathrm {G L M}} \rangle |} \\ \leq \varepsilon_ {g} + L _ {g} B _ {x} \left\| \widehat {\mathbf {w}} ^ {T} - \mathbf {w} _ {\mathrm {G D}} ^ {T} \right\| _ {2} + L _ {g} B _ {x} \left\| \mathbf {w} _ {\mathrm {G D}} ^ {T} - \mathbf {w} _ {\mathrm {G L M}} \right\| _ {2} \\ \leq \varepsilon_ {g} \left(1 + L _ {g} B _ {x} \cdot T \beta^ {- 1} B _ {x}\right) + L _ {g} B _ {x} \varepsilon_ {o}. \\ \end{array}
$$
Therefore, for any fixed  $\varepsilon > 0$ , we can take
$$
T = \lceil 2 \kappa \log (L _ {g} B _ {x} B _ {w} / \varepsilon) \rceil , \qquad \varepsilon_ {g} = \frac {1}{2} \frac {\varepsilon}{1 + T \cdot (L _ {g} B _ {x} ^ {2} \beta^ {- 1})},
$$
so that the  $\pmb{\theta}$  we construct above ensures  $|\widehat{y}_{N + 1} - g(\langle \mathbf{x}_{N + 1},\mathbf{w}_{\mathrm{GLM}}\rangle)|\leq \varepsilon$  for any input  $\mathbf{H}$  that satisfies (35). The upper bound on  $\| \pmb {\theta}\|$  follows immediately from Theorem D.1.
We summarize some basic and useful facts about GLM in the following theorem. Its proof is presented in Appendix G.3 - G.6.
Theorem G.3. Under Assumption  $B$ , the following statements hold with universal constant  $C_0$  and constant  $C_1, C_2$  that depend only on the parameters  $(K_x, K_y, B_\mu, B_w, \mu_x, L_g, \mu_g)$ .
(a) As long as  $N \geq C_1 \cdot d$ , the following event happens with probability at least  $1 - 2e^{-N / C_1}$ :
$$
\mathcal {E} _ {w}: \qquad \frac {1}{8} \mu_ {g} \mu_ {x} \leq \lambda_ {\min } (\nabla^ {2} \widehat {L} _ {N} (\mathbf {w})) \leq \lambda_ {\max } (\nabla^ {2} \widehat {L} _ {N} (\mathbf {w})) \leq 8 L _ {g} K _ {x} ^ {2}, \quad \forall \mathbf {w} \in \mathsf {B} _ {2} (B _ {w}).
$$
(b) For any  $\delta > 0$ , we have with probability at least  $1 - \delta$  that
$$
\varepsilon_ {\mathrm {s t a t}} := \sup _ {\mathbf {w} \in \mathsf {B} _ {2} (B _ {w})} \left\| \nabla_ {\mathbf {w}} \widehat {L} _ {N} (\mathbf {w}) - \nabla_ {\mathbf {w}} \mathbb {E} [ \widehat {L} _ {N} (\mathbf {w}) ] \right\| _ {2} \leq C _ {0} K _ {x} K _ {y} \max \left\{\sqrt {\frac {d \iota + \log (1 / \delta)}{N}}, \frac {d \iota + \log (1 / \delta)}{N} \right\},
$$
where we denote  $\iota = \log (2 + L_gK_x^2 B_w / K_y)$
(c) Condition on (a) holds and  $N \geq C_2 \cdot d$ , the event  $\mathcal{E}_r \coloneqq \{\| \mathbf{w}_{\mathrm{GLM}} \|_2 \leq B_w / 2\}$  happens with probability at least  $1 - e^{N / C_2}$ .
(d) For any  $\mathbf{w} \in \mathsf{B}_2(B_w)$ , it holds that
$$
L _ {p} (\mathbf {w}) - L _ {p} (\boldsymbol {\beta}) \leq \frac {4}{\mu_ {g} \mu_ {x}} \Bigl (\varepsilon_ {\mathrm {s t a t}} ^ {2} + \bigl \| \nabla \widehat {L} _ {N} (\mathbf {w}) \bigr \| _ {2} ^ {2} \Bigr).
$$
(e) (Realizable setting) As long as  $\mathbf{w}_{\mathrm{GLM}} \in \mathsf{B}_2(B_w)$ , it holds that
$$
\mathbb {E} _ {\mathbf {x}} \big (g (\langle \mathbf {x}, \mathbf {w} _ {\mathrm {G L M}} \rangle) - g (\langle \mathbf {x}, \boldsymbol {\beta} \rangle) \big) ^ {2} \leq \frac {L _ {g}}{\mu_ {x} \mu_ {g}} \varepsilon_ {\mathrm {s t a t}} ^ {2}.
$$
Therefore, we can set
$$
\alpha = \frac {\mu_ {g} \mu_ {x}}{8}, \quad \beta = 8 L _ {g} K _ {x} ^ {2},
$$
$$
B _ {x} = C _ {0} K _ {x} \sqrt {d \log (N / \delta)}, \qquad B _ {y} = C _ {0} K _ {y} \sqrt {\log (N / \delta)}.
$$
Consider the following good events
$$
\mathcal {E} _ {b} = \left\{\forall i \in [ N ], \| \mathbf {x} _ {i} \| _ {2} \leq B _ {x}, | y _ {i} | \leq B _ {y} \right\},
$$
$$
\mathcal {E} = \mathcal {E} _ {r} \cap \mathcal {E} _ {w} \cap \mathcal {E} _ {b} \cap \mathcal {E} _ {b, N + 1}.
$$
Under the event  $\mathcal{E}$  and our choice of  $\alpha, \beta$ , the problem (ICGLM) is well-conditioned (i.e. (35) holds).
Theorem G.1 implies that there exists a transformer  $\pmb{\theta}$  such that for any input  $\mathbf{H}$  of the form (3),  $\mathrm{TF}_{\pmb{\theta}}$  outputs  $\mathbf{h}_{N + 1}^{\prime} = [\mathbf{x}_{N + 1};\widetilde{\mathbf{y}}_{N + 1};\widetilde{\mathbf{w}};\mathbf{0}_{D - 2d - 3};1;0]$ , such that the output is given by  $\widehat{y}_{N + 1} = \widetilde{\mathrm{read}}_{\mathrm{y}}(\mathrm{TF}_{\pmb{\theta}}(\mathbf{H})) = \mathrm{clip}_{B_{\underline{y}}}(\widetilde{y}_{N + 1})$  and  $\widehat{\mathbf{w}} = \widetilde{\mathrm{read}}_{\mathbf{w}}(\mathrm{TF}_{\pmb{\theta}}(\mathbf{H})) := \mathrm{Proj}_{B_2(B_w)}(\widetilde{\mathbf{w}})$ , and the following holds on the good event  $\mathcal{E}$ :
(a)  $\widetilde{y}_{N + 1} = f_{\mathcal{D}}(\mathbf{x}_{N + 1})$  where  $f_{\mathcal{D}} = \mathcal{A}(\mathcal{D})$  is a predictor such that  $|f_{\mathcal{D}}(\mathbf{x}) - g(\langle \mathbf{x},\mathbf{w}_{\mathrm{GLM}}\rangle)|\leq$ $\varepsilon$  for all  $\mathbf{x}\in \mathsf{B}_2(B_x)$
(b)  $\widetilde{\mathbf{w}} = \widetilde{\mathbf{w}} (\mathcal{D})\in \mathsf{B}_2(B_w)$  depends only on  $\mathcal{D}$  (by the proof of Theorem G.1 and Theorem D.1), such that  $\left\| \nabla \widehat{L}_N(\widetilde{\mathbf{w}})\right\| _2\leq \frac{\beta\varepsilon}{L_gB_w}$ .
In the following, we show that  $\pmb{\theta}$  constructed above fulfills both (a) & (b) of Theorem G.2. The bounds on number of layers and heads and  $\| \pmb {\theta}\|$  follows from plugging our choice of  $B_{x},B_{y}$  in our proof of Theorem G.1.
Proof of Theorem G.2 (a). Notice that under the good event  $\mathcal{E}$ , we have  $\widehat{\mathbf{w}} = \widetilde{\mathbf{w}} = \widetilde{\mathbf{w}}(\mathcal{D})$  depends only on  $\mathcal{D}$ . Then we have
$$
\begin{array}{l} \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ \ell \left(\widehat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] \\ = \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} \right\} \ell \left(\hat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] + \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} ^ {c} \right\} \ell \left(\hat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] \\ = \mathbb {E} _ {(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1})} [ 1 \{\mathcal {E} \} \ell (\langle \mathbf {x} _ {N + 1}, \widetilde {\mathbf {w}} (\mathcal {D}) \rangle , y _ {N + 1}) ] + \mathbb {E} _ {(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1})} \left[ 1 \{\mathcal {E} ^ {c} \} \ell (\widehat {y} _ {N + 1} ^ {\mathrm {l i n}}, y _ {N + 1}) \right]. \\ \end{array}
$$
Thus, we can consider  $\mathcal{E}_0 = \mathcal{E}_r\cap \mathcal{E}_w\cap \mathcal{E}_b$  , and then
$$
\mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} \right\} \ell \left(\left\langle \mathbf {x} _ {N + 1}, \widetilde {\mathbf {w}} (\mathcal {D}) \right\rangle , y _ {N + 1} \right] \right.
$$
$$
\begin{array}{l} = \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} _ {0} \right\} \ell \left(\left\langle \mathbf {x} _ {N + 1}, \widetilde {\mathbf {w}} (\mathcal {D}) \right\rangle , y _ {N + 1}\right) \right] - \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} _ {0} - \mathcal {E} \right\} \ell \left(\left\langle \mathbf {x} _ {N + 1}, \widetilde {\mathbf {w}} (\mathcal {D}) \right\rangle , y _ {N + 1}\right) \right] \\ = \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} _ {0} \right\} L _ {p} (\widetilde {\mathbf {w}} (\mathcal {D})) \right] - \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} _ {0} - \mathcal {E} \right\} \ell \left(\left\langle \mathbf {x} _ {N + 1}, \widetilde {\mathbf {w}} (\mathcal {D}) \right\rangle , y _ {N + 1}\right) \right], \\ \end{array}
$$
where the second equality follows from  $L_{p}(\widehat{\mathbf{w}} (\mathcal{D})) = \mathbb{E}_{(\mathbf{x}_{N + 1},y_{N + 1})|\mathcal{D}}\ell (\langle \mathbf{x}_{N + 1},\widetilde{\mathbf{w}} (\mathcal{D})\rangle ,y_{N + 1}).$
$$
\begin{array}{l} \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ \ell \left(\hat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] - \mathbb {E} _ {\mathcal {D}} \left[ 1 \left\{\mathcal {E} _ {0} \right\} L _ {p} (\widetilde {\mathbf {w}} (\mathcal {D})) \right] \\ = \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} ^ {c} \right\} \ell \left(\widehat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] - \mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ 1 \left\{\mathcal {E} _ {0} - \mathcal {E} \right\} \ell \left(\left\langle \mathbf {x} _ {N + 1}, \widetilde {\mathbf {w}} (\mathcal {D}) \right\rangle , y _ {N + 1}\right) \right] \\ \leq 2 \sqrt {\mathbb {P} (\mathcal {E} ^ {c}) \cdot \max  \left\{\mathbb {E} \left[ \ell \left(\widehat {y} _ {N + 1} ^ {\mathrm {l i n}}, y _ {N + 1}\right) ^ {4} \right] , \mathbb {E} [ \ell \left(\langle \mathbf {x} _ {N + 1} , \widetilde {\mathbf {w}} (\mathcal {D}) \rangle , y _ {N + 1}\right) ^ {4} ] \right\}} = \mathcal {O} \left(\frac {B _ {\ell} ^ {2}}{N ^ {5}}\right), \\ \end{array}
$$
where the last line follows from Cauchy inequality and the fact  $\mathbb{P}(\mathcal{E}^c) = \mathcal{O}\left(N^{-10}\right)$ , and  $B_{\ell}$  is defined in Lemma G.1.
Notice that by Theorem G.3 (d), we have
$$
\mathbb {E} _ {\mathcal {D}} \left[ 1 \{\mathcal {E} _ {0} \} \left(L _ {p} (\widetilde {\mathbf {w}}) - \inf  L _ {p}\right) \right] \leq \frac {4}{\mu_ {g} \mu_ {x}} \left(\mathbb {E} \left[ \varepsilon_ {\text {s t a t}} ^ {2} \right] + \mathbb {E} \left[ 1 \{\mathcal {E} _ {0} \} \left\| \nabla \widehat {L} _ {N} (\widetilde {\mathbf {w}}) \right\| _ {2} ^ {2} \right]\right),
$$
and by Theorem G.3 (b) and taking integration over  $\delta > 0$ , we have
$$
\mathbb {E} \left[ \varepsilon_ {\text {s t a t}} ^ {2} \right] \leq \mathcal {O} (1) \cdot K _ {x} ^ {2} K _ {y} ^ {2} \left(\frac {d \iota}{N} + \left(\frac {d \iota}{N}\right) ^ {2}\right).
$$
Also, we have inf  $L_{p} = L_{p}(\beta^{\star})\leq B_{\ell}$  by Lemma G.1. Therefore, we can conclude that
$$
\mathbb {E} _ {\left(\mathcal {D}, \mathbf {x} _ {N + 1}, y _ {N + 1}\right)} \left[ \ell \left(\widehat {y} _ {N + 1} ^ {\text {l i n}}, y _ {N + 1}\right) \right] \leq \inf  L _ {p} + \mathcal {O} (1) \cdot \left(\frac {K _ {x} ^ {2} K _ {y} ^ {2} \iota}{\mu_ {g} \mu_ {x}} \frac {d}{N} + \frac {K _ {x} ^ {4}}{\mu_ {g} \mu_ {x} B _ {w}} \varepsilon^ {2} + \frac {B _ {\ell} ^ {2}}{N ^ {5}}\right).
$$
Taking  $\varepsilon^2\leq \frac{K_y^2\iota}{B_wK_x^2}\frac{d}{N}$  completes the proof.
Proof of Theorem G.2 (b). Similar to the proof of Corollary 6, we have
$$
\begin{array}{l} \mathbb {E} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} = \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \right] + \mathbb {E} \left[ 1 \{\mathcal {E} ^ {c} \} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \right] \\ \leq \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\widetilde {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \right] + \sqrt {\mathbb {P} \left(\mathcal {E} ^ {c}\right) \mathbb {E} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {4}}, \\ \end{array}
$$
where the inequality follows from  $y_{N + 1}\in [-B_y,B_y]$  on event  $\mathcal{E}$ . For the first part, we have
$$
\begin{array}{l} \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(\widetilde {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \right] = \mathbb {E} \left[ 1 \{\mathcal {E} \} \left(f _ {\mathcal {D}} \left(\mathbf {x} _ {N + 1}\right) - y _ {N + 1}\right) ^ {2} \right] \\ \left. \right. \leq \mathbb {E} _ {\mathcal {D}} \left[ 1 \left\{\mathcal {E} _ {0} \right\} \cdot \mathbb {E} _ {(\mathbf {x}, y) \sim \mathrm {P}} \left[ 1 \left\{\| \mathbf {x} \| _ {2} \leq B _ {x} \right\}\left(f _ {\mathcal {D}} (\mathbf {x}) - y\right) ^ {2} \right]\right], \\ \end{array}
$$
where we use the fact that the conditional distribution of  $(\mathbf{x}_{N + 1},y_{N + 1})|\mathcal{D}$  agrees with  $\mathsf{P}$ . Thus,
$$
\begin{array}{l} \mathbb {E} \left[ 1 \{\mathcal {E} \} (\widetilde {y} _ {N + 1} - y _ {N + 1}) ^ {2} \right] - \mathbb {E} _ {(\mathbf {x}, y) \sim \mathsf {P}} (g (\langle \mathbf {x}, \boldsymbol {\beta} \rangle) - y) ^ {2} \\ \leq \mathbb {E} _ {\mathcal {D}} \left[ 1 \left\{\mathcal {E} _ {0} \right\} \cdot \left(\mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {P}} 1 \left\{\| \mathbf {x} \| _ {2} \leq B _ {x} \right\} \left(f _ {\mathcal {D}} (\mathbf {x}) - y\right) ^ {2} - \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {P}} \left(g \left(\langle \mathbf {x}, \boldsymbol {\beta} \rangle\right) - y\right) ^ {2}\right) \right] \\ \leq \mathbb {E} _ {\mathcal {D}} \left[ 1 \left\{\mathcal {E} _ {0} \right\} \cdot \mathbb {E} _ {\mathbf {x}} 1 \left\{\| \mathbf {x} \| _ {2} \leq B _ {x} \right\} \left(f _ {\mathcal {D}} (\mathbf {x}) - g \left(\langle \mathbf {x}, \boldsymbol {\beta} \rangle\right)\right) ^ {2} \right] \\ \leq 2 \mathbb {E} _ {\mathcal {D}} \left[ 1 \{\mathcal {E} _ {0} \} \cdot \mathbb {E} _ {\mathbf {x}} 1 \{\| \mathbf {x} \| _ {2} \leq B _ {x} \} (f _ {\mathcal {D}} (\mathbf {x}) - g (\langle \mathbf {x}, \mathbf {w} _ {\mathrm {G L M}} \rangle)) ^ {2} \right] + 2 \mathbb {E} _ {\mathcal {D}} \left[ 1 \{\mathcal {E} _ {0} \} \cdot \mathbb {E} _ {\mathbf {x}} (g (\langle \mathbf {x}, \mathbf {w} _ {\mathrm {G L M}} \rangle) - g (\langle \mathbf {x}, \boldsymbol {\beta} \rangle)) ^ {2} \right] \\ \leq 2 \varepsilon^ {2} + \frac {2 L _ {g}}{\mu_ {x} \mu_ {g}} \mathbb {E} [ \varepsilon_ {\mathrm {s t a t}} ^ {2} ] \leq 2 \varepsilon^ {2} + \mathcal {O} (1) \cdot \frac {L _ {g} K _ {x} ^ {2} K _ {y} ^ {2} \iota}{\mu_ {x} \mu_ {g}} \frac {d}{N}. \\ \end{array}
$$
For the second part, we know  $\mathbb{P}(\mathcal{E}^c) = \mathcal{O}\left(N^{-10}\right)$  and
$$
\mathbb {E} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {4} \leq 8 \mathbb {E} \widehat {y} _ {N + 1} ^ {2} + 8 \mathbb {E} y _ {N + 1} ^ {4} = \mathcal {O} \left(B _ {y} ^ {4}\right).
$$
In conclusion, we have