SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left(\widehat {y} _ {N + 1} - y _ {N + 1}\right) ^ {2} \leq \mathbb {E} \left(y _ {N + 1} - g \left(\left\langle \mathbf {x} _ {N + 1}, \boldsymbol {\beta} \right\rangle\right)\right) ^ {2} + 2 \varepsilon^ {2} + \mathcal {O} (1) \cdot \frac {L _ {g} K _ {x} ^ {2} K _ {y} ^ {2} \iota}{\mu_ {x} \mu_ {g}} \frac {d}{N} + \mathcal {O} \left(\frac {B _ {y} ^ {2}}{N ^ {5}}\right).
$$
Taking  $\varepsilon^2\leq \frac{L_gK_x^2K_y^2\iota}{\mu_x\mu_g}\frac{d}{N}$  completes the proof.
Lemma G.1. Suppose that  $\mathbf{x} \sim \mathrm{SG}(K_x)$ ,  $y \sim \mathrm{SG}(K_y)$ , and  $\mathbf{w}$  is a (possibly random) vector such that  $\|\mathbf{w}\|_2 \leq B_w$ . Then
$$
\mathbb {E} \left[ \ell (\langle \mathbf {x}, \mathbf {w} \rangle , y) ^ {4} \right] ^ {1 / 4} \leq \mathcal {O} \left(L _ {g} K _ {x} ^ {2} B _ {w} ^ {2} d + K _ {x} K _ {y} B _ {w} d\right) =: B _ {\ell}.
$$
Proof. Notice that by our assumption,  $|g(0)| \leq 2K_y$ . Therefore, by the definition of  $\ell$
$$
| \ell (t, y) | = \left| - y t + \int_ {0} ^ {t} g (s) d s \right| \leq | t (g (0) - y) | + \left| \int_ {0} ^ {t} (g (s) - g (0)) d s \right| \leq | t | (2 K _ {y} + | y |) + 2 L _ {g} t ^ {2}.
$$
The proof is then done by bounding the moment by  $\mathbb{E}|y|^8\leq \mathcal{O}\left(K_y^8\right)$  and  $\mathbb{E}|\langle \mathbf{x},\mathbf{w}\rangle |^8\leq$ $B_w^8\mathbb{E}\| \mathbf{x}\| _2^8\leq \mathcal{O}\left((\sqrt{d} B_wK_x)^8\right)$ , which is standard (by utilizing the tail bound of sub-Gaussian/sub-Exponential random variable).
G.3 Proof of Theorem G.3 (a)
We begin with the upper bound on  $\lambda_{\mathrm{max}}(\nabla^2\widehat{L}_N(\mathbf{w}))$ . By Lemma B.3, as long as  $N\geq C_0\cdot d$ , the following event
$$
\mathcal {E} _ {w, 0}: \quad \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \right\| _ {\mathrm {o p}} \leq 8 K ^ {2}.
$$
happens with probability at least  $1 - \exp (-N / C_0)$ . By the assumption that  $\sup |g^{\prime}|\leq L_g$ , it is clear that when  $\mathcal{E}_{w,0}$  holds, we have  $\lambda_{\max}(\nabla^{2}\widehat{L}_{N}(\mathbf{w}))\leq 8L_{g}K_{x}^{2}\forall \mathbf{w}\in \mathbb{R}^{d}$ .
In the following, we analyze the quantity  $\lambda_{\mathrm{max}}(\nabla^2\widehat{L}_N(\mathbf{w}))$ . We have to invoke the following covering argument (see e.g. [85, Section 4.1.1]).
Lemma G.2. Suppose that  $\mathcal{V}$  is a  $\varepsilon$ -covering of  $\mathbb{S}^{d-1}$  with  $\varepsilon \in [0,1)$ . Then the following holds:
1. For any  $d \times d$  symmetric matrix  $A$ ,  $\| A \|_{\mathrm{op}} \leq \frac{1}{1 - 2\varepsilon} \max_{\mathbf{v} \in \mathcal{V}} |\mathbf{v}^\top A \mathbf{v}|$  and
$$
\lambda_ {\min } (A) \geq \min  _ {\mathbf {v} \in \mathcal {V}} \mathbf {v} ^ {\top} A \mathbf {v} - 2 \varepsilon \| A \| _ {\mathrm {o p}}
$$
2. For any vector  $\mathbf{x} \in \mathbb{R}^d$ ,  $\| \mathbf{x}\|_2 \leq \frac{1}{1 - \varepsilon} \max_{\mathbf{v} \in \mathcal{V}} |\langle \mathbf{v}, \mathbf{x}\rangle|$ .
$$
\begin{array}{l} \nabla^ {2} \widehat {L} _ {N} (\mathbf {w}) = \frac {1}{N} \sum_ {i = 1} ^ {N} g ^ {\prime} (\langle \mathbf {w}, \mathbf {x} _ {i} \rangle) \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \succeq \frac {1}{N} \sum_ {i = 1} ^ {N} \mu_ {g} \mathbb {I} (| \langle \mathbf {w}, \mathbf {x} _ {i} \rangle | \leq B _ {\mu}) \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \\ \succeq \frac {1}{N} \sum_ {i = 1} ^ {N} \mu_ {g} \left(1 - \frac {| \langle \mathbf {w} , \mathbf {x} _ {i} \rangle |}{B _ {\mu}}\right) _ {+} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}. \\ \end{array}
$$
Therefore, we can define  $h(t) \coloneqq (B_{\mu} - |t|)_{+}$  (which is a 1-Lipschitz function), and we have
$$
\nabla^ {2} \widehat {L} _ {N} (\mathbf {w}) \succeq \frac {\mu_ {g}}{B _ {\mu}} \underbrace {\frac {1}{N} \sum_ {i = 1} ^ {N} h (\langle \mathbf {w} , \mathbf {x} _ {i} \rangle) \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}} _ {=: A (\mathbf {w})}.
$$
In the following, we pick a  $\varepsilon_{\mathbf{v}}$ -covering  $\mathcal{V}$  of  $\mathbb{S}^{d-1}$  such that  $|\mathcal{V}| \leq (3 / \varepsilon_{\mathbf{v}})^{d}$  (we will specify  $\varepsilon_{\mathbf{v}}$  later in proof). Then for any  $\mathbf{w} \in \mathsf{B}_2(B_w)$ ,
$$
\lambda_ {\min } (A (\mathbf {w})) \geq \min  _ {\mathbf {v} \in \mathcal {V}} \mathbf {v} ^ {\top} A (\mathbf {w}) \mathbf {v} - 2 \varepsilon_ {\mathbf {v}} \| A (\mathbf {w}) \| _ {\mathrm {o p}}
$$
By our definition of  $A(\mathbf{w})$ , we have (for any fixed  $B_{xv}$ )
$$
\min  _ {\mathbf {v} \in \mathcal {V}} \mathbf {v} ^ {\top} A (\mathbf {w}) \mathbf {v} = \min  _ {\mathbf {v} \in \mathcal {V}} \frac {1}{N} \sum_ {i = 1} ^ {N} h \left(\left\langle \mathbf {w}, \mathbf {x} _ {i} \right\rangle\right) \left\langle \mathbf {v}, \mathbf {x} _ {i} \right\rangle^ {2}
$$
$$
\begin{array}{l} \geq \min  _ {\mathbf {v} \in \mathcal {V}} \underbrace {\frac {1}{N} \sum_ {i = 1} ^ {N} h (\langle \mathbf {w} , \mathbf {x} _ {i} \rangle) \min  \left\{\langle \mathbf {v} , \mathbf {x} _ {i} \rangle^ {2} , B _ {x v} ^ {2} \right\}} _ {=: U _ {\mathbf {v}} (\mathbf {w})} \\ \geq \min  _ {\mathbf {v} \in \mathcal {V}} \mathbb {E} \left[ U _ {\mathbf {v}} (\mathbf {w}) \right] + \min  _ {\mathbf {v} \in \mathcal {V}} \left(U _ {\mathbf {v}} (\mathbf {w}) - \mathbb {E} \left[ U _ {\mathbf {v}} (\mathbf {w}) \right]\right). \\ \end{array}
$$
By Lemma G.3, we can choose  $B_{xv} = K_x(15 + \log (K_x^2 /\mu_x))$ , and then  $\mathbb{E}[U_{\mathbf{v}}(\mathbf{w})]\geq 3B_{\mu}\mu_{x} / 8$ . Thus, combining the inequalities above, we can take  $\varepsilon_{\mathbf{v}} = \frac{128K_x^2}{\mu_x}$  in the following, so that under event  $\mathcal{E}_{w,0}$
$$
\lambda_ {\min} (\nabla^ {2} \widehat {L} _ {N} (\mathbf {w})) \geq \frac {\mu_ {g} \mu_ {x}}{8} + \frac {\mu_ {g}}{B _ {\mu}} \left(\frac {B _ {\mu} \mu_ {x}}{1 6} - \max _ {\mathbf {v} \in \mathcal {V}} \left(\mathbb {E} [ U _ {\mathbf {v}} (\mathbf {w}) ] - U _ {\mathbf {v}} (\mathbf {w})\right)\right).
$$
In the following, we consider the random process  $\left\{\overline{U}_{\mathbf{v}}(\mathbf{w}):= U_{\mathbf{v}}(\mathbf{w}) - \mathbb{E}[U_{\mathbf{v}}(\mathbf{w})]\right\}_{\mathbf{w}}$ , which is zero-mean and indexed by  $\mathbf{w}\in \mathsf{B}_2(B_w)$ . For any fixed  $\mathbf{v}$ , consider applying Proposition B.4 to the random process  $\left\{\overline{U}_{\mathbf{v}}(\mathbf{w})\right\}_{\mathbf{w}}$ . We need to verify the preconditions:
(a) With norm  $\rho (\mathbf{w},\mathbf{w}^{\prime}) = \| \mathbf{w} - \mathbf{w}^{\prime}\|_{2}$ ,  $\log \mathcal{N}(\mathtt{B}_{\rho}(\mathbf{w},r),\delta)\leq d\log (2Ar / \delta)$  with constant  $A = 2$
(b) Let  $f(\mathbf{x}; \mathbf{w}) \coloneqq h(\langle \mathbf{w}, \mathbf{x}_i \rangle) \min \left\{ \langle \mathbf{v}, \mathbf{x}_i \rangle^2, B_{xv}^2 \right\}$ , then  $|f(\mathbf{x}; \mathbf{w})| \leq B_\mu B_{xv}^2$  and hence in  $\mathrm{SG}(CB_\mu B_{xv}^2)$  for any random  $\mathbf{x}$ ;
(c) For  $\mathbf{w},\mathbf{w}^{\prime}\in \mathcal{W}$ , we have  $|h(\langle \mathbf{w},\mathbf{x}_i\rangle) - h(\langle \mathbf{w}^{\prime},\mathbf{x}_i\rangle)|\leq |\langle \mathbf{w} - \mathbf{w}^{\prime},\mathbf{x}_i\rangle|$ . Hence, because  $\mathbf{x}\sim$ $\mathrm{SG}(K_x)$ , the random variable  $h(\langle \mathbf{w},\mathbf{x}\rangle) - h(\langle \mathbf{w}^{\prime},\mathbf{x}\rangle)$  is  $\mathrm{SG}(CK_x\| \mathbf{w} - \mathbf{w}^{\prime}\| _2)$ , and the random variable  $f(\mathbf{x};\mathbf{w}) - f(\mathbf{x};\mathbf{w}^{\prime})$  is  $\mathrm{SG}(CK_xB_{xv}^2\| \mathbf{w} - \mathbf{w}^{\prime}\| _2)$ .
Therefore, we can apply Proposition B.4 to obtain that with probability  $1 - \delta_0$ , it holds
$$
\sup  _ {\mathbf {w}} \left| \overline {{U}} _ {\mathbf {v}} (\mathbf {w}) \right| \leq C ^ {\prime} B _ {\mu} B _ {x v} ^ {2} \left[ \sqrt {\frac {d \log (2 \kappa_ {g}) + \log (1 / \delta_ {0})}{N}} \right],
$$
where we denote  $\kappa_{g} = 1 + K_{x}B_{w} / B_{\mu}$ . Setting  $\delta_0 = \delta /|\mathcal{V}|$  and taking the union bound over  $\mathbf{v}\in \mathcal{V}$ , we obtain that with probability at least  $1 - \delta$
$$
\max _ {\mathbf {v} \in \mathcal {V}} \sup _ {\| \mathbf {w} \| _ {2} \leq B _ {w}} \left| \overline {{U}} _ {\mathbf {v}} (\mathbf {w}) \right| \leq C ^ {\prime} B _ {\mu} B _ {x v} ^ {2} \left[ \sqrt {\frac {d \log (8 \kappa_ {g} / \varepsilon_ {\mathbf {v}}) + \log (1 / \delta)}{N}} \right],
$$
where we use  $\log |\mathcal{V}| \leq d\log (4 / \varepsilon_{\mathbf{v}})$ . Therefore, we plug in the definition of  $\varepsilon_{\mathbf{v}}$  and  $B_{xv}$  to deduce that, if we set
$$
C _ {1} = \left(\frac {1 6 C ^ {\prime} B _ {x v} ^ {2}}{\mu_ {x}}\right) ^ {2} \log \left(8 \kappa_ {g} / \varepsilon_ {\mathbf {v}}\right), \qquad \varepsilon_ {\mathbf {v}} = \frac {1 2 8 K _ {x} ^ {2}}{\mu_ {x}}, \qquad B _ {x v} = K _ {x} (1 5 + \log \left(K _ {x} ^ {2} / \mu_ {x}\right)),
$$
then as long as  $N \geq C_1 \cdot d$ , it holds  $\max_{\mathbf{v} \in \mathcal{V}} \mathbb{E}[U_{\mathbf{v}}(\mathbf{w})] - U_{\mathbf{v}}(\mathbf{w}) \leq \frac{\mu_x B_\mu}{16}$  with probability at least  $1 - \exp(-N / C_1)$ . This is the desired result.
Lemma G.3. Under Assumption  $B$ , for  $B_{xv} = K_x(15 + \log (K_x^2 /\mu_x))$ , it holds
$$
\inf_ {\mathbf {w} \in \mathsf {B} _ {2} (B _ {w}), \mathbf {v} \in \mathbb {S} ^ {d - 1}} \mathbb {E} [ 1 \{| \mathbf {x} ^ {\top} \mathbf {w} | \leq B _ {\mu} / 2 \} (\mathbf {x} ^ {\top} \mathbf {v}) ^ {2} 1 \{| \mathbf {x} ^ {\top} \mathbf {v} | \leq B _ {x v} \} ] \geq 3 \mu_ {x} / 4.
$$
Proof. For any fixed  $\mathbf{w} \in \mathsf{B}_2(B_w)$ ,  $\mathbf{v} \in \mathbb{S}^{d-1}$
$$
\begin{array}{l} \mathbb {E} \left[ 1 \left\{\left| \mathbf {x} ^ {\top} \mathbf {w} \right| \leq B _ {\mu} / 2 \right\} \left(\mathbf {x} ^ {\top} \mathbf {v}\right) ^ {2} 1 \left\{\left| \mathbf {x} ^ {\top} \mathbf {v} \right| \leq B _ {x v} \right\} \right] \\ = \mathbb {E} \left[ \right. 1 \left\{\left| \mathbf {x} ^ {\top} \mathbf {w} \right| \leq B _ {\mu} / 2 \right\}\left(\mathbf {x} ^ {\top} \mathbf {v}\right) ^ {2} \left. \right\}\left. \right] - \mathbb {E} \left[ 1 \left\{\left| \mathbf {x} ^ {\top} \mathbf {w} \right| \leq B _ {\mu} / 2 \right\}\left(\mathbf {x} ^ {\top} \mathbf {v}\right) ^ {2} 1 \left\{\left| \mathbf {x} ^ {\top} \mathbf {v} \right| > B _ {x v} \right\}\right] \\ \geq \mu_ {x} - \mathbb {E} [ (\mathbf {x} ^ {\top} \mathbf {v}) ^ {2} 1 \{| \mathbf {x} ^ {\top} \mathbf {v} | > B _ {x v} \} ]. \\ \end{array}
$$
Because  $\mathbf{x} \sim \mathrm{SG}(K_x)$ ,  $\mathbf{x}^\top \mathbf{v} \sim \mathrm{SG}(K_x)$ , and a simple calculation yields
$$
\mathbb {E} [ (\mathbf {x} ^ {\top} \mathbf {v}) ^ {2} 1 \{| \mathbf {x} ^ {\top} \mathbf {v} | > t K _ {x} \} ] \leq 2 K _ {x} ^ {2} (t ^ {2} + 1) \exp (- t ^ {2}).
$$
Taking  $t = 15 + \log (K_x^2 /\mu_x)$  gives  $\mathbb{E}[(\mathbf{x}^\top \mathbf{v})^2 1\{| \mathbf{x}^\top \mathbf{v}| > B_{xv}\} ]\leq \mu_x / 4$  , which completes the proof.
G.4 Proof of Theorem G.3 (b)
$$
\nabla \widehat {L} _ {N} (\mathbf {w}) = \frac {1}{N} \sum_ {i = 1} ^ {N} (g (\langle \mathbf {w}, \mathbf {x} _ {i} \rangle) - y _ {i}) \mathbf {x} _ {i}.
$$
In the following, we pick a minimal  $1/2$ -covering of  $\mathbb{S}^{d-1}$  (so  $|\mathcal{V}| \leq 5^d$ ). Then by Lemma G.2, it holds
$$
\left\| \nabla \widehat {L} _ {N} (\mathbf {w}) - \mathbb {E} [ \nabla \widehat {L} _ {N} (\mathbf {w}) ] \right\| _ {2} \leq 2 \max  _ {\mathbf {v} \in \mathcal {V}} \left| \underbrace {\langle \nabla \widehat {L} _ {N} (\mathbf {w}) , \mathbf {v} \rangle - \mathbb {E} [ \langle \nabla \widehat {L} _ {N} (\mathbf {w}) , \mathbf {v} \rangle ]} _ {=: X _ {\mathbf {v}} (\mathbf {w})} \right.
$$
Fix a  $\mathbf{v} \in \mathbb{S}^{d-1}$  and set  $\delta' = \delta / |\mathcal{V}|$ . We proceed to bound  $\sup_{\mathbf{w}} |X_{\mathbf{v}}(\mathbf{w})|$  by applying Proposition B.4 to the random process  $\{X_{\mathbf{v}}(\mathbf{w})\}_{\mathbf{w}}$ . We need to verify the preconditions:
(a) With norm  $\rho (\mathbf{w},\mathbf{w}^{\prime}) = \| \mathbf{w} - \mathbf{w}^{\prime}\|_{2}$ ,  $\log N(\delta ;\mathsf{B}_{\rho}(r),\rho)\leq d\log (2Ar / \delta)$  with constant  $A = 2$
(b) For  $\mathbf{z} = [\mathbf{x};y]$ , we let  $f(\mathbf{z};\mathbf{w})\coloneqq (g(\langle \mathbf{w},\mathbf{x}\rangle) - y)\langle \mathbf{x},\mathbf{v}\rangle$ , then  $f(\mathbf{z};\mathbf{w})\sim \mathrm{SE}(CK_xK_y)$  for any  $\mathbf{w}$  by our assumption on  $(\mathbf{x},y)$ ;
(c) For  $\mathbf{w},\mathbf{w}^{\prime}\in \mathcal{W}$ , we have  $|g(\langle \mathbf{w},\mathbf{x}\rangle) - g(\langle \mathbf{w}^{\prime},\mathbf{x}\rangle)|\leq L_{g}|\langle \mathbf{w} - \mathbf{w}^{\prime},\mathbf{x}\rangle|$ . Hence, because  $\mathbf{x}\sim$ $\mathrm{SG}(K_x)$ , the random variable  $g(\langle \mathbf{w},\mathbf{x}_i\rangle) - g(\langle \mathbf{w}^{\prime},\mathbf{x}_i\rangle)$  is sub-Gaussian in  $\mathrm{SG}(K_xL_g\| \mathbf{w} - \mathbf{w}^{\prime}\| _2)$ . Thus,  $f(\mathbf{z};\mathbf{w}) - f(\mathbf{z};\mathbf{w}^{\prime})$  is sub-exponential in  $\mathrm{SE}(CK_x^2 L_g\| \mathbf{w} - \mathbf{w}^{\prime}\| _2)$ .
$$
\sup _ {\mathbf {w}} | X _ {\mathbf {v}} (\mathbf {w}) | \leq C ^ {\prime} K _ {x} K _ {y} \left[ \sqrt {\frac {d \log (2 \kappa_ {y}) + \log (1 / \delta_ {0})}{N}} + \frac {d \log (2 \kappa_ {y}) + \log (1 / \delta_ {0})}{N} \right],
$$
where we denote  $\kappa_y = 1 + L_g K_x^2 B_w / K_y$ . Setting  $\delta_0 = \delta / |\mathcal{V}|$  and taking the union bound over  $\mathbf{v} \in \mathcal{V}$ , we obtain that with probability at least  $1 - \delta$ ,
$$
\max  _ {\mathbf {v} \in \mathcal {V}} \sup  _ {\| \mathbf {w} \| _ {2} \leq B _ {w}} | X _ {\mathbf {v}} (\mathbf {w}) | \leq C ^ {\prime} K _ {x} K _ {y} \left[ \sqrt {\frac {d \log (1 0 \kappa_ {y}) + \log (1 / \delta)}{N}} + \frac {d \log (1 0 \kappa_ {y}) + \log (1 / \delta)}{N} \right].
$$
G.5 Proof of Theorem G.3 (c)
In the following, we condition on (a) holds, i.e.  $\widehat{L}_N$  is  $\alpha$ -strongly-convex and  $\beta$  smooth over  $\mathsf{B}_2(B_w)$  with  $\alpha = \mu_x\mu_g / 8$  and  $\beta = 8L_{g}K_{x}^{2}$ . We define
$$
\widetilde {\mathbf {w}} = \operatorname * {a r g   m i n} _ {\mathbf {w} \in \mathsf {B} _ {2} (B _ {w})} \widehat {L} _ {N} (\mathbf {w}).
$$
Then by standard convex analysis, we have
$$
\alpha \left\| \widetilde {\mathbf {w}} - \boldsymbol {\beta} ^ {\star} \right\| _ {2} ^ {2} \leq \left\langle \nabla \widehat {L} _ {N} (\widetilde {\mathbf {w}}) - \nabla \widehat {L} _ {N} (\boldsymbol {\beta} ^ {\star}), \widetilde {\mathbf {w}} - \boldsymbol {\beta} ^ {\star} \right\rangle \leq \left\langle - \nabla \widehat {L} _ {N} (\boldsymbol {\beta} ^ {\star}), \widetilde {\mathbf {w}} - \boldsymbol {\beta} ^ {\star} \right\rangle \leq \left\| \nabla \widehat {L} _ {N} (\boldsymbol {\beta} ^ {\star}) \right\| _ {2} \| \widetilde {\mathbf {w}} - \boldsymbol {\beta} ^ {\star} \| _ {2}.
$$
Notice that  $\left\| \nabla \widehat{L}_N(\beta^\star)\right\|_2 \leq \varepsilon_{\mathrm{stat}}$ , we can conclude that
$$
\| \widetilde {\mathbf {w}} \| _ {2} \leq \| \boldsymbol {\beta} ^ {\star} \| _ {2} + \frac {\varepsilon_ {\mathrm {s t a t}}}{\alpha}.
$$
Recall that we assume  $\| \beta^{\star}\|_{2}\leq B_{w} / 4$ , we can then consider  $\mathcal{E}_s\coloneqq \{\varepsilon_{\mathrm{stat}} < \alpha B_w / 4\}$ . Once  $\mathcal{E}_s$  holds, our argument above yields  $\| \widetilde{\mathbf{w}}\| _2 < B_w$ , which implies  $\nabla \widehat{L}_N(\widetilde{\mathbf{w}}) = 0$ . Therefore,  $\widetilde{\mathbf{w}} = \arg \min_{\mathbf{w}\in \mathbb{R}^d}\widehat{L}_N(\mathbf{w})$ . Further, by Theorem G.3, we can set
$$
C _ {2} := \max \left\{2 \iota \left(\frac {3 2 \alpha K _ {x} K _ {y}}{B _ {w}}\right) ^ {2}, 2 \iota \cdot \frac {3 2 \alpha K _ {x} K _ {y}}{B _ {w}} \right\},
$$
so that as long as  $N \geq C_2d$ , the event  $\mathcal{E}_s$  holds with probability at least  $1 - \exp(-N / C_2)$ . This is the desired result.
G.6 Proof of Theorem G.3 (d) & (e)
We first prove Theorem G.3 (d). Notice that
$$
\nabla^ {2} L _ {p} (\mathbf {w}) = \mathbb {E} \left[ g ^ {\prime} (\langle \mathbf {x}, \mathbf {w} \rangle) \mathbf {x x} ^ {\top} \right] \succeq \mathbb {E} \left[ \mu_ {g} \mathbb {I} (| \langle \mathbf {x}, \mathbf {w} \rangle | \leq B _ {\mu}) \mathbf {x x} ^ {\top} \right] \succeq \mu_ {g} \mu_ {x} \mathbf {I} _ {d}, \forall \mathbf {w} \in \mathsf {B} _ {2} (B _ {w}).
$$
Therefore,  $L_{p}$  is  $(\mu_g\mu_x)$ -strongly-convex over  $\mathsf{B}_2(B_w)$ . Therefore, because  $\beta^{\star} \in \mathsf{B}_2(B_w)$  is the global minimum of  $L_{p}$ , it holds that for all  $\mathbf{w} \in \mathsf{B}_2(B_w)$ ,
$$
L _ {p} (\mathbf {w}) - L _ {p} (\boldsymbol {\beta} ^ {\star}) \leq \frac {1}{2 \mu_ {g} \mu_ {x}} \| \nabla L _ {p} (\mathbf {w}) \| _ {2} ^ {2}.
$$
By the definition of  $\varepsilon_{\mathrm{stat}}$ ,  $\| \nabla L_p(\mathbf{w}) \|_2 \leq \varepsilon_{\mathrm{stat}} + \| \nabla \widehat{L}_N(\mathbf{w}) \|_2$ , and hence the proof of Theorem G.3 (d) is completed.
We next prove Theorem G.3 (e), where we assume that  $\mathbb{E}[y|\mathbf{x}] = g(\langle \mathbf{x},\boldsymbol {\beta}\rangle)$  (which implies  $\beta^{\star} = \beta$  directly) and  $\mathbf{w}_{\mathrm{GLM}}\in \mathsf{B}_2(B_w)$ . Notice that
$$
\nabla L _ {p} (\mathbf {w}) = \mathbb {E} \Big [ \nabla \widehat {L} _ {N} (\mathbf {w}) \Big ] = \mathbb {E} [ (g (\langle \mathbf {x}, \mathbf {w} \rangle) - y) \mathbf {x} ] = \mathbb {E} [ (g (\langle \mathbf {x}, \mathbf {w} \rangle) - g (\langle \mathbf {w}, \boldsymbol {\beta} \rangle)) \mathbf {x} ],
$$
$$
\begin{array}{l} \langle \nabla L _ {p} (\mathbf {w} _ {\mathrm {G L M}}), \mathbf {w} _ {\mathrm {G L M}} - \boldsymbol {\beta} \rangle = \mathbb {E} \left[ \left(g \left(\langle \mathbf {x}, \mathbf {w} _ {\mathrm {G L M}} \rangle\right) - g \left(\langle \mathbf {w}, \boldsymbol {\beta} \rangle\right)\right) \cdot \left(\left\langle \mathbf {x}, \mathbf {w} _ {\mathrm {G L M}} \right\rangle - \langle \mathbf {w}, \boldsymbol {\beta} \rangle\right) \right] \\ \geq \frac {1}{L _ {g}} \mathbb {E} \left[ \left(g \left(\langle \mathbf {x}, \mathbf {w} _ {\mathrm {G L M}} \rangle\right) - g \left(\langle \mathbf {w}, \boldsymbol {\beta} \rangle\right)\right) ^ {2} \right]. \\ \end{array}
$$
On the other hand, by the  $(\mu_g\mu_x)$ -strong-convexity of  $L_{p}$  over  $\mathsf{B}_2(B_w)$ , it holds that
$$
\langle \nabla L _ {p} (\mathbf {w} _ {\mathrm {G L M}}), \mathbf {w} _ {\mathrm {G L M}} - \boldsymbol {\beta} \rangle \leq \frac {1}{\mu_ {g} \mu_ {x}} \left\| \nabla L _ {p} (\mathbf {w} _ {\mathrm {G L M}}) \right\| _ {2} ^ {2}.
$$
Finally, using the definition of  $\mathbf{w}_{\mathrm{GLM}}$ , we have  $\nabla \widehat{L}_N(\mathbf{w}_{\mathrm{GLM}}) = 0$ , and hence  $\| \nabla L_p(\mathbf{w}_{\mathrm{GLM}})\|_2 \leq \varepsilon_{\mathrm{stat}}$ , which completes the proof of Theorem G.3 (e).
H Proofs for Section 3.2
H.1 Proof of Theorem 7
Fix  $\lambda_N \geq 0$ ,  $\beta > 0$  and  $B_w > 0$ , and consider any in-context data  $\mathcal{D}$  such that the precondition of Theorem 7 holds. Recall that
$$
L _ {\text {l a s s o}} (\mathbf {w}) := \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(\left\langle \mathbf {w}, \mathbf {x} _ {i} \right\rangle - y _ {i}\right) ^ {2} + \lambda_ {N} \| \mathbf {w} \| _ {1}
$$
denotes the lasso regression loss in (ICLasso), so that  $\mathbf{w}_{\mathrm{lasso}} = \arg \min_{\mathbf{w}\in \mathbb{R}^d}L_{\mathrm{lasso}}(\mathbf{w})$ . We further write
$$
\widehat {L} _ {N} ^ {0} (\mathbf {w}) := \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(\left\langle \mathbf {w}, \mathbf {x} _ {i} \right\rangle - y _ {i}\right) ^ {2}, \qquad \mathcal {R} (\mathbf {w}) := \lambda_ {N} \| \mathbf {w} \| _ {1}.
$$
Note that  $\nabla^2\widehat{L}_N^0 (\mathbf{w}) = \mathbf{X}^\top \mathbf{X} / N$  and thus  $\widehat{L}_N^0$  is  $\beta$ -smooth over  $\mathbb{R}^d$ .
Consider the proximal gradient descent algorithm on the ridge loss
$$
\mathbf {w} _ {\mathrm {P G D}} ^ {t + 1} = \mathbf {p r o x} _ {\eta \mathcal {R}} \left(\mathbf {w} _ {\mathrm {P G D}} ^ {t} - \eta \nabla \widehat {L} _ {N} ^ {0} \left(\mathbf {w} _ {\mathrm {P G D}} ^ {t}\right)\right)
$$
with initialization  $\mathbf{w}_{\mathrm{PGD}}^0 \coloneqq \mathbf{0}_d$ , learning rate  $\eta \coloneqq \beta^{-1}$ , and number of steps  $T$  to be specified later. Similar to the proof of Theorem 4, we can construct a transformer to approximate  $\mathbf{w}_{\mathrm{GD}}^T$ . Consider  $\ell(s,t) = \frac{1}{2}(s - t)^2$  and  $\mathcal{R}(\mathbf{w}) = \lambda_N\|\mathbf{w}\|_1$ , then  $\partial_s\ell(s,t)$  is  $(0, +\infty, 2, 4)$ -approximable by sum of relus (cf. Definition D.1), and  $\mathbf{prox}_{\eta\mathcal{R}}$  is  $(0, +\infty, 4d, 4 + 2\eta\lambda_N)$ -approximable by sum of relus (Proposition D.1). Therefore, we can apply Theorem D.2 with the square loss  $\ell$ , regularizer  $\mathcal{R}$ , learning rate  $\eta$  and accuracy parameter 0 to obtain that there exists a transformer  $\mathrm{TF}_{\theta}$  with  $(T + 1)$
layers, number of heads  $M^{(\ell)} = 2$  for all  $\ell \in [L]$ , and hidden dimension  $D' = 2d$ , such that the final output  $\mathbf{h}_{N+1}^{(L)} = [\mathbf{x}_{N+1}; \widehat{y}_{N+1}; \mathbf{w}_{\mathrm{PGD}}^T; *]$  with  $\widehat{y}_{N+1} = \langle \mathbf{w}_{\mathrm{PGD}}^T, \mathbf{x}_{N+1} \rangle$ . Further, the weight matrices have norm bounds  $\| \pmb{\theta} \| \leq 10R + (8 + 2\lambda_N)\beta^{-1}$ .
By the standard convergence result for proximal gradient descent (Proposition B.3), we have for all  $t \geq 1$  that
$$
L _ {\mathrm {l a s s o}} (\mathbf {w} _ {\mathrm {P G D}} ^ {t}) - L _ {\mathrm {l a s s o}} (\mathbf {w} _ {\mathrm {l a s s o}}) \leq \frac {\beta}{2 t} \left\| \mathbf {w} _ {\mathrm {l a s s o}} \right\| _ {2} ^ {2}.
$$
Plugging in  $\| \mathbf{w}_{\mathrm{lasso}}\| _2\leq B_w / 2$  and  $T = L - 1 = \left\lceil \beta B_w^2 /\varepsilon \right\rceil$  finishes the proof.
H.2 Sharper convergence analysis of proximal gradient descent for Lasso
Collection of parameters Throughout the rest of this section, we consider fixed  $N \geq 1$ ,  $\lambda_{N} = \sqrt{\frac{\rho\nu\log d}{N}}$  for  $\rho \geq 0$ ,  $\nu \geq 0$  fixed (and to be determined), fixed  $0 < \alpha \leq \beta$ , and fixed  $B_w^\star > 0$ . We write  $\kappa := \beta / \alpha$ ,  $\kappa_s := \beta (B_w^\star)^2 / \nu^2$ , and  $\omega_N := \frac{\rho s\log d}{\alpha N}$ .
Here we present a sharper convergence analysis on the proximal gradient descent algorithm for  $L_{\mathrm{lasso}}$  under the following well-conditionedness assumption, which will be useful for proving Theorem 8 in the sequel.
Assumption C (Well-conditioned property for Lasso). We say the (ICLasso) problem is well-conditioned with sparsity  $s$  if the following conditions hold:
1. The  $(\alpha, \rho)$ -RSC condition holds:
$$
\frac {\left\| \mathbf {X} \mathbf {w} \right\| _ {2} ^ {2}}{N} \geq \alpha \left\| \mathbf {w} \right\| _ {2} ^ {2} - \rho \frac {\log d}{N} \left\| \mathbf {w} \right\| _ {1} ^ {2}, \quad \forall \mathbf {w} \in \mathbb {R} ^ {d}. \tag {38}
$$
Further,  $\lambda_{\mathrm{max}}(\mathbf{X}^\top \mathbf{X} / N) \leq \beta$ .
2. The data  $(\mathbf{X},\mathbf{y})$  is "approximately generated from a  $s$ -sparse linear model": There exists a  $\mathbf{w}_{\star}\in \mathbb{R}^{d}$  such that  $\| \mathbf{w}_{\star}\|_{2}\leq B_{w}^{\star}$ ,  $\| \mathbf{w}_{\star}\|_{0}\leq s$  and for the residue  $\varepsilon = \mathbf{y} - \mathbf{X}\mathbf{w}_{\star}$ ,
$$
\left\| \mathbf {X} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {\infty} \leq \frac {1}{2} N \lambda_ {N}.
$$
3. It holds that  $N \geq 32\frac{\rho}{\alpha} \cdot s\log d$  (i.e.  $32\omega_{N} \leq 1$ ).