SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} D _ {h _ {k}} (\beta , \beta_ {k}) - D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) = \left[ h _ {k} (\beta) - \left(h _ {k} (\beta_ {k}) + \langle \nabla h _ {k} (\beta_ {k}), \beta - \beta_ {k} \rangle\right) \right] \\ \left. - \left[ h _ {k + 1} (\beta) - \left(h _ {k + 1} \left(\beta_ {k + 1}\right) + \langle \nabla h _ {k + 1} \left(\beta_ {k + 1}\right), \beta - \beta_ {k + 1} \rangle\right) \right] \right. \\ = h _ {k} (\beta) - h _ {k + 1} (\beta) + \left\langle \nabla h _ {k + 1} \left(\beta_ {k + 1}\right) - \nabla h _ {k} \left(\beta_ {k}\right), \beta - \beta_ {k + 1} \right\rangle \\ + h _ {k + 1} \left(\beta_ {k + 1}\right) - \left[ h _ {k} \left(\beta_ {k}\right) + \left\langle \nabla h _ {k} \left(\beta_ {k}\right), \beta_ {k + 1} - \beta_ {k} \right\rangle \right] \\ = h _ {k} (\beta) - h _ {k + 1} (\beta) + \left\langle \nabla h _ {k + 1} \left(\beta_ {k + 1}\right) - \nabla h _ {k} \left(\beta_ {k}\right), \beta - \beta_ {k + 1} \right\rangle \\ + h _ {k + 1} \left(\beta_ {k + 1}\right) - h _ {k} \left(\beta_ {k + 1}\right) + D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right). \\ \end{array}
$$
Rearranging and plugging in our mirror update we obtain that for all  $\beta$ :
$$
\begin{array}{l} \gamma_ {k} \langle \nabla \mathcal {L} (\beta_ {k}), \beta_ {k + 1} - \beta \rangle = D _ {h _ {k}} (\beta , \beta_ {k}) - D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) \\ - D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right) - \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right) + \left(h _ {k + 1} - h _ {k}\right) (\beta). \\ \end{array}
$$
From the convexity of  $\mathcal{L}$  and its  $L_{r}$ -relative smoothness we also have that:
$$
\mathcal {L} \left(\beta_ {k + 1}\right) \leqslant \mathcal {L} (\beta) + \left\langle \nabla \mathcal {L} \left(\beta_ {k}\right), \beta_ {k + 1} - \beta \right\rangle + L _ {r} D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right),
$$
Finally:
$$
\begin{array}{l} \gamma_ {k} (\mathcal {L} (\beta_ {k + 1}) - \mathcal {L} (\beta)) \leqslant D _ {h _ {k}} (\beta , \beta_ {k}) - D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) - (1 - \gamma_ {k} L _ {r}) D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) \\ + (h _ {k + 1} - h _ {k}) (\beta) - (h _ {k + 1} - h _ {k}) (\beta_ {k + 1}). \\ \end{array}
$$
Note that in our setting, for any  $\beta$ ,  $k \mapsto h_k(\beta)$  is increasing. We can therefore write that:
$$
\gamma_ {k} (\mathcal {L} (\beta_ {k + 1}) - \mathcal {L} (\beta)) \leqslant D _ {h _ {k}} (\beta , \beta_ {k}) - D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) - (1 - \gamma_ {k} L _ {r}) D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) + (h _ {k + 1} - h _ {k}) (\beta).
$$
In particular, for  $\beta = \beta^{*}$
$$
\begin{array}{l} \gamma_ {k} \mathcal {L} (\beta_ {k + 1}) \leqslant D _ {h _ {k}} (\beta^ {*}, \beta_ {k}) - D _ {h _ {k + 1}} (\beta^ {*}, \beta_ {k + 1}) - (1 - \gamma_ {k} L) D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) + (h _ {k + 1} - h _ {k}) (\beta^ {*}) \\ - \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right) \\ \leqslant D _ {h _ {k}} \left(\beta^ {*}, \beta_ {k}\right) - D _ {h _ {k + 1}} \left(\beta^ {*}, \beta_ {k + 1}\right) - \left(1 - \gamma_ {k} L _ {r}\right) D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right) + \left(h _ {k + 1} - h _ {k}\right) \left(\beta^ {*}\right) \\ \end{array}
$$
and in  $\beta = \beta_{k}$
$$
\begin{array}{l} \gamma_ {k} \mathcal {L} (\beta_ {k + 1}) \leqslant \gamma_ {k} \mathcal {L} (\beta_ {k}) - D _ {h _ {k + 1}} (\beta_ {k}, \beta_ {k + 1}) - (1 - \gamma_ {k} L _ {r}) D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) + (h _ {k + 1} - h _ {k}) (\beta_ {k}) \\ - \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right) \\ \leqslant \gamma_ {k} \mathcal {L} (\beta_ {k}) - D _ {h _ {k + 1}} (\beta_ {k}, \beta_ {k + 1}) - (1 - \gamma_ {k} L _ {r}) D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) + (h _ {k + 1} - h _ {k}) (\beta_ {k}) \\ \end{array}
$$
Proof of Proposition 1. We apply Proposition 14 for  $\beta = \beta_{k}$ , with  $L_{r} = 4BL$  (using Lemma 6) and replacing  $\mathcal{L}$  by  $\mathcal{L}_{\mathcal{B}_k}$ , to obtain:
$$
\begin{array}{l} \gamma_ {k} \left(\mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k + 1}\right) - \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right)\right) \leqslant - D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) - \left(1 - \gamma_ {k} L _ {r}\right) D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right) \\ + \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k}\right) - \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right), \\ \end{array}
$$
and thus, taking the mean wrt  $\mathcal{B}_k$
$$
\begin{array}{l} \gamma_ {k} \left(\mathbb {E} _ {\mathcal {B} _ {k}} \mathcal {L} \left(\beta_ {k + 1}\right) - \mathcal {L} \left(\beta_ {k}\right)\right) \leqslant - \mathbb {E} _ {\mathcal {B} _ {k}} D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) - \left(1 - \gamma_ {k} L _ {r}\right) \mathbb {E} _ {\mathcal {B} _ {k}} D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right) \\ + \mathbb {E} _ {\mathcal {B} _ {k}} \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k}\right) - \mathbb {E} _ {\mathcal {B} _ {k}} \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right) \\ \leqslant - \left(1 - \gamma_ {k} L _ {r}\right) \mathbb {E} _ {\mathcal {B} _ {k}} D _ {h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right) \\ + \mathbb {E} _ {\mathcal {B} _ {k}} \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k}\right) - \mathbb {E} _ {\mathcal {B} _ {k}} \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right). \\ \end{array}
$$
First, as in the proof of Proposition 10, using the fact that  $h_k$  is  $\ln(1 / \alpha_k)$  smooth,
$$
\begin{array}{l} D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) \geqslant \frac {1}{2 \ln (1 / \alpha_ {k})} \| \nabla h _ {k} (\beta_ {k}) - \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) - \nabla h _ {k} (\beta_ {k}) + \nabla h _ {k + 1} (\beta_ {k + 1}) - \nabla h _ {k} (\beta_ {k + 1}) \| _ {2} ^ {2} \\ \geqslant - \frac {1}{2 \ln \left(1 / \alpha_ {k}\right)} \| \nabla h _ {k} (\beta_ {k}) - \nabla h _ {k + 1} (\beta_ {k}) \| _ {2} ^ {2} + \frac {1}{4 \ln \left(1 / \alpha_ {k}\right)} \| \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \| _ {2} ^ {2}, \\ \end{array}
$$
$$
\mathbb {E} D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) \geqslant \mathbb {E} \left[ - \frac {1}{2 \ln (1 / \alpha_ {k})} \| \nabla h _ {k} (\beta_ {k}) - \nabla h _ {k + 1} (\beta_ {k}) \| _ {2} ^ {2} + \frac {\lambda_ {b}}{2 \ln (1 / \alpha_ {k})} \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B}} (\beta_ {k}) \right].
$$
Now, we apply Lemma 7 assuming that  $\| \beta^{\star}\|_{\infty},\| \beta_{k + 1}\|_{\infty}\leqslant B$  (which is satisfied since we are under the assumption of Theorem 2):
$$
\left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k}\right) - \left(h _ {k + 1} - h _ {k}\right) \left(\beta^ {\star}\right) \leqslant 2 4 B L \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right).
$$
Using  $|\nabla h_k(\beta) - \nabla h_{k + 1}(\beta)|\leqslant 2\delta_k$  where  $\delta_{k} = q(\gamma_{k}\nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k}))$  as in Proposition 10, we have:
$$
\mathbb {E} \| \nabla h _ {k} (\beta_ {k}) - \nabla h _ {k + 1} (\beta_ {k}) \| _ {2} ^ {2} \leqslant 1 6 B \gamma_ {k} ^ {2} \mathbb {E} \| \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \| ^ {2} \leqslant 3 2 B L \gamma_ {k} ^ {2} \mathbb {E} \mathcal {L} (\beta_ {k}).
$$
Wrapping everything together,
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} \left(\beta_ {k + 1}\right) - \mathcal {L} \left(\beta_ {k}\right) \right] \leqslant - (1 - \gamma_ {k} 4 B L) \frac {\lambda_ {b}}{2 \ln \left(1 / \alpha_ {k}\right)} \gamma_ {k} \mathbb {E} \mathcal {L} \left(\beta_ {k}\right) \\ + \left(\gamma_ {k} ^ {2} (1 - 4 \gamma_ {k} B L) 2 4 B L + \frac {3 2 B L}{\ln \left(1 / \alpha_ {k}\right)}\right) \gamma_ {k} ^ {2} \mathbb {E} \mathcal {L} (\beta_ {k}). \\ \end{array}
$$
Thus, for  $\gamma_{k} \leqslant \frac{c'}{LB\ln(1 / (\min_{i}\alpha_{k,i}))}$ , we have the first part of Proposition 1.
Using Lemma 3, we then have:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \beta_ {k} - \beta_ {\alpha_ {k}} ^ {\star} \right\| ^ {2} \right] = \mathbb {E} \left[ \mathbf {1} _ {\left\{\mathcal {L} \left(\beta_ {k}\right) \leqslant \frac {1}{2 \lambda_ {\max }} \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {2} \right\}} \left\| \beta_ {k} - \beta_ {\alpha_ {k}} ^ {\star} \right\| ^ {2} \right] \\ + \mathbb {E} \left[ \mathbf {1} _ {\left\{\mathcal {L} \left(\beta_ {k}\right) > \frac {1}{2 \lambda_ {\max }} \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {2} \right\}} \left\| \beta_ {k} - \beta_ {\alpha_ {k}} ^ {\star} \right\| ^ {2} \right] \\ \leqslant \mathbb {E} \left[ \mathbf {1} _ {\left\{\mathcal {L} \left(\beta_ {k}\right) \leqslant \frac {1}{2 \lambda_ {\max }} \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {2} \right\}} 2 B \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {- 1} \mathcal {L} \left(\beta_ {k}\right) \right] \\ + \mathbb {P} \left(\mathcal {L} \left(\beta_ {k}\right) > \frac {1}{2 \lambda_ {\max }} \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {2}\right) \times 4 B ^ {2} \\ \leqslant 2 B \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {- 1} \mathbb {E} \left[ \mathcal {L} \left(\beta_ {k}\right) \right] \\ + \frac {\mathbb {E} [ \mathcal {L} (\beta_ {k}) ]}{\frac {1}{2 \lambda_ {\max}} (\alpha^ {2} \lambda_ {\min} ^ {+}) ^ {2}} \times 4 B ^ {2} \\ = 2 B \left(\alpha^ {2} \lambda_ {\min } ^ {+}\right) ^ {- 1} \left(1 + \frac {4 B \lambda_ {\max }}{\alpha^ {2} \lambda_ {\min } ^ {+}}\right) \mathbb {E} \left[ \mathcal {L} \left(\beta_ {k}\right) \right]. \\ \end{array}
$$
G Proof of miscellaneous results mentioned in the main text
In this section, we provide proofs for results mentioned in the main text and that are not directly directed to the proof of Theorem 3.
G.1 Proof of Proposition 3 and the sum of the losses
We start by proving the following proposition, present as is in the first 9 pages of this paper. We then continue with upper and lower bounds (of similar magnitude) on the sum of the losses.
Proposition 3. Let  $\Lambda_b, \lambda_b > 0^5$  be the largest and smallest values, respectively, such that  $\lambda_bH \preceq \mathbb{E}_{\mathcal{B}}[H_{\mathcal{B}}^2] \preceq \Lambda_bH$ . For any stepsize  $\gamma > 0$  satisfying  $\gamma \leqslant \frac{c}{BL}$  (as in Theorem 2), initialisation  $\alpha \mathbf{1}$  and batch size  $b \in [n]$ , the magnitude of the gain satisfies:
$$
\lambda_ {b} \gamma^ {2} \sum_ {k} \mathbb {E L} (\beta_ {k}) \leqslant \mathbb {E} \left[ \| \operatorname {G a i n} _ {\gamma} \| _ {1} \right] \leqslant 2 \Lambda_ {b} \gamma^ {2} \sum_ {k} \mathbb {E L} (\beta_ {k}), \tag {10}
$$
Proof. From Lemma 5, for all  $-1/2 \leqslant x \leqslant 1/2$ , it holds that  $x^2 \leqslant q(x) \leqslant 2x^2$ . We have, using  $\|\gamma_k \nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_{\infty} \leqslant 1/2$  (which holds under the stepsize assumption):
$$
\begin{array}{l} \mathbb {E} \| \mathrm {G a i n} _ {\gamma} \| _ {1} = - \mathbb {E} \sum_ {i} \ln \left(\frac {\alpha_ {\infty , i}}{\alpha}\right) \\ = \sum_ {\ell <   \infty} \sum_ {i} \mathbb {E} q \left(\gamma_ {\ell} \nabla_ {i} \mathcal {L} _ {\mathcal {B} _ {\ell}} (\beta_ {\ell})\right) \\ \leqslant 2 \sum_ {\ell <   \infty} \sum_ {i} \mathbb {E} \left(\gamma_ {\ell} \nabla_ {i} \mathcal {L} _ {\mathcal {B} _ {\ell}} (\beta_ {\ell})\right) ^ {2} \\ = \sum_ {\ell <   \infty} \gamma_ {\ell} ^ {2} \mathbb {E} \| \nabla \mathcal {L} _ {\mathcal {B} _ {\ell}} (\beta_ {\ell}) \| _ {2} ^ {2} \\ \leqslant 4 \Lambda_ {b} \sum_ {\ell <   \infty} \gamma_ {\ell} ^ {2} \mathbb {E} \mathcal {L} _ {\mathcal {B} _ {\ell}} (\beta_ {\ell}), \\ \end{array}
$$
since  $\mathbb{E}\| \nabla \mathcal{L}_{\mathcal{B}_\ell}(\beta_\ell)\| _2^2\leqslant 2\Lambda_b\mathcal{L}_{\mathcal{B}_\ell}(\beta_\ell)$ . For the left hand side we use  $q(x)\geqslant x^{2}$  for  $|x|\leqslant 1 / 2$  and  $\mathbb{E}\| \nabla \mathcal{L}_{\mathcal{B}_k}(\beta_\ell)\| _2^2\geqslant 2\lambda_b\mathcal{L}_{\mathcal{B}_k}(\beta_\ell)$ . Finally, since  $\mathcal{B}_{\ell}$  independent from  $\beta_{\ell}$ , we have  $\mathbb{E}\mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell}) = \mathbb{E}\mathcal{L}(\beta_{\ell})$ .
Proposition 15. For stepsizes  $\gamma_k \equiv \gamma \leqslant \frac{c}{LB}$  (as in Theorem 2), we have:
$$
\sum_ {k \geqslant 0} \gamma^ {2} \mathbb {E} \mathcal {L} (\beta_ {k}) = \Theta \left(\gamma \| \beta^ {\star} \| _ {1} \ln (1 / \alpha)\right).
$$
Proof. We first lower bound  $\sum_{k < \infty} \gamma_k^2 \mathcal{L}_{\mathcal{B}_k}(\beta_k)$ . We have the following equality, that holds for any  $k$ :
$$
\begin{array}{l} D _ {h _ {k + 1}} \left(\beta^ {\star}, \beta_ {k + 1}\right) = D _ {h _ {k}} \left(\beta^ {\star}, \beta_ {k}\right) - 2 \gamma \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right) + D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) \\ + \left(h _ {k} - h _ {k + 1}\right) \left(\beta_ {k}\right) - \left(h _ {k} - h _ {k + 1}\right) \left(\beta^ {\star}\right), \\ \end{array}
$$
leading to, by summing for  $k \in \mathbb{N}$ :
$$
\sum_ {k <   \infty} 2 \gamma \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) = D _ {h _ {0}} (\beta^ {\star}, \beta_ {0}) - \lim  _ {k \to \infty} D _ {h _ {k}} (\beta^ {\star}, \beta_ {k}) + \sum_ {k <   \infty} D _ {h _ {k + 1}} (\beta_ {k}, \beta_ {k + 1}) + \sum_ {k <   \infty} \left(h _ {k} - h _ {k + 1}\right) (\beta_ {k}) - \left(h _ {k} - h _ {k + 1}\right) (\beta^ {\star}).
$$
First, since  $h_k \to h_\infty$ ,  $\beta_k \to \beta_\infty$ , we have  $\lim_{k \to \infty} D_{h_k}(\beta^\star, \beta_k) = 0$ . Then,  $D_{h_{k+1}}(\beta_k, \beta_{k+1}) \geqslant 0$ .
Finally,  $\left| \left( h_{k} - h_{k + 1} \right) (\beta_{k}) - \left( h_{k} - h_{k + 1} \right) (\beta^{\star}) \right| \leqslant 16BL_{2}\gamma^{2}\mathcal{L}_{\mathcal{B}_{k}}(\beta_{k})$ . Hence:
$$
\sum_ {k <   \infty} 2 \gamma (1 + 1 6 \gamma B L _ {2}) \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \geqslant D _ {h _ {0}} (\beta^ {\star}, \beta_ {0}),
$$
$^5\Lambda_b, \lambda_b > 0$  are data-dependent constants; for  $b = n$ , we have  $(\lambda_n, \Lambda_n) = (\lambda_{\min}^+(H), \lambda_{\max}(H))$  where  $\lambda_{\min}^+(H)$  is the smallest non-null eigenvalue of  $H$ ; for  $b = 1$ , we have  $\min_i \|x_i\|_2^2 \leqslant \lambda_1 \leqslant \Lambda_1 \leqslant \max_i \|x_i\|_2^2$ .
and thus  $\sum_{k < \infty} \gamma \mathcal{L}_{\mathcal{B}_k}(\beta_k) \geqslant D_{h_0}(\beta^\star, \beta_0) / 4$  for  $\gamma \leqslant c / (BL)$  (with  $c \geqslant 16$ ). This gives the RHS inequality. The LHS is a direct consequence of bounds proved in previous subsections.
$$
\gamma^ {2} \sum_ {k} \mathcal {L} (\beta_ {k}) = \Theta \left(\gamma D _ {h _ {0}} (\beta^ {\star}, \beta_ {0})\right).
$$
Noting that  $D_{h_0}(\beta^\star, \beta_0) = h_0(\beta^\star) = \Theta\left(\ln(1 / \alpha)\|\beta^\star\|_1\right)$  concludes the proof.
G.2  $\tilde{\beta}_0$  is negligible
In the following proposition we show that  $\tilde{\beta}_0$  is close to  $\mathbf{0}$  and therefore one should think of the implicit regularization problem as  $\beta_{\infty}^{\star} = \mathrm{argmin}_{\beta^{\star}\in S}\psi_{\alpha_{\infty}}(\beta^{\star})$
Proposition 16. Under the assumptions of Theorem 2,
$$
| \tilde {\beta} _ {0} | \leqslant \alpha^ {2},
$$
where the inequality must be understood coordinate-wise.
$$
\begin{array}{l} | \tilde {\beta} _ {0} | = \frac {1}{2} | \alpha_ {+} ^ {2} - \alpha_ {-} ^ {2} | \\ = \frac {1}{2} \alpha^ {2} \left| \exp \left(- \sum_ {k} q _ {+} \left(\gamma_ {k} \nabla \mathcal {L} \left(\beta_ {k}\right)\right) - \exp \left(- \sum_ {k} q _ {-} \left(\gamma_ {k} \nabla \mathcal {L} \left(\beta_ {k}\right)\right) \right. \right. \right. \\ \leqslant \alpha^ {2}, \\ \end{array}
$$
where the inequality is because  $q_{+}(\gamma_{k}\nabla \mathcal{L}(\beta_{k}))\geqslant 0,q_{-}(\gamma_{k}\nabla \mathcal{L}(\beta_{k}))\geqslant 0$  for all  $k$
G.3 Impact of stochasticity and linear scaling rule
Proposition 17. With probability  $1 - 2ne^{-d / 16} - 3 / n^2$  over the  $x_{i} \sim_{\mathrm{iid}} \mathcal{N}(0, \sigma^{2}I_{d})$ ,  $c_{1}\frac{d\sigma^{2}}{b}(1 + o(1)) \leqslant \lambda_{b} \leqslant \Lambda_{b} \leqslant c_{2}\frac{d\sigma^{2}}{b}(1 + o(1))$ ,
so that under these assumptions,
$$
\sum_ {k} \gamma_ {k} \mathbb {E} \mathcal {L} (\beta_ {k}) = \Theta \left(\frac {\gamma}{b} \sigma^ {2} \| \beta^ {\star} \| _ {1} \ln (1 / \alpha)\right).
$$
Proof. The bound on  $\lambda_b, \Lambda_b$  is a direct consequence of the concentration bound provided in Lemma 13.
G.4 (Stochastic) gradients at the initialisation
To understand the behaviour and the effects of the stochasticity and the stepsize on the shape of  $\mathrm{Gain}_{\gamma}$ , we analyse a noiseless sparse recovery problem under the following standard assumption 2 [10] and as common in the sparse recovery literature, we make the following assumption 3 on the inputs.
Assumption 2. There exists an  $s$ -sparse ground truth vector  $\beta_{\mathrm{sparse}}^{\star}$  where  $s$  verifies  $n = \Omega(s \ln(d))$ , such that  $y_i = \langle \beta_{\mathrm{sparse}}^{\star}, x_i \rangle$  for all  $i \in [n]$ .
Assumption 3. There exists  $\delta, c_1, c_2 > 0$  such that for all  $s$ -sparse vectors  $\beta$ , there exists  $\varepsilon \in \mathbb{R}^d$  such that  $(X^\top X)\beta = \beta + \varepsilon$  where  $\| \varepsilon \|_{\infty} \leqslant \delta \| \beta \|_2$  and  $c_1 \| \beta \|_2^2 \mathbf{1} \leqslant \frac{1}{n} \sum_i x_i^2 \langle x_i, \beta \rangle^2 \leqslant c_2 \| \beta \|_2^2 \mathbf{1}$ .
The first part of Assumption 3 closely resembles the classical restricted isometry property (RIP) and is relevant for GD while the second part is relevant for SGD. Such an assumption is not restrictive and holds with high probability for Gaussian inputs  $\mathcal{N}(0, \sigma^2 I_d)$  (see Lemma 10 in Appendix).
Based on the claim above, we analyse the shape of the (stochastic) gradient at initialisation. For GD and SGD, it respectively writes, where  $g_0 = \nabla \mathcal{L}_{i_0}(\beta_0)^2$ ,  $i_0 \sim \mathrm{Unif}([n])$ :
$$
\nabla \mathcal {L} (\beta_ {0}) ^ {2} = [ X ^ {\top} X \beta^ {\star} ] ^ {2}, \mathbb {E} _ {i _ {0}} [ g _ {0} ] = \frac {1}{n} \sum_ {i} x _ {i} ^ {2} \langle x _ {i}, \beta^ {\star} \rangle^ {2}.
$$
The following lemma then shows that while the initial stochastic gradients of SGD are homogeneous, it is not the case for that of GD.
Proposition 18. Under Assumption 3, the squared full batch gradient and the expected stochastic gradient at initialisation satisfy, for some  $\varepsilon$  verifying  $\| \varepsilon \|_{\infty} \ll \left\| \beta_{\mathrm{sparse}}^{\star} \right\|_{\infty}^{2}$ :
$$
\nabla \mathcal {L} \left(\beta_ {0}\right) ^ {2} = \left(\beta_ {\text {s p a r s e}} ^ {\star}\right) ^ {2} + \varepsilon , \tag {25}
$$
$$
\mathbb {E} _ {i _ {0}} \left[ \nabla \mathcal {L} _ {i _ {0}} \left(\beta_ {0}\right) ^ {2} \right] = \Theta \left(\| \beta^ {\star} \| _ {2} ^ {2} \mathbf {1}\right). \tag {26}
$$
Proof of Proposition 18. Under Assumption 3, we have using:
$$
\begin{array}{l} \nabla \mathcal {L} (\beta_ {0}) ^ {2} = (X ^ {\top} X \beta_ {\text {s p a r s e}} ^ {\star}) \\ = \left(\beta_ {\text {s p a r s e}} ^ {\star} + \varepsilon\right) ^ {2} \\ = \beta_ {\mathrm {s p a r s e}} ^ {\star} ^ {2} + \varepsilon^ {2} + 2 \varepsilon \beta_ {\mathrm {s p a r s e}} ^ {\star}. \\ \end{array}
$$
We have  $\left\| \varepsilon^2 + 2\varepsilon \beta_{\mathrm{sparse}}^\star \right\|_\infty \leqslant \left\| \varepsilon \right\|_\infty^2 + 2\left\| \varepsilon \right\|_\infty \left\| \beta_{\mathrm{sparse}}^\star \right\|_\infty$ , and we conclude by using  $\left\| \varepsilon \right\|_\infty \leqslant \delta \left\| \beta_{\mathrm{sparse}}^\star \right\|_2$ .
$$
\mathbb {E} _ {i \sim \mathrm {U n i f} ([ n ])} [ \nabla \mathcal {L} _ {i} (\beta_ {0}) ^ {2} ] = \frac {1}{n} x _ {i} ^ {2} \langle x _ {i}, \beta_ {\mathrm {s p a r s e}} ^ {\star} \rangle ,
$$
and we conclude using Assumption 3.
Proof of Proposition 4. The proof proceeds as that of Proposition 18.
G.5 Convergence of  $\alpha_{\infty}$  and  $\tilde{\beta}_0$  for  $\gamma \rightarrow 0$
Proposition 19. Let  $\beta_0(\gamma),\alpha_{\infty}(\gamma)$  be as defined in Theorem 1, for constant stepsizes  $\gamma_{k}\equiv \gamma$  . We have:
$$
\tilde {\beta} _ {0} (\gamma) \rightarrow 0, \quad \boldsymbol {\alpha} _ {\infty} \rightarrow \boldsymbol {\alpha} \mathbf {1},
$$
when  $\gamma \to 0$
Proof. We have, as proved previously, that
$$
\begin{array}{l} \left\| \sum_ {k} \gamma^ {2} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) ^ {2} \right\| _ {1} \leqslant \sum_ {k} \gamma^ {2} \left\| \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) ^ {2} \right\| _ {1} \\ = \sum_ {k} \gamma^ {2} \| \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \| _ {2} ^ {2} \\ \leqslant 2 L \gamma^ {2} \sum_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \\ \leqslant 2 L \gamma D _ {h _ {0}} \left(\beta^ {\star}, \beta_ {0}\right), \\ \end{array}
$$
for  $\gamma \leqslant \frac{c}{BL}$ . Thus,  $\sum_{k} \gamma^{2} \nabla \mathcal{L}_{\mathcal{B}_{k}} (\beta_{k})^{2} \to 0$  as  $\gamma \to 0$  (note that  $\beta_{k}$  implicitly depends on  $\gamma$ , so that this result is not immediate).
Then, for  $\gamma \leqslant \frac{c}{LB}$
$$
\left\| \ln \left(\boldsymbol {\alpha} _ {\infty} ^ {2} / \alpha^ {2}\right) \right\| _ {1} \leqslant \sum_ {k} \| q (\gamma \mathcal {L} (\beta_ {k}) \| _ {1} \leqslant 2 \sum_ {k} \gamma^ {2} \| \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) ^ {2} \| _ {1},
$$
which tends to 0 as  $\gamma \to 0$ . Similarly,  $\left\| \ln (\alpha_{+, \infty}^2 / \alpha^2) \right\|_1 \to 0$  and  $\left\| \ln (\alpha_{-, \infty}^2 / \alpha^2) \right\|_1 \to 0$  as  $\gamma \to 0$ , leading to  $\tilde{\beta}_0(\gamma) \to 0$  as  $\gamma \to 0$ .
H Technical lemmas
In this section we present a few technical lemmas, used and referred to throughout the proof of ??.
Lemma 4. Let  $\alpha_{+},\alpha_{-} > 0$  and  $x\in \mathbb{R}$  , and  $\beta = \alpha_{+}^{2}e^{x} - \alpha_{-}^{2}e^{-x}$  . We have:
$$
\operatorname {a r c s i n h} \left(\frac {\beta}{2 \alpha_ {+} \alpha_ {-}}\right) = x + \ln \left(\frac {\alpha_ {+}}{\alpha_ {-}}\right) = x + \operatorname {a r c s i n h} \left(\frac {\alpha_ {+} ^ {2} - \alpha_ {-} ^ {2}}{2 \alpha_ {+} \alpha_ {-}}\right).
$$
$$
\begin{array}{l} \frac {\beta}{2 \alpha_ {+} \alpha_ {-}} = \frac {1}{2} \left(\frac {\alpha_ {+}}{\alpha^ {-}} e ^ {x} - \left(\frac {\alpha_ {+}}{\alpha^ {-}}\right) ^ {- 1} e ^ {- x}\right) \\ = \frac {e ^ {x + \ln \left(\alpha_ {+} / \alpha_ {-}\right)} - e ^ {- x - \ln \left(\alpha_ {+} / \alpha_ {-}\right)}}{2} \\ = \sinh (x + \ln (\alpha_ {+} / \alpha_ {-})) , \\ \end{array}
$$
hence the result by taking the  $\operatorname{arcsinh}$  of both sides. Note also that we have  $\ln (\alpha_{+} / \alpha_{-}) = \operatorname{arcsinh}(\frac{\alpha_{+}^{2} - \alpha_{-}^{2}}{2\alpha_{+}\alpha_{-}})$ .
Lemma 5. If  $|x| \leqslant 1/2$  then  $x^2 \leqslant q(x) \leqslant 2x^2$ .
Lemma 6. On the  $\ell_{\infty}$  ball of radius  $B$ , the quadratic loss function  $\beta \mapsto \mathcal{L}(\beta)$  is  $4\lambda_{\max}\max(B, \alpha^2)$ -relatively smooth w.r.t all the  $h_k$ 's.
$$
\nabla^ {2} h _ {k} (\beta) = \mathrm {d i a g} \left(\frac {1}{2 \sqrt {\alpha_ {k} ^ {4} + \beta^ {2}}}\right) \succeq \mathrm {d i a g} \left(\frac {1}{2 \sqrt {\alpha^ {4} + \beta^ {2}}}\right),
$$
since  $\alpha_{k} \leqslant \alpha$  component-wise. Thus,  $\nabla^2 h_k(\beta) \succeq \frac{1}{2} \min \left( \min_{1 \leqslant i \leqslant d} \frac{1}{2|\beta_i|}, \frac{1}{2\alpha^2} \right) I_d = \frac{1}{\max(4\|\beta\|_{\infty}, 4\alpha^2)} I_d$ , and  $h_k$  is  $\frac{1}{\max(4B, 4\alpha^2)}$ -strongly convex on the  $\ell^\infty$  norm of radius  $B$ . Since  $\mathcal{L}$  is  $\lambda_{\max}$ -smooth over  $\mathbb{R}^d$ , we have our result.
Lemma 7. For  $k\geqslant 0$  and for all  $\beta \in \mathbb{R}^d$