SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {k} \leqslant \sup _ {\| \beta \| _ {\infty} \leqslant B} \| \nabla \mathcal {L} (\beta) \| _ {\infty} + 8 \lambda_ {\max } \sqrt {\| \boldsymbol {\alpha} \| _ {\infty} ^ {4} + B ^ {2}} \leqslant L B + 8 \lambda_ {\max } \sqrt {\| \boldsymbol {\alpha} \| _ {\infty} ^ {4} + B ^ {2}} \leqslant 1 0 L B,
$$
for  $B\geqslant \| \pmb {\alpha}\|_{\infty}^{2}$
We finally prove the bound on  $\| \beta_k\|_{\infty}$  independent of  $\alpha$  for a uniform initialisation  $\alpha = \alpha \mathbf{1}$ , using the monotonic property of  $\mathcal{L}$ .
Proposition 13. Assume that  $b = n$  (full batch setting). There exists some  $B = \mathcal{O}(\| \beta^{\star}\|_{1})$  such that for stepsizes  $\gamma_k\leqslant \frac{c}{BL}$ , we have  $\| \beta_k\|_{\infty}\leqslant B$  for all  $k\geqslant 0$ .
Proof. In this proof, we first let  $B$  be a bound on the iterates. Tuning stepsizes using this bound, we prove that the iterates are bounded by a some  $B' = \mathcal{O}(\|\beta^{\star}\|_1)$ . Finally, we conclude by using the continuity of the iterates (at a finite horizon) that this explicit bound can be used to tune the step sizes.
Writing the mirror descent with varying potentials, we have, since  $\nabla h_0(\beta_0) = 0$
$$
\nabla h _ {k} (\beta_ {k}) = - \sum_ {\ell <   k} \gamma_ {\ell} \nabla \mathcal {L} (\beta_ {\ell}),
$$
leading to, by convexity of  $h_k$ :
$$
h _ {k} \left(\beta_ {k}\right) - h _ {k} \left(\beta^ {\star}\right) \leqslant \langle \nabla h _ {k} \left(\beta_ {k}\right), \beta_ {k} - \beta^ {\star} \rangle = - \sum_ {\ell <   k} \langle \gamma_ {\ell} \nabla \mathcal {L} \left(\beta_ {\ell}\right), \beta_ {k} - \beta^ {\star} \rangle .
$$
We then write, using  $\nabla \mathcal{L}(\beta) = H(\beta -\beta^{\star})$  for  $H = XX^{\top}$ , that  $-\sum_{\ell < k}\langle \gamma_{\ell}\nabla \mathcal{L}(\beta_{\ell}),\beta_{k} - \beta^{\star}\rangle = -\sum_{\ell < k}\gamma_{\ell}\langle X^{\top}(\bar{\beta}_{k} - \beta^{\star}),X^{\top}(\beta_{k} - \beta^{\star})\rangle \leqslant \sum_{\ell < k}\gamma_{\ell}\sqrt{\mathcal{L}(\bar{\beta}_{k})\mathcal{L}(\beta_{k})}$ , leading to:
$$
h _ {k} (\beta_ {k}) - h _ {k} (\beta^ {\star}) \leqslant 2 \sqrt {\sum_ {\ell <   k} \gamma_ {\ell} \mathcal {L} (\bar {\beta} _ {k}) \sum_ {\ell <   k} \gamma_ {\ell} \mathcal {L} (\beta_ {k})} \leqslant 2 \sum_ {\ell <   k} \gamma_ {\ell} \mathcal {L} (\bar {\beta} _ {k}) \leqslant 2 D _ {h _ {0}} (\beta^ {\star}, \beta^ {0}),
$$
where the last inequality holds provided that  $\gamma_{k}\leqslant \frac{1}{CLB}$ . Thus,
$$
\psi_ {\boldsymbol {\alpha} _ {k}} (\beta_ {k}) \leqslant \psi_ {\boldsymbol {\alpha} _ {k}} (\beta^ {\star}) + 2 \psi_ {\boldsymbol {\alpha} _ {0}} (\beta^ {\star}) + \langle \phi_ {k}, \beta_ {k} - \beta^ {\star} \rangle .
$$
Then,  $\langle \phi_k,\beta_k - \beta^\star \rangle \leqslant \| \phi_k\| _1\| \beta_k - \beta^\star \|_\infty$  and  $\| \phi_k\| _1\leqslant C\lambda_{\max}\sum_{k <   K}\gamma_k^2\mathcal{L}(\beta^k)\leqslant$ $C\lambda_{\mathrm{max}}\gamma_{\mathrm{max}}h_0(\beta^\star)$  . Then, using
$$
\| \beta \| _ {\infty} - \frac {1}{\ln (1 / \alpha^ {2})} \leqslant \frac {\psi_ {\alpha} (\beta)}{\ln (1 / \alpha^ {2})} \leqslant \| \beta \| _ {1} \big (1 + \frac {\ln (\| \beta \| _ {1} + \alpha^ {2})}{\ln (1 / \alpha^ {2})} \big),
$$
$$
\begin{array}{l} \| \beta_ {k} \| _ {\infty} \leqslant \frac {1}{\ln (1 / \alpha^ {2})} + \| \beta^ {\star} \| _ {1} \left(1 + \frac {\ln (\| \beta^ {\star} \| _ {1} + \alpha^ {2})}{\ln (1 / \alpha^ {2})}\right) + \| \beta^ {\star} \| _ {1} \left(1 + \frac {\ln (\| \beta^ {\star} \| _ {1} + \alpha^ {2})}{\ln (1 / \alpha^ {2})}\right) \\ + B _ {0} C \lambda_ {\max } \gamma_ {\max } h _ {0} \left(\beta^ {\star}\right) / \ln \left(1 / \alpha^ {2}\right) \\ \leqslant R + B _ {0} C \lambda_ {\max } \gamma_ {\max } h _ {0} \left(\beta^ {\star}\right) / \ln \left(1 / \alpha^ {2}\right), \\ \end{array}
$$
where  $R = \mathcal{O}(\| \beta^{\star}\|_{1})$  is independent of  $\alpha$ . Hence, since  $B_0 = \sup_{k < \infty}\|\beta_k\|_{\infty} < \infty$ , we have:
$$
B _ {0} \left(1 - C \lambda_ {\max } \gamma_ {\max } h _ {0} \left(\beta^ {\star}\right) / \ln \left(1 / \alpha^ {2}\right)\right) \leqslant R \Rightarrow B _ {0} \leqslant 2 R,
$$
provided that  $\gamma_{\mathrm{max}} \leqslant 1 / (2C\lambda_{\mathrm{max}}h_0(\beta^{\star}) / \ln(1 / \alpha^2))$  (note that  $h_0(\beta^{\star}) / \ln(1 / \alpha^2)$  is independent of  $\alpha^2$ ).
Hence, if for all  $k$  we have  $\gamma_k \leqslant \frac{1}{C'LB}$  where  $B$  bounds all  $\|\beta_k\|_{\infty}$ , we have  $\|\beta_k\|_{\infty} \leqslant 2R$  for all  $k$ , where  $R = \mathcal{O}(\|\beta^{\star}\|_{1})$  is independent of  $\alpha$  and stepsizes  $\gamma_k$ .
Let  $K > 0$  be fixed, and
$$
\bar{\gamma} = \inf \left\{\gamma >0\quad \text{s.t.}\quad \sup_{k\leqslant K}\| \beta_{k}\|_{\infty} > 2R\right\} .
$$
For  $\gamma \geqslant 0$  a constant stepsize, let
$$
\varphi (\gamma) = \sup  _ {k \leqslant K} \| \beta_ {k} \| _ {\infty},
$$
which is a continuous function of  $\gamma$ . For  $r > 0$ , let  $\gamma_r = \frac{1}{C' L r}$ .
An important feature to notice is that if  $\gamma < \gamma_{r}$  and  $r$  bounds all  $\| \beta_k\|_{\infty}, k \leqslant K$ , then  $\varphi (\gamma) \leqslant R$ , as shown above. We will show that we have  $\bar{\gamma} \geqslant \gamma_{2R}$ . Reasoning by contradiction, if  $\bar{\gamma} < \gamma_{2R}$ : by continuity of  $\varphi$ , we have  $\varphi (\bar{\gamma}) \leqslant R$  and thus, there exists some small  $0 < \varepsilon < \gamma_{2R} - \bar{\gamma}$  such that for all  $\gamma \in [\bar{\gamma},\bar{\gamma} +\varepsilon]$ , we have  $\varphi (\bar{\gamma}) \leqslant 2R$ .
However, such  $\gamma$ 's verify both  $\varphi(\gamma) \leqslant 2R$  (since  $\gamma \in [\bar{\gamma}, \bar{\gamma} + \varepsilon]$  and by definition of  $\varepsilon$ ) and  $\gamma \leqslant \gamma_{2R}$  (by definition of  $\varepsilon$ ), and hence  $\varphi(\gamma) \leqslant R$ . This contradicts the infimum of  $\bar{\gamma}$ , and hence  $\bar{\gamma} \geqslant \gamma_{2R}$ . Thus, for  $\gamma \leqslant \gamma_{2R} = \frac{1}{2C'LR}$ , we have  $\| \beta_k \|_{\infty} \leqslant R$ .
F Proof of Theorem 1 and 2, and of Proposition 1
F.1 Proof of Theorem 1 and 2
We are now equipped to prove Theorem 1 and Theorem 2, condensed in the following Theorem.
Theorem 3. Let  $(u_{k},v_{k})_{k\geqslant 0}$  follow the mini-batch SGD recursion (3) initialised at  $u_0 = \sqrt{2}\alpha \in \mathbb{R}_{>0}^d$  and  $v_{0} = 0$ , and let  $(\beta_{k})_{k\geqslant 0} = (u_{k}\odot v_{k})_{k\geqslant 0}$ . There exists and explicit  $B > 0$  and a numerical constant  $c > 0$  such that:
1. For stepsizes satisfying  $\gamma_k \leqslant \frac{c}{LB}$ , the iterates satisfy  $\| \gamma_k \nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k) \|_{\infty} \leqslant 1$  and  $\| \beta_k \|_{\infty} \leqslant B$  for all  $k$ ;
2. For stepsizes satisfying  $\gamma_k \leqslant \frac{c}{LB}$ ,  $(\beta_k)_{k \geqslant 0}$  converges almost surely to some  $\beta_\infty^\star \in S$ ,
3. If  $(\beta_k)_k$  and the neurons  $(u_k, v_k)_k$  respectively converge to a model  $\beta_\infty^*$  and neurons  $(u_\infty, v_\infty)$  satisfying  $\beta_\infty^* \in S$  (and  $\beta_\infty^* = u_\infty \odot v_\infty$ ), then for almost all stepsizes (with respect to the Lebesgue measure), the limit  $\beta_\infty^*$  satisfies:
$$
\beta_ {\infty} ^ {\star} = \underset {\beta^ {\star} \in \mathcal {S}} {\operatorname {a r g m i n}} D _ {\psi_ {\alpha_ {\infty}}} (\beta^ {\star}, \tilde {\beta} _ {0}),
$$
for  $\alpha_{\infty} \in \mathbb{R}_{>0}^{d}$  and  $\tilde{\beta}_0 \in \mathbb{R}^d$  satisfying
$$
\boldsymbol {\alpha} _ {\infty} ^ {2} = \boldsymbol {\alpha} ^ {2} \odot \exp \left(- \sum_ {k = 0} ^ {\infty} q \left(\gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k})\right)\right),
$$
where  $q(x) = -\frac{1}{2}\ln \left(\left(1 - x^2\right)^2\right) \geqslant 0$  for  $|x| \leqslant \sqrt{2}$ , and  $\tilde{\beta}_0$  is a perturbation term equal to:
$$
\tilde {\beta} _ {0} = \frac {1}{2} \left(\boldsymbol {\alpha} _ {+} ^ {2} - \boldsymbol {\alpha} _ {-} ^ {2}\right),
$$
where,  $q_{\pm}(x) = \mp 2x - \ln ((1\mp x)^2)$ , and  $\pmb{\alpha}_{\pm}^{2} = \pmb{\alpha}^{2}\odot \exp (-\sum_{k = 0}^{\infty}q_{\pm}(\gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)))$
Proof. Point 1. The first point of the Theorem is a direct consequence of Corollary 1 and the bounds proved in appendix E.3.
Point 2. Then, for stepsizes  $\gamma_k \leqslant \frac{c}{LB}$ , using Proposition 8 for any interpolator  $\beta^* \in S$ :
$$
D _ {h _ {k + 1}} \left(\beta^ {\star}, \beta_ {k + 1}\right) \leqslant D _ {h _ {k}} \left(\beta^ {\star}, \beta_ {k}\right) - \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right). \tag {24}
$$
Hence, summing:
$$
\sum_ {k} \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \leqslant D _ {h _ {0}} (\beta^ {\star}, \beta_ {0}),
$$
so that the series converges.
Under our stepsize rule,  $\| \gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_{\infty}\leqslant \frac{1}{2}$ , leading to  $\| q(\gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_{\infty}\leqslant 3\|\gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_{\infty}^2$  by Lemma 5. Using  $\| \nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|^2\leqslant 2L_2\mathcal{L}_{\mathcal{B}_k}(\beta_k)$ , we have that  $\ln (\alpha_{\pm ,k})$ ,  $\ln (\alpha_k)$  all converge.
We now show that  $\sum_{k}\gamma_{k}\mathcal{L}(\beta_{k}) < \infty$ . We have:
$$
\sum_ {\ell <   k} \mathcal {L} (\beta_ {k}) = \sum_ {\ell <   k} \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) + M _ {k},
$$
where  $M_{k} = \sum_{\ell <  k}\gamma_{k}(\mathcal{L}(\beta_{k}) - \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k}))$  . We have that  $(M_{k})$  is a martingale with respect to the filtration  $(\mathcal{F}_k)$  defined as  $\mathcal{F}_k = \sigma (\beta_\ell ,\ell \leqslant k)$  . Using our upper-bound on  $\sum_{\ell <  k}\gamma_k\mathcal{L}_{\mathcal{B}_k}(\beta_k)$  , we have:
$$
M _ {k} \geqslant \sum_ {\ell <   k} \gamma_ {k} \mathcal {L} (\beta_ {k}) - \sum_ {\ell <   k} \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \geqslant - D _ {h _ {0}} (\beta^ {\star}, \beta_ {0}),
$$
and hence  $(M_{k})$  is a lower bounded martingale. Using Doob's first martingale convergence theorem (a lower bounded super-martingale converges almost surely, Doob [17]),  $(M_{k})$  converges almost surely. Consequently, since  $\sum_{\ell < k} \gamma_{k} \mathcal{L}(\beta_{k}) = \sum_{\ell < k} \gamma_{k} \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k}) + M_{k}$ , we have that  $\sum_{\ell < k} \gamma_{k} \mathcal{L}(\beta_{k})$  converges almost surely (the first term is upper bounded, the second converges almost surely).
We now prove the convergence of  $(\beta_{k})$ . Since it is a bounded sequence, let  $\beta_{\sigma (k)}$  be a convergent sub-sequence and let  $\beta_{\infty}^{\star}$  denote its limit:  $\beta_{\sigma (k)}\to \beta_{\infty}^{\star}$ .
Almost surely,  $\sum_{k}\gamma_{k}\mathcal{L}(\beta_{k}) < \infty$  and so  $\gamma_{k}\mathcal{L}(\beta_{k})\to 0$ , leading to  $\mathcal{L}(\beta_k)\to 0$  since the step sizes are lower bounded, so that  $\mathcal{L}(\beta_{\sigma (k)})\to 0$ , and hence  $\mathcal{L}(\beta_{\infty}^{\star}) = 0$ : this means that  $\beta_{\infty}^{\star}$  is an interpolator.
Since the quantities  $(\alpha_{k})_{k}$ ,  $(\alpha_{\pm ,k})_k$  and  $(\phi_k)_k$  converge almost surely to  $\alpha_{\infty}$ ,  $\alpha_{\pm}$  and  $\phi_{\infty}$ , we get that the potentials  $h_k$  uniformly converge to  $h_\infty = \psi_{\alpha_\infty} - \langle \phi_\infty, \cdot \rangle$  on all compact sets. Now notice that we can decompose  $\nabla h_{\infty}(\beta_{\infty}^{\star})$  as:
$$
\nabla h _ {\infty} (\beta_ {\infty} ^ {\star}) = \left(\nabla h _ {\infty} (\beta_ {\infty} ^ {\star}) - \nabla h _ {\infty} (\beta_ {\sigma (k)})\right) + \left(\nabla h _ {\infty} (\beta_ {\sigma (k)}) - \nabla h _ {\sigma (k)} (\beta_ {\sigma (k)})\right) + \nabla h _ {\sigma (k)} (\beta_ {\sigma (k)}).
$$
The first two terms converge to 0: the first is a direct consequence of the convergence of the extracted subsequence, the second is a consequence of the uniform convergence of  $h_{\sigma(k)}$  to  $h_\infty$  on compact sets. Finally the last term is always in  $\operatorname{Span}(x_1, \ldots, x_n)$  due to Proposition 5, leading to  $\nabla h_\infty(\beta_\infty^\star) \in \operatorname{Span}(x_1, \ldots, x_n)$ . Consequently,  $\nabla h_\infty(\beta_\infty^\star) \in \operatorname{Span}(x_1, \ldots, x_n)$ . Notice that from the definition of  $h_\infty$ , we have that  $\nabla h_\infty(\beta_\infty^\star) = \nabla \psi_{\alpha_\infty}(\beta_\infty^\star) - \phi_\infty$ . Now since  $\phi_\infty = \frac{1}{2} \operatorname{arcsinh}\left(\frac{\alpha_+^2 - \alpha_-^2}{2\alpha_\infty^2}\right)$ , one can notice that  $\tilde{\beta}_0$  is precisely defined such that  $\nabla \psi_{\alpha_\infty}(\tilde{\beta}_0) = \phi_\infty$ . Therefore  $\nabla \psi_{\alpha_\infty}(\beta_\infty^\star) - \nabla \psi_{\alpha_\infty}(\tilde{\beta}_0) \in \operatorname{Span}(x_1, \ldots, x_n)$ . This condition along with the fact that  $\beta_\infty^\star$  is an interpolator are exactly the optimality conditions of the convex minimisation problem:
$$
\min _ {\beta^ {\star} \in \mathcal {S}} D _ {\psi_ {\alpha_ {\infty}}} (\beta^ {\star}, \tilde {\beta} _ {0})
$$
Therefore  $\beta_{\infty}^{\star}$  must be equal to the unique minimiser of this problem. Since this is true for any sub-sequence we get that  $\beta_{k}$  converges almost surely to:
$$
\beta_ {\infty} ^ {\star} = \underset {\beta \in \mathcal {S}} {\operatorname {a r g m i n}} D _ {\psi_ {\alpha_ {\infty}}} (\beta^ {\star}, \tilde {\beta} _ {0}).
$$
Point 3. From what we just proved, note that it is sufficient to prove that  $\alpha_{k},\alpha_{\pm ,k},\phi_{k}$  converge to limits  $\alpha_{\infty},\alpha_{\pm ,\infty},\phi_{\infty}$  satisfying  $\alpha_{\infty},\alpha_{\pm ,\infty}\in \mathbb{R}_{>0}^{d}$  (with positive and non-null coordinates) and  $\phi_{\infty}\in \mathbb{R}^{d}$ . Indeed, if this holds and since we assume that the iterates converge to some interpolator, we proved just above that this interpolator is uniquely defined through the desired implicit regularization problem. We thus prove the convergence of  $\alpha_{k},\alpha_{\pm ,k},\phi_{k}$ .
Note that the convergence of  $u_{k}, v_{k}$  is equivalent to the convergence of  $w_{\pm, k}$  in the  $w_{+}^{2} - w_{-}^{2}$  parameterisation used in our proofs, that we use there too. We have:
$$
w _ {\pm , k + 1} = \left(1 \mp \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k})\right) \odot w _ {\pm , k},
$$
$$
\ln (w _ {\pm , k} ^ {2}) = \sum_ {\ell <   k} \ln ((1 \mp \gamma_ {\ell} \nabla \mathcal {L} _ {\mathcal {B} _ {\ell}} (\beta_ {\ell})) ^ {2}).
$$
We now assume that stepsizes are such that for all  $\ell \geqslant 0$  and  $i\in [d]$ , stepsizes are such that we have  $|\gamma_{\ell}\nabla_{i}\mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell})|\neq 1$ : this is true for all stepsizes except a countable number of stepsizes, and so this is true for almost all stepsizes. Since we assume that the iterates  $\beta_{k}$  converge to some interpolator, this leads to  $\gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell})\to 0$  if we assume that stepsizes do not diverge.
Taking the limit, we have
$$
\ln (w _ {\pm , \infty} ^ {2}) = \sum_ {\ell <   \infty} \ln ((1 \mp \gamma_ {\ell} \nabla \mathcal {L} _ {\mathcal {B} _ {\ell}} (\beta_ {\ell})) ^ {2}).
$$
This limit is in  $(\{-\infty \} \cup \mathbb{R})^d$  (since  $w_{\pm ,\infty}\in \mathbb{R}^d$ ), and a coordinate of the limit is equal to  $-\infty$  if and only if the sum on the RHS diverges to  $-\infty$  (note that from our assumption just above, no term of the sum can be equal to  $-\infty$ ).
We have  $\ln ((1\mp \gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell}))^{2})\sim \mp 2\gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell})$  as  $\ell \to \infty$ , so that if for some coordinate  $i$  we have  $\sum_{\ell}\gamma_{\ell}\nabla_{i}\mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell}) = \mp \infty$ , then the coordinate  $i$  of the limit satisfies  $\ln (w_{i,\pm ,\infty}^2) = +\infty$ , which is impossible. Hence, the sum  $\sum_{\ell}\gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell})$  is in  $\mathbb{R}^d$  (and is thus converging); consequently,  $\sum_{\ell}\gamma_{\ell}^{2}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell})^{2}$  converges and thus  $\sum_{\ell}q(\gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell}))$  and  $\sum_{\ell}q_{\pm}(\gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell}))$  all converge: the sequences  $\alpha_{k},\alpha_{\pm ,k}$  thus converge to limits in  $\mathbb{R}_>^d$ , and  $\phi_k$  converges, concluding our proof.
We begin with the following Lemma, that explicits the curvature of  $D_h$  around the set of interpolators.
Lemma 3. For all  $k \geqslant 0$ , if  $\mathcal{L}(\beta_k) \leqslant \frac{1}{2\lambda_{\max}} (\alpha^2\lambda_{\min}^+)^2$ , we have  $\left\| \beta_k - \beta_{\alpha_k}^\star \right\|^2 \leqslant 2B(\alpha^2\lambda_{\min}^+)^{-1}\mathcal{L}(\beta_k)$ .
Proof. Recall that the sequence  $\mathbf{z}^k = \nabla h_k(\beta^k)$  satisfies  $\mathbf{z}^0 = 0$  and  $\mathbf{z}^{k + 1} = \mathbf{z}^k -\gamma_k\mathcal{L}(\beta^k)$ , so that we have that  $\mathbf{z}^k\in V = \mathrm{Im}(\mathbf{X}\mathbf{X}^\top)$  for all  $k\geqslant 0$ . Then, let  $\beta_{k}^{\alpha}$  be the unique minimizer of  $h_k$  over  $\mathcal{S}$  the space of interpolators:  $\beta_{k}^{\alpha}$  is exactly characterized by  $\mathbf{X}^{\top}\beta_{k}^{\alpha} = \mathbf{Y}$  and  $\nabla h_k(\beta_k^\alpha)\in V$ . We define  $\mathbf{z}_k^\alpha \in V$  as  $\mathbf{z}_k^\alpha = \nabla h_k(\beta_k^\alpha)$ .
Now, fix  $\mathbf{z}^{\alpha} = \mathbf{z}_k^\alpha$  and  $h = h_k$ , and let us define  $\psi : \mathbf{z} \in V \to D_{h^*}(\mathbf{z}, \mathbf{z}^\alpha)$  and  $\phi : \mathbf{z} \in V \to \mathcal{L}(\nabla h^*(\mathbf{z}))$ . We next show that for all  $\mathbf{z} \in V$ , there exists  $\mu_z$  such that  $\nabla^2 \phi(\mathbf{z}) \geqslant \mu_z \nabla^2 \psi(\mathbf{z})$ , and that  $\mu_z \geqslant \mu$  for  $\mathbf{z}$  in an open convex set of  $V$  around  $\mathbf{z}^\alpha$ , for some  $\mu > 0$ . For  $A \in \mathbb{R}^{d \times d}$  an operator/matrix on  $\mathbb{R}^d$ , let us denote  $A_V$  its restriction/co-restriction to  $V$ .
First, for  $\mathbf{z} \in V$ , we have  $\nabla^2 \psi(\mathbf{z}) = \nabla^2 (h^*(\mathbf{z}) - h^*(\mathbf{z}) - \langle \nabla h^*(\mathbf{z}^\alpha), z - z^\alpha \rangle)(\mathbf{z}) = \nabla^2 h^*(\mathbf{z})_V$ . Then,  $\nabla \phi(\mathbf{z}) = \nabla^2 h^*(\mathbf{z}) \nabla \mathcal{L}(\nabla h^*(\mathbf{z}))$ , so that  $\nabla^2 \phi(\mathbf{z}) = \left( \nabla^2 h^*(\mathbf{z}) \nabla^2 \mathcal{L}(\nabla h^*(\mathbf{z})) \nabla^2 h^*(\mathbf{z}) \right)_V + \nabla^3 h^*(\mathbf{z})(\nabla \mathcal{L}(\nabla h^*(\mathbf{z})), \dots, \cdot)_V$ .
Since  $h$  is  $1/(2\alpha^2)$  smooth (on  $\mathbb{R}^d$  and thus on  $V$ ),  $h^*$  is  $2\alpha^2$  strongly convex (on  $V$  and on  $\mathbb{R}^d$ ). Using  $V = \operatorname{Im}(\mathbf{X}\mathbf{X}^\top)$  and  $\nabla^2\mathcal{L} \equiv \mathbf{X}\mathbf{X}^\top$ , we have  $\left(\nabla^2h^*(\mathbf{z})\nabla^2\mathcal{L}(\nabla h^*(\mathbf{z}))\nabla^2h^*(\mathbf{z})\right)_V = \nabla^2h^*(\mathbf{z})_V\nabla^2\mathcal{L}(\nabla h^*(\mathbf{z}))_V\nabla^2h^*(\mathbf{z})_V$ , and thus  $\left(\nabla^2h^*(\mathbf{z})\nabla^2\mathcal{L}(\nabla h^*(\mathbf{z}))\nabla^2h^*(\mathbf{z})\right)_V \succeq 2\alpha^2\lambda_{\min}^+ \nabla^2h^*(\mathbf{z})_V$ .
For the other term of  $\nabla^2\phi$ , namely  $\nabla^3 h^*(\mathbf{z})(\nabla \mathcal{L}(\nabla h^*(\mathbf{z})), \cdot, \cdot)_V$ , we compute  $\nabla_{ijk}^3 h^*(\mathbf{z}) = \mathbf{1}_{i = j = k} 2\alpha_{i,k}^2 \sinh(\mathbf{z}_i)$ , leading to:  $\nabla^3 h^*(\mathbf{z})(\nabla \mathcal{L}(\nabla h^*(\mathbf{z})), \cdot, \cdot)_V = \operatorname{diag}(2\alpha^2 \sinh(\mathbf{z})) \odot (\mathbf{X}\mathbf{X}^\top (2\alpha^2 \sinh(\mathbf{z}) - \beta^\alpha)))_V$ . Thus, writing  $\beta_{\mathbf{z}} = 2\alpha_{i,k}^2 \sinh(\mathbf{z}) = \nabla h^*(\mathbf{z})$  the primal surrogate of
$\mathbf{z}$  we have:
$$
\begin{array}{l} \nabla^ {3} h ^ {*} (\mathbf {z}) \left(\nabla \mathcal {L} \left(\nabla h ^ {*} (\mathbf {z})\right), \cdot , \cdot\right) _ {V} = \operatorname {d i a g} \left(2 \alpha_ {i, k} ^ {2} \sinh (\mathbf {z}) \odot \left(\mathbf {X X} ^ {\top} \left(\beta_ {\mathbf {z}} - \beta_ {k} ^ {\alpha}\right)\right)\right) _ {V} \\ \succeq - \left\| \mathbf {X} \mathbf {X} ^ {\top} \left(\beta_ {\mathbf {z}} - \beta_ {k} ^ {\alpha}\right) \right\| _ {\infty} \operatorname {d i a g} \left(2 \alpha_ {k} ^ {2} \odot | \sinh (\mathbf {z}) |\right) _ {V} \\ \succeq - \left\| \mathbf {X} \mathbf {X} ^ {\top} \left(\beta_ {\mathbf {z}} - \beta_ {k} ^ {\alpha}\right) \right\| _ {\infty} \operatorname {d i a g} \left(2 \alpha_ {k} ^ {2} \odot \cosh (\mathbf {z})\right) _ {V} \\ = - \left\| \mathbf {X} \mathbf {X} ^ {\top} \left(\beta_ {\mathbf {z}} - \beta_ {k} ^ {\alpha}\right) \right\| _ {\infty} \nabla^ {2} \psi (\mathbf {z}). \\ \end{array}
$$
Wrapping things together,
$$
\nabla^ {2} \phi (\mathbf {z}) \succeq \left(2 \alpha^ {2} \lambda_ {\min } ^ {+} - \left\| \mathbf {X X} ^ {\top} \left(\beta_ {\mathbf {z}} - \beta^ {\alpha}\right) \right\| _ {\infty}\right) \nabla^ {2} \psi (\mathbf {z}).
$$
Let  $\mathcal{Z} = \left\{\mathbf{z}\in V:\left\| \mathbf{X}\mathbf{X}^{\top}(\beta_{\mathbf{z}} - \beta_{k}^{\alpha})\right\|_{\infty} <   \alpha^{2}\lambda_{\min}^{+}\right\}$  that satisfies
$\left\{\beta \in V:\mathcal{L}(\beta_{\mathbf{z}}) < \frac{1}{2\lambda_{\max}} (\alpha^{2}\lambda_{\min}^{+})^{2}\right\} \subset \mathcal{Z}$ .  $\mathcal{Z}$  is an open convex set of  $V$  containing  $\mathbf{z}^{\alpha}$ . On  $\mathcal{Z}$ ,  $\nabla^2\phi \succeq \alpha^2\lambda_{\min}^+ \nabla^2\psi$ , and  $\psi(\mathbf{z}^{\alpha}) = \phi(\mathbf{z}^{\alpha}) = 0$ , so that for all  $\mathbf{z} \in \mathcal{Z}$ , we have  $\phi(\mathbf{z}) \geqslant \alpha^2\lambda_{\min}^+\psi(\mathbf{z})$ . Hence, for all  $\mathbf{z} \in \mathcal{Z}$ , we have  $D_{h_k}(\beta_k^\alpha, \beta_\mathbf{z}) \leqslant D_{h^\star}(\mathbf{z}, \mathbf{z}^\alpha) \leqslant (\alpha^2\lambda_{\min}^+)^{-1}\mathcal{L}(\beta_\mathbf{z})$ , and using the fact that  $D_{h_k}$  is  $\frac{1}{4B}$  strongly convex, we obtain, for  $\beta_\mathbf{z} = \beta_k$  (since  $\mathbf{z}^k \in V$ ): if  $\mathcal{L}(\beta_k) \leqslant \frac{1}{2\lambda_{\max}} (\alpha^2\lambda_{\min}^+)^2$ , we have  $\| \beta_k^\alpha - \beta_k \|_2^2 \leqslant (\alpha^2\lambda_{\min}^+)^{-1}\mathcal{L}(\beta_k)$ .
Proposition 14. As assume  $\mathcal{L}$  is  $L_{r}$ -relatively smooth with respect to all the  $h_k$ 's. Then for all  $\beta$  we have the following inequality.
$$
\begin{array}{l} \gamma_ {k} (\mathcal {L} (\beta_ {k + 1}) - \mathcal {L} (\beta)) \leqslant D _ {h _ {k}} (\beta , \beta_ {k}) - D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) - (1 - \gamma_ {k} L _ {r}) D _ {h _ {k}} (\beta_ {k + 1}, \beta_ {k}) \\ + \left(h _ {k + 1} - h _ {k}\right) (\beta) - \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k + 1}\right). \\ \end{array}
$$
Proof. For any  $\beta, \beta_k, \beta_{k+1}$ , the following holds (three points identity for time varying potentials, Proposition 9):