SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\psi_ {\tilde {\alpha}} (\beta) \underset {\alpha \rightarrow 0} {\sim} \ln \left(\frac {1}{\alpha}\right) \cdot \sum_ {i = 1} ^ {d} (1 + h _ {i}) | \beta_ {i} |.
$$
$$
\psi_ {\tilde {\alpha}} (\beta) = \frac {1}{2} \sum_ {i = 1} ^ {d} \left(\beta_ {i} \mathrm {a r c s i n h} \left(\frac {\beta_ {i}}{\tilde {\alpha} _ {i} ^ {2}}\right) - \sqrt {\beta_ {i} ^ {2} + \tilde {\alpha} _ {i} ^ {4}} + \tilde {\alpha} _ {i} ^ {2}\right)
$$
Using that  $\operatorname{arcsinh}(x) \underset{|x| \to \infty}{\sim} \operatorname{sgn}(x) \ln(|x|)$ , and that  $\ln \left(\frac{1}{\hat{\alpha}_i^2}\right) = (1 + h_i) \ln \left(\frac{1}{\alpha^2}\right)$  we obtain that
$$
\begin{array}{l} \psi_ {\tilde {\alpha}} (\beta) \underset {\alpha \rightarrow 0} {\sim} \frac {1}{2} \sum_ {i = 1} ^ {d} \operatorname {s g n} (\beta_ {i}) \beta_ {i} (1 + h _ {i}) \ln \left(\frac {1}{\alpha^ {2}}\right) \\ = \frac {1}{2} \ln (\frac {1}{\alpha^ {2}}) \sum_ {i = 1} ^ {d} (1 + h _ {i}) | \beta_ {i} |. \\ \end{array}
$$
The following Fig. 7 illustrates the effect of the non-uniform shape  $\alpha$  on the corresponding potential  $\psi_{\alpha}$ .
Figure 7: Left: Uniform  $\alpha = \alpha \mathbf{1}$ : a smaller scale  $\alpha$  leads to the potential  $\psi_{\alpha}$  being closer to the  $\ell_1$ -norm. Right: A non uniform  $\alpha$  can lead to the recovery of a solution which is very far from the minimum  $\ell_1$ -norm solution. The affine line corresponds to the set of interpolators when  $n = 1$ ,  $d = 2$  and  $s = 1$ .
More generally, for  $\alpha$  such that  $\alpha_{i}\rightarrow 0$  for all  $i\in [d]$  at rates such that  $\ln (1 / \alpha_i)\sim q_i\ln (1 / \max_i\alpha_i)$ , we retrieve a weighted  $\ell_1$  norm:
$$
\frac {\psi_ {\alpha} (\beta)}{\ln \left(1 / \alpha^ {2}\right)} \rightarrow \sum_ {i = 1} ^ {d} q _ {i} \left| \beta_ {i} \right|.
$$
Hence, even for arbitrary small  $\max_i\alpha_i$  , if the shape of  $\alpha$  is bad', the interpolator  $\beta_{\alpha}$  that minimizes  $\psi_{\alpha}$  can be arbitrary far away from  $\beta_{\ell^1}^{\star}$  the interpolator of minimal  $\ell_1$  norm.
We illustrate the importance of the previous proposition in the following example.
Example 1. We illustrate how, even for arbitrary small  $\max_i\alpha_i$ , the interpolator  $\beta_{\alpha}^{\star}$  that minimizes  $\psi_{\alpha}$  can be far from the minimum  $\ell_1$  norm solution, due to the shape of  $\alpha$  that is not uniform. The message of this example is that for  $\alpha \to 0$  non-uniformly across coordinates, if the coordinates of  $\alpha$  that go slowly to 0 coincide with the non-null coordinates of the sparse interpolator we want to retrieve, then  $\beta_{\alpha}^{\star}$  will be far from the sparse solution.
A simple counterexample can be built: let  $\beta_{\mathrm{sparse}}^{\star} = (1, \dots, 1, 0, \dots, 0)$  (with only the  $s = o(d)$  first coordinates that are non-null), and let  $(x_i)$ ,  $(y_i)$  be generated as  $y_i = \langle \beta_{\mathrm{sparse}}^{\star}, x_i \rangle$  with  $x_i \sim \mathcal{N}(0, 1)$ . For  $n$  large enough ( $n$  of order  $s \ln(d)$  where  $s$  is the sparsity), the design matrix  $X$  is RIP [10], so that the minimum  $\ell_1$  norm interpolator  $\beta_{\ell^1}^{\star}$  is exactly equal to  $\beta_{\mathrm{sparse}}^{\star}$ .
However, if  $\alpha$  is such that  $\max_i\alpha_i\to 0$  with  $h_i > > 1$  for  $j\leqslant s$  and  $h_i = 1$  for  $i\geqslant s + 1$  ( $h_i$  as in Proposition 7),  $\beta_{\alpha}^{\star}$  will be forced to verify  $\beta_{\alpha ,i}^{\star} = 0$  for  $i\leqslant s$  and hence  $\| \beta_{\alpha ,1}^{\star} - \beta_{\ell^{1}}^{\star}\|_{1}\geqslant s.$
E Main descent lemma and boundedness of the iterates
The goal of this section is to prove the following proposition, our main descent lemma: for well-chosen stepsizes, the Bregman divergences  $(D_{h_k}(\beta^\star ,\beta_k))_{k\geqslant 0}$  decrease. We then use this proposition to bound the iterates for both SGD and GD.
Proposition 8. There exist a constant  $c > 0$  and  $B > 0$  such that  $B = \mathcal{O}(\inf_{\beta^{\star}\in \mathcal{S}}\| \beta^{\star}\|_{\infty})$  for  $GD$  and  $B = \mathcal{O}(\ln (1 / \alpha)\inf_{\beta^{\star}\in \mathcal{S}}\| \beta^{\star}\|_{\infty})$  for SGD, such that if  $\gamma_k\leqslant \frac{c}{LB}$  for all  $k$ , then we have, for all  $k\geqslant 0$  and any interpolator  $\beta^{\star}\in \mathcal{S}$ :
$$
D _ {h _ {k + 1}} \left(\beta^ {\star}, \beta_ {k + 1}\right) \leqslant D _ {h _ {k}} \left(\beta^ {\star}, \beta_ {k}\right) - \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right).
$$
To prove this result, we first provide a general descent lemma for time-varying mirror descent (Proposition 9, appendix E.1), before proving the proposition for fixed iteration  $k$  and bound  $B > 0$  on the iterates infinity norm in Appendix E.2 (Proposition 10). We finally use this to prove a bound on the iterates infinity norm in appendix E.3.
E.1 Descent lemma for (stochastic) mirror descent with varying potentials
In the following we adapt a classical mirror descent equality but for time varying potentials, that differentiates from Orabona et al. [47] in that it enables us to prove the decrease of the Bregman divergences of the iterates. Moreover, as for classical MD, it is an equality.
Proposition 9. For  $h,g:\mathbb{R}^d\to \mathbb{R}$  functions, let  $D_{h,g}(\beta ,\beta^{\prime}) = h(\beta) - g(\beta^{\prime}) - \langle \nabla g(\beta^{\prime}),\beta -\beta^{\prime}\rangle^{4}$  for  $\beta ,\beta^{\prime}\in \mathbb{R}^{d}$ . Let  $(h_k)$  strictly convex functions defined  $\mathbb{R}^d$ $\mathcal{L}$  a convex function defined on  $\mathbb{R}^d$ . Let  $(\beta_{k})$  defined recursively through  $\beta_0\in \mathbb{R}^d$ , and
$$
\beta_ {k + 1} \in \operatorname {a r g m i n} _ {\beta \in \mathbb {R} ^ {d}} \left\{\gamma_ {k} \langle \nabla \mathcal {L} (\beta_ {k}), \beta - \beta_ {k} \rangle + D _ {h _ {k + 1}, h _ {k}} (\beta , \beta_ {k}) \right\},
$$
where we assume that the minimum is unique and attained in  $\mathbb{R}^d$ . Then,  $(\beta_{k})$  satisfies
$$
\nabla h _ {k + 1} \left(\beta_ {k + 1}\right) = \nabla h _ {k} \left(\beta_ {k}\right) - \gamma_ {k} \nabla \mathcal {L} \left(\beta_ {k}\right),
$$
and for any  $\beta \in \mathbb{R}^d$
$$
\begin{array}{l} D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) = D _ {h _ {k}} (\beta , \beta_ {k}) - \gamma_ {k} \langle \nabla \mathcal {L} (\beta_ {k}), \beta_ {k} - \beta \rangle + D _ {h _ {k + 1}} (\beta_ {k}, \beta_ {k + 1}) \\ - \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k}\right) + \left(h _ {k + 1} - h _ {k}\right) (\beta). \\ \end{array}
$$
Proof. Let  $\beta \in \mathbb{R}^d$ . Since we assume that the minimum through which  $\beta_{k + 1}$  is computed is attained in  $\mathbb{R}^d$ , the gradient of the function  $V_{k}(\beta) = \gamma_{k}\langle \nabla \mathcal{L}(\beta_{k}),\beta -\beta_{k}\rangle +D_{h_{k + 1},h_{k}}(\beta ,\beta_{k})$  evaluated at  $\beta_{k + 1}$  is null, leading to  $\nabla h_{k + 1}(\beta_{k + 1}) = \nabla h_k(\beta_k) - \gamma_k\nabla \mathcal{L}(\beta_k)$ .
for  $h = g$ , we recover the classical Bregman divergence that we denote  $D_{h} = D_{h,h}$
Then, since  $\nabla V_{k}(\beta_{k + 1}) = 0$ , we have  $D_{V_k}(\beta ,\beta_{k + 1}) = V_k(\beta) - V_k(\beta_{k + 1})$ . Using  $\nabla^2 V_k = \nabla^2 h_{k + 1}$ , we also have  $D_{V_k} = D_{h_{k + 1}}$ . Hence:
$$
D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) = \gamma_ {k} \langle \nabla \mathcal {L} (\beta_ {k}), \beta - \beta_ {k + 1} \rangle + D _ {h _ {k + 1}, h _ {k}} (\beta , \beta_ {k}) - D _ {h _ {k + 1}, h _ {k}} \left(\beta_ {k + 1}, \beta_ {k}\right).
$$
We write  $\gamma_{k}\langle \nabla \mathcal{L}(\beta_{k}),\beta -\beta_{k + 1}\rangle = \gamma_{k}\langle \nabla \mathcal{L}(\beta_{k}),\beta -\beta^{k}\rangle +\gamma_{k}\langle \nabla \mathcal{L}(\beta_{k}),\beta_{k} - \beta_{k + 1}\rangle$  . We also have  $\gamma_{k}\langle \nabla \mathcal{L}(\beta_{k}),\beta_{k} - \beta_{k + 1}\rangle = \langle \nabla h_{k}(\beta_{k}) - \nabla h_{k + 1}(\beta_{k + 1}),\beta_{k} - \beta_{k + 1}\rangle = D_{h_{k},h_{k + 1}}(\beta_{k},\beta_{k + 1})+$ $D_{h_{k + 1},h_k}(\beta_{k + 1},\beta^k)$  , so that  $\gamma_{k}\langle \nabla \mathcal{L}(\beta_{k}),\beta_{k} - \beta_{k + 1}\rangle -D_{h_{k + 1},h_{k}}(\beta_{k + 1},\beta^{k}) = D_{h_{k},h_{k + 1}}(\beta_{k},\beta_{k + 1})$  Thus,
$$
D _ {h _ {k + 1}} (\beta , \beta_ {k + 1}) = D _ {h _ {k + 1}, h _ {k}} (\beta , \beta_ {k}) - \gamma_ {k} \left(D _ {f} (\beta , \beta_ {k}) + D _ {f} (\beta_ {k}, \beta)\right) + D _ {h _ {k}, h _ {k + 1}} (\beta_ {k}, \beta_ {k + 1}),
$$
and writing  $D_{h,g}(\beta ,\beta^{\prime}) = D_{g}(\beta ,\beta^{\prime}) + h(\beta) - g(\beta)$  concludes the proof.
E.2 Proof of Proposition 10
In next proposition, we use Proposition 9 to prove our main descent lemma. To that end, we bound the error terms that appear in Proposition 9 as functions of  $\mathcal{L}_{\mathcal{B}_k}(\beta_k)$  and norms of  $\beta_{k},\beta_{k + 1}$ , so that for explicit step sizes, the error terms can be cancelled by half of the negative quantity  $-2\mathcal{L}_{\mathcal{B}_k}(\beta_k)$ .
Additional notation: let  $L_2, L_\infty > 0$  such that  $\forall \beta, \| H_{\mathcal{B}}\beta \|_2 \leqslant L\| \beta \|_2, \| H_{\mathcal{B}}\beta \|_\infty \leqslant L\| \beta \|_\infty$  for all batches  $\mathcal{B} \subset [n]$  of size  $b$ .
Proposition 10. Let  $k \geqslant 0$  and  $B > 0$ . Provided that  $\| \beta_k\|_{\infty},\| \beta_{k + 1}\|_{\infty},\| \beta^{\star}\|_{\infty} \leqslant B$  and  $\gamma_{k} \leqslant \frac{c}{LB}$  where  $c > 0$  is some numerical constant, we have:
$$
D _ {h _ {k + 1}} \left(\beta^ {\star}, \beta_ {k + 1}\right) \leqslant D _ {h _ {k}} \left(\beta^ {\star}, \beta_ {k}\right) - \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right). \tag {21}
$$
Proof. Let  $\beta^{\star} \in S$  be any interpolator. From Proposition 9:
$$
D _ {h _ {k + 1}} (\beta^ {\star}, \beta_ {k + 1}) = D _ {h _ {k}} (\beta^ {\star}, \beta_ {k}) - 2 \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) + D _ {h _ {k + 1}} (\beta_ {k + 1}, \beta_ {k}) - (h _ {k + 1} - h _ {k}) (\beta_ {k}) + (h _ {k + 1} - h _ {k}) (\beta^ {\star}).
$$
We want to bound the last three terms of this equality. First, to bound the last two we apply Lemma 7 assuming that  $\| \beta^{\star}\|_{\infty},\| \beta_{k + 1}\|_{\infty}\leqslant B$
$$
- \left(h _ {k + 1} - h _ {k}\right) \left(\beta_ {k}\right) + \left(h _ {k + 1} - h _ {k}\right) \left(\beta^ {*}\right) \leqslant 2 4 B L _ {2} \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right)
$$
We now bound  $D_{h_{k + 1}}(\beta_k,\beta_{k + 1})$ . Classical Bregman manipulations provide that
$$
\begin{array}{l} D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) = D _ {h _ {k + 1} ^ {*}} \left(\nabla h _ {k + 1} \left(\beta_ {k + 1}\right), \nabla h _ {k + 1} \left(\beta_ {k}\right)\right) \\ = D _ {h _ {k + 1} ^ {*}} \left(\nabla h _ {k} \left(\beta^ {k}\right) - \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right), \nabla h _ {k + 1} \left(\beta_ {k}\right)\right). \\ \end{array}
$$
From Lemma 6 we have that  $h_{k + 1}$  is  $\min(1 / (4\alpha_{k + 1}^2), 1 / (4B))$  strongly convex on the  $\ell^\infty$ -centered ball of radius  $B$  therefore  $h_{k + 1}^*$  is  $\max(4\alpha_{k + 1}^2, 4B) = 4B$  (for  $\alpha$  small enough or  $B$  big enough) smooth on this ball, leading to:
$$
\begin{array}{l} D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) \leqslant 2 B \| \nabla h _ {k} \left(\beta_ {k}\right) - \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right) - \nabla h _ {k + 1} \left(\beta_ {k}\right) \| _ {2} ^ {2} \\ \leqslant 4 B \left(\| \nabla h _ {k} (\beta_ {k}) - \nabla h _ {k + 1} (\beta_ {k}) \| _ {2} ^ {2} + \| \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \| _ {2} ^ {2}\right). \\ \end{array}
$$
Using  $|\nabla h_k(\beta) - \nabla h_{k + 1}(\beta)|\leqslant 2\delta_k$  where  $\delta_{k} = q(\gamma_{k}\nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k}))$  , we get that:
$$
D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) \leqslant 8 B \| \delta_ {k} \| _ {2} ^ {2} + 4 B L \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right).
$$
Now,  $\| \delta_k\| _2^2\leqslant \| \delta_k\| _1\| \delta_k\|_\infty$  and using Lemma 5,  $\| \delta_k\| _1\| \delta_k\|_\infty$ $\begin{array}{r}\leqslant \\ 4\| \gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\| _2^2\| \gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_\infty^2\\ \leqslant 2\| \gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\| _2^2\mathrm{since}\| \gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_\infty \end{array}$ $\gamma_{k}L_{\infty}\| \beta_{k} - \beta_{\infty}\| \leqslant \gamma_{k}\times 2LB\leqslant 1 / 2$  is verified for  $\gamma_{k}\leqslant 1 / (4LB)$ . Thus,
$$
D _ {h _ {k + 1}} \left(\beta_ {k}, \beta_ {k + 1}\right) \leqslant 4 0 B L _ {2} \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right).
$$
Hence, provided that  $\| \beta_k\|_{\infty}\leqslant B,\| \beta_{k + 1}\|_{\infty}\leqslant B$  and  $\gamma_{k}\leqslant 1 / (4LB)$ , we have:
$$
D _ {h _ {k + 1}} \left(\beta^ {\star}, \beta_ {k + 1}\right) \leqslant D _ {h _ {k}} \left(\beta^ {\star}, \beta_ {k}\right) - 2 \gamma_ {k} \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right) + 6 4 L _ {2} \gamma_ {k} ^ {2} B \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right),
$$
if  $\gamma_k \leqslant \frac{c}{BL}$ , where  $c = \frac{1}{64}$ .
E.3 Bound on the iterates
We now bound the iterates  $(\beta_{k})$  by an explicit constant  $B$  that depends on  $\| \beta^{\star}\| _1$  (for any fixed  $\beta^{\star}\in \mathcal{S}$ ).
The first bound we prove holds for both SGD and GD, and is of the form  $\mathcal{O}(\| \beta^{\star}\|_{1}\ln (1 / \alpha^{2})$  while the second bound, that holds only for GD  $(b = n)$  is of order  $\mathcal{O}(\| \beta^{\star}\|_{1})$  (independent of  $\alpha$ ). While a bound independent of  $\alpha$  is only proved for GD, we believe that such a result also holds for SGD, and in both cases  $B$  should be thought of order  $\mathcal{O}(\| \beta^{\star}\|_{1})$ .
E.3.1 Bound that depends on  $\alpha$  for GD and SGD
A consequence of Proposition 10 is the boundedness of the iterates, as shown in next corollary. Hence, Proposition 10 can be applied using  $B$  a uniform bound on the iterates  $\ell^{\infty}$  norm.
Corollary 1. Let  $B = 3\| \beta^{\star}\|_{1}\ln \left(1 + \frac{\|\beta^{*}\|_{1}}{\alpha^{2}}\right)$ . For stepsizes  $\gamma_{k} \leqslant \frac{c}{BL}$ , we have  $\| \beta_k\|_\infty \leqslant B$  for all  $k \geqslant 0$ .
Proof. We proceed by induction. Let  $k \geqslant 0$  such that  $\| \beta_k\|_{\infty} \leqslant B$  for some  $B > 0$  and  $D_{h_k}(\beta^\star, \beta_k) \leqslant D_{h_0}(\beta^\star, \beta_0)$  (note that these two properties are verified for  $k = 0$ , since  $\beta_0 = 0$ ). For  $\gamma_k$  sufficiently small (i.e., that satisfies  $\gamma_k \leqslant \frac{c}{B'L}$  where  $B' \geqslant \| \beta_{k+1}\|_{\infty}, \| \beta_k\|_{\infty}, \| \beta^\star\|_{\infty}$ ), using Proposition 10, we have  $D_{h_{k+1}}(\beta^\star, \beta_{k+1}) \leqslant D_{h_k}(\beta^\star, \beta_k)$  so that  $D_{h_{k+1}}(\beta^\star, \beta_{k+1}) \leqslant D_{h_0}(\beta^\star, \beta_0)$ , which can be rewritten as:
$$
\sum_ {i = 1} ^ {d} \alpha_ {k + 1, i} ^ {2} (\sqrt {1 + (\frac {\beta_ {k + 1 , i}}{\alpha_ {k + 1 , i} ^ {2}}) ^ {2}} - 1) \leqslant \sum_ {i = 1} ^ {d} \beta_ {i} ^ {\star} \operatorname {a r c s i n h} (\frac {\beta_ {k + 1 , i}}{\alpha^ {2}}).
$$
Hence,  $\| \beta_{k + 1}\| _1\leqslant \| \beta^\star \| _1\ln (1 + \frac{\|\beta_{k + 1}\|_1}{\alpha^2})$  . We then notice that for  $x,y > 0,x\leqslant y\ln (1 + x)\Rightarrow$ $x\leqslant 3y\ln (1 + y)$  : if  $x > y\ln (1 + y)$  and  $x > y$  , we have that  $y\ln (1 + y) <   y\ln (1 + x)$  , so that  $1 + y <   1 + x$  , which contradicts our assumption. Hence,  $x\leqslant \max (y,y\ln (1 + y))$  . In our case,  $x = \left\| \beta^{k + 1}\right\| _1 / \alpha^2,y = \left\| \beta^\star \right\| _1 / \alpha^2$  so that for small alpha,  $\ln (1 + y)\geqslant 1$
Hence, we deduce that  $\| \beta_{k + 1}\| _1\leqslant B$  , where  $B = \| \beta^{\star}\|_{1}\ln (1 + \frac{\|\beta^{\star}\|_{1}}{\alpha^{2}})$
This is true as long as  $\gamma_{k}$  is tuned using  $B^{\prime}$  a bound on  $\max (\| \beta_k\|_\infty ,\| \beta_{k + 1}\|_\infty)$ . Using the continuity of  $\beta_{k + 1}$  as a function of  $\gamma_{k}$  ( $\beta_{k}$  being fixed), we show that  $\gamma_{k}\leqslant \frac{1}{2}\times \frac{c}{BL}$  can be used using this  $B$ . Indeed, let  $\phi :\mathbb{R}^{+}\to \mathbb{R}^{d}$  be the function that takes as entry  $\gamma_{k}\geqslant 0$  and outputs the corresponding  $\| \beta_{k + 1}\|_{\infty}$ :  $\phi$  is continuous. Let  $\gamma_r = \frac{1}{2}\times \frac{c}{rL}$  for  $r > 0$  and  $\bar{r} = \sup \left\{r\geqslant 0:B < \phi (\gamma_r)\right\}$  (the set is upper-bounded; if is empty, we do not need what follows since it means that any stepsize leads to  $\| \beta_{k + 1}\|_{\infty}\leqslant B$ ). By continuity of  $\phi$ ,  $\phi (\gamma_{\bar{r}}) = B$ . Furthermore, for all  $r$  that satisfies  $r\geqslant \max (\phi (\gamma_r),B)\geqslant \max (\phi (\gamma_r),\| \beta_k\|_\infty,\| \beta^\star \|_\infty)$ , we have, using what is proved just above, that  $\| \beta_{k + 1}\|_{\infty}\leqslant B$  and thus  $\phi (\gamma_r)\leqslant B$  for such a  $r$ :
Lemma 1. For  $r > 0$  such that  $r\geqslant \max (\phi (\gamma_r),B)$  , we have  $\phi (\gamma_r)\leqslant B$
Now, if  $\bar{r} > B$ , by definition of  $\bar{r}$  and by continuity of  $\phi$ , since  $\phi(\bar{r}) = B$ , there exists some  $B < r < \bar{r}$  such that  $\phi(\gamma_r) > B$  (definition of the supremum) and  $\phi(\gamma_r) \leqslant 2B$  (continuity of  $\phi$ ). This particular choice of  $r$  thus satisfies  $r > B$  and and  $\phi(\gamma_r) \leqslant 2B \leqslant 2r$ , leading to  $\phi(\gamma_r) \leqslant B$ , using Lemma 1, hence a contradiction: we thus have  $\bar{r} \leqslant B$ .
This concludes the induction: for all  $r \geqslant B$ , we have  $r \geqslant \bar{r}$  so that  $\phi(\gamma_r) \leqslant B$  and thus for all stepsizes  $\gamma \leqslant \frac{c}{2LB}$ , we have  $\| \beta_{k+1} \|_\infty \leqslant B$ .
E.3.2 Bound independent of  $\alpha$
We here assume in this subsection that  $b = n$ . We prove that for gradient descent, the iterates are bounded by a constant that does not depend on  $\alpha$ .
Proposition 11. Assume that  $b = n$  (full batch setting). There exists some  $B = \mathcal{O}(\| \beta^{\star}\|_{1})$  such that for stepsizes  $\gamma_k\leqslant \frac{c}{BL}$ , we have  $\| \beta_k\|_{\infty}\leqslant B$  for all  $k\geqslant 0$ .
Proof. We first begin by proving the following proposition: for sufficiently small step sizes, the loss values decrease. In the following lemma we provide a bound on the gradient descent iterates  $(w_{+,k}, w_{-,k})$  which will be useful to show that the loss is decreasing.
Proposition 12. For  $\gamma_k \leqslant \frac{c}{LB}$  where  $B \geqslant \max(\|\beta_k\|_{\infty}, \|\beta_{k+1}\|_{\infty})$ , we have  $\mathcal{L}(\beta_{k+1}) \leqslant \mathcal{L}(\beta_k)$ .
Proof. Oddly, using the time-varying mirror descent recursion is not the easiest way to show the decrease of the loss, due to the error terms which come up. Therefore to show that the loss is decreasing we use the gradient descent recursion. Recall that the iterates  $w_{k} = (w_{+,k},w_{-,k})\in \mathbb{R}^{2d}$  follow a gradient descent on the non-convex loss  $F(w) = \frac{1}{2}\| y - \frac{1}{2} X(w_{+}^{2} - w_{-}^{2})\|_{2}$ .
For  $k \geqslant 0$ , using the Taylor formula we have that  $F(w_{k + 1}) \leqslant F(w_k) - \gamma_k(1 - \frac{\gamma_k L_k}{2})\| \nabla F(w_k)\|^2$  with the local smoothness  $L_{k} = \sup_{w \in [w_{k}, w_{k + 1}]}\lambda_{\max}(\nabla^{2}F(w))$ . Hence if  $\gamma_k \leqslant 1 / L_k$  for all  $k$  we get that the loss is non-increasing. We now bound  $L_{k}$ . Computing the hessian of  $F$ , we obtain that:
$$
\begin{array}{l} \nabla^ {2} F (w _ {k}) = \left( \begin{array}{c c} \mathrm {d i a g} (\nabla \mathcal {L} (\beta_ {k})) & 0 \\ 0 & - \mathrm {d i a g} (\nabla \mathcal {L} (\beta_ {k})) \end{array} \right) \\ + \left( \begin{array}{c c} \operatorname {d i a g} (w _ {+, k}) H \operatorname {d i a g} (w _ {+, k}) & - \operatorname {d i a g} (w _ {-, k}) H \operatorname {d i a g} (w _ {+, k}) \\ - \operatorname {d i a g} (w _ {+, k}) H \operatorname {d i a g} (w _ {-, k}) & \operatorname {d i a g} (w _ {-, k}) H \operatorname {d i a g} (w _ {-, k}) \end{array} \right). \tag {22} \\ \end{array}
$$
Let us denote by  $M = \begin{pmatrix} M_{+} & M_{+,-} \\ M_{+, - } & M_{-} \end{pmatrix} \in \mathbb{R}^{2d\times 2d}$  the second matrix in the previous equality. With this notation  $\| \nabla^2 F(w_k)\| \leqslant \| \nabla \mathcal{L}(\beta_k)\|_{\infty} + 2\| M\|$  (where the norm corresponds to the Schatten 2-norm which is the largest eigenvalue for symmetric matrices). Now, notice that:
$$
\begin{array}{l} \| M \| ^ {2} = \sup  _ {u \in \mathbb {R} ^ {2 d}, \| u \| = 1} \| M u \| ^ {2} \\ = \sup_{\substack{u_{+}\in \mathbb{R}^{d},\| u_{+}\| = 1\\ u_{-}\in \mathbb{R}^{d},\| u_{-}\| = 1\\ (a,b)\in \mathbb{R}^{2},a^{2} + b^{2} = 1}}\left\| M\left( \begin{array}{c}a\cdot u_{+}\\ b\cdot u_{-} \end{array} \right)\right\|^{2}. \\ \end{array}
$$
$$
\begin{array}{l} \left\| M \left( \begin{array}{c} a \cdot u _ {+} \\ b \cdot u _ {-} \end{array} \right) \right\| ^ {2} = \left\| \left( \begin{array}{c} a M _ {+} u _ {+} + b M _ {+ -} u _ {-} \\ a M _ {+ -} u _ {+} + b M _ {-} u _ {-} \end{array} \right) \right\| ^ {2} \\ = \left\| a M _ {+} u _ {+} + b M _ {+ -} u _ {-} \right\| ^ {2} + \left\| a M _ {+ -} u _ {+} + b M _ {-} u _ {-} \right\| ^ {2} \\ \leqslant 2 \left(a ^ {2} \| M _ {+} u _ {+} \| ^ {2} + b ^ {2} \| M _ {+ -} u _ {-} \| ^ {2} + a ^ {2} \| M _ {+ -} u _ {+} \| ^ {2} + b ^ {2} \| M _ {-} u _ {-} \| ^ {2}\right) \\ \leqslant 2 \left(\| M _ {+} \| ^ {2} + \| M _ {+ -} \| ^ {2} + \| M _ {-} \| ^ {2}\right). \\ \end{array}
$$
Since  $\| M_{\pm} \| \leqslant \lambda_{max} \cdot \| w_{\pm} \|_{\infty}^{2}$  and  $\| M_{+ - } \| \leqslant \lambda_{max} \| w_{+} \|_{\infty} \| w_{-} \|_{\infty}$  we finally get that
$$
\begin{array}{l} \left\| M \right\| ^ {2} \leqslant 6 \lambda_ {m a x} ^ {2} \cdot \max  \left(\left\| w _ {+} \right\| _ {\infty} ^ {2}, \left\| w _ {-} \right\| _ {\infty} ^ {2}\right) ^ {2} \\ \leqslant 6 \lambda_ {m a x} ^ {2} \left(\| w _ {+} ^ {2} \| _ {\infty} + \| w _ {-} ^ {2} \| _ {\infty}\right) ^ {2} \\ \leqslant 1 2 \lambda_ {m a x} ^ {2} \| w _ {+} ^ {2} + w _ {-} ^ {2} \| _ {\infty} ^ {2}. \\ \end{array}
$$
We now upper bound this quantity in the following lemma.
Lemma 2. For all  $k \geqslant 0$ , the following inequality holds component-wise:
$$
w _ {+, k} ^ {2} + w _ {-, k} ^ {2} = \sqrt {4 \boldsymbol {\alpha} _ {k} ^ {4} + \beta_ {k} ^ {2}}.
$$
Proof. Notice from the definition of  $w_{+,k}$  and  $w_{-,k}$  given in the proof of Proposition 5 that:
$$
\left| w _ {+, k} \right| \left| w _ {-, k} \right| = \alpha_ {-, k} \alpha_ {+, k} = \alpha_ {k} ^ {2}. \tag {23}
$$
And  $\alpha_0 = \alpha^2$ . Now since  $\alpha_{k}$  is decreasing coordinate-wise (under our assumptions on the stepsizes,  $\gamma_k^2\nabla \mathcal{L}(\beta_k)^2\leqslant (1 / 2)^2 < 1$ ), we get that:
$$
w _ {+, k} ^ {2} + w _ {-, k} ^ {2} = 2 \sqrt {\boldsymbol {\alpha} _ {k} ^ {4} + \beta_ {k} ^ {2}} \leqslant 2 \sqrt {\boldsymbol {\alpha} ^ {4} + \beta_ {k} ^ {2}}
$$
leading to  $w_{+,k}^2 + w_{-,k}^2 \leqslant \sqrt{4\alpha^4 + B^2}$ .
From Lemma 2,  $w_{+,k}^{2} + w_{-,k}^{2}$  is bounded by  $2\sqrt{\alpha^4 + B^2}$ . Putting things together we finally get that  $\| \nabla^2 F(w) \| \leqslant \| \nabla \mathcal{L}(\beta) \|_{\infty} + 8\lambda_{max}\sqrt{4\|\alpha\|_{\infty}^4 + B^2}$ . Hence,