SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(z _ {t + 1}\right) - f \left(z _ {t}\right) \right] \leq \mathbb {E} \left\langle \nabla f \left(z _ {t}\right), z _ {t + 1} - z _ {t} \right\rangle + \frac {L}{2} \mathbb {E} \| z _ {t + 1} - z _ {t} \| ^ {2} \\ = \mathbb {E} \left[ \frac {\beta}{1 - \beta} \langle \nabla f (z _ {t}), \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \rangle \right] - \mathbb {E} \left[ \langle \nabla f (z _ {t}), \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right] \\ - \mathbb {E} \left[ \left\langle \nabla f (z _ {t}), \alpha_ {t - 1} \delta_ {t - 1} \right\rangle \right] - \mathbb {E} \left[ \left\langle \nabla f (z _ {t}), (\alpha_ {t} - \alpha_ {t}) e _ {t - 1} \right\rangle \right] \\ + \mathbb {E} \left[ \frac {L}{2} \left\| \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} - \alpha_ {t} \frac {g _ {t}}{b _ {t}} - \alpha_ {t - 1} \delta_ {t - 1} - (\alpha_ {t} - \alpha_ {t - 1}) e _ {t - 1} \right\| ^ {2} \right] \\ = \mathbb {E} \left[ \frac {\beta}{1 - \beta} \langle \nabla f (z _ {t}), \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \rangle \right] - \mathbb {E} \left[ \langle \nabla f (z _ {t}), \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right] \\ + \mathbb {E} \left[ \frac {L}{2} \left\| \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} - \alpha_ {t} \frac {g _ {t}}{b _ {t}} - \alpha_ {t - 1} \delta_ {t - 1} - (\alpha_ {t} - \alpha_ {t - 1}) e _ {t - 1} \right\| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {\beta}{1 - \beta} \langle \nabla f (z _ {t}), \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \rangle \right] - \mathbb {E} \left[ \langle \nabla f (z _ {t}), \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right. \\ + L \mathbb {E} \left[ \left\| \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \right\| ^ {2} \right] + L \mathbb {E} \left[ \alpha_ {t} ^ {2} \left\| \frac {g _ {t}}{b _ {t}} \right\| ^ {2} \right] \\ + \frac {L}{2} \mathbb {E} \left[ \| \alpha_ {t - 1} \delta_ {t - 1} \| ^ {2} \right] + \frac {L}{2} \mathbb {E} \left[ \| \left(\alpha_ {t - 1} - \alpha_ {t}\right) e _ {t} \| ^ {2} \right] \tag {30} \\ \end{array}
$$
where the second equality holds due to  $\mathbb{E}[\delta_{t - 1}] = 0$  and  $\mathbb{E}[e_{t - 1}] = 0$ . The last inequality holds owing to  $\mathbb{E}[\| a + b\| ^2 ] = \mathbb{E}[\| a\| ^2 ] + \mathbb{E}[\| b\| ^2 ]$  if  $\mathbb{E}[a] = 0$  or  $\mathbb{E}[b] = 0$ , and  $\mathbb{E}[\| a + b\| ^2 ]\leq 2\mathbb{E}[\| a\| ^2 ] + 2\mathbb{E}[\| b\| ^2 ]$  if  $\mathbb{E}[a]\neq 0$  and  $\mathbb{E}[b]\neq 0$ .
$$
\begin{array}{l} \mathbb{E}[f(z_{T}) - f(z_{1})]\leq \underbrace{\frac{\beta}{1 - \beta}\mathbb{E}\left[\sum_{t = 1}^{T}\langle\nabla f(z_{t}),\left(\frac{\alpha_{t - 1}}{b_{t - 1}} -\frac{\alpha_{t}}{b_{t}}\right)\odot m_{t - 1}\rangle \right]}_{T_{1}} - \underbrace{\mathbb{E}\left[\sum_{t = 1}^{T}\langle\nabla f(z_{t}),\alpha_{t}\frac{g_{t}}{b_{t}}\rangle \right]}_{T_{2}} \\ + \underbrace {L \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\| \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \right\| ^ {2} \right]} _ {T _ {3}} \\ + \underbrace {L \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} ^ {2} \left\| \frac {g _ {t}}{b _ {t}} \right\| ^ {2} \right]} _ {T _ {4}} + \frac {L}{2} \mathbb {E} [ \sum_ {t = 1} ^ {T} \| \alpha_ {t - 1} \delta_ {t - 1} \| ^ {2} ] + \frac {L}{2} \mathbb {E} [ \sum_ {t = 1} ^ {T} \| (\alpha_ {t - 1} - \alpha_ {t}) e _ {t} \| ^ {2} ] \tag {31} \\ \end{array}
$$
Now we focus on bounding  $T_{1}$  below. From Assumption 4, we know  $\| g_t\| \leq G$  ( $t = 1,2,\dots,T$ ) and  $\| \nabla f(z_{t})\| \leq G$ . Due to  $m_{t} = \beta m_{t - 1} + (1 - \beta)g_{t}$  and  $m_{1} = g_{1}$ , it is easy to obtain  $\| m_t\| \leq G$  by complete induction.
Since  $\| \nabla f(z_{t})\| \leq G$  and  $\| m_t\| \leq G$ , we have
$$
\begin{array}{l} T _ {1} = \frac {\beta}{1 - \beta} \mathbb {E} \left[ \sum_ {i = 1} ^ {T} \langle \nabla f (z _ {i}), \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {i - 1} \rangle \right] \\ \stackrel {(i)} {\leq} \frac {\beta}{1 - \beta} \mathbb {E} \left[ \sum_ {i = 1} ^ {T} \| \nabla f (z _ {t}) \| \| m _ {t} \| \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right] \\ \stackrel {(i i)} {\leq} \frac {\beta}{1 - \beta} G ^ {2} \mathbb {E} \left[ \sum_ {i = 1} ^ {T} \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right] \tag {32} \\ \stackrel {(i i i)} {=} \frac {\beta}{1 - \beta} G ^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {T} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \right\| _ {1} \right] \\ \leq \frac {\beta}{1 - \beta} G ^ {2} \mathbb {E} \left[ \left\| \frac {\alpha_ {0}}{b _ {0}} \right\| _ {1} \right] \\ \stackrel {(i v)} {\leq} \frac {\alpha_ {0} \beta d}{(1 - \beta) \rho} G ^ {2}, \\ \end{array}
$$
where  $(i)$  holds size  $\| a\odot b\| \leq \| a\|\max_j|(b)_j|\leq \| a\| \| b\| _1$ $(ii)$  holds due to  $\| \nabla f(z_t)\| \leq G$  and  $\| m_t\| \leq G$  (iii) holds because  $\frac{\alpha_{t - 1}}{(b_{t - 1})_j} -\frac{\alpha_t}{(b_t)_j}\geq 0$  for any  $j\in [1,2,\dots,d]$ ,  $(iv)$  holds due to  $\min_j(b_t)_j\geq \rho >0$  for any  $j\in [1,2,\ldots ,d]$ .
Let us turn to bound  $T_{2}$ ,
$$
\begin{array}{l} T _ {2} = - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (z _ {t}), \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right] \\ = \underbrace {- \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (z _ {t}) - f (x _ {t}) , \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right]} _ {T _ {5}} \underbrace {- \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (x _ {t}) , \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right]} _ {T _ {6}} \tag {33} \\ \end{array}
$$
Taking telescope sum from 1 to  $T$  on the both sides of Eq.(30), we then have
where  $(i)$  holds by following  $\langle a,b\rangle \leq \frac{1}{2}\| a\| ^2 +\frac{1}{2}\| a\| ^2$ $(ii)$  holds due to Assumption 1,  $(iii)$  holds due to Assumption 1 owing to Eq.(21),  $(iii)$  holds since  $\mathbb{E}[\| a + b\| ^2 ] = \mathbb{E}[\| a\| ^2 ] + \mathbb{E}[\| b\| ^2 ]$  if  $\mathbb{E}[a] = 0$  or  $\mathbb{E}[b] = 0$ $(v)$  holds resulting from the updating rule in Eq. (14),  $(vi)$  holds due to  $\left|\frac{(m_t)_j}{(b_t)_j}\right|\leq 1,|\delta_j|\leq 2$  (the definition of  $\delta_{t}$  in Eq. (11)),  $\mathbb{E}[\| e_t\|^2 ]\leq d$  in Lemma 1,  $\| g_t\| \leq G$  in Assumption 2 and  $\min_j(b_t)_j\geq \rho >0$
We now analyze  $T_{5}$  below,
$$
T _ {6} = - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (x _ {t}), \alpha_ {t} \frac {g _ {t}}{b _ {t}} \rangle \right]
$$
$$
= - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (x _ {t}), \alpha_ {t} \frac {\nabla f (x _ {t})}{b _ {t}} \rangle \right] - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (x _ {t}), \alpha_ {t} \frac {g _ {t} - \nabla f (x _ {t})}{b _ {t}} \rangle \right]
$$
$$
\stackrel {(i)} {\leq} - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \nabla f (x _ {t}), \alpha_ {t} \frac {\nabla f (x _ {t}) - g _ {t}}{b _ {t}} \rangle \right]
$$
$$
= - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + \mathbb {E} \left[ \langle \nabla f (x _ {1}), \alpha_ {1} \frac {\nabla f (x _ {1}) - g _ {1}}{b _ {1}} \rangle \right]
$$
$$
+ \mathbb {E} \left[ \sum_ {t = 2} ^ {T} \langle \nabla f (x _ {t}), \nabla (f (x _ {t}) - g _ {t}) \odot \left(\frac {\alpha_ {t}}{b _ {t}} - \frac {\alpha_ {t - 1}}{b _ {t - 1}}\right) \rangle \right] + \mathbb {E} \left[ \sum_ {t = 2} ^ {T} \langle \nabla f (x _ {t}), \alpha_ {t - 1} \frac {\nabla f (x _ {t}) - g _ {t}}{b _ {t - 1}} \rangle \right]
$$
$$
\stackrel {(i i)} {=} - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + \mathbb {E} \left[ \langle \nabla f (x _ {1}), \alpha_ {1} \frac {\nabla f (x _ {1}) - g _ {1}}{b _ {1}} \rangle \right]
$$
$$
+ \mathbb {E} \left[ \sum_ {t = 2} ^ {T} \langle \nabla f (x _ {t}), (\nabla f (x _ {t}) - g _ {t}) \odot \left(\frac {\alpha_ {t}}{b _ {t}} - \frac {\alpha_ {t - 1}}{b _ {t - 1}}\right) \rangle \right]
$$
$$
\begin{array}{l} \stackrel {(i i i)} {\leq} - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + \mathbb {E} \left[ \| \nabla f (x _ {1}) \| \| \nabla f (x _ {1}) - g _ {1} \| \left\| \frac {\alpha_ {1}}{b _ {1}} \right\| _ {1} \right] \\ + \mathbb {E} \left[ \sum_ {t = 2} ^ {T} \| \nabla f (x _ {t}) \| \| \nabla f (x _ {t}) - g _ {t} \| \left\| \frac {\alpha_ {t}}{b _ {t}} - \frac {\alpha_ {t - 1}}{b _ {t - 1}} \right\| _ {1} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i v)} {\leq} - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + 2 G ^ {2} \mathbb {E} \left[ \left\| \frac {\alpha_ {1}}{b _ {1}} \right\| _ {1} + \sum_ {t = 2} ^ {T} \left\| \frac {\alpha_ {t}}{b _ {t}} - \frac {\alpha_ {t - 1}}{b _ {t - 1}} \right\| _ {1} \right] \\ \stackrel {(v)} {=} - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + 2 G ^ {2} \mathbb {E} \left[ \left\| \frac {\alpha_ {1}}{b _ {1}} + \sum_ {t = 2} ^ {T} \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right], \\ = - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + 4 G ^ {2} \mathbb {E} \left[ \left\| \frac {\alpha_ {1}}{b _ {1}} \right\| _ {1} \right], \\ \end{array}
$$
$$
\stackrel {(v i)} {\leq} - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + \frac {4 G ^ {2} \alpha_ {1} d}{\rho} \tag {35}
$$
where  $(i)$  holds due to  $\max_j(b_t)_j\leq \| b_t\| \leq G$  ,  $(ii)$  holds owing to  $\mathbb{E}[\nabla f(x_t) - g_t] = 0$  in Assumption 2 and  $g_{t},b_{t - 1}$  are independent,  $(iii)$  holds since  $\| a\odot b\| \leq \| a\|\max_j|(b)_j|\leq \| a\| \| b\| _1$ $(iv)$  holds resulting from  $\| \nabla f(x_t)\| \leq G$  and  $\| \nabla f(x_t) - g_t\| \leq \| \nabla f(x_t)\| +\| g_t\| \leq 2G$  , and  $(v)$  holds because  $\frac{\alpha_{t - 1}}{(b_{t - 1})_j} -\frac{\alpha_t}{(b_t)_j}\geq 0$  for any  $j\in [1,2,\dots,d]$ $(vi)$  holds due to  $\min_j(b_t)_j\geq \rho >0$  for any  $j\in [1,2,\ldots ,d]$
We then bound  $T_{6}$
Then, we pay attention to  $T_{3}$
$$
\begin{array}{l} T _ {3} = L \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\| \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \right\| ^ {2} \right] \\ \stackrel {(i)} {\leq} \frac {\beta^ {2} L}{(1 - \beta) ^ {2}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\| \begin{array}{c} \alpha_ {t - 1} \\ b _ {t - 1} \end{array} - \frac {\alpha_ {t}}{b _ {t}} \right\| ^ {2} \| m _ {t - 1} \| ^ {2} \right] \\ \stackrel {(i i)} {\leq} \frac {\beta^ {2} L G ^ {2}}{(1 - \beta) ^ {2}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| ^ {2} \right] \\ \stackrel {(i i i)} {\leq} \frac {\beta^ {2} L G ^ {2}}{(1 - \beta) ^ {2}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \max  _ {j} \left| \frac {\alpha_ {t - 1}}{\left(b _ {t - 1}\right) _ {j}} - \frac {\alpha_ {t}}{\left(b _ {t}\right) _ {j}} \right| \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right] \\ \stackrel {(i v)} {\leq} \frac {\alpha_ {0} \beta^ {2} L G ^ {2}}{\rho (1 - \beta) ^ {2}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \max  _ {j} \left(\frac {\alpha_ {t - 1}}{\left(b _ {t - 1}\right) _ {j}}\right) \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right] \tag {36} \\ \stackrel {(v)} {\leq} \frac {\alpha_ {0} \beta^ {2} L G ^ {2}}{\rho (1 - \beta) ^ {2}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right] \\ \stackrel {(v i)} {\leq} \frac {\alpha_ {0} \beta^ {2} L G ^ {2}}{\rho (1 - \beta) ^ {2}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\| \frac {\alpha_ {t - 1}}{b _ {t - 1}} \right\| _ {1} - \left\| \frac {\alpha_ {t}}{b _ {t}} \right\| _ {1} \right] \\ \stackrel {(v i i)} {\leq} \frac {\alpha_ {0} \beta^ {2} L G ^ {2}}{\rho (1 - \beta) ^ {2}} \mathbb {E} \left[ \left\| \frac {\alpha_ {0}}{b _ {0}} \right\| _ {1} - \left\| \frac {\alpha_ {T}}{b _ {T}} \right\| _ {1} \right] \\ \stackrel {(v i i)} {\leq} \frac {\alpha_ {0} ^ {2} \beta^ {2} L G ^ {2} d}{\rho^ {2} (1 - \beta) ^ {2}}, \\ \end{array}
$$
where  $(i)$  holds due to  $\|a \odot b\| \leq \|a\|\|b\|$ ,  $(ii)$  holds owing to  $\|m_{t-1}\| \leq G$ ,  $(ii)$  holds due to  $\|a\|^2 \leq \max_j |(a)_j||a||_1$ ,  $(iv)$  holds due to  $\frac{\alpha_{t-1}}{(b_{t-1})_j} - \frac{\alpha_t}{(b_t)_j} \geq 0$  and  $\frac{\alpha_t}{(b_t)_j} > 0$  for any  $j \in [1,2,\dots,d]$ ,  $(v)$  holds resulting from  $\min_j(b_t)_j \geq \rho > 0$  for any  $j$  and  $\alpha_t$  is non-increasing,  $(vi)$  holds resulting from  $\frac{\alpha_{t-1}}{(b_{t-1})_j} - \frac{\alpha_t}{(b_t)_j} \geq 0$  for any  $j \in [1,2,\dots,d]$ ,  $(vii)$  holds due to telescoping sum, and  $(viii)$  holds due to  $\min_j(b_t)_j \geq \rho > 0$  for any  $j \in [1,2,\dots,d]$ .
Now we turn attention to  $T_{4}$ ,
$$
\begin{array}{l} T _ {4} = L \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} ^ {2} \left\| \frac {g _ {t}}{b _ {t}} \right\| ^ {2} \right] + \frac {L}{2} \mathbb {E} [ \sum_ {t = 1} ^ {T} \| \alpha_ {t - 1} \delta_ {t - 1} \| ^ {2} ] + \frac {L}{2} \mathbb {E} [ \sum_ {t = 1} ^ {T} \| (\alpha_ {t - 1} - \alpha_ {t}) e _ {t} \| ^ {2} ] \tag {37} \\ \leq \left(L \frac {G ^ {2}}{\rho^ {2}} + 2 d L\right) \sum_ {t = 1} ^ {T} \alpha_ {t} ^ {2} + \frac {d L}{2} \sum_ {t = 1} ^ {T} \left(\alpha_ {t - 1} - \alpha_ {t}\right) ^ {2}, \\ \end{array}
$$
where the inequality holds owing to  $\| m_{t - 1}\| \leq G$  and  $\min_j(b_t)_j\geq \rho >0,\| (\delta_{t - 1})_j\| \leq 2$ , and  $\mathbb{E}[\| e_t\| ^2 ]\leq d$ . Combining Eq.(31-37), we can obtain
$$
\begin{array}{l} \mathbb {E} [ f (z _ {T}) - f (z _ {1}) ] \leq \frac {\alpha_ {0} \beta d}{(1 - \beta) \rho} G ^ {2} + \left(\frac {(8 \beta^ {2} + 1 0 \beta + 5) L ^ {2} d}{(1 - \beta) ^ {2}} + \frac {G ^ {2}}{2 \rho^ {2}}\right) \sum_ {t = 1} ^ {T} \alpha_ {t - 1} ^ {2} \\ - \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] + \frac {4 G ^ {2} \alpha_ {1} d}{\rho} + \frac {\alpha_ {0} ^ {2} \beta^ {2} L G ^ {2} d}{\rho^ {2} (1 - \beta) ^ {2}} \tag {38} \\ + \left(L \frac {G ^ {2}}{\rho^ {2}} + 2 d L\right) \sum_ {t = 1} ^ {T} \alpha_ {t} ^ {2} + \frac {d L}{2} \sum_ {t = 1} ^ {T} \left(\alpha_ {t - 1} - \alpha_ {t}\right) ^ {2}. \\ \end{array}
$$
Reformulating Eq.(38), we then have
$$
\begin{array}{l} \frac {1}{G} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \alpha_ {t} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq \mathbb {E} [ f (z _ {1}) - f (z _ {T}) ] \\ + \left(\frac {(8 \beta^ {2} + 1 0 \beta + 5) L ^ {2} d}{(1 - \beta) ^ {2}} + \frac {G ^ {2} (1 + L)}{2 \rho^ {2}} + 2 d L\right) \sum_ {t = 1} ^ {T} \alpha_ {t - 1} ^ {2} \tag {39} \\ + \frac {d L}{2} \sum_ {t = 1} ^ {T} \left(\alpha_ {t - 1} - \alpha_ {t}\right) ^ {2} \\ + \frac {\alpha_ {0} \beta d}{(1 - \beta) \rho} G ^ {2} + \frac {4 G ^ {2} \alpha_ {1} d}{\rho} + \frac {\alpha_ {0} ^ {2} \beta^ {2} L G ^ {2} d}{\rho^ {2} (1 - \beta) ^ {2}} \\ \end{array}
$$
It is known the learning rate saftifies  $\alpha_{t} = \frac{c}{\sqrt{t}}, \forall t \geq 1$  and  $\alpha_{0} = \alpha_{1} = c$ . Utilizing non-increasing  $\alpha_{t}$  and Cauchy-Schwarz inequality, we know  $\mathbb{E}\left[\sum_{t=1}^{T} \alpha_{t} \|\nabla f(x_{t})\|^{2}\right] \geq T \alpha_{T} \mathbb{E}\left[\frac{1}{T} \sum_{t=1}^{T} \|\nabla f(x_{t})\|\right]^{2} = \frac{\sqrt{T}}{c} \mathbb{E}\left[\frac{1}{T} \sum_{t=1}^{T} \|\nabla f(x_{t})\|\right]^{2}$ .  $\sum_{t=1}^{T} \alpha_{t-1}^{2} = \sum_{t=1}^{T} \frac{c^{2}}{t} \leq c^{2}(1 + \int_{1}^{T-1} \frac{1}{t} dt) \leq c^{2}(1 + \log T)$ , and  $\sum_{t=1}^{T} (\alpha_{t-1} - \alpha_{t})^{2} = \sum_{t=2}^{T} (\alpha_{t-1} - \alpha_{t})^{2} \leq \sum_{t=2}^{T} \frac{c^{2}}{4(t-1)^{3}} \leq \frac{c^{2}}{4}(1 + \int_{1}^{T-2} t^{-3} dt) = \frac{c^{2}}{4}\left(\frac{3}{2} - \frac{1}{2(T-2)}\right) \leq \frac{3c^{2}}{8}$ , we further have
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \| \nabla f (x _ {t}) \| \right] ^ {2} \leq \frac {C _ {1}}{\sqrt {T}} + \frac {C _ {2} (1 + \log T)}{\sqrt {T}}, \tag {40}
$$
$$
C _ {1} = c G \left(\mathbb {E} \left[ f \left(z _ {1}\right) - f ^ {*} \right] + \frac {3 c ^ {2} d L}{1 6} + \frac {\beta c d G ^ {2}}{(1 - \beta) \rho} + \frac {4 c d G ^ {2}}{\rho} + \frac {c ^ {2} \beta^ {2} L G ^ {2} d}{\rho^ {2} (1 - \beta) ^ {2}}\right), \tag {41}
$$
$$
C _ {2} = c ^ {3} G \left(\frac {\left(8 \beta^ {2} + 1 0 \beta + 5\right) L ^ {2} d}{(1 - \beta) ^ {2}} + \frac {G ^ {2} (1 + L)}{2 \rho^ {2}} + 2 d L\right). \tag {42}
$$
B Experiments for Comparing Vanilla SGD, SGDM, Adam, Birder and SoftSignSGD
To address the bottleneck in communication during distributed training, numerous gradient compression algorithms have been proposed, aiming to reduce the communication volume. Most of these algorithms can be reduced to Vanilla SGD without momentum if compression is not performed. Generally speaking, the epoch-wise convergence rate and inference performance a compressed algorithms is upper bounded by its uncompressed counterpart. In the experiments, we conducted empirical experiments to evaluate the training and inference performance of of Vanilla SGD, SGDM, Adam, Birder and its uncompressed version in training typical CNN-base, LSTM-base and Transformer-base DNNs.
Algorithm 1. SoftSignSGD
1: Input: model parameter  $x_0, x_1$ , the momentum  $m_0^{(i)} = 0$ ,  $b_0^{(i)} = 0$ , the exponential moving average factor  $\beta$ , the learning rate sequence  $\{\alpha_t\}$
2: for  $t = 1, \dots, T$  do
3: Randomly sample  $\xi_{t}$  and compute the gradient:  $g_{t} = \nabla f(x_{t};\xi_{t})$
4: Update the momentum  $m_t$ :  $m_t = \beta m_{t-1} + (1 - \beta) g_t$
5: Update the momentum  $b_{t}$ :  $b_{t} = \beta b_{t - 1} + (1 - \beta)|g_{t}|$
6: Update the model parameter  $x_{t + 1}$ :  $x_{t + 1} = x_t - \alpha_t\frac{m_t}{b_t}$
7: end for
We refer to the uncompressed Birder as SoftSignSGD. The implementation details for SoftSignSGD are presented in Algorithm 1. When comparing SoftSignSGD to Adam, there are two key differences. First, instead of using the square root of the exponential moving average of the squared gradient, denoted as  $\sqrt{v_t} = \sqrt{(1 - \beta_2)v_{t-1} + (1 - \beta_2)g_t^2}$ , SoftSignSGD utilizes the exponential moving average of the absolute gradient, represented as  $b_t = (1 - \beta)b_{t-1} + |g_t|$ . Second, in SoftSignSGD, the exponential moving factors for both the numerator  $m_t$  and the denominator  $b_t$  are the same. These differences ensure that each element of the updating amount in SoftSignSGD satisfies the condition  $-1 \leq \left(\frac{m_t}{b_t}\right)_j \leq 1$ .
B.1 Experimental Results for training ResNet-20
We evaluated the performance of five optimization algorithms: Vanilla SGD, SGDM, AdamW, Birder and SoftSignSGD, for training ResNet-20 on CIFAR100. Each batch consisted of a set of 128 examples sampled with replacement. For SGDM, we set the momentum parameter  $\beta$  to 0.9, while for SoftSignSGD and Birder, it was set to 0.95. For AdamW, the parameters  $\beta_{1}$  and  $\beta_{2}$  were set to 0.9 and 0.999, respectively. The weight decay was uniformly set to 0.0005 for Vanilla SGD and SGDM, and 0.05 for AdamW, Birder and SoftSignSGD. To simplify the tuning process and ensure fair comparisons, we initialized the learning rates at 0.1 for Vanilla SGD and SGDM, and 0.005 for AdamW, Birder and SoftSignSGD. We divided the learning rates by 10 after 75 and 130 epochs, and terminated the training after 150 epochs.
Figure 5 visually demonstrates that Vanilla SGD exhibits slower convergence speed and lower test accuracy compared to SGDM. In contrast, both Birder and SoftSignSGD show comparable training and inference performance to the commonly used SGDM and AdamW. This observation suggests that Birder outperforms existing gradient compression algorithms when training CNN-based ResNet-20 models.
(a) Train Loss
Figure 5: Training loss and test accuracy for ResNet-20 on CIFAR100.
B.2 Experimental Results for training LSTM
We conducted experiments to train a 3-layer LSTM model on the Penn TreeBank dataset to evaluate the performance of five optimization algorithms: Vanilla SGD, SGDM, AdamW, Birder and SoftSignSGD. Our implementations were built upon the code provided in the AdaBelief paper<sup>7</sup>, and we used the default experimental settings for SGDM and AdamW. For Vanilla SGD, we used the experimental settings of SGDM with the exception that we set the momentum parameter  $\beta$  to 0. For Birder and SoftSignSGD, we adopted the experimental settings of AdamW, except that we set the momentum parameter  $\beta$  to 0.99.
As visually illustrated in Figure 6, Vanilla SGD is still less effective than SGDM in terms of the convergence speed and the test accuracy, while the training and inference performance of SoftSignSGD and Birder are comparative to common-used SGDM and AdamW. It indicates the Birder is superior to exiting gradient compression algorithms for training LSTM.
B.3 Experimental Results for training ViT
We train ViT-B with Vanilla SGD, SGDM, AdamW, SoftSignSGD and Birder on the ILSVRC2012 with 32 GPUs (4 nodes). We use the Pytorch official implementation for ViT<sup>8</sup>. For AdamW, SoftSignSGD and Birder, we followed the recommended experimental settings, with the exception that we set the momentum parameter  $\beta$  to 0.95 for SoftSignSGD and Birder. As for Vanilla SGD and SGDM, we set the basic learning rate to 0.1 and the weight decay to 0.001, while keeping other settings the same as AdamW. Instead of the default 300 epochs, we uniformly set the total number of epochs to 150 for all optimizers
As visually illustrated in Figure 7, Vanilla SGD is still less effective than SGDM in terms of the convergence speed and the test accuracy, while the training and inference performance of SoftSignSGD and Birder are comparative to common-used SGDM and AdamW. Notably, the performance of SGD-type optimizers are substantially inferior to that of adaptive optimizers.
<sup>7</sup>https://github.com/juntang-zhuang/Adabelief-Optimizer
<sup>8</sup>https://github.com/pytorch/vision/tree/main/references/classification
(a) Train Loss
(b) Test Perplexity
Figure 6: Training loss and test perplexity (the lower, the better) for 3-layer LSTM on Penn TreeBank.
(a) Train Loss
Figure 7: Training loss and test accuracy for ViT-B-16 on ILSVRC2012.
C Experimental Results for pre-training BERT-Base
We employed BertAdam and Birder to pre-train BERT-Base on Wikipedia using 64 GPUs (8 nodes). The sequence length was set to 512, and the batch size per GPU was set to 16. The training process consisted of 37,000 iterations. The learning rate started at  $4 \times 10^{-4}$  and linearly increased in the first 12,500 iterations, after which it linearly decreased to 0 for the remaining iterations. For BertAdam, the parameter values  $[\beta_1, \beta_2]$  were set to [0.9, 0.999], and for Birder, the momentum parameter beta was set to 0.9.
As depicted in Figure 8, Birder demonstrates a comparable iteration-wise convergence rate to BertAdam. However, in terms of time-wise convergence, Birder achieves a  $4.2\mathrm{x}$  faster convergence rate compared to BertAdam.
D Experiments with InfiniBand connections