SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \stackrel {(i i i)} {=} (1 - \alpha) \mathbb {E} \left[ \left\| (1 - \eta) \left(w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})\right) + \nabla f _ {i} (x ^ {t + 1}) - g _ {i} ^ {t} \right\| ^ {2} \right] \\ + (1 - \alpha) \mathbb {E} \left[ \left\| \eta \mathcal {V} _ {i} ^ {t + 1} + (1 - \eta) \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] \\ = (1 - \alpha) \mathbb {E} \left[ \left\| \left(w _ {i} ^ {t} - g _ {i} ^ {t}\right) + \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) - \eta \left(w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + (1 - \alpha) \mathbb {E} \left[ \left\| \eta \mathcal {V} _ {i} ^ {t + 1} + (1 - \eta) \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i v)} {\leq} \quad (1 - \alpha) (1 + \rho) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - g _ {i} ^ {t} \right\| ^ {2} \right] \\ + (1 - \alpha) \left(1 + \rho^ {- 1}\right) \mathbb {E} \left[ \left\| \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) - \eta \left(w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + 2 (1 - \alpha) \eta^ {2} \mathbb {E} \left[ \left\| \mathcal {V} _ {i} ^ {t + 1} \right\| ^ {2} \right] + 2 (1 - \alpha) (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(v)} {=} \quad (1 - \theta) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - g _ {i} ^ {t} \right\| ^ {2} \right] \\ + \beta \mathbb {E} \left[ \left\| \left(\nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) - \eta \left(w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] \\ + 2 (1 - \alpha) \eta^ {2} \mathbb {E} \left[ \left\| \mathcal {V} _ {i} ^ {t + 1} \right\| ^ {2} \right] + 2 (1 - \alpha) (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(v i)} {\leq} \quad (1 - \theta) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - g _ {i} ^ {t} \right\| ^ {2} \right] + 2 \beta \eta^ {2} \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + 2 \beta \mathbb {E} \left[ \left\| \nabla f _ {i} \left(x ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] + 2 \ell_ {i} ^ {2} \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right] + 2 \eta^ {2} \sigma^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq (1 - \theta) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - g _ {i} ^ {t} \right\| ^ {2} \right] + 2 \beta \eta^ {2} \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] \\ + \left(2 \beta L _ {i} ^ {2} + \ell_ {i} ^ {2}\right) \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right] + 2 \eta^ {2} \sigma^ {2}, \\ \end{array}
$$
where  $(i)$  holds by Definition 1,  $(ii)$  follows from (60),  $(iii)$  holds by unbiasedness of  $\mathcal{V}_i^{t+1}$  and  $\mathcal{W}_i^t$  (57). In  $(iv)$  we use Young's inequality twice, in  $(v)$  we introduce the notation  $\theta := 1 - (1 - \alpha)(1 + \rho)$  and  $\beta := (1 - \alpha)(1 + \rho^{-1})$ , in  $(vi)$  we again use Young's inequality and the bound (58) and (59). The last step holds by smoothness of  $f_i(\cdot)$  (Assumption 1). The proof is complete by the choice  $\rho = \alpha/2$ , which guarantees  $1 - \theta \leq 1 - \alpha/2$ , and  $2\beta \leq 4/\alpha$ .
J Simplified Proof of SGDM: Time Varying Parameters and No Tuning for Momentum Sequence
In this section, we give a simplified proof of SGDM in the single node setting  $(n = 1)$  without compression  $(\alpha = 1)$ . The following theorem shows that the momentum parameter can be chosen in a parameter agnostic $^{22}$  way as  $\eta_t = 1 / \sqrt{t + 1}$  (or  $\eta_t = 1 / \sqrt{T + 1}$ ), instead of being a constant depending on problem parameters as it is suggested in our main Theorem 3. In other words, using SGDM with time varying momentum does not introduce any additional tuning of hyper-parameters.
Theorem 8. Let Assumptions 1, 2 hold. Let  $n = 1$  and Algorithm 1 run with identity compressor  $C$ , i.e.,  $\alpha = 1$ , and (possibly) time varying momentum  $\eta_t \in (0,1]$  and step-size parameters  $\gamma_t = \gamma \eta_t$  with  $\gamma \in (0,1/(3L)]$ . Let  $\hat{x}^T$  be sampled from the iterates of the algorithm with probabilities  $p_t = \eta_t / (\sum_{t=0}^{T-1} \eta_t)$ , then
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] \leq \frac {2 \Lambda_ {0} \gamma^ {- 1} + 2 \sigma^ {2} \sum_ {t = 0} ^ {T - 1} \eta_ {t} ^ {2}}{\sum_ {t = 0} ^ {T - 1} \eta_ {t}},
$$
where  $\Lambda_0\coloneqq f(x^0) - f^* +\gamma \mathbb{E}\left[\left\| v^0 -\nabla f(x^0)\right\| ^2\right]$  is the Lyapunov function.
Proof. By Lemma 2 denoting  $P_{t} \coloneqq \mathbb{E}\left[\| v^{t} - \nabla f(x^{t})\|^{2}\right]$ ,  $R_{t} \coloneqq \mathbb{E}\left[\left\| x^{t} - x^{t + 1}\right\|^{2}\right]$ , we have
$$
P _ {t + 1} \leq P _ {t} - \eta_ {t} P _ {t} + \frac {3 L ^ {2}}{\eta_ {t}} R _ {t} + \eta_ {t} ^ {2} \sigma^ {2}. \tag {62}
$$
By descent Lemma 1, we have for any  $\gamma_t > 0$
$$
\delta_ {t + 1} \leq \delta_ {t} - \frac {\gamma_ {t}}{2} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] - \frac {1}{2 \gamma_ {t}} (1 - \gamma_ {t} L) R _ {t} + \frac {\gamma_ {t}}{2} P _ {t}, \tag {63}
$$
where  $\delta_0\coloneqq \mathbb{E}\left[f(x^t) - f^*\right]$ . Define the Lyapunov function as  $\Lambda_t = \delta_0 + \gamma P_t$ . Then summing up (63) with a  $\gamma$  multiple of (62) and noticing that  $\gamma_t\leq \gamma$ , we get
$$
\Lambda_ {t + 1} \leq \Lambda_ {t} - \frac {\gamma_ {t}}{2} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] - \frac {1}{2 \gamma_ {t}} \left(1 - \gamma L - 6 \gamma^ {2} L ^ {2}\right) R _ {t} + \gamma \eta_ {t} ^ {2} \sigma^ {2}.
$$
Since  $\gamma \leq 1 / (3L)$ , we have  $1 - \gamma L - 6\gamma^2 L^2 \leq 0$ , and, therefore, by telescoping we can derive
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \left(\sum_ {t = 0} ^ {T - 1} \eta_ {t}\right) ^ {- 1} \sum_ {t = 0} ^ {T - 1} \eta_ {t} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ \leq \frac {2 \Lambda_ {0} \gamma^ {- 1} + 2 \sigma^ {2} \sum_ {t = 0} ^ {T - 1} \eta_ {t} ^ {2}}{\sum_ {t = 0} ^ {T - 1} \eta_ {t}}. \\ \end{array}
$$
The above theorem suggests that to ensure convergence, we can select any momentum sequence such that  $\sigma^2\sum_{t = 0}^{\infty}\eta_t^2 < \infty$ , and  $\sum_{t = 0}^{\infty}\eta_t^2\to \infty$  for  $t\to \infty$ . The parameter  $\gamma$ , which determines the step-size  $\gamma_{t} = \gamma \eta_{t}$ , should be set to  $\gamma = 1 / (3L)$  (to minimize the upper bound). Let us now consider some special cases.
Deterministic case. If  $\sigma = 0$ , we can set it to be any constant  $\eta_t = \eta \in (0,1]$  and derive
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] \leq \frac {2 \delta_ {0}}{\gamma \eta T} = \mathcal {O} \left(\frac {L \delta_ {0}}{\eta T}\right).
$$
22That is, independently of the problem specific parameters
Stochastic case. For  $\sigma^2 > 0$ , we can select time-varying  $\eta_t = \frac{1}{\sqrt{t + 1}}$  or constant  $\eta_t = \frac{1}{\sqrt{T + 1}}$ , which gives  $\sum_{t=0}^{T-1} \eta_t^2 = \mathcal{O}(\log(T))$ , and  $\sum_{t=0}^{T-1} \eta_t = \Omega(\sqrt{T})$ . Thus
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \widetilde {\mathcal {O}} \left(\frac {L \Lambda_ {0} + \sigma^ {2}}{\sqrt {T}}\right).
$$
Notice that if we set  $\eta_t$  as above, we do not need any tuning of momentum parameter. Only tuning of parameter  $\gamma$  is required to ensure convergence with optimal dependence on  $T$ , as in SGD without momentum. Of course, this rate is not yet optimal in other parameters, e.g.,  $\sigma^2$  and  $L$ . To make it optimal in all problem parameters, we can set  $\eta = \max \left\{1, \left(\frac{L\Lambda_0}{\sigma^2T}\right)^{1/2}\right\}$  similarly to the statement of Theorem 2.
K Revisiting EF14-SGD Analysis under BG and BGS Assumptions
In this section, we revisit the analysis of the original variant of error feedback (EF14-SGD) to showcase the difficulty in avoiding BG/BGS assumptions commonly used in the nonconvex analysis of this variant. In summary, the key reason for BG/BGS assumption is to bound the second term in (67) or (68).
Recall that EF14-SGD has the update rule [Stich et al., 2018]
$$
x ^ {t + 1} = x ^ {t} - g ^ {t}, \quad g ^ {t} = \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {i} ^ {t}, \tag {64}
$$
EF14-SGD:  $e_i^{t + 1} = e_i^t +\gamma \nabla f_i(x^t,\xi_i^t) - g_i^t,$  (65)
$$
g _ {i} ^ {t + 1} = \mathcal {C} \left(e _ {i} ^ {t + 1} + \gamma \nabla f _ {i} \left(x ^ {t + 1}, \xi_ {i} ^ {t + 1}\right)\right),
$$
where  $\{e_i^t\}_{t\geq 0}$  are error/memory sequences with  $e_i^0 = 0$  for each  $i = 1,\dots ,n$ . Let  $e^{t}:= \frac{1}{n}\sum_{i = 1}^{n}e_{i}^{t}$ . The proof of this method relies on so-called perturbed iterate analysis, for which one defines a "virtual sequence":  $\tilde{x}^t\coloneqq x^t -e^t$ . Then it is verified by direct substitution that for any  $t\geq 0$
$$
\tilde {x} ^ {t + 1} = \tilde {x} ^ {t} - \gamma \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f _ {i} (x ^ {t}, \xi_ {i} ^ {t}).
$$
If follows from Lemma 9 in [Stich and Karimireddy, 2021] that for any  $\gamma \leq 1/2L$  and  $t \geq 0$
$$
\mathbb {E} \left[ f (\tilde {x} ^ {t + 1}) \right] \leq \mathbb {E} \left[ f (\tilde {x} ^ {t}) \right] - \frac {\gamma}{4} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {\gamma L \sigma^ {2}}{2 n} + \frac {L ^ {2}}{2} \mathbb {E} \left[ \left\| e ^ {t} \right\| ^ {2} \right].
$$
Telescoping the recursion above and setting  $\delta_0\coloneqq f(x^0) - f^*$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla f \left(x ^ {t}\right) \right\| ^ {2} \right] \leq \frac {4 \delta_ {0}}{\gamma T} + \frac {2 \gamma L \sigma^ {2}}{n} + 2 L ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| e ^ {t} \right\| ^ {2} \right]. \tag {66}
$$
Now it remains to bound efficiently the average error term  $\mathbb{E}\left[\| e^t\|^2\right] = \mathbb{E}\left[\left\| \frac{1}{n}\sum_{i = 1}^{n}e_i^t\right\|^2\right]$ . By Jensen's inequality, we have
$$
\mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} e _ {i} ^ {t} \right\| ^ {2} \right] \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| e _ {i} ^ {t} \right\| ^ {2} \right],
$$
and develop a bound for each  $\mathbb{E}\left[\| e_i^t\|^2\right]$  individually. Denote by  $z\coloneqq e_i^t +\gamma \nabla f_i(x^t,\xi_i^t)$ , then
$$
\begin{array}{l} \mathbb {E} \left[ \left\| e _ {i} ^ {t + 1} \right\| ^ {2} \right] \leq \mathbb {E} \left[ \left\| \mathcal {C} (z) - z \right\| ^ {2} \right] \\ \leq (1 - \alpha) \mathbb {E} \left[ \left\| e _ {i} ^ {t} + \gamma \nabla f _ {i} (x ^ {t}, \xi_ {i} ^ {t}) \right\| ^ {2} \right] \\ \leq (1 - \alpha) \left(1 + \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| e _ {i} ^ {t} \right\| ^ {2} \right] + \left(1 + \frac {2}{\alpha}\right) \mathbb {E} \left[ \left\| \gamma \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) \right\| ^ {2} \right] \\ \leq \left(1 - \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| e _ {i} ^ {t} \right\| ^ {2} \right] + \frac {3 \gamma^ {2}}{\alpha} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) \right\| ^ {2} \right], \tag {67} \\ \end{array}
$$
where we used Definition 1 and Young's inequality.
BG assumption. If we assume bounded (stochastic) gradients (BG), i.e.,  $\mathbb{E}\left[\| \nabla f_i(x,\xi_i)\|^2\right] \leq G^2$  for all  $i = 1, \dots, n$ , then using (67) we can derive
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| e ^ {t} \right\| ^ {2} \right] \leq \frac {6 \gamma^ {2} G ^ {2}}{\alpha^ {2}}.
$$
Combining this bound with (66), we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \leq \frac {4 \delta_ {0}}{\gamma T} + \frac {2 \gamma L \sigma^ {2}}{n} + \frac {1 2 L ^ {2} \gamma^ {2} G ^ {2}}{\alpha^ {2}}.
$$
The step-size choice  $\gamma = \min \left\{\frac{1}{L},\left(\frac{\delta_0\alpha^2}{TL^2\sigma^2}\right)^{1 / 3},\left(\frac{n\delta_0}{TL\sigma^2}\right)^{1 / 2}\right\}$ , allows us to bound the RHS by  $\frac{12\delta_0}{\gamma T}$  and guarantees
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {L \delta_ {0}}{T} + \left(\frac {L \delta_ {0} G}{\alpha T}\right) ^ {2 / 3} + \left(\frac {L \delta_ {0}}{n T}\right) ^ {1 / 2}\right),
$$
or, equivalently,  $T = \mathcal{O}\left(\frac{L\delta_0}{\varepsilon^2} + \frac{L\delta_0G}{\alpha\varepsilon^3} + \frac{L\delta_0}{n\varepsilon^4}\right)$  sample complexity to find a stationary point. This analysis using BG assumption and derived sample complexity is essentially a simplified version of the one by Koloskova et al. [2020].<sup>23</sup>
BGS assumption. If we assume bounded gradient similarity (BGS), i.e.,  $\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\|\nabla f_i(x) - \nabla f(x)\|^2\right] \leq G^2$ , we can slightly modify the derivation in (67) as follows
$$
\begin{array}{l} \mathbb {E} \left[ \left\| e _ {i} ^ {t + 1} \right\| ^ {2} \right] \leq (1 - \alpha) \mathbb {E} \left[ \left\| e _ {i} ^ {t} + \gamma \nabla f _ {i} (x ^ {t}, \xi_ {i} ^ {t}) \right\| ^ {2} \right] \\ { = } { ( 1 - \alpha ) \mathbb { E } \left[ \left\| e _ { i } ^ { t } + \gamma \nabla f _ { i } ( x ^ { t } ) \right\| ^ { 2 } \right] + ( 1 - \alpha ) \gamma ^ { 2 } \mathbb { E } \left[ \left\| \nabla f _ { i } ( x ^ { t } , \xi _ { i } ^ { t } ) - \nabla f _ { i } ( x ^ { t } ) \right\| ^ { 2 } \right] } \\ \leq (1 - \alpha) \left(1 + \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| e _ {i} ^ {t} \right\| ^ {2} \right] + \left(1 + \frac {2}{\alpha}\right) \mathbb {E} \left[ \left\| \gamma \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] + \gamma^ {2} \sigma^ {2} \\ \leq \left(1 - \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| e _ {i} ^ {t} \right\| ^ {2} \right] + \frac {3 \gamma^ {2}}{\alpha} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] + \gamma^ {2} \sigma^ {2}. \tag {68} \\ \end{array}
$$
Averaging the above inequalities over  $i = 1, \ldots, n$  and using BGS assumption, i.e.,  $\frac{1}{n} \sum_{i=1}^{n} \mathbb{E}\left[\|\nabla f_i(x)\|^2\right] \leq \|\nabla f(x)\|^2 + G^2$ , we can derive via averaging over  $t = 0, \ldots, T - 1$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| e ^ {t} \right\| ^ {2} \right] \leq \frac {6 \gamma^ {2}}{\alpha^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {6 \gamma^ {2} G ^ {2}}{\alpha^ {2}} + \frac {2 \gamma^ {2} \sigma^ {2}}{\alpha},
$$
Combining the above inequality with (66), we have
$$
\left(1 - \frac {1 2 L ^ {2} \gamma^ {2}}{\alpha^ {2}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \leq \frac {4 \delta_ {0}}{\gamma T} + \frac {2 \gamma L \sigma^ {2}}{n} + \frac {1 2 \gamma^ {2} L ^ {2} G ^ {2}}{\alpha^ {2}} + \frac {4 \gamma^ {2} L ^ {2} \sigma^ {2}}{\alpha}.
$$
By setting  $\gamma = \min \left\{\frac{\alpha}{4L},\left(\frac{n\delta_0}{L\sigma^2T}\right)^{1 / 2},\left(\frac{\alpha^2\delta_0}{L^2G^2T}\right)^{1 / 3},\left(\frac{\alpha\delta_0}{L^2\sigma^2T}\right)^{1 / 3}\right\}$ , we have  $\left(1 - \frac{12L^2\gamma^2}{\alpha^2}\right)\geq \frac{1}{4}$ , and the RHS is at most  $\frac{16\delta_0}{\gamma T}$ . Therefore,
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {L \delta_ {0}}{\alpha T} + \left(\frac {L \delta_ {0} G}{\alpha T}\right) ^ {2 / 3} + \left(\frac {L \delta_ {0} \sigma}{\sqrt {\alpha} T}\right) ^ {2 / 3} + \left(\frac {L \delta_ {0}}{n T}\right) ^ {1 / 2}\right),
$$
or, equivalently,  $T = \mathcal{O}\left(\frac{L\delta_0}{\alpha\varepsilon^2} + \frac{L\delta_0G}{\alpha\varepsilon^3} + \frac{L\delta_0\sigma}{\sqrt{\alpha}\varepsilon^3} + \frac{L\delta_0}{n\varepsilon^4}\right)$  sample complexity. Notice that in the single node case ( $n = 1$ ), we have  $G = 0$ , and by Young's inequality  $\left(\frac{L\delta_0\sigma}{\sqrt{\alpha}T}\right)^{2/3} \leq \frac{1}{3}\frac{L\delta_0}{\alpha T} + \frac{2}{3}\left(\frac{L\delta_0\sigma^2}{T}\right)^{1/2}$ . Therefore, the above rate recovers the one by Stich and Karimireddy [2021] in the single node setting.
$^{23}$ Up to a smoothness constant and the fact that Koloskova et al. [2020] works in a more general decentralized setting.