SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u _ {i} ^ {t} - v _ {i} ^ {t} \right\| ^ {2} \right] \leq (1 - \eta) \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} \right\| ^ {2} \right] + 6 \eta \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t - 1}\right) \right\| ^ {2} \right] \\ + 6 L _ {i} ^ {2} \eta \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t - 1} \right\| ^ {2} \right] + \eta^ {2} \sigma^ {2}, \tag {37} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u ^ {t} - v ^ {t} \right\| ^ {2} \right] \leq (1 - \eta) \mathbb {E} \left[ \left\| u ^ {t - 1} - v ^ {t - 1} \right\| ^ {2} \right] + 6 \eta \mathbb {E} \left[ \left\| v ^ {t - 1} - \nabla f (x ^ {t - 1}) \right\| ^ {2} \right] \\ + 6 L ^ {2} \eta \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t - 1} \right\| ^ {2} \right] + \frac {\eta^ {2} \sigma^ {2}}{n}. \tag {38} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u _ {i} ^ {t} - v _ {i} ^ {t} \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - v _ {i} ^ {t} + \eta (v _ {i} ^ {t} - u _ {i} ^ {t - 1}) \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| v _ {i} ^ {t} - u _ {i} ^ {t - 1} \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| (1 - \eta) v _ {i} ^ {t - 1} + \eta \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) - u _ {i} ^ {t - 1} \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| \left(u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1}\right) + \eta \left(v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right)\right) \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| \left(u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1}\right) + \eta \left(v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t}\right)\right) + \eta \left(\nabla f _ {i} \left(x ^ {t}\right) - \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right)\right) \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \mathbb {E} _ {\xi_ {i} ^ {t}} \left[ \left\| \left(u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1}\right) + \eta \left(v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t}\right)\right) + \eta \left(\nabla f _ {i} \left(x ^ {t}\right) - \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right)\right) \right\| ^ {2} \right] \right] \\ = (1 - \eta) ^ {2} \left(\mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right\| ^ {2} \right] + \eta^ {2} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right]\right) \\ \leq (1 - \eta) ^ {2} \mathbb {E} \left[ \left| \left| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t}\right)\right) \right| \right| ^ {2} \right] + \eta^ {2} \sigma^ {2} \\ \leq (1 - \eta) ^ {2} \left(1 + \frac {\eta}{2}\right) \mathbb {E} \left[ \left| \left| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} \right| \right| ^ {2} \right] \\ + \left(1 + \frac {2}{\eta}\right) \eta^ {2} \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] + \eta^ {2} \sigma^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq (1 - \eta) \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} \right\| ^ {2} \right] + 3 \eta \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] + \eta^ {2} \sigma^ {2} \\ \leq (1 - \eta) \mathbb {E} \left[ \left| \left| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} \right| \right| ^ {2} \right] + 6 \eta \mathbb {E} \left[ \left| \left| v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t - 1}\right) \right| \right| ^ {2} \right] \\ + 6 \eta \mathbb {E} \left[ \left\| \nabla f _ {i} (x ^ {t}) - \nabla f _ {i} (x ^ {t - 1}) \right\| ^ {2} \right] + \eta^ {2} \sigma^ {2} \\ \leq (1 - \eta) \mathbb {E} \left[ \left| \left| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} \right| \right| ^ {2} \right] + 6 \eta \mathbb {E} \left[ \left| \left| v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t - 1}\right) \right| \right| ^ {2} \right] \\ + 6 L _ {i} ^ {2} \eta \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t - 1} \right\| ^ {2} \right] + \eta^ {2} \sigma^ {2}, \\ \end{array}
$$
where the first inequality holds Assumption 2, the second inequality holds by Young's inequality, and the last step uses smoothness of  $f_{i}(\cdot)$  (Assumption 1), which concludes the proof of (37).
For each  $t = 0, \dots, T - 1$ , define a random vector  $\xi^t := (\xi_1^t, \dots, \xi_n^t)$  and denote by  $\nabla f(x^t, \xi^{t+1}) := \frac{1}{n} \sum_{i=1}^{n} \nabla f_i(x^t, \xi_i^t)$ . Note that the entries of the random vector  $\xi^t$  are independent and  $\mathbb{E}_{\xi^t}[\nabla f(x^t, \xi^t)] = \nabla f(x^t)$ , then we have
$$
u _ {i} ^ {t} = u _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t} - u _ {i} ^ {t - 1}\right),
$$
where  $v^t \coloneqq \frac{1}{n} \sum_{i=1}^{n} v_i^t$ ,  $u^t \coloneqq \frac{1}{n} \sum_{i=1}^{n} u_i^t$  are auxiliary sequences. Therefore, we can similarly derive
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u ^ {t} - v ^ {t} \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| u ^ {t - 1} - v ^ {t} + \eta (v ^ {t} - u ^ {t - 1}) \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| v ^ {t} - u ^ {t - 1} \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| (1 - \eta) v ^ {t - 1} + \eta \nabla f (x ^ {t}, \xi^ {t}) - u ^ {t - 1} \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| \left(u ^ {t - 1} - v ^ {t - 1}\right) + \eta \left(v ^ {t - 1} - \nabla f \left(x ^ {t}, \xi^ {t}\right)\right) \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| \left(u ^ {t - 1} - v ^ {t - 1}\right) + \eta \left(v ^ {t - 1} - \nabla f \left(x ^ {t}\right)\right) + \eta (\nabla f \left(x ^ {t}\right) - \nabla f \left(x ^ {t}, \xi^ {t}\right)) \right\| ^ {2} \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \mathbb {E} _ {\xi^ {t}} \left[ \left\| \left(u ^ {t - 1} - v ^ {t - 1}\right) + \eta \left(v ^ {t - 1} - \nabla f \left(x ^ {t}\right)\right) + \eta \left(\nabla f \left(x ^ {t}\right) - \nabla f \left(x ^ {t}, \xi^ {t}\right)\right) \right\| ^ {2} \right] \right] \\ = (1 - \eta) ^ {2} \left(\mathbb {E} \left[ \left\| u ^ {t - 1} - v ^ {t - 1} + \eta \left(v ^ {t - 1} - \nabla f \left(x ^ {t}\right)\right) \right\| ^ {2} \right] + \eta^ {2} \mathbb {E} \left[ \left\| \nabla f \left(x ^ {t}, \xi^ {t}\right) - \nabla f \left(x ^ {t}\right) \right\| ^ {2} \right]\right) \\ \leq (1 - \eta) ^ {2} \mathbb {E} \left[ \left| \left| u ^ {t - 1} - v ^ {t - 1} + \eta \left(v ^ {t - 1} - \nabla f \left(x ^ {t}\right)\right) \right| \right| ^ {2} \right] + \frac {\eta^ {2} \sigma^ {2}}{n} \\ \leq (1 - \eta) ^ {2} \left(1 + \frac {\eta}{2}\right) \mathbb {E} \left[ \left| \left| u ^ {t - 1} - v ^ {t - 1} \right| \right| ^ {2} \right] \\ + \left(1 + \frac {2}{\eta}\right) \eta^ {2} \mathbb {E} \left[ \left\| v ^ {t - 1} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {\eta^ {2} \sigma^ {2}}{n} \\ \leq (1 - \eta) \mathbb {E} \left[ \left\| u ^ {t - 1} - v ^ {t - 1} \right\| ^ {2} \right] + 3 \eta \mathbb {E} \left[ \left\| v ^ {t - 1} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {\eta^ {2} \sigma^ {2}}{n} \\ \leq (1 - \eta) \mathbb {E} \left[ \left\| u ^ {t - 1} - v ^ {t - 1} \right\| ^ {2} \right] + 6 \eta \mathbb {E} \left[ \left\| v ^ {t - 1} - \nabla f (x ^ {t - 1}) \right\| ^ {2} \right] \\ + 6 \eta \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) - \nabla f (x ^ {t - 1}) \right\| ^ {2} \right] + \frac {\eta^ {2} \sigma^ {2}}{n} \\ \leq (1 - \eta) \mathbb {E} \left[ \left\| u ^ {t - 1} - v ^ {t - 1} \right\| ^ {2} \right] + 6 \eta \mathbb {E} \left[ \left\| v ^ {t - 1} - \nabla f (x ^ {t - 1}) \right\| ^ {2} \right] \\ + 6 L ^ {2} \eta \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t - 1} \right\| ^ {2} \right] + \frac {\eta^ {2} \sigma^ {2}}{n}, \\ \end{array}
$$
where the first inequality holds Assumption 2, the second inequality holds by Young's inequality, and the last step uses smoothness of  $f(\cdot)$  (Assumption 1), which concludes the proof of (38).
G.2 Controlling the error of contractive compression and double momentum estimator
Lemma 5. Let Assumption 1 be satisfied, and suppose  $\mathcal{C}$  is a contractive compressor. For every  $i = 1,\dots ,n$ , let the sequences  $\{v_i^t\}_{t\geq 0},\{u_i^t\}_{t\geq 0}$ , and  $\{g_i^t\}_{t\geq 0}$  be updated via
$$
g _ {i} ^ {t} = g _ {i} ^ {t - 1} + \mathcal {C} \left(u _ {i} ^ {t} - g _ {i} ^ {t - 1}\right).
$$
Then for every  $i = 1,\dots ,n$  and  $t\geq 0$  it holds
$$
\begin{array}{l} \mathbb {E} \left[ \left\| g _ {i} ^ {t} - u _ {i} ^ {t} \right\| ^ {2} \right] \leq \left(1 - \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| g _ {i} ^ {t - 1} - u _ {i} ^ {t - 1} \right\| ^ {2} \right] + \frac {6 \eta^ {2}}{\alpha} \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - v _ {i} ^ {t - 1} \right\| ^ {2} \right] \tag {39} \\ + \frac {6 \eta^ {4}}{\alpha} \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t - 1}, \xi_ {i} ^ {t - 1}\right) \right\| ^ {2} \right] + \frac {6 L _ {i} ^ {2} \eta^ {4}}{\alpha} \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t - 1} \right\| ^ {2} \right] + \eta^ {4} \sigma^ {2}. \\ \end{array}
$$
Proof. By the update rules of  $g_i^t$ ,  $u_i^t$  and  $v_i^t$ , we derive
$$
\begin{array}{l} \mathbb {E} \left[ \left\| g _ {i} ^ {t} - u _ {i} ^ {t} \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| g _ {i} ^ {t - 1} - u _ {i} ^ {t} + \mathcal {C} (u _ {i} ^ {t} - g _ {i} ^ {t - 1}) \right\| ^ {2} \right] \\ \stackrel {(i)} {\leq} \quad (1 - \alpha) \mathbb {E} \left[ \left\| u _ {i} ^ {t} - g _ {i} ^ {t - 1} \right\| ^ {2} \right] \\ \stackrel {(i i)} {=} (1 - \alpha) \mathbb {E} \left[ \left| \left| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1}\right) + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) - v _ {i} ^ {t - 1}\right) \right| \right| ^ {2} \right] \\ = (1 - \alpha) \mathbb {E} \left[ \| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1}\right) + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}\right) - v _ {i} ^ {t - 1}\right) \right. \\ \left. + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \| ^ {2} \right] \\ = (1 - \alpha) \mathbb {E} \left[ \| \mathbb {E} _ {\xi_ {i} ^ {t}} \left[ \| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1}\right) + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}\right) - v _ {i} ^ {t - 1}\right) \right. \right. \\ \left. + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) - \nabla f _ {i} \left(x ^ {t}\right)\right) \| ^ {2} \right] \bigg ] \\ = (1 - \alpha) \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} + \eta \left(v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1}\right) + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}\right) - v _ {i} ^ {t - 1}\right) \right\| ^ {2} \right] \\ + (1 - \alpha) \eta^ {4} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t}\right) - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i i)} {\leq} \quad (1 - \alpha) (1 + \rho) \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} \right\| ^ {2} \right] \\ + (1 - \alpha) (1 + \rho^ {- 1}) \mathbb {E} \left[ \left\| \eta \left(v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1}\right) + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t}\right) - v _ {i} ^ {t - 1}\right) \right\| ^ {2} \right] \\ + \eta^ {4} \sigma^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i v)} {=} (1 - \theta) \mathbb {E} \left[ \left| \left| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} \right| \right| ^ {2} \right] + \eta^ {4} \sigma^ {2} \\ + \beta \mathbb {E} \left[ \right.\left\| \right. \eta \left(v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1}\right) + \eta^ {2} \left(\nabla f _ {i} \left(x ^ {t - 1}\right) - v _ {i} ^ {t - 1}\right) + \eta^ {2} \left( \right.\nabla f _ {i} \left(x ^ {t}\right) - \nabla f _ {i} \left(x ^ {t - 1}\right)\left. \right\| ^ {2} \left. \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(v)} {\leq} (1 - \theta) \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} \right\| ^ {2} \right] + 3 \beta \eta^ {2} \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1} \right\| ^ {2} \right] \\ + 3 \beta \eta^ {4} \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - \nabla f _ {i} \left(x ^ {t - 1}\right) \right\| ^ {2} \right] \\ + 3 \beta \eta^ {4} \mathbb {E} \left[ \left\| \nabla f _ {i} (x ^ {t}) - \nabla f _ {i} (x ^ {t - 1}) \right\| ^ {2} \right] + \eta^ {4} \sigma^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \leq (1 - \theta) \mathbb {E} \left[ \left\| u _ {i} ^ {t - 1} - g _ {i} ^ {t - 1} \right\| ^ {2} \right] + 3 \beta \eta^ {2} \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - u _ {i} ^ {t - 1} \right\| ^ {2} \right] \\ + 3 \beta \eta^ {4} \mathbb {E} \left[ \left\| v _ {i} ^ {t - 1} - \nabla f _ {i} (x ^ {t - 1}) \right\| ^ {2} \right] \\ + 3 \beta L _ {i} ^ {2} \eta^ {4} \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t - 1} \right\| ^ {2} \right] + \eta^ {4} \sigma^ {2} \\ \end{array}
$$
where  $(i)$  is due to definition of a contractive compressor (Definition 1),  $(ii)$  follows by the update rule of  $v_{i}^{t}$  and  $u_{i}^{t}$ ,  $(iii)$  and  $(v)$  hold by Young's inequality for any  $\rho >0$ . In  $(iv)$ , we introduced
the notation  $\theta := 1 - (1 - \alpha)(1 + \rho)$ , and  $\beta := (1 - \alpha)(1 + \rho^{-1})$ . The last step follows by smoothness of  $f_{i}(\cdot)$  (Assumption 1). The proof is complete by the choice  $\rho = \alpha /2$ , which guarantees  $1 - \theta \leq 1 - \alpha /2$ , and  $3\beta \leq 6 / \alpha$ .
H EF21-SGDM with Absolute Compressor
In this section, we complement our theory by analyzing EF21-SGDM under a different class of widely used biased compressors, namely, absolute compressors, which are defined as follows.
Definition 2 (Absolute compressors). We say that a (possibly randomized) map  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}^d$  is an absolute compression operator if there exists a constant  $\Delta >0$  such that
$$
\mathbb {E} \left[ \| \mathcal {C} (x) - x \| ^ {2} \right] \leq \Delta^ {2}, \quad \forall x \in \mathbb {R} ^ {d}. \tag {40}
$$
This class includes important examples of compressors such as hard-threshold sparsifier [Sahu et al., 2021], (stochastic) rounding schemes with bounded error [Gupta et al., 2015] and scaled integer rounding [Sapio et al., 2021].
Algorithm 4 EF21-SGDM (abs)
1: Input: starting point  $x^0$ , step-size  $\gamma > 0$ , momentum  $\eta \in (0,1]$ , initial batch size  $B_{\mathrm{init}}$
2: Initialize  $v_{i}^{0} = g_{i}^{0} = \frac{1}{B_{\mathrm{init}}} \sum_{j=1}^{B_{\mathrm{init}}} \nabla f_{i}(x^{0}, \xi_{i,j}^{0})$  for  $i = 1, \dots, n$ ;  $g^{0} = \frac{1}{n} \sum_{i=1}^{n} g_{i}^{0}$
3: for  $t = 0,1,2,\ldots ,T - 1$  do
4: Master computes  $x^{t + 1} = x^t - \gamma g^t$  and broadcasts  $x^{t + 1}$  to all nodes
5: for all nodes  $i = 1,\dots ,n$  in parallel do
6: Compute momentum estimator  $v_{i}^{t + 1} = (1 - \eta)v_{i}^{t} + \eta \nabla f_{i}(x^{t + 1},\xi_{i}^{t + 1})$
7: Compress  $c_i^{t+1} = \mathcal{C}\left(\frac{v_i^{t+1} - g_i^t}{\gamma}\right)$  and send  $c_i^{t+1}$  to the master
8: Update local state  $g_{i}^{t + 1} = g_{i}^{t} + \gamma c_{i}^{t + 1}$
9: end for
10: Master computes  $g^{t + 1} = \frac{1}{n}\sum_{i = 1}^{n}g_{i}^{t + 1}$  via  $g^{t + 1} = g^t +\frac{1}{n}\sum_{i = 1}^{n}\gamma c_i^{t + 1}$
1: end for
To accommodate absolute compressors into our EF21-SGDM method, we need to make a slight modification to our algorithm, see Algorithm 4. At each iteration, before compressing the difference  $v_{i}^{t + 1} - g_{i}^{t}$ , we divide it by the step-size  $\gamma$ . Later, we multiply the compressed vector  $c_{i}^{t + 1}$  by  $\gamma$ , i.e., have
$$
g _ {i} ^ {t + 1} = g _ {i} ^ {t} + \gamma \mathcal {C} \left(\frac {v _ {i} ^ {t + 1} - g _ {i} ^ {t}}{\gamma}\right).
$$
Such modification is necessary for absolute compressors because by Definition 2 the compression error is not proportional to  $\| x\|^2$ , but merely an absolute constant  $\Delta^2$ . In fact, Algorithm 4 is somewhat more universal in the sense that it can be also applied for contractive compressors. We derive the following result for EF21-SGDM (abs).
Theorem 6. Let Assumptions 1 and 2 hold. Let  $\hat{x}^T$  be sampled uniformly at random from the iterates of the method. Let Algorithm 4 run with an absolute compressor (Definition 2). For all  $\eta \in (0,1]$  and  $B_{\mathrm{init}} \geq 1$ , with  $\gamma \leq \frac{\eta}{4L}$ , we have
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] \leq \mathcal {O} \left(\frac {\Psi_ {0}}{\gamma T} + \gamma^ {2} \Delta^ {2} + \frac {\eta \sigma^ {2}}{n}\right), \tag {41}
$$
where  $\Psi_0\coloneqq \delta_0 + \frac{\gamma}{\eta}\mathbb{E}\left[\| v^0 -\nabla f(x^0)\| ^2\right]$  is a Lyapunov function. With the following step-size, momentum parameter, and initial batch size
$$
\gamma = \frac {\eta}{4 L}, \quad \eta = \min  \left\{1, \left(\frac {L ^ {3} \delta_ {0}}{\Delta^ {2} T}\right) ^ {1 / 3}, \left(\frac {L \delta_ {0} n}{\sigma^ {2} T}\right) ^ {1 / 2} \right\}, \quad B _ {i n i t} = \frac {\sigma^ {2}}{L \delta_ {0} n} \tag {42}
$$
$$
\underline {{\mathbb {E}}} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] \leq \mathcal {O} \left(\frac {L \delta_ {0}}{T} + \left(\frac {\delta_ {0} \Delta}{T}\right) ^ {2 / 3} + \left(\frac {L \delta_ {0} \sigma^ {2}}{n T}\right) ^ {1 / 2}\right).
$$
20It is straightforward to modify the proof of our Theorem 3 for the case when Algorithm 4 is applied with a contractive compressor.
Corollary 4. Under the setting of Theorem 6, we have  $\mathbb{E}\left[\|\nabla f(\hat{x}^T)\|\right] \leq \varepsilon$  after  $T = \mathcal{O}\left(\frac{L\delta_0}{\varepsilon^2} + \frac{\Delta\delta_0}{\varepsilon^3} + \frac{\sigma^2L\delta_0}{n\varepsilon^4}\right)$  iterations.
Remark 3. The sample complexity result in Corollary 4 matches the one derived for DoubleSqueeze algorithm [Tang et al., 2020], which is different from Algorithm 4.
Proof. Similarly to the proof of Theorem 3, we control the error between  $g^t$  and  $\nabla f(x^t)$  by decomposing it into two terms
$$
\left\| g ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \leq 2 \left\| g ^ {t} - v ^ {t} \right\| ^ {2} + 2 \left\| v ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \leq 2 \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - v _ {i} ^ {t} \right\| ^ {2} + 2 \left\| v ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}.
$$
Again, for the second term above we can use the recursion developed for momentum estimator Lemma 2. However, since we use a different compressor here, we need to bound  $\| g_i^t - v_i^t\|^2$  term differently, thus we invoke Lemma 6 for absolute compressor.
Part I. Controlling the error of momentum estimator on average for  $v^t$ . Denote  $P_t \coloneqq \mathbb{E}\left[\|v^t - \nabla f(x^t)\|^2\right]$ ,  $R_t \coloneqq \mathbb{E}\left[\|x^t - x^{t+1}\|^2\right]$ . Similarly to Part I of the proof of Theorem 3, we have by Lemma 2
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} P _ {t} \leq \frac {3 L ^ {2}}{\eta^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} + \frac {\eta \sigma^ {2}}{n} + \frac {1}{\eta T} P _ {0}. \tag {43}
$$
Part II. Controlling the error of absolute compressor and momentum estimator. By Lemma 6 we have for any  $0 < \eta \leq 1$  and  $t \geq 0$
$$
\widetilde {V} _ {t} := \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| g _ {i} ^ {t} - v _ {i} ^ {t} \right\| ^ {2} \right] \leq \gamma^ {2} \Delta^ {2}. \tag {44}
$$
$$
\begin{array}{l} f \left(x ^ {t + 1}\right) \leq f \left(x ^ {t}\right) - \frac {\gamma}{2} \left\| \nabla f \left(x ^ {t}\right) \right\| ^ {2} - \frac {1}{4 \gamma} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| g ^ {t} - \nabla f \left(x ^ {t}\right) \right\| ^ {2} \tag {45} \\ \leq f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \frac {1}{4 \gamma} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \widetilde {V} _ {t} + \gamma P _ {t}. \\ \end{array}
$$
Subtracting  $f^*$  from both sides of (45), taking expectation and defining  $\delta_t \coloneqq \mathbb{E}\left[f(x^t) - f^*\right]$ , we derive
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \\ \leq \frac {2 \delta_ {0}}{\gamma T} + 2 \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \widetilde {V} _ {t} + 2 \frac {1}{T} \sum_ {t = 0} ^ {T - 1} P _ {t} - \frac {1}{2 \gamma^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} \\ \stackrel {(i)} {\leq} \quad \frac {2 \delta_ {0}}{\gamma T} + 2 \gamma^ {2} \Delta^ {2} + 2 \frac {1}{T} \sum_ {t = 0} ^ {T - 1} P _ {t} - \frac {1}{2 \gamma^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} \\ \stackrel {(i i)} {\leq} \frac {2 \delta_ {0}}{\gamma T} + 2 \gamma^ {2} \Delta^ {2} + \left(\frac {6 L ^ {2}}{\eta^ {2}} - \frac {1}{2 \gamma^ {2}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} + \frac {2 \eta \sigma^ {2}}{n} + \frac {1}{\eta T} P _ {0} \\ \leq \frac {2 \delta_ {0}}{\gamma T} + 2 \gamma^ {2} \Delta^ {2} + \frac {2 \eta \sigma^ {2}}{n} + \frac {1}{\eta T} P _ {0} \tag {46} \\ \end{array}
$$
where in (i) and (ii) we apply (43), (44), and in the last step we use the assumption on the step-size  $\gamma \leq \eta /(4L)$ .
Setting  $\gamma = \frac{\eta}{4L}$ , and taking  $\eta \leq \left(\frac{L^3\delta_0}{\Delta^2T}\right)^{1/3}$  we can ensure that  $\frac{\eta^2\Delta^2}{L^2} \leq \frac{L\delta_0}{\eta T}$ , since  $\eta \leq \left(\frac{L\delta_0n}{\sigma^2T}\right)^{1/2}$  we have  $\frac{\eta\sigma^2}{n} \leq \frac{L\delta_0}{\eta T}$ . Finally, by setting the initial batch-size to  $B_{init} = \frac{\sigma^2}{L\delta_0n}$ , we have  $\frac{1}{\eta T} P_0 = \frac{\sigma^2}{\eta T n B_{init}} \leq \frac{L\delta_0}{\eta T}$ . Therefore, we derive
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| \nabla f (x ^ {t}) \right\| ^ {2} \right] \leq \frac {2 \delta_ {0}}{\gamma T} + 2 \gamma^ {2} \Delta^ {2} + \frac {2 \eta \sigma^ {2}}{n} + \frac {1}{\eta T} P _ {0} \\ = \frac {8 L \delta_ {0}}{\eta T} + \frac {\eta^ {2} \Delta^ {2}}{8 L ^ {2}} + \frac {2 \eta \sigma^ {2}}{n} + \frac {\sigma^ {2}}{\eta T B _ {i n i t}} \\ = \mathcal {O} \left(\frac {L \delta_ {0}}{T} + \frac {\delta_ {0} ^ {2 / 3} \Delta^ {2 / 3}}{T ^ {2 / 3}} + \frac {\sigma (L \delta_ {0}) ^ {1 / 2}}{(n T) ^ {1 / 2}}\right). \tag {47} \\ \end{array}
$$
H.1 Controlling the error of absolute compression
Lemma 6. Let  $\mathcal{C}$  be an absolute compressor and  $g_i^{t+1}$  be updated according to Algorithm 4, then for  $t \geq 0$ , we have  $\frac{1}{n} \sum_{i=1}^{n} \mathbb{E}\left[\|g_i^t - v_i^t\|^2\right] \leq \gamma^2 \Delta^2$ .
Proof. By the update rule for  $g_i^{t+1}$  in Algorithm 4 and Definition 2, we can bound
$$
\begin{array}{l} \mathbb {E} \left[ \left\| g _ {i} ^ {t + 1} - v _ {i} ^ {t + 1} \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| \gamma \mathcal {C} \left(\frac {v _ {i} ^ {t + 1} - g _ {i} ^ {t}}{\gamma}\right) - \left(v _ {i} ^ {t + 1} - g _ {i} ^ {t}\right) \right\| ^ {2} \right] \\ { = } { \gamma ^ { 2 } \mathbb { E } \left[ \left\| \mathcal { C } \left( \frac { v _ { i } ^ { t + 1 } - g _ { i } ^ { t } } { \gamma } \right) - \frac { v _ { i } ^ { t + 1 } - g _ { i } ^ { t } } { \gamma } \right\| ^ { 2 } \right] \leq \gamma ^ { 2 } \Delta ^ { 2 } . } \\ \end{array}
$$