SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 5 EF21-STORM/MVR
1: Input:  $x^0$ , step-size  $\gamma > 0$ , parameter  $\eta \in (0,1]$ ,  $B_{\mathrm{init}} \geq 1$   
2: Initialize  $w_i^0 = g_i^0 = \frac{1}{B_{\mathrm{init}}} \sum_{j=1}^{B_{\mathrm{init}}} \nabla f_i(x^0, \xi_{i,j}^0)$  for  $i = 1, \dots, n$ ;  $g^0 = \frac{1}{n} \sum_{i=1}^{n} g_i^0$   
3: for  $t = 0,1,2, \dots, T-1$  do  
4: Master computes  $x^{t+1} = x^t - \gamma g^t$  and broadcasts  $x^{t+1}$  to all nodes  
5: for all nodes  $i = 1, \dots, n$  in parallel do  
6: Draw  $\xi_i^{t+1}$  and compute two (stochastic) gradients  $\nabla f_i(x^t, \xi_i^{t+1})$  and  $\nabla f_i(x^{t+1}, \xi_i^{t+1})$   
7: Compute variance reduced STORM/MVR estimator  
8:  $w_i^{t+1} = \nabla f_i(x^{t+1}, \xi_i^{t+1}) + (1-\eta)(w_i^t - \nabla f_i(x^t, \xi_i^{t+1}))$   
9: Compress  $c_i^{t+1} = \mathcal{C}(w_i^{t+1} - g_i^t)$  and send  $c_i^{t+1}$  to the master  
10: Update local state  $g_i^{t+1} = g_i^t + c_i^{t+1}$   
11: end for  
12: Master computes  $g^{t+1} = \frac{1}{n} \sum_{i=1}^{n} g_i^{t+1}$  via  $g^{t+1} = g^t + \frac{1}{n} \sum_{i=1}^{n} c_i^{t+1}$   
13: end for
Assumption 3 (Individual smoothness $^{21}$ ). For each  $i = 1, \dots, n$ , every realization of  $\xi_i \sim \mathcal{D}_i$ , the stochastic gradient  $\nabla f_i(x, \xi_i)$  is  $\ell_i$ -Lipschitz, i.e., for all  $x, y \in \mathbb{R}^d$
$$
\left\| \nabla f _ {i} (x, \xi_ {i}) - \nabla f _ {i} (y, \xi_ {i}) \right\| \leq \ell_ {i} \| x - y \|.
$$
We denote  $\widetilde{\ell^2} := \frac{1}{n}\sum_{i=1}^{n}\ell_i^2$
Theorem 7. Let Assumptions 1, 2 and 3 hold. Let  $\hat{x}^T$  be sampled uniformly at random from the iterates of the method. Let Algorithm 5 run with a contractive compressor. For all  $\eta \in (0,1]$  and  $B_{\mathrm{init}} \geq 1$ , with  $\gamma \leq \min \left\{\frac{\alpha}{8L}, \frac{\sqrt{\alpha}}{6\tilde{\ell}}, \frac{\sqrt{n\eta}}{8\tilde{\ell}}\right\}$ , we have
$$
\mathbb {E} \left[ \left\| \nabla f \left(\hat {x} ^ {T}\right) \right\| ^ {2} \right] \leq \mathcal {O} \left(\frac {\Psi_ {0}}{\gamma T} + \frac {\eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {\eta^ {2} \sigma^ {2}}{\alpha} + \frac {\eta \sigma^ {2}}{n}\right), \tag {48}
$$
where  $\Psi_0\coloneqq \delta_0 + \frac{\gamma}{\eta}\mathbb{E}\left[\left\| v^0 -\nabla f(x^0)\right\| ^2\right] + \frac{\gamma\eta}{\alpha^2}\frac{1}{n}\sum_{i = 1}^{n}\mathbb{E}\left[\left\| v_i^0 -\nabla f_i(x^0)\right\| ^2\right]$ . With the following step-size, momentum parameter, and initial batch size
$$
\gamma = \min \left\{\frac {\alpha}{8 \widetilde {L}}, \frac {\sqrt {\alpha}}{6 \widetilde {\ell}}, \frac {\sqrt {n \eta}}{8 \widetilde {\ell}} \right\}, \quad \eta = \min \left\{\alpha , \left(\frac {\widetilde {\ell} \delta_ {0} \alpha^ {2}}{\sigma^ {2} \sqrt {n T}}\right) ^ {2 / 7}, \left(\frac {\widetilde {\ell} \delta_ {0} \alpha}{\sigma^ {2} \sqrt {n T}}\right) ^ {2 / 5}, \left(\frac {\widetilde {\ell} \delta_ {0} \sqrt {n}}{\sigma^ {2} T}\right) ^ {2 / 3} \right\},
$$
and  $B_{\mathrm{init}} = \max \left\{\frac{\sigma^2}{L\delta_0n},\frac{\alpha n}{T}\right\}$ , we have
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] \leq \mathcal {O} \left(\frac {\widetilde {L} \delta_ {0}}{\alpha T} + \frac {\widetilde {\ell} \delta_ {0}}{\sqrt {\alpha} T} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma^ {1 / 3}}{\alpha^ {1 / 3} \sqrt {n T}}\right) ^ {6 / 7} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma^ {1 / 2}}{\alpha^ {1 / 4} \sqrt {n T}}\right) ^ {4 / 5} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma}{n T}\right) ^ {2 / 3}\right).
$$
Corollary 5. Under the setting of Theorem 7. we have  $\mathbb{E}\left[\|\nabla f(\hat{x}^T)\|\right] \leq \varepsilon$  after  $T = \mathcal{O}\left(\frac{\widetilde{\ell}\delta_0}{\alpha\varepsilon^2} + \frac{\widetilde{\ell}\delta_0\sigma^{1/3}}{\alpha^{1/3}\sqrt{n}\varepsilon^{7/3}} + \frac{\widetilde{\ell}\delta_0\sigma^{1/2}}{\alpha^{1/4}\sqrt{n}\varepsilon^{5/2}} + \frac{\widetilde{\ell}\delta_0\sigma}{n\varepsilon^3}\right)$  iterations.
Recently, Yau and Wai [2022] propose and analyze a DoCoM-SGT algorithm for decentralized optimization with contractive compressor under the above Assumption 3. When their method is specialized to centralized setting (with mixing constant  $\rho = 1$ ), their total sample complexity becomes  $\mathcal{O}\left(\frac{\widetilde{\ell}}{\alpha\varepsilon^2} + \frac{n^{4/5}\sigma^{3/2}}{\alpha^{9/4}\varepsilon^{3/2}} + \frac{\sigma^3}{n\varepsilon^3}\right)$  (see Table 1 or Theorem 4.1 in [Yau and Wai, 2022]). In contrast, the sample complexity given in our Corollary 5 improves the dependence on  $\sigma$  in the last term and, moreover, achieves the linear speedup in terms of  $n$  for all stochastic terms in the sample complexity.
I EF21-STORM/MVR
21This assumption can be also relaxed to so-called expected smoothness.
Proof. Similarly to the proof of Theorem 3, we control the error between  $g^{t}$  and  $\nabla f(x^{t})$  by decomposing it into two terms
$$
\left\| g ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \leq 2 \left\| g ^ {t} - w ^ {t} \right\| ^ {2} + 2 \left\| w ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \leq 2 \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - w _ {i} ^ {t} \right\| ^ {2} + 2 \left\| w ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}.
$$
In the following, we develop a recursive bound for each term above separately.
Part I. Controlling the variance of STORM/MVR estimator for each  $w_{i}^{t}$  and on average  $w^{t}$ . Recall that by Lemma 7-(55), we have for each  $i = 1, \dots, n$ , and any  $0 < \eta \leq 1$  and  $t \geq 0$
$$
\mathbb {E} \left[ \left\| w _ {i} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] \leq (1 - \eta) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] + 2 \ell_ {i} ^ {2} \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t + 1} \right\| ^ {2} \right] + 2 \eta^ {2} \sigma^ {2}. \tag {49}
$$
Averaging inequalities (49) over  $i = 1,\dots ,n$  and denoting  $\widetilde{P}_t\coloneqq \frac{1}{n}\sum_{i = 1}^n\mathbb{E}\left[\| w_i^t -\nabla f_i(x^t)\| ^2\right],$ $R_{t}\coloneqq \mathbb{E}\left[\| x^{t} - x^{t + 1}\|^{2}\right]$  we have
$$
\widetilde {P} _ {t + 1} \leq (1 - \eta) \widetilde {P} _ {t} + 2 \widetilde {\ell} ^ {2} R _ {t} + 2 \eta^ {2} \sigma^ {2}.
$$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \widetilde {P} _ {t} \leq \frac {2 \widetilde {\ell} ^ {2}}{\eta} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} + 2 \eta \sigma^ {2} + \frac {1}{\eta T} \widetilde {P} _ {0}. \tag {50}
$$
Similarly by Lemma 7-(56) denoting  $P_{t} \coloneqq \mathbb{E}\left[\| w^{t} - \nabla f(x^{t})\|^{2}\right]$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} P _ {t} \leq \frac {2 \widetilde {\ell} ^ {2}}{\eta n} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} + \frac {2 \eta \sigma^ {2}}{n} + \frac {1}{\eta T} P _ {0}. \tag {51}
$$
Part II. Controlling the variance of contractive compressor and STORM/MVR estimator. By Lemma 8 we have for each  $i = 1, \dots, n$ , and any  $0 < \eta \leq 1$  and  $t \geq 0$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| g _ {i} ^ {t + 1} - w _ {i} ^ {t + 1} \right\| ^ {2} \right] \leq \left(1 - \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| g _ {i} ^ {t} - w _ {i} ^ {t} \right\| ^ {2} \right] + \frac {4 \eta^ {2}}{\alpha} \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {4 L _ {i} ^ {2}}{\alpha} + \ell_ {i} ^ {2}\right) \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right] + 2 \eta^ {2} \sigma^ {2}. \tag {52} \\ \end{array}
$$
Averaging inequalities (52) over  $i = 1, \ldots, n$ , denoting  $\widetilde{V}_t := \frac{1}{n} \sum_{i=1}^n \mathbb{E}\left[\| g_i^t - w_i^t\|^2\right]$ , and summing up the resulting inequality for  $t = 0, \ldots, T-1$ , we obtain
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \widetilde {V} _ {t} \leq \frac {8 \eta^ {2}}{\alpha^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \widetilde {P} _ {t} + \left(\frac {8 \widetilde {L} ^ {2}}{\alpha^ {2}} + \frac {2 \widetilde {\ell} ^ {2}}{\alpha}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} + \frac {2 \eta^ {2} \sigma^ {2}}{\alpha} \\ \leq \left(\frac {8 \widetilde {L} ^ {2}}{\alpha^ {2}} + \frac {2 \widetilde {\ell} ^ {2}}{\alpha} + \frac {1 6 \eta \widetilde {\ell} ^ {2}}{\alpha^ {2}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} \\ + \frac {1 6 \eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {2 \eta^ {2} \sigma^ {2}}{\alpha} + \frac {8 \eta}{\alpha^ {2} T} \widetilde {P} _ {0}. \tag {53} \\ \end{array}
$$
$$
\begin{array}{l} f \left(x ^ {t + 1}\right) \leq f \left(x ^ {t}\right) - \frac {\gamma}{2} \left\| \nabla f \left(x ^ {t}\right) \right\| ^ {2} - \frac {1}{4 \gamma} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| g ^ {t} - \nabla f \left(x ^ {t}\right) \right\| ^ {2} \tag {54} \\ \leq f (x ^ {t}) - \frac {\gamma}{2} \left\| \nabla f (x ^ {t}) \right\| ^ {2} - \frac {1}{4 \gamma} \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} + \gamma \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| g _ {i} ^ {t} - w _ {i} ^ {t} \right\| ^ {2} + \gamma \left\| w ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2}. \\ \end{array}
$$
Subtracting  $f^*$  from both sides of (54), taking expectation and defining  $\delta_t \coloneqq \mathbb{E}\left[f(x^t) - f^*\right]$ , we derive
$$
\begin{array}{l} \stackrel {(i)} {\leq} \frac {2 \delta_ {0}}{\gamma T} + \left(\frac {1 6 \widetilde {L} ^ {2}}{\alpha^ {2}} + \frac {4 \widetilde {\ell} ^ {2}}{\alpha} + \frac {3 2 \eta \widetilde {\ell} ^ {2}}{\alpha^ {2}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} + 2 \frac {1}{T} \sum_ {t = 0} ^ {T - 1} P _ {t} - \frac {1}{2 \gamma^ {2}} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} \\ + \frac {3 2 \eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {4 \eta^ {2} \sigma^ {2}}{\alpha} + \frac {1 6 \eta}{\alpha^ {2} T} \tilde {P} _ {0} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i)} {\leq} \quad \frac {2 \delta_ {0}}{\gamma T} + \left(\frac {1 6 \widetilde {L} ^ {2}}{\alpha^ {2}} + \frac {4 \widetilde {\ell} ^ {2}}{\alpha} + \frac {3 2 \eta \widetilde {\ell} ^ {2}}{\alpha^ {2}} + \frac {4 \widetilde {\ell} ^ {2}}{\eta n} - \frac {1}{2 \gamma^ {2}}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} R _ {t} \\ + \frac {3 2 \eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {4 \eta^ {2} \sigma^ {2}}{\alpha} + \frac {4 \eta \sigma^ {2}}{n} + \frac {1 6 \eta}{\alpha^ {2} T} \widetilde {P} _ {0} + \frac {2}{\eta T} P _ {0} \\ \leq \frac {2 \delta_ {0}}{\gamma T} + \frac {3 2 \eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {4 \eta^ {2} \sigma^ {2}}{\alpha} + \frac {4 \eta \sigma^ {2}}{n} + \frac {1 6 \eta}{\alpha^ {2} T} \widetilde {P} _ {0} + \frac {2}{\eta T} P _ {0}, \\ \end{array}
$$
where in (i) we apply (53), in (ii) we use (51), and the last step follows by assumption on the step-size, which proves (48).
We now find the particular values of parameters. Using  $w_{i}^{0} = \frac{1}{B_{\mathrm{init}}} \sum_{j=1}^{B_{\mathrm{init}}} \nabla f_{i}(x^{0}, \xi_{i,j}^{0})$  for all  $i = 1, \ldots, n$ , we have
$$
P _ {0} = \mathbb {E} \left[ \left\| w ^ {0} - \nabla f (x ^ {0}) \right\| ^ {2} \right] \leq \frac {\sigma^ {2}}{n B _ {\mathrm {i n i t}}} \mathrm {a n d} \widetilde {P} _ {0} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| w _ {i} ^ {0} - \nabla f _ {i} (x ^ {0}) \right\| ^ {2} \right] \leq \frac {\sigma^ {2}}{B _ {\mathrm {i n i t}}}.
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {\delta_ {0}}{\gamma T} + \frac {\eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {\eta^ {2} \sigma^ {2}}{\alpha} + \frac {\eta \sigma^ {2}}{n} + \frac {\sigma^ {2}}{\eta n B _ {\mathrm {i n i t}} T} + \frac {\eta \sigma^ {2}}{\alpha^ {2} B _ {\mathrm {i n i t}} T}\right) \\ { = } { \mathcal { O } \left( \frac { \widetilde { L } \delta _ { 0 } } { \alpha T } + \frac { \widetilde { \ell } \delta _ { 0 } } { \sqrt { \alpha } T } + \frac { \widetilde { \ell } \delta _ { 0 } } { \sqrt { n \eta T } } + \frac { \eta ^ { 3 } \sigma ^ { 2 } } { \alpha ^ { 2 } } + \frac { \eta ^ { 2 } \sigma ^ { 2 } } { \alpha } + \frac { \eta \sigma ^ { 2 } } { n } + \frac { \sigma ^ { 2 } } { \eta n B _ { \mathrm { i n i t } } T } + \frac { \eta \sigma ^ { 2 } } { \alpha ^ { 2 } B _ { \mathrm { i n i t } } T } \right) . } \\ \end{array}
$$
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {\widetilde {L} \delta_ {0}}{\alpha T} + \frac {\widetilde {\ell} \delta_ {0}}{\sqrt {\alpha} T} + \frac {\widetilde {\ell} \delta_ {0}}{\sqrt {n \eta} T} + \frac {\eta^ {3} \sigma^ {2}}{\alpha^ {2}} + \frac {\eta^ {2} \sigma^ {2}}{\alpha} + \frac {\eta \sigma^ {2}}{n} + \frac {\eta \sigma^ {2}}{\alpha^ {2} B _ {\mathrm {i n i t}} T}\right).
$$
Notice that the choice of the momentum parameter such that  $\eta \leq \left(\frac{\widetilde{\ell}\delta_0\alpha^2}{\sigma^2\sqrt{n}T}\right)^{2 / 7},\eta \leq \left(\frac{\widetilde{\ell}\delta_0\alpha}{\sigma^2\sqrt{n}T}\right)^{2 / 5},$
$\eta \leq \left(\frac{\widetilde{\ell}\delta_0\sqrt{n}}{\sigma^2T}\right)^{2 / 3}$ , and  $\eta \leq \left(\frac{\widetilde{\ell}\delta_0\alpha^2B_{\mathrm{init}}}{\sigma^2\sqrt{n}}\right)^{2 / 3}$  ensures that  $\frac{\eta^3\sigma^2}{\alpha^2} \leq \frac{\widetilde{\ell}\delta_0}{\sqrt{n}\eta T}, \frac{\eta^2\sigma^2}{\alpha} \leq \frac{\widetilde{\ell}\delta_0}{\sqrt{n}\eta T}, \frac{\eta\sigma^2}{n} \leq \frac{\widetilde{\ell}\delta_0}{\sqrt{n}\eta T}$ .
and  $\frac{\eta\sigma^2}{\alpha^2B_{\mathrm{init}}T}\leq \frac{\widetilde{\ell}\delta_0}{\sqrt{n\eta T}}$  . Therefore, we have
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {\widetilde {L} \delta_ {0}}{\alpha T} + \frac {\widetilde {\ell} \delta_ {0}}{\sqrt {\alpha} T} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma^ {1 / 3}}{\alpha^ {1 / 3} \sqrt {n} T}\right) ^ {6 / 7} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma^ {1 / 2}}{\alpha^ {1 / 4} \sqrt {n} T}\right) ^ {4 / 5} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma}{n T}\right) ^ {2 / 3} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma}{\sqrt {n}}\right) ^ {2 / 3} \frac {\alpha^ {1 / 3}}{B _ {\mathrm {i n i t}} ^ {1 / 3} T}\right).
$$
Using  $B_{\mathrm{init}} \geq \frac{\alpha n}{T}$ , we obtain
$$
\mathbb {E} \left[ \left\| \nabla f (\hat {x} ^ {T}) \right\| ^ {2} \right] = \mathcal {O} \left(\frac {\widetilde {L} \delta_ {0}}{\alpha T} + \frac {\widetilde {\ell} \delta_ {0}}{\sqrt {\alpha} T} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma^ {1 / 3}}{\alpha^ {1 / 3} \sqrt {n} T}\right) ^ {6 / 7} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma^ {1 / 2}}{\alpha^ {1 / 4} \sqrt {n} T}\right) ^ {4 / 5} + \left(\frac {\widetilde {\ell} \delta_ {0} \sigma}{n T}\right) ^ {2 / 3}\right).
$$
I.1 Controlling the variance of STORM/MVR estimator
Lemma 7. Let Assumptions 2 and 3 be satisfied, and suppose  $0 < \eta \leq 1$ . For every  $i = 1, \dots, n$ , let the sequence  $\{w_i^t\}_{t \geq 0}$  be updated via  $w_i^{t+1} = \nabla f_i(x^{t+1}, \xi_i^{t+1}) + (1 - \eta)(w_i^t - \nabla f_i(x^t, \xi_i^{t+1}))$ . Define the sequence  $w^t := \frac{1}{n} \sum_{i=1}^{n} w_i^t$ . Then for every  $i = 1, \dots, n$  and  $t \geq 0$  it holds
$$
\mathbb {E} \left[ \left\| w _ {i} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right) \right\| ^ {2} \right] \leq (1 - \eta) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right) \right\| ^ {2} \right] + 2 \ell_ {i} ^ {2} \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t + 1} \right\| ^ {2} \right] + 2 \eta^ {2} \sigma^ {2}, \tag {55}
$$
$$
\mathbb {E} \left[ \left\| w ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \leq (1 - \eta) \mathbb {E} \left[ \left\| w ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {2 \widetilde {\ell} ^ {2}}{n} \mathbb {E} \left[ \left\| x ^ {t} - x ^ {t + 1} \right\| ^ {2} \right] + \frac {2 \eta^ {2} \sigma^ {2}}{n}. \tag {56}
$$
Proof. For each  $t = 0, \ldots, T - 1$ , define a random vector  $\xi^t \coloneqq (\xi_1^t, \ldots, \xi_n^t)$  and denote by  $\nabla f(x^t, \xi^t) \coloneqq \frac{1}{n} \sum_{i=1}^{n} \nabla f_i(x^t, \xi_i^t)$ . Note that the entries of the random vector  $\xi^t$  are independent and  $\mathbb{E}_{\xi^t}[\nabla f(x^t, \xi^t)] = \nabla f(x^t)$ , then we have
$$
w ^ {t + 1} = \nabla f (x ^ {t + 1}, \xi^ {t + 1}) + (1 - \eta) \left(w ^ {t} - \nabla f (x ^ {t}, \xi^ {t + 1})\right),
$$
where  $w^{t} = \frac{1}{n}\sum_{i = 1}^{n}w_{i}^{t}$  is an auxiliary sequence.
$$
\mathcal {V} _ {i} ^ {t} := \nabla f _ {i} (x ^ {t}, \xi_ {i} ^ {t}) - \nabla f _ {i} (x ^ {t}), \qquad \mathcal {V} ^ {t} := \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {V} _ {i} ^ {t},
$$
$$
\mathcal {W} _ {i} ^ {t} := \nabla f _ {i} (x ^ {t}) - \nabla f _ {i} (x ^ {t}, \xi_ {i} ^ {t + 1}) + \nabla f _ {i} (x ^ {t + 1}, \xi_ {i} ^ {t + 1}) - \nabla f _ {i} (x ^ {t + 1}), \qquad \mathcal {W} ^ {t} := \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {W} _ {i} ^ {t}.
$$
Then by Assumptions 2, we have
$$
\mathbb {E} \left[ \mathcal {V} _ {i} ^ {t} \right] = \mathbb {E} \left[ \mathcal {W} _ {i} ^ {t} \right] = \mathbb {E} \left[ \mathcal {V} ^ {t} \right] = \mathbb {E} \left[ \mathcal {W} ^ {t} \right] = 0, \tag {57}
$$
$$
\mathbb {E} \left[ \left\| \mathcal {V} _ {i} ^ {t} \right\| ^ {2} \right] \leq \sigma^ {2}, \quad \mathbb {E} \left[ \left\| \mathcal {V} ^ {t} \right\| ^ {2} \right] \leq \frac {\sigma^ {2}}{n}. \tag {58}
$$
Furthermore, we can derive
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathcal {W} ^ {t} \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] \\ { = } { \frac { 1 } { n ^ { 2 } } \mathbb { E } \left[ \left\| \sum _ { i = 1 } ^ { n } \mathcal { W } _ { i } ^ { t } \right\| ^ { 2 } \right] } \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] + \frac {1}{n ^ {2}} \sum_ {i \neq j} \mathbb {E} \left[ \langle \mathcal {W} _ {i} ^ {t}, \mathcal {W} _ {j} ^ {t} \rangle \right] \\ \stackrel {(i)} {=} \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] + \frac {1}{n ^ {2}} \sum_ {i \neq j} \langle \mathbb {E} \left[ \mathcal {W} _ {i} ^ {t} \right], \mathbb {E} \left[ \mathcal {W} _ {j} ^ {t} \right] \rangle \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \mathcal {W} _ {i} ^ {t} \right\| ^ {2} \right] \\ \leq \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \nabla f _ {i} \left(x ^ {t + 1}, \xi_ {i} ^ {t + 1}\right) - \nabla f _ {i} \left(x ^ {t}, \xi_ {i} ^ {t + 1}\right) \right\| ^ {2} \right] \\ \leq \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {2} \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right] = \frac {\widetilde {\ell} ^ {2}}{n} \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right], \\ \end{array}
$$
where  $(i)$  holds by the conditional independence of  $\mathcal{W}_i^t$  and  $\mathcal{W}_j^t$ , and the last inequality follows by the individual smoothness of stochastic functions (Assumption 3). Therefore, we have
$$
\mathbb {E} \left[ \| \mathcal {W} _ {i} ^ {t} \| ^ {2} \right] \leq \ell_ {i} ^ {2} \mathbb {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right], \quad \mathbb {E} \left[ \| \mathcal {W} ^ {t} \| ^ {2} \right] \leq \frac {\widetilde {\ell} ^ {2}}{n} \mathbb {E} \left[ \| x ^ {t + 1} - x ^ {t} \| ^ {2} \right], \tag {59}
$$
where the first inequality is obtained by using a similar derivation.
By the update rule for  $w^t$ , we can also derive
$$
\begin{array}{l} w ^ {t + 1} - \nabla f (x ^ {t + 1}) = (1 - \eta) \left(w ^ {t} - \nabla f (x ^ {t}, \xi^ {t + 1})\right) + \left(\nabla f (x ^ {t + 1}, \xi^ {t + 1}) - \nabla f (x ^ {t + 1})\right) \\ = (1 - \eta) \left(w ^ {t} - \nabla f \left(x ^ {t}\right)\right) + \eta (\nabla f \left(x ^ {t + 1}, \xi^ {t + 1}\right) - \nabla f \left(x ^ {t + 1}\right)) \\ + (1 - \eta) \left(\left(\nabla f (x ^ {t}) - \nabla f (x ^ {t}, \xi^ {t + 1}) + \nabla f (x ^ {t + 1}, \xi^ {t + 1}) - \nabla f (x ^ {t + 1})\right)\right) \\ = (1 - \eta) (w ^ {t} - \nabla f (x ^ {t})) + \eta \mathcal {V} ^ {t + 1} + (1 - \eta) \mathcal {W} ^ {t}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| w ^ {t + 1} - \nabla f (x ^ {t + 1}) \right\| ^ {2} \right] \leq \mathbb {E} \left[ \mathbb {E} _ {\xi^ {t + 1}} \left[ \left\| (1 - \eta) (w ^ {t} - \nabla f (x ^ {t})) + \eta \mathcal {V} _ {t + 1} + (1 - \eta) \mathcal {W} _ {t} \right\| ^ {2} \right] \right] \\ = (1 - \eta) ^ {2} \mathbb {E} \left[ \left\| w ^ {t} - \nabla f \left(x ^ {t}\right) \right\| ^ {2} \right] + \mathbb {E} \left[ \left\| \eta \mathcal {V} ^ {t + 1} + (1 - \eta) \mathcal {W} ^ {t} \right\| ^ {2} \right] \\ \leq (1 - \eta) \left\| w ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} + 2 \eta^ {2} \mathbb {E} \left[ \left\| \mathcal {V} ^ {t + 1} \right\| ^ {2} \right] + 2 \mathbb {E} \left[ \left\| \mathcal {W} ^ {t} \right\| ^ {2} \right] \\ \leq (1 - \eta) \mathbb {E} \left[ \left\| w ^ {t} - \nabla f (x ^ {t}) \right\| ^ {2} \right] + \frac {2 \sigma^ {2} \eta^ {2}}{n} + \frac {2 \widetilde {\ell} ^ {2}}{n} \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right], \\ \end{array}
$$
where the last inequality holds by (58) and (59). Similarly for each  $i = 1,\dots ,n$  , we have
$$
w _ {i} ^ {t + 1} - \nabla f _ {i} \left(x ^ {t + 1}\right) = (1 - \eta) \left(w _ {i} ^ {t} - \nabla f _ {i} \left(x ^ {t}\right)\right) + \eta \mathcal {V} _ {i} ^ {t + 1} + (1 - \eta) \mathcal {W} _ {i} ^ {t}. \tag {60}
$$
$$
\mathbb {E} \left[ \left\| w _ {i} ^ {t + 1} - \nabla f _ {i} (x ^ {t + 1}) \right\| ^ {2} \right] \leq (1 - \eta) \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] + 2 \sigma^ {2} \eta^ {2} + 2 \ell_ {i} ^ {2} R _ {t}.
$$
I.2 Controlling the variance of contractive compression and STORM/MVR estimator
Lemma 8. Let Assumptions 1, 2 and 3 be satisfied, and suppose  $0 < \eta \leq 1$ . For every  $i = 1, \dots, n$ , let the sequences  $\{w_i^t\}_{t \geq 0}$  and  $\{g_i^t\}_{t \geq 0}$  be updated via
$$
w _ {i} ^ {t + 1} = \nabla f _ {i} (x ^ {t + 1}, \xi_ {i} ^ {t + 1}) + (1 - \eta) (w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}, \xi_ {i} ^ {t + 1})),
$$
$$
g _ {i} ^ {t + 1} = g _ {i} ^ {t} + \mathcal {C} \left(w _ {i} ^ {t + 1} - g _ {i} ^ {t}\right).
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| g _ {i} ^ {t + 1} - w _ {i} ^ {t + 1} \right\| ^ {2} \right] \leq \left(1 - \frac {\alpha}{2}\right) \mathbb {E} \left[ \left\| g _ {i} ^ {t} - w _ {i} ^ {t} \right\| ^ {2} \right] + \frac {4 \eta^ {2}}{\alpha} \mathbb {E} \left[ \left\| w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t}) \right\| ^ {2} \right] \\ + \left(\frac {4 L _ {i} ^ {2}}{\alpha} + \ell_ {i} ^ {2}\right) \mathbb {E} \left[ \left\| x ^ {t + 1} - x ^ {t} \right\| ^ {2} \right] + 2 \eta^ {2} \sigma^ {2}. \tag {61} \\ \end{array}
$$
Proof. By the update rule of  $w_i^t$ ,  $g_i^t$ , and definition of  $\mathcal{V}_i^t$ ,  $\mathcal{W}_i^t$  given in the proof of Lemma 7, we can derive
$$
\begin{array}{l} \mathbb {E} \left[ \left\| g _ {i} ^ {t + 1} - w _ {i} ^ {t + 1} \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| \mathcal {C} (w _ {i} ^ {t + 1} - g _ {i} ^ {t}) - (w _ {i} ^ {t + 1} - g _ {i} ^ {t}) \right\| ^ {2} \right] \\ \stackrel {(i)} {\leq} (1 - \alpha) \mathbb {E} \left[ \left\| w _ {i} ^ {t + 1} - g _ {i} ^ {t} \right\| ^ {2} \right] \\ \stackrel {(i i)} {=} (1 - \alpha) \mathbb {E} \left[ \left\| (1 - \eta) \left(w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})\right) + \eta \mathcal {V} _ {i} ^ {t + 1} + (1 - \eta) \mathcal {W} _ {i} ^ {t} + \nabla f _ {i} (x ^ {t + 1}) - g _ {i} ^ {t} \right\| ^ {2} \right] \\ = (1 - \alpha) \mathbb {E} \left[ \mathbb {E} _ {\xi_ {i} ^ {t + 1}} \left[ \left\| (1 - \eta) \left(w _ {i} ^ {t} - \nabla f _ {i} (x ^ {t})\right) + \eta \mathcal {V} _ {i} ^ {t + 1} + (1 - \eta) \mathcal {W} _ {i} ^ {t} + \nabla f _ {i} (x ^ {t + 1}) - g _ {i} ^ {t} \right\| ^ {2} \right] \right] \\ \end{array}
$$