SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus,  $\bar{\theta}_{t + 1}$  is a non-decreasing sequence w.r.t.  $a_{t}$ . But the sequence  $a_{t}$  is non-increasing, therefore  $\bar{\theta}_{t + 1}$  is a non-increasing sequence w.r.t.  $t$ . It is left to use that  $\theta_{t + 1}$  is the minimum of  $\bar{\theta}_{t + 1}$  and the constant quantity.
E.2 Generic lemmas
First, we prove a well-known lemma from the theory of accelerated methods (Lan, 2020; Stonyakin et al., 2021).
Lemma E.2. Let us take vectors  $a, b, g \in \mathbb{R}^d$ , numerical quantities  $\alpha, \beta \geq 0$ , and
$$
u = \operatorname * {a r g   m i n} _ {x \in \mathbb {R} ^ {d}} \left\langle g, x \right\rangle + \frac {\alpha}{2} \left\| x - a \right\| ^ {2} + \frac {\beta}{2} \left\| x - b \right\| ^ {2}.
$$
$$
\langle g, x \rangle + \frac {\alpha}{2} \| x - a \| ^ {2} + \frac {\beta}{2} \| x - b \| ^ {2} \geq \langle g, u \rangle + \frac {\alpha}{2} \| u - a \| ^ {2} + \frac {\beta}{2} \| u - b \| ^ {2} + \frac {\alpha + \beta}{2} \| x - u \| ^ {2} \tag {34}
$$
for all  $x\in \mathbb{R}^d$
Proof. A function
$$
\widehat {f} (x) := \langle g, x \rangle + \frac {\alpha}{2} \| x - a \| ^ {2} + \frac {\beta}{2} \| x - b \| ^ {2}
$$
is strongly-convex with the parameter  $\alpha +\beta$ . From the strong-convexity and the optimality of  $u$ , we obtain
$$
\widehat {f} (x) \geq \widehat {f} (u) + \left\langle \widehat {f} (u), x - u \right\rangle + \frac {\alpha + \beta}{2} \| x - u \| ^ {2} = \widehat {f} (u) + \frac {\alpha + \beta}{2} \| x - u \| ^ {2}
$$
for all  $x\in \mathbb{R}^d$  . This inequality is equivalent to (34).
We assume that the conditional expectation  $\mathbb{E}_t[\cdot ]$  is conditioned on the randomness from the first  $t$  iterations. Also, let us define  $D_{f}(x,y)\coloneqq f(x) - f(y) - \langle \nabla f(y),x - y\rangle$
Lemma E.3. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. For Algorithm 1, the following inequality holds:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| g ^ {t + 1} - \nabla f (y ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq \frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2} + \frac {4 \omega L _ {\operatorname* {m a x}}}{n} \left(f \left(z ^ {t}\right) - f \left(y ^ {t + 1}\right) - \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right). \tag {35} \\ \end{array}
$$
Proof. Using the definition of  $g^{t + 1}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| g ^ {t + 1} - \nabla f (y ^ {t + 1}) \right| \right| ^ {2} \right] \\ = \mathbb {E} _ {t} \left[ \left\| h ^ {t} + \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {C} _ {i} ^ {D, y} \left(\nabla f _ {i} \left(y ^ {t + 1}\right) - h _ {i} ^ {t}\right) - \nabla f \left(y ^ {t + 1}\right) \right\| ^ {2} \right] \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t} \left[ \left\| \mathcal {C} _ {i} ^ {D, y} \left(\nabla f _ {i} \left(y ^ {t + 1}\right) - h _ {i} ^ {t}\right) - \left(\nabla f \left(y ^ {t + 1}\right) - h _ {i} ^ {t}\right) \right\| ^ {2} \right], \\ \end{array}
$$
where we use the independence and the unbiasedness of the compressors (see Assumption 2.4). Using Definition 2.3 and Assumption 1.1, we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left| \left| g ^ {t + 1} - \nabla f (y ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq \frac {\omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left| \left| \nabla f _ {i} (y ^ {t + 1}) - h _ {i} ^ {t} \right| \right| ^ {2} \\ \stackrel {(1 8)} {\leq} \frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - h _ {i} ^ {t} \right\| ^ {2} + \frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (y ^ {t + 1}) - \nabla f _ {i} (z ^ {t}) \right\| ^ {2} \\ \stackrel {L. D. 1} {\leq} \frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - h _ {i} ^ {t} \right\| ^ {2} + \frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} 2 L _ {i} (f _ {i} (z ^ {t}) - f _ {i} (y ^ {t + 1}) - \left\langle \nabla f _ {i} (y ^ {t + 1}), z ^ {t} - y ^ {t + 1} \right\rangle). \\ \end{array}
$$
Using that  $L_{\mathrm{max}} = \max_{i\in [n]}L_i$  we obtain (35).
Lemma E.4. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. For Algorithm 1, the following inequality holds:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f \left(z ^ {t + 1}\right) - f \left(x ^ {*}\right) \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f (z ^ {t}) - f \left(x ^ {*}\right)\right) \\ + \frac {2 p \omega}{n \bar {L}} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2}\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t} - x ^ {*} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - x ^ {*} \right\| ^ {2} \right]\right) \\ \end{array}
$$
$$
\begin{array}{l} + p \left(\frac {4 \omega L _ {\operatorname* {m a x}}}{n \bar {L}} + \theta_ {t + 1} - 1\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \frac {p L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right]. \\ \end{array}
$$
Proof. Using Assumption 1.2, we have
$$
f (x ^ {t + 1}) - f (x ^ {*}) \leq f (y ^ {t + 1}) - f (x ^ {*}) + \left\langle \nabla f (y ^ {t + 1}), x ^ {t + 1} - y ^ {t + 1} \right\rangle + \frac {L}{2} \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2}.
$$
Using the definition of  $x^{t + 1}$ , we obtain
$$
\begin{array}{l} f (x ^ {t + 1}) - f (x ^ {*}) \leq (1 - \theta_ {t + 1}) \left(f (y ^ {t + 1}) - f (x ^ {*}) + \left\langle \nabla f (y ^ {t + 1}), z ^ {t} - y ^ {t + 1} \right\rangle\right) \\ + \theta_ {t + 1} (f (y ^ {t + 1}) - f (x ^ {*}) + \left\langle \nabla f (y ^ {t + 1}), u ^ {t + 1} - y ^ {t + 1} \right\rangle) \\ + \frac {L}{2} \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \\ = \left(1 - \theta_ {t + 1}\right) \left(f \left(y ^ {t + 1}\right) - f \left(x ^ {*}\right) + \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right) \tag {36} \\ + \theta_ {t + 1} (f (y ^ {t + 1}) - f (x ^ {*}) + \left\langle g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle) \\ + \theta_ {t + 1} \left(\left\langle \nabla f (y ^ {t + 1}) - g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle\right) \\ + \frac {L}{2} \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2}, \\ \end{array}
$$
in the last inequality we add and subtract  $g^{t + 1}$ . Using the definition of  $u^{t + 1}$  and Lemma E.2 with  $x = x^*$ , we have
$$
\begin{array}{l} \left\langle g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle \leq \left\langle g ^ {t + 1}, x ^ {*} - y ^ {t + 1} \right\rangle + \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t} \right\| ^ {2} + \frac {\mu}{2} \left\| x ^ {*} - y ^ {t + 1} \right\| ^ {2} \\ - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} - \frac {\mu}{2} \left\| u ^ {t} - y ^ {t + 1} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t + 1} \right\| ^ {2}. \\ \end{array}
$$
We use the fact that  $\Gamma_{t + 1} = \Gamma_t + \gamma_{t + 1}$ . Since  $\left\| u^t - y^{t + 1} \right\|^2 \geq 0$ , we have
$$
\begin{array}{l} \left\langle g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle \leq \left\langle g ^ {t + 1}, x ^ {*} - y ^ {t + 1} \right\rangle + \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t} \right\| ^ {2} + \frac {\mu}{2} \left\| x ^ {*} - y ^ {t + 1} \right\| ^ {2} \\ - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t + 1} \right\| ^ {2}. \\ \end{array}
$$
By substituting this inequality to (36), we get
$$
\begin{array}{l} f (x ^ {t + 1}) - f (x ^ {*}) \\ \leq \left(1 - \theta_ {t + 1}\right) \left(f \left(y ^ {t + 1}\right) - f \left(x ^ {*}\right) + \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right) \\ + \theta_ {t + 1} (f (y ^ {t + 1}) - f (x ^ {*}) + \langle g ^ {t + 1}, x ^ {*} - y ^ {t + 1} \rangle) \\ + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t} \| ^ {2} + \frac {\mu}{2} \| x ^ {*} - y ^ {t + 1} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| u ^ {t + 1} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t + 1} \| ^ {2}\right) \\ + \theta_ {t + 1} \left(\left\langle \nabla f (y ^ {t + 1}) - g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle\right) \\ + \frac {L}{2} \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2}. \\ \end{array}
$$
Using  $\mu$ -strong convexity, we have
$$
f (x ^ {*}) \geq f (y ^ {t + 1}) + \left\langle \nabla f (y ^ {t + 1}), x ^ {*} - y ^ {t + 1} \right\rangle + \frac {\mu}{2} \left\| x ^ {*} - y ^ {t + 1} \right\| ^ {2}
$$
$$
\begin{array}{l} f (x ^ {t + 1}) - f (x ^ {*}) \\ \leq \left(1 - \theta_ {t + 1}\right) \left(f \left(y ^ {t + 1}\right) - f \left(x ^ {*}\right) + \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right) \\ \end{array}
$$
$$
\begin{array}{l} + \theta_ {t + 1} \left(\left\langle g ^ {t + 1} - \nabla f (y ^ {t + 1}), x ^ {*} - y ^ {t + 1} \right\rangle\right) \\ + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| u ^ {t + 1} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t + 1} \| ^ {2}\right) \\ + \theta_ {t + 1} \left(\left\langle \nabla f (y ^ {t + 1}) - g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle\right) \\ + \frac {L}{2} \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2}. \\ \end{array}
$$
Let us take the conditional expectation  $\mathbb{E}_t[\cdot ]$  conditioned on the randomness from the first  $t$  iterations:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (x ^ {t + 1}) - f (x ^ {*}) \right] \\ \leq \left(1 - \theta_ {t + 1}\right) \left(f \left(y ^ {t + 1}\right) - f \left(x ^ {*}\right) + \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right) \\ + \theta_ {t + 1} \left(\left\langle \mathbb {E} _ {t} \left[ g ^ {t + 1} - \nabla f (y ^ {t + 1}) \right], x ^ {*} - y ^ {t + 1} \right\rangle\right) \\ \left. \right. + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[\left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[\left\| x ^ {*} - u ^ {t + 1} \right\| ^ {2} \right]\right) \\ + \theta_ {t + 1} \mathbb {E} _ {t} \left[ \left\langle \nabla f (y ^ {t + 1}) - g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle \right] \\ + \frac {L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] \\ = \left(1 - \theta_ {t + 1}\right) \left(f \left(y ^ {t + 1}\right) - f \left(x ^ {*}\right) + \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right) \\ + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - u ^ {t} \| ^ {2} \right] - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| x ^ {*} - u ^ {t + 1} \| ^ {2} \right]\right) \\ + \theta_ {t + 1} \mathbb {E} _ {t} \left[ \left\langle \nabla f (y ^ {t + 1}) - g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle \right] \\ + \frac {L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right], \tag {37} \\ \end{array}
$$
where use that  $\mathbb{E}_t\left[g^{t + 1}\right] = \nabla f(y^{t + 1})$ . We can find  $u^{t + 1}$  analytically and obtain that
$$
u ^ {t + 1} = \frac {\bar {L} + \Gamma_ {t} \mu}{\bar {L} + \Gamma_ {t + 1} \mu} u ^ {t} + \frac {\mu \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} y ^ {t + 1} - \frac {\gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} g ^ {t + 1}.
$$
Therefore, using that  $\mathbb{E}_t\left[g^{t + 1}\right] = \nabla f(y^{t + 1})$  and  $u^t$  and  $y^{t + 1}$  are conditionally nonrandom, we obtain
$$
\mathbb {E} _ {t} \left[ \left\langle \nabla f (y ^ {t + 1}) - g ^ {t + 1}, u ^ {t + 1} - y ^ {t + 1} \right\rangle \right] = \frac {\gamma t + 1}{\bar {L} + \Gamma_ {t + 1} \mu} \mathbb {E} _ {t} \left[ \left\| g ^ {t + 1} - \nabla f (y ^ {t + 1}) \right\| ^ {2} \right]. \tag {38}
$$
Combining (35) from Lemma E.3 with (37) and (38), one can get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (x ^ {t + 1}) - f (x ^ {*}) \right] \\ \leq \left(1 - \theta_ {t + 1}\right) \left(f \left(y ^ {t + 1}\right) - f \left(x ^ {*}\right) + \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right) \\ + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| u ^ {t + 1} - u ^ {t} \| ^ {2} \right] - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| x ^ {*} - u ^ {t + 1} \| ^ {2} \right]\right) \\ + \frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \left(\frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2} + \frac {4 \omega L _ {\max}}{n} \left(f \left(z ^ {t}\right) - f \left(y ^ {t + 1}\right) - \left\langle \nabla f \left(y ^ {t + 1}\right), z ^ {t} - y ^ {t + 1} \right\rangle\right)\right) \\ + \frac {L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right]. \\ \end{array}
$$
Using the notation  $D_{f}(x,y)\coloneqq f(x) - f(y) - \langle \nabla f(y),x - y\rangle$  , we get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f (x ^ {t + 1}) - f (x ^ {*}) \right] \\ \leq \left(1 - \theta_ {t + 1}\right) \left(f (z ^ {t}) - f (x ^ {*}) - D _ {f} \left(z ^ {t}, y ^ {t + 1}\right)\right) \\ + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| x ^ {*} - u ^ {t + 1} \right\| ^ {2} \right]\right) \\ + \frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \left(\frac {2 \omega}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2} + \frac {4 \omega L _ {\max }}{n} D _ {f} \left(z ^ {t}, y ^ {t + 1}\right)\right) \\ + \frac {L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] \\ = \left(1 - \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + \frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \frac {2 \omega}{n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2}\right) \\ + \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| x ^ {*} - u ^ {t + 1} \right\| ^ {2} \right]\right) \\ + \left(\frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \frac {4 \omega L _ {\max }}{n} + \theta_ {t + 1} - 1\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \frac {L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] - \frac {\theta_ {t + 1} (\bar {L} + \Gamma_ {t} \mu)}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right]. \\ \end{array}
$$
In the last equality, we simply regrouped the terms. Using the definition of  $z^{t + 1}$ , we get
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f \left(z ^ {t + 1}\right) - f \left(x ^ {*}\right) \right] = p \mathbb {E} _ {t} \left[ f \left(x ^ {t + 1}\right) - f \left(x ^ {*}\right) \right] + (1 - p) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ \leq p \left(1 - \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \frac {2 \omega}{n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} (z ^ {t}) - h _ {i} ^ {t} \right\| ^ {2}\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| x ^ {*} - u ^ {t + 1} \| ^ {2} \right]\right) \\ + p \left(\frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \frac {4 \omega L _ {\max}}{n} + \theta_ {t + 1} - 1\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \frac {p L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] - \frac {p \theta_ {t + 1} (\bar {L} + \Gamma_ {t} \mu)}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right] + (1 - p) (f (z ^ {t}) - f (x ^ {*})) \\ = \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \frac {2 \omega}{n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2}\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \| x ^ {*} - u ^ {t} \| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \| x ^ {*} - u ^ {t + 1} \| ^ {2} \right]\right) \\ + p \left(\frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \frac {4 \omega L _ {\max}}{n} + \theta_ {t + 1} - 1\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \frac {p L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] - \frac {p \theta_ {t + 1} (\bar {L} + \Gamma_ {t} \mu)}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right]. \\ \end{array}
$$
In the last equality, we grouped the terms with  $f(z^t) - f(x^*)$ . In Algorithm 2, we choose the learning rates so that (see Lemma E.1)
$$
\bar {L} \theta_ {t + 1 \gamma_ {t + 1}} \leq \bar {L} + \Gamma_ {t} \mu .
$$
Since  $\Gamma_{t + 1}\geq \Gamma_t$  for all  $t\in \mathbb{N}_0$  , thus
$$
\frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t + 1} \mu} \leq \frac {\theta_ {t + 1} \gamma_ {t + 1}}{\bar {L} + \Gamma_ {t} \mu} \leq \frac {1}{\bar {L}}
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ f \left(z ^ {t + 1}\right) - f \left(x ^ {*}\right) \right] \\ \leq \left(1 - p \theta_ {t + 1}\right) \left(f \left(z ^ {t}\right) - f \left(x ^ {*}\right)\right) \\ + p \frac {2 \omega}{n \bar {L}} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(z ^ {t}\right) - h _ {i} ^ {t} \right\| ^ {2}\right) \\ + p \theta_ {t + 1} \left(\frac {\bar {L} + \Gamma_ {t} \mu}{2 \gamma_ {t + 1}} \left\| x ^ {*} - u ^ {t} \right\| ^ {2} - \frac {\bar {L} + \Gamma_ {t + 1} \mu}{2 \gamma_ {t + 1}} \mathbb {E} _ {t} \left[ \left\| x ^ {*} - u ^ {t + 1} \right\| ^ {2} \right]\right) \\ + p \left(\frac {4 \omega L _ {\operatorname* {m a x}}}{n \bar {L}} + \theta_ {t + 1} - 1\right) D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) \\ + \frac {p L}{2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] - \frac {p \theta_ {t + 1} ^ {2} \bar {L}}{2} \mathbb {E} _ {t} \left[ \left\| u ^ {t + 1} - u ^ {t} \right\| ^ {2} \right]. \\ \end{array}
$$
E.3 Construction of the Lyapunov function
In this section, we provide lemmas that will help us to construct a Lyapunov function.
Lemma E.5. Suppose that Assumptions 1.2, 1.1, 1.3 and 2.4 hold. The parameter  $\beta \leq 1 / \omega + 1$ . Then, for Algorithm 1, the following inequality holds:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right\| ^ {2} \right] (39) \\ \leq 8 p \left(1 + \frac {p}{\beta}\right) L _ {\max } D _ {f} \left(z ^ {t}, y ^ {t + 1}\right) + 4 p \left(1 + \frac {p}{\beta}\right) \widehat {L} ^ {2} \mathbb {E} _ {t} \left[ \left\| x ^ {t + 1} - y ^ {t + 1} \right\| ^ {2} \right] + \left(1 - \frac {\beta}{2}\right) \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t}\right) \right\| ^ {2}. (40) \\ \end{array}
$$
Proof. Using the definition of  $h_i^{t+1}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ = \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} + \beta \mathcal {C} _ {i} ^ {D, z} \left(\nabla f _ {i} \left(z ^ {t + 1}\right) - h _ {i} ^ {t}\right) - \nabla f _ {i} \left(z ^ {t + 1}\right) \right\| ^ {2} \right] \\ = \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right\| ^ {2} \right] \\ + \mathbb {E} _ {t} \left[ \frac {2 \beta}{n} \sum_ {i = 1} ^ {n} \left\langle h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t + 1}), \mathcal {C} _ {i} ^ {D, z} (\nabla f _ {i} (z ^ {t + 1}) - h _ {i} ^ {t}) \right\rangle + \frac {\beta^ {2}}{n} \sum_ {i = 1} ^ {n} \left\| \mathcal {C} _ {i} ^ {D, z} (\nabla f _ {i} (z ^ {t + 1}) - h _ {i} ^ {t}) \right\| ^ {2} \right]. \\ \end{array}
$$
Note that  $\mathbb{E}_{\mathcal{C}}\left[\mathcal{C}_i^{D,z}(\nabla f_i(z^{t + 1}) - h_i^t)\right] = \nabla f_i(z^{t + 1}) - h_i^t$  and
$$
\mathbb {E} _ {\mathcal {C}} \left[ \left\| \mathcal {C} _ {i} ^ {D, z} \left(\nabla f _ {i} \left(z ^ {t + 1}\right) - h _ {i} ^ {t}\right) \right\| ^ {2} \right] \leq (\omega + 1) \left\| \nabla f _ {i} \left(z ^ {t + 1}\right) - h _ {i} ^ {t} \right\| ^ {2},
$$
where  $\mathbb{E}_C[\cdot ]$  is a conditional expectation that is conditioned on  $z^{t + 1}$  and  $h_i^t$ . Therefore,
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \left| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right| \right| ^ {2} \right] \\ \leq \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t + 1}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} + \mathbb {E} _ {t} \left[ \frac {2 \beta}{n} \sum_ {i = 1} ^ {n} \left\langle h _ {i} ^ {t} - \nabla f _ {i} \left(z ^ {t + 1}\right), \nabla f _ {i} \left(z ^ {t + 1}\right) - h _ {i} ^ {t} \right\rangle + \frac {\beta^ {2} (\omega + 1)}{n} \sum_ {i = 1} ^ {n} \left| \left| \nabla f _ {i} \left(z ^ {t + 1}\right) - h _ {i} ^ {t} \right| \right| ^ {2} \right] \\ = \left(1 - 2 \beta + \beta^ {2} (\omega + 1)\right) \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right]. \\ \end{array}
$$
Since  $\beta \leq 1 / \omega +1$ , we have
$$
\mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t + 1} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right] \leq (1 - \beta) \mathbb {E} _ {t} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| h _ {i} ^ {t} - \nabla f _ {i} (z ^ {t + 1}) \right\| ^ {2} \right].
$$
Next, we use the definition of  $z^{t + 1}$  and obtain