SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In essence, when starting from  $x^0 = 0$ , the above linear-spanning property requires that any expansion of non-zero coordinates in vectors held by worker  $i$  ( $e.g., \mathcal{Y}_i^t, v_i^t$ ) are attributed to its past local gradient updates, local compression, or synchronization with the server. Meanwhile, it also requires that any expansion of non-zero coordinate in vectors held, including the final algorithmic output, in the server is due to the received compressed messages from workers.
Without loss of generality, we assume algorithms to start from  $x^0 = 0$  throughout the proofs. When  $\{f_i\}_{i=1}^n$  are further assumed to be zero-chain, following Definition 2, one can easily establish by induction that for any  $t \geq 1$ ,
$$
\max  _ {1 \leq r \leq t} \operatorname {p r o g} \left(u ^ {r}\right) \leq \max  _ {1 \leq r <   t} \max  _ {1 \leq i \leq n} \operatorname {p r o g} \left(\hat {v} _ {i} ^ {r}\right) \tag {10}
$$
$$
\begin{array}{l} \max  _ {1 \leq r \leq t} \operatorname {p r o g} (v _ {i} ^ {t}) \leq \max  _ {1 \leq r <   t} \max  \left\{\max  _ {1 \leq i \leq n} \operatorname {p r o g} (\hat {v} _ {i} ^ {r}), \operatorname {p r o g} (\mathcal {Y} _ {i} ^ {r}) \right\} \leq \max  _ {1 \leq r <   t} \max  _ {1 \leq i \leq n} \operatorname {p r o g} (\hat {v} _ {i} ^ {r}) + 1 \\ \operatorname {p r o g} \left(\hat {x} ^ {t}\right) \leq \max  _ {1 \leq r \leq t} \max  _ {1 \leq i \leq n} \operatorname {p r o g} \left(\hat {v} _ {i} ^ {r}\right) \\ \end{array}
$$
Next, we outline the proofs for the lower bounds presented in Theorem 2. For each case, we provide separate proofs for terms in the lower bound by constructing different hard-to-optimize examples, respectively. The construction of these proofs follows four steps:
- Constructing a set of zero-chain local functions  $\{f_i\}_{i=1}^n$ .
- Constructing a set of independent unbiased compressors  $\{C_i\}_{i=1}^n \subseteq \mathcal{U}_\omega^{\mathrm{ind}}$ . These compressors are delicately designed to impede algorithms from expanding the non-zero coordinates of model parameters.
- Establishing a limitation on zero-respecting algorithms that utilize the predefined compressor with  $t$  rounds of compressed communication on each worker. This limitation is based on the non-zero coordinates of model parameters.
- Translating the above limitation into the lower bound of the complexity measure defined in equation (3).
While the overall proof structure is similar to that of [19], our novel construction of functions and compressors enable us to derive lower bounds for independent compressors. These lower bounds clarify the unique properties and benefits of independent compressors.
We will use the following lemma in the analysis of the third step.
Lemma 3 ([19], Lemma 3). Given a constant  $p \in [0,1]$  and random variables  $\{B^t\}_{t=0}^\infty$  such that  $B^t \leq B^{(t-1)} + 1$  and  $\mathbb{P}(B^t \leq B^{t-1} \mid \{B^r\}_{r=0}^{t-1}) \geq 1 - p$  for any  $t \geq 1$ , it holds for  $t \geq 1/p$ , with probability at least  $1 - e^{-1}$ , that  $B^t \leq B^0 + ept$ .
C.1 Strongly-convex case
Below, we present two examples, each of which corresponding to a lower bound  $LB_{m}$  for  $T_{\epsilon}$ . We integrate the two lower bounds together and use the inequality
$$
T _ {\epsilon} \geq \max  _ {1 \leq m \leq 2} \left\{L B _ {m} \right\} = \Omega \left(L B _ {1} + L B _ {2}\right)
$$
to accomplish the lower bound for strongly-convex problems in Theorem 2.
Example 1. In this example, we prove the lower bound  $\Omega((1 + \omega)(1 + \sqrt{\kappa / n})\ln(\mu \Delta / \epsilon))$ .
(Step 1.) We assume the variable  $x \in \ell_2 \triangleq \{([x]_1, [x]_2, \ldots), \sum_{r=1}^{\infty} [x]_r^2 < \infty \}$  to be infinitely dimensional and square-summable for simplicity. It is easy to adapt the argument for finitely dimensional variables as long as the dimension is proportionally larger than  $t$ . Let  $M$  be
$$
M = \left[ \begin{array}{c c c c c} 2 & - 1 & & & \\ - 1 & 2 & - 1 & & \\ & - 1 & 2 & - 1 & \\ & & \ddots & \ddots & \ddots \end{array} \right] \in \mathbb {R} ^ {\infty \times \infty},
$$
then it is easy to see  $0 \preceq M \preceq 4I$ . Let  $\{f_i\}_{i=1}^n$  be as follows
$$
f _ {i} (x) = \left\{ \begin{array}{l l} \frac {\mu}{2} \| x \| ^ {2} + \frac {L - \mu}{4} \sum_ {r \geq 0} ([ x ] _ {n r + i} - [ x ] _ {n r + i + 1}) ^ {2}, & \text {i f} 1 \leq i \leq n - 1, \\ \frac {\mu}{2} \| x \| ^ {2} + \frac {L - \mu}{4} \left([ x ] _ {1} ^ {2} + \sum_ {r \geq 1} ([ x ] _ {n r} - [ x ] _ {n r + 1}) ^ {2} - 2 \lambda [ x ] _ {1}\right), & \text {i f} i = n. \end{array} \right.
$$
where  $\lambda \in \mathbb{R} \backslash \{0\}$  is to be specified. It is easy to see that  $\sum_{r \geq 0} ([x]_{nr + i} - [x]_{nr + i + 1})^2$  and  $[x]_1^2 + \sum_{r \geq 0} ([x]_{nr} - [x]_{nr + 1})^2 - 2\lambda [x]_1$  are convex and 4-smooth. Consequently, all  $f_i$ s are  $L$ -smooth and  $\mu$ -strongly convex. More importantly, it is easy to verify that all  $f_i$ s defined above are zero-chain functions and satisfy
$$
\operatorname {p r o g} \left(\nabla f _ {i} (x)\right) \left\{ \begin{array}{l l} = \operatorname {p r o g} (x) + 1, & \text {i f} \operatorname {p r o g} (x) \equiv i \mod n, \\ \leq \operatorname {p r o g} (x), & \text {o t h e r w i s e .} \end{array} \right. \tag {11}
$$
We further have  $f(x) = \frac{1}{n}\sum_{i=1}^{n}f_{i}(x) = \frac{\mu}{2}\|x\|^{2} + \frac{L-\mu}{4n}\left(x^{\top}Mx - 2\lambda[x]_{1}\right)$ . For the functions defined above, we also establish that
Lemma 4. Let  $\kappa \triangleq L / \mu \geq 1$ , it holds for any  $x$  that,
$$
f (x) - \min _ {x} f (x) \geq \frac {\mu}{2} \left(1 - 2 \left(1 + \sqrt {1 + \frac {2 (\kappa - 1)}{n}}\right) ^ {- 1}\right) ^ {2 \operatorname {p r o g} (x)} \| x ^ {0} - x ^ {\star} \| ^ {2}.
$$
Proof. The minimum  $x^{\star}$  of function  $f$  satisfies  $\left(\frac{L - \mu}{2n} M + \mu\right) x^{\star} - \lambda \frac{L - \mu}{2} e_{1} = 0$ , which is equivalent to
$$
\begin{array}{l} \frac {2 \kappa + 2 n - 2}{\kappa - 1} [ x ^ {\star} ] _ {1} - [ x ^ {\star} ] _ {2} = \lambda , \\ - \left[ x ^ {\star} \right] _ {j - 1} + \frac {2 \kappa + 2 n - 2}{\kappa - 1} \left[ x ^ {\star} \right] _ {j} - \left[ x ^ {\star} \right] _ {j + 1} = 0, \quad \forall j \geq 2. \tag {12} \\ \end{array}
$$
$$
q = \frac {\kappa + n - 1 - \sqrt {n (2 \kappa + n - 2)}}{\kappa - 1} = 1 - \frac {2}{1 + \sqrt {1 + \frac {2 (\kappa - 1)}{n}}}
$$
is the only root of the equation  $q^2 - \frac{2\kappa + 2n - 2}{\kappa - 1} q + 1 = 0$  that is smaller than 1. Then it is straight forward to check  $x^{\star} = ([x^{\star}]_{j} = \lambda q^{j})_{j\geq 1}$  satisfies (12). By the strong convexity of  $f$ ,  $x^{\star}$  is the unique solution. Therefore, we have that
$$
\| x - x ^ {\star} \| ^ {2} \geq \sum_ {j = \operatorname {p r o g} (x) + 1} ^ {\infty} \lambda^ {2} q ^ {2 j} = \lambda^ {2} \frac {q ^ {2 (r + 1)}}{1 - q ^ {2}} = q ^ {2 r} \| x ^ {0} - x ^ {\star} \| ^ {2}.
$$
Finally, using the strong convexity of  $f$  leads to the conclusion.
Following the proof of Lemma 4, we have
$$
\| x ^ {0} - x ^ {\star} \| ^ {2} = \lambda^ {2} \sum_ {j = 1} ^ {\infty} q ^ {2 j} = \lambda^ {2} \frac {q ^ {2}}{1 - q ^ {2}}
$$
Therefore, for any given  $\Delta > 0$ , letting  $\lambda = \sqrt{((1 - q^2)\Delta) / q^2}$  results in  $\| x^0 - x^\star \|^2 = \Delta$ . Consequently, our construction ensures  $\{f_i\}_{i=1}^n \in \mathcal{F}_{L,\mu}^\Delta$ .
(Step 2.) For the construction of  $\omega$ -unbiased compressors, we consider  $\{C_i\}_{i=1}^n$  to be independent random sparsification compressors. Building upon Example 1, we make a slight modification: during a round of communication on any worker, each coordinate is independently chosen with a probability of  $(1+\omega)^{-1}$  to be transmitted, and if selected, its value is scaled by  $(1+\omega)$  and then the scaled value is transmitted. Notably, the indices of chosen coordinates are not identical across all workers due to the independence of compressors. It can be easily verified that this construction ensures that  $\{C_i\}_{i=1}^n \subseteq \mathcal{U}_{\omega}^{\mathrm{ind}}$ .
(Step 3.) Since the algorithmic output  $\hat{x}^t$  calculated by the server lies in the linear manifold spanned by received messages, we can use (10) to obtain the following expression:
$$
\operatorname {p r o g} \left(\hat {x} ^ {t}\right) \leq \max  _ {1 \leq r \leq t} \max  _ {1 \leq i \leq n} \max  \left\{\operatorname {p r o g} \left(u ^ {r}\right), \operatorname {p r o g} \left(\hat {v} _ {i} ^ {r}\right) \right\} = \max  _ {1 \leq r \leq t} \max  _ {1 \leq i \leq n} \operatorname {p r o g} \left(\hat {v} _ {i} ^ {r}\right) \triangleq B ^ {t}. \tag {13}
$$
We next bound  $B^t$  with  $B^0 \coloneqq 0$  by showing that  $\{B^t\}_{t=0}^\infty$  satisfies Lemma 3 with  $p = (1 + \omega)^{-1}$ .
For any linear-spanning algorithm  $A$ , according to (11), the worker  $i$  can only attain one additional non-zero coordinate through local gradient-based updates when  $\mathrm{prog}(\mathcal{Y}_i^t) \equiv i \mod n$ . In other words, upon receiving messages  $\{u_i^r\}_{r=1}^t$  from the server, we have
$$
\operatorname {p r o g} (v _ {i} ^ {t}) \leq \left\{ \begin{array}{l l} \max  _ {1 \leq r \leq t} \operatorname {p r o g} (u _ {i} ^ {r}) + 1 \leq B ^ {t - 1} + 1, & \text {i f} \operatorname {p r o g} (\mathcal {Y} _ {i} ^ {t}) \equiv i \mod n, \\ \max  _ {1 \leq r \leq t} \operatorname {p r o g} (u _ {i} ^ {r}) \leq B ^ {t - 1}, & \text {o t h e r w i s e}. \end{array} \right.
$$
$$
\max  _ {1 \leq r \leq t} \operatorname {p r o g} (v _ {i} ^ {r}) \leq \max  _ {1 \leq r \leq t} B ^ {r - 1} + 1 = B ^ {t - 1} + 1.
$$
It then follows from the definition of the constructed  $C_i$  in Step 2 that  $\max_{1 \leq i \leq n} \operatorname{prog}(\hat{v}_i^t) \leq \max_{1 \leq i \leq n} \operatorname{prog}(v_i^t)$ , and therefore we have:
$$
B ^ {t} \leq \max  _ {1 \leq r \leq t} \max  _ {1 \leq i \leq n} \operatorname {p r o g} (v _ {i} ^ {r}) \leq B ^ {t - 1} + 1.
$$
Next, we aim to prove that  $B^t \leq B^{t-1} + 1$  with a probability of at least  $\omega/(1+\omega)$ . For any  $t \geq 1$ , let  $i \in \{1,\dots,n\}$  be such that  $B^{t-1} \equiv i \mod n$ . Due to the property in equation (11), during the  $t$ -th communication round, if  $\mathrm{prog}(\mathcal{Y}_i^t) = B^{t-1}$ , worker  $i$  can push the number of non-zero entries forward by 1, resulting in  $\mathrm{prog}(v_i^t) = B^{t-1} + 1$ , using local gradient updates. Note that any other worker  $j$  cannot achieve this even if  $\mathrm{prog}(\mathcal{Y}_j^t) = B^{t-1}$  due to equation (11).
Therefore, to achieve  $B^{t} = B^{t - 1} + 1$ , it is necessary for worker  $i$  to transmit a non-zero value at the  $(B^{t - 1} + 1)$ -th entry to the server. Otherwise, we have  $B^{t} \leq B^{t - 1}$ . However, since the compressor  $C_{i}$  associated with worker  $i$  has a probability  $\omega / (1 + \omega)$  to zero out the  $(B^{t - 1} + 1)$ -th entry in the  $t$ -th communication round, we have
$$
\mathbb {P} \left(B ^ {t} \leq B ^ {t - 1} \mid \left\{B ^ {r} \right\} _ {r = 0} ^ {t - 1}\right) \geq \omega / (1 + \omega).
$$
In summary, we have shown that  $B^{t} \leq B^{t - 1} + 1$  and  $\mathbb{P}\big(B^t \leq B^{t - 1} \mid \{B^r\}_{r = 0}^{t - 1}\big) \geq \omega / (1 + \omega)$ .
By applying Lemma 3, we can conclude that for any  $t \geq (1 + \omega)^{-1}$ , with a probability of at least  $1 - e^{-1}$ , it holds that  $B^t \leq et / (1 + \omega)$  and hence  $\mathrm{prog}(\hat{x}^t) \leq et / (1 + \omega)$  due to (13).
(Step 4.) Using Lemma 4 and that  $\mathrm{prog}(\hat{x}^t) \leq et / (1 + \omega)$  with probability at least  $1 - e^{-1}$ , we obtain
$$
\begin{array}{l} \mathbb {E} \left[ f \left(\hat {x} ^ {t}\right) \right] - \min  _ {x} f (x) \geq \frac {(1 - e ^ {- 1}) \mu \Delta}{2} \left(1 - 2 \left(1 + \sqrt {1 + \frac {2 (\kappa - 1)}{n}}\right) ^ {- 1}\right) ^ {2 e t / (1 + \omega)} \tag {14} \\ = \Omega \left(\mu \Delta \exp \left(- \frac {4 e t}{(\sqrt {\kappa / n} + 1) (1 + \omega)}\right)\right). \\ \end{array}
$$
Therefore, to ensure  $\mathbb{E}[f(\hat{x}^t)] - \min_x f(x) \leq \epsilon$ , relation (14) implies the lower bound  $T_{\epsilon} = \Omega((1 + \omega)(1 + \sqrt{\kappa / n})\ln (\mu \Delta /\epsilon))$ .
Example 2. Considering  $f_{1} = f$  to be homogeneous and  $C_i = I$  to be a loss-less compressor for all  $1 \leq i \leq n$ , the problem reduces to single-node convex optimization. In this case, the lower bound of  $\Omega(\sqrt{\kappa} \ln (\mu \Delta / \epsilon))$  is well-known in the literature, as shown in [44, 43].
With the two lower bounds achieved in Examples 1 and 2, we have
$$
\begin{array}{l} T _ {\epsilon} = \Omega \Big ((1 + \omega) (1 + \sqrt {\kappa / n}) \ln (\mu \Delta / \epsilon) + \sqrt {\kappa} \ln (\mu \Delta / \epsilon) \Big) \\ = \Omega \Big ((1 + \omega + \sqrt {\kappa / n} + \omega \sqrt {\kappa / n} + \sqrt {\kappa}) \ln (\mu \Delta / \epsilon) \Big) \\ = \Omega \left(\left(\omega + \omega \sqrt {\kappa / n} + \sqrt {\kappa}\right) \ln \left(\mu \Delta / \epsilon\right)\right) \\ \end{array}
$$
which is the result for the strongly-convex case in Theorem 2.
C.2 Generally-convex case
Below, we present three examples, each of which corresponding to a lower bound  $LB_{m}$  for  $T_{\epsilon}$ . We integrate the three lower bounds together and use the inequality
$$
T _ {\epsilon} \geq \max  _ {1 \leq m \leq 3} \left\{L B _ {m} \right\} = \Omega \left(L B _ {1} + L B _ {2} + L B _ {3}\right)
$$
to accomplish the lower bound for the generally-convex case in Theorem 2.
Example 1. In this example, we prove the lower bound  $\Omega((1 + \omega)(L\Delta/\epsilon)^{1/2})$ .
(Step 1.) We assume variable  $x \in \mathbb{R}^d$ , where  $d$  can be sufficiently large and will be determined later. Let  $M$  denote
$$
M = \left[ \begin{array}{c c c c c} 2 & - 1 & & & \\ - 1 & 2 & - 1 & & \\ & \ddots & \ddots & \ddots & \\ & & - 1 & 2 & - 1 \\ & & & - 1 & 2 \end{array} \right] \in \mathbb {R} ^ {d \times d},
$$
it is easy to verify  $0 \preceq M \preceq 4I$ . Similar to example 1 of the strongly-convex case, we consider
$$
f _ {i} (x) = \left\{ \begin{array}{l l} \frac {L}{4} \sum_ {r \geq 0} ([ x ] _ {n r + i} - [ x ] _ {n r + i + 1}) ^ {2}, & \text {i f} 1 \leq i \leq n - 1, \\ \frac {L}{4} \left([ x ] _ {1} ^ {2} + \sum_ {r \geq 1} ([ x ] _ {n r} - [ x ] _ {n r + 1}) ^ {2} - 2 \lambda [ x ] _ {1}\right), & \text {i f} i = n. \end{array} \right.
$$
where  $\lambda \in \mathbb{R} \backslash \{0\}$  is to be specified. It is easy to see that all  $f_i$ s are  $L$ -smooth. We further have  $f(x) = \frac{1}{n} \sum_{i=1}^{n} f_i(x) = \frac{L}{4n} (x^\top M x - 2\lambda [x]_1)$ . The  $f_i$  functions defined above are also zero-chain functions satisfying (11).
Following [43], it is easy to verify that the optimum of  $f$  satisfies
$$
x ^ {\star} = \left(\lambda \left(1 - \frac {k}{d + 1}\right)\right) _ {1 \leq k \leq d} \quad \text {a n d} \quad f (x ^ {\star}) = \min  _ {x} f (x) = - \frac {\lambda^ {2} L d}{4 n (d + 1)}.
$$
More generally, it holds for any  $0 \leq k \leq d$  that
$$
\min  _ {x: \operatorname {p r o g} (x) \leq k} f (x) = - \frac {\lambda^ {2} L k}{4 n (k + 1)}. \tag {15}
$$
Since  $\| x^0 -x^\star \| ^2 = \frac{\lambda^2}{(d + 1)^2}\sum_{k = 1}^{d}k^2 = \frac{\lambda^2d(2d + 1)}{6(d + 1)}\leq \frac{\lambda^2d}{3}$ , letting  $\lambda = \sqrt{3\Delta / d}$ , we have  $\{f_i\}_{i = 1}^n\in$ $\mathcal{F}_{L,0}^{\Delta}$ .
(Step 2.) Same as Step 2 of Example 1 of the strongly-convex case, we consider  $\{C_i\}_{i=1}^n$  to be independent random sparsification operators.
(Step 3.) Following the same argument as step 3 of example 1 of the strongly-convex case, we have that for any  $t \geq (1 + \omega)^{-1}$ , it holds with probability at least  $1 - e^{-1}$  that  $\mathrm{prog}(\hat{x}^t) \leq et / (1 + \omega)$ .
(Step 4.) Thus, combining (15), we have
$$
\begin{array}{l} \mathbb {E} [ f (\hat {x} ^ {t}) ] - \min _ {x} f (x) \geq (1 - e ^ {- 1}) \frac {\lambda^ {2} L}{4 n} \left(\frac {d}{d + 1} - \frac {e t / (1 + \omega)}{1 + e t / (1 + \omega)}\right) \\ = (1 - e ^ {- 1}) \frac {3 L \Delta}{4 n d} \left(\frac {d}{d + 1} - \frac {e t / (1 + \omega)}{1 + e t / (1 + \omega)}\right) \\ \end{array}
$$
Letting  $d = 1 + et / (1 + \omega)$ , we further have
$$
\mathbb {E} [ f (\hat {x} ^ {t}) ] - \min _ {x} f (x) \geq \frac {3 (1 - e ^ {- 1}) L \Delta}{8 n e t (1 + \omega) ^ {- 1} (1 + 2 e t (1 + \omega) ^ {- 1})} = \Omega \left(\frac {(1 + \omega) ^ {2} L \Delta}{n t ^ {2}}\right).
$$
Therefore, to ensure  $\mathbb{E}[f(\hat{x}^t)] - \min_x f(x) \leq \epsilon$ , the above inequality implies the lower bound to be  $T = \Omega((1 + \omega)(L\Delta/(n\epsilon))^{\frac{1}{2}})$ .
Example 2. Considering  $f_{1} = f$  to be homogeneous and  $C_i = I$  to be a loss-less compressor for all  $1 \leq i \leq n$ . The problem reduces to the single-node convex optimization. The lower bound  $\Omega(\sqrt{L\Delta/\epsilon})$  is well-known in literature, see, e.g., [44, 43].
Example 3. In this example, we prove the lower bound  $\Omega (\omega \ln (L\Delta /\epsilon))$
(Step 1.) We consider  $f_{1} = \dots = f_{n - 1} = L\| x\|^{2} / 2$  and  $f_{n} = L\| x\|^{2} / 2 + n\lambda \langle \mathbb{1}_{d},x\rangle$  where  $\mathbb{1}_d\in \mathbb{R}^d$  is the vector with all enries being 1 and  $\lambda \in \mathbb{R}$  is to be determined. By definition,  $\{f_i\}_{i = 1}^n$  are  $\mu$ -strongly-convex and  $L$ -smooth and the solution  $x^{\star} = -\frac{\lambda}{L}\mathbb{1}_d$ . Letting  $\lambda = L\sqrt{\Delta} /\sqrt{n}$ , we have  $\| x^{\star} - x^{0}\|^{2} = \Delta$ . Thus, the construction ensures  $\{f_i\}_{i = 1}^n\in \mathcal{F}_{L,\mu}^{\Delta}$ .
(Step 2.) Same as in Example 1, we consider  $\{C_i\}_{i=1}^n$  to be independent random sparsification operators.
(Step 3.) By the construction of  $\{f_i\}_{i=1}^n$ , we observe that the optimization process relies solely on transmitting the information of  $\mathbb{1}_d$  from worker  $n$  to the server. Let  $E^t$  denote the set of entries at which the server has received a non-zero value from worker  $n$  in the first  $t$  communication rounds. Note that for each entry, due to the construction of  $\{C_i\}_{i=1}^n$ , the server has a probability of at least  $(\omega/(1+\omega))^t$  of not receiving a non-zero value at that entry from worker  $n$ . Consequently,  $|(E^t)^c|$  is lower bounded by the sum of  $n$  independent Bernoulli  $(\omega^t/(1+\omega)^t)$  random variables. Therefore, we have  $\mathbb{E}[|(E^t)^c|] \geq d\omega^t/(1+\omega)^t$ .
(Step 4.) Given  $|E^t|$ , due to the linear-spanning property, we have  $\hat{x}^t \in \operatorname{span}\{e_j : j \in E^t\}$  where  $e_j$  is the  $j$ -th canonical vector. As a result, we have
$$
\begin{array}{l} \mathbb {E} [ f (\hat {x} ^ {t}) ] - \min _ {x} f (x) \\ \geq \mathbb {E} \left[ \min  _ {x \in \operatorname {s p a n} \{e _ {j}: j \in E ^ {t} \}} f (x) \right] - \min  _ {x} f (x) = \frac {L \Delta}{2} \frac {\mathbb {E} [ | (E ^ {t}) ^ {c} | ]}{d} \geq \frac {L \Delta}{2} \frac {\omega^ {t}}{(1 + \omega) ^ {t}}. \tag {16} \\ \end{array}
$$
Therefore, to ensure  $\mathbb{E}[f(\hat{x}^t)] - \min_x f(x) \leq \epsilon$ , (16) implies the lower bound  $T_{\epsilon} = \Omega (\omega \ln (L\Delta /\epsilon))$
With the three lower bounds achieved in Examples 1, 2, and 3, we have
$$
\begin{array}{l} T _ {\epsilon} = \Omega \Big (\sqrt {\frac {L \Delta}{\epsilon}} + (1 + \omega) \sqrt {\frac {L \Delta}{n \epsilon}} + \omega \ln (L \Delta / \epsilon) \Big) \\ = \Omega \left(\sqrt {\frac {L \Delta}{\epsilon}} + \omega \sqrt {\frac {L \Delta}{n \epsilon}} + \omega \ln (L \Delta / \epsilon)\right) \\ \end{array}
$$
which is the result for the generally-convex case in Theorem 2.
D.1 Strongly-convex case
We first present several important lemmas, followed by the definition of a Lyapunov function with delicately chosen coefficients for each term. Finally, we prove Theorem 3 by utilizing these lemmas. Throughout the convergence analysis, we use the following notations:
$$
\mathcal {W} ^ {k} = f \left(w ^ {k}\right) - f ^ {\star}, \quad \mathcal {Y} ^ {k} = f \left(y ^ {k}\right) - f ^ {\star}, \quad \mathcal {Z} ^ {k} = \| z ^ {k} - x ^ {\star} \| ^ {2},
$$
$$
\mathcal {H} ^ {k} = \frac {1}{n} \sum_ {i = 1} ^ {n} \| h _ {i} ^ {k} - \nabla f _ {i} (w ^ {k}) \| ^ {2}, \quad \mathcal {G} ^ {k} = \| g ^ {k} - \nabla f (x ^ {k}) \| ^ {2},
$$
$$
\mathcal {G} _ {w} ^ {k} = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (w ^ {k}) - \nabla f _ {i} (x ^ {k}) \| ^ {2}, \quad \mathcal {G} _ {y} ^ {k} = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (y ^ {k}) - \nabla f _ {i} (x ^ {k}) \| ^ {2}.
$$
We use  $\mathbb{E}_k$  or  $\mathbb{E}$  indicate the expectation with respect to the randomness in the  $k$ -th iteration or all historical randomness, respectively.
Lemma 5. If  $0 \leq \beta \leq 1$ , it holds for  $\forall k \geq 0$  that,
$$
\begin{array}{l} \mathcal {Z} ^ {k + 1} \leq 2 \gamma_ {k} \langle g ^ {k}, x ^ {\star} - x ^ {k} \rangle + \frac {2 \gamma_ {k} \beta \theta_ {2}}{\theta_ {1 , k}} \langle g ^ {k}, w ^ {k} - x ^ {k} \rangle + \frac {2 \gamma_ {k} \beta (1 - \theta_ {1 , k} - \theta_ {2})}{\theta_ {1 , k}} \langle g ^ {k}, y ^ {k} - x ^ {k} \rangle \\ + \beta \mathcal {Z} ^ {k} + (1 - \beta) \| x ^ {k} - x ^ {\star} \| ^ {2} + \gamma_ {k} ^ {2} \| g ^ {k} \| ^ {2}. \tag {17} \\ \end{array}
$$
Proof. Following the update rules in Algorithm 1, we have
$$
\begin{array}{l} \mathcal {Z} ^ {k + 1} = \left\| \beta z ^ {k} + (1 - \beta) x ^ {k} - x ^ {\star} + \frac {\gamma_ {k}}{\eta_ {k}} (y ^ {k + 1} - x ^ {k}) \right\| ^ {2} \\ = \| \beta \left(z ^ {k} - x ^ {\star}\right) + (1 - \beta) \left(x ^ {k} - x ^ {\star}\right) \| ^ {2} + \gamma_ {k} ^ {2} \| g ^ {k} \| ^ {2} \\ + \left\langle 2 \gamma_ {k} g ^ {k}, \beta z ^ {k} + (1 - \beta) x ^ {k} - x ^ {\star} \right\rangle . \tag {18} \\ \end{array}
$$
Since  $x^{k} = \theta_{1,k}z^{k} + \theta_{2}w^{k} + (1 - \theta_{1,k} - \theta_{2})y^{k}$ , we have
$$
\beta z ^ {k} + (1 - \beta) x ^ {k} - x ^ {\star} = \left(x ^ {k} - x ^ {\star}\right) + \frac {\beta \theta_ {2}}{\theta_ {1 , k}} \left(x ^ {k} - w ^ {k}\right) + \frac {\beta \left(1 - \theta_ {1 , k} - \theta_ {2}\right)}{\theta_ {1 , k}} \left(x ^ {k} - y ^ {k}\right). \tag {19}
$$
Plugging (19) into (18), using
$$
\left\| \beta \left(z ^ {k} - x ^ {\star}\right) + (1 - \beta) \left(x ^ {k} - x ^ {\star}\right) \right\| ^ {2} \leq \beta \| z ^ {k} - x ^ {\star} \| ^ {2} + (1 - \beta) \| x ^ {k} - x ^ {\star} \| ^ {2},
$$
we obtain (17).
Lemma 6. Under Assumption 1, if parameters satisfy  $\theta_{1,k},\theta_2,1 - \theta_{1,k} - \theta_2\in (0,1),\eta_k\in \left(0,\frac{1}{2L}\right]$ $\gamma_{k} = \frac{\eta_{k}}{2\theta_{1,k} + \eta_{k}\mu}$  and  $\beta = 1 - \gamma \mu = \frac{2\theta_{1,k}}{2\theta_{1,k} + \eta_k\mu}$ , then we have for any iteration  $k\geq 0$  that
$$
\begin{array}{l} \frac {2 \gamma_ {k} \beta}{\theta_ {1 , k}} \mathbb {E} _ {k} \left[ \mathcal {Y} ^ {k + 1} \right] + \mathbb {E} _ {k} \left[ \mathcal {Z} ^ {k + 1} \right] \leq \frac {2 \gamma_ {k} \beta \theta_ {2}}{\theta_ {1 , k}} \mathcal {W} ^ {k} + \frac {2 \gamma_ {k} \beta \left(1 - \theta_ {1 , k} - \theta_ {2}\right)}{\theta_ {1 , k}} \mathcal {Y} ^ {k} + \beta \mathcal {Z} ^ {k} + \frac {5 \gamma_ {k} \beta \eta_ {k}}{4 \theta_ {1 , k}} \mathcal {G} ^ {k} \\ - \frac {\gamma_ {k} \beta \theta_ {2}}{L \theta_ {1 , k}} \mathcal {G} _ {w} ^ {k} - \frac {\gamma_ {k} \beta \left(1 - \theta_ {1 , k} - \theta_ {2}\right)}{L \theta_ {1 , k}} \mathcal {G} _ {y} ^ {k}. \tag {20} \\ \end{array}
$$
Proof. By Assumption 1 and update rules in Algorithm 1, we have
$$
\begin{array}{l} f (y ^ {k + 1}) \leq f (x ^ {k}) + \langle \nabla f (x ^ {k}), y ^ {k + 1} - x ^ {k} \rangle + \frac {L}{2} \| y ^ {k + 1} - x ^ {k} \| ^ {2} \\ = f \left(x ^ {k}\right) - \left\langle \nabla f \left(x ^ {k}\right), \eta_ {k} g ^ {k} \right\rangle + \frac {L}{2} \eta_ {k} ^ {2} \| g ^ {k} \| ^ {2} \\ = f \left(x ^ {k}\right) - \eta_ {k} \left\langle \nabla f \left(x ^ {k}\right) - g ^ {k}, g ^ {k} \right\rangle + \left(\frac {L \eta_ {k} ^ {2}}{2} - \eta_ {k}\right) \| g ^ {k} \| ^ {2}. \tag {21} \\ \end{array}
$$