SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The most relevant works are FL with cyclic client participation (Cho et al., 2023) and FL with shuffling client participation (Malinovsky et al., 2023) (we note them when preparing this version).
Figure 5: Illustration of FL with cyclic client participation with  $M = 12$  clients divided into  $\overline{K} = 3$  groups. In each training round,  $N = 2$  clients are selected for training from the client group. All groups are traversed once in a cycle-epoch consisting of  $\overline{K}$  training rounds. (Cho et al., 2023).
Discussion about FL with cyclic client participation. Cho et al. (2023) consider the scenario where the total  $M$  clients are divided into  $\overline{K}$  non-overlapping client groups such that each group contains  $M / \overline{K}$  clients. In each training round, the sever selects a subset of  $N$  clients from a group without replacement for training in this round. One example is shown in Figure 5. As said in the paragraph "Cyclic Client Participation (CyCP)" in their Section 3 (Problem Formulation), the groups' training
order of FL with cyclic client participation is pre-determined and fixed. In contrast, the clients' training order of SFL (precisely, Algorithm 1) will be shuffled at the beginning of each round.
It can be verified by Cho et al. (2023)'s Theorem 2. Letting  $\overline{K} = 1$  and  $N = M / \overline{K} = M$  and  $\overline{K} = M$  and  $N = M / \overline{K} = 1$ , we get the bounds for PFL and SFL, respectively:
$$
\mathrm {P F L}: \quad \tilde {\mathcal {O}} \left(\frac {L \sigma^ {2}}{\mu^ {2} M K R} + \frac {L ^ {2} \zeta^ {2}}{\mu^ {3} M ^ {2} R ^ {2}}\right)
$$
where we have omitted the optimization term and changed their notations to ours (change  $\alpha$  to  $0$ ,  $\gamma$  to  $\zeta$ ,  $\nu$  to  $\zeta$ ,  $T$  to  $R$ ).
$$
\text {S F L :} \quad \tilde {\mathcal {O}} \left(\frac {L \sigma^ {2}}{\mu^ {2} K R} + \frac {L ^ {2} \zeta^ {2}}{\mu^ {3} R ^ {2}} + \frac {L ^ {2} \zeta^ {2}}{\mu^ {3} M ^ {2} R ^ {2}}\right)
$$
where we have omitted the optimization term and changed their notations to ours (change  $\alpha$  to  $\zeta$ ,  $\gamma$  to  $0$ ,  $\nu$  to  $\zeta$ ,  $T$  to  $MR$ ). As we can see, we do not see a clear advantage of SFL like ours.
Figure 6: Visualization of FL with shuffling client participation for 6 clients, each with 3 datapoints. Two clients are sampled in each communication round (Malinovsky et al., 2023).
Discussion about FL with shuffling client participation. At the beginning of each meta epoch, FL with shuffling client participation partition all  $M$  clients into  $M / C$  cohorts, each with size  $C$ . These cohorts are obtained using the without replacement sampling of clients. Each meta epoch contains  $R = M / C$  communication rounds. At each communication round, clients in a cohort participate in the training process. One example is shown in Figure 6. It is noteworthy that SGD-RR is used as the local solver in Malinovsky et al. (2023) while SGD is used in this paper.
Letting  $C = 1$ ,  $R = M$ ,  $T = R$  in their Theorem 6.1, we can get the bounds for SFL:
SFL (Malinovsky et al., 2023):  $\mathcal{O}\left(\frac{1}{\mu} \cdot LMK^2 \eta^2 \zeta_*^2 + D^2 \exp(-\mu MKR\eta)\right)$  with  $\eta \leq \frac{1}{L}$ .
SFL (Theorem 1):  $\mathcal{O}\left(LMK^2\eta^2\zeta_*^2 +\mu D^2\exp (-\mu MKR\eta)\right)$  with  $\eta \leq \frac{1}{LMK}$
where we only considered the heterogeneity terms (let  $\sigma_{\star}^{2} = 0$  for Malinovsky et al. (2023) and  $\sigma = 0$  for ours) and changed their notations to ours (change  $\gamma$  to  $\eta$ ,  $N$  to  $K$ ,  $\tilde{\sigma}_{\star}$  to  $\zeta_{*}$ ). Their bound almost matches ours, with some differences on the constants  $\mu$ ,  $L$  and restrictions on  $\eta$ , which is caused by using different local solvers. However, their results are limited to the case where the number of local steps equals the size of the local dataset. It is still uncertain whether their results for SGD-RR can be generalized to situations with varying numbers of local steps.
C Notations and technical lemmas
C.1 Notations
Table 5 summarizes the notations appearing in this paper.
Table 5: Summary of key notations.
Symbol	Description
R,r	number, index of training rounds
M,m	number, index of clients
K,k	number, index of local update steps
S	number of participating clients
π	{π1, π2, ..., πM} is a permutation of {1, 2, ..., M}
η	learning rate (or stepsize)
η	effective learning rate (η := MKη in SFL and η := Kη in PFL)
μ	μ-strong convexity constant
L	L-smoothness constant (Asm. 1)
σ	upper bound on variance of stochastic gradients at each client (Asm. 2)
β,ζ	constants in Asm. 3a to bound heterogeneity everywhere
ζ*	constants in Asm. 3b to bound heterogeneity at the optima
F/Fm	global objective/local objective of client m
x(r)	global model parameters in the r-th round
x(m,k)	local model parameters of the m-th client after k local steps in the r-th round
g(r)	g(r)_{πm,k} := ∇f_{πm}(x(r)_{m,k};ξ) denotes the stochastic gradients of F_{πm} regarding x(r)_{m,k}
ExDir(C,α)	Extended Dirichlet strategy with parameters C and α (see Sec. G.1)
C.2 Basic identities and inequalities
These identities and inequalities are mostly from Zhou (2018); Khaled et al. (2020); Mishchenko et al. (2020); Karimireddy et al. (2020); Garrigos and Gower (2023).
For any random variable  $\mathbf{x}$ , letting the variance can be decomposed as
$$
\mathbb {E} \left[ \| \mathbf {x} - \mathbb {E} [ \mathbf {x} ] \| ^ {2} \right] = \mathbb {E} \left[ \| \mathbf {x} \| ^ {2} \right] - \| \mathbb {E} [ \mathbf {x} ] \| ^ {2} \tag {12}
$$
In particular, its version for vectors with finite number of values gives
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} - \bar {\boldsymbol {x}} \| ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} \| ^ {2} - \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {x} _ {i} \right\| ^ {2} \tag {13}
$$
where vectors  $\pmb{x}_1, \dots, \pmb{x}_n \in \mathbb{R}^d$  are the values of  $\mathbf{x}$  and their average is  $\bar{\pmb{x}} = \frac{1}{n} \sum_{i=1}^{n} \pmb{x}_i$ .
Jensen's inequality. For any convex function  $h$  and any vectors  $x_{1},\ldots ,x_{n}$  we have
$$
h \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {x} _ {i}\right) \leq \frac {1}{n} \sum_ {i = 1} ^ {n} h (\boldsymbol {x} _ {i}). \tag {14}
$$
As a special case with  $h(\pmb{x}) = \| \pmb{x} \|^2$ , we obtain
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {x} _ {i} \right\| ^ {2} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} \| ^ {2}. \tag {15}
$$
Smoothness and general convexity, strong convexity. There are some useful inequalities with respect to  $L$ -smoothness (Assumption 1), convexity and  $\mu$ -strong convexity. Their proofs can be found in Zhou (2018); Garrigos and Gower (2023).
Bregman Divergence associated with function  $h$  and arbitrary  $x, y$  is denoted as
$$
D _ {h} (\boldsymbol {x}, \boldsymbol {y}) := h (\boldsymbol {x}) - h (\boldsymbol {y}) - \langle \nabla h (\boldsymbol {y}), \boldsymbol {x} - \boldsymbol {y} \rangle
$$
When the function  $h$  is convex, the divergence is strictly non-negative. A more formal definition can be found in Orabona (2019). One corollary (Chen and Teboulle, 1993) called three-point-identity is,
$$
D _ {h} (\boldsymbol {z}, \boldsymbol {x}) + D _ {h} (\boldsymbol {x}, \boldsymbol {y}) - D _ {h} (\boldsymbol {z}, \boldsymbol {y}) = \langle \nabla h (\boldsymbol {y}) - \nabla h (\boldsymbol {x}), \boldsymbol {z} - \boldsymbol {x} \rangle
$$
where  $x, y, z$  is three points in the domain.
Let  $h$  be  $L$ -smooth. With the definition of Bregman divergence, a consequence of  $L$ -smoothness is
$$
D _ {h} (\boldsymbol {x}, \boldsymbol {y}) = h (\boldsymbol {x}) - h (\boldsymbol {y}) - \left\langle \nabla h (\boldsymbol {y}), \boldsymbol {x} - \boldsymbol {y} \right\rangle \leq \frac {L}{2} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} \tag {16}
$$
Further, If  $h$  is  $L$ -smooth and lower bounded by  $h_*$ , then
$$
\left\| \nabla h (\boldsymbol {x}) \right\| ^ {2} \leq 2 L \left(h (\boldsymbol {x}) - h _ {*}\right). \tag {17}
$$
If  $h$  is  $L$ -smooth and convex (The definition of convexity can be found in Boyd et al. (2004)), then
$$
D _ {h} (\boldsymbol {x}, \boldsymbol {y}) \geq \frac {1}{2 L} \| \nabla h (\boldsymbol {x}) - \nabla h (\boldsymbol {y}) \| ^ {2}. \tag {18}
$$
The function  $h: \mathbb{R}^d \to \mathbb{R}$  is a strongly convex if and only if there exists a convex function  $g: \mathbb{R}^d \to \mathbb{R}$  such that  $h(\boldsymbol{x}) = g(\boldsymbol{x}) + \frac{\mu}{2} \| \boldsymbol{x} \|^2$ .
If  $h$  is  $\mu$ -strongly convex, it holds that
$$
\frac {\mu}{2} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} \leq D _ {h} (\boldsymbol {x}, \boldsymbol {y}) \tag {19}
$$
C.3 Technical lemmas
Lemma 1 (Karimireddy et al. (2020)). Let  $\{\xi_i\}_{i=1}^n$  be a sequence of random variables. And the random sequence  $\{\mathbf{x}_i\}_{i=1}^n$  satisfy that  $\mathbf{x}_i \in \mathbb{R}^d$  is a function of  $\xi_i, \xi_{i-1}, \ldots, \xi_1$  for all  $i$ . Suppose that the conditional expectation is  $\mathbb{E}_{\xi_i}[\mathbf{x}_i|\xi_{i-1}, \ldots, \xi_1] = \mathbf{e}_i$  (i.e., the vectors  $\{\mathbf{x}_i - \mathbf{e}_i\}$  form a martingale difference sequence with respect to  $\{\xi_i\}$ ), and the variance is bounded by  $\mathbb{E}_{\xi_i}\left[\| \mathbf{x}_i - \mathbf{e}_i\|^2 \Big| \xi_{i-1}, \ldots, \xi_1\right] \leq \sigma^2$ . Then it holds that
$$
\mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {n} \left(\mathbf {x} _ {i} - \mathbf {e} _ {i}\right) \right\| ^ {2} \right] = \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \mathbf {x} _ {i} - \mathbf {e} _ {i} \right\| ^ {2} \right] \leq n \sigma^ {2} \tag {20}
$$
Proof. This conclusion has appeared in Stich and Karimireddy (2019)'s Lemma 15, Karimireddy et al. (2020)'s Lemma 4 (separating mean and variance) and Wang et al. (2020)'s Lemma 2, which is useful for bounding the stochasticity.
$$
\mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {n} (\mathbf {x} _ {i} - \mathbf {e} _ {i}) \right\| ^ {2} \right] = \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \left\| \mathbf {x} _ {i} - \mathbf {e} _ {i} \right\| ^ {2} \right] + \sum_ {i = 1} ^ {n} \sum_ {j \neq i} ^ {n} \mathbb {E} \left[ (\mathbf {x} _ {i} - \mathbf {e} _ {i}) ^ {\top} (\mathbf {x} _ {j} - \mathbf {e} _ {j}) \right]
$$
Without loss of generality, we can assume that  $i < j$ . Then the cross terms in the preceding equation can be computed by the law of total expectation:
$$
\begin{array}{l} \mathbb {E} \left[ \left(\mathbf {x} _ {i} - \mathbf {e} _ {i}\right) ^ {\top} \left(\mathbf {x} _ {j} - \mathbf {e} _ {j}\right) \right] = \mathbb {E} \left[ \mathbb {E} \left[ \left(\mathbf {x} _ {i} - \mathbf {e} _ {i}\right) ^ {\top} \left(\mathbf {x} _ {j} - \mathbf {e} _ {j}\right) | \xi_ {i}, \dots , \xi_ {1} \right] \right] \\ = \mathbb {E} \left[ \left(\mathbf {x} _ {i} - \mathbf {e} _ {i}\right) ^ {\top} \underbrace {\mathbb {E} \left[ \left(\mathbf {x} _ {j} - \mathbf {e} _ {j}\right) | \xi_ {i} , \dots , \xi_ {1} \right]} _ {= 0} \right] \\ = 0 \\ \end{array}
$$
Here  $\mathbb{E}[(\mathbf{x}_j - \mathbf{e}_j)|\xi_i,\dots ,\xi_1] = \mathbf{0}$  can be proved by mathematical induction and the law of total expectation. Then,
$$
\mathbb {E} \left[\left\| \sum_ {i = 1} ^ {n} (\mathbf {x} _ {i} - \mathbf {e} _ {i}) \right\| ^ {2} \right] = \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \| \mathbf {x} _ {i} - \mathbf {e} _ {i} \| ^ {2} \right] = \sum_ {i = 1} ^ {n} \mathbb {E} \left[\left.\left\| \mathbf {x} _ {i} - \mathbf {e} _ {i} \right\| ^ {2} \right| \xi_ {i - 1}, \ldots , \xi_ {1} \right]\left. \right] \leq n \sigma^ {2},
$$
which is the claim of this lemma. Note that since  $\mathbb{E}_{\xi_i}[\mathbf{x}_i|\xi_{i - 1},\ldots \xi_1] = \mathbf{e}_i$ , the conditional expectation  $\mathbf{e}_i$  is not deterministic but a function of  $\xi_{i - 1},\dots ,\xi_1$ .
Lemma 2 (Karimireddy et al. (2020)). The following holds for any  $L$ -smooth and  $\mu$ -strongly convex function  $h$ , and any  $\pmb{x}, \pmb{y}, \pmb{z}$  in the domain of  $h$ :
$$
\langle \nabla h (\boldsymbol {x}), \boldsymbol {z} - \boldsymbol {y} \rangle \geq h (\boldsymbol {z}) - h (\mathbf {y}) + \frac {\mu}{4} \| \boldsymbol {y} - \boldsymbol {z} \| ^ {2} - L \| \boldsymbol {z} - \boldsymbol {x} \| ^ {2}. \tag {21}
$$
Proof. Using the three-point-identity, we get
$$
\langle \nabla h (\boldsymbol {x}), \boldsymbol {z} - \boldsymbol {y} \rangle = D _ {h} (\boldsymbol {y}, \boldsymbol {x}) - D _ {h} (\boldsymbol {z}, \boldsymbol {x}) + h (\boldsymbol {z}) - h (\boldsymbol {y})
$$
Then, we get the following two inequalities using smoothness and strong convexity of  $h$ :
$$
\langle \nabla h (\boldsymbol {x}), \boldsymbol {z} - \boldsymbol {y} \rangle \geq \frac {\mu}{2} \| \boldsymbol {y} - \boldsymbol {x} \| ^ {2} - \frac {L}{2} \| \boldsymbol {z} - \boldsymbol {x} \| ^ {2} + h (\boldsymbol {z}) - h (\boldsymbol {y})
$$
Further, using Jensen's inequality (i.e.,  $\| \pmb {y} - \pmb {z}\| ^2\leq 2(\| \pmb {x} - \pmb {z}\| ^2 +\| \pmb {y} - \pmb {x}\| ^2))$  , we have
$$
\frac {\mu}{2} \| \boldsymbol {y} - \boldsymbol {x} \| ^ {2} \geq \frac {\mu}{4} \| \boldsymbol {y} - \boldsymbol {z} \| ^ {2} - \frac {\mu}{2} \| \boldsymbol {x} - \boldsymbol {z} \| ^ {2}.
$$
Combining all the inequalities together we have
$$
\begin{array}{l} \langle \nabla h (\boldsymbol {x}), \boldsymbol {z} - \boldsymbol {y} \rangle \geq h (\boldsymbol {z}) - h (\boldsymbol {y}) + \frac {\mu}{4} \| \boldsymbol {y} - \boldsymbol {z} \| ^ {2} - \frac {L + \mu}{2} \| \boldsymbol {z} - \boldsymbol {x} \| ^ {2} \\ \geq h (\boldsymbol {z}) - h (\boldsymbol {y}) + \frac {\mu}{4} \| \boldsymbol {y} - \boldsymbol {z} \| ^ {2} - L \| \boldsymbol {z} - \boldsymbol {x} \| ^ {2} \quad \triangleright \mu \leq L \\ \end{array}
$$
which is the claim of this lemma.
Lemma 3 (Simple Random Sampling). Let  $\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_n$  be fixed units (e.g., vectors). The population mean and population variance are given as
$$
\overline {{\boldsymbol {x}}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {x} _ {i} \quad \zeta^ {2} := \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \boldsymbol {x} _ {i} - \overline {{\boldsymbol {x}}} \right\| ^ {2}
$$
Draw  $s \in [n] = \{1, 2, \dots, n\}$  random units  $\mathbf{x}_{\pi_1}, \mathbf{x}_{\pi_2}, \dots, \mathbf{x}_{\pi_s}$  randomly from the population. There are two possible ways of simple random sampling, well known as "sampling with replacement (SWR)" and "sampling without replacement (SWOR)". For these two ways, the expectation and variance of the sample mean  $\overline{\mathbf{x}}_{\pi} \coloneqq \frac{1}{s} \sum_{p=1}^{s} \mathbf{x}_{\pi_p}$  satisfies
$$
S W R: \mathbb {E} [ \overline {{\mathbf {x}}} _ {\pi} ] = \overline {{\boldsymbol {x}}} \quad \mathbb {E} \left[ \| \overline {{\mathbf {x}}} _ {\pi} - \overline {{\boldsymbol {x}}} \| ^ {2} \right] = \frac {\zeta^ {2}}{s} \tag {22}
$$
$$
S W O R: \mathbb {E} [ \overline {{\mathbf {x}}} _ {\pi} ] = \overline {{\boldsymbol {x}}} \quad \mathbb {E} \left[ \| \overline {{\mathbf {x}}} _ {\pi} - \overline {{\boldsymbol {x}}} \| ^ {2} \right] = \frac {n - s}{s (n - 1)} \zeta^ {2} \tag {23}
$$
Proof. The proof of this lemma is mainly based on Mishchenko et al. (2020)'s Lemma 1 (A lemma for sampling without replacement) and Wang et al. (2020)'s Appendix G (Extension: Incorporating Client Sampling). Since the probability of each unit being selected equals  $\frac{1}{n}$  in each draw, we can get the expectation and variance of any random unit  $\mathbf{x}_{\pi_p}$  at the  $p$ -th draw:
$$
\begin{array}{l} \mathbb {E} \left[ \mathbf {x} _ {\pi_ {p}} \right] = \sum_ {i = 1} ^ {n} \boldsymbol {x} _ {i} \cdot \operatorname * {P r} (\mathbf {x} _ {\pi_ {p}} = \boldsymbol {x} _ {i}) = \sum_ {i = 1} ^ {n} \boldsymbol {x} _ {i} \cdot \frac {1}{n} = \overline {{\boldsymbol {x}}}, \\ \mathrm {V a r} (\mathbf {x} _ {\pi_ {p}}) = \mathbb {E} \left[ \left\| \mathbf {x} _ {\pi_ {p}} - \overline {{\boldsymbol {x}}} \right\| ^ {2} \right] = \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} - \overline {{\boldsymbol {x}}} \| ^ {2} \cdot \mathrm {P r} (\mathbf {x} _ {\pi_ {p}} = \boldsymbol {x} _ {i}) = \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} - \overline {{\boldsymbol {x}}} \| ^ {2} \cdot \frac {1}{n} = \zeta^ {2}, \\ \end{array}
$$
where the preceding equations hold for both sampling ways. Thus, we can compute the expectations of the sample mean for both sampling ways as
$$
\mathbb {E} \left[ \overline {{\mathbf {x}}} _ {\pi} \right] = \mathbb {E} \left[ \frac {1}{s} \sum_ {p = 1} ^ {s} \mathbf {x} _ {\pi_ {p}} \right] = \frac {1}{s} \sum_ {p = 1} ^ {s} \mathbb {E} \left[ \mathbf {x} _ {\pi_ {p}} \right] = \overline {{\mathbf {x}}},
$$
which indicates that the sample means for both ways are unbiased estimators of the population mean. The variance of the sample mean can be decomposed as
$$
\mathbb {E} \left\| \overline {{\mathbf {x}}} _ {\pi} - \overline {{\mathbf {x}}} \right\| ^ {2} = \mathbb {E} \left\| \frac {1}{s} \sum_ {p = 1} ^ {s} (\mathbf {x} _ {\pi_ {p}} - \overline {{\mathbf {x}}}) \right\| ^ {2} = \frac {1}{s ^ {2}} \sum_ {p = 1} ^ {s} \mathrm {V a r} (\mathbf {x} _ {\pi_ {p}}) + \frac {1}{s ^ {2}} \sum_ {p = 1} ^ {s} \sum_ {q \neq p} ^ {s} \mathrm {C o v} (\mathbf {x} _ {\pi_ {p}}, \mathbf {x} _ {\pi_ {q}})
$$
Next, we deal with these two ways separately:
- SWR: It holds that  $\mathrm{Cov}(\mathbf{x}_{\pi_p}, \mathbf{x}_{\pi_q}) = 0, \forall p \neq q$  since  $\mathbf{x}_{\pi_p}, \mathbf{x}_{\pi_q}$  are independent for SWR. Thus, we can get  $\mathbb{E} \left\| \overline{\mathbf{x}}_{\pi} - \overline{\mathbf{x}} \right\|^{2} = \frac{1}{s^{2}} \sum_{p=1}^{s} \mathrm{Var}(\mathbf{x}_{\pi_p}) = \frac{\zeta^{2}}{s}$ .
- SWOR: For  $p \neq q$ , we have
$$
\mathrm {C o v} (\mathbf {x} _ {\pi_ {p}}, \mathbf {x} _ {\pi_ {q}}) = \mathbb {E} \left[ \left\langle \mathbf {x} _ {\pi_ {p}} - \overline {{\boldsymbol {x}}}, \mathbf {x} _ {\pi_ {q}} - \overline {{\boldsymbol {x}}} \right\rangle \right] = \sum_ {i = 1} ^ {n} \sum_ {j \neq i} ^ {n} \langle \boldsymbol {x} _ {i} - \overline {{\boldsymbol {x}}}, \boldsymbol {x} _ {j} - \overline {{\boldsymbol {x}}} \rangle \cdot \mathrm {P r} (\mathbf {x} _ {\pi_ {p}} = \boldsymbol {x} _ {i}, \mathbf {x} _ {\pi_ {q}} = \boldsymbol {x} _ {j}),
$$
Since there are  $n(n - 1)$  possible combinations of  $(\mathbf{x}_{\pi_p}, \mathbf{x}_{\pi_q})$  and each has the same probability, we get  $\operatorname*{Pr}(\mathbf{x}_{\pi_p} = \mathbf{x}_i, \mathbf{x}_{\pi_q} = \mathbf{x}_j) = \frac{1}{n(n - 1)}$ . As a consequence, we have
$$
\begin{array}{l} \operatorname {C o v} (\mathbf {x} _ {\pi_ {p}}, \mathbf {x} _ {\pi_ {q}}) = \frac {1}{n (n - 1)} \sum_ {i = 1} ^ {n} \sum_ {j \neq i} ^ {n} [ \langle \mathbf {x} _ {i} - \overline {{\mathbf {x}}}, \mathbf {x} _ {j} - \overline {{\mathbf {x}}} \rangle ] \\ = \frac {1}{n (n - 1)} \left\| \sum_ {i = 1} ^ {n} \left(\boldsymbol {x} _ {i} - \overline {{\boldsymbol {x}}}\right) \right\| ^ {2} - \frac {1}{n (n - 1)} \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} - \overline {{\boldsymbol {x}}} \| ^ {2} \\ = - \frac {\zeta^ {2}}{n - 1} \tag {24} \\ \end{array}
$$
Thus we have  $\mathbb{E}\left\| \overline{\mathbf{x}}_{\pi} - \overline{\mathbf{x}}\right\| ^2 = \frac{\zeta^2}{s} -\frac{s(s - 1)}{s^2}\cdot \frac{\zeta^2}{n - 1} = \frac{(n - s)}{s(n - 1)}\zeta^2.$
When  $n$  is infinite (or large enough), we get  $\frac{(\mathbb{E}||\overline{\mathbf{x}}_{\pi} - \overline{\mathbf{x}} ||^{2})_{\mathrm{SWOR}}}{(\mathbb{E}||\overline{\mathbf{x}}_{\pi} - \overline{\mathbf{x}} ||^{2})_{\mathrm{SWR}}} \approx 1 - \frac{s}{n}$ . This constant has appeared in Karimireddy et al. (2020)'s Lemma 7 (one round progress) and Woodworth et al. (2020b)'s Section 7 (Using a Subset of Machines in Each Round).
Lemma 4. Under the same conditions of Lemma 3, use the way "sampling without replacement" and let  $b_{m,k}(i) = \begin{cases} K - 1, & i \leq m - 1 \\ k - 1, & i = m \end{cases}$ . Then for  $S \leq M$  ( $M \geq 2$ ), it holds that
$$
\sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right) \right\| ^ {2} \right] \leq \frac {1}{2} S ^ {2} K ^ {3} \zeta^ {2} \tag {25}
$$
Proof. Let us focus on the term in the following:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} (\mathbf {x} _ {\pi_ {i}} - \overline {{\mathbf {x}}}) \right\| ^ {2} \right] = \mathbb {E} \left[ \left\| K \sum_ {i = 1} ^ {m - 1} (\mathbf {x} _ {\pi_ {i}} - \overline {{\mathbf {x}}}) + k (\mathbf {x} _ {\pi_ {m}} - \overline {{\mathbf {x}}}) \right\| ^ {2} \right] \\ = K ^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m - 1} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right) \right\| ^ {2} \right] + k ^ {2} \mathbb {E} \left[ \left\| \mathbf {x} _ {\pi_ {m}} - \overline {{\boldsymbol {x}}} \right\| ^ {2} \right] + 2 K k \mathbb {E} \left[ \left\langle \sum_ {i = 1} ^ {m - 1} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right), \left(\mathbf {x} _ {\pi_ {m}} - \overline {{\boldsymbol {x}}}\right) \right\rangle \right] \tag {26} \\ \end{array}
$$
For the first term on the right hand side in (26), using (23), we have
$$
K ^ {2} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m - 1} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right) \right\| ^ {2} \right] \stackrel {(2 3)} {=} \frac {(m - 1) (M - (m - 1))}{M - 1} K ^ {2} \zeta^ {2}.
$$
For the second term on the right hand side in (26), we have
$$
k ^ {2} \mathbb {E} \left[ \left\| \mathbf {x} _ {\pi_ {m}} - \overline {{\boldsymbol {x}}} \right\| ^ {2} \right] = k ^ {2} \mathbb {E} \left[ \left\| \mathbf {x} _ {\pi_ {m}} - \overline {{\boldsymbol {x}}} \right\| ^ {2} \right] = k ^ {2} \zeta^ {2}.
$$
For the third term on the right hand side in (26), we have
$$
2 K k \mathbb {E} \left[ \left\langle \sum_ {i = 1} ^ {m - 1} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right), \left(\mathbf {x} _ {\pi_ {m}} - \overline {{\boldsymbol {x}}}\right) \right\rangle \right] = 2 K k \sum_ {i = 1} ^ {m - 1} \mathbb {E} \left[ \left\langle \mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}, \mathbf {x} _ {\pi_ {m}} - \overline {{\boldsymbol {x}}} \right\rangle \right] \stackrel {(2 4)} {=} - \frac {2 (m - 1)}{M - 1} K k \zeta^ {2},
$$
where we use (24) in the last equality, since  $i \in \{1, 2, \ldots, m - 1\} \neq m$ . With these three preceding equations, we get
$$
\mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right) \right\| ^ {2} \right] = \frac {(m - 1) (M - (m - 1))}{M - 1} K ^ {2} \zeta^ {2} + k ^ {2} \zeta^ {2} - \frac {2 (m - 1)}{M - 1} K k \zeta^ {2}
$$
Then summing the preceding terms over  $m$  and  $k$ , we can get
$$
\begin{array}{l} \sum_ {m = 1} ^ {S} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\mathbf {x}}}\right) \right\| ^ {2} \right] \\ = \frac {M K ^ {3} \zeta^ {2}}{M - 1} \sum_ {m = 1} ^ {S} (m - 1) - \frac {K ^ {3} \zeta^ {2}}{M - 1} \sum_ {m = 1} ^ {S} (m - 1) ^ {2} + S \zeta^ {2} \sum_ {k = 0} ^ {K - 1} k ^ {2} - \frac {2 K \zeta^ {2}}{M - 1} \sum_ {m = 1} ^ {S} (m - 1) \sum_ {k = 0} ^ {K - 1} k ^ {2}. \\ \end{array}
$$
Then applying the fact that  $\sum_{k=1}^{K-1} k = \frac{(K-1)K}{2}$  and  $\sum_{k=1}^{K-1} k^2 = \frac{(K-1)K(2K-1)}{6}$ , we can simplify the preceding equation as
$$
\begin{array}{l} \sum_ {m = 1} ^ {M} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left\| \sum_ {i = 1} ^ {m} \sum_ {j = 0} ^ {b _ {m, k} (i)} \left(\mathbf {x} _ {\pi_ {i}} - \overline {{\boldsymbol {x}}}\right) \right\| ^ {2} \\ = \frac {1}{2} S K ^ {2} (S K - 1) - \frac {1}{6} S K (K ^ {2} - 1) - \frac {1}{M - 1} (S - 1) S \left(\frac {1}{6} (2 S - 1) K - \frac {1}{2}\right) \leq \frac {1}{2} S ^ {2} K ^ {3} \zeta^ {2}, \\ \end{array}
$$
D Proofs of Theorem 1
In this section, we provide the proof of Theorem 1 for the strongly convex, general convex and non-convex cases in D.1, D.2 and D.3, respectively.