SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yipeng Li and Xinchen Lyu *
National Engineering Research Center for Mobile Network Technologies
Beijing University of Posts and Telecommunications
Beijing, 100876, China
{liyipeng, lvxinchen}@bupt.edu.cn
There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) sequential FL (SFL), where clients train models in a sequential manner. In contrast to that of PFL, the convergence theory of SFL on heterogeneous data is still lacking. In this paper, we establish the convergence guarantees of SFL for strongly/general/non-convex objectives on heterogeneous data. The convergence guarantees of SFL are better than that of PFL on heterogeneous data with both full and partial client participation. Experimental results validate the counterintuitive analysis result that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.
Federated Learning (FL) (McMahan et al., 2017) is a popular distributed machine learning paradigm, where multiple clients collaborate to train a global model. To preserve data privacy and security, data must be kept in clients locally cannot be shared with others, causing one severe and persistent issue, namely "data heterogeneity". In cross-device FL, where data is generated and kept in massively distributed resource-constrained devices (e.g., IoT devices), the negative impact of data heterogeneity would be further exacerbated (Jhunjunwala et al., 2023).
There are two categories of methods in FL to enable distributed training across multiple clients (Qu et al., 2022): i) parallel FL (PFL), where models are trained in a parallel manner across clients with synchronization at intervals, e.g., Federated Averaging (FedAvg) (McMahan et al., 2017); and ii) sequential FL (SFL), where models are trained in a sequential manner across clients, e.g., Cyclic Weight Transfer (CWT) (Chang et al., 2018). However, both categories of methods suffer from the "client drift" (Karimireddy et al., 2020), i.e., the local updates on heterogeneous clients would drift away from the right direction, resulting in performance degradation.
Motivation. Recently, SFL (more generally, the sequential training manner, see Algorithm 1) has attracted much attention in the FL community (Lee et al., 2020). Specifically, SFL demonstrates advantages on training speed (in terms of training rounds) (Zaccone et al., 2022) and small datasets (Kamp et al., 2023), and both are crucial for cross-device FL. Furthermore, the sequential manner has played a great role in Split learning (SL) (Gupta and Raskar, 2018; Thapa et al., 2022), an emerging distributed learning technology at the edge side (Zhou et al., 2019), where the full model is split into client-side and server-side portions to alleviate the excessive computation overhead for resource-constrained devices. Appendix A will show that the convergence theory in this work is also applicable to SL.
* Xinchen Lyu is the corresponding author.
Convergence theory is critical for analyzing the learning performance of algorithms on heterogeneous data in FL. So far, there are numerous works to analyze the convergence of PFL (Li et al., 2019; Khaled et al., 2020; Koloskova et al., 2020) on heterogeneous data. However, the convergence theory of SFL on heterogeneous data, given the complexity of its sequential training manner, has not been well investigated in the literature, with only limited preliminary empirical studies Gao et al. (2020, 2021). This paper aims to establish the convergence guarantees for SFL and compare the convergence results of PFL and SFL.
Setup. In the following, we provide some preliminaries about SFL and PFL.
Problem formulation. The basic FL problem is to minimize a global objective function:
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \left\{F (\mathbf {x}) := \frac {1}{M} \sum_ {m = 1} ^ {M} \left(F _ {m} (\mathbf {x}) := \mathbb {E} _ {\xi \sim \mathcal {D} _ {m}} [ f _ {m} (\mathbf {x}; \xi) ]\right) \right\},
$$
where  $F_{m}$ ,  $f_{m}$  and  $\mathcal{D}_m$  denote the local objective function, the loss function and the local dataset of client  $m$  ( $m\in [M]$ ), respectively. In particular, when  $\mathcal{D}_m$  has finite data samples  $\{\xi_m^i:i\in [|D_m|]\}$ , the local objective function can also be written as  $F_{m}(\mathbf{x}) = \frac{1}{|\mathcal{D}_{m}|}\sum_{i = 1}^{|\mathcal{D}_{m}|}f_{m}(\mathbf{x};\xi_{m}^{i})$ .
Update rule of SFL. At the beginning of each training round, the indices  $\pi_1, \pi_2, \ldots, \pi_M$  are sampled without replacement from  $\{1, 2, \ldots, M\}$  randomly as the clients' training order. Within a round, each client i) initializes its model with the latest parameters from its previous client; ii) performs  $K$  steps of local updates over its local dataset; and iii) passes the updated parameters to the next client. This process continues until all clients finish their local training. Let  $\mathbf{x}_{m,k}^{(r)}$  denote the local parameters of the  $m$ -th client (i.e., client  $\pi_m$ ) after  $k$  local steps in the  $r$ -th round, and  $\mathbf{x}^{(r)}$  denote the global parameter in the  $r$ -th round. With SGD (Stochastic Gradient Descent) as the local solver, the update rule of SFL is as follows:
Local update:  $\mathbf{x}_{m,k + 1}^{(r)} = \mathbf{x}_{m,k}^{(r)} - \eta \mathbf{g}_{\pi_m,k}^{(r)}$  initializing as  $\mathbf{x}_{m,0}^{(r)} = \left\{ \begin{array}{ll}\mathbf{x}^{(r)}, & m = 1\\ \mathbf{x}_{m - 1,K}^{(r)}, & m > 1 \end{array} \right.$
Global model:  $\mathbf{x}^{(r + 1)} = \mathbf{x}_{M,K}^{(r)}$
where  $\mathbf{g}_{\pi_m,k}^{(r)}\coloneqq \nabla f_{\pi_m}(\mathbf{x}_{m,k}^{(r)};\xi)$  denotes the stochastic gradient of  $F_{\pi_m}$  regarding parameters  $\mathbf{x}_{m,k}^{(r)}$  and  $\eta$  denotes the learning rate. See Algorithm 1. Notations are summarized in Appendix C.1.
Update rule of PFL. Within a round, each client i) initializes its model with the global parameters; ii) performs  $K$  steps of local updates; and iii) sends the updated parameters to the central server. The server will aggregate the local parameters to generate the global parameters. See Algorithm 2
In this work, unless otherwise stated, we use SFL and PFL to represent the classes of algorithms that share the same update rule as Algorithm 1 and Algorithm 2, respectively.
Algorithm 1: Sequential FL
Output:  $\bar{\mathbf{x}}^{(R)}$  : weighted average on  $\mathbf{x}^{(r)}$    
1 for training round  $r = 0,1,\dots ,R - 1$  do Sample a permutation  $\pi_1,\pi_2,\ldots ,\pi_M$  of  $\{1,2,\dots ,M\}$    
3 for  $m = 1,2,\dots ,M$  in sequence do  $\mathbf{x}_{m,0}^{(r)} = \left\{ \begin{array}{ll}\mathbf{x}^{(r)}, & m = 1\\ \mathbf{x}_{m - 1,K}^{(r)}, & m > 1 \end{array} \right.$    
5 for local step  $k = 0,\dots ,K - 1$  do  $\mathbf{x}_{m,k + 1}^{(r)} = \mathbf{x}_{m,k}^{(r)} - \eta \mathbf{g}_{\pi_m,k}^{(r)}$    
7 Global model:  $\mathbf{x}^{(r + 1)} = \mathbf{x}_{M,K}^{(r)}$
Algorithm 2: Parallel FL
Output:  $\bar{\mathbf{x}}^{(R)}$  : weighted average on  $\mathbf{x}^{(r)}$    
1 for training round  $r = 0,1,\ldots ,R - 1$  do   
2 for  $m = 1,2,\dots ,M$  in parallel do   
3  $\mathbf{x}_{m,0}^{(r)} = \mathbf{x}^{(r)}$    
4 for local step  $k = 0,\dots ,K - 1$  do   
5  $\begin{array}{r}\mathbf{x}_{m,k + 1}^{(r)} = \mathbf{x}_{m,k}^{(r)} - \eta \mathbf{g}_{m,k}^{(r)} \end{array}$    
6 Global model:  $\mathbf{x}^{(r + 1)} = \frac{1}{M}\sum_{m = 1}^{M}\mathbf{x}_{m,K}^{(r)}$
2 Contributions
Brief literature review. The most relevant work is the convergence of PFL and Random Reshuffling (SGD-RR). There are a wealth of works that have analyzed the convergence of PFL on data heterogeneity (Li et al., 2019; Khaled et al., 2020; Karimireddy et al., 2020; Koloskova et al., 2020; Woodworth et al., 2020b), system heterogeneity (Wang et al., 2020), partial client participation (Li et al., 2019; Yang et al., 2021; Wang and Ji, 2022) and other variants (Karimireddy et al., 2020; Reddi et al., 2021). In this work, we compare the convergence bounds between PFL and SFL (see Subsection 3.3) on heterogeneous data.
SGD-RR (where data samples are sampled without replacement) is deemed to be more practical than SGD (where data samples are sample with replacement), and thus attracts more attention recently. Gurbuzbalaban et al. (2021); Haochen and Sra (2019); Nagaraj et al. (2019); Ahn et al. (2020); Mishchenko et al. (2020) have proved the upper bounds and Safran and Shamir (2020, 2021); Rajput et al. (2020); Cha et al. (2023) have proved the lower bounds of SGD-RR. In particular, the lower bounds in Cha et al. (2023) are shown to match the upper bounds in Mishchenko et al. (2020). In this work, we use the bounds of SGD-RR to exam the tightness of that of SFL (see Subsection 3.2).
Recently, the shuffling-based method has been applied to FL (Mishchenko et al., 2022; Yun et al., 2022; Cho et al., 2023; Malinovsky et al., 2023). The most relevant works are FL with cyclic client participation (Cho et al., 2023) and FL with shuffling client participation (Malinovsky et al., 2023). The detailed comparisons are given in Appendix B.
Challenges. The theory of SGD is applicable to SFL on homogeneous data, where SFL can be reduced to SGD. However, the theory of SGD can be no longer applicable to SFL on heterogeneous data. This is because for any pair of indices  $m$  and  $k$  (except  $m = 1$  and  $k = 0$ ) within a round, the stochastic gradient is not an (conditionally) unbiased estimator of the global objective:
$$
\mathbb {E} \left[ \nabla f _ {\pi_ {m}} \left(\mathbf {x} _ {m, k}; \xi\right) \mid \mathbf {x} \right] \neq \nabla F \left(\mathbf {x} _ {m, k}\right).
$$
In general, the challenges of establishing convergence guarantees of SFL mainly arise from (i) the sequential training manner across clients and (ii) multiple local steps of SGD at each client.
Sequential training manner across clients (vs. PFL). In PFL, local model parameters are updated in parallel within each round and synchronized at the end of the round. In this case, the local updates across clients are mutually independent when conditional on all the randomness prior to the round. However, in SFL, client's local updates additionally depend on the randomness of all previous clients. This makes bounding the client drift of SFL more complex than that of PFL.
Multiple local steps of SGD at each client (vs. SGD-RR). SGD-RR samples data samples without replacement and then performs one step of gradient descent (GD) on each data sample. Similarly, SFL samples clients without replacement and then performs multiple steps of SGD on each local objective (i.e., at each client). In fact, SGD-RR can be regarded as a special case of SFL. Thus, the derivation of convergence guarantees of SFL is also more complex than that of SGD-RR.
Contributions. The main contributions are as follows:
- We derive convergence guarantees of SFL for strongly convex, general convex and non-convex objectives on heterogeneous data with the standard assumptions in FL in Subsection 3.2.
- We compare the convergence guarantees of PFL and SFL, and find a counterintuitive comparison result that the guarantee of SFL is better than that of PFL (with both full participation and partial participation) in terms of training rounds on heterogeneous data in Subsection 3.3.
- We validate our comparison result with simulations on quadratic functions (Subsection 4.1) and experiments on real datasets (Subsection 4.2). The experimental results exhibit that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.
3 Convergence theory
We consider three typical cases for convergence theory, i.e., the strongly convex case, the general convex case and the non-convex case, where all local objectives  $F_{1}, F_{2}, \ldots, F_{M}$  are  $\mu$ -strongly convex, general convex  $(\mu = 0)$  and non-convex.
3.1 Assumptions
We assume that (i)  $F$  is lower bounded by  $F^{*}$  for all cases and there exists a minimizer  $\mathbf{x}^*$  such that  $F(\mathbf{x}^{*}) = F^{*}$  for strongly and general convex cases; (ii) each local objective function is  $L$ -smooth (Assumption 1). Furthermore, we need to make assumptions on the diversities: (iii) the assumptions on the stochasticity bounding the diversity of  $\{f_{m}(\cdot ;\xi_{m}^{i}):i\in [|D_{m}|]\}$  with respect to  $i$  inside each client (Assumption 2); (iv) the assumptions on the heterogeneity bounding the diversity of local objectives  $\{F_m:m\in [M]\}$  with respect to  $m$  across clients (Assumptions 3a, 3b).
Assumption 1 (L-Smoothness). Each local objective function  $F_{m}$  is L-smooth,  $m\in \{1,2,\dots ,M\}$ , i.e., there exists a constant  $L > 0$  such that  $\| \nabla F_m(\mathbf{x}) - \nabla F_m(\mathbf{y})\| \leq L\| \mathbf{x} - \mathbf{y}\|$  for all  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$ .
Assumptions on the stochasticity. Since both Algorithms 1 and 2 use SGD (data samples are chosen with replacement) as the local solver, the stochastic gradient at each client is an (conditionally) unbiased estimate of the gradient of the local objective function:  $\mathbb{E}_{\xi \sim \mathcal{D}_m}[f_m(\mathbf{x};\xi)|\mathbf{x}] = \nabla F_m(\mathbf{x})$ . Then we use Assumption 2 to bound the stochasticity, where  $\sigma$  measures the level of stochasticity.
Assumption 2. The variance of the stochastic gradient at each client is bounded:
$$
\mathbb {E} _ {\xi \sim \mathcal {D} _ {m}} \left[ \| \nabla f _ {m} (\mathbf {x}; \xi) - \nabla F _ {m} (\mathbf {x}) \| ^ {2} \mid \mathbf {x} \right] \leq \sigma^ {2}, \quad \forall m \in \{1, 2, \dots , M \} \tag {1}
$$
Assumptions on the heterogeneity. Now we make assumptions on the diversity of the local objective functions in Assumption 3a and Assumption 3b, also known as the heterogeneity in FL. Assumption 3a is made for non-convex cases, where the constants  $\beta$  and  $\zeta$  measure the heterogeneity of the local objective functions, and they equal zero when all the local objective functions are identical to each other. Further, if the local objective functions are strongly and general convex, we use one weaker assumption 3b as Koloskova et al. (2020), which bounds the diversity only at the optima.
Assumption 3a. There exist constants  $\beta^2$  and  $\zeta^2$  such that
$$
\frac {1}{M} \sum_ {m = 1} ^ {M} \| \nabla F _ {m} (\mathbf {x}) - \nabla F (\mathbf {x}) \| ^ {2} \leq \beta^ {2} \| \nabla F (\mathbf {x}) \| ^ {2} + \zeta^ {2} \tag {2}
$$
Assumption 3b. There exists one constant  $\zeta_{*}^{2}$  such that
$$
\frac {1}{M} \sum_ {m = 1} ^ {M} \| \nabla F _ {m} \left(\mathbf {x} ^ {*}\right) \| ^ {2} = \zeta_ {*} ^ {2} \tag {3}
$$
where  $\mathbf{x}^{*}\in \arg \min_{\mathbf{x}\in \mathbb{R}^{d}}F(\mathbf{x})$  is one global minimizer.
3.2 Convergence analysis of SFL
Theorem 1. For SFL (Algorithm 1), there exist a constant effective learning rate  $\tilde{\eta} \coloneqq MK\eta$  and weights  $w_{r}$ , such that the weighted average of the global parameters  $\bar{\mathbf{x}}^{(R)} \coloneqq \frac{1}{W_R}\sum_{r=0}^{R} w_r \mathbf{x}^{(r)}$  ( $W_R = \sum_{r=0}^{R} w_r$ ) satisfies the following upper bounds:
Strongly convex: Under Assumptions 1, 2, 3b, there exist a constant effective learning rate  $\frac{1}{\mu R} \leq \tilde{\eta} \leq \frac{1}{6L}$  and weights  $w_{r} = (1 - \frac{\mu\tilde{\eta}}{2})^{-(r + 1)}$ , such that it holds that
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] \leq \frac {9}{2} \mu D ^ {2} \exp \left(- \frac {\mu \tilde {\eta} R}{2}\right) + \frac {1 2 \tilde {\eta} \sigma^ {2}}{M K} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{M K} + \frac {1 8 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2}}{M} \tag {4}
$$
General convex: Under Assumptions 1, 2, 3b, there exist a constant effective learning rate  $\tilde{\eta} \leq \frac{1}{6L}$  and weights  $w_{r} = 1$ , such that it holds that
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] \leq \frac {3 D ^ {2}}{\tilde {\eta} R} + \frac {1 2 \tilde {\eta} \sigma^ {2}}{M K} + \frac {1 8 L \tilde {\eta} ^ {2} \sigma^ {2}}{M K} + \frac {1 8 L \tilde {\eta} ^ {2} \zeta_ {*} ^ {2}}{M} \tag {5}
$$
Non-convex: Under Assumptions 1, 2, 3a, there exist a constant effective learning rate  $\tilde{\eta} \leq \frac{1}{6L(\beta + 1)}$  and weights  $w_{r} = 1$ , such that it holds that
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \| \nabla F (\mathbf {x} ^ {(r)}) \| ^ {2} \right] \leq \frac {3 A}{\tilde {\eta} R} + \frac {3 L \tilde {\eta} \sigma^ {2}}{M K} + \frac {2 7 L ^ {2} \tilde {\eta} ^ {2} \sigma^ {2}}{8 M K} + \frac {2 7 L ^ {2} \tilde {\eta} ^ {2} \zeta^ {2}}{8 M} \tag {6}
$$
where  $D\coloneqq \left\| x^{(0)} - x^{*}\right\|$  for the convex cases and  $A\coloneqq F(\mathbf{x}^{(0)}) - F^{*}$  for the non-convex case.
The effective learning rate  $\tilde{\eta} \coloneqq MK\eta$  is used in the upper bounds as Karimireddy et al. (2020); Wang et al. (2020) did. All these upper bounds consist of two parts: the optimization part (the first term) and the error part (the last three terms). Setting  $\tilde{\eta}$  larger can make the optimization part vanishes at a higher rate, yet cause the error part to be larger. This implies that we need to choose an appropriate  $\tilde{\eta}$  to achieve a balance between these two parts, which is actually done in Corollary 1. Here we choose the best learning rate with a prior knowledge of the total training rounds  $R$ , as done in the previous works (Karimireddy et al., 2020; Reddi et al., 2021).
Corollary 1. Applying the results of Theorem 1. By choosing a appropriate learning rate (see the proof of Theorem 1 in Appendix D), we can obtain the convergence bounds for SFL as follows:
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] = \tilde {\mathcal {O}} \left(\frac {\sigma^ {2}}{\mu M K R} + \frac {L \sigma^ {2}}{\mu^ {2} M K R ^ {2}} + \frac {L \zeta_ {*} ^ {2}}{\mu^ {2} M R ^ {2}} + \mu D ^ {2} \exp \left(- \frac {\mu R}{1 2 L}\right)\right) \tag {7}
$$
$$
\mathbb {E} \left[ F \left(\bar {\mathbf {x}} ^ {(R)}\right) - F \left(\mathbf {x} ^ {*}\right) \right] = \mathcal {O} \left(\frac {\sigma D}{\sqrt {M K R}} + \frac {\left(L \sigma^ {2} D ^ {4}\right) ^ {1 / 3}}{\left(M K\right) ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L \zeta_ {*} ^ {2} D ^ {4}\right) ^ {1 / 3}}{M ^ {1 / 3} R ^ {2 / 3}} + \frac {L D ^ {2}}{R}\right) \tag {8}
$$
$$
\min  _ {0 \leq r \leq R} \mathbb {E} \left[ \| \nabla F (\mathbf {x} ^ {(r)}) \| ^ {2} \right] = \mathcal {O} \left(\frac {\left(L \sigma^ {2} A\right) ^ {1 / 2}}{\sqrt {M K R}} + \frac {\left(L ^ {2} \sigma^ {2} A ^ {2}\right) ^ {1 / 3}}{(M K) ^ {1 / 3} R ^ {2 / 3}} + \frac {\left(L ^ {2} \zeta^ {2} A ^ {2}\right) ^ {1 / 3}}{M ^ {1 / 3} R ^ {2 / 3}} + \frac {L \beta A}{R}\right) \tag {9}
$$
where  $\mathcal{O}$  omits absolute constants,  $\tilde{\mathcal{O}}$  omits absolute constants and polylogarithmic factors,  $D\coloneqq \left\| x^{(0)} - x^{*}\right\|$  for the convex cases and  $A\coloneqq F(\mathbf{x}^{(0)}) - F^{*}$  for the non-convex case.
Convergence rate. By Corollary 1, for sufficiently large  $R$ , the convergence rate is determined by the first term for all cases, resulting in convergence rates of  $\tilde{\mathcal{O}}(1 / MKR)$  for strongly convex cases,  $\mathcal{O}(1 / \sqrt{MKR})$  for general convex cases and  $\mathcal{O}(1 / \sqrt{MKR})$  for non-convex cases.
SGD-RR vs. SFL. Recall that SGD-RR can be seen as one special case of SFL, where one step of GD is performed on each local objective  $F_{m}$  (i.e.,  $K = 1$  and  $\sigma = 0$ ). The bound of SFL turns to  $\tilde{\mathcal{O}}\left(\frac{L\zeta_*^2}{\mu^2MR^2} + \mu D^2\exp\left(-\frac{\mu R}{L}\right)\right)$  when  $K = 1$  and  $\sigma = 0$  for the strongly convex case. Then let us borrow the upper bound from Mishchenko et al. (2020)'s Corollary 1,
$$
\text {(S t r o n g l y c o n v e x)} \quad \mathbb {E} \left[ \| \mathbf {x} ^ {(R)} - \mathbf {x} ^ {*} \| ^ {2} \right] = \tilde {\mathcal {O}} \left(\frac {L \zeta_ {*} ^ {2}}{\mu^ {3} M R ^ {2}} + D ^ {2} \exp \left(- \frac {\mu M R}{L}\right)\right).
$$
As we can see, the bound of SGD-RR only has an advantage on the second term (marked in red), which can be omitted for sufficiently large  $R$ . The difference on the constant  $\mu$  is because their bound is for  $\mathbb{E}\left[\| \mathbf{x}^{(R)} - \mathbf{x}^*\|^2\right]$  (see Stich (2019b)). Furthermore, our bound also matches the lower bound  $\Omega \left(\frac{L\zeta_s^2}{\mu^2MR^2}\right)$  of SGD-RR suggested by Cha et al. (2023)'s Theorem 3.1 for sufficiently large  $R$ . For the general convex and non-convex cases, the bounds of SFL (when  $K = 1$  and  $\sigma = 0$ ) also match that of SGD-RR (see Mishchenko et al. (2020)'s Theorems 3, 4). These all suggest our bounds are tight. Yet a specialized lower bound for SFL is still required.
Effect of local steps. Two comments are included: i) It can be seen that local updates can help the convergence with proper learning rate choices (small enough) by Corollary 1. Yet this increases the total steps (iterations), leading to a higher computation cost. ii) Excessive local updates do not benefit the dominant term of the convergence rate. Take the strongly convex case as an example.
When  $\frac{\sigma^2}{\mu MKR} \leq \frac{L\zeta_*^2}{\mu^2 MR^2}$ , the latter turns dominant, which is unaffected by  $K$ . In other words, when the value of  $K$  exceeds  $\tilde{\Omega}\left(\sigma^2/\zeta_*^2 \cdot \mu/L \cdot R\right)$ , increasing local updates will no longer benefit the dominant term of the convergence rate. Note that the maximum value of  $K$  is affected by  $\sigma^2/\zeta_*^2$ ,  $\mu/L$  and  $R$ . This analysis follows Reddi et al. (2021); Khaled et al. (2020).
3.3 PFL vs. SFL on heterogeneous data
Table 1: Upper bounds in the strongly convex case with absolute constants and polylogarithmic factors omitted. All results are for heterogeneous settings.
Method	Bound (D = ||x(0) - x* ||)
SGD (Stich, 2019b)	σ2/μMKR + LD2 exp(-μR/L) (1)
PFL	σ2/μMKR + Lσ2/μ2KR2 + Lζ2/μ2R2 + μD2 exp(-μR/L) (2)
(Karimireddy et al., 2020)	σ2/μMKR + Lσ2/μ2KR2 + Lζ2/μ2R2 + LKD2 exp(-μR/L) (3)
(Theorem 2)	σ2/μMKR + Lσ2/μ2KR2 + Lζ2/μ2R2 + μD2 exp(-μR/L)
SFL	σ2/μMKR + Lσ2/μ2MKR2 + Lζ2/μ2MR2 + μD2 exp(-μR/L)
(Theorem 1)
(1) SGD with a large mini-batch size. We get the bound in the table by replacing  $\sigma^2$  in the Stich (2019b)'s result with  $\frac{\sigma^2}{MK}$ . See Woodworth et al. (2020b) for more details about Minibatch SGD.
(2) Karimireddy et al. (2020) use  $\frac{1}{M}\sum_{m = 1}^{M}\| \nabla F_m(\mathbf{x})\|^2\leq B^2\| \nabla F(\mathbf{x})\| +G^2$  to bound the heterogeneity, which is equivalent to Assumption 3a. The global learning rate is not considered in this work.
(3) Koloskova et al. (2020) use  $\sigma_{*}^{2} \coloneqq \frac{1}{M} \sum_{m=1}^{M} \mathbb{E}\left[\|\nabla f_{m}(\mathbf{x}^{*};\xi) - \nabla F_{m}(\mathbf{x}^{*})\|^{2}\right]$  to bound the stochasticity, which is weaker than Assumption 3b.
Unless otherwise stated, our comparison is in terms of training rounds, which is also adopted in Gao et al. (2020, 2021). This comparison (running for the same total training rounds  $R$ ) is fair considering the same total computation cost for both methods.
Convergence results of PFL. We summarize the existing convergence results of PFL for the strongly convex case in Table 1. Here we slightly improve the convergence result for strongly convex cases by combining the works of Karimireddy et al. (2020); Koloskova et al. (2020). Besides, we note that to derive a unified theory of Decentralized SGD, the proofs of Koloskova et al. (2020) are different from other works focusing on PFL. So we reproduce the bounds for general convex and non-convex cases based on Karimireddy et al. (2020). All our results of PFL are in Theorem 2 (see Appendix E).