SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The convergence guarantee of SFL is better than PFL on heterogeneous data. Take the strongly convex case as an example. According to Table 1, the upper bound of SFL is better than that of PFL, with an advantage of  $\frac{1}{M}$  on the second and third terms (marked in red). This benefits from its sequential and shuffling-based training manner. Besides, we can also note that the upper bounds of both PFL and SFL are worse than that of minibatch SGD.
Partial client participation. In the more challenging cross-device settings, only a small fraction of clients participate in each round. Following the works (Li et al., 2019; Yang et al., 2021), we provide the upper bounds of PFL and SFL with partial client participation as follows:
$$
\text {P F L :} \tilde {\mathcal {O}} \left(\frac {\sigma^ {2}}{\mu S K R} + \frac {\zeta_ {*} ^ {2}}{\mu R} \frac {M - S}{S (M - 1)} + \frac {L \sigma^ {2}}{\mu^ {2} K R ^ {2}} + \frac {L \zeta_ {*} ^ {2}}{\mu^ {2} R ^ {2}} + \mu D ^ {2} \exp \left(- \frac {\mu R}{L}\right)\right) \tag {10}
$$
$$
\text {S F L :} \tilde {\mathcal {O}} \left(\frac {\sigma^ {2}}{\mu S K R} + \frac {\zeta_ {*} ^ {2}}{\mu R} \frac {(M - S)}{S (M - 1)} + \frac {L \sigma^ {2}}{\mu^ {2} S K R ^ {2}} + \frac {L \zeta_ {*} ^ {2}}{\mu^ {2} S R ^ {2}} + \mu D ^ {2} \exp \left(- \frac {\mu R}{L}\right)\right) \tag {11}
$$
where  $S$  clients are selected randomly without replacement. There are additional terms (marked in blue) for both PFL and SFL, which is due to partial client participation and random sampling (Yang et al., 2021). It can be seen that the advantage of  $1 / S$  (marked in red) of SFL also exists, similar to the full client participation setup.
We run experiments on quadratic functions (Subsection 4.1) and real datasets (Subsection 4.2) to validate our theory. The main findings are i) in extremely heterogeneous settings, SFL performs better than PFL, ii) while in moderately heterogeneous settings, this may not be the case.
4.1 Experiments on quadratic functions
According to Table 1, SFL outperforms PFL on heterogeneous settings (in the worst case). Here we show that the counterintuitive result (in contrast to Gao et al. (2020, 2021)) can appear even for simple one-dimensional quadratic functions (Karimireddy et al., 2020).
Results of simulated experiments. As shown in Table 2, we use four groups of experiments with various degrees of heterogeneity. To further catch the heterogeneity, in addition to Assumption 3b, we also use bounded Hessian heterogeneity in Karimireddy et al. (2020):
$$
\max _ {m} \left\| \nabla^ {2} F _ {m} (\mathbf {x}) - \nabla^ {2} F (\mathbf {x}) \right\| \leq \delta .
$$
Choosing larger values of  $\zeta_*$  and  $\delta$  means higher heterogeneity. The experimental results of Table 2 are shown in Figure 1. When  $\zeta_* = 0$  and  $\delta = 0$ , SFL outperforms PFL (Group 1). When  $\zeta_* = 1$  and  $\delta = 0$ , the heterogeneity has no bad effect on the performance of PFL while hurts that of SFL significantly (Group 2). When the heterogeneity continues to increase to  $\delta > 0$ , SFL outperforms PFL with a faster rate and better result (Groups 3 and 4). This in fact tells us that the comparison between PFL and SFL can be associated with the data heterogeneity, and SFL outperforms PFL when meeting high data heterogeneity, which coincides with our theoretical conclusion.
Limitation and intuitive explanation. The bounds (see Table 1) above suggest that SFL outperforms PFL regardless of heterogeneity (the value of  $\zeta_*$ ), while the simulated results show that it only holds in extremely heterogeneous settings. This inconsistency is because existing theoretical works (Karimireddy et al., 2020; Koloskova et al., 2020) with Assumptions 3a, 3b may underestimate the capacity of PFL, where the function of global aggregation is omitted. In particular, Wang et al. (2022) have provided rigorous analyses showing that PFL performs much better than the bounds suggest in moderately heterogeneous settings. Hence, the comparison turns vacuous under this condition. Intuitively, PFL updates the global model less frequently with more accurate gradients (with the global aggregation). In contrast, SFL updates the global model more frequently with less accurate gradients. In homogeneous (gradients of both are accurate) and extremely heterogeneous settings (gradients of both are inaccurate), the benefits of frequent updates become dominant, and thus SFL outperforms PFL. In moderately heterogeneous settings, it's the opposite.
Table 2: Settings of simulated experiments. Each group has two local objectives (i.e.,  $M = 2$ ) and shares the same global objective. The heterogeneity increases from Group 1 to Group 4.
Group 1	Group 2	Group 3	Group 4
Settings	{F1(x) = 1/2x2F2(x) = 1/2x2}	{F1(x) = 1/2x2 + xF2(x) = 1/2x2 - x}	{F1(x) = 2/3x2 + xF2(x) = 1/3x2 - x}	{F1(x) = x2 + xF2(x) = -x}
ζ*,δ	ζ* = 0, δ = 0	ζ* = 1, δ = 0	ζ* = 1, δ = 1/3	ζ* = 1, δ = 1
Figure 1: Simulations on quadratic functions. It displays the experimental results from Group 1 to Group 4 in Table 2 from left to right. Shaded areas show the min-max values.
4.2 Experiments on real datasets
Extended Dirichlet strategy. This is to generate arbitrarily heterogeneous data across clients by extending the popular Dirichlet-based data partition strategy (Yurochkin et al., 2019; Hsu et al., 2019). The difference is to add a step of allocating classes (labels) to determine the number of classes per client (denoted by  $C$ ) before allocating samples via Dirichlet distribution (with concentrate parameter  $\alpha$ ). Thus, the extended strategy can be denoted by  $\operatorname{ExDir}(C, \alpha)$ . The implementation is as follows (with more details deferred to Appendix G.1):
- Allocating classes: We randomly allocate  $C$  different classes to each client. After assigning the classes, we can obtain the prior distribution  $q_{c}$  for each class  $c$ .
- Allocating samples: For each class  $c$ , we draw  $\pmb{p}_c \sim \mathrm{Dir}(\alpha \pmb{q}_c)$  and then allocate a  $p_{c,m}$  proportion of the samples of class  $c$  to client  $m$ . For example,  $q_{c} = [1, 1, 0, 0, \dots, ]$  means that the samples of class  $c$  are only allocated to the first 2 clients.
Experiments in cross-device settings. We next validate the theory in cross-device settings (Kairouz et al., 2021) with partial client participation on real datasets.
Setup. We consider the common CV tasks training VGGs (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016) on CIFAR-10 (Krizhevsky et al., 2009) and CINIC-10 (Darlow et al., 2018). Specifically, we use the models VGG-9 (Lin et al., 2020) and ResNet-18 (Acar et al., 2021). We partition the training sets of CIFAR-10 into 500 clients / CINIC-10 into 1000 clients by ExDir(1, 10.0) and ExDir(2, 10.0); and spare the test sets for computing test accuracy. As both partitions share the same parameter  $\alpha = 10.0$ , we use  $C = 1$  (where each client owns samples from one class) and  $C = 2$  (where each client owns samples from two classes) to represent them, respectively. Note that these two partitions are not rare in FL (Li et al., 2022). They are called extremely heterogeneous data and moderately heterogeneous data respectively in this paper. We fix the number of participating clients to 10 and the mini-batch size to 20. The local solver is SGD with learning rate being constant, momentum being 0 and weight decay being 1e-4. We apply gradient clipping to both algorithms (Appendix G.2) and tune the learning rate by grid search (Appendix G.3).
The best learning rate of SFL is smaller than that of PFL. We have the following observations from Figure 2: i) the best learning rates of SFL is smaller than that of PFL (by comparing PFL and SFL), and ii) the best learning rate of SFL becomes smaller as data heterogeneity increases (by comparing the top row and bottom row). These observations are critical for hyperparameter selection.
Effect of local steps. Figure 3 is aimed to study the effects of local steps. In both plots, it can be seen that the performance of SFL improves as  $K$  increases from 1 to 5. This validates the theoretical conclusion that local steps can help the convergence of SFL even on heterogeneous data. Then, the performance of SFL deteriorates as  $K$  increases from 5 to 10, whereas the upper bound of SFL always diminishes as long as  $K$  increases. This is because when  $K$  exceeds one threshold, the dominant term of the upper bound will be immune to its change as stated in Subsection 3.2. Then, considering "catastrophic forgetting" (Kirkpatrick et al., 2017; Sheller et al., 2019) problems in SFL, it can be expected to see such phenomenon.
SFL outperforms PFL on extremely heterogeneous data. The test accuracy results for various tasks are collected in Table 3. When  $C = 1$  (extremely heterogeneous), the performance of SFL is better than that of PFL across all
Figure 2: Test accuracies after training VGG-9 on CIFAR-10 for 1000 training rounds with different learning rates.
Figure 3: Effect of local steps.
tried settings. When  $C = 2$  (moderately heterogeneous), PFL can achieve the close or even slightly better performance than SFL in some cases (e.g., CIFAR-10/  $C = 2 / K = 50$ ). This is consistent with our observation and analysis in Subsection 4.1. Notably, on the more complicated dataset CINIC-10, SFL shows better for all settings, which may be due to higher heterogeneity.
Table 3: Test accuracy results in cross-device settings. We run PFL and SFL for 4000 training rounds on CIFAR-10 and CINIC-10. Results are computed across three random seeds and the last 100 training rounds. The better results (with larger than  $2\%$  test accuracy gain) between PFL and SFL in each setting are marked in bold.
Setup	C=1	C=2
Dataset	Model	Method	K=5	K=20	K=50	K=5	K=20	K=50
CIFAR-10	VGG-9	PFL	67.61±4.02	62.00±4.90	45.77±5.91	78.42±1.47	78.88±1.35	78.01±1.50
SFL	78.43±2.46	72.61±3.27	68.86±4.19	82.56±1.68	82.18±1.97	79.67±2.30
ResNet-18	PFL	52.12±6.09	44.58±4.79	34.29±4.99	80.27±1.52	82.27±1.55	79.88±2.18
SFL	83.44±1.83	76.97±4.82	68.91±4.29	87.16±1.34	84.90±3.53	79.38±4.49
CINIC-10	VGG-9	PFL	52.61±3.19	45.98±4.29	34.08±4.77	55.84±0.55	53.41±0.62	52.04±0.79
SFL	59.11±0.74	58.71±0.98	56.67±1.18	60.82±0.61	59.78±0.79	56.87±1.42
ResNet-18	PFL	41.12±4.28	33.19±4.73	24.71±4.89	57.70±1.04	55.59±1.32	46.99±1.73
SFL	60.36±1.37	51.84±2.15	44.95±2.97	64.17±1.06	58.05±2.54	56.28±2.32
In this paper, we have derived the convergence guarantees of SFL for strongly convex, general convex and non-convex objectives on heterogeneous data. Furthermore, we have compared SFL against PFL, showing that the guarantee of SFL is better than PFL on heterogeneous data. Experimental results validate that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.
Future directions include i) lower bounds for SFL (this work focuses on the upper bounds of SFL), ii) other potential factors that may affect the performance of PFL and SFL (this work focuses on data heterogeneity) and iii) new algorithms to facilitate our findings (no new algorithm in this work).
This work was supported in part by the National Key Research and Development Program of China under Grant 2021YFB2900302, in part by the National Science Foundation of China under Grant 62001048, and in part by the Fundamental Research Funds for the Central Universities under Grant 2242022k60006.
We thank the reviewers in NeurIPS 2023 for the insightful suggestions. We thank Sai Praneeth Karimireddy for helping us clear some doubts when proving the bounds.
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=B7v4QMR6Z9w.
Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. Sgd with shuffling: optimal rates without component convexity and large epoch requirements. Advances in Neural Information Processing Systems, 33:17526-17535, 2020.
Ghadir Ayache and Salim El Rouayheb. Private weighted random walk stochastic gradient descent. IEEE Journal on Selected Areas in Information Theory, 2(1):452-463, 2021.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Jaeyoung Cha, Jaewook Lee, and Chulhee Yun. Tighter lower bounds for shuffling sgd: Random permutations and beyond. arXiv preprint arXiv:2303.07160, 2023.
Ken Chang, Niranjan Balachandar, Carson Lam, Darwin Yi, James Brown, Andrew Beers, Bruce Rosen, Daniel L Rubin, and Jayashree Kalpathy-Cramer. Distributed deep learning networks
Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning framework. arXiv preprint arXiv:2107.11621, 2021.
Fan Zhou and Guojing Cong. On the convergence properties of a  $k$ -step averaging stochastic gradient descent algorithm for nonconvex optimization. arXiv preprint arXiv:1708.01012, 2017.
Xingyu Zhou. On the fenchel duality between strong convexity and lipschitz continuous gradient. arXiv preprint arXiv:1803.06573, 2018.
Zhi Zhou, Xu Chen, En Li, Liekang Zeng, Ke Luo, and Junshan Zhang. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proceedings of the IEEE, 107(8): 1738-1762, 2019.
A Applicable to Split Learning 15
B Related work 17
C Notations and technical lemmas 19
C.1 Notations 19
C.2 Basic identities and inequalities 19
C.3 Technical lemmas 20
D Proofs of Theorem 1 23
D.1 Strongly convex case 24
D.1.1 Finding the recursion 24
D.1.2 Bounding the client drift with Assumption 3b 26
D.1.3 Tuning the learning rate 27
D.1.4 Proof of strongly convex case of Theorem 1 and Corollary 1 29
D.2 General convex case 30
D.2.1 Tuning the learning rate 30
D.2.2 Proof of general convex case of Theorem 1 and Corollary 1 31
D.3 Non-convex case 31
D.3.1 Bounding the client drift with Assumption 3a 32
D.3.2 Proof of non-convex case of Theorem 1 and Corollary 1 34
E Proofs of Theorem 2 35
E.1 Strongly convex case 36
E.1.1 Find the per-round recursion 36
E.1.2 Bounding the client drift with Assumption 3b 38
E.1.3 Proof of strongly convex case of Theorem 2 39
E.2 General convex case 40
E.2.1 Proof of general convex case of Theorem 2 and Corollary 2 40
E.3 Non-convex case 40
E.3.1 Bounding the client drift with Assumption 3a 42
E.3.2 Proof of non-convex case of Theorem 2 43
F Simulations on quadratic functions 44
G More experimental details 45
G.1 Extended Dirichlet partition 45
G.2 Gradient clipping. 46
G.3 Grid search 49
G.4 More experimental results 50
A Applicable to Split Learning
Split Learning is proposed to address the computation bottleneck of resource-constrained devices, where the full model is split into two parts: the client-side model (front-part) and the server-side model (back-part). There are two typical algorithms in SL, i.e., Sequential Split Learning (SSL)<sup>2</sup> (Gupta and Raskar, 2018) and Split Federated Learning (SplitFed)<sup>3</sup> (Thapa et al., 2022). The overviews of these four paradigms are illustrated in Figure 4
Figure 4: Overviews of paradigms in FL and SL. The top row shows the FL algorithms, SFL and PFL. The bottom row shows the SL algorithms, SSL and SplitFed.
Training process of SSL. Each client keeps one client-side model and the server keeps one server-side model. Thus, each client and the server can collaborate to complete the local training of a full model (over the local data kept in clients). Each local step of local training includes the following operations: 1) the client executes the forward pass on the local data, and sends the activations of the cut-layer (called smashed data) and labels to the server; 2) the server executes the forward pass with received the smashed data and computes the loss with received labels; 3) the server executes the backward pass and send the gradients of the smashed data to the client; 4) the client executes the backward pass with the received gradients. After finishing the local training, the client sends the updated parameters of its client-side model to the next client. This process continues until all clients complete their local training. See the bottom-left subfigure and Algorithms 3, 4. For clarity, we adjust the superscripts and subscripts in this section, and provide the required notations in Table 4.
Training process of SplitFed. Each client keeps one client-side model and the server keeps (named as main server) keeps multiple server-side models, whose quantity is the same as the number of clients. Thus, each client and its corresponding server-side model in the main server can complete the local training of a full model in parallel. The local training operations of each client in SplitFed are identical to that in SSL. After the local training with the server, clients send the updated parameters to the fed server, one server introduced in SplitFed to achieve the aggregation of client-side models. The fed server aggregates the received parameters and sends the aggregated (averaged) parameters to the clients. The main server also aggregates the parameters of server-side models it kept and updates them accordingly. See the bottom-right subfigure and Thapa et al. (2021)'s Algorithm 2.
Applicable to SL. According to the complete training process of SSL and SplitFed, we can conclude the relationships between SFL and SSL, and, PFL and SplitFed as follows:
- SSL and SplitFed can be viewed as the practical implementations of SFL and PFL respectively in the context of SL.
- SSL and SplitFed share the same update rules with SFL (Algorithm 1) and PFL (Algorithm 2) respectively, and hence, the same convergence results.
2In SSL, client-side model parameters can be synchronized in two modes, the peer-to-peer mode and centralized mode. In the peer-to-peer mode, parameters are sent to the next client directly, while in the centralized mode, parameters are relayed to the next client through the server. This paper considers the peer-to-peer mode.
There are two versions of SplitFed and the first version is considered in this paper by default.
Table 4: Additional notations for Section A.
Symbol	Description
τm,k	number, index of local update steps (when training) with client πm
xm(r,k)/xc,m/(r,k)/xs,m	full/client-side/server-side local model parameters (xm(r,k)=[xc,m];xs,m)) after k local updates with client πm in the r-th round
Xm(r,k)/Ym(r,k)/Ym(r,k)	features/labels/predictors after k local updates with client πm in the r-th round
x(r)/xc(r)/xs(r)	full/client-side/server-side global model parameters in the r-th round
Am(r,k)	smashed data (activation of the cut layer) after k local updates with client πm in the r-th round
lπm	loss function with client πm
∇lπm(xs,m)(r,k);Am(r,k)	gradients of the loss regarding xs,m on input Am(r,k)
∇lπm(A(m(r,k);xs,m)(r,k))	gradients of the loss regarding A(m(r,k) on parameters xs,m)
∇lπm(xc,m)(r,k);Xm(r,k)	gradients of the loss regarding xc,m on input Xm(r,k)
Algorithm 3: Sequential Split Learning (Server-side operations)
Main Server executes:  
1 Initialize server-side global parameters  $\mathbf{x}_s^{(0)}$   
2 for round  $r = 0, \ldots, R - 1$  do  
3 Sample a permutation  $\pi_1, \pi_2, \ldots, \pi_M$  of  $\{1, 2, \ldots, M\}$  as clients' update order  
4 for  $m = 1, 2, \ldots, M$  in sequence do  
5 Initialize server-side local parameters:  $\mathbf{x}_{s,m}^{(r,0)} \gets \left\{ \begin{array}{ll} \mathbf{x}_s^{(r)}, & m = 1 \\ \mathbf{x}_{s,m-1}^{(r,\tau_{m-1})}, & m > 1 \end{array} \right.$   
6 for local update step  $k = 0, \ldots, \tau_m - 1$  do  
7 Receive  $(A_m^{(r,k)}, Y_m^{(r,k)})$  from client  $m$  // Com.  
8 Execute forward passes with smashed data  $A_m^{(r,k)}$   
9 Calculate the loss with  $(\hat{Y}_m^{(r,k)}, Y_m^{(r,k)})$   
10 Execute backward passes and compute  $\nabla \ell_{\pi_m}(\mathbf{x}_{s,m}^{(r,k)}; A_m^{(r,k)})$   
11 Send  $\nabla \ell_{\pi_m}(A_m^{(r,k)}; \mathbf{x}_{s,m}^{(r,k)})$  to client  $m$  // Com.  
12 Update server-side parameters:  $\mathbf{x}_{s,m}^{(r,k+1)} \gets \mathbf{x}_{s,m}^{(r,k)} - \eta \nabla \ell_{\pi_m}(\mathbf{x}_{s,m}^{(r,k)}; A_m^{(r,k)})$
Algorithm 4: Sequential Split Learning (Client-side operations)
Client  $\pi_{m}$  executes: 1 Request the latest client-side parameters from the previous client // Com. 2 Initialize client-side parameters:  $\mathbf{x}_{c,m}^{(r,0)}\gets \left\{ \begin{array}{ll}\mathbf{x}_c^{(r)}, & m = 1\\ \mathbf{x}_{c,m - 1}^{(r,\tau_{m - 1})}, & m > 1 \end{array} \right.$  3 for local update step  $k = 0,\ldots ,\tau_{m} - 1$  do 4 Execute forward passes with data features  $X_{m}^{(r,k)}$  5 Send  $(A_{m}^{(r,k)},Y_{m}^{(r,k)})$  to the server // Com. 6 Receive  $\nabla \ell_{\pi_m}(A_m^{(r,k)};\mathbf{x}_{s,m}^{(r,k)})$  // Com. 7 Execute backward passes and compute  $\nabla \ell_{\pi_m}(\mathbf{x}_{c,m}^{(r,k)};\mathbf{X}_m^{(r,k)})$  8 Update client-side parameters:  $\mathbf{x}_{c,m}^{(r,k + 1)}\gets \mathbf{x}_{c,m}^{(r,k)} - \eta \nabla \ell_{\pi_m}(\mathbf{x}_{c,m}^{(r,k)};\mathbf{X}_m^{(r,k)})$
B Related work
Convergence of PFL. The convergence of PFL (also known as Local SGD, FedAvg) has developed rapidly recently, with weaker assumptions, tighter bounds and more complex scenarios. Zhou and Cong (2017); Stich (2019a); Khaled et al. (2020); Wang and Joshi (2021) analyzed the convergence of PFL on homogeneous data. Li et al. (2019) derived the convergence guarantees for PFL with the bounded gradients assumption on heterogeneous data. Yet this assumption has been shown too stronger (Khaled et al., 2020). To further catch the heterogeneity, Karimireddy et al. (2020); Koloskova et al. (2020) assumed the variance of the gradients of local objectives is bounded either uniformly (Assumption 3a) or on the optima (Assumption 3b). Moreover, Li et al. (2019); Karimireddy et al. (2020); Yang et al. (2021) also consider the convergence with partial client participation. The lower bounds of PFL are also studied in Woodworth et al. (2020a,b); Yun et al. (2022). There are other variants in PFL, which show a faster convergence rate than the vanilla one (Algorithm 2), e.g., SCAFFOLD (Karimireddy et al., 2020).
Convergence of SGD-RR. Random Reshuffling (SGD-RR) has attracted more attention recently, as it (where data samples are sampled without replacement) is more common in practice than its counterpart algorithms SGD (where data samples are sample with replacement). Early works (Gürüzbalaban et al., 2021; Haochen and Sra, 2019) prove the upper bounds for strongly convex and twice-smooth objectives. Subsequent works (Nagaraj et al., 2019; Ahn et al., 2020; Mishchenko et al., 2020) further prove upper bounds for strongly convex, convex and non-convex cases. The lower bounds of SGD-RR are also investigated in the quadratic case Safran and Shamir (2020, 2021) and the strongly convex case (Rajput et al., 2020; Cha et al., 2023). In particular, the lower bounds in Cha et al. (2023) are shown to match the upper bounds in Mishchenko et al. (2020) for the both strongly convex and general convex cases. These works have reached a consensus that SGD-RR is better than SGD a least when the number of epochs (passes over the data) is large enough. In this paper, we use the bounds of SGD-RR to exam the tightness of our bounds of SFL.
There are also works studying the randomized incremental gradient methods are also relevant (Ram et al., 2009; Johansson et al., 2010; Ayache and El Rouayheb, 2021; Mao et al., 2020; Cyffers and Bellet, 2022), which consider a single update at each client and focus on random walks.
Shuffling-based methods in FL. Recently, shuffling-based methods have appeared in FL (Mishchenko et al., 2022; Yun et al., 2022; Cho et al., 2023). Mishchenko et al. (2022) gave the convergence result of Federated Random Reshuffling (FedRR) as a application to Federated Learning of their theory for Proximal Random Reshuffling (ProxRR). Yun et al. (2022) analyzed the convergence of Minibatch RR and Local RR, the variants of Minibatch SGD and Local SGD (Local SGD is equivalent to PFL in this work), where clients perform SGD-RR locally (in parallel) instead of SGD. Both FedRR and Local RR are different from SFL from the algorithm perspective. See Yun et al. (2022)'s Appendix A for comparison.