SPFL: Sequential updates with Parallel aggregation for Enhanced Federated Learning under Category and Domain Shifts
Authors: Haoyuan Liang
Shilei Cao
Guowen Li
Zhiyu Ye
Haohuan Fu
Juepeng Zheng
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning， Domain Shift， Category Shift
Primary area: other
TL;DR: Sequential updates with Parallel aggregation for Enhanced Federated Learning under Category and Domain Shifts

Abstract: Federated learning (FL) has recently emerged as the primary approach to overcoming data silos, 
enabling collaborative model training without sharing sensitive or proprietary data.
Parallel federated learning (PFL) aggregates models trained independently on each client’s local data, which can lead to suboptimal convergence due to limited data exposure.
In contrast, Sequential Federated Learning (SFL) allows models to traverse client datasets sequentially, enhancing data utilization. 
However, SFL effectiveness is limited in real-world non-IID scenarios characterized by category shift (inconsistent class distributions) and domain shift (distribution discrepancies).
These shifts cause two critical issues: update order sensitivity, where model performance varies significantly with the sequence of client updates, and catastrophic forgetting, where the model forgets previously learned features when trained on new client data.
We propose SPFL, a novel updating method that can be integrated into existing FL methods, integrating sequential updates with parallel aggregation to enhance data utilization and ease update order sensitivity. At the same time, we give the convergence analysis of SPFL under strong convex, general convex, and non-convex conditions, proving that this update scheme is significantly better than PFL and SFL.
Additionally, we introduce the Global-Local Alignment Module to mitigate catastrophic forgetting by aligning the predictions of the global model with those of the local and previous models during training. 
Our extensive experiments demonstrate that integrating SPFL into existing PFL methods significantly improves performance under category and domain shifts.

SPFL: Sequential Updates with Parallel Aggregation for Enhanced Federated Learning Under Category and Domain Shifts
Haoyuan Liang', Shilei Cao"*, Guowen Li Zhiyu Ye Haohwan Fu Juepeng Zheng 1.2
School of Artificial Intelligence, Sun Yat-sen University, China National Supercomputing Center in Shenzhen, China 3 Tsinghua Shenzhen International Graduate School, China
{ fliang caoshlei 1igw8) Cmail2. yysu.edu, en zhengjjp@mmil. sysu. edu cn yezy25dmails tsinghua. edu..
Abstract
Federated Learning (FL) has recently emerged as the primary approach to over- coming data silos, enabling collaborative model training without sharing sensitive proprietary data. Parallel Federated Learning (PFL) aggregates models trained independently on each client's local data, which could prevent the model from con- verging to the optimal solution due to limited data exposure. In contrast, Sequential Federated Learning (SFL.) allows models to traverse client datasets sequentially, enhancing data utilization. However, SFL effectiveness is limited in real-world Non-IID scenarios characterized by category shift (inconsistent class distributions) and domain shift (distribution discrepancies). These shifts cause two critical issues: update order sensitivity, where model performance varies significantly with the sequence of client updates; and catastrophic forgettt where the model forgets previously learned features when trained on new client data. Therefore, based on SFL, propose novel updating framework, SPFL (Sequential updates with Parallel aggregation Federated Learning). that can he integrated into existing PFI. methods. It integrates sequential updates with parallel aggregation to enhance data utilization and ease update order sensitivity. Meanwhile, we give the con- vergence analysis of SPFL under strong convex, general convex, and non-convex conditions, proving that this update scheme is significantly better than PFL and SFL. Additionally,
introduce the GLAM (Global-Local Alignment Module) to mitigate catastrophic forgetting by aligning the predictions of the local model with those of previous models and the global model during training. Our extensive experiments demonstrate that integrating the SPFL framework into existing PFL methods significantly improves performance under category and domain shifts.
1 Introduction
Deep learning models have demonstrated immense potential across various vision tasks, typically depending on large-scale data training [ 3] As privacy concerns regarding the centralized collection of extensive data continue escalate, traditional centralized training methods increasingly
fail to address clients' privacy needs. In response, Federated Learning (FL) 5 6 has emerged as decentralized solution, designed specifically to prioritize data privacy by allowing models to be trained collaboratively without centralizing sensitive information. Aligning with the foundational
"quual contributions. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Categery Shift
Clict?
Facleng
SFL
1D
47
111
Cheri
Cherl
Oleer'
20
37
Cllent
Clicne
Clocse
(e)
(c)
(0)
SBL
Domain Shift
Acc Clkst Cior. Clisi
Fedavg
SFL
50
Clkas
Clier
nond!
(b)
Cliest
Clicet2
Note: represents the start of updates from client represents the direction of transmission
Figure 1: Top: Parallel Federated Learning (PFL): The server aggregates the local models trained inde- pendently on each client. Bottom: Sequential Feder- ated Learning (SFL.): The local model will be trained on the clients sequentially without centralized sever.
Figure 2: (a) and (b) present cases of category shift and domain shift. (c) and (d) illustrate the performance of three starting points for client training (i.e., Client I->liiet 2 Cliii 3, Client 2-Client C-ieit 1, and Client 3-Client I-llie 2) on the three clients' data under category and domain shift, respectively. The solid lines in (e) and (f) compare the performance of Client 1 data between FedAvg, which repeatedly learns solely on Client I's data, and SFI,, which iteratively learns sequentially across different client datasets under category and domain shift, respectively. The dashed lines represent the sequential training process of SFL.
method FedAvg 77). existing FL methods 8 9 commonly employ Parallel Federated Learning (PFL). which aggregates models trained independently on each client at a central server, as shown in Fig. (top). While PFL effectively enhances the global model's performance through iterative aggregation, does not fully exploit all available data since each client's model trained solely on its local data, potentially leading to poor model generalization. This raises the question: what strategies can be explored leverage client data better and improve the model generalization?
One promising strategy is to allow the model to traverse each client's dataset sequentially, as demonstrated in Fig. [ (bottom), leading Sequential Federated Learning (SFL) I0 12). Although current SFL methods perform effectively in Independently and Identically Distributed (IID) settings across clients enhance generalization and leverage more data, their performance remains constrained in real-world Non-IID scenarios. The challenges arise because the Non-IID nature across clients mainly manifests in two forms in image tasks: category shift and domain shift. The category shift refers to inconsistent class distributions across clients. For instance, dog samples constitute 70% _
of the Client I's data but only 10% of the Client 2's as shown in Fig. (a). The domain shift involves distribution discrepancies arising from different domains, such as images sampled from paintings in one client and real photos in another, as depicted in Fig. (b).
To analyze the impact of two types of shifts in SFL, we test SFL on two datasets characterized by the category shift [13 14 (i.e. CIFAR-10) and the domain shift [15 16] (i.e. PACS) following previous work, respectively. The results presented in Fig. 2 reveal two critical issues impacting SFL under Non-IID conditions: (1) Update Order Sensitivity: The model becomes highly sensitive to the order in which clients are updated. Different update sequences can lead to inconsistent results in the same client dataset, illustrated in Fig. (c) and (d). For example, Fig. (c) demonstrates that starting the training process from Client 2 (Client 2->Client 3->Client /) leads to better performance in the Client / dataset than starting from Client (Client 3->Client 1->Client 2) under category shift. This may be because the latter update order causes the model to become trapped in saddle point, resulting in consistently poor performance. In addition, we present two examples to theoretically demonstrate the existence of this phenomenon in Appendix Section H However, solving this by exhaustively exploring all possible update sequences to determine the optimal order is impractical due to the high computational and time costs, especially in real-world applications involving large

---

number of clients. (2) Catastrophic Forgetting: As the model updates sequentially across clients with different data distributions, it tends to forget previously learned features when trained on new client data, a phenomenon known as catastrophic forgetting in continual learning [17 15 18 This issue is evident in Fig. (e) and (f), where SFL performs worse in Client I's data after repeated iterations under category or domain shift, compared to SFL., which learns repeatedly on a single client's dataset.
To effectively leverage the available client data and overcome these challenges under category shift and domain shift, we propose novel FL framework named Sequential updates with Parallel aggregation Federated Learning (SPFL) and a Global-Local Alignment Module (GLAM).
Specifically, SPFL harnesses the strengths of both parallel and sequential learning. The server initially distributes the global model to all clients. Each client trains the model on its local data before passing the updated model to the next client for further training. After completing the sequential update round starting from different clients, the server aggregates these models in parallel to form a new global model for the next iteration. This hybrid approach maximizes data utilization and mitigates the risk of suboptimal update sequences, improving model robustness within a single computation round. Moreover, SPFL eliminates the need to share dataset sizes for weighting during aggregation, which increases privacy in applications where dataset sizes may leak sensitive information. Additionally, SPFL can be integrated into existing FL optimization approaches as a new updating strategy to address category shift, such as Scaffold [19] and FedDyn [20]. Furthermore, our proposed GLAM tackles catastrophic forgetting by continuously aligning the predictions of the global model with the local model during training. It also aligns the predictions of the local model with those from the previous client, effectively preserving learned knowledge among clients. The GLAM demonstrates significant effectiveness in improving the performance of SPFL. under domain shift. We also provide a convergence analysis of SPFL under strong convex, general convex, and non-convex conditions, demonstrating that this update scheme outperforms both PFL and SFL in Appendix Section G. In summary, the contributions of our paper are summarized as follows:
We identify and analyze the limitations of existing SFL methods in Non-IID scenarios, highlighting the issues of update order sensitivity and catastrophic forgetting under category and domain shifts,
We propose SPFL., a novel federated learning framework that combines sequential updates with parallel aggregation. SPFL. improves data utilization and model robustness by aggregating models with different starting clients to traverse each client's ' dataset sequentially. We also design GLAM to align predictions of the local model, both with global model and previous local model mitigate catastrophic forgetting, ensuring that the model retains knowledge across different client updates.
We provide convergence analysis under strong convex, general convex, and non-convex conditions. Extensive experiments demonstrate the effectiveness of SPFL under Non-IID conditions (domain shift and category shift), as well as their compatibility with traditional PFL methods.
2 Related Work
2.1 Parallel Federated Learning
As privacy concerns grow, companies become increasingly hesitant to upload their data the cloud for centralized model training. To effectively utilize these distributed data, FedAvg 7 pioneer federated learning community that serves the foundation of Parallel Federated Learning (PFL. 2T Building upon this work, most federated optimization algorithms have adopted the PFL paradigm, improving FedAvg through various improvements [22 23
24 25 For example. SCAFFOLD [19] mitigates client drift in federated learning through control variates. FedSAM (25) addresses the decline in global model generalization caused by inconsistent data distributions across clients by employing the Sharpness Aware Minimization optimizer. Although these FL methods significantly enhance the handling of traditional Non- MID-III issues, they are less effective on complex image tasks involving Non-IID data, such as domain shift (66 and category shift [33. Therefore, to address domain shift, FedCSA (27) addresses the inconsistency in the global model due to Non-IID data, using a model bias-based client data clustering method. Furthermore, to address the category shift, FedDisco 28 tackles the poor convergence of the global model under the category shift. However, existing PFL methods fall short in fully exploiting the available client data, since each local model is
19
3

---

prediction
Lop
Cleee
Client
Cliest
Cllent
Wyrevionus
Wintermediate
glocoal
L
Secoetat
Claar2 2
Clann
Ciient
W undermediade
Log
Cliest
Client
W
prreoioue
new
global
Figure 3: (Left) The process of our update method Sequential Updates with Parallel Aggregation (SPFL). The serial numbers indicate the steps to be executed sequentially. (Right) The module that solves the forgetting problem when the domain differences of each client are too large, that is, the module that Global-Local Alignment Module (GLAM). Lines of different colors represent the update order of different starts, and the black represents aggregation to the central server.
limited access to its corresponding client data. This limitation may result in suboptimal convergence of model training due insufficient data exposure, which motivates our proposed SPFL.
2.2 Sequential Federated Learning
Nowadays, studies on Sequential Federated Learning (SFL) remain relatively limited. Some research addresses the issue of information leakage in federated learning by performing membership inference attacks in the SFL setting [29]. Although other research uses sequential updates for privacy protection [29], they do not address the issue of poor model performance Non-IID data. FedGSP (30] seeks to address the decline in model performance caused by Non- MID data in federated learning through a dynamic collaborative training approach that transitions from sequential to grouped parallel updates. However, this approach deviates from the standard SFL setting. Similarly, FedSeq [10] mitigates the issues of slow convergence and performance degradation caused by computational heterogeneity in federated learning by sequentially training on heterogeneous client groups, referred to as super-clients. Although some experimental results demonstrate the advantages of FedSeq on Non-IID data, and some scholars theoretically prove the convergence of SFL under Non-IID conditions [31], these models still perform poorly on complex tasks involving domain shift and category shift.
3 Method
In this section, first introduce the problem formulation. Then, provide the convergence analysis of SPFL., comparing it with PFL. and SFL. Next, to tackle the challenges associated with domain shift for SPFL, propose Global-Local Alignment Module (GLAM). The workflow of SPFL is illustrated in Fig. 3 Limited by space, notations are summarized in Appendix D Meanwhile, the general assumptions for convergence are provided in Appendix
3.1 Preliminaries
In our paper, we adopt similar setting to the traditional FL and assume that each client m has its dataset D2 {(x,"' v/"h } where represents the amount of data from client kand performs model updates this way where x E X' is the input and y E y its corresponding label. For each client k; have separate models f(: wx:) The optimization objective of PFL can be written 1
K
4m
min
Jm
Lm (fax"" wm),y"")
4
Unlike PFL, the server is no longer needed in SFL, and our optimization objective is written 2
ane
ΣΣΣ fm (f w..), y."
min
10

---

By comparing and we observe that the gradient of Lm is solely dependent on the dataset of client m, whereas the gradient C. in is influenced by all the datasets. Meanwhile, we can see that SFL. does not need to introduce the dataset size in the optimization. However, as we analyzed in Fig- inconsistent starting points have a great impact on model performance. To solve this problem, we proposed Sequential updates with Parallel aggregation (SPFL., which can avoid the worst solution of the model, but of course, this does not represent the best model performance. To distinguish the different models at each starting point, we add conditional probability to the optimization target to fully consider this issue, Meanwhile, because SPFL goes through each client, the dataset that undergoes gradient descent is the same size, and we longer need to consider the impact of the dataset size on the model, which could proved in Fig. 4c
3.2 Sequential updates with Parallel aggregation
Our paper proposes new framework of updating Sequential updates with Parallel aggregation (SPFL), similar the client setting in PFL. Unlike PFL, our updating framework can continuously learn from different client datasets. From the perspective of global optimization, are different from PFL because the aggregation process different from the SFL aggregation process, as shown:
M
M
K
gen
(Sle f(x Tim
min M n=
Vmmg.
w 1.) M.
However, to compare the subsequent convergence analysis, according to Assumption [ this deviation can be incorporated into the a in Assumption 3 See 45 for more details.
For client m, the update method simplified as (4)
M
K
w (r+1)
w
M
8
(4)
n=1
k==
To help readers better understand SPFL, we provide the pscudo-code in Appendix Algorithm I
3.3 Convergence analysis of SPFL
Theorem 1. For SPFL, there exists constant effective learning rate i) = MKn and weights Or, such that the weighted average of the global parameters (I)
Wa En (t) (where
Wn Em= 0x) satisfies the following upper bounds:
Strongly convex: Under Assumptions and (r+1)
5 there exists a constant effective learning rate such that the following holds:
I and weights 0, 11-1
I ń GT.
E ((0)1R)) - -(ww)) <
9 2 uA exp
1 WnR
+
12
18L7 18LiP
+
(5)
M
MK
MK
and there exists a constant effective learning rate π) S 1 and weights 0, 1, such that the following holds: 6L
General convex: Under Assumptions
3.42
12ij²
18L
E (() - [(w²)
(6)
RR
M
MK
MK
Non-convex: Under Assumptions 2
and there exists a constant effective learning rate r (L(++] and weights O, = such that the following holds:
3
S
min
E ()(((()))
19
3B
3Lio²
27L2n8002
27L2
OSTER
πR
M
8MK
SM
A |0 (0) 205 for the convex cases and B ((w)()) L" for the non-convex case.
We include Corollary i the appendix. Additionally, we compare SPFL, SFL, and PFL. As shown in Tab. D SPFL achieves smaller convergence upper bound than both SFL and PFI. indicating faster convergence rate. Compared to PFL and SFL, proposed SPFL achieves a tighter upper
5

---

Table The upper bounds in the strongly convex case, with absolute constants and polylogarithmic factors omitted, are presented. The general convex and non-convex cases are included in Corollary
Method
Bound (A (0)
w='
PFL (FedAvg)
31
HMKR
+
# R +
L
exp n² + uA
LC?
HR
L
PFL (Scaffold) 19
0a
p²KR + aa + 1A? exp
Lo²
LCa
HR
PMKR
SFL 31
o²
+
La²
MA uA exp
LC3
(-4A
μMKR
PAMKR
SPFL (ours)
pM-KR
d²
+
MAMRR
Loa
LC?
+ HLA
exp
bound in the strongly convex case, notably improving the variance term from ((1/(MKR)) to ((1((M2 RR). Therefore, increasing the number of Clients can enhance the convergence speed of the model compared to other methods.
As shown in Fig. 3 we set an update order for the startup of each client and perform aggregation after each step, enabling more effective integration with traditional personalized federated learning (PFL). This sequential federated learning (SFL) strategy allows the model traverse larger volume of data, leading to 3% improvement in handling category shift. However, SFL alone remains insufficient for addressing domain shift due to the inherently uneven data distributions across clients. While SPFL inherits the PtL-style weighted aggregation, which helps maintain competitive performance under domain shift, it still faces limitations. To mitigate these issues, we introduce global-local alignment module (GLAM) specifically designed to enhance robustness against domain discrepancies.
3.4 Global-Local Alignment Module (GLAM)
In SFL with client data, if the model does not learn enough in the current domain and quickly transitions to the next domain, it may cause the model to forget the features learned in the previous round when moving to the next. For example, in the PACS task, when the model transitions from learning comics to the photo domain, we aim to offset the style features during the learning process. However, seems that some content information is also lost, making the SFL method significantly inferior to the PFL method in domain transfer. Parameter are summarized in Appendix D
In real-world federated learning, the data distribution of each client is unknown, the model needs to be robust enough to perform well across a variety of potential scenarios. Traditional solutions to the forgetting problem typically store part of the source domain data, which is not permissible in federated learning. In SPFL., each client maintains not only its own trained model but also a global model and model passed from the previous client- an advantage that neither SFL nor PFL possesses. To save resources as much as possible and fully leverage the advantages of SPFL, (wwgi)) and f(w -) are used solely for inference prediction, while they assist model (was in performing gradient descent. We fully leverage this advantage and develop a global-local alignment module (GLAM) based on these three models. To prevent the local model from excessively forgetting previous information, we designed two additional loss terms.
To ensure the convergence of the training process, first use the cross-entropy loss function the basis for gradient descent. The specific loss function used during the training is as follows:
B Lee = B Σ y." (o )
(8)
To mitigate catastrophic forgetting during training, we incorporate the predictions of the global model with those the model under training by using a mutual cross-entropy loss, as shown in (9). This loss Lag is named Approach to the global model, ensures consistency at the output level, and facilitates alignment between local and global models.
1 Σ sw.gi T. log( (f 7. Fi Lag
For client, we can also use the model passed from the previous client to align, thereby reducing forgetting of the previous client. Unlike before, use KL divergence to make the distributions of

---

the two outputs consistent. We refer to this loss as the Local Alignment with the Previous Output Caps which is defined as follows:
1
Cap
(L()))))+()))))sttt. P f(w;:z f (10)
2B
Table 2: Comparison of Classic and State-of-the-Are Algorithms in Federated Learning with and without SPFL on Cifar- 10, Cif,r-100, CINIC-10 and Fmmist in Category Shift
Cifar- 10
Cifar- 100
CINIC-10
Fmniit
Method
resnet18 (mum=10)
resnet18 (num=10)
simple-cnn (num=10)
resnet18 (num=10)
Avg
FodAvg X
75.00
57.55
39,74
81.13
63.35
SPFL
(+3.88)
(+6.78)
84.75(+3.62)
67.24 (+3.89)
FedD> 12
80.52
64.44
40.92
83.44
67.33
SPFL.
85.90(+5.38)
(+5.44)
(+3.23)
87.67(+4.23)
(+4.57)
FedDyn 20
77.91
64.11
40,91
81,35
66.03
SPFT.
80.90(+2.99)
64.24 (+0.13)
43.00 33.06 (+2.15)
83.23( (.88)
67.86 (+ L.3))
FodNova
76.02
57.64
81.38
63,46
SPFL
79.31(+3.29)
64.47 (+6.83)
41.32 (+1.50)
(4.85(+3.47)
67.49(+41.03)
76.04
57.56
+ SPFI.
77.25(+1.21 1,21
60.46 (+2.90)
40.27 (+0.62)
83.89(+2.66)
65.47 (+115)
MOON (5
78.70
58.44
40.11
81.29
63.68
-SPFI.
79.98 (+1.28)
63.54 (+5.20)
41.11 (+100)
84.83(+3.54)
67.37(+3.69)
SCAFFOLD
76.38
56,
36.00
80.71
62.31
SPFL
(+2.35)
(+8.90)
(+3.52)
85.80(55.09)
(+4.95)
FedDisco
28
76.32
$7.50
39.67
81.24
63.68
78.58(+2.26)
(+6.94)
(+1.12)
84.27(+3.02)
67.13(+3.45)
Table 3: Comparison of Classic Algorithms in FL with and without SPFL on Cifar- 10 in IID
Fodreg
FedDyn
FodNNovs
FedlProx
MOON [S
SCAFPOI
FodDison
PPL
SPHL
89,85 (+6.54
93.08 (+11.33)
81.77 (+1.11)
90000 (#6.61)
86.12/22.18)
90.16 (6.13)
91.97 (+9.53)
89.92 (+5.89)
This approach helps to maintain consistent output distribution across clients, thus facilitating smoother transition and reducing catastrophic forgetting. Combining (10) and (8) we can derive the final loss function C for model, shown below:
L = Lap + pLag + Lee
(11)
where r and / are hyperparameters used to control the influence of the previous client's model and the global model on the current gradient descent, which affects the convergence of the model.
In this paper, we enhance SFL to develop SPFL and introduce the GLAM module, making SPFL more effective update method than PFL. SPFL achieves significant improvements under category shift and delivers domain shift performance comparable to that of PFL. These insights are empirically validated through extensive experiments.
4 Experiments
4.1
Set up
In this section, we introduce the setup. Unless otherwise specified, all experiments will be conducted under the following conditions. We include the dataset descriptions and important experimental details in Appendix Sections [.] and [.2 Here, we present only the comparison methods:
Comparing Methods. We compared proposed method, SPFL, with the most classic algorithms federated learning. both with and without integration. This comparison aimed to evaluate the performance cnhancement that the integration of SPFL brings to the federated learning process. Including: (1) Fedavg 7 is the first algorithm for FL and PFL: (2) FedProx [34], SCAFFOLD [19],
FedDyn 20 MOON [35,, and FedDC[32 focus dynamically adjusting the client models; (3)
FedNova 3 adjusts the iteration counts from a global perspective to optimize the federated learning
process; (4) FedDisco
28 is the first paper that focuses addressing category shift in FL.
To comprehensively evaluate the impact the SPFL update framework under domain shift, we compare not only with traditional PFL methods but also with domain generalization approaches (c.g..
7

---

77.5
80
75.0
80
72.5
Acqurayy
P9
70.0
Accuracy
70
67.5
65
60
SPFL
Round
Round
Round
(a) IID
(b) Non-IID
(c) Without/With weighted average
Figure 4: (a) Comparison of the performance of SPFL and PFL on IID data in the cifar 10; (b) Comparison of the performance of SPFL and PFL on Non-IID in the cifar 10; (c) indicates that the Weighted Average shows no significant difference from the Arithmetic Average in SPFL.
Table 4: Comparison between PFL method and GLAM under the updated framework of SPFL
Method
PTL.
with SPFL.
Photo
Art
Cortoon
Sterch
92.16
Fodavg
90.96
76.17
72.31
65. 65,84
76.32
Fedprox
91.62
76.07
71.20
67.68
76.64
Scallold
19
91.32
76.61
67.45
67.12
75.62
AM
36
92.16
81.01
67.15
68.01
77.08
RSC
89.82
75.05
72.40
66.94
76.05
CCNet
39
91.92
75.10
65.10
70.78
75.73
Fedseq
70.54
59.01
52.28
FedDG-GA 38
90.78
69.63
69.58
63.76
73.44
SPEL-GLAM (ours)
92.28
77.44
74.42
67.32
77.87
AM
36
RSC 37 and federated domain generalization methods (e.g., FedDG-GA 33,, CCNet
Ditto 40 is personalized FL. method that incorporates model distillation. Since our approach also involves distillation components, we include it in our comparisons as well. As shown in Tab. [] our proposed GLAM effectively mitigates the impact of domain shift within the SPFL framework. Currently, the performance of most federated sequential learning methods heavily depends on data distribution and hierarchical aggregation strategies. For comparison, include FedSeq 41] which demonstrates limited effectiveness under domain shift scenarios.
39
4.2 Results
The performance IID. From Tab. 3 it can be observed that for client data under IID, SPFL can fully leverage its capability to encounter more data. SPFL can perfectly adapt to each method, achieving average improvement of over 6%. Additionally, SPFL demonstrates fast convergence rate; the model can reach from 40% accuracy to 70% in just one round, as shown in Fig. 4a However, real-world data is Non-IID. So focus
the performance of SPFL. in the domain and category shift. In Fig. 4b SPFL. also shows strong performance in the category shift.
Performance under Category Shift. We evaluate SPFI. on four datasets (CIFAR-10 10 42 CIFAR- 100 [42), CINIC-10 43), and FMNIST [44]), as shown in Tab. 2 SPFL can be effectively integrated with existing federated learning methods, yielding average performance gain of approximately 3%. This integration highlights the compatibility and potential of SPFL in enhancing federated learning algorithms across diverse data distributions under category shift.
Performance under Domain Shift. In Tab. the proposed SPFL framework alone does not perform well under domain shift. To address this limitation, we introduce the Global-Local Alignment Module (GLAMM., which significantly narrows the performance gap. With the integration of GLAAM, SPFL's performance improved from 2.05% below the expected baseline to just 0.5% below, demonstrating the module's effectiveness enhancing SPFL's robustness to domain shift. Additionally, most PFI. methods integrated into the SPFL framework show suboptimal performance, as illustrated in Tab. 4
4.3 Ablation Studies
Impact of Hyperparameters т and p in the model. As shown Fig. 5a and 5b
we observe
that setting and p too high leads the model overly align with the global models and previous

---

FTL
Vils
$2.24 5012
Number
Epoc
(a) Sensitivity to
(b) Sensitivity
(c) The efficiency of SPFL
Figure 5: (a) and (b) illustrate the sensitivity to GLAM hyperparameters: (c) demonstrates increasing the number of PFL's local Epoch under category shift does not lead to significant improvement.
Table 5: The combination experiments of the two different loss functions Cap and Lag conducted on the PACS dataset using a tre-trained ResNet- 18 (Best in bold)
Method
Lag
Cap
Photo
Art
Cartoon
Sketch
Avg
PFL (Fedavg)
X
90.48
76.95
74.87
71.21
78.37
SFL (Fedseq)
X
70.54
59.01
58.53
52.28
60.04
SPFL.
X
90.96
76.17
72.31
65.84
76.32
SPFL.
x
91.50
73.73
72.40
66.58
76.05
SPFL
V
92.22
76.12
71.46
64.34
76.04
SPFL
92.28
77.44
74.42
67.32
77.87
rounds' models, which in turn degrades local training performance.Throee our sensitivity analysis, we observe that the combination T 0.3 and p yields the best model performance.
Impact of the GLAM Module. As shown in Tab. 5 SPFL alone is insufficient to effectively address domain shift. However, with the introduction the proposed GLAM module, the SPFL update method becomes competitive under domain shift scenarios. Each component of GLAM (Cap and Lag) contributes the overall performance improvement. In addition, we added in the appendix Tab. 10 if PFL is used first to improve the overall ability of the model, and then SPFL is used, the generalization of the model can be further improved.
The Impact of Uploaded Dataset Size on the Model. In PFL, the contribution of each client is weighted based the size of its dataset. In contrast, SPFL iteratively updates the model over all client datasets, effectively treating them as having equal size. As a result, SPFL does not require uploading client data, offering stronger privacy protection compared to PFL. As shown in Fig. 4c, our experimental results further confirm that the presence or absence of client data upload has no significant impact on model performance in SPFL.
The impact of Epoch on the SPFL framework. In our setting, we trained for epoch with 10 clients, resulting in computational cost that is 10 times higher than PFL. However, shown in Fig. [cc] when the number of epochs is set to 10, the resource consumption
PFL and SPFL becomes comparable. In Fig. 5c we conduct 200 rounds of training to approximate the theoretical upper bound. It can be observed that SPFL improves the upper limit of model convergence when addressing category shift, which is consistent with the results reported in Tab. [ Notably, when the number of PFL epochs is increased to 21 21-douning the computational cost compared to SPFL.-the model performance of PFL remains 0.6 % lower than that of SPFL!
Communication cost analysis
When directly applying SPFL to approximate the performance of PFL, it often incurs significant communication and computational overhead. To demonstrate the advantage of SPFL in achieving a lower convergence bound, the pre-trained model reported Tab. 6 and Tab. CIFAR-10 10 with FedAvg for 200 rounds across 10 clients. Furthermore, to show that SPFL can attain a superior convergence upper bound, we use SPFL to match the effect of PFL within only 10 rounds, surpassing the performance of PFL trained for thousands of rounds, while simultancously
trained on

---

Table Efficiency Comparison on CIFAR-I0 (Dirichlet a=0.1, Number of clients=10)
Method
Accuracy($)
Rounds
Comm. Cost (GB)
Total Time(s)
Compute Cost(GB)
PFL
79.15
500
116.90
3419.91
175.35
SPFL(OUrs)
84.17
15
21.042
678.6
87.675
Table Efficiency Comparison on CIFAR-10 (Dirichlet a=0.1, Number of clients=100)
Method
Accuracy(%)
Rounds
Comm. Cost (TB)
Total Time(s)
Compute Cost(TB)
PFL
81.38
2000 10
18.30
30259.82
27 46
SPFL(Ours)
86.68
4.68
14713.13
22.85
reducing both communication and computational costs. This provides new direction for the practical application of SPFL.
6 Conclusion
In this paper, we first identify and analyze the limitations of existing Sequential Federated Learning (SFL) methods under non-independent and identically distributed (Non-IID) scenarios, with focus on addressing category and domain shifts caused by update order sensitivity and catastrophic forgetting. To tackle these challenges, we propose SPFL, a novel framework that integrates seamlessly with existing federated learning (FL) methods. SPFL combines sequential updates with parallel aggregation to enhance data utilization and reduce sensitivity to update order. this framework, the model is distributed to all clients, where each client performs local training and sequentially passes the model. The server then aggregates the updated models from each starting client to generate a new global model. We provide a convergence analysis of the SPFL. update scheme, showing that it achieves faster convergence than both PFL and SFL across strongly convex, general convex, and non-convex settings. Additionally, we introduce the Global-Local Alignment Module (GLAM) address catastrophic forgetting by aligning the predictions of the global model with those of the local and previous models during training. Extensive experiments demonstrate the effectiveness of SPFL and GLAM under Non-IIID conditions specificalllyl domain and category shifts anfts-aadd  confirm their compatibility with traditional Parallel Federated Learning (PFL) methods.
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant T2125006 and Grant 42401415; in part by Shenzhen Science and Technology Program under Grant KCXF/20040900009359000 and Grant KI/ID202309231 5106012; part by the Fundamental Research Funds for the Central Universities, Sun Yat-sen University, under Project 24xkjc002; and in part by Jiangsu Innovation Capacity Building Program under Project BM2022028.
References
[1] Alexey Dosovitskiy. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
[2] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[4] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. survey on federated learning, Knowledge- Boosed Systems, 216: 106775 2021.
10

---

[5] Yichen Li. Haozhao Wang, Wenchho Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang. Xin Yang. Rui Zhang, Shui Yu, Song Guo, and Ruixuan Li. Unleashing the power of continual learning on non-centralized devices: A survey, 2024.
[6] Haoyuan Liang, Xinyu Zhang, Shilei Cao, Guowen Li, and Juepeng Zheng. Tta-feddg: Lever- aging test-time adaptation to address federated domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18658-18666, 2025.
[7] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effie -ffficientf learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[8] Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20412-20421, 2023.
[9] Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang. Wei Wang, Yi Wang, and Ruixuan Li. Per- sonalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings of the ACM on Web Conference 2025, WWW '25, page 2709, New York, NY, USA, 2025. Association for Computing Machinery.
[10] Riccardo Zaccone, Andrea Rizzardi, Debora Caldarola, Marco Ciccone, and Barbara Caputo. Speeding up heterogeneous federated learning with sequentially trained superclients. In 2022 26th International Conference on Puttern Recognition (ICPR), pages 3376-3382. IERE, 2022.
[11] Andrea Silvi, Andrea Rizzardi, Debora Caldarola, Barbara Caputo, and Marco Ciccone. Ac- celerating federated learning via sequential training of grouped heterogeneous clients. IEEE Access, 2024.
[12] Xuyang Li, Weizhuo Zhang, Yue Yu, Wei-Shi Zheng, Tong Zhang, and Ruixuan Wang. Sift: A serial framework with textual guidance for federated learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 655-665. Springer, 2024.
[13] Edvin Listo Zec, Adam Breitholtz, and Fredrik Johansson. Overcoming label shift in targeted federated learning, arXiv preprint arXiv:2411.03790 2024.
[14] Zhuang Qi, Lei Meng, Zhaochuan Li, Han Hu, and Xiangxu Meng. Cross-silo feature space alignment for federated learning on clients with imbalanced data. In The 39th Annual AAAI Conference on Artificial Intelligence (AAAI-25), pages 19986-19994, 2025.
[15] Yichen Li, Wenchao Xu, Yining Qi, Haozhao Wang, Ruixuan Li, and Song Guo. Sr-fdil: Synergistic replay for federated Nomain-incrementt learning. IEEE Transactions on Parallel and Distributed Systems, 35(1 181999-1890, 2024.
[16] Zhuang Qi. Sijin Zhou, Lei Meng, Han Hu, Han Yu, and Xiangxu Meng. Federated de- confounding and debiasing learning for diitt--diitiitutin generalization. arXiv preprint arXiv:2505.04979, 2025.
[17] Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, and Ruixuan Li. Fedssi: Rehearsal-free continual federated learning with synergistic synaptic intelligence. In Forty- second International Conference on Machine Learning.
[18] Xuankun Rong, Jianshu Zhang, Kun He, and Mang Ye. Can: Leveraging clients as navigators for generative replay in federated continual learning. In Forty-second International Conference on Machine Learning, 2025.
[19] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sehastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pages 1132-5143. PMLR, 2020.
[20] Cheng Jin, Xuandong Chen, Yi Gu, and Qun Li. Feddyn: A dynamic and efficient federated distillation approach on recommender system. In 2022 IEEE 28th international conference on parallel and distributed systems (ICPADS), pages 786-793, IEEE, 2023.
11

---

[21] Xuezheng Liu, Zhicong Zhong, Yipeng Zhou, Di Wu, Xu Chen, Min Chen, and Quan Z Sheng. Accelerating federated learning via parallel servers: A theoretically guaranteed approach. IEEF/ACM Transactions on Networking, 30(5):2201 2022.
[22] Yichen Li, Qunwei Li, Haozhao Wang. Ruixuan Li, Wenliang Zhong, and Guannan Zhang. Towards efficient replay in federated incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12820-12829, June 2024.
[23] Haozhao Wang, Haoran Xu, Yichen Li. Yuan Xu, Ruixuan Li, and Tianwei Zhang. Fed- CDA: Federated learning with cross-rounds divergence-awart aggregation. In The Twelfth International Conference on Learning Representations, 2024.
[24] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:21
2021.
[25] Zhe Qu. Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International conference on machine learning, pages 18280. PMLR, 2022.
[26] Yichen Li, Wenchao Xu, Haozhao Wang, Vining Qi, Jingcai Guo, and Ruixuan Li. Personalized federated
learning based on adaptive knowledge matching. In European conference on computer vision, pages 127-144. Springer, 2024.
[27] Zhen Wang, Daniyal M Alghazzawi, Li Cheng, Gaoyang Liu, Chen Wang, Zeng Cheng, and Yang Yang. Fedesa: Boosting the convergence speed of federated unlearning under data heterogeneity. In 2023 IEEE Intl Conf on Parallel Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking
pages 388-393. IEEE, 2023.
[28] Rui Ye, Mingkai Xu, Jianyu Wang. Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with discrepancy-aware collaboration. In International Conference on Machine Learning, pages 39879-39902. PMLR, 2023.
[29] Anastasia Pustozerova and Rudolf Mayer. Information leaks in federated learning. In Pro- ceedings of the network and distributed system security symposium, volume 10, page 122, 2020.
[30] Shenglai Zeng, Zonghang Li, Hongfang Yu, Yihong He, Zenglin Xu, Dusit Niyato, and Han Yu. Heterogeneous federated learning via grouped sequential to to-parallel training. In International Conference on Database Systems for Advanced Applications, pages 455-471. Springer, 2022.
[31] Yipeng Li and Xinchen Lyu. Convergence analysis of sequential federated learning hetero- geneous data. Advances Neural Information Processing Systems, 36, 2024.
[32] Liang Gao. Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10112-10121, 2022.
[33] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623, 2020.
[34] Xiaotong Yuan and Ping Li. On convergence of fedprox: Local dissimilarity invariant bounds, non-smoothness and beyond. Advances in Neural Information Processing Systems, 35:10752. 10765, 2022.
[35] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713-10722, 2021.
12

---

[36] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 14383-14392, 2021.
[37] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross- domain generalization. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, Augusi 23-28, 2020, proceedings, part II 16, pages 124-140. Springer, 2020.
[38] Ruipeng Zhang, Qinwei Xu, Jiangehao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Feder- ated domain generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3954 0364,,933, 2023.
[39] Ahmed Radwan and Mohamed Shehata. Fedpartwhole: federated domain generalization via consistent phrt-whole hierarchies. Pattern Analysis and Applications, 28(2):61, 2025.
[40] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pages 6357-6368, PMLR, 2021.
[41] Zhikun Chen, Daofeng Li, Rui Ni, Jinkang Zhu, and Sihai Zhang, Fedseq; A hybrid federated learning framework based on sequential in-cluster training. IEEE Systems Journal, 17(3):0088- 4049, 2023.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[43] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- 10 is not imagenet cifar-10. arXiv preprint arXiv:/8/0.03500 2018.
[44] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1708.07747, 2017.
[45] Guowen Li, Xintong Liu, Shilei Cao, Haoyuan Liang, Mengxuan Chen, Lixian Zhang, Jinxiao Zhang, Jiuke Wang, Meng Jin, Juepeng Zheng, al. Tianquan-climate: A -ob-essonalloo-seasoaall global weather model via incorporate climatology state. arXiv preprint arXiv:2504.09940, 2025.
[46] Lei Meng, Zhuang Qi, Lei Wu, Xiaoyu Du Zhaochuan Li, Lizhen Cui, and Xiangxu Meng. Improving global generalization and local personalization for federated learning. IEEE Trans- actions on Neural Networks and Learning Systems, 36, 2024.
[47] Zhuang Qi, Lei Meng, Zitan Chen, Han Hu, Hui Lin, and Xiangxu Meng. Cross-silo proto- typical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3099-3107, 2023.
[48] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng- Heng. Feddg: Federated domain generalization medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1013-1023, 2021.
[49] Meirui Jiang, Zirui Wang, and Qi Dou. Harmofl: Harmonizing local and global drifts in federated learning on heterogeneous medical images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1087-1095, 2022.
[50] Junming Chen, Meirui Jiang, Qi Dou, and Qifeng Chen. Federated domain generalization for image recognition via cross-client style transfer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 361-370, 2023.
[51] Enyi Jiang, Yibo Jacky Zhang, and Sanmi Koyejo. Principled federated domain adaptation: Gradient projection and auto weito-weggting. arXiv preprint :a:::2222.000999 2023.
[52] Trong-Binh Nguyen, Minh-Duong Nguyen, Jinsun Park, Quoe- Quoc-Viet Pham, and Won Joo Hwang. Federated domain generalization with data-free on-server matching gradient. arxiv preprint arXiv:2501.14653, 2025.
13

---

[53] Xingyu Zhou. On the fenchel duality between strong convexity and lipschitz continuous gradient. arXiv preprint 1803.0880..06 2018.
[54] Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International conference artificial intelligence and statistics, pages 4519 4529. PMLR, 2020.
[55] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International conference on machine learning, pages 5381-5393. PMLR, 2020.
[56] Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint IrXIv:2301.11255, 2023.
[57] Francesco Orabona. modern introduction to online learning. CoRR, abs/1912.13213, 2019.
[58] Gong Chen and Marc Teboulle. Convergence analysis of proximal-like minimization algorithm using bregman functions. SIAM Journal Optimization, 3(3):538-55) 0088-533311 1993.
[59] S. Boyd, L. Vandenberghe, and L. Faybusovich. Convex optimization. IEEE Transactions Automatic Control, 51(1 1):
2006.
[60] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[61] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pages 3542-5550, 2017.
[62] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5018-5027, 2017.
[63] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[64] Farhad Pourpanah, Mahdiyar Molahasani, Milad Soltany, Michael Greenspan, and Ali Etemad. Federated unsupervised domain generalization using global and local alignment of gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19948-19955 2025.
[65] Yichen Li, Haozhao Wang, Yining Qi, Wei Liu, and Ruixuan Li. Re-fed+: better replay strategy for federated incremental learning, IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the scope of applicability and the computation and communication overhead in detail in section J
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
We have provided the full set of assumptions and a complete (and correct) proof. Please refer to Section in the main paper and Section G in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided detailed implementation details in Section 4 of the main paper and Section I.I and L.2 of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided detailed implementation details in Section of the main paper, as well as in Sections and and 1.2 of the Appendix. The source code is included in the
supplementary material. All datasets used are publicly available and properly cited in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sublic/guidess httpe//uuuiide/eeeuuuueeeeeeee
(oodeSubmissionoool1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpa////uideee//eeuuubbssiase http:/ddeeeeeeeeeeeeeaasiiiiooaa wattamisssiiooooiiioooo cilips.cc/publle httee//ullccggguddeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed implementation details in Section of the main paper, as well as in Sections 1.1 and 1.2 of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: result.
Our experimental results are computed three times and report the average
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: My research group supports me in computer resources. The specific hardware
used for the experiments is described in Appendix Section I.2
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper fully adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discussed both potential positive societal impacts and negative societal impacts in Section of the Appendix.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The data and code used in this paper have obtained legal permissions.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: The large language model (LLM) used in this paper is solely employed for text polishing and formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

SPFL: Sequential Updates with Parallel Aggregation for Enhanced Federated Learning Under Category and Domain Shifts (Supplementary material)
Table of Contents in Appendix
Pseudocode
23
B
More Related Work
23
Federated Domain Generalization
23
Corollary
24
Notations
24
Assumption
24
Technical Lemmas
25
Basic Identities and Inequalities
25
F2 Technical Lemmas
27
Proofs of Theorem
32
G.1
Strongly Convex Case
32
G.1.1
Finding the recursion
................................
32
G. G.1.2
Bounding the client drift with Assumption.
................................
35
G.1.3
Proof of strongly convex case of Theorem
37
G
G.2
General Convex Case
37
G.2.1
Proof of general convex case of Theorem
................................
37
G.3
Nonconvex Case
38
G.3.1
Bounding the client drift with Assumption
40
G.3.2 2
Proof of nonconvex case of Theorem
41
H Theoretical Analysis of Update Order Sensitivity
42
Loss Function Differences
42
H.2
The Data Distribution Differences
43
More Experimental Details
44
Dataset.
44
I.2
Implementation Details.
45
1.3
More comparative experiments
45
1.4
More Ablation Study
46
3
Limitation
47
Broader impacts
48
22

---

A
Pseudocode
Here we show our algorithm process.
Algorithm 1 Sequential updates With Parallel aggregation Federated learning (SPFL.)
Require: Initial global model wh. the number of clients is k the client dataset D
[D1. D2..., }x), (Hyperparameters:loca epoch 7, total aggregation round R)
Ensure: Final global model wa
1: Server:
2:
Deploy the global model wo to each client to obtain each client model:uf wo for each round i 1, 2, R do for each client for each client
3:
4:
2.,.... in parallel do
5:
c...., k&=l,
do
6:
if +1 S k
7:
ui e++ + Client (e, w/
8:
else:
9:
W11 Client (e, w)
10:
W + R Σx=1 w
11:
Deploy w++1 to all clients.
12: Client:
13:
for each local epoch & 1, 1,2,..., ......T do
14:
for Batch = 1, ..., B do
15:
(L( [B, Pa) = V Lee + + Lag + Lap (++(++1) f- (++(() 1/ (( -
16:
17:
k: % R
18:
return wi+1 (T) to Next Client
19:
else:
20:
return Wi+1 (i+(T) to Server
B
More Related Work
Domain generalization is also important criterion for evaluating the model. Examining the generalization ability of the global model is crucial, as it better reflects the overall performance of the model rather than its tendency to overfit. Therefore, we introduce related work on Federated Domain Generalization in this section.
B.1 Federated Domain Generalization
In recent years, Domain Generalization (DG) has developed rapidly, aiming to learn models that generalize well to unseen domains. However, in practice, multi-source data often exhibit significant domain shifts for example, between ceneral-purposs datasets and meteorological datasets (45) Federated Domain Generalization (FedDG) [66 [7] is an emerging field where each domain trains local model, which then aggregated, considering the generalization ability of the global model on target clients (domains). Domain shift is one of the most important issues in FedDG, which is consistent with this paper. Despite recent progress, research in this area is still limited. For example, ELCES 48 generalizes the model by sharing spectra across domains, yet this approach risks privacy leaks. 49 further proposes flatness-awee aware optimization method for better generalization on local updates. CCST [50] could lead to more uniform distributions of source clients and make each local model learn fit the image styles of all the clients avoid the different model biases. An adaptive weighting method, named GA (38] is proposed to achieve tighter generalization bound through explicit re-weighted aggregation. However, the abovementioned studies only focus on the Federated Aggregation. To this end, AutoFedGP [51] tackles the domain adaptation (DA) problem by projecting local gradients and assigning adaptive weights to align source and target domains. For domain generalization (DG), FedOMG (52) enhances robustness to unseen domains by aligning gradient directions across clients without access to target data. However, both approaches focus on model-level alignment, which may be insufficient when domain shifts induce severe forgetting during sequential
23

---

client updates. In contrast, our proposed SPFL explicitly targets this issue by introducing GLAM. This global-local aligmment mechanism concurrently aligns each client with both its predecessor and the global model, effectively mitigating the forgetting problem caused by domain shift.
C Corollary 1
Due to space limitations, we introduce Corollary 1 here.
Corollary 1. Applying the results Theorem 7 and by selecting an appropriate learning rate (see the proof of Theorem 7 in the Appendix Section G). we derive the following convergence bounds for SPFL:
Strongly convex: Under Assumptions
and
there exists constant effective learning rate such that the following holds:
(r+1)
I i S eI and weights 8, =
E [(w (() [(w")) Ō
o²
L
T.C.
FAA cxp
(-12))
|RR
(12)
WMRKR
MRR
12M
12L
and there exists a constant effective learning rate îj S 6L and weights 0, 1, such that the following holds:
General convex: Under Assumptions
E (n) L(w0)) =0 <MMKRR (MK) M1/3
( OA (LO 1/3
(LC
LAA
(13)
Non-convex: Under Assumptions 2
3 and there exists constant effective learning rate ñ S 6[[++]] and weights 0, 1, such that the following holds:
(
(Lo2B)
(T202 (22 1/3
min E [VV OSrSR
)P] -0 -
+
L
(14)
VM3KR
(MK) M2/3R2/3
R
where O omits absolute constants, Ô omits absolute constants and polylogarithmic factors, A (0) w"1 for the convex cases, and B = ((w(()) L for the non-convex case.
D Notations
Tab summarizes the notations appearing in this paper. We follow the same settings as in 31 to
compare the convergence bounds better.
E
Assumption
We assume that:
C is lower bounded by C for all cases and there exists minimizer w" such that C(w*)
C for strongly and generally convex cases;
each local objective function is L-smooth(( (Assuoott(stsmttionn 2
Furthermore, need to make assumptions about the diversities: the assumptions the
stochasticity bounding the diversity of [L.m (:: ( ) : [[Dm]]] with respect to i inside each client (Assumption
the assumptions the heterogeneity bounding the diversity of local objectives {Cm
[M]] with respect to m across clients (Assumptions 5
Assumption (Gradient Boundedness). There exists constant such that for all clients m,
any parameter z E Rd, and any sample &, the following holds:
V (w; E)]l S G.
(15)
24

---

Table Summary of key notations.
Symbol
Description
R
number, index of training rounds number, index of clients number, index local update steps Number of datasets for client m Number of batch size number of participating clients Client the starting Client
M. m
K,k
qm
B
N
S
S represents S clients selected from M clients. When all clients participate.M =
( 72. TM is permutation of {1,2,....
x(++))
((++)) R (I(r) = n2 m3 1
(r) (r)
(r) (r)
learning rate (or stepsize)
effective learning rate (7) == MKT in SFL and SPFL and řj Kn in PFL)
μ-strong convexity constant
L
L-smoothness constant (Asm.
upper bound on variance of stochastic gradients at each client (Asm.
3.C
constants in Asm. to bound heterogeneity everywhere constants in Asm. 3b to bound heterogeneity the optima
S-
C/ C.
global objective/loca objective of client m global model parameters in the r-th round
(r)
ugg
the global model [(; Wiy) where Wig represents the global model parameters.
w;
Represents the parameters of the model during training,and w; Wantermediate
Wp
Indicates the parameters of the model passed from the previous client training, and w,, = previoou
w (r)
When client is the starting client, local model parameters of the m-th client after k local steps in the r-th round
m.k.n
(r)
When client is the starting client, 8
(r) V.ny (w mnk E) denotes
the stochastic gradients of Lez regarding un (r) m.kkn
Assumption 2 (L-Smoothness). Each local objective function L. L-smooth,m E (1, i.e. there exists constant L > O such that [m (x) v Lm (y) S || yllrfor all æ, y R. Assumption 3. The variance of the stochastic gradient at each client is bounded:
Be [|V (w: v V (w)l
[
<
Vm E (1,2,..... M}
(16)
Assumption For strongly convex and generally convex functions, there exist constants 3 and ? such that
M
Σ
VL (w) VL(w)))1 s 8 [[V[(w) +
(17)
m=1
Assumption 5. For non-convex functions, there exists one constant C? such that
1
fw 2
(18)
M m12
F Technical Lemmas
F.1 Basic Identities and Inequalities
These identities and inequalities are mostly from analytical ideas provided by these works.
55
Thanks for the
25

---

E-norm identity. (1) For any random variable letting the variance can be decomposed as
w E [w]]
|| = |
|
E
[w]l
(19)
(2) In particular, its version for vectors with finite number of values gives
n
n
72 x x11
Σ
Σ
2
(20)
where vectors Wiy..... Wnn E Rd Lemma 1. Under standard discrete summation rules, the following closed-form expressions hold for any integer K 1:
are the values of w and their average の - wi,
K-1 k
K 1)K
K-1
R2 (K
12K
K-1 Σ
2 (K - 1)K (KK)
Σ
k3
(21)
6
Lemma 2 (Subadditivity of Concave Power Functions). Let 0 < 1 and a, 0. The power function f(x) = x² is concave, and the following inequality holds:
(a + b)" S a" + b
Jensen's inequality. For any convex function h and any vectors 2 In we have
x2
S
h(x;))
(22)
71
72
As special case with h(x)) = || we obtain
|
(23)
Smoothness and general convexity, strong convexity. There are some useful inequalities concerning L-smoothness (Assumption 2) convexity, and μ strong converity. Their proofs can be found in
53 and [56).
Bregman Divergence associated with function h and arbitrary is denoted as
Da (xx y) == h(x) - -h(y) - (Vh(y), x - y)
(24)
The divergence is strictly non-negative when the function h is convex. A more formal definition can be found in [57] One corollary (58] called the three point identity,
(= . + Dr y) Da(z, y) = (Vh(y) Vh(x), x)) where æ, y, z are three points the set.
(25)
When h is L-moooth, with the definition of Bregman divergence, consequence of L-smoothness is L || yll²
(x y) h(())) h(y) (Vh(y), y) 2
(26)
Further, if h is L-smooth and lower bounded by ht, then
|[h(=>)] < 2L (h(x)) h.)
(27)
If h is smooth and convex in L (the definition of convexity be found in 55 then
1
Da v)
2T. [[h((=)
Vh(y))l
(28)
The function : Rd + I. is μ strongly convex if and only if there exists convex function g IRd + I such that l(x)) = g(x) + ||
If h is strongly convex, it holds that
r vll < D2, (x. y)
(29)
26

---

F.2 Technical Lemmas
We obtain a recurrence relation suitable for SPFL based on previous work 31 as shown below: Lemma [ L.Lemma and lemma the relation
| Two nonnegative sequences {rihr20 { which satisfy
T++1 S (1 an)rr bus + cini + c2ni,
(30)
for all and for parameters 0, a, c1 02 2 and non-negative learning rates fmer20 with
Tt 3. Vt 2 for a parameter d 2 a,
Lemma 3 (linear convergence rate and Constant Step sizes), fri)r20, frehr2o as in 30 and a > 0.
Then there exists a constant step size "/t 7 S such that for weights O₂ = (1 a7)) -1+1) and
W. = Ei=o Wi holds:
b
s,B, I 3aro - (T+1) + + c2?
WT
(31)
S exprexp(-ard + +1]] + 617 + 027
b
0,0= + arr+1- = = (drr exp
aT -
C2
+ aT
(32)
WT
r2T2
Proof. We start by rearranging 30) and multiplying both sides with Og:
bs,0, Oz - (1)1-))) 0g40+1 +0n00 c2>"0 Wh-1T' 01r+11 + c170, c2220
7
7
By summing from = t l. = T, we obtain telescoping sum:
T
WT 010,
Σ
S
(wo - -a7ro wTTTH) + + +07 e2?
WTT
1=
and hence
T
T
V
sr0p <
s²0, +
WrTT+1
<
To
+ C17 + c2?
WT
WT
WWT
WWT
1=0
1=0
With the estimates
WT = -1-ar)-(7+1) ΣT Et=o a7)²
WT az
(here we leverage a7 S a/2 S 1)
and WT wT (1 (IT+1)
We can further simplify the left and right-hand sides:
b
Σ t=0
swwi + arr+1 (1 (T 7
+ar +ar+oon exp
xxpl-ay(T + 1)] + c1 c2r²
WT
7
Now the lemma follows by carefully tuning 7-Y (2, /ea)) and y =
of
In(max (2, 2 n fex)) yielding two choices of Consider the three cases:
af
If 1n /c1)) then we choose 7 = In (2,""nT
/e1]) and get that Eq
32
af
a/''
① (aroT Texpeepp----- In(max //1 1)1 + aT 2TT2 =
=
C2
2T
in case 2 > rrroT2/a it holds a
If 1 In(max In(max(2.02ro") ro / then we choose y
In (max In(max(2aa rg /02))
and get that Eq 32
aT
o7'
ô (aro 2 /oz1)) +
aT 2
2T
in case 2 > aoroTT/o2 it holds aroT <
20
2.72
27

---

In(max (a,nnx((2nnna 2o 1000/// /e1.0" ro"""" /o2))
If otherwise
then pick ? 2 and get that Eq 32
of
0T
dro exp
d
of
ci In(max (2, a² d'ro'' /rTI2//1,,00 (orooT///2)
<
dro exp
aT
62 In (max (2, roT 2/1, /c20)
T2T
= dro exp
aT
of
2272
Lemma 4 (linear convergence rate and Decreasing Step sizes). fribr20 fsetr20 in (30 and >
Then there exist step sizes 7/2 = r S à and weights 0, 2 0. W, Et wa such that:
sr02 + art+1 S 32dro exp Σ t=0
T
aT
36c1
+
3602
(33)
WT
2d
aT
odT
Proof. Let frshr20 {st be as in 30 for > O and for constant stepsizes %/t :=
Vt 0. Then, it holds for all T 2 O. We have
T-1
rT S (1 ay)rr-1- an + < (1 ay) t
E (1
(1- az)² + to2
3
ayy)
t=
(=0
C17
c22
(1
a
al
C1
C2
rn exp
ad
ad2
(34)
Let (ri)r20 {seJr2o as in (30) for > Oand for decreasing steps "
22
Vt 0, with
a(itoj
2d weights wt == (x + t) and Wr == -E- Gg. Then
parameter x
we a7)rr
wer++1 + 61 7/444 + +0272
bsywi S
7
%
a(k () + 1-tt-2) 2)r, a(N + r
(35)
S <a(k + - 1)?r, - 1) 3
ad
where the equality follows from the definition of ?/e and lr, and the inequality from (x++)((++-2) (R + t 1)2 < (Is + 1)² Again, we have a telescoping sum:
T
w;w, + als + +TPTT+
ax²rO WT
Ci (T +1)
C2 +1)
+
WT 1=0
WT
aWr
adWr
(36)
2an2ro
20
aT + 20g
T
adT
with
Wy ΣT B, ( (2++))(++)
> T(T+1)
> T
2
and WT (T++))(++1)
2(+T)1++T < (k+ 2 for K
2d >
2
2
By applying (34 and 36 we conclude the proof.
For the integer T 2 O, we choose the step sizes and weights as follows:
if T
1
0, (1 - (1-ay)) (1 11-9. 4) -((+1)
28

---

if T > d andt < to:
Yt d'
0:
=0, 0,
if T d and t z to:
2 7tt a(x+t-1o)'
0, = (N t to)²,
for K
2d and to
[ We will now show that these choices imply the claimed result.
This case similar to the proof of the Lemma. and suffices to We start with the case T S 말
a
consider Eq 35 for the choice "/t = d- We observe that Eq 35 simplifies to
aT
c1
C2 d
aT
C2 22T2
drg exp
+
S dra exp
oT +
d
If L. then from Eq 34
we obtain the following
af
C1
T1o ro exp
+
+
2d
ad
ad?
From Eq 36 we have for the second half of the iterates:
b
T
8ax2rio
4e1
402
0, +
arT+1
0, + arr+1 S
+
WT
WT
7
aT
adT
Now we observe that the restart condition ftp satisfies:
aK rio
ax2ro exp
2d
x
02 d27
<
4aro exp
aT
+
Ac1
402
T
T
dT
2d
aT
odT
Because T 2 These inequalities show the claim:
al
36c1 aT
36c2
Σ t=0
s.0, art+1 S 32dro exp
WT
2d
adT
Lemma Sub-linear Convergence rate from 31). (re)r2o, (re)r2o as in 30 and 0. Then
there exists the step size 7t. S à such that for weights 0g == and W. == Σt=o wi holds:
When our step size a constant, that is, ÿtt = S we have
b
T( 1 + ciTy c2
When we dynamically adjust the step size Yo, we have
V < 20
ro
+ 20
To
+
dra
(37)
T T+1
T+1
T+1
Proof. For constant learning rates 7t- = 7 S 1 we can derive the estimate
1 (n+ r.41) ci7y c2T 5
VT 7T
+ o27
7T +1)
which is the first result of this lemma. yielding two choices of 7 .y
5 Let 77+1]
= ciY and
027
7T+
(1+ and γ
.Teen choosing Y
cT(T+15)
min ((-,))))) 1/ , (T+
to
<
d there are three cases:
d
29

---

7 2. which implies that 7
7= 5
and 7 ==25
,then:
c1(+1)
62 (2(++1]
VT 5 dro
C1
22
dro
+
Ci
d2
T
+
T
T + 1
7 (1(7+1]
wwhich implies that
Eo
x
Tn
then:
(((+TT)
02(T+1)
T S 2c1
To (T +1)
+
ro
S 200
To T++
02
To T+1 1
CT +1)
If y (2(T++))
which implies that y
then:
c2(T+1))
(1(+1)
ro
To T+1
TO T ++1
ro
VT Sa
+ 202
202
C2 (T (T
T 1
Combining these three cases, we get the second result of this lemma:
T 2ef
ro
2c
TO T T+1 1
dro T +
T+1
Lemma (Simple Random Sampling from BII). Let w1 W2..... The population mean and population variance are given as
Wn be fixed units (e.g., vectors).
W
Ws 2² := Σ
w; m
(38)
1=1
Draw s E =1, n random units Wx, Way Wn, randomly from the population. There possible ways of simple random sampling, well known "sampling with replacement (SWR)" " and "sampling without replacement (SWOR)". For these two ways, the expectation and variance of the sample mean w Ep=1 Wap satisfy
5
SW R
E
W
W 2
(39)
8
SWOR:
E[w=]]
T
E
|| [+
w0|33
s
(40)
s(1 1)
We can easily get the relationship between variance and expectation, as well as Eq 2q If you want this proof, refer to [3I.
Proof.
19 and
Lemma 7. Under the same conditions of Lemma Juse the way "sampling without replacement" and i m -
K - 1,
let b,, (1)
Then for S S SM(M 2 2), it holds that
k 1
i=m
27
K
E
Σ
m
k(1)
(wzp - w
S 2 $2
1
(41)
S
2
1=1
m=1
K=
1=0
Proof. As shown below, the idea refers to
31
h (8)
w
w
=
(w n.
w + k (Wmn w
1=1
=
t=1
K2E
m
Σ
W n. w)
27
+ 12
(wan w1|²] + 2KAE
Σ (way w) (w w) -w))
-
30

---

For the first term on the right-hand side in Eq 40
using Lemma 6 we have
1) (m 1)) K26²
K
Σ
W
w
M
For the second term on the right-hand side in Eq
40
we have
w
k
2
w -===
2 12] =?
For the third term on the right-hand side in Eq
40
we have
m--1
m
2(m 1)
2K
Σ
w) (Way.
20)
2K
IE
[ (way
w. 20 w)
KkÇ²
M
where we use Lemma 6 in the last equality, since preceding equations, we get
(1,2,..., 1} + m. With these three
B (5)
(m 1)(M (m (01-1). 1)) M --1
2(m 1))
E
Σ
w
w
R² - k2
M - KhC
Then summing the preceding terms over m and k, we can get
K-1
bm. (i)
E
x w)
5
1=1
m=
k=0
f=0
R
8
K
m 1)
m
1)2 + Sç²
k2
1) S(M
S(M 1)
n=1 m=1
==
m=1
k=0 0
2KÇ²
5
S
K-1
(m
1)
Σ
k
S(M
1)
k=
Then, applying the lemma
D
we can simplify the preceding equation as follows:
E
Σ
W
w)
m=1
m=1
k=
0=
1 M- -1s ( S
= 2
Sk*
-1) -15K-1)
SK(K -
Lemma 8 (from 19).. Let f be a sequence of random variables. And the random sequence [w,li_, satisfy that w, E Rd is a function of Es. &1 for all i. Suppose that the conditional expectation is Eg, [w.lks-1,.....5
e; (i.e., the vectors {w, e,) form martingale difference
sequence with respect to { and the variance is bounded by Eg, ||w. e:ll? E.- E o². Then holds that
27
n
E
Σ w
e;)
Σ
E wi c S no²
(42)
Proof. For details, please refer to [99's Lemma and 31'ss Lemma 1.
Lemma (from (1)). The following holds for any L-smooth and p-stroongyy convex function h, and any x, y, z in the domain of h;
(Vh(m), - 4) M2)) h (y) +
11
211² T.
|[ =| xx
(43)
Proof. can be easily obtained using the three-point identity 24 and Jensen's 's inequality details, please refer to [3]] Here, only use relevant conclusions to assist our proof.
22 For
31

---

G Proofs of Theorem 1
In this section, we provide the proof of Theorem for the strongly convex, general convex, and non-convex cases in G.1. G.2 and G.3 respectively.
In the following proof, we consider the partial client participation setting. So we assume that x" {xī", ....... ....... xM) is permutation of [1,2,... M jin certain training round and only the first S se- lected clients {Tí m...... T will participate in this round. Without otherwise stated, we use B[-] to represent the expectation concerning both types of randomness (i.e., sampling data samples ( and sam-
pling clients π). R represents rotation, that is, π (n+1)
(a+1) (n+1)
(x+1) (m+1))
"22
n3
..... RM .11
R.
(1)
2
(a)
[m) 33 M ,whee n represents the client starting with client
(m) (n)
G.1 Strongly Convex Case
G.1.1 Finding the recursion
Lemma 10. Let Assumptions 2 and hold, and assume that all the local objectives are p-strongly 1
convex. If the learning rate satisfies 7 S
GLSK then it holds that
w (r+1) w² |1] $ (1- #SK#
M-S -
E
w --1111 + 4K
K2
S(M
- 1)
(44)
8 L
SKnDo(w, w²)
E
1 km
w/
3
k=
Proof. According to the pseudocode of our algorithm, the overall model updates SFL after one complete training round shown in (45). However, compare the subsequent convergence analysis, according to Assumption I this deviation can be incorporated into the Assumption For client m, the update method simplified (45
M
K Ien
m -
(r+1) = (t) - -IM
.),, Vi
V
1=
j=1
M
M
K
= (r)
M
ma
k=U
(45)
More generally, when rrpresents the number of clients selected to participate, we have
K
Aw (r+1) wr()) --1
Σ
where BT = Vnn (w (.n is the stochastic gradient of Cun regarding the vector w (r) Thus,
K
Aw
E
V (W),
A=
In the following, we focus on the recurrence of adjacent training rounds, so we omit the superscript for while, e.g., writing w w..kn Wm In particular, would like to use replace 11.0.1 Without otherwise stated, the expectation is conditioned on w".
32

---

We start by substituting the overall updates:
[|w + Aw - ==]l]]
||w - w" 112 + +E (w0 w., Aw)] + E ||wl]
K
K
||w - w" 112 20
(VLn (wm,k,n, w- - w=)l + 1
#=
k=0
(46)
We can apply Lemma with æ Wm.kn y = '" , = w and h = Lup, for the second term on the right-hand side in 46
K
21
E (V Lun (Wm.k.n) w*))
5
n=l
m=
k=
K-1
2m
1
E Cap (w) Cuy (20)) + H
w
w0112- L Wm
201²
(47)
8 K
<
S-25KiDe(uo, w²)
uSKnllw
w011 -00
2 5
Wmmkk,n - -0
11
k=0
For the third term on the right-hand side in (7) using Jensen's 's inequality, we have
E
8x
n=1
m= k=
8
K-1
< 4E
Bay
V Log (w (Wm,k.))
S
1m
k=
$
K
+ 4E
Cyp (wm.k.)) - V V (w))
S
#=1 m= k=0
S
K-1
27
5
K-1
27
+ 4F
(Vrr - V (w²)
+ 4E
Lrn
w
S
S
k=0
(48)
Seeing the data sample km,k.n the stochastic gradient gay .s the gradient V V (Lm. as E,,w0; and in Lemma respectivvvy and applying the result of Lemma6] the first term on the right-hand side (48) can be bounded by 4Ko2:
K
E
Br &k V Lrn (Wm,k.n))
S
K
(49)
E (gr A V.ra (Wm.k.))) 1|2
<
4Ko²
33

---

For the second term on the right-hand side in 48
we have
S
K-1
4E
V Crr (w))
28
4
K
S
K
V Loggy (Wm.k.)) V Lorg (20) 112
(50)
2
0=
8
8
K-1
AL'K
E
[| - w01?
==1
m=1
k=
For the third term on the right-hand side in
48
we have
K
4
V Cry (w)
Vorg (20)))
n=
m==
k=U
23
4
K
E V (20) - V Crm (w )|
(51)
k=0
SLK
E
Denn (w, (w,
26
LLS K2 Dc(w, w"),
We explain it follows, because De (w, w=) is linear concerning L(w)), so it satisfies the formula:
E [ (w, w )] Dcww, w²)
The fourth term on the right hand side in 48 can be bounded by Lemma 6 as follows:
S
K-1
(Lem 6 S
M - S
4E
ΣΣ
V Cag (wx)
KS
S(M - 1) c
(52)
With the preceding four inequalities, we can bound the third term on the right hand side in (48)
S S K-1
E
S
n=1 m==1
k=0
(53)
< 4Ko² + 4LPK
E
[mm - w012
m=1 k=
M S
+ 8LS2 R De(w w0)
+ 2
22
S(M 1)
Then substituting 47
and 53
into 46
we have
μSKI
M S
E || + Aw w
<
| 1w-w"' w0
4K1200 + K2
S(M 1) c
2
5
K-1
Lin (1 + 2LSKg)
+ 5
E Wm.kkn 2
n= m- k=
25Kn(1 4LSKg) Deww, w²)
Here substitute 7
6L.S to get Lemma
10
follows:
μSKn)
M S
E. || Aw w01 <
||
w1
+ 4K + K3
S(M 1) c
2
5
K
2 SKnDe(W,
8
w²)
E Wm. - w0 2
3
3
mL k=
34

---

G.1.2 Bounding the client drift with Assumption.
Similar to the "client drift" in PFL
19 and SFL
31.
we define the client drift in SPFL:
K-1
E, S 5EE k=0
E
w (r) m.k.n
(r))
(54)
Lemma 11. When Assumptions. 2 Hand Sand5hholl holdd and assuming that all local objective functions are ju-strongly convex, then if the learning rate satisfies 17 S BL.KK the client drift is bounded:
9 E, S 52 2 33LS | ,w-)
9
(55)
Proof. (r)
According our SPFL pseudo code, we can get the model updates of SPFL from w(rr to
w
is
m,k,n
bm & (8)
w (r) k,
(0)))
11
B
1=1 ==
i K- k 1,
with (1)
In the following, we focus single training round,
i= m
and hence we drop the superscript r for a while, e.g., writing Wm.k. to replace w m, IIn particular, we would like to use w to replace w 11,0,1- otherwise stated, the expectation conditioned on wt We use Jensen's inequality to bound the term Wmm.kn w|]²
E
(()) j-
B
E km, w0|²
bm (1)
< 2
(g VLag (wijin))
i=1
d (1)
+ 1
Σ
V Lor (wi.j..) - V VL (w))
dm, k (5)
b &e (8)
+ TT
(VZzy VL (w=))
+ 2 TT
V L (w²)
Applying Lemma to the first term and Jensen's 's inequality to the second, third terms on the right hand side in the preceding inequality, respectively, we can get
E
w Wm,knn
Am. (i)
4n²
E
B
V Von (wi...))l?
3
bm. (m..(())
2m
E V (Wi,j.)) - V. (w) [l²
m (1)
m
b x
+ 4m 2C Cm.k
[[) VL (w0 + 44
Σ
V (w")
j=0
(56)
35

---

where Cm,k- ii=1=
m Σ (+)) = (m 1)K + k. The first term the right-hand side in (56) is
i=
bounded by 4Cm,ke o². For the second term the right-hand side in (56), we have
Asm L'
E
| (Wijin) - V E (w)l
| w0ll
For the third term on the right-hand side in (66.. we have
E Lxz - VL (w") |F 2LE De.p (w, 20) 2LD,cww, w²)
Since wt the optimal solution, its gradient VL(ww)) = As a resuult we can get
M(]
[1mmk..- - u01²] 4Cm, 2 C
E ||wiiji - w0l2 Cm, Deww, w')
m Bm (1)
4n E
V Lmp (2=))
コ=
(57)
Then, returning E, = S 5
K- E 2k
|| Wm
w w|]
we have
5 K-1
K
m
(
E, S 4n 0²
2 2
Cm, + 11
Cm,
& [[] - w||²]
k=
j=0
K-1
8 K-1
b
+ 8Ly²
C Dc(w, 20) 4n7
E
V x (20)
m=1 0=
m=
km
j=0
Applying Lemma
with Wn. = VLn; (w*) (w=) and w0 VL(w") = 0 and Lemmal that
K
SK(SK - 11) 1 52k
Cm,k
2
8
K
1 ) 53
(SK - 11SK(SK
3
2
1=1 m=1
k=0
we can simplify the preceding inequality:
E, $ 22
+ f D' ((),
After rearranging the preceding inequality, we get
8 k35 De(w, w²) (1 11-2225k32))) < 25 K2 22
Finally, using the condition that 1) < 6L5K which implies - 2T. K2 8. have
E, 5 2 = 3
The claim follows after recovering the superscripts and taking unconditional expectations.
36

---

G.1.3 Proof of strongly convex case of Theorem 1
Proof of the strongly convex case of Theorem 1. Substituting Lemma n I 6LSK. we can simplify the recursion as follows:
11 into Lemma
T0 and using
w (rt w"
|9
μSKI
M S 1 - w01l =4KK200 + K3
S(M 1) c
-
2
2
8
SK (KDDDCCue Dr(w,, w²) LnnEr
3
3
1 (1
μSK1
1 w0 1
SK De (wo .00))
E
2
M - S
+ 4Krj0 0² + 45 12,"
2 + 6LS 12 6
S(M 1)
Let ij = MKn, we have
E
w (+++)) w0
1 1
E
w"
1
27
" E (e((ol"),
2
3
(58)
47j² 0²
417 (M
S)Ç?
6
6
52
S(M - 1)
SK
s
Applying Lemma 3 with 1 = r(T R), 7 it = E (1(0))
w01]"]
1 5,
3 8r
E [Dc(w(t) ww) 0. = (1- _ - (r+1),
C1 5
20²
+ 4MM
S(M-1)
02
6L SK
+
61.c2
SK
and
6L ( = MKn S a it follows that
1
E [( [(ww) - WR
1 Σ 0,E ((wo(1)) - [(wo')
9
(0) w/ exp
- (-furrt)
12ñ0²
127(M - 5)S2
18L7
18L7
S
24
+
+
S2K
S(M
SK
SK
(59)
where i (R) = WR Σ r= 0 w, (r) and we use Jensen's 's inequality (C is convex) in the first inequality. Applying Lemma to Eq 59 and using a suitable dynamic learning rate yields:
E Clwo (R)) L(w=))
(-Hui)
(M S)²
LO2
LC
(60)
- [uAeexs exp
12T. (-12T.)
KR
1sS WSR(M-1) 1)
J2SKR
S
where A : |0)()) (0)
w
||.
Eq. (59) and Eq. 60 are the upper bounds with partial client participation.
In particular, when = M. we can get the clarm of the strongly convex case of Theorem |
G.2 General Convex Case
G.2.1 Proof of general convex case of Theorem 1
Proof of the general convex case of Theorem 1, Letting μ = 0 in Eq. 588 we get the recursion of the general convex case,
E
w (r+1)
w01
w0
< (0)) - w -00
7
(
)
w0)
3
41j00
52K + 1v72(M - S)
6L
6L
+
S(M 1)
SK
S
Applying Lemma 5 with t = r(T = R), n. E (1
w' w=1[]]
a = 0,b = 3: 8:-
1
E. (Dc(ur(r), w0]] 0, = =(1 -1-20) -
5 + 3(
402
4M
6l
SK +
6LL?
SK
and =
37

---

69 ( MKn S
1
,it follows that
GL
R
((ww(R) [(ww))
1
Σ
0, -
-
WR
r=
(61)
3 w0 w0
12ñj0² S
12î M - S)C2
18L1Po²
18L1
+
S(M - +
RR
SK
SK
where (R) = Wa En (, and we use Jensen's 's inequality (C is convex) in the first inequality. By using suitable dynamic learning rate, we get
4(M -
S)C3 5(M 1))
$ EE
w (0)
w
11
((mo(R)) [(w") 52
S2K
R +
6Lo
6LL²2
(10000)
6LE
||
[ww(0)
w
|1]
+2 2
+
SK
SK
R RR-1
R + 1
Due to the concave nature of the power function(Leemas
2). we can get
2
(
[402
40
(4(M- -3533
14(M
E w w
[(w - [(w")
52
52k
+
8(M - 1)
R
+
(
66Lo*
G
E
(w(()
w
11
6LE
w
+
+
SK
SK
R+1
R + 1
After finishing, we can get:
AA
S
C.A
(L02 1/3
(LC. 1/3
LAA2
(1) (1011)))((w)))
+
+
VS
M
VSR (SK) S1/3
R
(62)
where A = ()) w=' Eq. 66 and Eq. 62 are the upper bounds with partial client participa- tion. particular, when S M, can claim the strongly convex case of Theorem and Corollary 1.
G.3
Nonconvex Case
Lemma 12. Let Assumptions 2 andd hold. If the learning rate satisfies 1) = 6L.K then holds
that
E [((w(r-1)) [(2o(>)) ]]
s
SK
E
((
LSK
2
I.2n
K-1
Σ
-
(r)
(63)
Proof. According to the Pseudocode of SPFL, the overall model updates of SPFL after one complete training round (with S clients selected for training)
S
K
Aw = w (r+1) (r)
2
- -7
8
g
m==
m, E) is the stochastic gradient of Cxn. regarding the vector un where B = V.r (r) Thus,
(r)
km
K
Awi --
E V C.p. (wm,k.))
12
mum
k=
38

---

In the following, we focus on the recurrence of adjacent training rounds, so we omit the superseript r for while, e.g., writing w k.kk.n as wm,k.. In particular, we would like use w replace 11001 Without otherwise stated, the expectation conditioned on w".
Starting from the smoothness of F (applying Eq. (53). Dz(x, y) S 1/2 | yll with x + Aw,
= w.and substituting the overall updates, we have
E [[ + Aw) [(w)]
L < B [VV(() Aw)]
||
8
Lup²
K
S - 5
[VL(()) Vary (wm.k,))) +
E
gm
2
S
k=0
k=0
(64)
For the first term on the right-hand side in Eq. 64), using the fact that 2 (a, b) = ||a|| + ||6||
|la bll? with ((w) and b Va (Wm,k,)) we have
K
E (VL(w) VLrg (Wm.k,a))
S 1
1)
E
+ VLrz (w (wm,k.))) V Lra (wm,k.) - VL(w)]
2
m=1 km
Asm 2
SK
|L(w)]|
17
$ K
VL (wm,k,n))
2
25
R=1
k=0
L' 8
K-1
+
Σ
E
[|W.kkin
w|1
2
S
m==mm=1
K=
(65)
For the third term on the right hand side in Eq.
64
using Jensen's 's inequality, we have
La7
9
K
E
2
S
m=
k=0
9
9
K-1
S
S
K-1
V Cgn (w (mm)ka))
3
g k
n=1
m=1
k=0
n=1
m=1
km
(66)
5
S
+ E
Von (Wm,k.))
0=0
S K-1
$ LKKI + LSK'' S
| (2mm
111m k=0
where we apply Lemma 6 by seeing the data sample Sm,k.n the stochastic gradient grn,k the gradi- ent V Crr (Sm,k.n) as ε,, wi e; respectively, Lemma for the first term and Jensen's inequality for the second term in the preceding inequality. Substituting Eq. (65) and Eq. (66) into Eq. (64) we have
SKn 2
L'n 1
K
E [C(w + Aw)- L(w)] S
[V(w))| + LK
Σ
E Wmmkk. w|
S
==1
m==
Am
S
7(1 (1-2LLKK)
V (wm,k,n)) 11
V
S
K=U
39

---

Since - 1 the last tern on the right-hand side in the preceding inequality is negative. Then
6LSK
SKn [[(W))
L
L2n
S
K-1
E [C(( + Aw) - [(w)] 2
Σ E
[w wll?
2
S
==1 m==
This conclusion can be obtained after restoring the superscript and taking the unconditional expecta- tion.
G.3.1 Bounding the client drift with Assumption
Since the proof of Lemma.
IT
uses Eq. (28). which is only applicable convex functions; we cannot Next, we use Assumption 4 to bound the client drift (defined in Eq.
use the result of Lemma. 11 (54)
Lemma 13. Assumptions 2 and 4 hold. If the learning rule satisfies 1 S 6LSK the client drift is bounded
E, < 122 + 9 s²
9
3 2 -
E [<( 19]
(67)
Proof.
Similar to the "client drift" in PFL 19 and SFL
31,.
define the client drift in SPFL:
== 1 Σ
E
1 (r)
(())
(68)
with (0) = K- k: 1, i=m m
i i m 1
In the following, we focus on single training round,
(r)
and hence we drop the superscript r for a while, e.g., writing Wm.k.n to replace m. In particular, we would like to use w to replace 11001- W1.0,--Wthhot otherwise stated, the expectation conditioned on ur. We use Jensen's inequality to bound the term E Wmmkk. - unl"
th
gaf J
E
[/ k.n
w
bm. (()
< =
[x77 V (wij.))
iml
0m, (5)
+ -
Σ
Cv (Wi.j.) - V VLn (w))
1=
2
m Um, (i)
bm, (1)
+ 41y
V (VZz -
+4n
VL(w)
i=0
1=
T
Applying Lemma. g Jensen's inequality and Jensen's inequality to the first, third, and fourth terms on the right side of the previous inequality, respectively, we can get
E
km
bm.. (i)
4n
E
[| - V Cnz (wa.sn))l
(69)
1=1
1=0
+ 4
E V (wism))
Vany (e)) + 2T + 4C2 4Ct,e0 [VL(w)]]
40

---

where Cm.kk == Σ Eis i= = (m 1)K + k. The first term the right-hand side in 669 is the second term the right-hand
bounded by &Cm, 1 with Assumption side in (56) can be defined as
With Assumption 2
be (t)
Dm (1)
41j 41f" Cm.k
E
V (Wijin)
V Razp (wm)ll² S 1/
E [[W....n - -w w]]]]
1=
Then, returning E, = E 2m=1 k-
E
Wm.
w|
we have
S
1
K-1
m (
E, < 41
Cm,k + 12
Cm..k
E w
w w1|
S
k=0
k=0
j=
K
k(f)
41 +
E
- (2)
8
==1 m=1 k=0
==
K
+
4nj
C k
V 2(w))
5
m=
k=0
(70)
Applying Lemma with way
VLe and T VL(w) to the third term and
S "M EM= Σx=o Cm.k I 2 3 terms on the right Hand side of the preceding inequality, we can simplify
K2 and 1 En E K-1 C I 1 533 to the other
3
E, 28 22 2 25
VL:(w) Vl(w)))
M
4 5 5 V[(w)]]
Asm
+2 +
|V
+ 53 [
After rearranging the preceding inequality, ge
4 + $ x² [V[(w)]
Finally, using the condition that // < BLSK which implies 1 - 22
3/ we have
E, S 9 s233120²++ 9 52
3 S [[]())]]
The claim follows after recovering the superscripts and taking unconditional expectations.
G.3.2 Proof of nonconvex case of Theorem
Proof. Substituting Lemma 12 into Lemma 13 and using S
1 6LSK min(1, y5
we can simplify
the recursion as follows:
E )
1 - SKnE [[ + " 2
9
9
Letting îj := =SK Subtracting L from both sides and rearranging the terms, we have
((w((++))) - c*) s ((w(r)) c*] - E \V(
Lij 9L'
9
52K
8SK
85
41

---

Then applying Lemma
with t r(T = R).7 7. = E [[(w(r) [*] = St
E [[V[((((>))
||
Lo² 3:K.2 2
9626² + 9L'52 and 3/2 67 min 11, v/8 (7
1,
oL(3+1) - min {1, v3
we have
E [[(2 - LT] 3
Lo²
((w(1)) - ((w0) 52
S²K
R+1
#
9I
E. ()) L³]
61E [L(w(()) L*
+
+
8SK
85
R R+1
R+1
Due the concave nature of the power function(Lemma 2). can get
1/2
Loz
E [(w(()) - Lr]
[(wo(R) [(ww) 52
52K
R+1
(
9L
(9L25²
]
3
E [[ww ---
$
6LE [((w(()) - L-]
+ 2
+
8SK
85
R+1
R+1
After finishing, get:
3 (I(20")-2")
3L
27L2"1002
27L"
min E USrER
+
52K +
(71)
iR
8SK
88
where use minogrer E [
S म Df E
[[ ||]]
Then, using ñ
1 61(3+]] S min
{1, y3 and by dynamically adjusting the learning rate, we have
-0 (L1O2)) 1/2
(L'0²B3) 1/3
(L2 1/3
LBB
min E oSrSR
+
(72)
VSKR (52K)1/3Re13
81/3 p2/3
R
where B = L(w0)) -
L" Eq 71 and Eq. are upper bounds for the case of partial client participation. In particular, when M, we obtain the conclusions of Theorem in the non-convex case.
H Theoretical Analysis of Update Order Sensitivity
What differentiates PFL and SFL is that the current updated gradient dependent on the gradient from the previous round; thus, we represent their dependency using conditional probability, as follows This why order sensitivity exists.
Eq. 73
Cup &e
[(1 ((x/Tt0 w
V
# Lun.i (f(x) Tm
Wx x)), 8i (iink
(73)
We provide two examples to illustrate that the update order in SFL introduces variance: one from the perspective of loss function differences, and the other from the data distribution perspective.
H.1 Loss Function Differences
We consider two clients A and B, each taking K steps of SGD in each training round, with learning rate of n).
Definition:
(r);; the global model at the beginning of round r;
42

---

The local gradient of client is (2(2) (x)))) and that of client B is gB()) VFb(z).
Order (A - B):
(r) A I1 = x(1) (1ga(z!r)) Bg x(1+1) -198(21)
Thus:
x/411) x11)) (gg( 1998 (zl1) non (20
Order (B A):
(r)) b 2 x(()) ngg((a!)) - (r+1) x (1/)((1)) alt
Thus:
(r+1) r()) ngg(a!))) - 79a (gg
x alt
Here the translated and refined version in academic style:
As example, let (x) {x FB(x) f 1)?, representing left- and right- night-biased objectives, respectively:
That is, VFA(=) = = 1, (F) = x + 1. We get:
(r+1)) =1) 19a (x 198 (x( ()9a(x(11))
(r)r) ()x((r) -1) - (x:x1))) (r) -n((x(1) 1) +
(1 2n + n² (r)
(r+1)
I (r) - n9n (z (r) (2 - ()gn(z!")))
alt
(1) (1)(1)) + +1) ()(x(t) 17(x0(1) +] - 1)
= =(1 - 27 + )²)p)(11) (r( +
By substituting specific values (e.g., (r) = 0, = 0.1), we can verify that the results differ under
different update orders. Cieerry, Lait (r++1) # ((++1)_ When the loss functions are inconsistent, the update order in SFL influences the gradient used for model updates.
H.2 The Data Distribution Differences
Suppose two clients, A and B, have different local data distributions DA + DB. Then their corresponding local objective functions differ:
(x(x) Lg FF) Bg [):
Therefore, the same x; their gradients are also different:
Fx) # FFB(.)
(74)
This defines the mathematical nature of data heterogeneity: gradient diversity, i.e., inconsistency in gradient distributions.
Example: Suppose use MSE loss (Mean Squared Error)
For linear regression task:
Client A's data: target value 0.
Client B's data: target value y ba
Let the model be 9 = Ox. The loss function is:
f(x: {) -0 -)²
1 (wx y)
(75)
Then:
43

---

The expected loss gradient for client A is:
R. F
u. axr) Ey (wur ax)
dw
The expected loss gradient for client is:
(1 (wzz bz)² ba)²)] Eg [[wx bx)
FFB Eg
Ow
If the distribution of x is the same (e.g., x follows a standard normal distribution), then:
VF((w) = -w--a) VFn(w)) = (w b)
(76)
Order (A - B):
((++1) = (r) - - F) FB (w(t)) - 1/V FA (1)(1)))
(r) (r) 0)) - (7(((1)) a)E[(² -) [[²
w 2mpu + 17 w (a b)))B[o - an/ (x²
Order (B + A):
(r+1) w Wat
(()) -/V('(((())) 1/ F/FFAA (w(+) - -1VVF'V
= (T( - n7(w(() D) 1 17 (w (((((r) - b)E[2²] - a 0[²
= 2nwol[[>> wE? [x²]] - (a + b)n|B[²?] - 2
By substituting specific values (e.g., a 1, b- 2), we can verify that the results differ under different update Clearly, wit # (++1), When the Data Distributions are inconsistent, the update order SFL influences the gradient used for model updates.
I More Experimental Details
L.1 Dataset.
We researched our proposed method using two types of datasets. One category consists of datasets with category shift, which necessitates partitioning. Our partitioning is based the Dirichlet distribution 35 Dir B3 Where 3 (default 0.5) is an argument correlated with the heterogeneity level. set the default number of clients 10. There are three data sets need to
In this problem, divide here, CIFAR- 10 (2), CIFAR-100F 100 and CINIC- 43 CIFAR-10 10 is a widely used dataset consisting of 60,000 32x32 color images in 10 classes, with 6,000 images per class, primarily used for image classification tasks.CIFAR CIFAR- 100 is similar to CIFAR-10, but with 100 classes containing 600 images each, providing more granular challenge for image classification tasks. CINIC-IO is an extended version CIFAR-10, containing 270,000 images split into 10 classes, designed to bridge the gap between CIFAR-10 10 and ImageNet for improved training scalability.
For domain shift, we conducted experiments on two benchmark datasets: PACS [6 and Office- Homel62 (i) The PACS [6]] dataset contains four distinct domains (Photos, Art Paintings, Cartoons, and Sketches), with total of 9,991 images. Each domain shares the same 7-class label space, despite the variations in image styles.
(ii) The Office-Home 62 dataset contains approximately 15,500 images across 65 categories from four domains (Art, Clipart, Product, and Real- Seal-World), offering diverse range of categories that better test the robustness of our method. We have included the Office-Hone experiment in the supplementary materials.
44

---

Table 9: Comparison of SPFL and PFL on Tmny-ImageNet and CIFAR-I00 across various FL methods
(Dirichlet = 0.1).
Method
Imageette
Tiny-ImageNet (SPFL)
CIFAR-100
CIFAR-100 (SPFL)
FedAvg
15.69
33.82 (+18.13)
57.55
64.33 (+6.78)
FedProx
15.47
16.65 (+1.18)
57.56
60.46 (+2.90)
MOON
15,42
16.47 (+1.05)
58.44
63.54 (+5.20)
FedDyn
15.22
17.66 (+2.44)
64.11
64.24 (+0.13)
Scaffold
14.58
18.16 (+3.58)
56.15
65.05 (+8.90)
FedDC
15.37
16.45 (+1.08)
64.44
69.88 (+5.44)
FedNova
15.53
16.45 (+0.98)
57.64
64.47 (+6.83)
FedDisco
33.55
34.50 (+0.95)
57.50
64.44 (+6.94)
Table 10: Accuracy (%) PACS under Domain Shift Setting.
Method
S
Avg
PFL (120R)
91.02
68.61
70.85
66.84
74.33
PFL (IOOR)+SPLL (20R)
93.11
73.24
69.36
64.92
75.16
PFL (1OOR)+SPFL. with GLAM (2OR)
93.71
74,45
70.21
64.45
75.71
Table 11:
Performance comppparicccomprissonnn of GLAM+SPFL with recent FL methods under domain shift.
Method
P
A
Avg
CAN [8]+
90.06
70.32
77.63
66.47
76.12
FedDA [SI 4SPFL
90.41
73.83
69.01
74.01
76.82
FedGALA +SPFL
87.58
68.58
68.17
66.56
72.72
GLAM+SPFL (Ours)
92.28
77.44
74.40
67.32
77.86
I.2 Implementation Details.
When we are in the domain shift setting, for local model training across the PACS and OfficeHome datasets, we utilize architectures ResNet18 and ResNet50, as detailed by (3) which are pre-trained on the ImageNet [60] We adopt tave--oeedooaile
evaluation method for all benchmarks, where one domain is reserved for testing and the remaining domains are used for training and validation. To ensure consistency and fairness in experiments, we standardize the batch size and learning rate at 128 and 0.2 in local training. Furthermore, to guarantee that the local models reach convergence within each training phase, we set the number of local epochs E to and define the communication rounds R as 100.
In the category shift setting, we utilized pre-trained model from PFL (FedAvg) that was trained for 100 rounds. Subsequently, we ran our new algorithm for an additional 50 rounds. For datasets Cifar10 and Cifarl 00, we employed the ResNet18 model framework, while for dataset CINIC- 10, we utilized a Simple-CNI CNN 633 model framework. We consistently employed Accuracy (acc) as the performance metric for our evaluations.
In the training phase, conducted our experiments used four NVIDIA Tesla A800 GPUs during testing to obtain results more quickly. However, the experiments only required 5-10GB of GPU memory due batch size of 128.
single NVIDIA Tesla A800 GPU. We
I.3 More comparative experiments
Performance comparison on complex datasets. Tab. compares SPFL and PFL on Tiny- ImageNet and CIFAR- 100 under different FL methods. SPFL consistently outperforms PFL across all settings, showing significant gains in both accuracy and convergence speed. These results demonstrate that SPFL achieves better generalization with most FL methods.
Performance of SPFL and GLAM under hybrid strategies. As shown in Tab. 10 when adopting hybrid strategy, SPFL maintains lower convergence margin the main shift but converges more
45

---

Label
Client and category distribution relationship
Client
Client_num num
0
0-1290
200
3
1067
110
142
1555
529
1364
1
53
295
1407
649
2067
86
191
3000
2
2500
127
2028
1599
146
1107
514
71
2000
4-1871
0
64
1129
133
59
290
728
75
5
235
837
2823
806
1573
0
0
0
1500
6
9
127
73
211
293
1654
361
1723
1250
13
1000
7
70
539
55
1619
9166
40
120
744
1689
859
8
2-009
692
130
183
250
544
3
233
1393
500
9
216
872
555
735
137
440
423
3464
Client_
id
Figure 6: Distribution relationship between client and category
Table 12: The combination experiments of the two different loss functions were conducted on the Office-Home dataset using pre-trained ResNet- 18 (Best in bold)
Method Lag
Cap
Product
Art
Clipart
Real-World
Avg
SPFL
57.22
43.10
46.07
59.22
51,40
SPFL
v
x
58.08
43.88
47.10
59.10
52.04
SPFL
X
v
56.75
44.27
47.62
59.49
52.03
SPFL
V
v
57.40
44.47
47.40
59.20
52.14
slowly. Therefore, first applying PFL to reach the boundary, followed by a low number of SPFI. rounds, can achieve lower convergence margin. Meanwhile, GLAM remains effective under this hybrid strategy.
Comparison with state-of-the-art methods. CAN 18 represents the latest federated continual learning approach [65 26), while FedDA (51) and FedGala (64) are recent advances in federated domain generalization. However, unlike GLAM, these methods do not align the model transmitted from the previous client, leading limited performance gains when adapting to new domains. In contrast, GLAM provides more comprehensive alignment mechanism, allowing flexible adjustment of the comparison strength through parameters and p. as shown Tab,
1.4 More Ablation Study
Performance in Office-Home. Through ablation studies on PACS and Office-Home, the effective- of the GLAM module is thoroughly demonstrated, with each component contributing to the overall model performance, as shown Fig. 12
Comparison of different update methods. We compare the two existing update schemes, PFL and SFL, with our proposed SPFL. As shown in Tab. [3] SPFL consistently outperforms both methods across various FL frameworks, demonstrating its superior effectiveness and adaptability. To quickly converge, the initial model was initialized using a FedAvg mre-traieed for 200 rounds on CIFAR-10, with 10 clients.
Comparison different Dirichlet a. We conduct an ablation study on parameter a. As shown in Tab. [4] the performance improvement of SPFI. becomes more pronounced as the degree of non-IlI increases. To quickly converge, the initial model was initialized using FedAvg pre-trained for 200 rounds CIFAR-10 10, with 10 clients.
46

---

Table 13: Comparison of different update strategies (PFL, SFL, and our SPFL) under various FL
methods on CIFAR-10 (Dirichlet a = 0.1).
UpdatelMethod FedAvg
FedProx
FedDC
Moon
PFL
55.60
56.65
55.16
56.33
SFL
68.59
53.75
54.41
55.24
SPFL (ours)
71.39
57.66
55.26
58.44
Table 14: Accuracy (%) SPFL and PFL (FedAvg) under different Dirichlet a settings.
UpdatelDir
0.1
0.2
0.5
100
PFL (FedAvg)
39.25
43,40
42.85
46.03
53.37
SPFL (Ours)
47.55
50.96
49.38
49.47
55.27
Table 15: Accuracy (%) of PFL, multiple SFL variants, and SPFL under different Dirichlet settings.
Method
PFL
SFL(1)
SFL(2)
SFL(3)
SFL(4)
SFL(5)
SPFL (ours)
Dir(0.1)
41.85
49.56
54.45
49.51
47.98
55.56
55.33
Din(0.5)
43.60
50.22
47.95
55.18
54.20
49.52
55.16
Comparison of different update orders. We conduct experiments with five clients for training, where SFL((i) denotes using client as the starting point. As shown in Tab. [5] different starting points result in significant performance variations, highlighting the update sensitivity problem in SFL. In contrast, SPFL effectively mitigates this issue, achieving more stable and consistent performance across clients.
Effect of client participation rate. Tab. I6 compares PFL and SPFL under different client participation rates (E) with 100 clients. As E increases, SPFL consistently outperforms PFL, showing greater stability and accuracy, which demonstrates its stronger adaptability to varying participation levels.
Effect under Category and Domain Shifts. As shown in Tab. 17 SPFL significantly improves performance under both category and domain shifts compared to PFL and SFI., demonstrating its stronger generalization capability. Furthermore, integrating the GLAM module further enhances performance, especially under domain shift, by effectively aligning inter-domain representations. This confirms that SPFL addresses update sensitivity, while GLAM strengthens cross-domain consistency and adaptability.
J Limitation
Although SPFL demonstrates strong performance in addressing category shift and achieves com- parable results to PFL under domain shift, this improvement imposes stricter requirements on the deployment environment, specifically, it assumes that clients remain continuously online. Addition- ally, while increasing the number of clients can accelerate convergence and lower the theoretical convergence upper bound, also introduces substantial communication overhead. As a result, SPFL is better suited for cross-silo scenarios rather than cross-device settings.
Moreover, cross-silo environments typically entail higher computational costs, whereas SPFL is able to match the performance of multi-epoch PFL using significantly fewer epochs. Therefore. although SPFL is theoretically applicable cross-device scenarios, it is more practically aligned with cross-silo applications. detailed comparison of computational and communication costs is presented in Tab.
Where M is the model size, M the number of clients, E is the epoch, and the number of communication rounds.
47

---

Table 16: Accuracy comparison between PFL and SPFL under different client participation rates (E)
with 100 total clients.
UpdatelE 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
PFL
41.78
46.99
52,43
53.31
54.04
54.52
54.47
54.45
54.35
54.59
SPFL
51.72
63.57
72.33
76.19
80.84
82.68
84.43
85.41
85.98
89.95
Table 17: Performance under Category and Domain Shift with PFL., SFL, SPFL, and SPFL+GLAM.
Shift Type
PFT
SFL
SPFL
SPFL+GLAM
Category Shift
55.60
68.59
71.39 76.32
71.71
Domain Shift (Avg)
78.37
60.04
77.87
Table 18: Cost calculation analysis
Method
Calculate costs
Communication costs
PFL
T M x E 3 x M Batch_Size
x 4
SFL
T M x E 3 x M Batch_Size
T x M M
SPFL
Tx
M x Bulch_Sized T X X M +122)
K
Broader impacts
The proposed SPFL framework enhances the robustness and generalization of federated learning systems under category and domain shifts, potentially benefiting applications involving privacy- sensitive and heterogeneous data, such as healthcare, finance, and education. By reducing the dependency on data centralization, it supports data sovereignty and regulatory compliance.
However, stronger model generalization across clients may inadvertently increase the risk of model inversion membership inference attacks. Additionally, more complex update schemes could amplify computational inequality between clients with varying resources. Future work should consider fairness and security safeguards mitigate these risks.
48


Reviews:
[{"review_id": "bN44m3xKDn", "reviewer": "Reviewer_BJo3", "summary": "This paper introduces SPFL (Sequential updates with Parallel aggregation Federated Learning), a novel framework that combines the strengths of Parallel Federated Learning (PFL) and Sequential Federated Learning (SFL). While PFL (e.g., FedAvg) enables parallel client training but suffers from limited data exposure, SFL allows a single model to traverse clients sequentially, enhancing data utilization. However, SFL is vulnerable to update order sensitivity and catastrophic forgetting, especially under Non-IID conditions. SPFL addresses these issues by organizing sequential client updates and periodically performing parallel aggregation, mitigating order bias and improving convergence. The authors also propose GLAM (Global-Local Alignment Module) to reduce forgetting by aligning local predictions with both the global and previous client models. The paper provides rigorous convergence analysis under various convexity settings and demonstrates, through extensive experiments on category and domain shifts, that SPFL significantly improves performance and stability when integrated into existing FL methods like FedAvg, FedProx, and FedDC. Overall, SPFL effectively bridges PFL and SFL, offering a robust solution for federated learning under heterogeneous data conditions.", "questions": "In tables like Table 2, only SPFL enhanced versions of FL methods are shown. Including original (non-SPFL) baselines such as FedAvg, FedProx, and FedDC would better quantify SPFL’s contribution.\n\nCurrent results are based on CIFAR-10 and similar datasets. To demonstrate scalability and robustness, evaluations on more complex datasets (e.g., CIFAR-100, subsets of ImageNet) are recommended.\n\nWhile the focus is on Non-IID scenarios, the paper should detail the partitioning strategies used. Including comparisons across varying levels of data heterogeneity (IID vs extreme Non-IID) would strengthen the claims.\n\n\nSPFL introduces additional communication through sequential and parallel updates in line 246. A comparison of communication/computation overhead versus standard methods like FedAvg would help clarify trade offs.\n\nSince update order sensitivity is a central issue, empirical evidence (e.g., accuracy variance across different client sequences) should be presented to support SPFL’s robustness against order changes.", "limitations": "As addressed above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper addresses a key limitation in FL under Non-IID data by combining sequential updates with parallel aggregation. This hybrid strategy (SPFL) effectively balances broader data exposure from Sequential Federated Learning (SFL) and order bias control via periodic parallel aggregation a notable conceptual contribution.\nThe authors provide convergence analysis for SPFL under strongly convex, convex, and non-convex settings. These theoretical results show SPFL’s stability and improved convergence bounds over FedAvg and pure SFL, adding credibility to the proposed method.\nThe GLAM module aligns client model predictions with both the previous local model and the global model, helping preserve earlier knowledge and reduce forgetting. This is a practical and validated addition.\n\nExperiments under category and domain shift conditions show consistent improvements when SPFL is applied to baseline methods like FedAvg and FedProx. The results highlight SPFL’s generalizability and effectiveness in challenging IID settings.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BJo3：\n\nWe sincerely thank you for the detailed and constructive feedback. We are encouraged that the reviewer appreciates our contributions in proposing SPFL as a hybrid solution bridging PFL and SFL, our theoretical convergence analysis, and the design of the GLAM module. Below, we address the reviewer’s insightful questions and concerns.\n\n**Note: To quickly converge,the most model below was initialized using a FedAvg pre-trained checkpoint trained for 200 rounds on CIFAR-10,the number of clients is 10.**\n### Response to Question\n>**Q1**:In tables like Table 2, only SPFL enhanced versions of FL methods are shown. Including original (non-SPFL) baselines such as FedAvg, FedProx, and FedDC would better quantify SPFL’s contribution.\n\nThank you for your valuable comments.  We compare three update paradigms: PFL, SFL, and our proposed SPFL. Since SFL is decentralized, to facilitate the evaluation，we use the final client’s model as the global model. As shown in the table,  SPFL reintroduces server aggregation and demonstrates strong adaptability across FL algorithms like FedProx, consistently improving performance.\n\n**Tab 1:Comparison of different update strategies (PFL, SFL, and our SPFL) under various FL methods on CIFAR-10 (Dirichlet α = 0.1).**\n\n|Update\\Method|FedAvg|FedProx|FedDC|Moon|\n|-|-|-|-|-|\n|PFL|55.60|56.65|55.16|56.33|\n|SFL|68.59|53.75|54.41|55.24|\n|SPFL(ours)|**71.39**|**57.66**|**55.26**|**58.44**|\n>**Q2**:Current results are based on CIFAR-10 and similar datasets. To demonstrate scalability and robustness, evaluations on more complex datasets (e.g., CIFAR-100, subsets of ImageNet) are recommended.\n\nWe appreciate your suggestion regarding the evaluation of more complex datasets. We have addressed this by adding results on more complex datasets, including CIFAR-100 and Tiny-ImageNet, as shown in the table below. These results further validate the scalability and robustness of SPFL across diverse FL methods.\n\n**Tab 2:Comparison of SPFL and PFL on Tiny-ImageNet and CIFAR-100 across various FL methods.(Dirichlet α = 0.1)**\n\n|Method|Tiny-ImageNet(PFL)|Tiny-ImageNet(SPFL)|CIFAR-100(PFL)| CIFAR-100(SPFL)|\n|-|-|-|-|-|\n|FedAvg|15.69|**33.82 (+18.13)**| 57.55| **64.33 (+6.78)**|\n|FedProx|15.47|16.65 (+1.18)|57.56| 60.46 (+2.90)|\n|MOON|15.42| 16.47 (+1.05)| 58.44|63.54 (+5.20)|\n|FedDyn|15.22|17.66 (+2.44)|64.11|64.24 (+0.13)|\n|Scaffold|14.58|18.16 (+3.58)|56.15|65.05 (+8.90)|\n|FedDC|15.37|16.45 (+1.08)|**64.44**|**69.88 (+5.44)**|\n|FedNova|15.53|16.45 (+0.98)|57.64|64.47 (+6.83)|\n|FedDisco|**33.55**|**34.50 (+0.95)**|57.50|64.44 (+6.94)|\n>**Q3**:While the focus is on Non-IID scenarios, the paper should detail the partitioning strategies used. Including comparisons across varying levels of data heterogeneity (IID vs extreme Non-IID) would strengthen the claims.\n\nWe thank the reviewer for pointing out the importance of evaluating model performance under varying degrees of data heterogeneity. Our main experiments were conducted under the widely adopted Non-IID setting with Dirichlet \\$\\alpha=0.1\\$, which we believe best reflects realistic federated learning scenarios. We also reported results under the IID setting in the main line 211. As suggested, we have included a comprehensive comparison across a range of Dirichlet partitioning parameters ($\\alpha$), which control the degree of Non-IID-ness—from highly skewed ($\\alpha=0.1$) to nearly IID ($\\alpha=100$). The tab 3 below reports the accuracy of SPFL and a representative PFL baseline (FedAvg):\n\n**Tab 3: Accuracy (%) of SPFL and PFL (FedAvg) under Different Dirichlet $\\alpha$ Settings**\n\n|Update\\Dir|0.1|0.2|0.5|1|100|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|39.25|43.40|42.85|46.03|53.37|\n|SPFL(Our)|47.55|50.96|49.38|49.47|55.27|\n\nWe will integrate this analysis into the revised manuscript and provide a clearer description of our partitioning strategy in the appendix for completeness and reproducibility.\n\n>**Q4**:SPFL introduces additional communication through sequential and parallel updates in line 246. A comparison of communication/computation overhead versus standard methods like FedAvg would help clarify trade-offs.\n\nThank you for pointing this out. We agree that SPFL introduces additional coordination for sequential and parallel updates. We acknowledge that SPFL increases the per-round communication volume by a factor of M/2 and the computational cost by approximately 5M/3 during training. **However, since SPFL converges faster, SPFL effectively reduces the number of training rounds, thereby reducing the overall communication and computation costs, as shown in Tab 4 and 5.**\n\n\n**Tab 4: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=10)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost (GB)|Total Time(seconds)|Compute Cost(GB)|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|79.15|500|116.90|3419.91|175.35|\n|SPFL(Ours)|84.17|15|21.042|678.6|87.675|\n\n**Tab 5: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=100)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost(TB)|Total Time(seconds)|Compute Cost(TB)|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|81.38|2000|18.30|30259.82|27.46|\n|SPFL(Ours)|86.68|10|4.68|14713.13|22.85|\n\n\n>**Q5**:Since update order sensitivity is a central issue, empirical evidence (e.g., accuracy variance across different client sequences) should be presented to support SPFL’s robustness against order changes.\n\nThank you for highlighting this important point. While we briefly illustrated SFL’s instability in Fig. 2 of the main, we now provide a more comprehensive analysis. As shown in Table 6, after sequential training across five clients, SFL exhibits substantial performance variance due to different update orders—for instance, accuracy ranges from 47.98% to 55.56% when α = 0.1. In contrast, SPFL achieves consistently great performance, closely matching the best SFL case while eliminating sensitivity to update order. This confirms that SPFL robustly mitigates order sensitivity, and we will incorporate this expanded experiment in the final version. Here, SFL(i) denotes a sequential update process where client i initiates the update order.\n\n**Tab 6: Accuracy (%) of PFL, multiple SFL variants, and SPFL under different Dirichlet α settings**\n\n|Method|PFL|SFL(1)|SFL(2)|SFL(3)|SFL(4)|SFL(5)|SPFL(ours)|\n|-|--|-|-|-|-|-|-|\n|Dir(0.1)|41.85|49.56|54.45|49.51|47.98|55.56|55.33|\n|Dir(0.5)|43.60|50.22|47.95|55.18|54.20|49.52|55.16|\n\n\n**We sincerely appreciate your insightful feedback and have addressed your concerns to the best of our ability. Should you have any further questions or recommendations, we would be glad to engage in further discussion.**"}, {"review_id": "ZEu14fjcSj", "reviewer": "Reviewer_ARDv", "summary": "This paper presents a new FL framework named SPFL, designed to enhance learning performance in the presence of category shift and domain shift, two prevalent types of data heterogeneity in FL. The proposed SPFL integrates the strengths of both SFL and PFL by training models sequentially across clients while aggregating in parallel. To address catastrophic forgetting—a common challenge in sequential updates—the authors propose GLAM, which aligns local predictions with global and prior local models. The authors also provide convergence analysis under different convexity assumptions and demonstrate improved performance across various datasets.", "questions": "1.\tCan the authors elaborate on the computational cost and communication latency of SPFL compared to traditional PFL in large-scale real-world settings (e.g., >100 clients)?\n2.\tHow sensitive is SPFL’s performance to the choice of update order, especially under extreme domain shift? Does parallel aggregation fully mitigate the update-order sensitivity or only reduce it?\n3.\tDoes the proposed GLAM introduce privacy risks by requiring clients to align outputs with previous client models? How is privacy preserved if models are indirectly influenced by earlier clients' data distributions?\n4.\tCan SPFL work in asynchronous or partial participation settings where not all clients are available in each round?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper identifies limitations of SFL in the presence of non-IID data and proposes a principled and well-motivated solution (SPFL) that combines the advantages of both SFL and PFL paradigms.\n2.\tThe convergence analysis is carefully presented for strongly convex, general convex, and non-convex settings. The results suggest a theoretically grounded improvement over SFL and PFL baselines.\n3.\tSPFL is designed as a plug-in update strategy, showing compatibility with many popular FL optimizers.\n\nWeaknesses:\n\n1.\tThe paper is at times difficult to follow due to dense writing and imprecise descriptions. For example, the mathematical expressions could benefit from clearer notation, and the explanations in sections like 3.2 and 3.4 could be streamlined.\n2.\tWhile SPFL is shown to improve accuracy, its communication and computation overheads—especially due to sequential updates across clients and the GLAM module—are not fully discussed. Practical feasibility for large-scale FL deployments remains unclear.\n3.\tWhile the paper compares with several popular FL methods, it misses comparisons with recent works focused explicitly on catastrophic forgetting or domain adaptation in FL.\n4.\tThe ablation on hyperparameters is appreciated, but more insight into how much each component (sequential update, parallel aggregation, GLAM) contributes independently to performance would be valuable.", "was_revised": true, "final_justification": "The authors have addressed the main concerns effectively, particularly the issues related to clarity, communication, and computation costs of SPFL. They have also added valuable comparisons with recent works on catastrophic forgetting and domain adaptation, and provided further experimental results and ablation studies to clarify the contributions of SPFL and GLAM. There are no major unresolved issues, and the revisions have significantly improved the clarity and depth of the paper. Based on the thoroughness of the authors’ responses and the quality of the additional experiments, I recommend accepting this paper.", "rebuttal": "Dear Reviewer ARDv：\n\nThank you for your thoughtful and constructive feedback. We appreciate your recognition of our contributions and  we provide detailed responses to the concerns and questions below.\n\n**Note: In order to quickly converge, the most model below was initialized using a FedAvg pre-trained checkpoint trained for 200 rounds on CIFAR-10, the number of clients is 10.**\n### Response to Weakness\n>**W1**:The paper is at times difficult to follow due to dense writing and imprecise descriptions. For example, the mathematical expressions could benefit from clearer notation, and the explanations in sections like 3.2 and 3.4 could be streamlined.\n\nThank you for the helpful suggestion. We will move key definitions from the appendix into the main in the final version and simplify the corresponding descriptions for better clarity. Thanks again for your insightful comments.\n\n>**W2&Q1**:While SPFL is shown to improve accuracy, its communication and computation overheads—especially due to sequential updates across clients and the GLAM module—are not fully discussed. Practical feasibility for large-scale FL deployments remains unclear.&Can the authors elaborate on the computational cost and communication latency of SPFL compared to traditional PFL in large-scale real-world settings (e.g., >100 clients)?\n\nThank you for your suggestions. We acknowledge that SPFL introduces higher communication and computation costs in both large-scale (>100 clients) and small-scale (< 10 clients) settings. Based on your feedback, we observed that SPFL’s early-stage performance gains were not sufficient to offset the increased communication cost. To address this, we adopted a hybrid strategy: pre-training the model with PFL, followed by SPFL fine-tuning. As shown in Tab 1 and 2, this significantly accelerates convergence, achieving strong performance in just a few rounds while reducing total communication and computation overhead.\n\n**Tab 1: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=10)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost (GB)|Total Time(seconds)|Compute Cost(GB)|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|79.15|500|116.90|3419.91|175.35|\n|SPFL(Ours)|84.17|15|21.042|678.6|87.675|\n\n**Tab 2: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=100)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost(TB)|Total Time(seconds)|Compute Cost(TB)|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|81.38|2000|18.30|30259.82|27.46|\n|SPFL(Ours)|86.68|10|4.68|14713.13|22.85|\n\n>**W3**:While the paper compares with several popular FL methods, it misses comparisons with recent works focused explicitly on catastrophic forgetting or domain adaptation in FL.\n\nWe thank you for pointing out the lack of comparison with recent methods addressing catastrophic forgetting and domain adaptation in FL. Following your suggestion, we added comparisons with recent methods targeting catastrophic forgetting and domain adaptation in FL. We selected FedDA (ICLR-2024) and FedGALA (AAAI-2025) as representative methods that explicitly address domain adaptation and CAN (ICML-2025) for forgetting. All methods were implemented under the same sequential federated learning setting on CIFAR10 with 10 clients and Dir(α=0.1) distribution to ensure a fair comparison.\nAs shown in the Tab 3, most existing methods focus on server-side alignment in the PFL setting, while the primary challenge of SPFL arises from **cross-client**, where catastrophic forgetting occurs across clients. Consequently, these methods are less effective in addressing the degradation of SPFL under domain shift. In contrast, our proposed GLAM module explicitly mitigates cross-client forgetting and demonstrates improved robustness in such scenarios.\n\n**Tab 3: Performance comparison of GLAM+SPFL with recent FL methods under domain shift.**\n\n|Method|P|A|C|S|Avg|\n|-|-|-|-|-|-|\n|CAN(ICML-2025)+SPFL|90.06|70.32|**77.63**|66.47|76.12|\n|FedDA(ICLR-2024)+SPFL|90.41|73.83|69.01|**74.01**|76.82|\n|FedGALA(AAAI-2025)+SPFL|87.58|68.58|68.17|66.56|72.72|\n|GLAM+SPFL(Ours)|**92.28**|**77.44**|74.40|67.32|**77.86**|\n\n\n>**W4**:The ablation on hyperparameters is appreciated, but more insight into how much each component (sequential update, parallel aggregation, GLAM) contributes independently to performance would be valuable\n\nThank you for your valuable feedback. As shown in Tab 4 and 5 (to be added in the appendix), we further demonstrate the roles of the SPFL and GLAM modules under both category shift and domain shift. The GLAM module does not provide significant gains under category shift; instead, it serves as a compensatory mechanism to mitigate forgetting caused by the performance degradation of SPFL under domain shift. This will enable SPFL to serve as a viable alternative to PFL in cross-island settings (i.e., limited client base).\n\n**Tab 4: Performance under Category and Domain Shift with PFL, SFL, SPFL, and SPFL+GLAM**\n\n|Shift Type|PFL|SFL|SPFL|SPFL+GLAM|\n|-|-|-|-|-|\n|Category Shift|55.60|68.59|71.39|71.71|\n|Domain Shift(Avg)|78.37|60.04|76.32|77.87|\n\n\n### Response to Question\n>**Q2:** How sensitive is SPFL’s performance to the choice of update order, especially under extreme domain shift? Does parallel aggregation fully mitigate the update-order sensitivity or only reduce it?\n\nThank you for the insightful question. SPFL reduces update-order sensitivity by **aggregating multiple SFL orders**, avoiding the instability of any single update sequence. As shown in Tab 5, while SFL accuracy varies significantly across orders (up to 7.5% under α = 0.1), SPFL achieves stable and strong performance.\n\nHowever, under extreme domain shift, the main challenge is **catastrophic forgetting**. To address this, SPFL incorporates **GLAM**, which aligns local and global predictions to maintain consistency across clients. This design further enhances robustness in the face of severe heterogeneity. Here, SFL(i) denotes a sequential update process where client i initiates the update order.\n\n**Tab 5: Accuracy (%) of PFL, multiple SPL Different Order, and SPFL under different Dirichlet \\alpha settings**\n\n|Method|PFL|SFL(1)|SFL(2)|SFL(3)|SFL(4)|SFL(5)|SPFL(ours)|\n|-|--|-|-|-|-|-|-|\n|Dir(0.1)|41.85|49.56|54.45|49.51|47.98|55.56|55.33|\n|Dir(0.5)|43.60|50.22|47.95|55.18|54.20|49.52|55.16|\n\n>**Q3:** Does the proposed GLAM introduce privacy risks by requiring clients to align outputs with previous client models? How is privacy preserved if models are indirectly influenced by earlier clients' data distributions?\n\nThank you for raising this important point. GLAM utilizes the previous client's model solely for local inference, which differs fundamentally from data or gradient sharing and is considered a lower-risk form of model sharing. The process is entirely local—no data or intermediate outputs are transmitted—serving only as a reference for alignment. Similar strategies have been adopted in prior works such as FedDF and FedGen, which have demonstrated low privacy risk. In future work, we plan to further enhance security through techniques such as model encryption or differential privacy.\n\n>**Q4:** Can SPFL work in asynchronous or partial participation settings where not all clients are available in each round?\n\nThank you for your insight comment. We conducted experiments with 100 clients under a partial participation setting, where only a fraction of clients ($E \\in [0.1, 1.0]$) were randomly selected to participate in each round. As shown in the table below, SPFL consistently outperforms PFL across all participation rates, demonstrating strong robustness even with limited client availability. These results further validate the effectiveness of SPFL in partial participation scenarios, aligning well with our theoretical insights.\n\n**Tab 6: Accuracy comparison between PFL and SPFL under different client participation rates (\\$E\\$) with 100 total clients.**\n\n|Update\\E|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1|\n|-|-|-|-|-|-|-|-|-|-|-|\n|PFL|41.78|46.99|52.43|53.31|54.04|54.52|54.47|54.45|54.35|54.59| \n|SPFL|51.72|63.57|72.33|76.19|80.84|82.68|84.43|85.41|85.98|89.95| \n\n**We sincerely appreciate your insightful feedback and have addressed your concerns to the best of our ability. Should you have any further questions or recommendations, we would be glad to engage in further discussion.**"}, {"review_id": "eIg7ek94uw", "reviewer": "Reviewer_cWLJ", "summary": "This paper proposes a novel update framework, SPFL, which effectively addresses category and domain shift in federated learning. By combining the data efficiency of sequential updates with the stability of parallel aggregation, SPFL significantly reduces update order sensitivity and improves model generalization in non-IID settings. The introduction of the GLAM module further alleviates catastrophic forgetting by aligning local and global predictions. Theoretical convergence analysis under strong, general, and non-convex cases, along with extensive experiments on benchmarks like CIFAR-10 and PACS, demonstrates that SPFL consistently outperforms existing methods and offers a powerful new direction for federated optimization. In addition, by solving category shift, SPFL points the way for cross-soli FL, showing strong innovation and practical significance. The authors further demonstrate through experiments that even when PFL is allocated twice the computational budget of SPFL, it still struggles to outperform SPFL. This result presents a remarkable departure from conventional Cognition in the FL, which may lead to the SPFL replacing the PFL in cross-soli scenarios in the future.", "questions": "1. The author shows that SPFL is compatible with multiple PFL algorithms, such as FedAvg, FedDyn, etc. Is it necessary to make special adjustments to each method when adapting? Are there any optimization methods that are not compatible?\n2. The number of clients set by the authors in the experiment seems to be fixed, and the impact of the change in the number of clients on the performance of the method has not been explored. However, in the convergence analysis, the number of clients has an important impact on the convergence speed. Therefore, it is recommended that the authors further evaluate the actual impact of the number of clients on the effect and convergence performance of the SPFL method.\n3. I encourage the authors to move the key experimental settings from the appendix to the main text in the final version. Presenting these details upfront would improve the clarity and accessibility of the experimental section, making it easier for readers to follow and reproduce the results.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper proposes a novel \"multi-start sequential update and parallel aggregation\" strategy, which significantly improves data utilization across clients. This hybrid approach bridges the gap between federated learning and centralized learning, yielding remarkable empirical results. SPFL provides a novel solution idea for category shift in PFL, enabling global models to achieve improved generalization in heterogeneous data. Meanwhile, it establishes a new paradigm for SFL by alleviating the challenge of update order sensitivity in SFL.\n2. The authors provide a thorough convergence analysis under strong convexity, general convexity, and non-convex conditions. This theoretical rigor not only validates the effectiveness of SPFL but also contributes valuable insights to the broader understanding of optimization dynamics in SPFL.\n3. The GLAM leverages KL divergence and mutual cross-entropy loss to align the current model with both the previous local model and the global model. This design effectively mitigates catastrophic forgetting during SFL, providing a practical solution to one of the core challenges in SPFL. This work represents the first attempt to introduce alignment mechanisms within the SPFL framework, offering valuable insights and a strong reference for future research in SFL.\n4. The experiment covers category shift and domain shift, multiple baseline methods (FedAvg, FedProx, SCAFFOLD, etc.), multiple data sets (CIFAR-10, PACS, etc.), and provides ablation experiments and hyperparameter sensitivity analysis\n5. SPFL does not rely on weighted aggregation of client data volume, improves privacy protection capabilities, and has good adaptability in privacy-sensitive application scenarios.\n\n**Weaknesses:**\n\n1. Although the authors point out that the problem of high computational cost per round tends to converge after multiple rounds of training, SPFL performs multiple sequential trainings in each round of updates, which may not be friendly to resource-constrained devices.\n2. Category shift has always been a common problem in federated learning, but the author's experiment involved Dirichlet distribution and did not seem to consider the impact of different $\\beta$ on the experiment.", "was_revised": true, "final_justification": "I thank the authors for their response, which has addressed most of my concerns. While I still have some remaining questions and concerns, I’ve seen relevant experiments and analyses in the responses to other reviewers. The transferability of unlearnable examples in FL is indeed a novel and promising direction that deserves more attention from the community. Overall, I think the current theoretical analysis and experimental results are sufficient to support the SPFL work. I will maintain my score and recommend acceptance.", "rebuttal": "Dear Reviewer cWLJ：\n\nWe sincerely thank you for the encouraging and thoughtful feedback. We are glad that you recognize the novelty of our SPFL framework and the module of GLAM, the theoretical and empirical contributions, and the practical significance of addressing category and domain shift in federated learning. We also appreciate the constructive suggestions regarding client scalability analysis, compatibility evaluation, and presentation clarity, which we will add in the final version.\n\n**Note: In order to quickly converge,the most model below was initialized using a FedAvg pre-trained checkpoint trained for 200 rounds on CIFAR-10,the number of clients is 10.**\n### Response to Weakness\n>**W1**:Although the authors point out that the problem of high computational cost per round tends to converge after multiple rounds of training, SPFL performs multiple sequential trainings in each round of updates, which may not be friendly to resource-constrained devices.\n\nThank you for pointing this out. We agree that SPFL introduces additional coordination for sequential and parallel updates. We acknowledge that SPFL increases the per-round communication volume by a factor of M/2 and the computational cost by approximately 5M/3 during training.\n**However, since SPFL converges faster, SPFL effectively reduces the number of training rounds, thereby reducing the overall communication and computation costs, as shown in Tab 1 and 2.**\n\n\n**Tab 1: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=10)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost (GB)|Total Time(seconds)|Compute Cost(GB)|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|79.15|500|116.90|3419.91|175.35|\n|SPFL(Ours)|84.17|15|21.042|678.6|87.675|\n\n**Tab 2: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=100)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost(TB)|Total Time(seconds)|Compute Cost(TB)|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|81.38|2000|18.30|30259.82|27.46|\n|SPFL(Ours)|86.68|10|4.68|14713.13|22.85|\n\n>**W2**:Category shift has always been a common problem in federated learning, but the author's experiment involved the Dirichlet distribution and did not seem to consider the impact of different on the experiment.\n\nThank you for highlighting the importance of evaluating category shift. While our main experiments were conducted under a strong category shift setting ($\\alpha = 0.1$), showing performance under varying heterogeneity levels is vital. We have therefore added results across a broader range of $\\alpha$ values, including the IID case ($\\alpha = 100$). As shown in tab 3 below, SPFL consistently outperforms PFL (FedAvg), with larger gains under stronger shifts, demonstrating its robustness to different degrees of category imbalance.\n\n**Tab 3: Accuracy (%) of SPFL and PFL (FedAvg) under Different Dirichlet $\\alpha$ Settings**\n\n|Update\\Dir|0.1|0.2|0.5|1|100|\n|-|-|-|-|-|-|\n|PFL(FedAvg)|39.25|43.40|42.85|46.03|53.37|\n|SPFL(Our)|47.55|50.96|49.38|49.47|55.27|\n\nWe will include these additional results and corresponding discussion in the updated manuscript.\n\n### Response to Question\n\n>**Q1:** The author shows that SPFL is compatible with multiple PFL algorithms, such as FedAvg, FedDyn, etc. Is it necessary to make special adjustments to each method when adapting? Are there any optimization methods that are not compatible?\n\nWe thank you for the valuable question. Indeed, Sequential Federated Learning (SFL) is inherently decentralized, which makes it difficult to couple with mainstream Parallel Federated Learning (PFL) methods that rely on server-mediated aggregation. This structural incompatibility limits the practical extension of SFL to more advanced optimization strategies.\n\nIn contrast, our proposed SPFL framework reintroduces the server as a coordinator, enabling a seamless integration with existing PFL algorithms such as FedAvg, FedDyn, and others. Since SPFL retains the client-level sequential update pathway while supporting centralized aggregation, it naturally bridges the gap between SFL and PFL.\n\nImportantly, SPFL does not require any modifications to the local update rules of existing methods. Instead, it wraps them into a multi-start sequential training process with parallel aggregation, preserving their theoretical and empirical advantages. This design not only ensures broad compatibility but also enhances convergence in challenging non-IID settings.We will further clarify this coupling capability in the revised manuscript.\n\n>**Q2:** The number of clients set by the authors in the experiment seems to be fixed, and the impact of the change in the number of clients on the performance of the method has not been explored. However, in the convergence analysis, the number of clients has an important impact on the convergence speed. Therefore, it is recommended that the authors further evaluate the actual impact of the number of clients on the effect and convergence performance of the SPFL method.\n\nThank you for your question. We conducted experiments with 100 clients under a partial participation setting, where only a fraction of clients ($E \\in [0.1, 1.0]$) were randomly selected to participate in each round. As shown in the table below, SPFL consistently outperforms PFL across all participation rates, demonstrating strong robustness even with limited client availability. These results further validate the effectiveness of SPFL in partial participation scenarios, aligning well with our theoretical insights.\n\n**Tab 4: Accuracy comparison between PFL and SPFL under different client participation rates (\\$E\\$) with 100 total clients.**\n\n|Update\\E|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1|\n|-|-|-|-|-|-|-|-|-|-|-|\n|PFL|41.78|46.99|52.43|53.31|54.04|54.52|54.47|54.45|54.35|54.59| \n|SPFL|51.72|63.57|72.33|76.19|80.84|82.68|84.43|85.41|85.98|89.95| \n\n>**Q3:** I encourage the authors to move the key experimental settings from the appendix to the main text in the final version. Presenting these details upfront would improve the clarity and accessibility of the experimental section, making it easier for readers to follow and reproduce the results.\n\nThank you for the helpful suggestion. Taking into account the opinions of most reviewers, we will move the key experimental settings (e.g., dataset partitions, client numbers, Dirichlet $\\alpha$, model architectures, and training schedules) from the appendix to the main text in the final version to improve clarity and reproducibility.\n\n**We sincerely appreciate your insightful feedback and have addressed your concerns to the best of our ability. Should you have any further questions or recommendations, we would be glad to engage in further discussion.**"}, {"review_id": "TjvdDURjbB", "reviewer": "Reviewer_wzqr", "summary": "This paper proposes SPFL, a novel federated learning framework that combines sequential updates with parallel aggregation to enhance data utilization and mitigate update order sensitivity in non-identically distributed (non-IID) settings. By introducing a Global-Local Alignment Module (GLAM) to mitigate catastrophic forgetting and providing convergence guarantees under various conditions, SPFL achieves better or comparable performance compared to traditional PFL and SFL methods in scenarios involving category and domain shifts.", "questions": "- Are there better ways to combine sequential and parallel FL to better balance computational cost and performance?\n- See weaknesses above.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The setting is interesting, and the problem is important to investigate for domain and category shifts in FL. Also, it is intuitively useful and helpful to investigate combining sequential and parallel FL for better performance under distributional shifts.\n- The authors provide theoretical guarantees (compared with other baselines) for their framework.\n\nWeaknesses:\n- In sections 3.1 and 3.2, it is unclear to me what the differences are among k, m, and n for the loss equations and their lack of explanations for that (which is confusing). \n- The performance of this paper's framework is kind of underwhelming for me - on PACS, it is worse than FedAvg on average, yet the theoretical analysis seems to show it should be better than FedAvg.\n- I felt like the trade-off between efficiency and performance is not addressed very well in this paper: SPFL will increase the computational cost significantly, yet the performance improvements are limited. \n- The related work is not sufficiently discussed, especially the works on federated domain adaptation [1] /generalization [2], e.g., they used gradient matching methods to find the optimal solution for the new data distribution - It would be better to write a paragraph on this topic in the related work for distribution shifts in federated learning.\n\n[1] Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting.\n\n[2] Federated Domain Generalization with Data-free On-server Matching Gradient.", "was_revised": true, "final_justification": "Thank you to the authors for their detailed response. After reviewing all the reviews and responses, I observe that the authors adopt a new two-stage training method, first using standard PFL, followed by a small number of SPFL rounds, to mitigate performance issues. This hybrid training scheme requires modifications to the original algorithm and presentation. Therefore, I have decided to maintain my original rating.", "rebuttal": "Dear Reviewer wzqr：\n\nWe sincerely thank you for the constructive and insightful feedback. We greatly appreciate your recognition of the significance of addressing domain and category shifts in federated learning, as well as the intuitive value of integrating sequential and parallel update strategies. Your comments regarding theoretical clarity, the trade-off between performance and efficiency, and the discussion of related work are highly valuable and will substantially enhance the quality of the final version. Below, we provide detailed responses to each of your concerns.\n\n**Note: To quickly converge, the most model below was initialized using a FedAvg pre-trained checkpoint trained for 200 rounds on CIFAR-10; the number of clients is 10.**\n### Response to Weakness\n>**W1** :In sections 3.1 and 3.2, it is unclear to me what the differences are among k, m, and n for the loss equations and their lack of explanations for that (which is confusing).\n\nWe thank you for raising this question. As also noted by other reviewers, the current placement of key notations may hinder readability. Although the full definitions of $k$, $m$, and $n$ are provided in the appendix, we agree that introducing these symbols earlier in the main text would significantly improve clarity. For completeness, we briefly clarify them here:\n\n* $K , k$  denotes the **index of local update steps** ;\n* $M , m$ denotes the number of participating clients and the **Client index** in the current sequential update order;\n* $ n$ represents   the starting **Client index**.\n\nIn the final version, we will revise Sections 3.1 and 3.2 to move the definitions of key notations closer to where they are first used, and we have a summary table of notations for quick reference in the appendix. We appreciate your attention to this detail, as it helps improve the overall accessibility and precision of the theoretical exposition.\n\n>**W2:** The performance of this paper's framework is kind of underwhelming for me - on PACS, it is worse than FedAvg on average, yet the theoretical analysis seems to show it should be better than FedAvg.\n\nThank you for your insightful observation. We acknowledge that the performance of SPFL on PACS appears less impressive, particularly in the presence of a significant domain shift. Theoretically, most existing convergence analyses—including ours—assume **bounded model variance $\\sigma$**. However, in cross-domain scenarios like PACS, this assumption no longer holds, as domain-induced distribution gaps can cause the variance  $\\sigma$ to grow unbounded, leading to loose or invalid convergence guarantees. We acknowledge this limitation and will consider extending our analysis to the unbounded variance regime in future work.\n\nMoreover, SPFL was originally designed to tackle **category shift**, where it demonstrates consistent advantages over PFL. Nevertheless, to adapt SPFL to more challenging **domain shift** scenarios and in response to reviewer suggestions, we developed a **two-stage training strategy**: we first perform standard PFL to enable the model to acquire basic recognition ability, and then apply a few SPFL rounds to enhance generalization by exposing the model to more diverse domain knowledge through client rotation. And our GLAM module mitigates domain-induced forgetting by aligning client updates and improving consistency.\n\n As shown in Tab 1, this hybrid strategy—first training with PFL followed by a few rounds of SPFL with GLAM—demonstrates that **SPFL can be effectively leveraged to improve performance on PACS**, despite its limited effectiveness when used alone. R represents the number of training rounds.\n\n**Tab 1: Accuracy (%) on PACS under Domain Shift Setting**\n\n|Method|P|A|C|S|Avg|\n|-|-|-|-|-|-|\n|PFL(120R)|91.02|68.61|**70.85**|**66.84**|74.33|\n|PFL(100R)+SPFL(20R)|93.11|73.24|69.36|64.92|75.16|\n|PFL(100R)+SPFL *with* GLAM (20R)|**93.71**|**74.45**|70.21|64.45| **75.71** |\n\n\n>**W3:** I felt like the trade-off between efficiency and performance is not addressed very well in this paper: SPFL will increase the computational cost significantly, yet the performance improvements are limited.\n\nWe thank you for raising this important point regarding the trade-off between computational efficiency and performance. While SPFL introduces additional per-round overhead due to its sequential update mechanism, it converges substantially faster than traditional PFL methods, thereby reducing the total communication and computation burden across training. We fully agree that if SPFL were applied uniformly throughout all training rounds, the per-round cost might outweigh its benefits. \n\n**Motivated by reviewer suggestions, we adopt a two-stage training strategy to mitigate this issue: the model is first trained using FedAvg to reach a reasonable initialization, followed by a few rounds of SPFL to rapidly refine and boost generalization. This design retains the efficiency of PFL while leveraging SPFL’s robustness and effectiveness in handling category shifts.**\nAs shown in Tab 2 and 3, SPFL achieves superior performance with dramatically fewer communication rounds and lower computational cost:\n\n**Tab 2: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=10)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost (GB)|Total Time(seconds)|Compute Cost(GB)|\n|-|-|-|-|-|-|\n|PFL|79.15|500|116.90|3419.91|175.35|\n|SPFL(Ours)|84.17|15|21.042|678.6|87.675|\n\n**Tab 3: Efficiency Comparison on CIFAR-10 (Dirichlet α=0.1,Number of clients=100)**\n\n|Method|Accuracy(%)|Rounds|Comm. Cost(TB)|Total Time(seconds)|Compute Cost(TB)|\n|-|-|-|-|-|-|\n|PFL|81.38|2000|18.30|30259.82|27.46|\n|SPFL(Ours)|86.68|10|4.68|14713.13|22.85|\n\nThese results demonstrate that SPFL’s higher per-round cost is more than offset by its faster convergence, leading to better final performance and greater efficiency. We will highlight this trade-off more clearly in the revised manuscript and foreground the supporting empirical evidence accordingly.\n\n>**W4:** The related work is not sufficiently discussed, especially the works on federated domain adaptation [1] /generalization [2], e.g., they used gradient matching methods to find the optimal solution for the new data distribution - It would be better to write a paragraph on this topic in the related work for distribution shifts in federated learning.\n\nWe thank you for pointing out the missing discussion on recent gradient matching methods for federated domain adaptation and generalization. As noted, we have included a section in the appendix discussing more domain generalization (DG) approaches. However, the recent works leveraging gradient matching, such as [1] and [2], represent an important and emerging direction that was not sufficiently addressed in the current draft.\nIn the final version, we will revise the Related Work section to explicitly highlight these approaches and move key discussions from the appendix to the main for better visibility. Below is the paragraph we plan to add:\n\n>While most federated learning (FL) methods have made progress in addressing category shift, domain shift remains a more persistent challenge. To this end, AutoFedGP [1] tackles the domain adaptation (DA) problem by projecting local gradients and assigning adaptive weights to align source and target domains. For domain generalization (DG), FedOMG [2] enhances robustness to unseen domains by aligning gradient directions across clients without access to target data. However, both approaches focus on model-level alignment, which may be insufficient when domain shifts induce severe forgetting during sequential client updates. In contrast, our proposed SPFL explicitly targets this issue by introducing GLAM. This global-local alignment mechanism concurrently aligns each client with both its predecessor and the global model, effectively mitigating the forgetting problem caused by domain shift.\n\n[1] Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting.\n\n[2] Federated Domain Generalization with Data-free On-server Matching Gradient.\n\n### Response to Question\n>**Q1**:Are there better ways to combine sequential and parallel FL to better balance computational cost and performance?\nWe appreciate your interest and recognition of our work. Like most reviewers, we hope that SPFL can serve as a practical and effective alternative to PFL in cross-silo federated learning.\n\nTo address the concerns regarding efficiency and optimization, we propose a two-stage improvement strategy informed by reviewers' feedback. First, **we adopt a hybrid training scheme where the model is initially trained using standard PFL, followed by a small number of SPFL rounds to refine the model and enhance generalization under category shifts.** Second, we explore the impact of the client participation ratio during training. Specifically, we conduct experiments with 100 clients and vary the participation ratio $E \\in [0.1, 1.0]$. As shown in the tab 4, we find that a moderate participation ratio (e.g., 50%) offers the best trade-off between communication/computation cost and performance, highlighting the robustness of SPFL under partial participation.\n\n**Tab 4: Accuracy comparison between PFL and SPFL under different client participation rates (\\$E\\$) with 100 total clients.**\n\n|Update\\E|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1|\n|-|-|-|-|-|-|-|-|-|-|-|\n|PFL|41.78|46.99|52.43|53.31|54.04|54.52|54.47|54.45|54.35|54.59| \n|SPFL|51.72|63.57|72.33|76.19|80.84|82.68|84.43|85.41|85.98|89.95| \n\n**We sincerely appreciate your insightful feedback and have addressed your concerns to the best of our ability. Should you have any further questions or recommendations, we would be glad to engage in further discussion.**"}]

Author rebuttal:

Meta-review: This paper proposes SPFL, a novel federated learning framework combining parallel and serial updates, with a solid convergence analysis. It effectively mitigates update-order sensitivity under category and domain shifts, with GLAM addressing the latter more effectively. While one reviewer maintained a low score over presentation choices, most concerns were addressed through additional experiments and clarifications. Therefore, I recommend acceptance.
Decision: Accept (poster)