Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift
Authors: Junbao Chen
Jingfeng Xue
Yong Wang
Zhenyan Liu
Lu Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: federated learning
concept drift
data heterogeneity
Primary area: optimization_for_deep_networks
TL;DR: We propose FedCCFA, a federated learning framework with classifier clustering and feature alignment.

Abstract: Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted to tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is a largely unexplored area. In this work, we focus on real drift, where the conditional distribution $P(\mathcal{Y}|\mathcal{X})$ changes. We first study how distributed concept drift affects the model training and find that local classifier plays a critical role in drift adaptation. Moreover, to address data heterogeneity, we study the feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution $P(\mathcal{Y}|\mathcal{X})$ and the degree of data heterogeneity. Motivated by the above findings, we propose FedCCFA, a federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution $P(\mathcal{Y})$, alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at https://github.com/Chen-Junbao/FedCCFA.

Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift
Junbao Chen Beijing Institute of Technology junbaochenObit. edu.
Jingfeng Xue Beijing Institute of Technology xuej edu. en
Yong Wang
Zhenyan Liu'
Beijing Institute of Technology wangyong@bit.cm edu. cn
Beijing Institute of Technology zhenyanl iu@bit........ edu. on
Lu Huang
Beijing Institute of Technology luhuang@bit. edu. cn
Abstract
Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is largely unexplored area. this work, we focus on real drift, where the conditional distribution P())))))) changes. We first study how distributed concept drift affects the model training and find that local classifier plays critical study the
role in drift adaptation. Moreover, address data heterogeneity, feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution P() and the degree data heterogeneity. Motivated by the above findings, we propose FedCCFA, federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution P(J)). alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at http://twwwooooo/eeeeeanaaaa///a
1 Introduction
Federated Learning (FL) 28 is an emerging allows multiple clients collaboratively train global model without sharing their data. In FL,, clients train the models on their local data and send the updated models to the server for aggregation. Driven the growing need for privacy protection, FL. has been widely applied in various real-world
machine learning paradigm that
scenarios g [6 30 44).
One significant challenge in FL is data heterogeneity, which denotes the discrepancies in the data distributions across clients. Such discrepancies can hinder the convergence of the global model.
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, existing works neglect real-world setting where data heterogeneity and distributed concept drift simultaneuusly exist. Unlike conventional concept drift in centralized machine learning distributed concept drift [19 involves multiple clients experiencing different concept drifts at different times. For example, for the same medical image, diagnoses can vary among doctors (i.e., concept drift across clients), and even a doctor may offer different diagnoses at different times (i.e., concept drift across times). Moreover, the distribution of medical images varies across hospitals (i.e., data heterogeneity). This setting significantly degrades the performance of most FL methods, especially those using a single global model, because the model cannot provide different outputs
[1] 25 26 38
for the same input. A similar research problem is multistream classification [4] [5 46), where sampling bias may exist between the distributions represented by source stream and target stream. Different from this problem, distributed concept drift focuses on the changing conditional distribution P() across clients and over time. For each client at any round, training and test data distribution assumed to be similar.
Several recent works have recognized the concept drift problem in FL.. However, as discussed above, these single-model solutions 12
3 5 14 32 are ill-suited for distributed concept drift. FedDrift [I] considers distributed concept drift and employs multiple models to address this problem. However, FedDrift significantly increases the computation overhead for its distance measure and the storage overhead for multiple entire models. In addition, since the data heterogeneity can affect the loss estimation, FedDrift may fail to merge the models under the same concept. Experimental details are provided in Appendix C.7
To tackle distributed concept drift, we analyze how it affects the model training in FL.. In this work, we focus on real drift [1] 38 in concept drift, where the conditional distribution P))
Test Accuracy
Reprosentation Update
80
Accuracy (%) (44)
Frabenius Norm
200
changes. When P() varies across clients, the representation should not be affected, as the marginal distribution P(X') is invariant. How- ever, as shown in Figure when vanilla FedAvg is adopted and distributed concept drift occurs at round 100, the Frobenius norm of represen- tation update increases drastically, suggesting that the drift leads to large gradients in the repre- sentation. Furthermore, the accuracy of vanilla FedAvg drops rapidly and cannot recover to the accuracy before the drift, indicating the unsuit- ability of the single-model solution.
60
1.50
40-
20
100
50
100
150
200
100
150
2000
Number of Rounds
Number of Rounds
FodAvg
Decoupled
Clcssplid-Clusstriing
Figure I: The impact of distributed concept drift on model training. Distributed concept drift occurs round 100. Decoupled: learning method. Decoupled- Decoupled-Cluustrring: Decou- pled method with classifier clustering.
ceessfier-thnneextreee
Motivated by the observations and analyses above, we decouple the network into an extrac- tor and classifier, and first train the classifier while fixing the extractor. After the classifier learns new conditional distribution, small gradients will be back pack-propagater to the extractor, and then we train the extractor. This decoupled method can effectively adapt to distributed concept drift, demonstrated by the much higher accuracy than FedAvg in Figure []] The small gradient norm of representation at drift round also indicates the small gradients brok-propag from classifier. However, some clients may share similar P(Y) and pure decoupled methods neglect the fine-grained collaboration between local classifiers. This will make each client's local classifier overfit to its local data. To enhance generalization performance, we develop a class-level classifier clustering method. Our clustering method separates the classifier for each class (referred as class classifier), and then aggregates clients' class classifiers trained under the same conditional distribution P()|)). The clients under the same conditional distribution share the aggregated class classifiers. This aggregation reduces the bias introduced by any single client's 's data, contributing to improved generalization performance. As shown in Figure [ with the benefit of classifier clustering, the generalization performance is further improved, demonstrated by higher accuracy and smaller gradient norm. To remedy the data heterogeneity under distributed concept drift, we propose an adaptive feature alignment method, which aligns the feature spaces of the clients with the same conditional distribution P()) and adjusts the alignment weight according to the entropy of label distribution.
To summarize our contributions:
2

---

1. We explore the impact of distributed concept drift on FL training and propose a class-level classifier clustering approach that not only adapts to this drift but also enhances generalization performance (Section 4.
2. We propose clustered feature anchors to achieve feature alignment under distributed concept drift and propose an adaptive alignment weight to prevent severe data heterogeneity from impeding main task learning (Section 4.2
3. We propose FedCCTA, federated learning framework with classifier clustering and feature alignment (see Section 4.3) In Section 5| conduct extensive experiments and empirical results demonstrate that FedCCFA can effectively adapt to distributed concept drift under data heterogeneity and significantly outperforms existing methods.
2 Related work
Concept drift in FL.. Concept drift has been extensively studied in centralized machine learning
11
26 38). Several recent works have recognized the concept drift problem in FL and
proposed methods tackle this issue, including regularization (5), continual learning [3] [4] and
adaptive learning rate 22312 These works assume that the conditional distributions P() of all
clients' dataset are the same, so only a single global model is trained. However, as proposed in [19 and discussed in Section M. distributed concept drift may exist in FL setting and the single-model solution fails to adapt to this drift. To address this problem, FedDrift [19 creates new models based on drift detection and adaptively merges models by hierarchical clustering. However, FedDrift still faces some challenges: 1) incorrect model merging caused by data heterogeneity; 2) high computation and communication overhead for measuring cluster distance; 3) high storage cost for maintaining multiple global models.
Data heterogeneity in FL. Data heterogeneity hinders fast convergence when using vanilla FedAvg (88) In this work, we focus on the heterogeneity of label distributions (referred to as label distribution
skew). To alleviate this issue, previous works either focus on local training [ 20 22
24
29)
global aggregation 17
To further study how data heterogeneity affects FI., several recent works have focused on the representation of model, such dimensional collapse [36] and
18 33 41.
inconsistent feature spaces [43 45 48, 50]. To align clients' feature spaces, FedFA (50] and FedPAC
43 regularize the la distance between local features and global anchors. To promote more precise alignment, FedFM 45 uses a contrastive-guiding method to further maximize the distance between the feature and correppoognn anchors. However, severe data heterogeneity may significantly increase the loss of feature alignment, hindering the model convergence.
Clustered FL and personalized FL. Some clustered FL methods group clients with the same data distribution into cluster, which can also adapt to distributed concept drift. To group clients, several
works measure the similarity based on gradient information (9 [0] 335 or training loss [3] 27).
However, these methods face the following challenges: 1) unknown number of clusters; 2) large computation and communication overhead for estimating the training loss; 3) considerable storage
43 40 are also robust to distributed concept drift, since each client trains a local model for its local data distribution. However, most personalized FL methods neglect the classifier collaboration among the clients with similar data distributions, limiting the performance of models. Different from personalized FL approaches, this work focuses on generalized FL, aiming to enhance generalization performance.
cost for multiple global models. Personalized FL methods 66 23
31
37
3 Problem formulation
We consider a FL setting where there are K clients and a central server. The k-thh client has local dataset D₁ with data distribution Pa ))), where X' is the input space and y is the label space with C' classes. Let f(w; x, y) be the task-specific loss function associated with model w and data sample (x, y). The global objective of FL. can be formulated as:
K
min (F(w) = Σ Pk ((()) weRd
(1)
1=1
3

---

where pkk is the aggregation weight of the k-th client and (w( = B(x,y)- (w: X, y)] is the (w;;
local objective.
To distinguish different types of concept drift, we decompose the joint distribution P(X,)) as: P(X, P(X) P( P()) P(X In this work, focus on the real drift in concept drift. Real drift means that the conditional distribution at round may be different from that at the previous round, ie., p()) ()] * p((-1) ()) In addition, under distributed concept drift, this conditional distribution may vary across clients, i.e., (D)()) + Pf ()1x).
To address distributed concept drift, we decouple the model parameterized by w {0, Φ) into feature extractor So parameterized by 0 and classifier f parameterized by Φ. Given sample (x, y) E X' x y/ the feature extractor ʃe X' + 2 maps the input into a feature vector 2 = So (x) in the feature space Z, and then the classifier fo Z + y maps the feature vector z to the label space y. All clients share the feature extractor and each client maintains its local classifier. To align clients' feature spaces, we introduce a regularization term G3 O0; As) into the local objective function. Here, A₁ represents form of global information that client i uses to align its feature space. Let + = { 22..... Φx) denote the set of all clients' local classifier parameters and denote the parameters of shared feature extractor. The global objective can be reformulated as:
K
min F(0, ②) = 0.
Σ
(0. + (0;
(2)
where À is the weight of feature alignment.
4 Methodology
FedCCFA decouples the network into a feature extractor fo and a classifier fg. In FedCCFA, the classifier is the last layer of the model (i.e., a linear classifier), and the feature extractor is composed of the remaining layers. this section, we first present two methods used in FedCCFA and then describe the design of FedCCFA.
4.1 Classifier clustering
For better classifier collaboration, we introduce new method for client clustering. Specifically, all clients share the same extractor and train their personal classifiers. Given the identical dimension reduction [x). the personal classifier learns the local data distribution. Therefore, the linear classifiers with similar weights exhibit similar data distributions, especially the conditional distribution P()IX). Different from existing clustered FL methods, FedCCFA directly uses the classifier weights for clustering, which significantly reduces the computation and communication cost.
However, if using the weights of local classifiers, clustering may be disturbed by factors: (1) variation in the classifier weights before training, and (2) class imbalance. For the first one, since clients' classifiers are different before local training, the classifiers trained under the same conditional distribution may differ significantly, which can lead to wrong clustering results. For the second one, the classifier can be easily biased towards head classes with massive training data, the classifiers trained under the similar marginal distributions ((X) may be grouped, although their conditional distributions P() are different.
Balanced classifier training. To address the above two problems, before local classifier training, FedCCFA trains balanced classifier and uses it for classifier clustering. Specifically, ensure that the classifier weights before training are identical, each selected client & E I(() updates its classifier (1(9) by the initial global classifier of (0). Then, address class imbalance, client samples a balanced
batch (( from ( to train its classifier of (t) while fixing its extractor 0 o²)
No V V F1 (())
where Vi denotes the learning rate for classifier.
(1)
After a training iterations, client l saves the balanced classifier d and sends it to the server after local training. Note that, to reduce additional computation cost, randomly select 5 samples for

---

each class E [C]] (i.e., the balanced batch size [6 is 5 * C), and set training iterations to a small value.
Class-level clustering. Unlike FedPAC 43 that uses entire classifier for collaboration, FedCCFA conducts classifier clustering class-level. Specifically, after receiving all balanced classifiers
(t) {Øt J. the server separates these classifiers for each class (referred to as class classifier). Then, for each class the server measures the class-level distance Dz(i, between client i and client
(+) by their class classifiers di.c and &) (t)
where j. E I(O, To effectively measure distance between
high-dimensional vectors, we exploit MADD 34 and realize a measure based on cosine distance:
Do(i,5)=
Σ [ gEZ
(t) (1) Cos(@i ( Cos( j.s 200)
(f)
Vi,j E E
(4)
where Cos(a, b) denotes the cosine distance between vector and b.
After getting the distance matrix De, DBSCAN clustering algorithm is used to get the class-level
clusters s(t)) For each cluster Sitt E s!!,, clustered class classifier aggregated as:
(t)
VS E E
m.C
Pkec
Sm.ee
AES!!
where | denotes the number of clients in cluster Sm,e- Each client k E (it) updates its local class classifier &he by ( Here we use uniform aggregation considering privacy concerns. More details about aggregation method are shown in Appendix C.6
4.2
Adaptive feature alignment
To alleviate the inconsistency in feature spaces 45
we introduce a regularization term into the local objective function align clients' feature spaces. Compared to existing aligmment methods
43 45 50), our method is robust to two challenges: concept drift and the degree of data heterogeneity.
Clustered feature anchors. Existing feature alignment methods suppose that all clients' condi- tional distributions P() are identical. Based on this, global feature anchors 45 50 (also known as feature centroids 453 are leveraged to align clients' feature spaces. However, under distributed concept drift scenario, the conditional distribution P() may vary across clients, which can result in the differences in feature anchors. Therefore, clients require the global anchors that match their conditional distributions; otherwise, incorrect global anchors can mislead feature alignment. Moti- vated by this, we propose a feature alignment method using clustered feature anchors. Specifically, at round f, each client uses its feature extractor to compute the local feature anchor a( for each class
(())
1
Σ
fog (x), Vc E [C]
(x.c)E
Then, with the help of our class-level clustering, we compute the global feature anchor A of class for each cluster s!!)
A00
Σ te
VS(I) E S(!) m.e
Sm.el
Finally, each client k E s(t) uses its global anchors -A align its feature space during local training. FedCCFA leverages the contrastive-guiding loss proposed in FedFM [45), but uses clustered feature anchors. Let sim(u, v) denote the cosine similarity between and v. Then the alignment loss function for sample (x, c) with label è is defined as:
exp(sim([ (x), Gi Go = log Ee
(8)
exp(sim Logg (x), (1)
where f0g (x) is the representation vector of input x and 7 denotes a temperature parameter.
5

---

S
Balanced classitte training
Client clustering
Belenced diesiiera
Closs closs flors
S
Balanoed classifler training
spir
Eatedoo
Class
6,
Local data
Balanced
ONe
Balanced diensiher
6", g/
Class
Clustering
balcch
Local training
Client clusters
Extidarr
Aggregation
Chucs
Choos
6
Class
Class
[ Closo
Local
Adactive aligrment
Local anchors
Clsseleer aggregetion
Anchor
g1743)
Extrctor
Frozen
Feature
Classifiar
Trainable
anchers
Server
Extractor aggregation
Figure 2: An overview the proposed FedCCFA. Clients train balanced classifiers and local models, and then generate local anchors. The server performs client clustering with the help of balanced classifiers, and then aggregates local models and local anchors.
the alignment weight is fixed and all clients use the same weight. However, severe data heterogeneity will signif- icantly increase the gradients alignment term, impeding the main task learning. Experimental results are presented in Table [ To balance the main task learning and feature alignment, it is essential to reduce the alignment weight under severe data heterogeneity. Note that, in this work, we focus on the label distribution skew and the degree of this heterogeneity can be reflected by the marginal distribution p(. (J)) Based this intuition, leverage the entropy of marginal distribution pl ())), denoted as H((P)) ())). to adaptively determine the alignment weight:
Adaptive alignment weight. In existing feature alignment methods 43 45 50
HL(PK) (Q1) GL(OR)
- No Vo
(00)
(9)
7
where t/ denotes the learning rate for extractor and the scaling factor. For the round T, start feature alignment, we use the empirical value T, = 20 proposed in FedFM [5]
4.3 FedCCFA
We now present FedCCFA, federated learning framework to adapt to distributed concept drift under data heterogeneity. The pipeline of FedCCFA is shown in Figure 2 The procedure of FedCCFA is formally presented in Algorithm Appendix A
At each round f, each selected client & E (+)) updates its extractor 0 by the global extractor ((t) and then starts local training procedure. Specifically, each client trains balanced classifier (Equation]) 33 for client clustering. Then, it fixes its feature extractor and trains local classifier adapt to concept drift. Finally, it trains its feature extractor (Fquation 9 Since classifier (f) learns the conditional distribution p(() ()))2) concept drift will not lead to larges gradients in the representation. After local training, each client generates its local anchors for each class (Equation 6).
Local training.
Global aggregation. aggregation procedure. Specifically, the server performs client clustering with the help of balanced classifiers. Then, the server aggregates all feature extractors:
After receiving the local parameters and local anchors, the server starts the

---

((+11)
1
(10)
ΣKEZI 1
KET
Finally, according to the clustering results, the server aggregates local classifiers and local anchors for each cluster.
Experiments
5.1 Experimental setup
Datasets and models. We conduct experiments three datasets, namely Fashion-MNIST 42 CIFAR-I0 21 and CINIC-10 77]. We construct two different CNN models for Fashion-MNIST and CIFAR-IO/CINIC- 10, respectively. Details of datasets and models provided in Appendix B.
Baselines. We compare FedCCFA against the methods falling under the following categories:
(1) single-model methods without drift adaptation: FedAvg (28), FedProx (24). SCAFFOLD [20] and FedFM [45]: (2) single-model methods with drift adaptation: Adaative-FedAAv (shortened to
AdapFedAvg) and Flash [2]; (3) personalized FL methods: pFedMe [37] Ditto 23 FedRep
6 FedBABU 31 and FedPAC 4 (4) clustered FL. methods: IFCA 13 and FedDrift 19
Federated learning settings. We consider two FL scenarios: 20 clients with full participation and
100 clients with 20% participation. We use a widely considered non-IID setting [22 39 47]: for each class E [C], we sample probability vector P. (Pc,1, Pe.2..P........ .P.Kk. Dirk (a) and allocate
a Pe.k proportion of instances of class e to client E [K]. where Dirk (a) denotes the Dirichlet distribution with concentration parameter o;; smaller means unbalanced partition. To evaluate the adaptability to concept drift, each client's 's training set contains at least 5 samples per class.
Concept drift settings. In this work, we focus on the distributed concept drift proposed [1]], where the conditional distribution P())XX)) varies over time and across clients. We use three label swapping settings to simulate the conditional distribution changes across clients: for client E [K]. (1) class and class are swapped if &%10 3: (2) class 3 and class 4 are swapped if 3 <= k%10 <= 5; (3) class 5 and class 6 are swapped 4%10 > 5. We simulate three patterns of conditional distribution changes over time: (1) Sudden Drift. All label swapping settings occur at round 100; (2) Incremental Drift. Label swapping settings (1), (2) and (3) occur at round 100, 110 and 120, respectively; (3) Reoccurring Drift. All label swapping settings occur round 100, and these settings occur again at round 150.
Implementation details. We run 200 communication rounds for all experiments. To comprehen- sively evaluate the generalized performance and drift adaptability, for each client, report the generalized accuracy of its model on the whole test set. In particular, the conditional distributions P()IX))) of each client's training set and test set are the same every round. For clustered and single-model methods, each client evaluates its global model; for the other methods, each client evaluates its personal model (or the combination of global extractor and personal classifier). Finally, we report the average accuracy across clients at the last round. For local training, we use SGD optimizer. The weight decay 0.00001 and the SGD momentum is 0.9. For all datasets, we set local epochs E 5. For the decoupled methods, extractor learning rate is 0.01 and classifier learning rate is 0.1; for the other methods, local learning rate is 0.01. For FedCCFA, scaling factor y 20.0, iterations of balanced training s is 5 and balanced batch size is 5 C (5 samples per class). For classifier clustering, maximum distance e and minimum samples used in DBSCAN algorithm are and 1, respectively. More details of hyperparameters are provided in Appendix B.2
5.2 Experimental results
We focus three points in our experiments: (1) the generalization performance of FedCCFA; (2) the adaptability to distributed concept drift compared to existing methods; (3) the effects of our classifier clustering and adaptive alignment weight. Full experimental results are provided in Appendix C
7

---

Table Generalized accuracy under Dir(0.5). The sample ratio 100% and 20% for 20 clients and 100 clients, respectively. All results are averaged over 3 runs (mean + std).
Fashion-MNIST
CIFAR-10
CINIC-10
Method
clients
100 clients
20 clients
100 clients
20 clients
100 clients
FedAvg
90.57:1:.18
90.22:10.33
78.56:10.20
74.05+:0.22
62.23:10.28
58.28:10.21
FedProx
90.29:+0.17
90.22:+0.02
78.49:10.20
73.50:10.61
58.40-+0.86 02.17
SCAFFOLD
90.33=10004
87.72:10.19
75.84:10.71
59.0110.74
58.92:+.122
47 17.69:10.35
FedFM
90.03:10.26
90.2010.35
78.9710043
74.8910.42
62.96:10.27
58.66:10.133
AdapFedAvg
90.42++0.25
90.39++011
78.28+0.38
73.91+0066
61.96++0.30
58 08.80+00.42
Flash
90.19+0..05
89.94+0.30
78.01+0.61
75.13+0.23
61.11+0.43
60.30++0.225
PFedMe
85.06110.32
84,8110.31
63,08111228
51.4910.08
43.75:10.07
4180110.49
Ditto
90.3311..15
89.7311.119
77.03-10.39
72.26110.4
59.3011.225
56.33110.1
FedRep
81.70=+0.10
80.39+0.30
60.08
12.19+
43.63++0.38
38.48+0.12
FedBABU
16.83+1.66
80.69=0.54
58.99+0.25
55.42:+0.21
40.74+0.21
41 03
FedPAC
05.01
87.61+0.26
67 67.96+0.40
67.77+0.36
43.83+0.69
47.50+0.38
IFCA
1.17
88.6110.51
64.10.5
72.53110.30
57.84.10.18 18.43
FedDrift
90.38::0.36
90.33:+.16
78.59+0.35
73.70++0.7
02.1233
58,48+0.72
FedCCFA
89.81
89.74:10.18
78.38
74.44:10.26
61.13:10.25
58.70:10.33
Table Generalized accuracy under sudden drift setting and (0.(.. The sample ratio is 100%
and 20% for 20 clients and 100 clients, respectively. All results are averaged over 3 runs (mean + std).
Fashion-MNIS:
CIFAR-10
CINIC-10
Method
20 clients
100 clients
20 clients
100 clients 20 clients
100 clients
FedAvg
1.12
68. 08.1511.990
60.76+0.37
18.15+0.23
49.69
16.18
FedProx
69.25=0.60
68.47::0.19
61 6..25
57.68=00111
49.45=0.40
46.11=00.59
SCAFFOLD
69.92-1:1.24
69.66:1:0.24
58.67-10.39
45.87+11.29
46.34:10.70
3..94
FedFM
68.48-10.79
69.12:1:.34
60.61 +..27
57.51
50.25:+0.11
46.23:10.81
AdapFedAvg
1.06
68. 0.55
60.92:10.20
58.2310.30
49.86-10.36
46.34110.1
Flash
10.00=10.00
71.16:10.28
59.84-10.75
60.49:1027
49.4411041
49.28:10.40
PFedMe
82.37++0.09
07.66++0.17
58.92+1.66
44.15+11.08
41.34++0.2
37.76-1-0.78
Ditto
78.51+0082
79.62+0.31
67.45+00.01
63.35+0.67
01.06++0.26
48.40+-0.34
FedRep
81.9911..37
81.0710.29
64.5110088
53.3011.51
44.00110.26
38.1510..14
FedBABU
79.83110.18
81.6810.110
58.8710.550
55.2910.61
41.24:10.90
40.49:10.30
FedPAC
84.00-10.47
87,2410.16
66.47.10.30
64.73111.22
44.38
46.22-10.47
IFCA
87.02-t0.06
88.25+:021
61.24:10.22
57.31=:0.44
44.04+22.00
45.43+-0.22
FedDrift
78.95+1..30
87.75+00.4
48.17442.43
57.26+10.36
34.23++0.04
45.60+1.07
FedCCFA
89.50-10.10
89.3911.14
76.65.10.63
73.21
51.12.11.70
52.62-10.51
Performance without concept drift. First, evaluate all methods under the no drift setting, and these results are the base performance compared against various concept drift settings. For each method, we three trials and report the mean and standard deviation. Table T presents the generalized accuracy under Dir(0.5). We observe that FedCCFA outperforms all decoupled methods (FedRep, FedBABU and FedPAC) and personalized FL methods (pFedMe and Ditto). indicating that FedCCFA enhances the generalization performance of local classifiers. Besides, the performance of IFCA under 20 clients with 100% participation is worse than the performance under 100 clients with 20% participation. This is because multiple global models are trained under similar data distribution (i.e., each global model is trained with much less data). contrast, with the help of our classifier clustering, clients with similar data distribution will share the same classifier. Compared with single-model methods, FedCCFA achieves lower accuracy, which is attributed to decoupled training. Specifically, for decoupled methods (e.g., FedPAC, FedRep and our FedCCFA), training the classifier first might cause to the initial features, resulting gradient attenuation during backpropagation. This will restrict the subsequent extractor's training process and prevent it from optimally learning

---

Table 3: Ablation study on the clustering imput. We run experiments under sudden drift setting and Dir(0.5). We apply partial participation.
Table 4: Ablation study on alignment methods. We run experiments under sudden drift setting and Dir(0.5). We apply partial participation.
Clustering imput
CIFAR-10
Alignment methods
CIFAR-10
local classifiers
68.39:#0.71
w/o alignment
71.40::0.43
balanced classifiers
73.21
w/o clustered
72.77=0.71
oracle
73.55++0.27
w/ clustered
73.21+00.2
Table 5: Ablation study on adaptive alignment weight. We apply partial participation. Experiments are run on CIFAR-I0 and all results are averaged over runs (mean + std).
Weight
Dir(0.01)
Dir(0.1)
(0.().5)
fixed (0)
0..4
69.33
72.58
fixed (0.1)
1..8
70.7
73.83++0.53
fixed (I.0)
40.8941.977
67.43
74.80+1041
7 = 10
60.30:10.79
69.75:10.19
74.97110.48
7 20
61.29:10.59
70,21
1..44
7 = 50
62.15-10.49
70.11
73.32+00.6
features. In contrast, when training the entire model together, all layers can simultaneously adapt to the training data, allowing them synergize effectively.
Performance under distributed concept drift. Next, we evaluate all methods under the distributed concept drift setting.
the generalized accuracy under this setting. We observe that FedCCFA significantly outperforms all baselines. Specifically, under full participation (i.c., 20 clients with 100% participation), FedCCFA outperforms the second-best method 2.48%, 9.50% and 0.06% on Fashion-MNIST. CIFAR-10 10 and CINIC-10, respectively; under partial participation (i.e., 100 clients with 20% participation), FedCCFA outperforms the second-best method 1.4,, 8.48% and 3.34% on Bashion-MNiST CIFAR-10 10 and CINIC-10, respectively. Compared with the no drift setting, the results of accuracy decrease are shown in Appendix C2.2 These results also show that (1) personalized FI. methods can effectively adapt to distributed concept drift, demonstrated by the smaller accuracy decrease compared with single-model methods; (2) clustered FL methods may fail to adapt to distributed concept drift under data heterogeneity, because clustering results may be affected by data heterogeneity.
Effects of balanced classifier training. As analyzed in Section 4.1 balanced classifier training is crucial for classifier clustering. We conduct experiments under sudden drift setting and Table 3 shows the results of our ablation study on the input of clustering. We see that using the weights of balanced classifiers can significantly increase the generalized accuracy. Furthermore, to demonstrate that accurate clustering results contribute to this improvement, we compare our method against oracle method which has knowledge about the accurate clustering results at each round, Experimental results show that oracle method indeed increases the generalized accuracy and our method is comparable to the oracle method. These results demonstrate that classifier clustering can enhance the generalization performance and our method achieves accurate clustering.
Effects of clustered feature anchors. To verify the efficacy of clustered feature anchors, we ablate FedCCFA by variety of alignment methods. From Table 4 we see that (1) feature alignment can boost the accuracy under data heterogeneity; (2) clustered feature anchors can facilitate more precise feature alignment under distributed concept drift.
Effects of adaptive alignment weight. We turn the degree of data heterogeneity (a E {0.01, 0.1,0.5) and smaller a corresponds to more severe heterogeneity), and report the average accuracy in Table 5 First, we conduct experiments with various fixed alignment weights (0, 0.1, 1.0). Experimental results show that small alignment weight is better under severe data heterogeneity. In addition, under Dir(0.01), training without feature alignment outperforms training with feature alignment, indicating that feature alignment impedes main task learning under this setting. Then, turn the

---

scaling factor {20, 50, 100}. We see that our adaptive alignment weight is robust to the degree of data heterogeneity and the performance is not sensitive to the scaling factor.
6 Conclusion
In this work, we take a further step towards distributed concept drift in federated learning. We have shown that FedCCFA can effectively adapt distributed concept drift, and outperforms existing methods under various concept drift settings. Further experiments suggest that our classifier clustering method significantly enhances the generalization performance for decoupled FL methods. Besides, our clustered feature anchors can enhance the precision of feature alignment and our adaptive alignment weight can stabilize the training process when using feature alignment under severe heterogeneity. We hope that FedCCFA can inspire more works on classifier collaboration and other types of concept drift in federated learning.
Limitations. FedCCFA takes some steps to train balanced classifiers, which increases computation cost. Although we reduce it by setting small iteration and batch size, it is preferable remove these steps. We will investigate other efficient methods in the future. Besides, feature alignment, used in FedCCFA, FedFM and FedPAC, could impede main task learning under severe data heterogeneity. Therefore, research on the feature alignment under severe heterogeneity is also of interest.
Acknowledgements
This work was supported by the National Natural Science Foundation of China [No.62 NN..6212242]..
References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In the 9th Interna- lional Conference on Learning Representations, 2021.
[2] Giuseppe Canonaco, Alex Bergamasco, Alessio Mongelluzzo, and Manuel Roveri. Adaptive federated learning in presence of concept drift. In 2021 International Joint Conference
Neural Networks (IJCNN), pages 1-7. IEEE, 2021.
[3] Fernando E Casado, Dylan Lema, Marcos F Criado, Roberto Iglesias, Carlos V Regueiro, and Senén Barro. Concept drift detection and adaptation for federated and continual learning. Multimedia Tools and Applications, pages 1-23, 2022.
[4] Swarup Chandra, Ahsanul Haque, Latifur Khan, and Charu Aggarwal. An adaptive framework for multistream classification. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 1181-1190, 2016.
[5] Yujing Chen, Zheng Chai, Yue Cheng, and Huzefa Rangwala. Asynchronous federated learning for sensor data with concept drift. In 2021 IEEE International Conference on Big Data (Big Data), pages
IEEE, 2021.
[6] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning, pages 2089-2099, 2021.
[7] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- 10 is not imagenet or cifar- 10. arXiv preprint arXiv:/810.033055, 2018.
[8] Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z. Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford Wood, Chien-Sung Tsai, al. Fed- erated learning for predicting clinical outcomes in patients with covid-19. Nature medicine, 17350):735 2021.
[9] Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Fedgroup: Efficient clustered federated learning via decomposed data-driven measure. arXiv preprint arXiv:2010.06870, 2020.
10

---

[10] Moming Duan, Duo Liu, Xinyuan Ji, Yu Wu, Liang Liang, Xianzhang Chen, Yujuan Tan, and Ao Ren. Flexible clustered federated learning for client-level data distribution shift. IEEE Transactions on Purallel and Distributed Systems, 33(1 1):2661-2674, 2021.
[11] João Gama, Indré Žliobaitè, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. survey on concept drift udaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
[12] Y. Gao, Swarup Chandra, Yifan Li, L. Khan, and Bhavani M. Thuraisingham. Saccos: A semi-supervised framework for emerging class detection and concept drift adaption over data streams. IEEE Transactions on Knowledge and Data Engineering. 34:1416-1495 2022.
[13] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33: 19586- 19597, 2020.
[14] Yongxin Guo, Tao Lin, and Xiaoying Tang. Towards federated learning time-evolving heterogeneous data, arXiv preprint arXiv:2112.13244 2021.
[15] Ahsanul Haque, Zhuoyi Wang, Swarup Chandra, Bo Dong, Latifur Khan, and Kevin W Hamlen. Fusion: An online method for multistream classification. P Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 919-928, 2017.
[16] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 181 I.06004, 2018.
[17] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.0633 2019.
[18] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. In the 11th International Conference on Learning Representations, 2023.
[19] Ellango Jothimurugesan, Kevin Hsieh, Jianyu Wang, Gauri Joshi, and Phillip Gibbons. Federated learning under distributed concept drift. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, pages 5834 5853. PMLR, 2023.
[20] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, pages 5132-5143,
2020.
[21] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University Toronto, 2009.
[22] Qinbin Li, Bingsheng He, and Dawn Song. contr--cnt federated learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10713-10722, 2021.
[23] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pages 6357-6368, 2021.
[24] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
[25] Wendi Li, Xiao Yang, Weiging Liu, Yingce Xia, and Jiang Bian. Ddg-da: Data distribution generation for predictable concept drift adaptation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, volume 36, pages 4092-4100 2022.
[26] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12):2346 2363, 2018

---

[27] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications federated learning. arXiv preprint arXiv:2002.16119 2020.
[28] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficcee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282, 2017.
[29] Matias Mendieta, Taojiannan Yang. Pu Wang, Minwoo Lce, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8397-8406,
2022.
[30] Viraaji Mothukuri, Prachi Khare, Reza M Parizi, Seyedamin Pouriyeh, Ali Dehghantanha, and Gautam Srivastava. Federated- Seteratted-lamming-bbbeee anomaly detection for iot security attacks. IEEE Internet of Things Journal, 94)):2545-2554, 2021.
[31] Jachoon Oh, SangMook Kim, and Se- Young Yun. FedBABU: Toward enhanced represen- tation for federated image classification. In the 10th International Conference on Learning Representations, 2022.
[32] Kunjal Panchal. Sunav Choudhary, Subrata Mitra, Koyel Mukherjee, Somdeb Sarkhel, Saayan Mitra, and Hui Guan. Flash: Concept drift adaptation in federated learning. In Proceedings of the 40th International Conference on Machine Learning, pages 26931-26999 2023.
[33] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečny, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In the 9th International Conference on Learning Representations, 2021.
[34] Soham Sarkar and Anil K Ghosh. On perfect clustering of high dimension, low sample size data. IEEE transactions pattern analysis and machine intelligence, 42(9):2257-2277, 2019.
[35] Felix Sattler, Klaus-Robers Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):3710-3722 2020.
[36] Yujun Shi, Jian Liang, Wenqing Zhang, Chuhui Xue, Vincent Y. F. Tan, and Song Bai. Un- derstanding and mitigating dimensional collapse in federated learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1l))2933-2999 2024.
[37] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394-21405 2020.
[38] Ashraf Tahmmasi,, Ellango Jothimuragesan, Srikanta Tirthapura, and Phillip B Gibbons. Drift- surf: Stable-state / reactive-state learning under concept drift. In Proceedings of the 38th International Conference on Machine Learning, pages 10054-10064, 2021.
[39] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In the 8th International Conference on Learning Representations, 2020.
[40] Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020.
[41] Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Im- proving communication-etticinn distributed sgd with slow momentum. arXiv preprint 110000191, 2019.
[42] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:170.007:00 2017.
[43] Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In the 11th International Conference on Learning Representations, 2023.
12

---

[44] Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of healthcare informatics research, 5:1-19, 2021.
[45] Rui Ye, Zhenyang Ni, Chenxin Xu, Jianyu Wang, Siheng Chen, and Yonina C. Eldar. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. IEEE Transactions on Signal Processing, 71:4224 4239, 2023.
[46] En Yu, Jie Lu, Bin Zhang, and Guangquan Zhang. Online boosting adaptive learning under concept drift for multistream classification. In Proceedings of the AAAl Conference Artificial Intelligence, volume 38, pages 16522-16530, 2024.
[47] Mikhail Yurochkin. Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pages
2019.
[48] Lin Zhang, Yong Luo, Yan Bai, Bo Du, and Ling- Yu Duan. Federated learning for non-iid data via unified feature learning and optimization objective alignment. In Proceedings of the IEEE/CVF international conference computer vision, pages 4420-4428, 2021.
[49] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021.
[50] Tailin Zhou, Jun Zhang, and Danny HK Tsang. Fedfa: Federated learning with feature anchors align features and classifiers for heterogeneous data. IEEE Transactions on Mobile Computing, 2023.
13

---

A Algorithm
The procedure of FedCCFA is formally presented in Algorithm [ methods 6 [11, the sections with green color and red color are the additional computation and communication steps, respectively.
Compared with decoupled FL
Local training, At each round t, each selected client A E I(( first updates its extractor 0 by the global extractor ( Then, starts local training procedure. Lines 6-7, Line Line 9 and Lines 10-12 correspond to balanced classifier training, local classifier training, extractor training and local anchor calculation, respectively. Since the classifier Φ learns the conditional distribution pl ()), concept drift will not lead to larges gradients in the representation.
Global aggregation. After receiving the local parameters and local anchors, the server starts the aggregation procedure. Line 15, Lines 17-18 and Line correspond to extractor aggregation, class-level clustering and the aggregation of classifiers and anchors, respectively. Note that, for better description, each client updates its classifiers and global anchors in a double loop, as described in Line 21. In the implementation, the server sends aggregated class classifiers and global anchors to each client at the end of round f.
Algorithm FedCCFA
1: Input: number of communication rounds T, initial model {0, Φ). extractor learning rate Ne. classifier learning rate Veo, number of local epochs E, iterations of balanced training 8, maximum distance
2: Initialize w (0) = (0 (0)
3: for ...,..,,
do
4: Sample clients (+)) S [K] and broadcast (0()) "0 each selected client
5: for client I(() in parallel do
6: Set 0000 = (())
and sample a balanced batch b
7:
Update Φ (t)
for iterations, and save it as Φ
(t)
train balanced classifier
8:
Set @ = Φ
and update (h as (3) for 1 epoch
train local classifier
Update (t)) as for epochs
9:
train extractor
10:
for class E [C]do
compute the local feature anchor akee (6 end for
13:
Send ox end for
(1)
and {a
(C 1 to server
14:
15:
Update global extractor: (++1)
( ΣMEI
10(0)000)
Exert
for class E (C] do
Vi.j (t); calculate class-level distance Di(i.) between &i and (2) Φj
17:
5 + D-BSCAN(D,,.
for each cluster sit) E s!t) do
19:
get (1)) and (t) as and respectively
client sets (0t) Φ
Pi and appends A to (() Ak E slt)
21:
22: end for
23:
end for
24:
end for
14

---

B Experimental details
All experiments are conducted using PyTorch and all results are produced with an Intel Core i7- 13700K CPU and a NVIDIA GeForce RTX 4090 GPU. The required memory is around 2GB 7GB.
B.1 Datasets and models
16
15
Clentic
Chicnta
(a) CIFAR- 10 under (ir(0.1)
(b) CIFAR-I0 10 under Dir(0.5)
Cleests
Clieds
(c) CINIC-10 under Dir(0.1)
(d) CINIC-10 10 under (ir(0.5)
Figure Data partition visualization under clients with full participation.
Cloase
10
12
14
15
15
27
9 11 12
13
15 16 17
Chamns
Chir
(a) CIFAR-10 10 under (ir(0.1)
(b) CIFAR- 10 under Dir(0.5)
Classss
10
12
15
9 11
13
15 16
Clenii
Clicata
(c) CINIC-10 under Dir(0.1)
(d) CINIC- 10 under Dir(0.5)
Figure 4: Data partition visualization under 100 clients with 20% participation.
We evaluate our method and all baselines on three datasets: (1) Fashion-
42 dataset
comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories (7,000 images category), where the training set has 60,000 images and the test set has 10,000 images; (2) CIFAR-I0 (1 a dataset consisting 60,000 32*32 color images in 10 classes (6,000 images per class), where the training set has 50,000 images and the test set has 10,000 images; (3) CINIC-10 (7): an augmented extension CIFAR-10. containing the images from CIFAR-10 (60,000 images) and selection of ImageNet (210,000 images downsampled to 32*32). We use the same model architectures
15

---

as those used in FedPAC [4]] For Fashion-Moi MNIST, we construct CNN model consisting of two convolution layers with 16 5 and 32 5*5 filters respectively, each followed by a max pooling layer, and two fully-connected layers with 128 and 10 neurons before softmax layer. For CIFAR-10 10 and CINIC-10, the CNN model similar the model used for Fashion-MNIST but has one more convolution layer with 64 5*5 filters.
We visualize the data partition for CIFAR-I0 and CINIC-I0 under two degrees of data heterogeneity (i.e., (Or(0.1) and (05().))). Figure 3 shows the partition visualization under full participation (i.e.. 20 clients with 100% participation). Since CINIC-10 is much larger than CIFAR-10 and we consider the setting where clients have different data sizes (i.e., quantity skew), data distribution of CINIC-10 more biased. Figure shows the partition visualization under partial participation (i.e.. 100 clients with 20% participation). We randomly choose 20 clients for visualization. We see that data heterogeneity is more severe under full participation, due to more severe quantity skew.
B.2 Hyperparameters
We employ SGD as the optimizer
methods. The batch size set to 64, the momentum is set to 0.9 and the weight decay is set to 0.00001. For all datasets, set local epochs E 5. In particular, for the decoupled methods, we first train the classifier for epoch with learning rate no = 0.1, and then train the extractor for 5 epochs with learning rate пe 0.01, aiming to reduce the computation cost; for the other methods, we set local learning rate 0.01.
For FedCCFA, we turn the scaling factor according to the results in Figure we turn the maximum distance E {0.05, 0.10, 0.15, 0.20J, and set it to 0.10 according to the results in Figure 6
(5.0, 10.0, 20.0, 50.0, 100.0). and set it to 20.0
For IFCA, we the number of clusters k 4, which is the oracle knowledge of the ground-truth clustering. For the following baselines, we tune their hyperparameters on CIFAR-10 under no concept drift and (0.(0.5), and select the best hyperparameters for all settings:
FedDrift: we turn the detection threshold E (.21, 0.5, 1.0, 5.0), and set to 0.5.
FedFM: we turn the penalty coefficient À E (0.1, 1.0, 5.0, 10.0, 50.0), and set it to 1.0.
FedPAC: turn the penalty coefficient (0.1,0.5, 10.0, and it to 1.0.
Flash: we turn the server learning rate 1/2 E 0,0,000,.001,002,.0.011
and set to 0.01.
SCAFFOLD: we turn the server learning rate 7/ E 0.100,0,.01,0000,0001,,
and set it to 1.0.
PFedMe: turn K (1,5,7) 7), and set it to 5; we turn À (0.1, 0.5, 1.0, 1.0, 5.0, 10.0, 15.0). and set it to 1.0.
Ditto: we turn penalty coefficient À (0.1, 0.5, 1.0, 5.0), and set it to 5.0.
80
80
70
70
Accuracy (%)
Accuracy (%)
60
60
Dir(0.01)
50
Dir(0.1)
50
FedCCFA
Dir(0.5)
oneele
40
40
5
20
100
0.1
Y
Figure 5: Ablation study on scaling factor 7.
Figure 6: Ablation study on maximum distance in DBSCAN, 0.1 an optimal selection.
7 20 an optimal selection.
16

---

C Additional experimental results
C.1 Performance under Dir(0.1)
We evaluate all methods under no drift setting and Dir(0.1). 6 the generalized accuracy under (ir(0.1). We observe that, on CIFAR-10 and CINIC-10 with 20% participation, FedCCFA outperforms baselines. Besides, FedCCFA outperforms all decoupled methods and personalized FL methods, showing that FedCCFA boosts the generalization performance of classifiers under this setting. We find that gradient explosion occurs when adopting FedPAC and FedRep on CINIC-10 with full participation. Under this setting, data heterogeneity much severe (as shown Figure 3 and classifier is extremely biased. Such classifier could lead to gradient explosion when using high classifier learning rate, so the accuracy of FedRep may drop to 10% and FedPAC fails to finish FL training due to abnormal inputs its classifier collaboration. Since FedBABU uses the initial global classifier during training and FedCCFA uses clustered classifiers, this problem is alleviated.
Table 6: Generalized accuracy under Dir(0.1). The sample ratio 100% and 20% for 20 clients and 100 clients, respectively. All results are averaged over 3 runs (mean std). FedPAC fails to finish FL training due to abnormal inputs to its classifier collaboration.
Pashion-MNIS,
CIFAR-I0
CINIC-10
Method 20 clients
100 clients
20 clients
clients
clients
100 clients
FedAvg
88.8910.30
89.0111..24
70.8510.40
1.09011.03
53.17:10.44
$1.84.1.099
FedProx
88.75+0.11
88.92+00.57
71.25:+0.3
67.48
52.97=0.44
0..8
SCAFFOLD
86.56+0.225
85.52+t0.06
65.03+0078
01.60:00411
36.24+3.56
1..24
FedFM
88.43:+:0.66
88.6840.18 18
0..53++0.7
68.31+1.31
53.86+0.38
50.05
AdapFedAvg
88.78-10.25
88.91110.20
71.23:10.72
66.91
53.1510.11
$1.1510.117
Flash
89.47:10.27
88.76:10.21
69.60:10.01
68.34:10.48
51.53:10,22
52.37
PFedMe
77.89:10.07
83.0740.18
51.35+0.11
50.89+0.04
34.07++0.21
39.89=10555
Ditto
87.224:0.02
88.00-L0.08
66.74.10.37
0..00
46.49-L0.35
48.05
FedRep
71.17+-0.67
16.61
44.00++0.30
45.09
16.00+110.39
+0.89+0.60
FedBABU
1..29+11.36
78.39+-0.97
42.23++0.02
48.88
26.3441.445
02.50
FedPAC
74.0910.33
84.42:10.11
44.48
55.4411.41
34.3411.64
IFCA
76.52-10.14
84.71-10.09
49.07
61.3342.18
36.33
43.40.1..39
FedDrift
88.86-10002
88.88+00.19
70.31
66.99
52.34+0.09
51.64++0.2
FedCCFA
88.2110.40
88.3211..22
70.7410.30
70.21110.48
50.1310.62
53.5010.055
C.2 Performance under sudden drift
We evaluate all methods under distributed concept drift and Dir(0.1). Table presents the generalized accuracy under sudden drift setting. We find that FedCCFA outperforms all baselines on all datasets except for CINIC-10 with full participation. This weakness is caused by wrong clustering results. Specifically, in this case, the performance of global extractor limited (the accuracy of the best method is only 53.86% under no drift setting), so the poor extractor cannot generate good feature vectors for the classifier. For this reason, the classifier cannot accurately learn the data distribution and its weights may provide wrong information. Besides, we find that gradient explosion also occurs when using FedFM on CINIC- with full participation. This is because severe hetcrogeneity will significantly increase the loss of feature alignment term when distributed concept drift occurs. FL methods without feature alignment are not affected, such as FedAvg, Flash and IFCA.
We present the generalized accuracy decrease under sudden drift setting to show the adaptability to distributed concept drift. These results show that FedCCFA can effectively adapt to distributed concept drift except for 1I with full participation (due to the poor extractor discussed above). For personalized FL. methods, thanks to their personal models or classifiers, distributed concept drift will not affect them.
17

---

Table Generalized accuracy under sudden drift and Dir(0.1). The sample ratio is 100% and 20% for 20 clients and 100 clients, respectively. All results averaged over runs (mean + std).
Fashion-MNIST
CIFAR-10
CINIC-10
Method
20 clients
100 elients
20 elients
100 clients
20 clients
100 clients
FedAvg
67.39:10.72
68.00:10.35
$4.87:10.46
53.71:10.95
42.15:10.02
37.50:1:.440
FedProx
68.43-1:.33
68. 16
55.03:10.19
53.57:10.52
42.27:10.09
38.58:10.22
SCAFFOLD
63.98:10.76
66.30+0003
18,25+1.65
41.10:10.64
27.78=0.10
34.25+0.96
FedFM
67.89=0.81
67.29:1.114
55.04:10.87
53.01:0.211
10.00:0.00
36.84:1.21
AdapFedAvg
66.91-0.774
68.07++0.32
55.11++.112
52.58800.54
0.14
36.98
Flash
03.8
68.96+2.09
54.44
54.20+0.59
41.35++0.53
41.40++0.80
PFedMe
70.69:1032
71.1710.07
44.54:1..55
41.94:10477
30.77+0.08
34.54:10.03
Ditto
72.88:10.33
73.2910.18
55.9911.41
$2.3911.14
39.6010.559
40.22:1:045
FedRep
71.49+0.07
76.79+10.37
43.99+0.27
45.81+
28.27+00.00
31.17
FedBABU
1.75
76.93+0..0
39.12+1.74
46.51#0..0
25.80+00.4
30.97++0.40
FedPAC
69.33+0055
79.96+0.42
41.35+00011
50.10+0.21
31.60=+0.51
IFCA
69.45111.20
82.9010..97
43.70.10.21
50.34
33.84.1..18
31.70.LO.30
FedDrift
63.01
10.48.11.61
38.06.13.16
46.85.11007
28.32-11.31
32.40.12.84
FedCCFA
85.22:10.08
87.63:10.15
65.23:10.48
66.34:10.96
40.94+0.11
46.02:10.42
Table 8: Generalized accuracy decrease under sudden drift and Dir(0.5). All results are averaged
over 3 runs.
Sashion-MNIST"
CIFAR-10
CINIC-10
Method
20 clients
100 clients
clients
clients
clients
100 clients
FedAvg
-22.76
-22.07
-17.6
-15.9
-12.54
-12.1
FedProx
-21.04
-21.75
-17.24
-15.82
-12.72
-12.29
SCAFFOLD
-20.41
-18.06
-17.17
-13.14
-12.58
-9.75
FedEM
-21.55
-21.08
-18.36
-17.38
-12.71
-12.43
AdapFedAvg
-21.12
-21.33
-17.36
-15.68
-12.1
-12.46
Flash
-80.19
-18.78
-18.17
-14.64
-11.67
-11.02
PFedMe
-2.69
-7.15
-4.66
-7.34
-2.1
-4.04
Ditto
-11.82
-10.11
-9,58
-8.91
-8.24
-7.93
FedRep
0.29
0.68
0.43
0.37
-0.33
FedBABU
3
0.99
-0.12
-0.13
0.5
-0.94
FedPAC
-1.01
-0.37
-1.49
-3.04
0.55
-1.28
IFCA
0.03
-0.36
-2.66
-15.22
-4.39
-12.41
FedDrift
-11.43
-2.58
-30.42
-16.44
-27.89
-12.88
FedCCFA
-0.31
-0.35
-1.43
-1.23
-10.01
-6.08
18

---

C.3 Performance under incremental drift
For all methods, we validate the adaptability incremental drift, where three different concept drifts occur at round 100, 10 and 120, respectively. Table shows that, under Dir(0.5), FedCCFA can also adapt this drift setting and outperforms all baselines.
Table 9: Generalized accuracy under incremental drift and Dir(0.5). The sample ratio is 100% and 20% for 20 clients and 100 clients, respectively. All results are averaged over runs (mean std).
Beshion-MNIST
CIFAR-10
CINIC-10
Method
20 clients
100 clients
20 clients
100 clients
20 clients
100 clients
FedAvg
6..47:1.113
68.15
6..31+00.00
08.1744049
49.77+0.33
46.33-+0.32
FedProx
70.04++0.70
69.06++0.00
61 01.0
07.77=0043
50.04+0.12
46.42=+0.43
SCAFFOLD
68.18+00.38
69.15++0.44
58.14+1.00
45.39++0.46
46.53+0.28
38.06=00.33
FedFM
68.54.10.38
68.86110.51
61.02.10.05
57.68.10.48
50.86.10.10
46.52.10.36
AdapFodAvg
68.75=0049
68. 68.86+0.39
60.74=0.10
57.93
49.92:10.22
46,00
Flash
67.82+0.447
70.41
60.61
60.66:10222
49.38++0.06
49.77
PFedMe
82.38:10.41
77.18:10.20
58.5311043
44.01:10.55
30.53:114.22
37.84.10.81
Ditto
77.90:10.69
79.56:10.21
67.57+0.225
63.02-10.41
50.85+10.3
48.61
FedRep
82.10:10.14
80.85+L0.17
63.89+1..00
$1.87-0..9
43.80:10.47
38.53-10.30
FedBABU
1..8
81.54:10.46
58.6310.36
54.98
41.0610.87
40.04
FedPAC
84.1010..36
86.66+10.31
67.2110.60
64.78
44.07=0.166
46.24410.25
IFCA
06.3
88.07++0.29
59.74+4.33
57.63+00.23
45.08+2.60
45.51+00.66
FedDrift
79.62+22.15
87.36+40.31
46.05++0.07
56.3242.33
33.63++0.93
45.11+00.5
Ours
89,4911.110
88.9610..22
76.06:10.39
03.42:10.66
52.1510.34
53.02:10.46
C.4
Performance under reoccurring drift
To verify whether all methods could recover the same accuracies once the original data distributions come back, we conduct experiments under the reoccurring drift setting. Table 10 shows that all methods can achieve this goal under reoccurring drift and Dir(0.5).
Table 10: Generalized accuracy under reoccurring drift and Dir(0.5). The sample ratio is 100% and 20% for 20 clients and 100 clients, respectively. All results are averaged over runs (mean std).
MNSST
CIFAR-10
CINIC-10
Method
20 clients
100 clients
20 clients
100 clients
20 clients
100 elients
FedAvg
90.38-10.14
90.02-0.16
78.37-10.56
73.56-10.28
62.16.10.18
58.66-10.04
FedProx
90.48:10.08
90.15-0.20
78.40-10.33
74.10-10.64
62.40-10.16
58.45-10.52
SCAFFOLD
90.22-10.16
87336=0.36
75.23+40.06
57.26-10.95
58.86:10.12
46.64=1..50
FedFM
90.37=1.006
90.21=0.14
79.25:00.25
73.74=0023
62.56:10.46
58.66=0.57
AdapFedAvg
90.49++0.03
90.08=0.13
78.36
73.24+0.37
62.05
58.72-+0.15
Flash
90.04+0..13
89.85-0.25
77.59+0.60
75.69
12.1220.32
59.72-+0.46
PFedMe
85.43.10.15
84.77-0.24
62.35:10.08
51.11=0.144
32.61+15.99
41.34=
Ditto
90.26:10.02
89.93-0.16
77.3110.03
72.27-0.68
58.6311043
55.83-0.01
FedRep
33.70++41.05
81.29-0.26
64.09+4.37
52.71=0045
43.6440.56
38.60=0.30
FedBABU
80.85+1.02
81.41=0.88
60.11+00.23
55.49+0.700
41.4440.43
41.33+0.52
FedPAC
85.26+0.21
87.50=0.36
67.45++0.2
65.89-0065
44.92+0.11
47.28=0.57
IFCA
86.96110.16
0.76
69.82.10.57
73.43
$1.3512.19
57.90
FedDrift
88.2911.722
89.43-0.47
68.9111.81
73.21
36.9110.08
56.92-10.44
Ours
90.31+0042
89.67=0.03
78.2440.49
73.70=0.43
60.86+0.27
59.18=0.25
19

---

C.5 Visualization of distance matrix
To demonstrate the effectiveness of our classifier clustering, we conduct experiments on CIFAR-I0 under sudden drift setting with full participation, and then visualize the distance matrices (i.e., the input of DBSCAN algorithm) in Figure 7 The left shows the distance matrix of class 1. According to the concept drift setting in Section 5.1 for class 1, clients {0, 1, 10, 11, 12) should be grouped and the others should be grouped. The right shows the distance matrix of class 5. For this class, clients {6, 7,8, 9, 16, 17, 18, 19} should be grouped and the others should be grouped. These two matrices demonstrate that our clustering method can effectively measure the distance between each client.
Chonts
Chees
Figure 7: The visualization of distance matrices at round 199 under sudden drift setting.
C.6 Ablation study on aggregation method
FedCCFA involves class-level classifier aggregation (Equation 5 and class-level feature anchor aggregation (Equation 77 An intuitive manner is sample-number-basee weighted aggregation, which is similar to the model aggregation in FedAvg. However, this manner requires clients to upload the sample number for each class, which may leak privacy about clients' category distributions. To address this problem. we consider the uniform aggregation, where all clients share the same aggregation weight. This aggregation manner does not involve other private information. We compare the two manners of aggregation. As shown in Table [1] we observed that uniform aggregation even performs better, especially under concept drift setting. This may be because that uniform aggregation can prevent the classifier from overfitting to the classifier that is significantly biased due to data heterogeneity.
Table II: Ablation study on aggregation method, We apply partial participation under (0..0.
Fashion-MNIST
CIFAR-10
CINIC-10
no drift (weighted)
89,39-0.16
73.06:10.69
57,41
no drift (uniform)
8.174
74.44
58.7010.33
sudden drift (weighted)
19.1 8.13
66.49 73.21
45.60
sudden drift (uniform)
52.62
C.7 FedDrift under data heterogeneity and distributed concept drift
FedDrift [19 is the first solution for distributed concept drift, which creates new global models based on drift detection and merges models by hierarchical clustering. However, data heterogeneity may disturb model merging, because cluster distance could exceed the threshold under data heterogeneity. To validate this conjecture, we present the accuracy curves and the number of used global models during training process. The latter denotes the number of global models used by all clients each round. We conduct experiments under the full participation setting.
20

---

We first present the accuracy curves and the number of used global models under homogeneous setting in Figure We see that FedDrift can creates new models under various drift settings and accurately merges the models under the same concepts (i.e., the same conditional distributions P())))) However, since new global models are created at drift rounds and these models need be retrained, FedDrift cannot rapidly reach steady state.
Test Accuracy
Global models
90
18
Accuracy (%)
Model number
16
14
12
85
10
80
0
50
100
150
200
50
100
150
200
Number of Rounds
Number of Rounds
no drift
sudden
incremental
reoccurring
Figure 8: FedDrift on homogeneous Fashion-MNIST
Then, present the two indices under Dir(0.1) in Figure We find that FedDrift can creates new model at drift rounds, but it cannot accurately merge the models under the same concepts. This is because, in addition different concepts, different label distributions can also increase the cluster distances estimated by the loss value, which will prevent model merging. Since all clients choose the only global model at the beginning of training process and the loss delta will not exceed the threshold if no drift occurs, new global models will not be created under the no drift setting.
Test Accuracy
Global models
20
18
16
Accuracy (%)
Model number
14
12
10
60
40
0
50
100
150
200
50
100
150
200
Number of Rounds
Number of Rounds
no drift
sudden
incremental
reoccurring
Figure 9: FedDrift on heterogeneous MaShiOn-MNISTT.
C.8 Computation cost comparison
To enhance client collaboration, FedPAC 43 introduces classifier combination method, which uses feature statistics to estimate optimal combining weights. However, according to our experimental results, this operation is time-consuming. We compare the computation cost of classifier collaboration between FedCCFA and FedPAC. We count the time consumption for classifier collaboration after sudden drift occurs, and report the average value of 10 rounds. Table shows that FedPAC is more time-consuming than FedCCFA, especially large datasets (e.g., CINIC-10).
21

---

Table 12: Time consumption (seconds) for classifier collaboration. All reults are averaged over 10 rounds.
Fashion-MNIST
CIFAR-10
CINIC-10
FedPAC FedCCFA
1.35 0.49
1.67
2.43
0.71
0.70
C.9 Personalized performance
Though this work does not focus on personalized FL, we study whether more generalized model could bring benefits to personalization. We distribute test samples by the same Dirichlet distribution used for training data partition, and evaluate the personalized accuracy over each client's 's personal model (or global extractor followed by personal classifier). After FL training. we fine-tune the classifier with epochs and classifier learning rate 0.01. Table 13 shows that FedCCFA with classifier fine-tuning is comparable to the most performing personalized methods.
Table 13: Personalized accuracy under no drift setting. FedCCFA-FT denotes FedCCFA with classifier fine-tuning. We apply 20% participation with 100 clients and all experiments are run on CIFAR-10 and results are averaged over runs.
pFedMe
Ditto
FedRep
FedBABU
FedPAC
FedCCFA
FedCCFA-FT
(01(
65.94
78.84
74.26
74.46
80.24 79.47
71.99 75.03
80.52
(0.5
60.42
79.20
68.85
71.29
79.13
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of efficiency and feature alignment in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

We empirically verify the effectiveness of our method in Section 5.2 and Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We present the details our method Section We provide the details of experimental settings and hyperparameter selection in Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have released our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/pulli/guuddee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details of experimental settings and hyperparameter selection in Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We explain how we calculate the error bars Section 5.1 and the results are averaged over runs (mean + std).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources and deep learning framework in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://:///rrrrsssccpptcltt///// http://s....cppoc/ccttcccccuceaa
Answer: [Yes]
Justification:
The research conducted this paper conform with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This work focuses distributed concept drift in federated learning. Users' personal data is usually protected in federated learning and we do not find any possible adverse effects society caused by this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers that produced the datasets, including Fashion- MNIST, CIFAR-10 and CINIC-10.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "swePlSEHaX", "reviewer": "Reviewer_U5Wz", "summary": "The paper introduces a solutions based on clustering to group classifiers in the federated learning setting. Such an approach is meant to deal with concept drift in data so as to track and adapt the evolutions of the classifiers of the clients in federated learning. On of the core points is to align features and this is carried out by ether adaptive feature alignment procedure suggested in the paper.", "questions": "- What's the need to partition the models into feature extractor and classifier? \n- Which is the assumption guaranteeing the clustering of classifier parameters in the parameter space?\n- What's the reason of the reduction in accuracy in no-change condition (i.e., Table 1) and how to mitigate this point?", "limitations": "see previous box", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea of letting the global/local models evolve over time is interesting and relevant\n- The clustering is based on the classifier space (the feature extraction phase is fixed)\n- Results in the concept drift are promising", "weaknesses": "Some technical points of the paper are not adequately addressed such as:\n- the partitioning of feature extractor + classifier is relevant and requires to share a common feature extractor. It is not clear to this reviewer whether the feature extractor is fixed or adapted over time\n- clustering of models requires to assume that the distribution of the model parameters shares some \"locality\" (e.g., this holds for linear models but not in general for NNs). \n- performance in stationary-conditions are less relevant. This is a crucial point (e.g., not suffered by other solutions) that requires to be further investigated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and kind words to our work. Below we address specific questions:\n\n> W1: the partitioning of feature extractor + classifier is relevant and requires to share a common feature extractor. It is not clear to this reviewer whether the feature extractor is fixed or adapted over time.\n\nA1: We will clarify this part of our manuscript. As we mentioned in Section 4.3 (FedCCFA), the feature extractor is trained locally by each selected client (line 9 in Algorithm 1) and the server aggregates these clients' local extractors (line 15 in Algorithm 1). The aggregated extractor is broadcasted to the clients at the next round (line 4 in Algorithm 1). The training and aggregation of feature extractors can further enhance generalization performance. In particular, to mitigate the impact of concept drift on the model training, local classifier training (line 8) is conducted before local extractor training, which can effectively adapt to real drift (i.e., the conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ changes).\n\n> W2: clustering of models requires to assume that the distribution of the model parameters shares some \"locality\" (e.g., this holds for linear models but not in general for NNs).\n\nA2: We believe there may have been a misunderstanding regarding our clustering method. Our clustering method is also suitable for NNs. Specifically, in our FedCCFA framework and other decoupled methods (e.g., FedBABU [1] and FedPAC [2]), the classifier is the last fully-connected layer (i.e., a simple linear classifier) and the feature extractor is composed of the remaining layers. The extractor is shared across all clients and maps the input to a low-dimensional feature vector. Once concept drift occurs (i.e., the conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ changes), different linear classifiers will be learned and the parameters of these classifiers reflect corresponding $P(\\mathcal{Y}|\\mathcal{X})$. Finally, the server separates these classifiers for each class (referred to as class classifier) and measures the class-level distances between each client's class classifier, which can group the clients according to their $P(\\mathcal{Y}|\\mathcal{X})$. The class classifier is still a linear model that outputs the logit for the specific class. Therefore, our method is also suitable for NNs, as long as the last fully-connected layer is used as the local classifier. Our experiments conducted on two CNNs also demonstrate this feasibility. To avoid further confusion, we will clarify this section of our manuscript.\n\n> W3: performance in stationary-conditions are less relevant. This is a crucial point (e.g., not suffered by other solutions) that requires to be further investigated.\n\nA3: The lower accuracy under no drift setting is attributed to decoupled training (i.e., training the classifier first and then training the feature extractor). Specifically, for decoupled methods (e.g., FedPAC [2], FedRep [3] and our FedCCFA), training the classifier first might cause it to fit the initial features, resulting in gradient attenuation during backpropagation. This will restrict the subsequent extractor's training process and prevent it from optimally learning features. In contrast, when training the entire model together, all layers can simultaneously adapt to the training data, allowing them to synergize effectively. However, in the context of concept drift, decoupled training is crucial because the classifier can effectively adapt to new conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ and small gradients will be back-propagated to the extractor. This can stabilize the training of the feature extractor. Besides, the results in Table 1 show that our method still outperforms the majority of baseline methods.\n\n> Q1: What's the need to partition the models into feature extractor and classifier?\n\nA4: In this paper, we focus on real drift, where the conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ changes. Since the marginal distribution $P(\\mathcal{X})$ is invariant, the representation learning should not be affected by this drift. That is, the feature extractor can be shared across clients to enhance generalization performance. The key to adapt to this drift is the classifier, as it map the representation to the label space (i.e., learning the distribution $P(\\mathcal{Y}|\\mathcal{X})$). Therefore, decoupling the model into a feature extractor and a classifier can effectively adapt to concept drift while enhancing the generalization performance. The superiority of shared feature extractor can be demonstrated by the better performance than other clustered FL methods under various drift settings.\n\n> Q2: Which is the assumption guaranteeing the clustering of classifier parameters in the parameter space?\n\nA5: Since the classifier is the last fully-connected layer (i.e., a simple linear classifier), the clustering of its parameters makes sense. Please refer to A2 for more details.\n\n> Q3: What's the reason of the reduction in accuracy in no-change condition (i.e., Table 1) and how to mitigate this point?\n\nA6: Please refer to A3 for the reason of this reduction. To mitigate this point, it can be useful to fine-tune the entire model after training the extractor.\n\n\n**References:**\n\n[1] Oh, J., Kim, S., & Yun, S. Y. (2022). FedBABU: Towards enhanced representation for federated image classification. International Conference on Learning Representations.\n\n[2] Xu, J., Tong, X., & Huang, S. L. (2023). Personalized federated learning with feature alignment and classifier collaboration. International Conference on Learning Representations.\n\n[3] Collins, L., Hassani, H., Mokhtari, A., & Shakkottai, S. (2021). Exploiting shared representations for personalized federated learning. In International conference on machine learning (pp. 2089-2099). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "72FPRb44TY", "reviewer": "Reviewer_HDwb", "summary": "This paper proposes to consider two cross-coupled important issues, i.e., federated learning and concept drift. To address such a challenging problem, the FedCCFA framework has been proposed composed of classifier clustering and feature alignment modules. The former is designed to cope with concept drift and enhance the generalization performance of the model. The latter is utilized to prevent data heterogeneity of clients. The overall design is sound, interesting, and intuitive. The whole paper is well-organized and well-written. The experimental evaluation is comprehensive and convincing.", "questions": "See the weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A relatively new and important problem of federated learning under concept drift is addressed.\n- Contributions of this work are in the framework perspective and the module/strategy design perspective.\n- Both classifier clustering and feature alignment are sound and interesting ideas.\n- The whole paper is well-organized.\n- Comprehensive evaluation comparing many counterparts (including SOTAs) on benchmark datasets.", "weaknesses": "- Lack of a framework pipeline to intuitively explain the overall design.\n- Time complexity analysis is missing.\n- Visual classification results on the datasets are preferable.\n- The source code is not opened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and kind words to our work. Below we address specific questions:\n\n> W1: Lack of a framework pipeline to intuitively explain the overall design.\n\nA1: Thank you for your valuable feedback. Please see the attached PDF in the top-level comment for our framework pipeline. We will incorporate this pipeline into our final version.\n\n> W2: Time complexity analysis is missing.\n\nA2: Thank you for your constructive comment. We conducted a comprehensive time complexity analysis at client side and compared our FedCCFA to other methods, including FedFM [1], FedRep [2] and FedPAC [3]. $E$ and $B$ denotes the epoch and batch num, respectively. In our FedCCFA, we train a balanced classifier for $s$ iteration with a batch of data. $m$, $f$ and $c$ denotes the computation cost for training local model, extractor and classifier, respectively. Note that, to reduce computation cost, $s$ is small.\n\n| Method | Complexity |\n| --------- | ------------- |\n| FedFM | O(E\\*B\\*m) |\n| FedRep | O(E\\*B\\*f+B\\*c) |\n| FedPAC | O(E\\*B\\*f+B\\*c) |\n| FedCCFA | O(E\\*B\\*f+B\\*c+s\\*c) |\n\nIn particular, at server side, our classifier collaboration method is more efficient than that of FedPAC, which can be demonstrated by the results in Appendix B.6.\n\n> W3: Visual classification results on the datasets are preferable.\n\nA3: Thank you for your valuable feedback. Please see the attached PDF in the top-level comment for the visualization of distance matrix. We think it can be helpful to understand our clustering method.\n\n> W4: The source code is not opened.\n\nA4: Thank you for your valuable feedback. According to the FAQ of NeurIPS 2024, it is not allowed to add any link in any part of the rebuttal. We will provide our code once this paper is accepted.\n\n\n**References:**\n\n[1] Ye, R., Ni, Z., Xu, C., Wang, J., Chen, S., & Eldar, Y. C. (2023). Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. IEEE Transactions on Signal Processing.\n\n[2] Collins, L., Hassani, H., Mokhtari, A., & Shakkottai, S. (2021). Exploiting shared representations for personalized federated learning. In International conference on machine learning (pp. 2089-2099). PMLR.\n\n[3] Xu, J., Tong, X., & Huang, S. L. Personalized Federated Learning with Feature Alignment and Classifier Collaboration. In The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4STJ69E39", "reviewer": "Reviewer_erp1", "summary": "The paper proposes a federated learning framework called FedCCFA, which addresses the challenges posed by distributed concept drift and data heterogeneity. The authors introduce innovative solutions such as classifier clustering and adaptive feature alignment to enhance collaboration and improve model performance under various concept drift settings. The authors demonstrate the effectiveness of their approach through experimental results showing significant improvements over existing methods.", "questions": "1. In the experiments, which classifiers were ultimately clustered in the class-level clustering? What does the final distance matrix look like?\n2. Equation (5) uses uniform aggregation to avoid privacy concerns. Why does this method avoid privacy concerns? Could taking the average directly lead to performance loss in classifiers?", "limitations": "The authors point out that FedCCFA requires training balanced classifiers, which will increase computational complexity to some extent.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles an important and relatively unexplored problem in federated learning, namely concept drift and data heterogeneity.\n2. The extensive experimental results demonstrate the effectiveness of FedCCFA, showing significant performance improvements over existing methods in various concept drift scenarios.", "weaknesses": "1. I am unable to fully understand the motivation of the paper. Figure 1 does not clearly illustrate how Decoupled Clustering addresses the real drift problem compared to previous decoupled methods. While it shows better performance than FedAvg and pure Decoupled methods, the underlying reasons remain unclear. Why does introducing clustering improve performance?\n2. The writing of the paper can be improved. For example, the specific meaning of \\(\\phi_{i,c}\\) is not explained, leading to significant confusion in understanding equation (4). The authors need to explain each symbol to enhance the readability of the paper. Additionally, the term \"balanced batch\" in line 160 needs to be explained, specifically how it was sampled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and kind words to our work. Below we address specific questions:\n\n> W1: I am unable to fully understand the motivation of the paper. Figure 1 does not clearly illustrate how Decoupled Clustering [...] the underlying reasons remain unclear. Why does introducing clustering improve performance?\n\nA1: Thank you for your constructive feedback. We will clarify this part of our manuscript. When real drift occurs, decoupled methods can adapt to this drift by training the classifier while fixing the extractor. Each local classifier learns corresponding conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$. However, some clients may share similar $P(\\mathcal{Y}|\\mathcal{X})$, and pure decoupled methods neglect the fine-grained collaboration between local classifiers. This will make each client's local classifier overfit to its local data. To address this problem, we propose a novel class-level classifier clustering method, which aggregates the class classifiers within the same cluster. This aggregation reduces the bias introduced by any single client's data, contributing to improved generalization performance. That is, individual local classifiers trained on local data exhibit biased parameter estimation, while the aggregation of multiple classifiers mitigates these biases and improves the overall parameter estimation.\n\n> W2: The writing of the paper can be improved. For example, the specific meaning of $\\boldsymbol{\\phi}\\_{i,c}$ is not explained, [...] Additionally, the term \"balanced batch\" in line 160 needs to be explained, specifically how it was sampled.\n\nA2: Thank you for your valuable feedback, and we will incorporate the suggestions into our updated draft.\n\n1. The meaning of $\\{\\hat{\\boldsymbol{\\phi}}\\_{k, c}^{(t)}\\}$ : At round $t$, the server separates each client $k$'s balanced classifier $\\hat{\\boldsymbol{\\phi}}\\_k^{(t)}$ for each class $c$ ( referred to as class classifier $\\hat{\\boldsymbol{\\phi}}\\_{k, c}^{(t)}$ ). Then, for each class $c$, the server measures the class-level distances $\\mathcal{D}\\_c(i, j)$ between clients $i$ and $j$ by their class classifiers $\\hat{\\boldsymbol{\\phi}}\\_{i, c}^{(t)}$ and $\\hat{\\boldsymbol{\\phi}}\\_{j, c}^{(t)}$.\n\n2. The meaning of \"balanced batch\": To prevent class imbalance from misleading classifier clustering, FedCCFA uses a balanced batch $b\\_k^{(t)}$ to train a balanced classifier $\\hat{\\boldsymbol{\\phi}}\\_k^{(t)}$ for distance estimation. To reduce computation cost, we randomly and uniformly select 5 samples for each class $c \\in [C]$, i.e., the balanced batch size $|b\\_k^{(t)}|$ is $5 \\* C$.\n\n> Q1: In the experiments, which classifiers were ultimately clustered in the class-level clustering? What does the final distance matrix look like?\n\nA3: In our experiments, for each class $c \\in [C]$, all selected clients' class classifiers $\\{\\hat{\\boldsymbol{\\phi}}\\_{k, c}^{(t)}\\}$ are used for class-level clustering, where $k \\in \\mathcal{I}^{(t)}$. That is, for each class $c$, the server computes a $|\\mathcal{I}^{(t)}| \\* |\\mathcal{I}^{(t)}|$ distance matrix $\\mathcal{D}\\_c$ to measure the distance between each client's class classifier. Please see the attached PDF in the top-level comment for the visualization of distance matrix.\n\n> Q2: Equation (5) uses uniform aggregation to avoid privacy concerns. Why does this method avoid privacy concerns? Could taking the average directly lead to performance loss in classifiers?\n\nA4: FedCCFA involves class-level classifier aggregation (Eq. (5)) and class-level feature anchor aggregation (Eq. (7)). An intuitive manner is sample-number-based weighted aggregation, which is similar to the model aggregation in FedAvg. However, this manner requires clients to upload the sample number for each class, which may leak privacy about clients' category distributions. To address this problem, we consider the uniform aggregation, where all clients share the same aggregation weight. This aggregation manner does not involve other private information.\nWe compare the two manners of aggregation. Interestingly, we observed that uniform aggregation even performs better, especially under concept drift setting. This may be because that uniform aggregation can prevent the classifier from overfitting to the classifier that is significantly biased due to data heterogeneity.\n\n|  | Fashion-MNIST | CIFAR10 | CINIC-10 |\n| ------------- | -------- | -------- | ----|\n| no drift (weighted) | 89.39$\\pm$0.16 | 73.06$\\pm$0.69 | 57.41$\\pm$0.12 |\n| no drift (uniform) | **89.74$\\pm$0.18** | **74.44$\\pm$0.26** | **58.70$\\pm$0.33** |\n| sudden drift (weighted) | 89.14$\\pm$0.31 | 66.49$\\pm$1.21 | 45.60$\\pm$0.38 |\n| sudden drift (uniform) | **89.39$\\pm$0.14** | **73.21$\\pm$0.82** | **52.62$\\pm$0.51** |\n\nThank you for your valuable comments. We will incorporate these results into our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bj6w5MvEBR", "reviewer": "Reviewer_EQ7w", "summary": "This paper explores the impact of distributed concept drift on federated learning (FL), and proposes a novel FL framework, FedCCFA, to adapt to distributed concept drift with data heterogeneity (i.e., label distribution shift). Extensive experiments demonstrated the effectiveness and generality of the proposed method.", "questions": "discussed in weaknesses.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Federated learning (FL) under concept drift holds significant academic and practical value. The method proposed in this paper is both reasonable and effective.\n2. The experiments are extensive, not only comparing the proposed method with SOTA methods but also simulating various types of drift. This demonstrates the method's generality across different drift scenarios.\n3. The paper is well-organized and well-written, enhancing the clarity and impact of its findings.", "weaknesses": "1. In the introduction, the authors use the distribution of medical images as an example to describe data heterogeneity. However, the actual task addressed in the paper is label distribution shift, which does not align with the previous example. It is recommended that the authors directly use label distribution shift to describe the problem addressed in the paper\n2. The authors calculate the global feature anchor by averaging (i.e., Eq. (7)). However, given the presence of distributed concept drift among different clients, simply averaging may be unreasonable. It raises the question of whether concept drift adaptation should be considered.\n3. The adaptive alignment weight is calculated based on the entropy of the label distribution. However, there lacks the relevant motivation, references, and theoretical justification for this strategy.\n4. Federated learning (FL) under distributed concept drift primarily addresses concept drift across clients and over time. This is similar to the research problem of multistream classification under concept drift [1-3]. It is recommended that the authors discuss these works to enhance readers' understanding of related research.\n    [1] An adaptive framework for multistream classification, CIKM 2016 \n    [2] Fusion: An online method for multistream classification, CIKM 2017\n    [3] Online boosting adaptive learning under concept drift for multistream classification, AAAI 2024\n5. The paper does not provide open-source code and datasets, which compromises the reproducibility of the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. Below we address specific questions.\n\n> W1: In the introduction, the authors use the distribution of medical images [...] It is recommended that the authors directly use label distribution shift to describe the problem addressed in the paper.\n\nA1: We believe there may have been a misunderstanding regarding the primary problem our research tackles. To clarify, the primary problem our paper addresses is distributed concept drift with data heterogeneity, not just label distribution shift. We guess that the \"label distribution shift\" you mentioned is \"data heterogeneity\" in our paper, where the marginal distribution $P(\\mathcal{Y})$ varies across clients. In our paper, we consider a new problem named distributed concept drift. Specifically, the conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ may vary across clients (e.g., diagnoses can vary among doctors) and change over time (e.g., a doctor may offer different diagnoses at different times). These two kinds of changes constitute distributed concept drift. As for data heterogeneity, we think it is a common setting for federated learning, so we conduct experiments under two Dirichlet distribution (Dir(0.1) and Dir(0.5)) to simulate data heterogeneity.\n\n> W2: The authors calculate the global feature anchor by averaging (i.e., Eq. (7)). [...] It raises the question of whether concept drift adaptation should be considered.\n\nA2: Thank you for your valuable feedback. We would like to clarify that this issue has already been addressed in our manuscript. As detailed in \"Clustered feature anchors\" in Section 4.2, given the presence of distributed concept drift, simply averaging clients' local anchors could generate incorrect global anchors, which can mislead feature alignment. To address this problem, we propose clustered feature anchors (i.e., $\\mathcal{A}\\_{m, c}^{(t)}$ in Eq. (7)), where the local anchors are averaged \"for each cluster $\\mathcal{S}\\_{m, c}^{(t)}$\". Since the conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ of clients in the same cluster is similar, the \"clustered feature anchors\" can correctly guide the feature alignment. The experimental results in Table 4 demonstrate the efficacy of our clustered feature anchors.\n\n> W3: The adaptive alignment weight is calculated based on the entropy of the label distribution. However, there lacks the relevant motivation, references, and theoretical justification for this strategy.\n\nA3: Thank you for your constructive comments. In our ablation study on alignment weight (Table 5), we observed that alignment weight should be reduced under severe heterogeneity. Then, we propose adaptive alignment weight based on the intuition that the degree of data heterogeneity can be reflected by the entropy of marginal distribution $P(\\mathcal{Y})$. This motivation can be supported by Shannon's information theory [1]. Besides, He and Garcia [2] also note that imbalanced distributions result in lower entropy because the majority class dominates, reducing the unpredictability of the outcomes. We will clarify this part of our manuscript. We acknowledge the importance of providing a thorough theoretical analysis for our adaptive alignment weight. However, our current work primarily aims to address the feature alignment under distributed concept drift. Further theoretical analysis and other adaptive method will be considered in subsequent studies.\n\n> W4: Federated learning (FL) under distributed concept drift [...] This is similar to the research problem of multistream classification under concept drift [1-3]. It is recommended that the authors discuss these works [...].\n\nA4: We appreciate your pointer to these related works [3-5]. These works focus on multistream classification, which involves two independent non-stationary data generating processes (i.e., source stream and target stream). A sampling bias may exist between the distributions represented by these two streams of data. That is, training and test data distributions are not guaranteed to be similar. Different from multistream classification, distributed concept drift focus on the changing conditional distribution $P(\\mathcal{Y}|\\mathcal{X})$ across clients and over time. For each client at any round, training and test data distributions are assumed to be similar. We will cite these works and discuss the relationship in our final version, as you suggested.\n\n> W5: The paper does not provide open-source code and datasets, which compromises the reproducibility of the experiments.\n\nA5: Thank you for your valuable feedback. According to the FAQ of NeurIPS 2024, it is not allowed to add any link in any part of the rebuttal. We will provide our code once this paper is accepted.\n\n\n**References:**\n\n[1] Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3), 379-423.\n\n[2] He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9), 1263-1284.\n\n[3] Chandra, S., Haque, A., Khan, L., & Aggarwal, C. (2016, October). An adaptive framework for multistream classification. In Proceedings of the 25th ACM international on conference on information and knowledge management (pp. 1181-1190).\n\n[4] Haque, A., Wang, Z., Chandra, S., Dong, B., Khan, L., & Hamlen, K. W. (2017, November). Fusion: An online method for multistream classification. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (pp. 919-928).\n\n[5] Yu, E., Lu, J., Zhang, B., & Zhang, G. (2024, March). Online boosting adaptive learning under concept drift for multistream classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 15, pp. 16522-16530).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMkZkXYz7E", "reviewer": "Reviewer_uegQ", "summary": "The paper addresses the significant challenge of data heterogeneity and distributed concept drift in federated learning. The authors propose a novel framework, which integrates classifier clustering and feature alignment to improve model performance and collaboration among clients facing different concept drifts.\nThe key contributions of the paper are as follows:\n1. FedCCFA Framework: The proposed framework includes a method for clustering local classifiers at the class level and generating clustered feature anchors to enhance feature alignment. This clustering helps clients with similar data distributions share classifiers, thus improving the generalization performance of the global model.\n2. Feature Alignment: The framework introduces an adaptive feature alignment technique that aligns clients' feature spaces based on the entropy of the label distribution. This method helps to alleviate the inconsistencies in feature space due to data heterogeneity.\n3. Experimental Validation: Extensive experiments demonstrate that FedCCFA significantly outperforms existing federated learning methods under various concept drift settings. The results show that FedCCFA effectively adapts to distributed concept drift and enhances generalization performance.", "questions": "1. Computational Overhead\nHave you explored alternative methods for balanced classifier training to reduce computational overhead? It’s better to consider leveraging advanced optimization techniques or lightweight pre-processing steps to achieve balanced classifier training more efficiently.\n2. Evaluation on Diverse Datasets\nDo you plan to evaluate FedCCFA on more diverse, real-world datasets reflecting practical distributed concept drift scenarios? It’s better to extend evaluation to real-world datasets would provide a comprehensive understanding of the framework's applicability and robustness.\n3. Sensitivity Analysis of Hyperparameters\nHave you conducted a thorough sensitivity analysis of key hyperparameters across different datasets and settings? A detailed sensitivity analysis could help understand the impact of these hyperparameters on performance and provide guidelines for their selection.\n4. Handling Extreme Data Heterogeneity\nWhat solutions have you considered for effectively handling extreme cases of data heterogeneity and preventing gradient explosions? It’s better to investigate adaptive methods that dynamically adjust based on real-time training stability monitoring to handle extreme data heterogeneity more robustly", "limitations": "1. Adaptability to Various Drift Patterns: The paper evaluates performance under specific concept drift scenarios but may not cover all possible drift patterns. Expanding the evaluation to include a wider variety of drift patterns would provide a more comprehensive understanding of the framework’s adaptability.\n2. Real-World Implementation: The paper demonstrates effectiveness in controlled experimental settings. Discussing potential challenges and solutions for deploying FedCCFA in real-world environments, including scalability and communication efficiency, would strengthen the paper.\n3. Detailed Analysis of Classifier Clustering: The paper proposes class-level classifier clustering but provides limited analysis on the clustering's dynamics and potential pitfalls. A deeper analysis of how clustering decisions impact overall model performance and stability would be beneficial.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Comprehensive Validation: The research is supported by well-designed experiments and thorough ablation studies, demonstrating the effectiveness of the FedCCFA framework. The mathematical formulations and algorithms are clearly presented, and the results are statistically significant.\n2. Clear Presentation: The paper is clearly written and well-structured, making complex concepts accessible. Visual aids, such as graphs and tables, effectively support the explanations. Detailed breakdowns of the experimental setup enhance the reproducibility of the results.\n3. Significant Impact: This work addresses a critical gap in federated learning, with potential applications in diverse fields like healthcare, finance, and mobile device collaboration. By improving the adaptability and generalization performance of federated models, the proposed FedCCFA framework provides a robust foundation for future studies to build upon, advancing the current state of federated learning research.", "weaknesses": "1. Computation Overhead: The proposed FedCCFA framework involves additional computational steps to train balanced classifiers, which increases the overall computational cost. While the authors attempt to mitigate this by setting small iterations and batch sizes, exploring more efficient methods to achieve balanced classifier training would be beneficial.\n2. Limited Evaluation Scenarios: The experiments are primarily conducted on standard datasets. Including more diverse datasets, especially those with real-world distributed concept drift scenarios, would strengthen the validation of the framework's general applicability and robustness.\n3. Sensitivity to Hyperparameters: The effectiveness of FedCCFA relies on several hyperparameters, such as the maximum distance in DBSCAN and the scaling factor. While the authors provide some tuning, a more thorough analysis of the sensitivity and robustness to these hyperparameters across different datasets and settings would be valuable.\n4. Handling Extreme Data Heterogeneity: The paper addresses data heterogeneity, but extreme cases of data heterogeneity can still pose challenges, as noted with gradient explosions in some scenarios. Further discussion and potential solutions to handle such extreme cases more effectively would improve the robustness of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and thoughtful comments. Below we address specific questions:\n\n> W1 & Q1: Computation Overhead: The proposed FedCCFA framework involves additional computational steps to train balanced classifiers [...] exploring more efficient methods to achieve balanced classifier training would be beneficial.\n\nA1: Thank you for your insightful comments. We appreciate your suggestion to consider more efficient alternatives. In our current work, balanced classifier is trained to prevent classifier bias from misleading client clustering. It is preferable to address classifier bias during local training, such as CCVR [1], FedETF [2] and FedBR [3]. However, (1) CCVR requires virtual representations sampled from an approximated gaussian mixture model and re-trains classifier using these representations; (2) FedETF utilizes a fixed ETF classifier to address classifier bias, which requires finetuning and is not suitable for concept drift setting; (3) FedBR generates pseudo-data to reduce bias in local classifiers, which also requires additional computation cost. We agree that future work could explore more efficient methods, and we plan to investigate this in subsequent studies. For example, some global information (e.g., global feature anchors) can be used to calibrate the local classifier.\n\n> W2 & Q2: Limited Evaluation Scenarios: The experiments are primarily conducted on standard datasets. Including more diverse datasets [...] would strengthen the validation of the framework's general applicability and robustness.\n\nA2: Thank you for your constructive comments. We agree that testing on real-world data is crucial for demonstrating the practical applicability of our method. Unfortunately, there are currently no datasets that fit the specific requirements of our study, since publicly available real-world datasets usually assume that $P(\\mathcal{Y}|\\mathcal{X})$ is invariant. Therefore, existing works focusing on concept drift in federated learning (e.g., FedDrift [4] and Flash [5]) use synthetic datasets. Note that FMoW dataset used in FedDrift does not contain the changes in $P(\\mathcal{Y}|\\mathcal{X})$, as described in Section 2.1 of the original paper.\n\n> W3 & Q3: Sensitivity to Hyperparameters: The effectiveness of FedCCFA relies on several hyperparameters, [...] a more thorough analysis of the sensitivity and robustness to these hyperparameters across different datasets and settings would be valuable.\n\nA3: Thank you for your valuable comments. We have conducted additional experiments to thoroughly analyze the sensitivity of key hyperparameters across different datasets and concept drift settings. Experimental results show that $\\gamma$ ranging from 10 to 50 tends to perform better, and $\\epsilon \\le 0.1$ is better.\n\n| | $\\gamma$ = 5 | 10 | 20 | 50 | 100 |\n|---|---|---|---|---|---|\n|Fashion-MNIST (no drift) | 89.84$\\pm$0.11 | 89.63$\\pm$0.23 | 89.74$\\pm$0.18 | 89.70$\\pm$0.19 | 89.37$\\pm$0.07 |\n|Fashion-MNIST (sudden drift) | 89.73$\\pm$0.32 | 89.00$\\pm$0.19 | 89.39$\\pm$0.14 | 89.01$\\pm$0.17 | 89.15$\\pm$0.22 |\n|CIFAR-10 (no drift) | 75.19$\\pm$0.15 | 74.97$\\pm$0.48 | 74.44$\\pm$0.26 | 73.32$\\pm$0.06 | 73.47$\\pm$0.41 |\n|CIFAR-10 (sudden drift) | 70.94$\\pm$0.42 | 73.17$\\pm$0.77 | 73.21$\\pm$0.82 | 72.25$\\pm$0.32 | 71.45$\\pm$0.68 |\n\n| | $\\epsilon$ = 0.05 | 0.1 | 0.15 | 0.2 |\n|---|---|---|---|---|\n|Fashion-MNIST (no drift) | 89.13$\\pm$0.29 | 89.74$\\pm$0.18 | 89.53$\\pm$0.10 | 89.76$\\pm$0.16 |\n|Fashion-MNIST (sudden drift) | 88.29$\\pm$0.41 | 89.39$\\pm$0.14 | 89.13$\\pm$0.34 | 89.37$\\pm$0.15 |\n|CIFAR-10 (no drift) | 74.00$\\pm$0.18 | 74.44$\\pm$0.26 | 74.18$\\pm$0.13 | 73.28$\\pm$0.69 |\n|CIFAR-10 (sudden drift) | 72.99$\\pm$0.78 | 73.21$\\pm$0.82 | 71.40$\\pm$0.84 | 69.33$\\pm$0.50 |\n\n> W4 & Q4: Handling Extreme Data Heterogeneity: The paper addresses data heterogeneity, but extreme cases of data heterogeneity [...] Further discussion and potential solutions to handle such extreme cases more effectively would improve the robustness of the framework.\n\nA4: Thank you for your insightful comments. In FL, different clients may suffer from various level of data heterogeneity. Requiring all clients to use the same hyperparameters for local training is not suitable. To address this problem, we propose adaptive alignment weight. We agree that other adaptive methods that dynamically adjust based on real-time training stability monitoring are more helpful to address this problem. An intuition is gradient clipping. According to our observations, gradient clipping can effectively address the gradient explosions when using FedFM, FedPAC, etc. Since the main focus in this work is distributed concept drift adaptation, we will investigate other adaptive methods to improve the performance under severe heterogeneity.\n\n**References:**\n\n[1] Luo, M., Chen, F., Hu, D., Zhang, Y., Liang, J., & Feng, J. (2021). No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34, 5972-5984.\n\n[2] Li, Z., Shang, X., He, R., Lin, T., & Wu, C. (2023). No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5319-5329).\n\n[3] Guo, Y., Tang, X., & Lin, T. (2023, July). Fedbr: Improving federated learning on heterogeneous data via local learning bias reduction. In International Conference on Machine Learning (pp. 12034-12054). PMLR.\n\n[4] Jothimurugesan, E., Hsieh, K., Wang, J., Joshi, G., & Gibbons, P. B. (2023, April). Federated learning under distributed concept drift. In International Conference on Artificial Intelligence and Statistics (pp. 5834-5853). PMLR.\n\n[5] Panchal, K., Choudhary, S., Mitra, S., Mukherjee, K., Sarkhel, S., Mitra, S., & Guan, H. (2023, July). Flash: concept drift adaptation in federated learning. In International Conference on Machine Learning (pp. 26931-26962). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to thank all reviewers for providing constructive comments that helped us to improve our paper. We are encouraged that reviews think our paper:

- "The idea of letting the global/local models evolve over time is interesting and relevant" (Reviewer U5Wz),
- "A relatively new and important problem of federated learning under concept drift is addressed" and "Contributions of this work are in the framework perspective and the module/strategy design perspective." (Reviewer HDwb),
- "The paper tackles an important and relatively unexplored problem in federated learning" (Reviewer erp1) and "This work addresses a critical gap in federated learning" (Reviewer uegQ),
- "The experiments are extensive" (Reviewer EQ7w) and "the results show significant performance improvements over existing methods in various concept drift scenarios" (Reviewer erp1)

We have been working diligently on improving the paper on several fronts, addressing your critique. Below, we provide a PDF containing more helpful **pipeline of our FedCCFA** (as suggested by Reviewer HDwb) and **visualization of distance matrix after concept drift** (as suggested by Reviewer erp1). Besides, according to the FAQ of NeurIPS 2024, it is not allowed to add any link in any part of the rebuttal. We will provide our code once this paper is accepted.


Please see our reviewer-specific feedback for more information.

Meta-review: I have read all comments and responses. Reviews appear to be consentaneous, with five scores of 7, 6, 6, 5, and 6. All concerns of reviewers have been fixed. It is recommended to accept this manuscript.
Decision: Accept (poster)