Title: Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift
Venue: NeurIPS 2024
Accepted: yes
Keywords: federated learning

concept drift

data heterogeneity
Abstract: Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted to tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is a largely unexplored area. In this work, we focus on real drift, where the conditional distribution $P(\mathcal{Y}|\mathcal{X})$ changes. We first study how distributed concept drift affects the model training and find that local classifier plays a critical role in drift adaptation. Moreover, to address data heterogeneity, we study the feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution $P(\mathcal{Y}|\mathcal{X})$ and the degree of data heterogeneity. Motivated by the above findings, we propose FedCCFA, a federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution $P(\mathcal{Y})$, alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at https://github.com/Chen-Junbao/FedCCFA.

Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift

Abstract

Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted to tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is a largely unexplored area. In this work, we focus on real drift, where the conditional distribution P ( Y|X ) changes. We first study how distributed concept drift affects the model training and find that local classifier plays a critical role in drift adaptation. Moreover, to address data heterogeneity, we study the feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution P ( Y|X ) and the degree of data heterogeneity. Motivated by the above findings, we propose FedCCFA, a federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution P ( Y ) , alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at https://github.com/Chen-Junbao/FedCCFA.

1 Introduction

Federated Learning (FL) [28] is an emerging privacy-preserving machine learning paradigm that allows multiple clients to collaboratively train a global model without sharing their raw data. In FL, clients train the models on their local data and send the updated models to the server for aggregation. Driven by the growing need for privacy protection, FL has been widely applied in various real-world scenarios [8, 16, 30, 44].

One significant challenge in FL is data heterogeneity, which denotes the discrepancies in the data distributions across clients. Such discrepancies can hinder the convergence of the global model.

However, existing works neglect a real-world setting where data heterogeneity and distributed concept drift simultaneously exist. Unlike conventional concept drift in centralized machine learning [11, 25, 26, 38], distributed concept drift [19] involves multiple clients experiencing different concept drifts at different times. For example, for the same medical image, diagnoses can vary among doctors (i.e., concept drift across clients), and even a doctor may offer different diagnoses at different times (i.e., concept drift across times). Moreover, the distribution of medical images varies across hospitals (i.e., data heterogeneity). This setting significantly degrades the performance of most FL methods, especially those using a single global model, because the model cannot provide different outputs for the same input. A similar research problem is multistream classification [4, 15, 46], where a sampling bias may exist between the distributions represented by source stream and target stream. Different from this problem, distributed concept drift focuses on the changing conditional distribution P ( Y|X ) across clients and over time. For each client at any round, training and test data distribution are assumed to be similar.

Several recent works have recognized the concept drift problem in FL. However, as discussed above, these single-model solutions [2, 3, 5, 14, 32] are ill-suited for distributed concept drift. FedDrift [19] considers distributed concept drift and employs multiple models to address this problem. However, FedDrift significantly increases the computation overhead for its distance measure and the storage overhead for multiple entire models. In addition, since the data heterogeneity can affect the loss estimation, FedDrift may fail to merge the models under the same concept. Experimental details are provided in Appendix C.7.

To tackle distributed concept drift, we analyze how it affects the model training in FL. In this work, we focus on real drift [11, 38] in concept drift, where the conditional distribution P ( Y|X ) changes. When P ( Y|X ) varies across clients, the representation should not be affected, as the marginal distribution P ( X ) is invariant. However, as shown in Figure 1, when vanilla FedAvg is adopted and distributed concept drift occurs at round 100, the Frobenius norm of representation update increases drastically, suggesting that the drift leads to large gradients in the representation. Furthermore, the accuracy of vanilla FedAvg drops rapidly and cannot recover to the accuracy before the drift, indicating the unsuitability of the single-model solution.

Motivated by the observations and analyses above, we decouple the network into an extrac-

0

50

100

150

200

Number of Rounds

20

40

60

80

Accuracy (%)

Test Accuracy

0

50

100

150

200

Number of Rounds

100

150

200

Representation Update

FedAvg

Decoupled

Decoupled-Clustering

tor and a classifier, and first train the classifier while fixing the extractor. After the classifier learns new conditional distribution, small gradients will be back-propagated to the extractor, and then we train the extractor. This decoupled method can effectively adapt to distributed concept drift, demonstrated by the much higher accuracy than FedAvg in Figure 1. The small gradient norm of representation at drift round also indicates the small gradients back-propagated from classifier. However, some clients may share similar P ( Y|X ) , and pure decoupled methods neglect the fine-grained collaboration between local classifiers. This will make each client's local classifier overfit to its local data. To enhance generalization performance, we develop a class-level classifier clustering method. Our clustering method separates the classifier for each class (referred to as class classifier), and then aggregates clients' class classifiers trained under the same conditional distribution P ( Y|X ) . The clients under the same conditional distribution share the aggregated class classifiers. This aggregation reduces the bias introduced by any single client's data, contributing to improved generalization performance. As shown in Figure 1, with the benefit of classifier clustering, the generalization performance is further improved, demonstrated by higher accuracy and smaller gradient norm. To remedy the data heterogeneity under distributed concept drift, we propose an adaptive feature alignment method, which aligns the feature spaces of the clients with the same conditional distribution P ( Y|X ) and adjusts the alignment weight according to the entropy of label distribution.

To summarize our contributions:

We explore the impact of distributed concept drift on FL training and propose a class-level classifier clustering approach that not only adapts to this drift but also enhances generalization performance (Section 4.1).

We propose clustered feature anchors to achieve feature alignment under distributed concept drift and propose an adaptive alignment weight to prevent severe data heterogeneity from impeding main task learning (Section 4.2).

We propose FedCCFA, a federated learning framework with classifier clustering and feature alignment (see Section 4.3). In Section 5, we conduct extensive experiments and empirical results demonstrate that FedCCFA can effectively adapt to distributed concept drift under data heterogeneity and significantly outperforms existing methods.

2 Related work

Concept drift in FL. Concept drift has been extensively studied in centralized machine learning [11, 12, 25, 26, 38]. Several recent works have recognized the concept drift problem in FL and proposed methods to tackle this issue, including regularization [5], continual learning [3, 14] and adaptive learning rate [2, 32]. These works assume that the conditional distributions P ( Y|X ) of all clients' dataset are the same, so only a single global model is trained. However, as proposed in [19] and discussed in Section 1, distributed concept drift may exist in FL setting and the single-model solution fails to adapt to this drift. To address this problem, FedDrift [19] creates new models based on drift detection and adaptively merges models by hierarchical clustering. However, FedDrift still faces some challenges: 1) incorrect model merging caused by data heterogeneity; 2) high computation and communication overhead for measuring cluster distance; 3) high storage cost for maintaining multiple global models.

Data heterogeneity in FL. Data heterogeneity hinders fast convergence when using vanilla FedAvg [28]. In this work, we focus on the heterogeneity of label distributions (referred to as label distribution skew). To alleviate this issue, previous works either focus on local training [1, 20, 22, 24, 29] or global aggregation [17, 18, 33, 39-41]. To further study how data heterogeneity affects FL, several recent works have focused on the representation of model, such as dimensional collapse [36] and inconsistent feature spaces [43, 45, 48, 50]. To align clients' feature spaces, FedFA [50] and FedPAC [43] regularize the ℓ 2 distance between local features and global anchors. To promote more precise alignment, FedFM [45] uses a contrastive-guiding method to further maximize the distance between the feature and non-corresponding anchors. However, severe data heterogeneity may significantly increase the loss of feature alignment, hindering the model convergence.

Clustered FL and personalized FL. Some clustered FL methods group clients with the same data distribution into a cluster, which can also adapt to distributed concept drift. To group clients, several works measure the similarity based on gradient information [9, 10, 35] or training loss [13, 27]. However, these methods face the following challenges: 1) unknown number of clusters; 2) large computation and communication overhead for estimating the training loss; 3) considerable storage cost for multiple global models. Personalized FL methods [6, 23, 31, 37, 43, 49] are also robust to distributed concept drift, since each client trains a local model for its local data distribution. However, most personalized FL methods neglect the classifier collaboration among the clients with similar data distributions, limiting the performance of models. Different from personalized FL approaches, this work focuses on generalized FL, aiming to enhance generalization performance.

3 Problem formulation

We consider a FL setting where there are K clients and a central server. The k -th client has a local dataset D k with data distribution P k ( X , Y ) , where X is the input space and Y is the label space with C classes. Let ℓ ( w ; x , y ) be the task-specific loss function associated with model w and data sample ( x , y ) . The global objective of FL can be formulated as:

where p k is the aggregation weight of the k -th client and F k ( w ) := E ( x ,y ) ∼ D k [ ℓ k ( w ; x , y )] is the local objective.

̸

To distinguish different types of concept drift, we decompose the joint distribution P ( X , Y ) as: P ( X , Y ) = P ( X ) P ( Y|X ) = P ( Y ) P ( X|Y ) . In this work, we focus on the real drift in concept drift. Real drift means that the conditional distribution at round t may be different from that at the previous round, i.e., P ( t ) ( Y|X ) = P ( t -1) ( Y|X ) . In addition, under distributed concept drift, this conditional distribution may vary across clients, i.e., P ( t ) i ( Y|X ) = P ( t ) j ( Y|X ) .

̸

To address distributed concept drift, we decouple the model parameterized by w = { θ , ϕ } into a feature extractor f θ parameterized by θ and a classifier f ϕ parameterized by ϕ . Given a sample ( x , y ) ∈ X × Y , the feature extractor f θ : X → Z maps the input x into a feature vector z = f θ ( x ) in the feature space Z , and then the classifier f ϕ : Z → Y maps the feature vector z to the label space Y . All clients share the feature extractor and each client maintains its local classifier. To align clients' feature spaces, we introduce a regularization term G k ( θ ; A k ) into the local objective function. Here, A k represents a form of global information that client k uses to align its feature space. Let Φ = { ϕ 1 , ϕ 2 , . . . , ϕ k } denote the set of all clients' local classifier parameters and θ denote the parameters of shared feature extractor. The global objective can be reformulated as:

where λ is the weight of feature alignment.

4 Methodology

FedCCFA decouples the network into a feature extractor f θ and a classifier f ϕ . In FedCCFA, the classifier is the last layer of the model (i.e., a linear classifier), and the feature extractor is composed of the remaining layers. In this section, we first present two methods used in FedCCFA and then describe the design of FedCCFA.

4.1 Classifier clustering

For better classifier collaboration, we introduce a new method for client clustering. Specifically, all clients share the same extractor and train their personal classifiers. Given the identical dimension reduction z = f θ ( x ) , the personal classifier learns the local data distribution. Therefore, the linear classifiers with similar weights exhibit similar data distributions, especially the conditional distribution P ( Y|X ) . Different from existing clustered FL methods, FedCCFA directly uses the classifier weights for clustering , which significantly reduces the computation and communication cost.

However, if using the weights of local classifiers, clustering may be disturbed by two factors: (1) variation in the classifier weights before training, and (2) class imbalance. For the first one, since clients' classifiers are different before local training, the classifiers trained under the same conditional distribution may differ significantly, which can lead to wrong clustering results. For the second one, the classifier can be easily biased towards head classes with massive training data, so the classifiers trained under the similar marginal distributions P ( X ) may be grouped, although their conditional distributions P ( Y|X ) are different.

Balanced classifier training. To address the above two problems, before local classifier training, FedCCFA trains a balanced classifier and uses it for classifier clustering. Specifically, to ensure that the classifier weights before training are identical, each selected client k ∈ I ( t ) updates its classifier ϕ ( t ) k by the initial global classifier ϕ (0) . Then, to address class imbalance, client k samples a balanced batch b ( t ) k from D ( t ) k to train its classifier ϕ ( t ) k while fixing its extractor θ ( t ) k :

where η ϕ denotes the learning rate for classifier.

After s training iterations, client k saves the balanced classifier ˆ ϕ ( t ) k and sends it to the server after local training. Note that, to reduce additional computation cost, we randomly select 5 samples for

each class c ∈ [ C ] (i.e., the balanced batch size | b ( t ) k | is 5 ∗ C ), and set training iterations s to a small value.

Class-level clustering. Unlike FedPAC [43] that uses entire classifier for collaboration, FedCCFA conducts classifier clustering at class-level . Specifically, after receiving all balanced classifiers { ˆ ϕ ( t ) k } , the server separates these classifiers for each class (referred to as class classifier). Then, for each class c , the server measures the class-level distance D c ( i, j ) between client i and client j by their class classifiers ˆ ϕ ( t ) i,c and ˆ ϕ ( t ) j,c , where i, j ∈ I ( t ) . To effectively measure distance between high-dimensional vectors, we exploit MADD [34] and realize a measure based on cosine distance:

where Cos ( a , b ) denotes the cosine distance between vector a and b .

After getting the distance matrix D c , DBSCAN clustering algorithm is used to get the class-level clusters S ( t ) c . For each cluster S ( t ) m,c ∈ S ( t ) c , clustered class classifier is aggregated as:

where |S ( t ) m,c | denotes the number of clients in cluster S ( t ) m,c . Each client k ∈ S ( t ) m,c updates its local class classifier ϕ ( t ) k,c by ¯ ϕ ( t ) m,c . Here we use uniform aggregation considering privacy concerns. More details about aggregation method are shown in Appendix C.6.

4.2 Adaptive feature alignment

To alleviate the inconsistency in feature spaces [45], we introduce a regularization term into the local objective function to align clients' feature spaces. Compared to existing alignment methods [43, 45, 50], our method is robust to two challenges: concept drift and the degree of data heterogeneity.

Clustered feature anchors. Existing feature alignment methods suppose that all clients' conditional distributions P ( Y|X ) are identical. Based on this, global feature anchors [45, 50] (also known as feature centroids [43]) are leveraged to align clients' feature spaces. However, under distributed concept drift scenario, the conditional distribution P ( Y|X ) may vary across clients, which can result in the differences in feature anchors. Therefore, clients require the global anchors that match their conditional distributions; otherwise, incorrect global anchors can mislead feature alignment. Motivated by this, we propose a feature alignment method using clustered feature anchors. Specifically, at round t , each client uses its feature extractor θ ( t ) k to compute the local feature anchor a ( t ) k,c for each class c :

Then, with the help of our class-level clustering, we compute the global feature anchor A ( t ) m,c of class c for each cluster S ( t ) m,c :

Finally, each client k ∈ S ( t ) m,c uses its global anchors A ( t ) k = {A ( t ) m,c } C c =1 to align its feature space during local training. FedCCFA leverages the contrastive-guiding loss proposed in FedFM [45], but uses clustered feature anchors. Let sim ( u , v ) denote the cosine similarity between u and v . Then the alignment loss function for a sample ( x , c ) with label c is defined as:

where f θ ( t ) k ( x ) is the representation vector of input x and τ denotes a temperature parameter.

Extractor

Classifier

Frozen

Trainable

⋯

⋯

Feature

anchors

Server

𝜙 "

!

(#)

𝜙 "

%

(#)

⋯

𝜙 "

!,'

(#)

𝜙 "

!,(

(#)

⋯

𝜙 "

%,'

(#)

𝜙 "

%,(

(#)

⋯

𝜙 "

!,!

(#)

𝜙 "

%,!

(#)

⋯

⋯

⋯

Classifier

split

Class classifiers

Balanced classifiers

⋯

𝒟

'

𝒟

!

𝒟

(

Class-level distances

Class

0

⋯

Class

𝑐

Client clusters

Clustering

⑤

Client clustering

Client

1

Local

data

Balanced

batch

Extractor

Classifier

Balanced classifier

𝜙 "

%

#

Classifier

Extractor

Extractor

Local model

Classifier

𝜙

%

(#)

𝜃

%

(#)

②

Balanced classifier training

Local training

⋯

Adaptive

feature

alignment

⋯

Feature

anchors

update

Client

𝑘

Local

data

Balanced

batch

Extractor

Classifier

Balanced classifier

𝜙 "

%

#

Classifier

Extractor

Extractor

Local model

Classifier

𝜙

%

(#)

𝜃

%

(#)

②

Balanced classifier training

③

Local training

⋯

Adaptive

feature

alignment

⋯

Local

anchors

update

⑥

Aggregation

Class

0

𝜙

!,(

(#)

𝜙

),(

(#)

𝜙

%,(

(#)

𝜙 -

!,(

(#)

⋯

𝜙

*,(

(#)

𝜙

+,(

(#)

𝜙 -

!,(

(#)

Class

𝑐

𝜙

!,(

(#)

𝜙

),(

(#)

𝜙

%,(

(#)

𝜙 -

!,(

(#)

⋯

𝜙

*,(

(#)

𝜙

+,(

(#)

𝜙 -

),(

(#)

⋯

Classifier aggregation

Class

0

⋯

Class

𝑐

⋯

⋯

Anchor aggregation

𝜃

!

(#)

𝜃

)

(#)

𝜃

%,!

(#)

𝜃

%

(#)

⋯

𝜃

(#-!)

Extractor aggregation

①

𝜃

!

, 𝜙

(#)

𝜃

%

!

, 𝜙

%

!

, 𝜙

%

%

!

④

⑦

𝜙

%

!

, 𝒜

%

(!)

88

8

★★

Adaptive alignment weight. In existing feature alignment methods [43, 45, 50], the alignment weight is fixed and all clients use the same weight. However, severe data heterogeneity will significantly increase the gradients of alignment term, impeding the main task learning. Experimental results are presented in Table 5. To balance the main task learning and feature alignment, it is essential to reduce the alignment weight under severe data heterogeneity. Note that, in this work, we focus on the label distribution skew and the degree of this heterogeneity can be reflected by the marginal distribution P ( t ) k ( Y ) . Based on this intuition, we leverage the entropy of marginal distribution P ( t ) k ( Y ) , denoted as H ( P ( t ) k ( Y )) , to adaptively determine the alignment weight:

where η θ denotes the learning rate for extractor and γ is the scaling factor. For the round T s to start feature alignment, we use the empirical value T s = 20 proposed in FedFM [45].

4.3 FedCCFA

We now present FedCCFA, a federated learning framework to adapt to distributed concept drift under data heterogeneity. The pipeline of FedCCFA is shown in Figure 2. The procedure of FedCCFA is formally presented in Algorithm 1 in Appendix A.

Local training. At each round t , each selected client k ∈ I ( t ) updates its extractor θ ( t ) k by the global extractor θ ( t ) , and then starts local training procedure. Specifically, each client trains a balanced classifier (Equation 3) for client clustering. Then, it fixes its feature extractor and trains local classifier to adapt to concept drift. Finally, it trains its feature extractor (Equation 9). Since the classifier ϕ ( t ) k learns the conditional distribution P ( t ) k ( Y|X ) , concept drift will not lead to larges gradients in the representation. After local training, each client generates its local anchors for each class (Equation 6).

Global aggregation. After receiving the local parameters and local anchors, the server starts the aggregation procedure. Specifically, the server performs client clustering with the help of balanced classifiers. Then, the server aggregates all feature extractors:

Finally, according to the clustering results, the server aggregates local classifiers and local anchors for each cluster.

5 Experiments

5.1 Experimental setup

Datasets and models. We conduct experiments on three datasets, namely Fashion-MNIST [42], CIFAR-10 [21] and CINIC-10 [7]. We construct two different CNN models for Fashion-MNIST and CIFAR-10/CINIC-10, respectively. Details of datasets and models are provided in Appendix B.1.

Baselines. We compare FedCCFA against the methods falling under the following categories: (1) single-model methods without drift adaptation: FedAvg [28], FedProx [24], SCAFFOLD [20] and FedFM [45]; (2) single-model methods with drift adaptation: Adaptive-FedAvg (shortened to AdapFedAvg) [2] and Flash [32]; (3) personalized FL methods: pFedMe [37], Ditto [23], FedRep [6], FedBABU [31] and FedPAC [43]; (4) clustered FL methods: IFCA [13] and FedDrift [19].

Federated learning settings. We consider two FL scenarios: 20 clients with full participation and 100 clients with 20% participation. We use a widely considered non-IID setting [22, 39, 47]: for each class c ∈ [ C ] , we sample a probability vector p c = ( p c, 1 , p c, 2 , . . . , p c,K ) ∼ Dir K ( α ) and allocate a p c,k proportion of instances of class c to client k ∈ [ K ] , where Dir K ( α ) denotes the Dirichlet distribution with concentration parameter α ; smaller α means more unbalanced partition. To evaluate the adaptability to concept drift, each client's training set contains at least 5 samples per class.

Concept drift settings. In this work, we focus on the distributed concept drift proposed in [19], where the conditional distribution P ( Y|X ) varies over time and across clients. We use three label swapping settings to simulate the conditional distribution changes across clients: for client k ∈ [ K ] , (1) class 1 and class 2 are swapped if k %10 < 3 ; (2) class 3 and class 4 are swapped if 3 < = k %10 < = 5 ; (3) class 5 and class 6 are swapped if k %10 > 5 . We simulate three patterns of conditional distribution changes over time: (1) Sudden Drift. All label swapping settings occur at round 100; (2) Incremental Drift. Label swapping settings (1), (2) and (3) occur at round 100, 110 and 120, respectively; (3) Reoccurring Drift. All label swapping settings occur at round 100, and these settings occur again at round 150.

Implementation details. We run 200 communication rounds for all experiments. To comprehensively evaluate the generalized performance and drift adaptability, for each client, we report the generalized accuracy of its model on the whole test set . In particular, the conditional distributions P ( Y|X ) of each client's training set and test set are the same at every round. For clustered and single-model methods, each client evaluates its global model; for the other methods, each client evaluates its personal model (or the combination of global extractor and personal classifier). Finally, we report the average accuracy across clients at the last round. For local training, we use SGD optimizer. The weight decay is 0.00001 and the SGD momentum is 0.9. For all datasets, we set local epochs E = 5 . For the decoupled methods, extractor learning rate is 0.01 and classifier learning rate is 0.1; for the other methods, local learning rate is 0.01. For FedCCFA, scaling factor γ is 20.0, iterations of balanced training s is 5 and balanced batch size is 5 ∗ C (5 samples per class). For classifier clustering, maximum distance ϵ and minimum samples used in DBSCAN algorithm are 0.1 and 1, respectively. More details of hyperparameters are provided in Appendix B.2.

5.2 Experimental results

We focus on three points in our experiments: (1) the generalization performance of FedCCFA; (2) the adaptability to distributed concept drift compared to existing methods; (3) the effects of our classifier clustering and adaptive alignment weight. Full experimental results are provided in Appendix C.

Performance without concept drift. First, we evaluate all methods under the no drift setting, and these results are the base performance compared against various concept drift settings. For each method, we run three trials and report the mean and standard deviation. Table 1 presents the generalized accuracy under Dir (0 . 5) . We observe that FedCCFA outperforms all decoupled methods (FedRep, FedBABU and FedPAC) and personalized FL methods (pFedMe and Ditto), indicating that FedCCFA enhances the generalization performance of local classifiers. Besides, the performance of IFCA under 20 clients with 100% participation is worse than the performance under 100 clients with 20% participation. This is because multiple global models are trained under similar data distribution (i.e., each global model is trained with much less data). In contrast, with the help of our classifier clustering, clients with similar data distribution will share the same classifier. Compared with single-model methods, FedCCFA achieves lower accuracy, which is attributed to decoupled training. Specifically, for decoupled methods (e.g., FedPAC, FedRep and our FedCCFA), training the classifier first might cause it to fit the initial features, resulting in gradient attenuation during backpropagation. This will restrict the subsequent extractor's training process and prevent it from optimally learning

features. In contrast, when training the entire model together, all layers can simultaneously adapt to the training data, allowing them to synergize effectively.

Performance under distributed concept drift. Next, we evaluate all methods under the distributed concept drift setting. Table 2 presents the generalized accuracy under this setting. We observe that FedCCFA significantly outperforms all baselines. Specifically, under full participation (i.e., 20 clients with 100% participation), FedCCFA outperforms the second-best method 2.48%, 9.50% and 0.06% on Fashion-MNIST, CIFAR-10 and CINIC-10, respectively; under partial participation (i.e., 100 clients with 20% participation), FedCCFA outperforms the second-best method 1.14%, 8.48% and 3.34% on Fashion-MNIST, CIFAR-10 and CINIC-10, respectively. Compared with the no drift setting, the results of accuracy decrease are shown in Appendix C.2. These results also show that (1) personalized FL methods can effectively adapt to distributed concept drift, demonstrated by the smaller accuracy decrease compared with single-model methods; (2) clustered FL methods may fail to adapt to distributed concept drift under data heterogeneity, because clustering results may be affected by data heterogeneity.

Effects of balanced classifier training. As analyzed in Section 4.1, balanced classifier training is crucial for classifier clustering. We conduct experiments under sudden drift setting and Table 3 shows the results of our ablation study on the input of clustering. We see that using the weights of balanced classifiers can significantly increase the generalized accuracy. Furthermore, to demonstrate that accurate clustering results contribute to this improvement, we compare our method against an oracle method which has knowledge about the accurate clustering results at each round. Experimental results show that oracle method indeed increases the generalized accuracy and our method is comparable to the oracle method. These results demonstrate that classifier clustering can enhance the generalization performance and our method achieves accurate clustering.

Effects of clustered feature anchors. To verify the efficacy of clustered feature anchors, we ablate FedCCFA by a variety of alignment methods. From Table 4, we see that (1) feature alignment can boost the accuracy under data heterogeneity; (2) clustered feature anchors can facilitate more precise feature alignment under distributed concept drift.

Effects of adaptive alignment weight. Weturn the degree of data heterogeneity ( α ∈ { 0 . 01 , 0 . 1 , 0 . 5 } and smaller α corresponds to more severe heterogeneity), and report the average accuracy in Table 5. First, we conduct experiments with various fixed alignment weights { 0 , 0 . 1 , 1 . 0 } . Experimental results show that small alignment weight is better under severe data heterogeneity. In addition, under Dir (0 . 01) , training without feature alignment outperforms training with feature alignment, indicating that feature alignment impedes main task learning under this setting. Then, we turn the

scaling factor γ ∈ { 20 , 50 , 100 } . We see that our adaptive alignment weight is robust to the degree of data heterogeneity and the performance is not sensitive to the scaling factor.

6 Conclusion

In this work, we take a further step towards distributed concept drift in federated learning. We have shown that FedCCFA can effectively adapt to distributed concept drift, and outperforms existing methods under various concept drift settings. Further experiments suggest that our classifier clustering method significantly enhances the generalization performance for decoupled FL methods. Besides, our clustered feature anchors can enhance the precision of feature alignment and our adaptive alignment weight can stabilize the training process when using feature alignment under severe heterogeneity. We hope that FedCCFA can inspire more works on classifier collaboration and other types of concept drift in federated learning.

Limitations. FedCCFA takes some steps to train balanced classifiers, which increases computation cost. Although we reduce it by setting small iteration and batch size, it is preferable to remove these steps. We will investigate other efficient methods in the future. Besides, feature alignment, used in FedCCFA, FedFM and FedPAC, could impede main task learning under severe data heterogeneity. Therefore, research on the feature alignment under severe heterogeneity is also of interest.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In the 9th International Conference on Learning Representations , 2021.

Giuseppe Canonaco, Alex Bergamasco, Alessio Mongelluzzo, and Manuel Roveri. Adaptive federated learning in presence of concept drift. In 2021 International Joint Conference on Neural Networks (IJCNN) , pages 1-7. IEEE, 2021.

Fernando E Casado, Dylan Lema, Marcos F Criado, Roberto Iglesias, Carlos V Regueiro, and Senén Barro. Concept drift detection and adaptation for federated and continual learning. Multimedia Tools and Applications , pages 1-23, 2022.

Swarup Chandra, Ahsanul Haque, Latifur Khan, and Charu Aggarwal. An adaptive framework for multistream classification. In Proceedings of the 25th ACM international on conference on information and knowledge management , pages 1181-1190, 2016.

Yujing Chen, Zheng Chai, Yue Cheng, and Huzefa Rangwala. Asynchronous federated learning for sensor data with concept drift. In 2021 IEEE International Conference on Big Data (Big Data) , pages 4822-4831. IEEE, 2021.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning , pages 2089-2099, 2021.

Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine , 27(10):1735-1743, 2021.

Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Fedgroup: Efficient clustered federated learning via decomposed data-driven measure. arXiv preprint arXiv:2010.06870 , 2020.

Moming Duan, Duo Liu, Xinyuan Ji, Yu Wu, Liang Liang, Xianzhang Chen, Yujuan Tan, and Ao Ren. Flexible clustered federated learning for client-level data distribution shift. IEEE Transactions on Parallel and Distributed Systems , 33(11):2661-2674, 2021.

João Gama, Indr˙ e Žliobait˙ e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR) , 46(4):1-37, 2014.

Y. Gao, Swarup Chandra, Yifan Li, L. Khan, and Bhavani M. Thuraisingham. Saccos: A semi-supervised framework for emerging class detection and concept drift adaption over data streams. IEEE Transactions on Knowledge and Data Engineering , 34:1416-1426, 2022.

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems , 33:1958619597, 2020.

Yongxin Guo, Tao Lin, and Xiaoying Tang. Towards federated learning on time-evolving heterogeneous data. arXiv preprint arXiv:2112.13246 , 2021.

Ahsanul Haque, Zhuoyi Wang, Swarup Chandra, Bo Dong, Latifur Khan, and Kevin W Hamlen. Fusion: An online method for multistream classification. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 919-928, 2017.

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 , 2018.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. In the 11th International Conference on Learning Representations , 2023.

Ellango Jothimurugesan, Kevin Hsieh, Jianyu Wang, Gauri Joshi, and Phillip B Gibbons. Federated learning under distributed concept drift. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , pages 5834-5853. PMLR, 2023.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning , pages 5132-5143, 2020.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10713-10722, 2021.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning , pages 6357-6368, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Wendi Li, Xiao Yang, Weiqing Liu, Yingce Xia, and Jiang Bian. Ddg-da: Data distribution generation for predictable concept drift adaptation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence , volume 36, pages 4092-4100, 2022.

Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering , 31(12):23462363, 2018.

Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619 , 2020.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282, 2017.

Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8397-8406, 2022.

Viraaji Mothukuri, Prachi Khare, Reza M Parizi, Seyedamin Pouriyeh, Ali Dehghantanha, and Gautam Srivastava. Federated-learning-based anomaly detection for iot security attacks. IEEE Internet of Things Journal , 9(4):2545-2554, 2021.

Jaehoon Oh, SangMook Kim, and Se-Young Yun. FedBABU: Toward enhanced representation for federated image classification. In the 10th International Conference on Learning Representations , 2022.

Kunjal Panchal, Sunav Choudhary, Subrata Mitra, Koyel Mukherjee, Somdeb Sarkhel, Saayan Mitra, and Hui Guan. Flash: Concept drift adaptation in federated learning. In Proceedings of the 40th International Conference on Machine Learning , pages 26931-26962, 2023.

Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In the 9th International Conference on Learning Representations , 2021.

Soham Sarkar and Anil K Ghosh. On perfect clustering of high dimension, low sample size data. IEEE transactions on pattern analysis and machine intelligence , 42(9):2257-2272, 2019.

Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems , 32(8):3710-3722, 2020.

Yujun Shi, Jian Liang, Wenqing Zhang, Chuhui Xue, Vincent Y. F. Tan, and Song Bai. Understanding and mitigating dimensional collapse in federated learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(5):2936-2949, 2024.

Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems , 33:21394-21405, 2020.

Ashraf Tahmasbi, Ellango Jothimurugesan, Srikanta Tirthapura, and Phillip B Gibbons. Driftsurf: Stable-state / reactive-state learning under concept drift. In Proceedings of the 38th International Conference on Machine Learning , pages 10054-10064, 2021.

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In the 8th International Conference on Learning Representations , 2020.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623, 2020.

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643 , 2019.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In the 11th International Conference on Learning Representations , 2023.

Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of healthcare informatics research , 5:1-19, 2021.

Rui Ye, Zhenyang Ni, Chenxin Xu, Jianyu Wang, Siheng Chen, and Yonina C. Eldar. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. IEEE Transactions on Signal Processing , 71:4224-4239, 2023.

En Yu, Jie Lu, Bin Zhang, and Guangquan Zhang. Online boosting adaptive learning under concept drift for multistream classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 16522-16530, 2024.

Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning , pages 7252-7261, 2019.

Lin Zhang, Yong Luo, Yan Bai, Bo Du, and Ling-Yu Duan. Federated learning for non-iid data via unified feature learning and optimization objective alignment. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4420-4428, 2021.

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations , 2021.

Tailin Zhou, Jun Zhang, and Danny HK Tsang. Fedfa: Federated learning with feature anchors to align features and classifiers for heterogeneous data. IEEE Transactions on Mobile Computing , 2023.

A Algorithm

The procedure of FedCCFA is formally presented in Algorithm 1. Compared with decoupled FL methods [6, 31], the sections with green color and red color are the additional computation and communication steps, respectively.

Local training. At each round t , each selected client k ∈ I ( t ) first updates its extractor θ ( t ) k by the global extractor θ ( t ) . Then, it starts local training procedure. Lines 6-7, Line 8, Line 9 and Lines 10-12 correspond to balanced classifier training, local classifier training, extractor training and local anchor calculation, respectively. Since the classifier ϕ ( t ) k learns the conditional distribution P ( t ) k ( Y|X ) , concept drift will not lead to larges gradients in the representation.

Global aggregation. After receiving the local parameters and local anchors, the server starts the aggregation procedure. Line 15, Lines 17-18 and Line 20 correspond to extractor aggregation, class-level clustering and the aggregation of classifiers and anchors, respectively. Note that, for better description, each client updates its classifiers and global anchors in a double loop, as described in Line 21. In the implementation, the server sends aggregated class classifiers and global anchors to each client at the end of round t .

Algorithm 1 FedCCFA

B Experimental details

All experiments are conducted using PyTorch and all results are produced with an Intel Core i713700K CPU and a NVIDIA GeForce RTX 4090 GPU. The required memory is around 2GB 7GB.

B.1 Datasets and models

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(a) CIFAR-10 under

Dir

(0

.

1)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(b) CIFAR-10 under

Dir

(0

.

5)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(c) CINIC-10 under Dir (0 . 1)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(d) CINIC-10 under Dir (0 . 5)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(a) CIFAR-10 under

Dir

(0

.

1)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(b) CIFAR-10 under

Dir

(0

.

5)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(c) CINIC-10 under Dir (0 . 1)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

Classes

(d) CINIC-10 under Dir (0 . 5)

We evaluate our method and all baselines on three datasets: (1) Fashion-MNIST [42]: a dataset comprising of 28×28 grayscale images of 70,000 fashion products from 10 categories (7,000 images per category), where the training set has 60,000 images and the test set has 10,000 images; (2) CIFAR-10 [21]: a dataset consisting of 60,000 32*32 color images in 10 classes (6,000 images per class), where the training set has 50,000 images and the test set has 10,000 images; (3) CINIC-10 [7]: an augmented extension of CIFAR-10, containing the images from CIFAR-10 (60,000 images) and a selection of ImageNet (210,000 images downsampled to 32*32). We use the same model architectures

as those used in FedPAC [43]. For Fashion-MNIST, we construct a CNN model consisting of two convolution layers with 16 5*5 and 32 5*5 filters respectively, each followed by a max pooling layer, and two fully-connected layers with 128 and 10 neurons before softmax layer. For CIFAR-10 and CINIC-10, the CNN model is similar to the model used for Fashion-MNIST but has one more convolution layer with 64 5*5 filters.

We visualize the data partition for CIFAR-10 and CINIC-10 under two degrees of data heterogeneity (i.e., Dir (0 . 1) and Dir (0 . 5) ). Figure 3 shows the partition visualization under full participation (i.e., 20 clients with 100% participation). Since CINIC-10 is much larger than CIFAR-10 and we consider the setting where clients have different data sizes (i.e., quantity skew), data distribution of CINIC-10 is more biased. Figure 4 shows the partition visualization under partial participation (i.e., 100 clients with 20% participation). We randomly choose 20 clients for visualization. We see that data heterogeneity is more severe under full participation, due to more severe quantity skew.

B.2 Hyperparameters

We employ SGD as the optimizer for all methods. The batch size is set to 64, the momentum is set to 0.9 and the weight decay is set to 0.00001. For all datasets, we set local epochs E = 5 . In particular, for the decoupled methods, we first train the classifier for 1 epoch with learning rate η ϕ = 0 . 1 , and then train the extractor for 5 epochs with learning rate η θ = 0 . 01 , aiming to reduce the computation cost; for the other methods, we set local learning rate η = 0 . 01 .

For FedCCFA, we turn the scaling factor γ ∈ { 5 . 0 , 10 . 0 , 20 . 0 , 50 . 0 , 100 . 0 } , and set it to 20.0 according to the results in Figure 5; we turn the maximum distance ϵ ∈ { 0 . 05 , 0 . 10 , 0 . 15 , 0 . 20 } , and set it to 0.10 according to the results in Figure 6.

For IFCA, we set the number of clusters k = 4 , which is the oracle knowledge of the ground-truth clustering. For the following baselines, we tune their hyperparameters on CIFAR-10 under no concept drift and Dir (0 . 5) , and select the best hyperparameters for all settings:

FedDrift: we turn the detection threshold δ ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 5 . 0 } , and set it to 0.5.

FedFM: we turn the penalty coefficient λ ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 , 50 . 0 } , and set it to 1.0.

FedPAC: we turn the penalty coefficient λ ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 , 50 . 0 } , and set it to 1.0.

Flash: we turn the server learning rate η g ∈ { 0 . 005 , 0 . 01 , 0 . 02 , 0 . 1 , 1 . 0 } , and set it to 0.01.

SCAFFOLD: we turn the server learning rate η g ∈ { 0 . 005 , 0 . 01 , 0 . 02 , 0 . 1 , 1 . 0 } , and set it to 1.0.

pFedMe: we turn K ∈ { 1 , 5 , 7 } , and set it to 5; we turn λ ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 , 15 . 0 } , and set it to 1.0.

Ditto: we turn penalty coefficient λ ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 } , and set it to 5.0.

5

10

20

50

100

40

50

60

70

80

Accuracy (%)

Dir

(0.01)

Dir

(0.1)

Dir

(0.5)

0.05

0.1

0.15

0.2

40

50

60

70

80

Accuracy (%)

FedCCFA

oracle

C Additional experimental results

C.1 Performance under Dir(0.1)

We evaluate all methods under no drift setting and Dir (0 . 1) . Table 6 shows the generalized accuracy under Dir (0 . 1) . We observe that, on CIFAR-10 and CINIC-10 with 20% participation, FedCCFA outperforms baselines. Besides, FedCCFA outperforms all decoupled methods and personalized FL methods, showing that FedCCFA boosts the generalization performance of classifiers under this setting. We find that gradient explosion occurs when adopting FedPAC and FedRep on CINIC-10 with full participation. Under this setting, data heterogeneity is much severe (as shown in Figure 3) and classifier is extremely biased. Such classifier could lead to gradient explosion when using high classifier learning rate, so the accuracy of FedRep may drop to 10% and FedPAC fails to finish FL training due to abnormal inputs to its classifier collaboration. Since FedBABU uses the initial global classifier during training and FedCCFA uses clustered classifiers, this problem is alleviated.

C.2 Performance under sudden drift

Weevaluate all methods under distributed concept drift and Dir (0 . 1) . Table 7 presents the generalized accuracy under sudden drift setting. We find that FedCCFA outperforms all baselines on all datasets except for CINIC-10 with full participation. This weakness is caused by wrong clustering results. Specifically, in this case, the performance of global extractor is limited (the accuracy of the best method is only 53.86% under no drift setting), so the poor extractor cannot generate good feature vectors for the classifier. For this reason, the classifier cannot accurately learn the data distribution and its weights may provide wrong information. Besides, we find that gradient explosion also occurs when using FedFM on CINIC-10 with full participation. This is because severe heterogeneity will significantly increase the loss of feature alignment term when distributed concept drift occurs. FL methods without feature alignment are not affected, such as FedAvg, Flash and IFCA.

We present the generalized accuracy decrease under sudden drift setting to show the adaptability to distributed concept drift. These results show that FedCCFA can effectively adapt to distributed concept drift except for CINIC-10 with full participation (due to the poor extractor discussed above). For personalized FL methods, thanks to their personal models or classifiers, distributed concept drift will not affect them.

C.3 Performance under incremental drift

For all methods, we validate the adaptability to incremental drift, where three different concept drifts occur at round 100, 110 and 120, respectively. Table 9 shows that, under Dir (0 . 5) , FedCCFA can also adapt to this drift setting and outperforms all baselines.

C.4 Performance under reoccurring drift

To verify whether all methods could recover to the same accuracies once the original data distributions come back, we conduct experiments under the reoccurring drift setting. Table 10 shows that all methods can achieve this goal under reoccurring drift and Dir (0 . 5) .

C.5 Visualization of distance matrix

To demonstrate the effectiveness of our classifier clustering, we conduct experiments on CIFAR-10 under sudden drift setting with full participation, and then visualize the distance matrices (i.e., the input of DBSCAN algorithm) in Figure 7. The left shows the distance matrix of class 1. According to the concept drift setting in Section 5.1, for class 1, clients { 0 , 1 , 2 , 10 , 11 , 12 } should be grouped and the others should be grouped. The right shows the distance matrix of class 5. For this class, clients { 6 , 7 , 8 , 9 , 16 , 17 , 18 , 19 } should be grouped and the others should be grouped. These two matrices demonstrate that our clustering method can effectively measure the distance between each client.

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0.00

0.05

0.04

0.52

0.50

0.53

0.51

0.51

0.48

0.48

0.07

0.02

0.06

0.53

0.51

0.52

0.52

0.52

0.49

0.50

0.05

0.00

0.03

0.49

0.47

0.49

0.48

0.48

0.45

0.45

0.10

0.04

0.10

0.49

0.48

0.49

0.49

0.49

0.46

0.47

0.04

0.03

0.00

0.49

0.47

0.49

0.48

0.48

0.45

0.45

0.10

0.04

0.10

0.49

0.48

0.49

0.49

0.49

0.45

0.47

0.52

0.49

0.49

0.00

0.02

0.01

0.02

0.03

0.05

0.05

0.57

0.52

0.56

0.02

0.01

0.01

0.01

0.02

0.04

0.02

0.50

0.47

0.47

0.02

0.00

0.03

0.02

0.02

0.03

0.03

0.55

0.50

0.55

0.03

0.02

0.02

0.02

0.03

0.03

0.01

0.53

0.49

0.49

0.01

0.03

0.00

0.03

0.03

0.06

0.06

0.57

0.52

0.57

0.01

0.02

0.01

0.01

0.02

0.05

0.03

0.51

0.48

0.48

0.02

0.02

0.03

0.00

0.01

0.03

0.03

0.56

0.50

0.55

0.03

0.01

0.02

0.02

0.03

0.02

0.01

0.51

0.48

0.48

0.03

0.02

0.03

0.01

0.00

0.03

0.03

0.56

0.50

0.55

0.03

0.02

0.03

0.03

0.04

0.02

0.02

0.48

0.45

0.45

0.05

0.03

0.06

0.03

0.03

0.00

0.02

0.53

0.48

0.52

0.06

0.04

0.05

0.05

0.05

0.02

0.03

0.48

0.45

0.45

0.05

0.03

0.06

0.03

0.03

0.02

0.00

0.53

0.47

0.52

0.06

0.04

0.05

0.05

0.05

0.02

0.03

0.07

0.10

0.10

0.57

0.55

0.57

0.56

0.56

0.53

0.53

0.00

0.07

0.01

0.58

0.57

0.57

0.57

0.57

0.54

0.55

0.02

0.04

0.04

0.52

0.50

0.52

0.50

0.50

0.48

0.47

0.07

0.00

0.06

0.52

0.51

0.52

0.52

0.52

0.49

0.50

0.06

0.10

0.10

0.56

0.55

0.57

0.55

0.55

0.52

0.52

0.01

0.06

0.00

0.57

0.56

0.56

0.56

0.56

0.53

0.54

0.53

0.49

0.49

0.02

0.03

0.01

0.03

0.03

0.06

0.06

0.58

0.52

0.57

0.00

0.02

0.02

0.01

0.01

0.05

0.03

0.51

0.48

0.48

0.01

0.02

0.02

0.01

0.02

0.04

0.04

0.57

0.51

0.56

0.02

0.00

0.02

0.02

0.03

0.03

0.01

0.52

0.49

0.49

0.01

0.02

0.01

0.02

0.03

0.05

0.05

0.57

0.52

0.56

0.02

0.02

0.00

0.01

0.03

0.04

0.03

0.52

0.49

0.49

0.01

0.02

0.01

0.02

0.03

0.05

0.05

0.57

0.52

0.56

0.01

0.02

0.01

0.00

0.02

0.04

0.02

0.52

0.49

0.49

0.02

0.03

0.02

0.03

0.04

0.05

0.05

0.57

0.52

0.56

0.01

0.03

0.03

0.02

0.00

0.05

0.03

0.49

0.46

0.45

0.04

0.03

0.05

0.02

0.02

0.02

0.02

0.54

0.49

0.53

0.05

0.03

0.04

0.04

0.05

0.00

0.02

0.50

0.47

0.47

0.02

0.01

0.03

0.01

0.02

0.03

0.03

0.55

0.50

0.54

0.03

0.01

0.03

0.02

0.03

0.02

0.00

Class 1

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Clients

0.00

0.02

0.03

0.03

0.04

0.01

0.46

0.41

0.32

0.39

0.02

0.01

0.02

0.03

0.02

0.02

0.46

0.46

0.43

0.41

0.02

0.00

0.02

0.04

0.06

0.02

0.45

0.40

0.30

0.37

0.02

0.03

0.02

0.04

0.03

0.02

0.44

0.45

0.41

0.39

0.03

0.02

0.00

0.04

0.06

0.03

0.44

0.39

0.30

0.36

0.03

0.04

0.02

0.04

0.03

0.02

0.43

0.44

0.40

0.39

0.03

0.04

0.04

0.00

0.03

0.03

0.47

0.43

0.33

0.40

0.04

0.03

0.03

0.01

0.04

0.04

0.47

0.47

0.44

0.42

0.04

0.06

0.06

0.03

0.00

0.05

0.49

0.44

0.35

0.42

0.05

0.04

0.06

0.03

0.05

0.06

0.49

0.49

0.46

0.44

0.01

0.02

0.03

0.03

0.05

0.00

0.47

0.42

0.33

0.39

0.02

0.01

0.02

0.03

0.02

0.02

0.46

0.47

0.43

0.41

0.46

0.45

0.44

0.47

0.49

0.47

0.00

0.07

0.17

0.09

0.46

0.47

0.45

0.48

0.47

0.45

0.02

0.02

0.04

0.06

0.41

0.40

0.39

0.43

0.44

0.42

0.07

0.00

0.11

0.03

0.41

0.42

0.40

0.42

0.42

0.40

0.06

0.06

0.05

0.03

0.32

0.30

0.30

0.33

0.35

0.33

0.17

0.11

0.00

0.08

0.32

0.33

0.31

0.33

0.33

0.31

0.16

0.16

0.13

0.11

0.39

0.37

0.36

0.40

0.42

0.39

0.09

0.03

0.08

0.00

0.39

0.40

0.37

0.40

0.40

0.37

0.08

0.09

0.06

0.04

0.02

0.02

0.03

0.04

0.05

0.02

0.46

0.41

0.32

0.39

0.00

0.03

0.03

0.04

0.02

0.03

0.46

0.46

0.43

0.41

0.01

0.03

0.04

0.03

0.04

0.01

0.47

0.42

0.33

0.40

0.03

0.00

0.03

0.03

0.02

0.03

0.47

0.47

0.44

0.42

0.02

0.02

0.02

0.03

0.06

0.02

0.45

0.40

0.31

0.37

0.03

0.03

0.00

0.03

0.03

0.02

0.44

0.45

0.41

0.40

0.03

0.04

0.04

0.01

0.03

0.03

0.48

0.42

0.33

0.40

0.04

0.03

0.03

0.00

0.03

0.04

0.47

0.47

0.44

0.42

0.02

0.03

0.03

0.04

0.05

0.02

0.47

0.42

0.33

0.40

0.02

0.02

0.03

0.03

0.00

0.03

0.46

0.47

0.43

0.42

0.02

0.02

0.02

0.04

0.06

0.02

0.45

0.40

0.31

0.37

0.03

0.03

0.02

0.04

0.03

0.00

0.44

0.45

0.41

0.39

0.46

0.44

0.43

0.47

0.49

0.46

0.02

0.06

0.16

0.08

0.46

0.47

0.44

0.47

0.46

0.44

0.00

0.01

0.03

0.05

0.46

0.45

0.44

0.47

0.49

0.47

0.02

0.06

0.16

0.09

0.46

0.47

0.45

0.47

0.47

0.45

0.01

0.00

0.04

0.06

0.43

0.41

0.40

0.44

0.46

0.43

0.04

0.05

0.13

0.06

0.43

0.44

0.41

0.44

0.43

0.41

0.03

0.04

0.00

0.04

0.41

0.39

0.39

0.42

0.44

0.41

0.06

0.03

0.11

0.04

0.41

0.42

0.40

0.42

0.42

0.39

0.05

0.06

0.04

0.00

Class 5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Distance

C.6 Ablation study on aggregation method

FedCCFA involves class-level classifier aggregation (Equation 5) and class-level feature anchor aggregation (Equation 7). An intuitive manner is sample-number-based weighted aggregation, which is similar to the model aggregation in FedAvg. However, this manner requires clients to upload the sample number for each class, which may leak privacy about clients' category distributions. To address this problem, we consider the uniform aggregation, where all clients share the same aggregation weight. This aggregation manner does not involve other private information. We compare the two manners of aggregation. As shown in Table 11, we observed that uniform aggregation even performs better, especially under concept drift setting. This may be because that uniform aggregation can prevent the classifier from overfitting to the classifier that is significantly biased due to data heterogeneity.

C.7 FedDrift under data heterogeneity and distributed concept drift

FedDrift [19] is the first solution for distributed concept drift, which creates new global models based on drift detection and merges models by hierarchical clustering. However, data heterogeneity may disturb model merging, because cluster distance could exceed the threshold under data heterogeneity. To validate this conjecture, we present the accuracy curves and the number of used global models during training process. The latter denotes the number of global models used by all clients at each round. We conduct experiments under the full participation setting.

We first present the accuracy curves and the number of used global models under homogeneous setting in Figure 8. We see that FedDrift can creates new models under various drift settings and accurately merges the models under the same concepts (i.e., the same conditional distributions P ( Y|X ) ). However, since new global models are created at drift rounds and these models need to be retrained, FedDrift cannot rapidly reach a steady state.

0

50

100

150

200

Number of Rounds

80

85

90

Accuracy (%)

Test Accuracy

0

50

100

150

200

Number of Rounds

0

2

4

6

8

10

12

14

16

18

20

Model number

Global models

no drift

sudden

incremental

reoccurring

Then, we present the two indices under Dir (0 . 1) in Figure 9. We find that FedDrift can creates new model at drift rounds, but it cannot accurately merge the models under the same concepts. This is because, in addition to different concepts, different label distributions can also increase the cluster distances estimated by the loss value, which will prevent model merging. Since all clients choose the only global model at the beginning of training process and the loss delta will not exceed the threshold if no drift occurs, new global models will not be created under the no drift setting.

0

50

100

150

200

Number of Rounds

40

60

80

Accuracy (%)

Test Accuracy

0

50

100

150

200

Number of Rounds

0

2

4

6

8

10

12

14

16

18

20

Model number

Global models

no drift

sudden

incremental

reoccurring

C.8 Computation cost comparison

To enhance client collaboration, FedPAC [43] introduces a classifier combination method, which uses feature statistics to estimate optimal combining weights. However, according to our experimental results, this operation is time-consuming. We compare the computation cost of classifier collaboration between FedCCFA and FedPAC. We count the time consumption for classifier collaboration after sudden drift occurs, and report the average value of 10 rounds. Table 12 shows that FedPAC is more time-consuming than FedCCFA, especially on large datasets (e.g., CINIC-10).

C.9 Personalized performance

Though this work does not focus on personalized FL, we study whether more generalized model could bring benefits to personalization. We distribute test samples by the same Dirichlet distribution used for training data partition, and evaluate the personalized accuracy over each client's personal model (or global extractor followed by personal classifier). After FL training, we fine-tune the classifier with 5 epochs and classifier learning rate is 0.01. Table 13 shows that FedCCFA with classifier fine-tuning is comparable to the most performing personalized methods.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims reflect this paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of efficiency and feature alignment in Conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We empirically verify the effectiveness of our method in Section 5.2 and Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present the details of our method in Section 4. We provide the details of experimental settings and hyperparameter selection in Section 5.1 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have released our code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the details of experimental settings and hyperparameter selection in Section 5.1 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We explain how we calculate the error bars in Section 5.1 and the results are averaged over 3 runs (mean ± std).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information on the computer resources and deep learning framework in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focuses on distributed concept drift in federated learning. Users' personal data is usually protected in federated learning and we do not find any possible adverse effects on society caused by this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original papers that produced the datasets, including FashionMNIST, CIFAR-10 and CINIC-10.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the significant challenge of data heterogeneity and distributed concept drift in federated learning. The authors propose a novel framework, which integrates classifier clustering and feature alignment to improve model performance and collaboration among clients facing different concept drifts.\nThe key contributions of the paper are as follows:\n1. FedCCFA Framework: The proposed framework includes a method for clustering local classifiers at the class level and generating clustered feature anchors to enhance feature alignment. This clustering helps clients with similar data distributions share classifiers, thus improving the generalization performance of the global model.\n2. Feature Alignment: The framework introduces an adaptive feature alignment technique that aligns clients' feature spaces based on the entropy of the label distribution. This method helps to alleviate the inconsistencies in feature space due to data heterogeneity.\n3. Experimental Validation: Extensive experiments demonstrate that FedCCFA significantly outperforms existing federated learning methods under various concept drift settings. The results show that FedCCFA effectively adapts to distributed concept drift and enhances generalization performance.\n\nSTRENGTHS:\n1. Comprehensive Validation: The research is supported by well-designed experiments and thorough ablation studies, demonstrating the effectiveness of the FedCCFA framework. The mathematical formulations and algorithms are clearly presented, and the results are statistically significant.\n2. Clear Presentation: The paper is clearly written and well-structured, making complex concepts accessible. Visual aids, such as graphs and tables, effectively support the explanations. Detailed breakdowns of the experimental setup enhance the reproducibility of the results.\n3. Significant Impact: This work addresses a critical gap in federated learning, with potential applications in diverse fields like healthcare, finance, and mobile device collaboration. By improving the adaptability and generalization performance of federated models, the proposed FedCCFA framework provides a robust foundation for future studies to build upon, advancing the current state of federated learning research.\n\nWEAKNESSES:\n1. Computation Overhead: The proposed FedCCFA framework involves additional computational steps to train balanced classifiers, which increases the overall computational cost. While the authors attempt to mitigate this by setting small iterations and batch sizes, exploring more efficient methods to achieve balanced classifier training would be beneficial.\n2. Limited Evaluation Scenarios: The experiments are primarily conducted on standard datasets. Including more diverse datasets, especially those with real-world distributed concept drift scenarios, would strengthen the validation of the framework's general applicability and robustness.\n3. Sensitivity to Hyperparameters: The effectiveness of FedCCFA relies on several hyperparameters, such as the maximum distance in DBSCAN and the scaling factor. While the authors provide some tuning, a more thorough analysis of the sensitivity and robustness to these hyperparameters across different datasets and settings would be valuable.\n4. Handling Extreme Data Heterogeneity: The paper addresses data heterogeneity, but extreme cases of data heterogeneity can still pose challenges, as noted with gradient explosions in some scenarios. Further discussion and potential solutions to handle such extreme cases more effectively would improve the robustness of the framework.\n\nQUESTIONS:\n1. Computational Overhead\nHave you explored alternative methods for balanced classifier training to reduce computational overhead? It’s better to consider leveraging advanced optimization techniques or lightweight pre-processing steps to achieve balanced classifier training more efficiently.\n2. Evaluation on Diverse Datasets\nDo you plan to evaluate FedCCFA on more diverse, real-world datasets reflecting practical distributed concept drift scenarios? It’s better to extend evaluation to real-world datasets would provide a comprehensive understanding of the framework's applicability and robustness.\n3. Sensitivity Analysis of Hyperparameters\nHave you conducted a thorough sensitivity analysis of key hyperparameters across different datasets and settings? A detailed sensitivity analysis could help understand the impact of these hyperparameters on performance and provide guidelines for their selection.\n4. Handling Extreme Data Heterogeneity\nWhat solutions have you considered for effectively handling extreme cases of data heterogeneity and preventing gradient explosions? It’s better to investigate adaptive methods that dynamically adjust based on real-time training stability monitoring to handle extreme data heterogeneity more robustly\n\nLIMITATIONS:\n1. Adaptability to Various Drift Patterns: The paper evaluates performance under specific concept drift scenarios but may not cover all possible drift patterns. Expanding the evaluation to include a wider variety of drift patterns would provide a more comprehensive understanding of the framework’s adaptability.\n2. Real-World Implementation: The paper demonstrates effectiveness in controlled experimental settings. Discussing potential challenges and solutions for deploying FedCCFA in real-world environments, including scalability and communication efficiency, would strengthen the paper.\n3. Detailed Analysis of Classifier Clustering: The paper proposes class-level classifier clustering but provides limited analysis on the clustering's dynamics and potential pitfalls. A deeper analysis of how clustering decisions impact overall model performance and stability would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the impact of distributed concept drift on federated learning (FL), and proposes a novel FL framework, FedCCFA, to adapt to distributed concept drift with data heterogeneity (i.e., label distribution shift). Extensive experiments demonstrated the effectiveness and generality of the proposed method.\n\nSTRENGTHS:\n1. Federated learning (FL) under concept drift holds significant academic and practical value. The method proposed in this paper is both reasonable and effective.\n2. The experiments are extensive, not only comparing the proposed method with SOTA methods but also simulating various types of drift. This demonstrates the method's generality across different drift scenarios.\n3. The paper is well-organized and well-written, enhancing the clarity and impact of its findings.\n\nWEAKNESSES:\n1. In the introduction, the authors use the distribution of medical images as an example to describe data heterogeneity. However, the actual task addressed in the paper is label distribution shift, which does not align with the previous example. It is recommended that the authors directly use label distribution shift to describe the problem addressed in the paper\n2. The authors calculate the global feature anchor by averaging (i.e., Eq. (7)). However, given the presence of distributed concept drift among different clients, simply averaging may be unreasonable. It raises the question of whether concept drift adaptation should be considered.\n3. The adaptive alignment weight is calculated based on the entropy of the label distribution. However, there lacks the relevant motivation, references, and theoretical justification for this strategy.\n4. Federated learning (FL) under distributed concept drift primarily addresses concept drift across clients and over time. This is similar to the research problem of multistream classification under concept drift [1-3]. It is recommended that the authors discuss these works to enhance readers' understanding of related research.\n    [1] An adaptive framework for multistream classification, CIKM 2016 \n    [2] Fusion: An online method for multistream classification, CIKM 2017\n    [3] Online boosting adaptive learning under concept drift for multistream classification, AAAI 2024\n5. The paper does not provide open-source code and datasets, which compromises the reproducibility of the experiments.\n\nQUESTIONS:\ndiscussed in weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a federated learning framework called FedCCFA, which addresses the challenges posed by distributed concept drift and data heterogeneity. The authors introduce innovative solutions such as classifier clustering and adaptive feature alignment to enhance collaboration and improve model performance under various concept drift settings. The authors demonstrate the effectiveness of their approach through experimental results showing significant improvements over existing methods.\n\nSTRENGTHS:\n1. The paper tackles an important and relatively unexplored problem in federated learning, namely concept drift and data heterogeneity.\n2. The extensive experimental results demonstrate the effectiveness of FedCCFA, showing significant performance improvements over existing methods in various concept drift scenarios.\n\nWEAKNESSES:\n1. I am unable to fully understand the motivation of the paper. Figure 1 does not clearly illustrate how Decoupled Clustering addresses the real drift problem compared to previous decoupled methods. While it shows better performance than FedAvg and pure Decoupled methods, the underlying reasons remain unclear. Why does introducing clustering improve performance?\n2. The writing of the paper can be improved. For example, the specific meaning of \\(\\phi_{i,c}\\) is not explained, leading to significant confusion in understanding equation (4). The authors need to explain each symbol to enhance the readability of the paper. Additionally, the term \"balanced batch\" in line 160 needs to be explained, specifically how it was sampled.\n\nQUESTIONS:\n1. In the experiments, which classifiers were ultimately clustered in the class-level clustering? What does the final distance matrix look like?\n2. Equation (5) uses uniform aggregation to avoid privacy concerns. Why does this method avoid privacy concerns? Could taking the average directly lead to performance loss in classifiers?\n\nLIMITATIONS:\nThe authors point out that FedCCFA requires training balanced classifiers, which will increase computational complexity to some extent.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes to consider two cross-coupled important issues, i.e., federated learning and concept drift. To address such a challenging problem, the FedCCFA framework has been proposed composed of classifier clustering and feature alignment modules. The former is designed to cope with concept drift and enhance the generalization performance of the model. The latter is utilized to prevent data heterogeneity of clients. The overall design is sound, interesting, and intuitive. The whole paper is well-organized and well-written. The experimental evaluation is comprehensive and convincing.

STRENGTHS:
- A relatively new and important problem of federated learning under concept drift is addressed.
- Contributions of this work are in the framework perspective and the module/strategy design perspective.
- Both classifier clustering and feature alignment are sound and interesting ideas.
- The whole paper is well-organized.
- Comprehensive evaluation comparing many counterparts (including SOTAs) on benchmark datasets.

WEAKNESSES:
- Lack of a framework pipeline to intuitively explain the overall design.
- Time complexity analysis is missing.
- Visual classification results on the datasets are preferable.
- The source code is not opened.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper introduces a solutions based on clustering to group classifiers in the federated learning setting. Such an approach is meant to deal with concept drift in data so as to track and adapt the evolutions of the classifiers of the clients in federated learning. On of the core points is to align features and this is carried out by ether adaptive feature alignment procedure suggested in the paper.

STRENGTHS:
- The idea of letting the global/local models evolve over time is interesting and relevant
- The clustering is based on the classifier space (the feature extraction phase is fixed)
- Results in the concept drift are promising

WEAKNESSES:
Some technical points of the paper are not adequately addressed such as:
- the partitioning of feature extractor + classifier is relevant and requires to share a common feature extractor. It is not clear to this reviewer whether the feature extractor is fixed or adapted over time
- clustering of models requires to assume that the distribution of the model parameters shares some "locality" (e.g., this holds for linear models but not in general for NNs). 
- performance in stationary-conditions are less relevant. This is a crucial point (e.g., not suffered by other solutions) that requires to be further investigated

QUESTIONS:
- What's the need to partition the models into feature extractor and classifier? 
- Which is the assumption guaranteeing the clustering of classifier parameters in the parameter space?
- What's the reason of the reduction in accuracy in no-change condition (i.e., Table 1) and how to mitigate this point?

LIMITATIONS:
see previous box
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15