Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift

Authors: junbao  chen
jingfeng  xue
yong  wang
zhenyan  liu
lu  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:08:25

Abstract: Data heterogeneity is one of the key challenges in federated learning, and many efforts have been devoted to tackling this problem. However, distributed concept drift with data heterogeneity, where clients may additionally experience different concept drifts, is a largely unexplored area. In this work, we focus on real drift, where the conditional distribution $P(\mathcal{Y}|\mathcal{X})$ changes. We first study how distributed concept drift affects the model training and find that local classifier plays a critical role in drift adaptation. Moreover, to address data heterogeneity, we study the feature alignment under distributed concept drift, and find two factors that are crucial for feature alignment: the conditional distribution $P(\mathcal{Y}|\mathcal{X})$ and the degree of data heterogeneity. Motivated by the above findings, we propose FedCCFA, a federated learning framework with classifier clustering and feature alignment. To enhance collaboration under distributed concept drift, FedCCFA clusters local classifiers at class-level and generates clustered feature anchors according to the clustering results. Assisted by these anchors, FedCCFA adaptively aligns clients' feature spaces based on the entropy of label distribution $P(\mathcal{Y})$, alleviating the inconsistency in feature space. Our results demonstrate that FedCCFA significantly outperforms existing methods under various concept drift settings. Code is available at https://github.com/Chen-Junbao/FedCCFA.

Main paper: 
 introduction: 
 Federated Learning (FL) [28] is an emerging privacy-preserving machine learning paradigm that allows multiple clients to collaboratively train a global model without sharing their raw data. In FL, clients train the models on their local data and send the updated models to the server for aggregation. Driven by the growing need for privacy protection, FL has been widely applied in various real-world scenarios [8, 16, 30, 44] .One significant challenge in FL is data heterogeneity, which denotes the discrepancies in the data distributions across clients. Such discrepancies can hinder the convergence of the global model. However, existing works neglect a real-world setting where data heterogeneity and distributed concept drift simultaneously exist. Unlike conventional concept drift in centralized machine learning [11, 25, 26, 38] , distributed concept drift [19] involves multiple clients experiencing different concept drifts at different times. For example, for the same medical image, diagnoses can vary among doctors (i.e., concept drift across clients), and even a doctor may offer different diagnoses at different times (i.e., concept drift across times). Moreover, the distribution of medical images varies across hospitals (i.e., data heterogeneity). This setting significantly degrades the performance of most FL methods, especially those using a single global model, because the model cannot provide different outputs for the same input. A similar research problem is multistream classification [4, 15, 46] , where a sampling bias may exist between the distributions represented by source stream and target stream. Different from this problem, distributed concept drift focuses on the changing conditional distribution P (Y|X ) across clients and over time. For each client at any round, training and test data distribution are assumed to be similar.Several recent works have recognized the concept drift problem in FL. However, as discussed above, these single-model solutions [2, 3, 5, 14, 32] are ill-suited for distributed concept drift. FedDrift [19] considers distributed concept drift and employs multiple models to address this problem. However, FedDrift significantly increases the computation overhead for its distance measure and the storage overhead for multiple entire models. In addition, since the data heterogeneity can affect the loss estimation, FedDrift may fail to merge the models under the same concept. Experimental details are provided in Appendix C.7. To tackle distributed concept drift, we analyze how it affects the model training in FL. In this work, we focus on real drift [11, 38] in concept drift, where the conditional distribution P (Y|X ) changes. When P (Y|X ) varies across clients, the representation should not be affected, as the marginal distribution P (X ) is invariant. However, as shown in Figure 1 , when vanilla FedAvg is adopted and distributed concept drift occurs at round 100, the Frobenius norm of representation update increases drastically, suggesting that the drift leads to large gradients in the representation. Furthermore, the accuracy of vanilla FedAvg drops rapidly and cannot recover to the accuracy before the drift, indicating the unsuitability of the single-model solution.Motivated by the observations and analyses above, we decouple the network into an extractor and a classifier, and first train the classifier while fixing the extractor. After the classifier learns new conditional distribution, small gradients will be back-propagated to the extractor, and then we train the extractor. This decoupled method can effectively adapt to distributed concept drift, demonstrated by the much higher accuracy than FedAvg in Figure 1 . The small gradient norm of representation at drift round also indicates the small gradients back-propagated from classifier. However, some clients may share similar P (Y|X ), and pure decoupled methods neglect the fine-grained collaboration between local classifiers. This will make each client's local classifier overfit to its local data. To enhance generalization performance, we develop a class-level classifier clustering method. Our clustering method separates the classifier for each class (referred to as class classifier), and then aggregates clients' class classifiers trained under the same conditional distribution P (Y|X ). The clients under the same conditional distribution share the aggregated class classifiers. This aggregation reduces the bias introduced by any single client's data, contributing to improved generalization performance. As shown in Figure 1 , with the benefit of classifier clustering, the generalization performance is further improved, demonstrated by higher accuracy and smaller gradient norm. To remedy the data heterogeneity under distributed concept drift, we propose an adaptive feature alignment method, which aligns the feature spaces of the clients with the same conditional distribution P (Y|X ) and adjusts the alignment weight according to the entropy of label distribution.To summarize our contributions:1. We explore the impact of distributed concept drift on FL training and propose a class-level classifier clustering approach that not only adapts to this drift but also enhances generalization performance (Section 4.1).2. We propose clustered feature anchors to achieve feature alignment under distributed concept drift and propose an adaptive alignment weight to prevent severe data heterogeneity from impeding main task learning (Section 4.2).3. We propose FedCCFA, a federated learning framework with classifier clustering and feature alignment (see Section 4.3) . In Section 5, we conduct extensive experiments and empirical results demonstrate that FedCCFA can effectively adapt to distributed concept drift under data heterogeneity and significantly outperforms existing methods. 
related work: 
 Concept drift in FL. Concept drift has been extensively studied in centralized machine learning [11, 12, 25, 26, 38] . Several recent works have recognized the concept drift problem in FL and proposed methods to tackle this issue, including regularization [5] , continual learning [3, 14] and adaptive learning rate [2, 32] . These works assume that the conditional distributions P (Y|X ) of all clients' dataset are the same, so only a single global model is trained. However, as proposed in [19] and discussed in Section 1, distributed concept drift may exist in FL setting and the single-model solution fails to adapt to this drift. To address this problem, FedDrift [19] creates new models based on drift detection and adaptively merges models by hierarchical clustering. However, FedDrift still faces some challenges: 1) incorrect model merging caused by data heterogeneity; 2) high computation and communication overhead for measuring cluster distance; 3) high storage cost for maintaining multiple global models.Data heterogeneity in FL. Data heterogeneity hinders fast convergence when using vanilla FedAvg [28] . In this work, we focus on the heterogeneity of label distributions (referred to as label distribution skew). To alleviate this issue, previous works either focus on local training [1, 20, 22, 24, 29] or global aggregation [17, 18, 33, [39] [40] [41] . To further study how data heterogeneity affects FL, several recent works have focused on the representation of model, such as dimensional collapse [36] and inconsistent feature spaces [43, 45, 48, 50] . To align clients' feature spaces, FedFA [50] and FedPAC [43] regularize the ℓ 2 distance between local features and global anchors. To promote more precise alignment, FedFM [45] uses a contrastive-guiding method to further maximize the distance between the feature and non-corresponding anchors. However, severe data heterogeneity may significantly increase the loss of feature alignment, hindering the model convergence.Clustered FL and personalized FL. Some clustered FL methods group clients with the same data distribution into a cluster, which can also adapt to distributed concept drift. To group clients, several works measure the similarity based on gradient information [9, 10, 35] or training loss [13, 27] . However, these methods face the following challenges: 1) unknown number of clusters; 2) large computation and communication overhead for estimating the training loss; 3) considerable storage cost for multiple global models. Personalized FL methods [6, 23, 31, 37, 43, 49] are also robust to distributed concept drift, since each client trains a local model for its local data distribution. However, most personalized FL methods neglect the classifier collaboration among the clients with similar data distributions, limiting the performance of models. Different from personalized FL approaches, this work focuses on generalized FL, aiming to enhance generalization performance.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of efficiency and feature alignment in Conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 We consider a FL setting where there are K clients and a central server. The k-th client has a local dataset D k with data distribution P k (X , Y), where X is the input space and Y is the label space with C classes. Let ℓ(w; x, y) be the task-specific loss function associated with model w and data sample (x, y). The global objective of FL can be formulated as:EQUATIONwhere p k is the aggregation weight of the k-th client and F k (w) := E (x,y)∼D k [ℓ k (w; x, y)] is the local objective.To distinguish different types of concept drift, we decompose the joint distribution P (X , Y) as: P (X , Y) = P (X )P (Y|X ) = P (Y)P (X |Y). In this work, we focus on the real drift in concept drift.Real drift means that the conditional distribution at round t may be different from that at the previous round, i.e., P (t) (Y|X ) ̸ = P (t-1) (Y|X ). In addition, under distributed concept drift, this conditional distribution may vary across clients, i.e., Pi (Y|X ) ̸ = P (t) j (Y|X ). To address distributed concept drift, we decouple the model parameterized by w = {θ, ϕ} into a feature extractor f θ parameterized by θ and a classifier f ϕ parameterized by ϕ. Given a sample (x, y) ∈ X × Y, the feature extractor f θ : X → Z maps the input x into a feature vector z = f θ (x) in the feature space Z, and then the classifier f ϕ : Z → Y maps the feature vector z to the label space Y. All clients share the feature extractor and each client maintains its local classifier. To align clients' feature spaces, we introduce a regularization term G k (θ; A k ) into the local objective function. Here, A k represents a form of global information that client k uses to align its feature space. Let Φ = {ϕ 1 , ϕ 2 , . . . , ϕ k } denote the set of all clients' local classifier parameters and θ denote the parameters of shared feature extractor. The global objective can be reformulated as:EQUATION)where λ is the weight of feature alignment.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
methodology: 
 FedCCFA decouples the network into a feature extractor f θ and a classifier f ϕ . In FedCCFA, the classifier is the last layer of the model (i.e., a linear classifier), and the feature extractor is composed of the remaining layers. In this section, we first present two methods used in FedCCFA and then describe the design of FedCCFA.For better classifier collaboration, we introduce a new method for client clustering. Specifically, all clients share the same extractor and train their personal classifiers. Given the identical dimension reduction z = f θ (x), the personal classifier learns the local data distribution. Therefore, the linear classifiers with similar weights exhibit similar data distributions, especially the conditional distribution P (Y|X ). Different from existing clustered FL methods, FedCCFA directly uses the classifier weights for clustering, which significantly reduces the computation and communication cost. EQUATIONwhere η ϕ denotes the learning rate for classifier. Class-level clustering. Unlike FedPAC [43] that uses entire classifier for collaboration, FedCCFA conducts classifier clustering at class-level. Specifically, after receiving all balanced classifiers { φ(t) k }, the server separates these classifiers for each class (referred to as class classifier). Then, for each class c, the server measures the class-level distance D c (i, j) between client i and client j by their class classifiers φ(t) i,c and φ(t) j,c , where i, j ∈ I (t) . To effectively measure distance between high-dimensional vectors, we exploit MADD [34] and realize a measure based on cosine distance:EQUATIONwhere Cos(a, b) denotes the cosine distance between vector a and b.After To alleviate the inconsistency in feature spaces [45] , we introduce a regularization term into the local objective function to align clients' feature spaces. Compared to existing alignment methods [43, 45, 50] , our method is robust to two challenges: concept drift and the degree of data heterogeneity.Clustered feature anchors. Existing feature alignment methods suppose that all clients' conditional distributions P (Y|X ) are identical. Based on this, global feature anchors [45, 50] (also known as feature centroids [43] ) are leveraged to align clients' feature spaces. However, under distributed concept drift scenario, the conditional distribution P (Y|X ) may vary across clients, which can result in the differences in feature anchors. Therefore, clients require the global anchors that match their conditional distributions; otherwise, incorrect global anchors can mislead feature alignment. Motivated by this, we propose a feature alignment method using clustered feature anchors. Specifically, at round t, each client uses its feature extractor θ EQUATIONThen, with the help of our class-level clustering, we compute the global feature anchorA (t)m,c of class c for each cluster S (t) m,c :EQUATION|S (t) m,c | k∈S (t) m,c a (t) k,c , ∀S (t) m,c ∈ S (t) c (7) Finally, each client k ∈ S (t)m,c uses its global anchorsA (t) k = {A (t)m,c } C c=1 to align its feature space during local training. FedCCFA leverages the contrastive-guiding loss proposed in FedFM [45] , but uses clustered feature anchors. Let sim(u, v) denote the cosine similarity between u and v. Then the alignment loss function for a sample (x, c) with label c is defined as:G k (θ (t) k ; A (t) k ) = -log exp(sim(f θ (t) k (x), A (t) k,c )/τ ) C i=1 exp(sim(f θ (t) k (x), A (t) k,i )/τ ) (8)where f θ (t) k (x) is the representation vector of input x and τ denotes a temperature parameter. 𝜃 ! (#) 𝜃 ) (#) 𝜃 %,! (#) 𝜃 % (#) ⋯ 𝜃 (#-!)Extractor aggregation Adaptive alignment weight. In existing feature alignment methods [43, 45, 50] , the alignment weight is fixed and all clients use the same weight. However, severe data heterogeneity will significantly increase the gradients of alignment term, impeding the main task learning. Experimental results are presented in Table 5 . To balance the main task learning and feature alignment, it is essential to reduce the alignment weight under severe data heterogeneity. Note that, in this work, we focus on the label distribution skew and the degree of this heterogeneity can be reflected by the marginal distribution P ① 𝜃 ! , 𝜙 (#) 𝜃 % ! , 𝜙 % ! , 𝜙 % % ! ④ ⑦ 𝜙 % ! , 𝒜 % (!)EQUATIONwhere η θ denotes the learning rate for extractor and γ is the scaling factor. For the round T s to start feature alignment, we use the empirical value T s = 20 proposed in FedFM [45] .We now present FedCCFA, a federated learning framework to adapt to distributed concept drift under data heterogeneity. The pipeline of FedCCFA is shown in Figure 2 . The procedure of FedCCFA is formally presented in Algorithm 1 in Appendix A.Local training. At each round t, each selected client k ∈ I (t) updates its extractor θ(t)k by the global extractor θ (t) , and then starts local training procedure. Specifically, each client trains a balanced classifier (Equation 3) for client clustering. Then, it fixes its feature extractor and trains local classifier to adapt to concept drift. Finally, it trains its feature extractor (Equation 9). Since the classifier ϕ (t) k learns the conditional distribution P (t) k (Y|X ), concept drift will not lead to larges gradients in the representation. After local training, each client generates its local anchors for each class (Equation 6).Global aggregation. After receiving the local parameters and local anchors, the server starts the aggregation procedure. Specifically, the server performs client clustering with the help of balanced classifiers. Then, the server aggregates all feature extractors:θ (t+1) = 1 k∈I t |D (t) k | k∈I t |D (t) k |θ (t) k (10)Finally, according to the clustering results, the server aggregates local classifiers and local anchors for each cluster. 
experimental setup: 
 Datasets and models. We conduct experiments on three datasets, namely Fashion-MNIST [42] , CIFAR-10 [21] and CINIC-10 [7] . We construct two different CNN models for Fashion-MNIST and CIFAR-10/CINIC-10, respectively. Details of datasets and models are provided in Appendix B.1.Baselines. We compare FedCCFA against the methods falling under the following categories:(1) single-model methods without drift adaptation: FedAvg [28] , FedProx [24] , SCAFFOLD [20] and FedFM [45] ; (2) single-model methods with drift adaptation: Adaptive-FedAvg (shortened to AdapFedAvg) [2] and Flash [32] ; (3) personalized FL methods: pFedMe [37] , Ditto [23] , FedRep [6] , FedBABU [31] and FedPAC [43] ; (4) clustered FL methods: IFCA [13] and FedDrift [19] .Federated learning settings. We consider two FL scenarios: 20 clients with full participation and 100 clients with 20% participation. We use a widely considered non-IID setting [22, 39, 47] Implementation details. We run 200 communication rounds for all experiments. To comprehensively evaluate the generalized performance and drift adaptability, for each client, we report the generalized accuracy of its model on the whole test set. In particular, the conditional distributions P (Y|X ) of each client's training set and test set are the same at every round. For clustered and single-model methods, each client evaluates its global model; for the other methods, each client evaluates its personal model (or the combination of global extractor and personal classifier). Finally, we report the average accuracy across clients at the last round. For local training, we use SGD optimizer. The weight decay is 0.00001 and the SGD momentum is 0.9. For all datasets, we set local epochs E = 5. For the decoupled methods, extractor learning rate is 0.01 and classifier learning rate is 0.1; for the other methods, local learning rate is 0.01. For FedCCFA, scaling factor γ is 20.0, iterations of balanced training s is 5 and balanced batch size is 5 * C (5 samples per class). For classifier clustering, maximum distance ϵ and minimum samples used in DBSCAN algorithm are 0.1 and 1, respectively. More details of hyperparameters are provided in Appendix B.2.We focus on three points in our experiments: (1) the generalization performance of FedCCFA; (2) the adaptability to distributed concept drift compared to existing methods; (3) the effects of our classifier clustering and adaptive alignment weight. Full experimental results are provided in Appendix C. Performance without concept drift. First, we evaluate all methods under the no drift setting, and these results are the base performance compared against various concept drift settings. For each method, we run three trials and report the mean and standard deviation. 3 shows the results of our ablation study on the input of clustering. We see that using the weights of balanced classifiers can significantly increase the generalized accuracy. Furthermore, to demonstrate that accurate clustering results contribute to this improvement, we compare our method against an oracle method which has knowledge about the accurate clustering results at each round. Experimental results show that oracle method indeed increases the generalized accuracy and our method is comparable to the oracle method. These results demonstrate that classifier clustering can enhance the generalization performance and our method achieves accurate clustering.Effects of clustered feature anchors. To verify the efficacy of clustered feature anchors, we ablate FedCCFA by a variety of alignment methods. From Table 4 , we see that (1) feature alignment can boost the accuracy under data heterogeneity; (2) clustered feature anchors can facilitate more precise feature alignment under distributed concept drift.Effects of adaptive alignment weight. We turn the degree of data heterogeneity (α ∈ {0.01, 0.1, 0.5} and smaller α corresponds to more severe heterogeneity), and report the average accuracy in Table 5 . First, we conduct experiments with various fixed alignment weights {0, 0.1, 1.0}. Experimental results show that small alignment weight is better under severe data heterogeneity. In addition, under Dir(0.01), training without feature alignment outperforms training with feature alignment, indicating that feature alignment impedes main task learning under this setting. Then, we turn the scaling factor γ ∈ {20, 50, 100}. We see that our adaptive alignment weight is robust to the degree of data heterogeneity and the performance is not sensitive to the scaling factor. 
conclusion: 
 In this work, we take a further step towards distributed concept drift in federated learning. We have shown that FedCCFA can effectively adapt to distributed concept drift, and outperforms existing methods under various concept drift settings. Further experiments suggest that our classifier clustering method significantly enhances the generalization performance for decoupled FL methods. Besides, our clustered feature anchors can enhance the precision of feature alignment and our adaptive alignment weight can stabilize the training process when using feature alignment under severe heterogeneity. We hope that FedCCFA can inspire more works on classifier collaboration and other types of concept drift in federated learning.Limitations. FedCCFA takes some steps to train balanced classifiers, which increases computation cost. Although we reduce it by setting small iteration and batch size, it is preferable to remove these steps. We will investigate other efficient methods in the future. Besides, feature alignment, used in FedCCFA, FedFM and FedPAC, could impede main task learning under severe data heterogeneity. Therefore, research on the feature alignment under severe heterogeneity is also of interest. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work focuses on distributed concept drift in federated learning. Users' personal data is usually protected in federated learning and we do not find any possible adverse effects on society caused by this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original papers that produced the datasets, including Fashion-MNIST, CIFAR-10 and CINIC-10.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The procedure of FedCCFA is formally presented in Algorithm 1. Compared with decoupled FL methods [6, 31] , the sections with green color and red color are the additional computation and communication steps, respectively.Local training. At each round t, each selected client k ∈ I (t) first updates its extractor θ Sample clients I (t) ⊆ [K] and broadcast {θ (t) , ϕ (0) } to each selected client 5:for client k ∈ I (t) in parallel do end forEQUATIONk and {a Update global extractor:θ (t+1) = 1 k∈I t |D (t) k | k∈I t |D (t) k |θ (t) k 16: for class c ∈ [C] do 17: ∀i, j ∈ I (t) : calculate class-level distance D c (i, j) between φ(t) i,c and φ(t) j,c18:S (t) c ← DBSCAN(D c , ϵ) 19:for each cluster S end for 24: end forAll experiments are conducted using PyTorch and all results are produced with an Intel Core i7-13700K CPU and a NVIDIA GeForce RTX 4090 GPU. The required memory is around 2GB -7GB. We evaluate our method and all baselines on three datasets: ( an augmented extension of CIFAR-10, containing the images from CIFAR-10 (60,000 images) and a selection of ImageNet (210,000 images downsampled to 32*32). We use the same model architectures as those used in FedPAC [43] . For Fashion-MNIST, we construct a CNN model consisting of two convolution layers with 16 5*5 and 32 5*5 filters respectively, each followed by a max pooling layer, and two fully-connected layers with 128 and 10 neurons before softmax layer. For CIFAR-10 and CINIC-10, the CNN model is similar to the model used for Fashion-MNIST but has one more convolution layer with 64 5*5 filters.We visualize the data partition for CIFAR-10 and CINIC-10 under two degrees of data heterogeneity (i.e., Dir(0.1) and Dir(0.5)). Figure 3 shows the partition visualization under full participation (i.e., 20 clients with 100% participation). Since CINIC-10 is much larger than CIFAR-10 and we consider the setting where clients have different data sizes (i.e., quantity skew), data distribution of CINIC-10 is more biased. Figure 4 shows the partition visualization under partial participation (i.e., 100 clients with 20% participation). We randomly choose 20 clients for visualization. We see that data heterogeneity is more severe under full participation, due to more severe quantity skew.We employ SGD as the optimizer for all methods. The batch size is set to 64, the momentum is set to 0.9 and the weight decay is set to 0.00001. For all datasets, we set local epochs E = 5. In particular, for the decoupled methods, we first train the classifier for 1 epoch with learning rate η ϕ = 0.1, and then train the extractor for 5 epochs with learning rate η θ = 0.01, aiming to reduce the computation cost; for the other methods, we set local learning rate η = 0.01.For FedCCFA, we turn the scaling factor γ ∈ {5.0, 10.0, 20.0, 50.0, 100.0}, and set it to 20.0 according to the results in Figure 5 ; we turn the maximum distance ϵ ∈ {0.05, 0.10, 0.15, 0.20}, and set it to 0.10 according to the results in Figure 6 .For IFCA, we set the number of clusters k = 4, which is the oracle knowledge of the ground-truth clustering. For the following baselines, we tune their hyperparameters on CIFAR-10 under no concept drift and Dir(0.5), and select the best hyperparameters for all settings:FedDrift: we turn the detection threshold δ ∈ {0.1, 0.2, 0.5, 1.0, 5.0}, and set it to 0.5.FedFM: we turn the penalty coefficient λ ∈ {0.1, 0.5, 1.0, 5.0, 10.0, 50.0}, and set it to 1.0.FedPAC: we turn the penalty coefficient λ ∈ {0.1, 0.5, 1.0, 5.0, 10.0, 50.0}, and set it to 1.0.Flash: we turn the server learning rate η g ∈ {0.005, 0.01, 0.02, 0.1, 1.0}, and set it to 0.01.SCAFFOLD: we turn the server learning rate η g ∈ {0.005, 0.01, 0.02, 0.1, 1.0}, and set it to 1.0.pFedMe: we turn K ∈ {1, 5, 7}, and set it to 5; we turn λ ∈ {0.1, 0.5, 1.0, 5.0, 10.0, 15.0}, and set it to 1.0.Ditto: we turn penalty coefficient λ ∈ {0.1, 0.5, 1.0, 5.0}, and set it to 5.0. We evaluate all methods under no drift setting and Dir(0.1). Table 6 shows the generalized accuracy under Dir(0.1). We observe that, on CIFAR-10 and CINIC-10 with 20% participation, FedCCFA outperforms baselines. Besides, FedCCFA outperforms all decoupled methods and personalized FL methods, showing that FedCCFA boosts the generalization performance of classifiers under this setting. We find that gradient explosion occurs when adopting FedPAC and FedRep on CINIC-10 with full participation. Under this setting, data heterogeneity is much severe (as shown in Figure 3 ) and classifier is extremely biased. Such classifier could lead to gradient explosion when using high classifier learning rate, so the accuracy of FedRep may drop to 10% and FedPAC fails to finish FL training due to abnormal inputs to its classifier collaboration. Since FedBABU uses the initial global classifier during training and FedCCFA uses clustered classifiers, this problem is alleviated. We evaluate all methods under distributed concept drift and Dir(0.1). Table 7 presents the generalized accuracy under sudden drift setting. We find that FedCCFA outperforms all baselines on all datasets except for CINIC-10 with full participation. This weakness is caused by wrong clustering results. Specifically, in this case, the performance of global extractor is limited (the accuracy of the best method is only 53.86% under no drift setting), so the poor extractor cannot generate good feature vectors for the classifier. For this reason, the classifier cannot accurately learn the data distribution and its weights may provide wrong information. Besides, we find that gradient explosion also occurs when using FedFM on CINIC-10 with full participation. This is because severe heterogeneity will significantly increase the loss of feature alignment term when distributed concept drift occurs. FL methods without feature alignment are not affected, such as FedAvg, Flash and IFCA.We present the generalized accuracy decrease under sudden drift setting to show the adaptability to distributed concept drift. These results show that FedCCFA can effectively adapt to distributed concept drift except for CINIC-10 with full participation (due to the poor extractor discussed above).For personalized FL methods, thanks to their personal models or classifiers, distributed concept drift will not affect them. For all methods, we validate the adaptability to incremental drift, where three different concept drifts occur at round 100, 110 and 120, respectively. Table 9 shows that, under Dir(0.5), FedCCFA can also adapt to this drift setting and outperforms all baselines. To verify whether all methods could recover to the same accuracies once the original data distributions come back, we conduct experiments under the reoccurring drift setting. Table 10 shows that all methods can achieve this goal under reoccurring drift and Dir(0.5). To demonstrate the effectiveness of our classifier clustering, we conduct experiments on CIFAR-10 under sudden drift setting with full participation, and then visualize the distance matrices (i.e., the input of DBSCAN algorithm) in Figure 7 . The left shows the distance matrix of class 1. According to the concept drift setting in Section 5.1, for class 1, clients {0, 1, 2, 10, 11, 12} should be grouped and the others should be grouped. The right shows the distance matrix of class 5. For this class, clients {6, 7, 8, 9, 16, 17, 18, 19} should be grouped the others should be grouped. These two matrices demonstrate that our clustering method can effectively measure the distance between each client. FedCCFA involves class-level classifier aggregation (Equation 5) and class-level feature anchor aggregation (Equation 7). An intuitive manner is sample-number-based weighted aggregation, which is similar to the model aggregation in FedAvg. However, this manner requires clients to upload the sample number for each class, which may leak privacy about clients' category distributions.To address this problem, we consider the uniform aggregation, where all clients share the same aggregation weight. This aggregation manner does not involve other private information. We compare the two manners of aggregation. As shown in Table 11 , we observed that uniform aggregation even performs better, especially under concept drift setting. This may be because that uniform aggregation can prevent the classifier from overfitting to the classifier that is significantly biased due to data heterogeneity. Then, we present the two indices under Dir(0.1) in Figure 9 . We find that FedDrift can creates new model at drift rounds, but it cannot accurately merge the models under the same concepts. This is because, in addition to different concepts, different label distributions can also increase the cluster distances estimated by the loss value, which will prevent model merging. Since all clients choose the only global model at the beginning of training process and the loss delta will not exceed the threshold if no drift occurs, new global models will not be created under the no drift setting. To enhance client collaboration, FedPAC [43] introduces a classifier combination method, which uses feature statistics to estimate optimal combining weights. However, according to our experimental results, this operation is time-consuming. We compare the computation cost of classifier collaboration between FedCCFA and FedPAC. We count the time consumption for classifier collaboration after sudden drift occurs, and report the average value of 10 rounds. Table 12 shows that FedPAC is more time-consuming than FedCCFA, especially on large datasets (e.g., CINIC-10). 13 shows that FedCCFA with classifier fine-tuning is comparable to the most performing personalized methods. Justification: The main claims reflect this paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: I have read all comments and responses. Reviews appear to be consentaneous, with five scores of 7, 6, 6, 5, and 6. All concerns of reviewers have been fixed. It is recommended to accept this manuscript.

Reviews: review: paper_summary: paper_summary: The paper introduces a solutions based on clustering to group classifiers in the federated learning setting. Such an approach is meant to deal with concept drift in data so as to track and adapt the evolutions of the classifiers of the clients in federated learning. On of the core points is to align features and this is carried out by ether adaptive feature alignment procedure suggested in the paper.
strength_weakness: strength_weakness: Some technical points of the paper are not adequately addressed such as:
- the partitioning of feature extractor + classifier is relevant and requires to share a common feature extractor. It is not clear to this reviewer whether the feature extractor is fixed or adapted over time
- clustering of models requires to assume that the distribution of the model parameters shares some "locality" (e.g., this holds for linear models but not in general for NNs). 
- performance in stationary-conditions are less relevant. This is a crucial point (e.g., not suffered by other solutions) that requires to be further investigated
questions: questions: - What's the need to partition the models into feature extractor and classifier? 
- Which is the assumption guaranteeing the clustering of classifier parameters in the parameter space?
- What's the reason of the reduction in accuracy in no-change condition (i.e., Table 1) and how to mitigate this point?
limitations: limitations: see previous box
paper_summary: paper_summary: This paper proposes to consider two cross-coupled important issues, i.e., federated learning and concept drift. To address such a challenging problem, the FedCCFA framework has been proposed composed of classifier clustering and feature alignment modules. The former is designed to cope with concept drift and enhance the generalization performance of the model. The latter is utilized to prevent data heterogeneity of clients. The overall design is sound, interesting, and intuitive. The whole paper is well-organized and well-written. The experimental evaluation is comprehensive and convincing.
strength_weakness: strength_weakness: - Lack of a framework pipeline to intuitively explain the overall design.
- Time complexity analysis is missing.
- Visual classification results on the datasets are preferable.
- The source code is not opened.
questions: questions: See the weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes a federated learning framework called FedCCFA, which addresses the challenges posed by distributed concept drift and data heterogeneity. The authors introduce innovative solutions such as classifier clustering and adaptive feature alignment to enhance collaboration and improve model performance under various concept drift settings. The authors demonstrate the effectiveness of their approach through experimental results showing significant improvements over existing methods.
strength_weakness: strength_weakness: 1. I am unable to fully understand the motivation of the paper. Figure 1 does not clearly illustrate how Decoupled Clustering addresses the real drift problem compared to previous decoupled methods. While it shows better performance than FedAvg and pure Decoupled methods, the underlying reasons remain unclear. Why does introducing clustering improve performance?
2. The writing of the paper can be improved. For example, the specific meaning of \(\phi_{i,c}\) is not explained, leading to significant confusion in understanding equation (4). The authors need to explain each symbol to enhance the readability of the paper. Additionally, the term "balanced batch" in line 160 needs to be explained, specifically how it was sampled.
questions: questions: 1. In the experiments, which classifiers were ultimately clustered in the class-level clustering? What does the final distance matrix look like?
2. Equation (5) uses uniform aggregation to avoid privacy concerns. Why does this method avoid privacy concerns? Could taking the average directly lead to performance loss in classifiers?
limitations: limitations: The authors point out that FedCCFA requires training balanced classifiers, which will increase computational complexity to some extent.
paper_summary: paper_summary: This paper explores the impact of distributed concept drift on federated learning (FL), and proposes a novel FL framework, FedCCFA, to adapt to distributed concept drift with data heterogeneity (i.e., label distribution shift). Extensive experiments demonstrated the effectiveness and generality of the proposed method.
strength_weakness: strength_weakness: 1. In the introduction, the authors use the distribution of medical images as an example to describe data heterogeneity. However, the actual task addressed in the paper is label distribution shift, which does not align with the previous example. It is recommended that the authors directly use label distribution shift to describe the problem addressed in the paper
2. The authors calculate the global feature anchor by averaging (i.e., Eq. (7)). However, given the presence of distributed concept drift among different clients, simply averaging may be unreasonable. It raises the question of whether concept drift adaptation should be considered.
3. The adaptive alignment weight is calculated based on the entropy of the label distribution. However, there lacks the relevant motivation, references, and theoretical justification for this strategy.
4. Federated learning (FL) under distributed concept drift primarily addresses concept drift across clients and over time. This is similar to the research problem of multistream classification under concept drift [1-3]. It is recommended that the authors discuss these works to enhance readers' understanding of related research.
    [1] An adaptive framework for multistream classification, CIKM 2016 
    [2] Fusion: An online method for multistream classification, CIKM 2017
    [3] Online boosting adaptive learning under concept drift for multistream classification, AAAI 2024
5. The paper does not provide open-source code and datasets, which compromises the reproducibility of the experiments.
questions: questions: discussed in weaknesses.
limitations: limitations: yes
paper_summary: paper_summary: The paper addresses the significant challenge of data heterogeneity and distributed concept drift in federated learning. The authors propose a novel framework, which integrates classifier clustering and feature alignment to improve model performance and collaboration among clients facing different concept drifts.
The key contributions of the paper are as follows:
1. FedCCFA Framework: The proposed framework includes a method for clustering local classifiers at the class level and generating clustered feature anchors to enhance feature alignment. This clustering helps clients with similar data distributions share classifiers, thus improving the generalization performance of the global model.
2. Feature Alignment: The framework introduces an adaptive feature alignment technique that aligns clients' feature spaces based on the entropy of the label distribution. This method helps to alleviate the inconsistencies in feature space due to data heterogeneity.
3. Experimental Validation: Extensive experiments demonstrate that FedCCFA significantly outperforms existing federated learning methods under various concept drift settings. The results show that FedCCFA effectively adapts to distributed concept drift and enhances generalization performance.
strength_weakness: strength_weakness: 1. Computation Overhead: The proposed FedCCFA framework involves additional computational steps to train balanced classifiers, which increases the overall computational cost. While the authors attempt to mitigate this by setting small iterations and batch sizes, exploring more efficient methods to achieve balanced classifier training would be beneficial.
2. Limited Evaluation Scenarios: The experiments are primarily conducted on standard datasets. Including more diverse datasets, especially those with real-world distributed concept drift scenarios, would strengthen the validation of the framework's general applicability and robustness.
3. Sensitivity to Hyperparameters: The effectiveness of FedCCFA relies on several hyperparameters, such as the maximum distance in DBSCAN and the scaling factor. While the authors provide some tuning, a more thorough analysis of the sensitivity and robustness to these hyperparameters across different datasets and settings would be valuable.
4. Handling Extreme Data Heterogeneity: The paper addresses data heterogeneity, but extreme cases of data heterogeneity can still pose challenges, as noted with gradient explosions in some scenarios. Further discussion and potential solutions to handle such extreme cases more effectively would improve the robustness of the framework.
questions: questions: 1. Computational Overhead
Have you explored alternative methods for balanced classifier training to reduce computational overhead? It’s better to consider leveraging advanced optimization techniques or lightweight pre-processing steps to achieve balanced classifier training more efficiently.
2. Evaluation on Diverse Datasets
Do you plan to evaluate FedCCFA on more diverse, real-world datasets reflecting practical distributed concept drift scenarios? It’s better to extend evaluation to real-world datasets would provide a comprehensive understanding of the framework's applicability and robustness.
3. Sensitivity Analysis of Hyperparameters
Have you conducted a thorough sensitivity analysis of key hyperparameters across different datasets and settings? A detailed sensitivity analysis could help understand the impact of these hyperparameters on performance and provide guidelines for their selection.
4. Handling Extreme Data Heterogeneity
What solutions have you considered for effectively handling extreme cases of data heterogeneity and preventing gradient explosions? It’s better to investigate adaptive methods that dynamically adjust based on real-time training stability monitoring to handle extreme data heterogeneity more robustly
limitations: limitations: 1. Adaptability to Various Drift Patterns: The paper evaluates performance under specific concept drift scenarios but may not cover all possible drift patterns. Expanding the evaluation to include a wider variety of drift patterns would provide a more comprehensive understanding of the framework’s adaptability.
2. Real-World Implementation: The paper demonstrates effectiveness in controlled experimental settings. Discussing potential challenges and solutions for deploying FedCCFA in real-world environments, including scalability and communication efficiency, would strengthen the paper.
3. Detailed Analysis of Classifier Clustering: The paper proposes class-level classifier clustering but provides limited analysis on the clustering's dynamics and potential pitfalls. A deeper analysis of how clustering decisions impact overall model performance and stability would be beneficial.

Comments: comment: Thank you for the authors' response. All of my concerns have been addressed, and I will increase the score to 6.
Thank you for your responses, and I will raise my score.
Thanks for the authors' detailed response and additional experiments. My concerns have been adequately addressed. I will keep my positive score.
Thank you for your responses, I will stick my score with a higher confidence.