Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains

Authors: lei  wang
jieming  bian
letian  zhang
chen  chen
jie  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 04:50:25

Abstract: Federated learning (FL) allows collaborative machine learning training without sharing private data. While most FL methods assume identical data domains across clients, real-world scenarios often involve heterogeneous data domains. Federated Prototype Learning (FedPL) addresses this issue, using mean feature vectors as prototypes to enhance model generalization. However, existing FedPL methods create the same number of prototypes for each client, leading to cross-domain performance gaps and disparities for clients with varied data distributions. To mitigate cross-domain feature representation variance,  we introduce FedPLVM, which establishes variance-aware dual-level prototypes clustering and employs a novel $\alpha$-sparsity prototype loss. The dual-level prototypes clustering strategy creates local clustered prototypes based on private data features, then performs global prototypes clustering to reduce communication complexity and preserve local data privacy. The $\alpha$-sparsity prototype loss aligns samples from underrepresented domains, enhancing intra-class similarity and reducing inter-class similarity. Evaluations on Digit-5, Office-10, and DomainNet datasets demonstrate our method's superiority over existing approaches.

Main paper: 
 introduction: 
 Federated Learning [21] (FL) is a novel distributed learning framework that enables clients to collaboratively train a global model using their respective local datasets, thereby preserving data privacy. FL offers distinct advantages over traditional distributed learning methodologies by mitigating communication costs and addressing privacy concerns, which has led to its increased adoption across various sectors. Despite these benefits, FL comes with its own challenges, especially in terms of data heterogeneity [41] . In FL, clients gather private data from unique sources, resulting in non-independent and identically distributed (non-IID) datasets. Such non-IID distributions can lead clients toward their local optima, potentially diverging from the global objective. Consequently, this may impede convergence rates and diminish overall model performance [17] .To address the aforementioned issue of data heterogeneity, numerous FL techniques have been devised [20, 5, 41, 29, 14, 12] . While these methods have indeed improved convergence, their focus on non-IID scenarios remains largely restricted. Typically, they assume that the data across clients Figure 1 : Illustration of federated learning with heterogeneous data domains. The Vanilla column depicts the local feature distribution of the standard FedPL approach, obtaining average local and global prototypes directly. Proposed method showcased in the adjacent column yields a larger inter-class distance and a reduced intra-class distance. Note that without capturing variance information, even for hard domains, local averaged prototypes for each class can be well distinguished while the feature vectors are still mixed up. Both methods illustrate noticeable variations in domain characteristics across datasets, as detailed in Fig. 4 .pertain to a single domain, attributing non-IID distribution to label skew alone. Yet, in more realistic scenarios, clients gather data according to their unique preferences, making it impractical to assume identical domain origins for local data. Instead, the data often stems from heterogeneous domains, resulting in varied feature distributions.Current approaches [10, 32] to FL in heterogeneous domains aim to develop a global prototype for each label category. These prototypes are designed to minimize the distance between the individual training samples and their corresponding category's global prototype. Typically, a global prototype is computed as the mean of the local prototypes from each client, whereas a local prototype is itself the average of the representations of samples within the same category. Several studies have proposed unique designs to enhance training performance in these settings. For example, one work [33] enables contrastive learning at the client level by facilitating the exchange of local prototypes, thus promoting inter-client knowledge transfer. However, this exchange raises privacy concerns and substantially increases communication overhead. Another study [10] opts for a clustering approach to identify representative prototypes, thereby preserving domain diversity and preventing bias towards predominant domains. This technique has proven effective, especially in scenarios with disproportionate client distribution across domains.Although these methods improve overall performance across domains, they do not address the unequal learning challenges that arise from domain diversity. For instance, on the Digits dataset [42] , methods may perform well in domains like MNIST [7] but underperform in more challenging domains such as SVHN [24] . This discrepancy is evident in representation distributions, as exemplified in Fig. 1 , where 'easy' domains show tight clustering of samples within the same category and clear separation between different categories, facilitating accurate classification. In contrast, 'hard' domains display looser clustering, increasing the likelihood of misclassification, particularly for samples near category boundaries. Addressing these disparities is crucial for the equitable advancement of FL methodologies across diverse client domains.Considering the challenges posed by unequal learning due to domain diversity, we introduce a novel method, termed FedPLVM (short for Federated Prototype Learning with Variance Mitigation). FedPLVM devises two main mechanisms. Firstly, we develop a dual-level prototype clustering mechanism that adeptly captures variance information, a significant improvement over previous methodologies that rely on averaging local training samples' representations to derive local prototypes. Our local-level clustering generates multiple local clustered prototypes within each domain. To further mitigate increased communication costs and privacy concerns arising from transferring a comprehensive set of diverse prototypes for each class from every client, we employ global-level prototype clustering on the server side. Secondly, by capturing the variance information through clustered prototypes, we design an innovative α-sparsity prototype loss to enhance the training process. To prevent the intermingling of feature distributions from 'hard' clients, instead of simply maximizing the feature-level distance between each local instance and the prototypes of other classes, we refine this distance by elevating it to an α power, with α being a value between 0 and 1. This modification effectively repels other prototypes, thereby introducing greater sparsity in the inter-class feature distributions. Moreover, we incorporate an intra-class similarity correction term to lessen the feature-level distance among intra-class samples, thereby concentrating the α-sparsity prototype loss on amplifying the feature-level distance across inter-class samples. Collectively, these two strategies empower FedPLVM to reasonably leverage variance information, thus equilibrating fairness between 'hard' and 'easy' learning domains and enhancing overall learning performance. Consequently, FedPLVM emerges as a reliable approach for Federated Learning in contexts characterized by heterogeneous data domains. Our main contributions are outlined as follows:• This study delves into FL with heterogeneous data domains, examining why models exhibit varying performance across different domains. We identify a fundamental limitation in existing methods: their inability to effectively address the disparate learning challenges inherent in diverse domains.• To tackle these uneven learning challenges, we introduce a novel approach, FedPLVM. This method incorporates a dual-level prototype clustering method, capturing the rich sample representation variance information while ensuring communication efficiency. Additionally, we develop a new α-sparsity prototype loss to address learning difficulties more equitably.• Extensive experiments conducted on the Digit-5 [42] , Office-10 [8], and DomainNet [26] datasets demonstrate the superior performance of our proposed method when compared with multiple state-of-the-art approaches.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Details can be found in Sec. 1. 
related work: 
 Federated Learning. FL aims to train a global model through collaboration among multiple clients while preserving their data privacy. FedAvg [21] , the pioneering work in FL, demonstrates the advantages of this approach in terms of privacy and communication efficiency by aggregating local model parameters to train a global model. A significant challenge in FL is data heterogeneity, often manifested as non-IID (independently and identically distributed) data. Subsequent research, following FedAvg, has primarily focused on addressing data heterogeneity to enhance training performance in FL environments. Specifically, studies such as [18, 2, 30] have improved performance by incorporating a global penalty term to mitigate discrepancies. Other works, e.g., [32, 23] , have sought to maximize feature-level agreement between local and global models to further boost performance. Recent works [10, 32, 19, 40] have explored data heterogeneity arising from clientspecific domain diversity, while overlooking the challenges of unequal learning across different domains. In this paper, we propose a novel approach using dual-level prototype clustering to capture essential local variance information. Additionally, we introduce a new α-sparsity loss, specifically designed to tackle the challenges of learning in diverse domains, thereby facilitating the development of a more generalizable global model.Prototype Learning. Prototype learning has been extensively explored in various tasks, such as transfer learning [27, 13] , few-shot learning [31] , zero-shot learning [11] , and unsupervised learning [34] . The concept of a prototype in this context refers to the average feature vectors of samples within the same class. In the FL literature, prototypes serve to abstract knowledge while preserving privacy. Specifically, approaches like FedProc [23] and FedProto [32] focus on achieving feature-wise alignment with global prototypes. FedPCL [33] employs prototypes to capture knowledge across clients, constructing client representations in a prototype-wise contrastive manner using a set of pre-trained models. FPL [10] highlights the use of cluster-based and unbiased global prototypes to tackle the challenges in FL where clients possess domain-diverse data. Our study addresses a similar issue as FPL but with distinct emphases. While FPL concentrates on the disparities in the number of clients across domains, aiming to mitigate the bias in the overall model caused by domains with more clients, our work focuses on the intrinsic learning challenges that vary across domains. Contrastive Learning. Contrastive learning is a promising self-supervised learning technique. The work by [25] constructs pairs of positives and negatives for each sample and applies the InfoNCE loss to compare these pairs. Another work by [15] extends contrastive learning from self-supervised to fully supervised settings, utilizing both label information and contrastive methods. Additionally, some studies [16, 39] integrate contrastive learning into local training to enhance performance in FL. Instead of the conventional InfoNCE loss, our approach introduces a new α-sparsity loss, aiming to further reduce similarity among inter-class sample features while amplifying similarity among intra-class samples.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Details can be found in Sec. J.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Regular FL Scenario. Consider the classic FL scenario: there exist K clients and one server, which aims to assist all clients to train a common machine learning model without sharing their private data, denoted by D k = {x i , y i } N k i for client k. Formally, the global objective of FL can be formulated as:EQUATIONwhere L k is the local loss function for client k, w and N = K k=1 N k denote the shareable model and the total number of samples among all clients respectively. Domain Shift in FL. In the simplest FL setting, the label and feature distribution are the same between two clients i and j, formally P i (y) = P j (y) and P i (x|y) = P j (x|y). Extending such a scenario to the heterogeneous data setting brings us to the point of domain shift, also denoted as feature non-IID setting. Domain shift is caused by distinctive feature distributions among clients, that is P i (x|y) ̸ = P j (x|y), though their label space is still the same. Federated Prototype Learning. To handle the FL domain shift problem, previous works [32, 10] divide the classification network into two parts: feature extractor and classifier. The feature extractor h : R V → R D maps a sample x ∈ R V into the feature space and generates the corresponding feature vector z = h(x) ∈ R D . Then the classifier f : R D → R M outputs the M -class prediction f (z) = f (h(x)) ∈ R M given the feature vector z. The intuition of FedPL is adjusting the feature extractor to generate a consistent feature distribution among different domains. Similar to the global shareable model, the straightforward solution computes the average feature vector of all samples belonging to the same class for further processing, called the prototype. Due to the distributed property of FL, we cannot collect all samples via the server, which makes the above-mentioned idea a two-step procedure:EQUATIONwhere K = {1, 2, . . . , K} is the set of clients, M = {1, 2, . . . , M } is the set of classes. pm k and ḡm denotes the local prototype of class m on client k and the global prototype of class m, respectively. D m k is the subset that all samples belonging to class m in the local dataset D k . Upon obtaining the prototypes, most FedPL works tend to design a loss function that approximates the prototype of one's own class while staying away from the prototypes of others. Local Prototype Clustering. To mitigate the impact of domain variance in FedPL, we revise the two-step averaged prototype generation process to a dual-level clustering algorithm. In Fig. 1 , the pronounced diversity in domain variances among clients becomes evident, intricately linked to the complexity of their datasets. For instance, comparing the feature distribution between Synth and MNIST shows a visibly more scattered pattern in Synth due to its higher complexity, whereas MNIST, being comparatively easier to learn, displays a more structured distribution. It becomes evident that computing a single local averaged prototype for one class per client is 'unfair', given the differing richness of feature distribution information among clients. Particularly, for 'hard' clients with complex datasets such as SVHN, employing multiple local prototypes becomes imperative to capture the scattered feature distribution comprehensively. Hence, we propose the first-level prototype generation, namely local prototype clustering. Instead of a straightforward averaging approach, our method involves initially clustering the feature vectors of all same-class local samples, forming several local clustered prototypes as a set of local representations.EQUATIONwhere J k,m and p m k,j represents the number of local clustered prototypes and the j-th local clustered prototype of class m on client k clustered from the set of feature vectors {h(x i )|(x i , y i ) ∈ D m k }. It is important to note that the number of local clustered prototypes may differ across various classes and clients. To determine these prototypes, we employ the parameter-free clustering algorithm, FINCH [28] , utilizing cosine similarity as the clustering metric. We also conduct additional experiments on comparison with other clustering methods in Supplementary Material. This choice ensures the alignment of the number of local clustered prototypes with the sparsity of the domain distribution. By leveraging this approach, we enhance the representation of distinct feature distributions, preventing the overdrift of local averaged prototypes towards densely concentrated regions in the feature space.Global Prototype Clustering on Server. Distributing all local clustered prototypes among clients poses challenges due to the extra communication cost and privacy concerns. Hence, we introduce the second level of prototype generation, namely global prototype clustering, which can be formulated as:EQUATIONwhere C m and g m j denote the number of global clustered prototypes and the j-th global clustered prototype of class m on the server from the local collected prototypes set P m = {P m k } K k=1 . Through the second-level global clustering, we significantly reduce the number of prototypes that the server must distribute to clients compared to distributing all locally clustered prototypes, alleviating potential communication costs. Dual-level clustering also addresses privacy concerns that arise from original local clustered prototypes potentially revealing client-specific features and the corresponding results can be found in Supplementary Material.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We will provide the code soon in the following github link https://github. com/TriStonesWang/FedPLVM. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
α-sparsity prototype loss: 
 Unlike previous methods in FedPL, which generate a single prototype per class, our approach employs dual-level clustering to create multiple prototypes for each class, thereby capturing valuable variance information. This multiplicity of prototypes could potentially lead to overlapping feature representations among different classes, especially in challenging client scenarios. To mitigate this risk, we introduce a novel α-sparsity prototype loss, inspired by the InfoNCE-based loss. Our newly designed α-sparsity prototype loss enhances inter-class feature distribution sparsity and maintains balanced feature representation distances within classes, unlike the traditional InfoNCE-based loss. The formulation of the α-sparsity prototype loss is detailed below:EQUATIONThe first contrastive term can be formulated as:EQUATION)where τ is the temperature hyper-parameter that controls the concentration strength of the similarity [35] , and G = {G m } M m=1 is the set of all global clustered prototypes. s α (•, •) in the first term is the modified α-sparsity cosine similarity metric between the feature vector h(x i ) and the prototype g m from class m and can be formulated as:EQUATIONwhere α ∈ (0, 1). This metric serves to compel the feature extractor to generate feature outputs closely aligned with the global clustered prototypes of their respective classes while distancing them from other global prototypes. It achieves this by maximizing intra-class similarity and minimizing inter-class similarity. As our feature vectors have positive values, cosine similarity always falls within the range of [0, 1]. By introducing a sparsity factor α to the similarity and applying it as a power, all similarity values are elevated. However, due to the smaller denominator component representing similarity with other classes in the L α function, the impact of the concave function (•) α is more pronounced. This emphasis on maximizing inter-class distance is what we denote as α-sparsity operation, which directs more attention toward expanding the overall feature distribution to a broader range. Consequently, it mitigates the issue of feature distributions within one class being excessively dispersed and overlapping with feature distributions of other classes.This modification does indeed lead to an unintended consequence: an increase in intra-class distance due to heightened similarity between the feature vector and prototypes of the corresponding class, resulting from the concavity of (•) α . This brings us to the second correction term of our α-sparsity prototype loss:EQUATIONThe latter term within the α-sparsity prototype loss serves as a corrective measure, which focuses on pushing the average cosine similarity closer to 1. This correction measure aims to counterbalance the increase in intra-class distance stemming from the adjustment introduced by α.An additional Cross-Entropy (CE) loss [6] is employed to train the classifier and derive prediction results, which can be formulated as:EQUATIONThe total local loss, combining the previously mentioned loss functions, is expressed as follows:EQUATIONHere, λ serves as a hyper-parameter that regulates the balance between the α-sparsity prototype loss and the CE loss. This formulation allows for a unified and weighted consideration of the α-sparsity prototype loss.  
experiments: 
 Datasets. We evaluate our proposed algorithm on three datasets: Digit-5 [42] , Office-10 [8] and DomainNet [26] . Digit-5 is a dataset for digits recognition, consisting of 5 domains: MNIST, SVHN, USPS, Synth and MNIST-M. Office-10 is a dataset for office item recognition, consisting of 4 domains: Amazon, Caltech, DSLR and Webcam. DomainNet is a large-scale classification dataset, consisting of 6 domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch.Baselines. We compare our algorithm with classic FL methods: FedAvg [21] , FedProx [18] , FedPL methods: FedProto [32] , FedPCL [33] , FPL [10] and FL method on feature skew: FedFA [43] .Implement Details. We employ the ResNet10 [9] as our backbone model, configuring the feature vectors' dimension to 512. The optimization is done using the SGD optimizer, employing a learning rate of 0.01, momentum of 0.5, and a weight decay of 1e -5. For Digit-5, Office-10 and DomainNet, we use 5, 4 and 6 clients respectively. The client data is independent and identically distributed (i.i.d.) and non i.i.d. results can be found in Supplementary Material. It is important to note that each client operates within distinct data domains, meaning different datasets. for DomainNet. The batch size is set at 32 for all datasets. Details regarding hyper-parameter settings will be elaborated in Sec. 5.2. For fair comparisons, we conduct each setting for 5 experiments and report the average result.We compare our proposed method with the state-of-the-art methods using the Digit-5, Office-10 and DomainNet datasets, as detailed in Tab. 1, Tab. 2 and Tab. 6. Our method demonstrates significant improvements in average accuracy over baseline methods for both datasets. A closer examination of domain-specific results reveals a more pronounced enhancement in performance on domains that are more challenging to learn. For instance, within the Digit-5 dataset, our method achieves a 5.3% increase in accuracy for the SVHN domain, which is more difficult, and a 0.58% increase for the MNIST dataset, considered easier. This aligns with the goal of our method, which is to To evaluate the effectiveness of each component within our proposed methodology, we conducted a series of ablation studies using the Digit-5 dataset.In this subsection, we focus on examining the impact of dual-level clustered prototypes, as demonstrated by the results in Tab. 3. The first row illustrates outcomes derived from calculating local prototypes by averaging features of samples within the same class, and global prototypes through the direct averaging of these local prototypes. This approach, utilizing straightforward averaging for both levels, yields the least effective performance. This is attributed to its failure to capture variance information, a non-trivial aspect in this context. Moreover, adopting a solely global clustering approach does not significantly enhance performance. This is attributed to the fact that while global clusters incorporate inter-domain variance, they overlook the critical aspect of high intra-domain sample variance, particularly in domains that are challenging to learn. To elucidate this, we present a t-SNE visualization analysis in Fig. 3 comparing the three prototype generation methods. Our approach fosters a more generalizable decision boundary, as illustrated in the visualization. This ability to effectively capture variance at both local and global levels is key to why our dual-level clustering method outperforms the others. preserves variance information but raises significant privacy concerns, unlike our dual-level clustering method, which offers enhanced privacy protection. As demonstrated in Tab. 4, directly transferring all local prototypes yields performance comparable to our dual-cluster approach. However, it requires transmitting approximately five times as many prototypes in each training round. To evaluate the specific impact of our proposed α-sparsity prototype loss, we conduct experiments comparing both contrastive and corrective loss terms. Results in Tab. 5 demonstrates that employing the contrast term led to a 1.46% improvement in final average accuracy, while the correction term resulted in a 0.99% enhancement. Combining all components yields the best performance, showcasing a 2.29% improvement. Our investigation also focuses on the sparsity parameter α, depicted in Fig. 4 . We find that varying α within the range of (0, 1) consistently outperforms the baseline setting with α = 1. After careful consideration of numerical stability, we opt for an α value of 0.250.Comparing our method to the prior work FPL, which also integrates its proposed prototype loss with the CE loss, we explore different prototype loss weights, denoted by λ. The results depicted in Fig. 4 substantiate the superiority of our approach over FPL across various λ settings. To assess the impact of the contrastive temperature (τ ) on our model's performance, we conduct the following experiment. The results, depicted in Fig. 5 , demonstrate that our method consistently surpasses the baselines across a range of temperatures (refer to Tab. 1 and Tab. 2 for details, where FPL exhibits the highest baseline performance at 66.88% and 58.55% respectively). Further analysis indicates that an optimal temperature setting for our model on Digit-5 is τ = 0.070. Another results on Office-10 identifies an optimal temperature setting for our model at τ = 0.045, while we maintain τ = 0.070 for stability in performance comparisons. It is noteworthy that temperatures either significantly higher or lower than this value lead to training difficulties due to numerical instability.  
conclusion: 
 In this paper, we start by noting the significant difference in domain-specific representation variance across various datasets within the context of federated learning involving heterogeneous data domains. Traditional methods relying on averaged prototypes, calculated as the mean values of feature representations from samples within the same class, consistently fail to capture this essential local information. Our proposed approach, FedPLVM, addresses this issue by implementing a dual-level prototype generation method. This method leverages first-level local clustering to manage variance information and employs second-level global clustering to streamline communication complexity while ensuring privacy. Additionally, we introduce an α-sparsity prototype loss that prioritizes expanding the inter-class distance, considering the diverse cross-domain variances, and includes a correction term to subsequently reduce the intra-class distance. Comprehensive experiments have been conducted to validate the effectiveness of FedPLVM, demonstrating a significant accuracy improvement over state-of-the-art federated prototype learning methods.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Details can be found in Sec. 5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Details can be found in Sec. 5.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Details can be found in Sec. J.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Details can be found in Sec. 5.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The supplementary material is organized into the following sections:B: Detailed algorithm of our proposed method, FedPLVM.C: Further results of our experiments.D: Additional quantitative results comparing FedPLVM and other baseline methods under the label non-i.i.d. setting.E: Additional quantitative results comparing different clustering algorithms for the prototype clustering procedure of FedPLVM.F: Additional quantitative results comparing FedPLVM and FPL [10] under the unbalanced client domain distribution setting.FedPLVM.H: Additional quantitative results comparing different global prototype generation methodologies of FedPLVM.I: Additional quantitative results analyzing the effectiveness of local prototypes clustering on different domains.J: Discussion on the wider range of scenario of our method, including limitation and broader impact.We summarize our proposed method, FedPLVM in the Alg. 1.Input: Communication rounds Due to page limitation, we put some further experimental results in Sec. 5.1 here: We expanded our method's assessment on Digit-5 and Office-10 within a non-i.i.d. label setting.Utilizing the Dirichlet method (α = 0.5), we shaped the data distribution and generated non-i.i.d. datasets for individual clients. Notably, our approach showcased substantial accuracy improvements compared to baseline methods across both datasets. Non-i.i.d. distributions are visualized in Fig. 8 , the variation in features across non-identically distributed data is reflected by the colors of the dots, whereas the non-identical label distribution (α = 0.5) is denoted by the sizes of the dots. For the top figure in Fig. 8 , client index 0, 1, 2, 3, 4 own the datasets MNIST, SVHN, USPS, Synth and MNIST-M respectively; for the bottom figure, client index 0, 1, 2, 3 represent the dataset domains Amazon, Caltech, DSLR and Webcam repectively. The outcomes are presented in Tab. 7 and Tab. 8. Consistent with our findings in the i.i.d. label setting, we observed a significant performance boost in domains that pose greater learning challenges. For instance, within the Digit-5 dataset, our method achieved a 2.24% accuracy surge in the Synth domain-a more complex task-while showing a 0.94% increase in accuracy for the comparatively easier USPS dataset. This reiterates our method's aim: addressing disparate learning difficulties among domains, fostering fairness, and bolstering performance, especially in tougher domains. Similar advancements were evident within the Office-10 dataset. In our paper, FINCH is employed for our dual-level prototype clustering due to its parameter-free nature.Here we select the one with the minimum number of cluster centers among several possible clustering schemes generated by FINCH. This characteristic obviates the need for hyper-parameter tuning typically required in clustering methods, such as selecting the number of centers in K-Means algorithm. However, this does not imply exclusivity to the FINCH clustering algorithm within our method. Our approach is compatible with various clustering algorithms, including simpler methods like K-Means. The experiments result in Tab. 9 show that with a carefully tuned K-Means algorithm (precisely determining the appropriate number of centers), our method with K-Means achieves performance comparable to that achieved using FINCH. Conversely, poorly tuned parameters result in weaker performance with the K-Means clustering. This reinforces our choice of FINCH, for its parameter-free advantage. In our method, the server only receives clustered local prototypes, making it challenging to reconstruct the clients' local datasets. We also employ a differential privacy (DP) technology to validate the impact of FedPLVM. Each client trains local model by DP-SGD [1] to perturb model parameters.The noise multiplier is determined by [22, 4, 3] . The privacy budget ϵ and approximate parameter δ are set as 4.0 and 1e -5 respectively for a (ϵ, δ)-DP setting. Meanwhile, we also incorporate the clustered local prototypes with the privacy protection technique. We set the scale parameter s = 0.05 and the perturbation coefficient p = 0.1 for the Gaussian noise distribution generation of our local clustered prototypes. As shown in Tab. 11, the average accuracy drops at most 0.77% if we employ the privacy protection only for the prototypes. The approximate DP causes a slightly larger decrease in accuracy, up to 2.45%. Note that even with the privacy protection technologies, our method reaches a comparable performance to the most advanced baseline in Tab. 1. The determination of the number of global prototypes for a class is contingent on the associated learning challenges. In instances where a class within a domain presents significant learning difficulties, it is characterized by large variance. Consequently, employing a single global prototype, reducing the sufficiency of variance information, could decay the learning performance. Incorporating multiple global prototypes for such a class can introduce additional variance information, thereby enhancing the learning process. We provide averaged experiment results in Tab. 12. (Note in the main paper we only report the single experiment to be consistent with the corresponding visualization.) The results validate the necessity of our dual-level clustering. Local prototype clustering is beneficial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Easy domains tend to show tight clustering within the same category and clear distinctions between different categories, facilitating accurate classification. In contrast, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average data suffices for easy domains but falls short for hard domains.Our proposed method captures more feature distribution variation information by introducing the local prototypes clustering, since clustering provides more prototypes compared to simply averaging especially considering the sparse distribution in hard domains as shown in Fig. 3 . For easy domains, where the average is sufficient, our method generates fewer prototypes.To demonstrate this, we refer to Fig. 9 . The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08% in experiment of impact of local prototypes clustering in Tab. 3 also supports this observation.J Discussion on Wider Range of Scenarios J.1 Limitation.While FedPLVM introduces innovative strategies to address domain heterogeneity in federated learning, it also comes with certain limitations and challenges. For example, noisy labels, or incorrectly labeled data instances, are a common issue in real-world datasets. A few previous works has discussed the related impact in FL scenario [37, 38, 36] . This noisy case will bring two problems for prototype learning in FL: Prototype Distortion. Noisy labels can lead to distorted prototype representations, as the local prototypes clustered for each class may be influenced by mislabeled instances. This distortion can propagate throughout the training process, affecting the model's ability to accurately capture the true underlying data distribution. Prototype Ambiguity. In the presence of noisy labels, the distinction between different classes becomes less clear, leading to ambiguity in prototype definitions. Prototypes may no longer accurately represent the true characteristics of their respective classes, making it difficult for the model to generalize effectively. These make noisy labels in FL scenario an interesting direction for future extension to our proposed FedPLVM.FedPLVM primarily addresses the challenges of cross-domain federated learning, where data from heterogeneous domains need to be collaboratively utilized while preserving privacy. This capability enables the application of FedPLVM in various real-life scenarios, including: Healthcare Data Collaboration. Different hospitals or healthcare institutions often possess diverse patient datasets with varying characteristics and distributions. FedPLVM allows these institutions to collaboratively train machine learning models for tasks such as disease diagnosis, patient outcome prediction, or personalized treatment recommendation while ensuring data privacy and security. Internet of Things (IoT) Networks. IoT devices deployed in different locations or environments generate heterogeneous data streams, including sensor readings, environmental data, and user interactions. FedPLVM enables federated learning across IoT networks, facilitating tasks such as anomaly detection, predictive maintenance, or environmental monitoring without centralized data collection, thus preserving user privacy and reducing communication overhead.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper presents a variance-aware dual-level prototype clustering and a $\alpha$-sparsity prototype loss to tackle performance loss on hard domains during federated prototype learning. A majority of the reviewers believe that the paper is very well written and addresses an important shortcoming of the prior work federated prototype learning. 

The authors present strong experimental results to support their methodology and provide appropriate intuitions and visualizations to further improve the readability of the work. The additional results presented by the authors in the rebuttal resolved most of the concerns that were raised by the reviewers regarding (i) space and time complexity, (ii) differences with respect to the prior works, (iii) differences in performance gap among baselines on different datasets, and (iv) details of the FINCH clustering.

I would request the authors to consider these forum discussions and appropriately reorganize the paper before final submission. I believe it would improve the overall quality of the work and help the readers understand the method better.

Reviews: review: paper_summary: paper_summary: The domain gap among multiple clients impedes the generalization of federated learning. To mitigate cross-domain feature representation variance, the authors introduce FedPLVM, which establishes variance-aware dual-level prototypes clustering and employs a novel α-sparsity prototype loss. To verify the effectiveness of the proposed method, extensive experiments are conducted.
strength_weakness: strength_weakness: 1. It makes sense to perform global level prototype clustering on the server side, while performing intra-class prototype clustering on the local side makes unclear sense, why it is advantageous to get more prototypes for hard domain samples?
2. In Eq. (8), what is meaning of C_y?
3. From Fig. 4, \alpha has a significant impact on the results, which is not conducive to the robustness of the algorithm.
4. The latest compared method in the paper is from 2023 and lacks the latest comparison method.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Please refer to the weaknesses.
paper_summary: paper_summary: The paper introduces FedPLVM, a federated learning approach that improves federated prototype learning (FedPL) in heterogeneous data domain setting. Traditional FedPL methods create the same number of prototypes for each client, leading to performance disparities across clients with different data distributions. FedPLVM mitigates this by introducing variance-aware dual-level prototype clustering and an α-sparsity prototype loss. Specifically, each client first clusters the feature vectors of all same-class local samples to get local clustered prototypes. Then,  the server conducts global prototype clustering to reduce the number of prototypes for each class. Experiments show that FedPLVM outperforms the other baselines on the data from multiple domains.
strength_weakness: strength_weakness: 1. The paper misses important baselines. Personalized federated learning studies are not studied, which should be suitable in the studied setting. 

2. The computation and communication cost of FedPLVM compared with other studies is not presented.

3. The clustering method needs more details. It is not clear why clustering can address the heterogeneous domain issue. 

4. The work is incremental work based on FedPL.
questions: questions: 1. From Algorithm 1, the goal of FedPLVM is to train a single global model. From the experiments, the model is tested on the local datasets. Given data from different domains, why don’t the parties train personalized local models with personalized federated learning?

2. The paper mentioned that more prototypes are needed for harder domains. For the clustering method, is there a guarantee that the harder domain will also get more local prototypes? The paper may add more analysis or experiments to demonstrate it.

3. Will FedPL have significant higher computation and communication cost than FedAVg?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper aims to investigate the federated prototype learning problem with data heterogeneity. To handle the cross-domain representation variance problem, a new method termed FedPLM is proposed, which includes a dual-level prototype clustering mechanism and an alpha-sparsity prototype loss. Experiments are conducted to show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. On several datasets (e.g. MNIST and USPS), the performance of the proposed method is not significant enough. Can the author give more explanation for this point?

2. The running efficiency of the proposed method is not given. Complexity analysis and efficiency-related experiments are expected to discuss the efficiency of the proposed method.
questions: questions: The authors are expected to address the concerns in the block of "Weaknesses". Also, I have one more question: How to balance the two sub-terms (contra and corr) in alpha loss? As they may be on different scales, is that feasible to directly add them together?
limitations: limitations: Limitations are discussed in the paper. For the negative societal impact, I didn't find any concern from my side.
paper_summary: paper_summary: This paper focuses on Federated Prototypes Learning and reveals that existing methods create the same prototypes for different clients, which neglects the distribution diversity. In this work, the authors introduce the variance-aware dual-level prototype clustering and alpha sparsity prototype loss. Various experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: This paper has several drawbacks.

1. Motivation for the α-Sparsity Prototype Loss. It is a little strange that "This multiplicity of prototypes could potentially lead to overlapping feature representations among different classes, especially in challenging client scenarios". As the authors claim the feature overlapping, the naive solution is to use other ways to cover the local feature behavior rather than multiple local prototypes.  Furthermore, I assume that authors could employ the OT (optimal transport) to require them to concentrate on different parts. I do not think the current operation is a suitable way to deal with the feature overlapping.

2. The paper architecture is not suitable. I wonder why authors spend a lot of space to write meaningless or complicated words and even leave no space for DomainNet and Office-10. Furthermore, I encourage to consider the label skew effect, ie, with different label skew degree. Besides, the experiments on large scale of clients are also important.

3. The Figure 1 is confusing. Why monitor would capture the digitis figure? It is a not soundable problem figure.
questions: questions: I encourage the author to take careful thinking for the federated prototype learning field. The existing solution shares a high similarity with the FPL (CVPR'23)  and seems like an incremental work.
limitations: limitations: Yes, the authors have discussed.
paper_summary: paper_summary: This paper studies federated learning under heterogeneous data domains and introduces a federated prototype learning strategy, denoted as FedPLVM, to mitigate the problem. A dual-level prototype generation method is proposed to address domain variance between hard and easy domains, reducing the communication burden between clients and the server. Moreover, an \alpha-sparsity prototype loss is proposed to enhance the sparsity of the prototype representations. The experiment results demonstrate the efficacy of the proposed method and the necessity of each proposed module.
strength_weakness: strength_weakness: 1. The proposed dual-level clustering strategy appears similar to FPL. What is the key difference between them? The primary distinction lies in the proposed \alpha-sparsity loss, which introduces incremental innovation to the paper.
2. The proposed dual-level prototype strategy utilizes FINCH for clustering. However, the clustering results from FINCH can vary at different steps. Which step's result is selected for use? Do results from different steps vary?
3. The impact of \alpha is only evaluated on small-scale datasets. Does the trend consist with big-scale datasets like DomainNet?
4. In Sec. G, the authors claim that the evaluation protocol used in FPL is unfair. However, the protocol used in FPL is sensitive to both easy and hard datasets.
questions: questions: This paper is well-organized and the experiment results significantly surpass existing methods. But the level of innovation is questionable: the dual-level prototype generation is similar to FPL, and the only key difference is the \alpha-sparsity prototype loss. Moreover, I also have some concerns on this paper. For more details, please check the weakness part.
limitations: limitations: Yes. The authors addressed limitations of the paper.

Comments: comment: Thank you for your responses. In light of the novelty and other reviewers' comments, I tend to maintain my score.
Thanks for your response. I maintain my score.
Thank you for your detailed response and addressed my concerns. Thus, I would like to raise my score.
Thank you for your continued feedback. We appreciate the opportunity to further clarify our approach.

1. We would like to further clarify the distinctions between our method and FPL. Our dual-level prototype clustering significantly differs from the single-level global prototype clustering  FPL, both in purpose and effectiveness, as we detailed in our previous response and demonstrated in the visualization results of our paper. The local prototype clustering is crucial because it captures essential variance information, rather than just representing the average feature as in FPL. This is particularly important in challenging domains. For instance, as shown in Figure 1 of the attached PDF, the easier domain (MNIST) generates noticeably fewer prototypes compared to the harder domain (SVHN), which underscores the necessity of our local prototype clustering approach. Moreover, our proposed $\alpha$-sparsity loss introduces an innovative and effective enhancement to our dual-level prototype clustering. We conducted additional experiments to highlight the advantages of the  $\alpha$--sparsity loss in terms of both convergence speed and final accuracy. In the Digit-5 experiment, we performed 100 global rounds as stated in our paper and compared the convergence speed with and without the $\alpha$-sparsity loss. The results, which report the average test accuracy after every 10 rounds, clearly show that the $\alpha$--sparsity loss leads to faster convergence (achieving it in 70 rounds compared to 80 rounds without the proposed loss) and higher final accuracy.
| Rounds            |  10   |  20   |  30   |  40   |  50   |  60   |  70   |  80   |  90   | 100   |
|:-----------------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
| w/o \(\alpha\)-sparsity loss | 35.14 | 56.44 | 57.58 | 62.20 | 64.90 | 65.72 | 65.02 | 66.66 | 66.32 | 66.06 |
| w/ \(\alpha\)-sparsity loss  | 54.12 | 62.44 | 65.32 | 66.12 | 67.32 | 68.20 | 69.62 | 69.22 | 69.86 | 69.26 |

2. We appreciate your insightful comment. Following your suggestion, we have calculated the additional average accuracy using the protocol in FPL. Our updated results show that the average accuracy of our proposed method is 77.07%, compared to 75.54% for FPL. This further demonstrates the superior performance of our method.
Thank you for your response, some of my concerns are solved. However, I still do not find a distinction between dual-cluster and FPL. And for R4, the weighted average could be another valuable evaluation protocol but I do not think the protocol used in FPL and other methods is unfair. Thus, I would like to maintain my score.
Dear Reviewers,

We sincerely appreciate your extensive efforts in reviewing and commenting on our work. We hope that our rebuttal has effectively addressed your comments and concerns.

As the author-reviewer discussion period is approaching to its end on August 13th, we kindly encourage you to reach out if you have any further questions or points of discussion. Your feedback is highly valued, and we are eager to address any additional remaining concerns you may have.

Thank you very much for your attention to this matter.

Anonymous Authors
Thank you for your further comments.

For the details of the review:
1. FedFA aims to augment the feature statistic by constructing a vicinity Gaussian distribution for each data point and change the empirical risk to the vicinal risk. Such an augmentation operation can enrich the feature information after the feature extractor layer. However, this approach has its limitations, especially in cases where data volume is small or distribution is uneven. In these scenarios, the constructed Gaussian distribution might not fully capture the true diversity of the data and can have bias. Also, it requires additional computation regarding the distribution parameters locally for each feature extractor layer with augmentation.  In contrast, our proposed FedPLVM utilizes only the feature representation of the final feature extractor layer to cluster several local prototypes regardless of the even or uneven feature distribution and further performs global clustering and combines with the $\alpha$-sparsity loss in the local training. Lastly, FedFa is one of the baselines we compared in our paper and the result shows our method outperforms FedFA in Digit-5, Office-10 and DomainNet.

2. We appreciate your suggestions regarding the organization of our paper. However, we must respectfully disagree with the comment that excessive space is devoted to preliminary discussions at the expense of comparisons with existing methods. Firstly, the preliminary section occupies only half a page and concisely presents essential knowledge required for understanding our proposed method. This information is crucial and cannot be further condensed without loss of clarity. Secondly, we have integrated comparisons with existing methods throughout the introduction and methodology sections. Specifically, our comparison with FPL, which is most relevant to our work, includes dedicated paragraphs discussing the differences (see lines 232-247).

For Questions:

1. We appreciate for suggesting the paper on FedOTP. While FedOTP is indeed a noteworthy contribution, we believe it addresses a different aspect from that of our paper. FedOTP focuses on developing efficient collaborative prompt learning strategies to integrate pre-trained visual-language models into federated learning frameworks. This topic, although valuable, is not closely related to the core focus of our research.

2. The icon in Figure 1 of our paper mentioned in your question is not that of a monitor but a laptop. In FL, the client device doesn’t necessarily need to capture the data itself. In our illustration, the laptop processes the stored digital data without needing to capture figures or photos directly. 

3. We thank you for your advice on including a convergence speed analysis. We agree that theoretical convergence analysis is a crucial aspect of federated learning. Typically, such analyses in existing FL research depend heavily on strong assumptions about gradient bounds and the differences between global and local loss functions. Importantly, incorporating prototype learning alters the loss function itself. Since FL convergence analyses require the specific assumptions regarding the loss functions, the advantages of integrating prototype learning into the loss function might not be readily apparent. This limitation is also why existing FL studies that utilize prototype learning, such as FPL, do not provide theoretical convergence analysis. Instead, to address concerns regarding convergence speed, we have provided experimental results that clearly demonstrate the improvements our method achieves. In our Digit-5 experiment, we performed 100 global rounds as stated in our paper and compared the convergence speed with and without our proposed $\alpha$-sparsity loss. We reported the average test accuracy after each 10 rounds training. Obviously with the $\alpha$-sparsity loss we have a faster convergence speed (70 rounds compared to 80 rounds without the proposed loss function) and a higher converged accuracy: 
| Round | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| w/o $\alpha$-sparsity loss | 35.14 | 56.44 | 57.58 | 62.20 | 64.90 | 65.72 | 65.02 | 66.66 | 66.32 | 66.06 |
| w/ $\alpha$-sparsity loss | 54.12 | 62.44 | 65.32 | 66.12 | 67.32 | 68.20 | 69.62 | 69.22 | 69.86 | 69.26 |

4. We appreciate your suggestion to include visualizations to support our claims. However, during the author-reviewer discussion period, we are unable to add new figures. Should our paper be accepted, we will certainly include the recommended visualizations in the final version of the manuscript as advised.
Thanks for the author's response. Some of my concerns have been addressed. I'll increase my score to 5.
Thank you for the AC tips. I have provided more details in my response to the review.

As for the details of the review,
1. The naive solution is to utilize a Gaussian distribution to describe class information. For example, FedFA [1] is based on Gaussian modeling of feature statistic augmentation. I encourage the authors to compare the conceptual differences between local prototype clustering and class Gaussian construction. Additionally, the authors should clearly explain the advantages of utilizing a dual-level prototype approach.
2. With respect to paper organization, the authors should spend more space on concept comparison with existing methods. But the authors focus on preliminary discussion. 


Thank you for authors feedback. I still have the following questions:
1. You refer to a paper from 2019, but many papers related to Optimal Transport (OT) have been published in recent years. For instance, FedOTP [2] introduces the OT loss in regularization.
2. The authors did not address question 3: “Figure 1 is confusing. Why would a monitor capture the digits? It does not seem like a logical problem illustration.”
3. You mentioned that “This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.” Could the authors provide the corresponding theoretical or convergence analysis? Additionally, regarding “which mitigates the potential risk of feature representation overlapping,” could the authors offer visualizations to support this claim rather than relying on terms like “potential”?

[1] FedFA: Federated Feature Augmentation
[2] Global and Local Prompts Cooperation via Optimal Transport for Federated Learning. CVPR 2024