Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains
Authors: Lei Wang
Jieming Bian
Letian Zhang
Chen Chen
Jie Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Prototype Learning
Domain Heterogeneity
Primary area: optimization_for_deep_networks

Abstract: Federated learning (FL) allows collaborative machine learning training without sharing private data. While most FL methods assume identical data domains across clients, real-world scenarios often involve heterogeneous data domains. Federated Prototype Learning (FedPL) addresses this issue, using mean feature vectors as prototypes to enhance model generalization. However, existing FedPL methods create the same number of prototypes for each client, leading to cross-domain performance gaps and disparities for clients with varied data distributions. To mitigate cross-domain feature representation variance,  we introduce FedPLVM, which establishes variance-aware dual-level prototypes clustering and employs a novel $\alpha$-sparsity prototype loss. The dual-level prototypes clustering strategy creates local clustered prototypes based on private data features, then performs global prototypes clustering to reduce communication complexity and preserve local data privacy. The $\alpha$-sparsity prototype loss aligns samples from underrepresented domains, enhancing intra-class similarity and reducing inter-class similarity. Evaluations on Digit-5, Office-10, and DomainNet datasets demonstrate our method's superiority over existing approaches.

Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains
Lei Wang University of Florida Gainesville, FL 32611 leiwangl@ufl.. edu
Jieming Bian
University of Florida Gainesville, FL 32611 j j jimming. bian@ufl edu
Letian Zhang
Chen Chen
Jie Xu
Middle Tennessee State University Murfreesboro, TN 37132 letian. zhang@mtsu. edu
University of Central Florida Orlando, FL 32816 chen. cwwnwcrcv.cccccccc ucf edu
University of Florida Gainesville, FL 3261 jie.xu@ufl.ee xu@ufl.edu
Abstract
Federated learning (FL) allows collaborative machine learning training without sharing private data. While most FL methods assume identical data domains across clients, real-world scenarios often involve heterogeneous data domains. Federated Prototype Learning (FedPL) addresses this issue, using mean feature vectors prototypes to enhance model generalization. However, existing FedPL. methods create the same number of prototypes for each client, leading to cross-domain performance gaps and disparities for clients with varied data distributions. To miti- gate cross-domain feature representation variance, we introduce FedPLVM. which establishes variance-aware dual-level prototypes clustering and employs a novel a- sparsity prototype loss. The dual-level prototypes clustering strategy creates local clustered prototypes based on private data features, then performs global prototypes clustering to reduce communication complexity and preserve local data privacy. The a-sparsity prototype loss aligns samples from underrepresented domains, en- hancing intra-class similarity and reducing inter-class similarity. Evaluations on Digit-5, Office-100 10, and DomainNet datasets demonstrate our method's superiority over existing approaches.
Introduction
Federated Learning (1 (FL) is novel distributed learning framework that enables clients collabo- ratively train a global model using their respective local datasets, thereby preserving data privacy. FL offers distinct advantages over traditional distributed learning methodologies by mitigating communi- cation costs and addressing privacy concerns, which has led to its increased adoption across various sectors. Despite these benefits, FL comes with its own challenges. especially in terms of data hetero- geneity [41]. In FL, clients gather private data from unique sources, resulting in non-independent and identically distributed (non-IID) datasets. Such non-IlD distributions can lead clients toward their local optima, potentially diverging from the global objective. Consequently, this may impede convergence rates and diminish overall model performance 17
To address the aforementioned issue of data heterogeneity, numerous FL techniques have been devised 220 41 29 [2). While these methods have indeed improved convergence, their focus on non-lID scenarios remains largely restricted. Typically, they assume that the data across clients
'The first two authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MNIST (Easy Domain)
Proposed
Vanilla
05
D
5
protasyes morkl
/
4
Client
MNIST
Client 2
Averaged Protstype
MNIST-M (Hard Domain)
Synth
Server
Client K
MNIST-M
Figure 1: Illustration of federated learning with heterogeneous data domains. The Vanilla column depicts the local feature distribution of the standard FedPL approach, obtaining average local and global prototypes directly. Proposed method showcased in the adjacent column yields larger inter-class distance and a reduced intra-class distance. Note that without capturing variance information, even for hard domains, local averaged prototypes for each class can be well distinguished while the feature vectors are still mixed up. Both methods illustrate noticeable variations in domain characteristics across datasets, detailed in Fig.
pertain to a single domain, attributing non-Ill distribution to label skew alone. Yet, in more realistic scenarios, clients gather data according to their unique preferences, making it impractical to assume identical domain origins for local data. Instead, the data often stems from heterogeneous domains, resulting in varied feature distributions..
Current approaches [10 32 to FL in heterogeneous domains aim to develop a global prototype for each label category. These prototypes are designed to minimize the distance between the individual training samples and their corresponding category's global prototype. Typically, global prototype is computed as the mean of the local prototypes from each client, whereas a local prototype is itself the average of the representations of samples within the same category. Several studies have proposed unique designs to enhance training performance in these settings. For example, one work 33 enables contrastive learning at the client level by facilitating the exchange of local prototypes, thus promoting inter-client knowledge transfer. However, this exchange raises privacy concerns and substantially increases communication overhead. Another study [T0] opts for clustering approach to identify representative prototypes, thereby preserving domain diversity and preventing bias towards predominant domains. This technique has proven effective, especially in scenarios with disproportionate client distribution across domains.
Although these methods improve overall performance across domains, they do not address the unequal learning challenges that arise from domain diversity. For instance, on the Digits dataset [42], methods may perform well in domains like MNIST 7 but underperform in more challenging domains such as SVHN (24). This discrepancy is evident in representation distributions, as exemplified in Fig. [] where "easy' domains show tight clustering samples within the same category and clear separation between different categories, facilitating accurate classification. In contrast, 'hard' domains display looser clustering, increasing the likelihood of misclassification, particularly for samples near category boundaries. Addressing these disparities is crucial for the equitable advancement of FL methodologies across diverse client domains.
Considering the challenges posed by unequal learning due to domain diversity, we introduce novel method, termed FedPLVM (short for Federated Prototype Learning with Variance Mitigation). FedPLVM devises two main mechanisms. Firstly, we develop a dual-level prototype clustering mechanism that adeptly captures variance information, significant improvement over previous methodologies that rely on averaging local training samples' representations to derive local prototypes. Our local-level clustering generates multiple local clustered prototypes within each domain. To further mitigate increased communication costs and privacy concerns arising from transferring

---

comprehensive set of diverse prototypes for each class from every client, we employ global-level prototype clustering on the server side. Secondly, by capturing the variance information through clustered prototypes, we design an innovative o-Sparsity prototype loss to enhance the training process. To prevent the intermingling of feature distributions from 'hard' clients, instead of simply maximizing the feature-level distance between each local instance and the prototypes of other classes, we refine this distance by clevating it to an a power, with a being a value between and 1. This modification effectively repels other prototypes, thereby introducing greater sparsity in the inter-class feature distributions. Moreover, we incorporate an intra-class similarity correction term to lessen the feature-Ievel distance among intra-class samples, thereby concentrating the a-sparsity prototype loss on amplifying the feature-level distance across inter-class samples. Collectively, these two strategies empower FedPLVM to reasonably leverage variance information, thus equilibrating fairness between 'hard' and 'easy" learning domains and enhancing overall learning performance. Consequently, FedPLVM emerges as reliable approach for Federated Learning in contexts characterized by heterogeneous data domains. Our main contributions are outlined as follows:
This study delves into FL with heterogeneous data domains, examining why models exhibit varying performance across different domains. We identify a fundamental limitation in existing methods: their inability to effectively address the disparate learning challenges inherent diverse domains.
To tackle these uneven learning challenges, we introduce a novel approach, FedPLVM. This method incorporates dual-level prototype clustering method, capturing the rich sample representation variance information while ensuring communication efficiency. Additionally, we develop a new c-sparsity prototype loss to address learning difficulties more equitably.
Extensive experiments conducted on the Digit-5 42 Office-10 []] and DomainNet [26] datasets demonstrate the superior performance of our proposed method when compared with multiple state-of-the-art approaches.
2
Related Work
Federated Learning. FL aims to train a global model through collaboration among multiple clients while preserving their data privacy. FedAvg (21), the pioneering work in FL, demonstrates the advantages of this approach in terms of privacy and communication efficiency by aggregating local model parameters to train a global model. A significant challenge in FL is data heterogeneity, often manifested as non-llD (independently and identically distributed) data. Subsequent research, following FedAvg, has primarily focused on addressing data heterogeneity to enhance training performance in FL environments. Specifically, studies such as [8 2 30 have improved performance by incorporating a global penalty term to mitigate discrepancies. Other works, e.g., [32] (23. have sought to maximize fcature-level agreement between local and global models to further boost performance. Recent works [10] 32 19 40] have explored data heterogeneity arising from client- specific domain diversity, while overlooking the challenges of unequal learning across different domains. In this paper, we propose novel approach using dual-level prototype clustering to capture essential local variance information. Additionally. we introduce a new a-sparsity loss, specifically designed to tackle the challenges of learning in diverse domains, thereby facilitating the development of more generalizable global model.
Prototype Learning. Prototype learning has been extensively explored in various tasks, such as transfer learning [27 [3]
few-shot learning (31), zero-shot learning [1. and unsupervised learning [3]. The concept of a prototype in this context refers to the average feature vectors of samples within the same class. In the FL literature, prototypes serve to abstract knowledge while preserving privacy. Specifically, approaches like FedProc (3
and FedProto (32] focus on achieving feature-wise alignment with global prototypes. FedPCL 33 employs prototypes to capture knowledge across clients, constructing client representations a prototype-we -wise contrastive manner using a set of pre-traie trained models. FPL [10 highlights the use of cluster-based and unbiased global prototypes to tackle the challenges in FL where clients possess donain-diverse data, Our study addresses a similar issue as FPL but with distinct emphases. While FPI. concentrates on the disparities in the number of clients across domains, aiming to mitigate the hias in the overall model caused by domains with more clients, our work focuses on the intrinsic learning challenges that vary across domains.
3

---

Clients
Server
Clients
L
LCE
6
hg
4
hg
:
(a) Embedding
(b) Local Clustering
Global Clastering
Local Training
Generation
pload
Download
Model Aggregation
hx
jx
PgK
L
CE
Feature Extractor
Classifier
Sample Embedding
Local Clustered Prototype
Clustered Prototype
An overview of our proposed FedPLVM framework. Once the sample embedding is generated by the feature extractor, the client conducts the first-level local clustering, following Eq. Subsequently, the server gathers all local clustered prototypes and local models (comprising feature extractors and classifiers), initiates the second-level global clustering based E and averages the local models to form global model. Finally, clients utilize the received global clustered prototypes update the local model, employing loss functions La from Eq. and LCF from Eq.
Figure 2:
Contrastive Learning. Contrastive learning a promising superviee learning technique. The work by [25] constructs pairs of positives and negatives for each sample and applies the InfoNCE loss to compare these pairs. Another work by [5] extends contrastive learning from self-supervised to fully supervised settings, utilizing both label information and contrastive methods. Additionally, some studies [6 39) integrate contrastive learning into local training to enhance performance FL.. a-sparsity loss, aiming to further reduce similarity among inter-class sample features while amplifying similarity among intra-class samples.
Instead of the conventional InfoNCE loss, our approach introduces
3 Preliminary
Regular FL Scenario. Consider the classic FL scenario: there exist K clients and one server, which aims assist all clients train common machine learning model without sharing their private data, denoted by Dj {x N. for client k. Formally, the global objective of FL be formulated as:
K NE N
min
Cu(w; Dk),
=k=1 where L3 is the local loss function for client k, w and N - ΣK N₁. denote the shareable model and the total number of samples among all clients respectively.
Domain Shift in FL. In the simplest FL setting, the label and feature distribution are the same between two clients and j. formally Pv()) = P3 (v) and Pr(xly) P3 (arly. Extending such scenario to the heterogeneous data setting brings us to the point of domain shift, also denoted as feature non-IlD setting. Domain shift is caused by distinctive feature distributions among clients, that (x(x)) * (xly), though their label space is still the same.
Federated Prototype Learning. To handle the FL domain shift problem, previous works [32 10 divide the classification network into two parts: feature extractor and classifier. The feature extractor h RV + R maps a sample x R into the feature space and generates the corresponding feature vector z h(x) E RR,, Then the classifier
RD R outputs the M-class prediction M given the feature vector z. The intuition of FedPL is adjusting the feature extractor to generate a consistent feature distribution among different domains. Similar the global shareable model, the straightforward solution computes the average feature vector of all samples belonging to the same class for further processing, called the prototype. Due to the distributed property of FL, cannot collect all samples via the server, which makes the above-mentiones idea
fz) (h(x)) E R

---

a two-step procedure:
K K
Pk
Σ (X..)))D
h(x;);
8"h
vk KK
D
where K" denotes the local prototype of class m on client and the global prototype of class m, respectively. DE is the subset that all samples belonging to class m in the local dataset DR. Upon obtaining the prototypes, most FedPL works tend to design a loss function that approximates the prototype of one's own class while staying away from the prototypes of others.
K} is the set of clients, M 21,2,..... M) } is the set of classes. Pk and gt
4 FedPLVM: FedPL with Variance Mitigation
4.1 Dual-Level Prototype Generation
Local Prototype Clustering. To mitigate the impact of domain variance in FedPL., we revise the two-step averaged prototype generation process to a dual-level clustering algorithm. In Fig. [ the pronounced diversity in domain variances among clients becomes evident, intricately linked to the complexity of their datasets. For instance, comparing the feature distribution between Synth and MNIST shows visibly more scattered pattern in Synth due its higher complexity, whereas MNIST, being comparatively easier to learn, displays a more structured distribution. It becomes evident that computing a single local averaged prototype for one class per client is 'unfair', given the differing richness of feature distribution information among clients. Particularly, for 'hard' clients with complex datasets such as SVHN, employing multiple local prototypes becomes imperative to capture the scattered feature distribution comprehensively Hence, we propose the first-level prototype generation, namely local prototype clustering. Instead of a straightforward averaging approach, our method involves initially clustering the feature vectors of all same-class local samples, forming several local clustered prototypes as set of local representations.
P.r {Pk.))
Ja.m |
Cluster
(h(x;) y1) D
where Jk.m and Pkj represents the number of local clustered prototypes and the j-th local clustered prototype of class important to note that the number local clustered prototypes may differ across various classes and clients. To determine these prototypes, we employ the parameter-free clustering algorithm, FINCH (28). utilizing cosine similarity as the clustering metric. We also conduct additional experiments comparison with other clustering methods in Supplementary Material. This choice ensures the alignment of the number of local clustered prototypes with the sparsity of the domain distribution. By enhance the representation of distinct feature distributions, preventing
client & clustered from the set of feature vectors {/(x:))(x,, my) E D ). It is
leveraging this approach, the overdrift of local averaged prototypes towards densely concentrated regions in the feature space.
Global Prototype Clustering on Server. Distributing all local clustered prototypes among clients poses challenges due to the extra communication cost and privacy concerns. Hence, we introduce the second level of prototype generation, namely global prototype clustering, which can be formulated as:
gTh = (990
Cm Cluster pm = {P 1==1
(4)
j=1
where C... and 0²" denote the number of global clustered prototypes and the j-th global clustered prototype of class m on the server from the local collected prototypes set pm = (P Jk=1 Through the second-level global clustering, must distribute to clients compared to distributing all locally clustered prototypes, alleviating potential communication costs. Dual-level clustering also addresses privacy concerns that arise from original local clustered prototypes potentially revealing client-specific features and the corresponding results can be found in Supplementary Material.
significantly reduce the number of prototypes that the server
4.2 a-Sparsity Prototype Loss
Unlike previous methods in FedPI., which generate a single prototype per class, our approach employs dual-level clustering to create multiple prototypes for each class, thereby capturing valuable
5

---

variance information. This multiplicity of prototypes could potentially lead to overlapping feature representations among different classes, especially in challenging client scenarios. To mitigate this risk, we introduce a novel a-sparsity prototype loss, inspired by the InfoNCE-based loss. Our newly designed o-sparsity prototype loss enhances inter-class feature distribution sparsity and maintains balanced feature representation distances within classes, unlike the traditional InfoNeE- loss. The formulation the a-sparsity prototype loss is detailed below:
L Leontra Leorr-
(5)
The first contrastive term can be formulated as:
Σ exp (h
Leantra = lloo
(6)
Σ exp (sa (sa(h( gEg
where r is the temperature B5 and G { is the set of all global clustered prototypes. sa(-,-) -) in the first term is the modified a-sparsity cosine similarity metric between the feature vector h(x,) and the prototype g" from class m and can be formulated
that controls the concentration strength the similarity
J.M
h(xx))
g T
saa sa(h(xi):9)))
[|x(xx)]
11
where a E 1). This metric serves to compel the feature extractor to generate feature outputs closely aligned with the global clustered prototypes of their respective classes while distancing them from other global prototypes. It achieves this by maximizing intra-class similarity and minimizing inter-class similarity. As our feature vectors have positive values, cosine similarity always falls within the range of [0, 1]. By introducing a sparsity factor a to the similarity and applying it as a power, all similarity values are elevated. However, due the smaller denominator component representing similarity with other classes in the L., function, the impact of the concave function (+) is more pronounced. This emphasis on maximizing inter-class distance is what we denote a-sparsity operation, which directs more attention toward expanding the overall feature distribution to a broader range. Consequently, it mitigates the issue of feature distributions within one class being excessively dispersed and overlapping with feature distributions of other classes.
This modification does indeed lead to unintended consequence: increase in intra-class distance due to heightened similarity between the feature vector and prototypes of the corresponding class, resulting from the concavity of (+) This brings to the second correction term of our a-sparsity prototype loss:
Ceory
Σ g EG"
sa (a(xx)) gh:) Cy.
(8)
The latter term within the a-sparsity prototype loss serves corrective measure, which focuses on pushing the average cosine similarity closer to 1. This correction measure aims to counterbalance the increase in intra-class distance stemming from the adjustment introduced by a.
An additional Cross- Cooss-Entropy (CE) loss 6 is employed to train the classifier and derive prediction results, which can be formulated as:
LCF =
Σ
1,y (og()()(x))))-
(x..)))
The total local loss, combining the previously mentioned loss functions, is expressed follows:
Ciocal XCa + Lcg.
(10)
Here, À serves as a pppee-parmmttrr that regulates the balance between the a-sparsity prototype loss and the CE loss. This formulation allows for unified and weighted consideration of the a-sparsity prototype loss.
In summary, our FedPLVM operates as follows in each training round: Initially, each client generates feature vectors for all local samples and clusters these into several local clustered prototypes. These local clustered prototypes are then uploaded to the server, which aggregates them into distinct

---

prototype sets for various classes and further clusters them to form global clustered prototypes. Concurrently, the server collects local models from all clients and consolidates them into a unified global model. The server then distributes all global clustered prototypes and global model to clients. Subsequently, each client utilizes the global clustered prototypes to train the global model on its private dataset using Flocal, obtaining its local model. Finally, clients employ their local models to generate new feature vectors and repeat the aforementioned procedure in the next training round. For detailed insight into FedPLVM, refer to Algorithm 1 in the Supplementary Material.
Comparison with FPL [0]. FPL studies on FL among clients with distinet domain datasets and is considered a state-of- oatte-of-hh-aaee method. FPL is specifically designed to address imbalances in client distribution across domains, aiming to neutralize the skewed influence of domains with more clients on the global model training. In contrast, our study concentrates on the unequal learning obstacles that vary by domain, a challenge that persists regardless of equal client distribution, leading to distinct learning hurdles across domains and thereby leading to our two key operations. Firstly, although both our method and FPL. employ prototype clustering, the objectives and implementations markedly differ. FPL's clustering is intended to harmonize the impact of local prototypes from each domain, involving only global (server-level) clustering. Conversely, our method integrates dual-level clustering at both the client (local) and server (global) levels. Our technique distinguishes itself by performing local prototype clustering. capturing critical variance and not just the mean data, which is particularly crucial in hard domains. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, thereby enhancing both efficiency and privacy. Secondly, we introduce an innovative c-sparsity prototype loss that features corrective component to reasonably utilize the variance information to reduce feature similarity across different classes while boosting it within the same class, promoting more effective and stable learning.
Experiments
Datasets. We evaluate our proposed algorithm on three datasets: Digit-5 42 Office- 10 B and DomainNet [26). Digit-5 is a dataset for digits recognition, consisting of 5 domains: MNIST, SVHN, USPS, Synth and MNIST-M. Office-10 is a dataset for office item recognition, consisting of domains: Amazon, Caltech, DSLR and Webcam. DomainNet is a large-scale classification dataset, consisting of 6 domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch.
Baselines. We compare our algorithm with classic FL methods: FedAvg (2I], FedProx [8], FedPL methods: FedProto 32) FedPCL [33] FPL 10 and FL method on feature skew: FedFA 43
Implement Details. We employ the ResNet10 (9] as our backbone model, configuring the feature vectors' dimension to 512. The optimization is done using the SGD optimizer, employing learning rate of 0.01, momentum of 0.5, and a weight decay of le 5. For Digit-S, Office- 10 and DomainNet, we use 5, 4 and 6 clients respectively. The client data is independent and identically distributed (i.i.d.) and non i.i.d. results can be found in Supplementary Material. operates within distinct data domains, meaning different datasets. For Digit-5 and Office- 10, each client possessed 100 training samples and 1000 test samples. Global communication rounds are fixed at T = 50 for Digit-5 and T 80 for Office-10. Each local training epoch consists of E = 2 iterations. We maintain default hyper-parametee values: T = 0.07, a = 0.25, and À = 100. As for DomainNet, we followed the setup in FedPCL using a 10-class subset. Each client employs 300 training samples and all test samples (approximately 1000, vary on domains). x = 1 and T = =220 for DomainNet. The batch size is set at 32 for all datasets. Details regarding will be elaborated in Sec. 5.2. For fair comparisons, we conduct each setting for experiments and report the average result.
important to note that each client
settings
5.1 Performance Comparison
We compare our proposed method with the sate-of-theeane methods using the Digit-5, Office- 10 and DomainNet datasets, as detailed in Tab Tab. 2 and Tab. [6] Our method demonstrates significant improvements in average accuracy over baseline methods for both datasets. A closer examination of domain-specifie results reveals a more pronounced enhancement in performance on domains that are more challenging to learn. For instance, within the Digit-5 dataset, our method achieves a 5.3% increase in accuracy for the SVHN domain, which is more difficult, and a 0.58% increase for the MNIST dataset, considered casier. This aligns with the goal of our method, which is to
7

---

Table 1: Test accuracy Digit-5. Avg means average results among all clients. Details in Sec. 5.1
Digit-5
Methods
MNIST
SVIIN
USPS
Synth
MNIST-M
Avg
FedAvg
84.98 + 0.92
29.38 + 1.06
82.36 + 1.18
47.00 + 0.73
53.14 + 0.78
59.37
FedProx
85.72 #: 1.50
28.86 + 1.23
82.30 0.75
46.78 + 1.10
52.60 = 2.37
59.25
-0.12
FedProto
88.60 0.72
31.94
1.58
85.54 0.34
51.82 - 1.12
56.86 0.42
62.95
+3.58
FedPCL
88.84 1.08
39.70
2.25
84.74
54.70 1: 1.18
59.96 1 1.34
65.59
+6.22
FedFA
38.96
85.86
0.38
58.04
+ 1.06
61.38 = 0.98
66.74
+7.37
FPL
90.12 1.39
36.78
1.88
86.10
0.66
57.36
1 1,96
64.02
1 1.38
66.88
+7.51
Ours
90.70 1: 0.39
42.08
1.59
86.24
1.37
60.08
1 1.47
67.16 1 0.77
69.25
+9.88
Table Test accuracy on Office-10. Details in Sec, 5.1
Office- 10
Methods
Amazon
Caltech
DSLR
Webcam
Avg
A
FedAvg
48.26 + 1.92
35.11 + 0.96
57.29 + 1,47
71.75
0.80
53.10
FedProx
47.74 0.65
36.44 i 1.92
56.25 4.42
73.45 0.80
53.47
+0.37
FedProto
49.31 2.18
36.07 + 0.91
57.38 + 2.55
79.05
2.40
55.45
+2.35
FedPCL
53.65 2.33
38.93 = 2.73
58.13 + 5.08
78.64 = 0.83
57.34
+4.24
FedFA
56.46 2.15
40.91 + 2.39
60.00 4.68
78.58 1.86
58.99
+5.89
FPL
54.38 + 1.02
38.24
+ 2.38
61.25 + 3.19
80.34 = 1.73
58.55
+5.45
Ours
57.03 + 1.45
42.71 + 1.04
61.50 + 2.02
81.36 1.86
60.65
+7.55
address the disparate learning challenges across various domains, enhancing fairness and aiding in performance improvement in harder domains. Similar improvements can also be observed in Office-10 and DomainNet in Sec. C of Appendix.
5.2 Ablation Study
To evaluate the effectiveness of each component within our proposed methodology, we conducted series of ablation studies using the Digit-5 dataset.
5.2.1 Impact of Dual-Level Prototype Generation.
In this subsection, we focus on examining the impact of dual-level clustered prototypes, as demon- strated by the results in Tab. g The first row illustrates outcomes derived from calculating local prototypes by averaging features samples within the same class, and global prototypes through the direct averaging of these local prototypes. This approach, utilizing straightforward averaging for both levels, yields the least effective performance. This is attributed to its failure to capture variance information, non-trivial aspect in this context. Morcover, adopting solely global clustering approach does not significantly enhance performance. This is attributed to the fact that while global clusters incorporate inter-domain variance, they overlook the eritical aspect of high intra-domain sample variance, particularly in domains that are challenging to learn. To elucidate this, we present i-SNE visualization analysis in Fig. comparing the three prototype generation methods. Our approach fosters a more generalizable decision boundary, as illustrated in the visualization. This ability to effectively capture variance both local and global levels is key why our dual-level clustering method outperforms the others.
Table 3: Comparison on prototype generation methods. Variance means the average distance from the normalized feature vector of one sample to its corresponding class feature center (i.e. the averaged prototype). Results are then used for visualization in Fig. 3 Details in Sec. 5.2.1
Local
Global
MNIST
SVHN
USPS
Synth
MNIST-M
Avg
Variance
Avg
Avg.
86.90
33.10
83,90
53.40
61.40
63.54
1.725
Avg
Cluster
89.40
37.00
85.50
56.60
63.50
66.40
1.393
Cluster
Cluster
90.20
43.70
86.90
61.20
65.40
69.48
0.825
To further explore alternative methods, we examine the efficacy of directly transferring all local prototypes to each client without employing any aggregation clustering techniques. This approach

---

Local Global Avg
Local Avg Global Cluster
Local Cluster
Global Cluster
MNIST
SVHN
USPS
Synth
MNIST-M
Total
Figure 3: Visualization of different prototype generation methods. The first row averages feature vectors locally and averages local prototypes globally. The second row averages feature vectors locally and clusters local prototypes globally. The last row (ours) clusters feature vectors locally and clusters local clustered prototypes globally. The last column Total the visualization of mixing the feature vectors from all datasets. Details in Sec. 5.2.
preserves variance information but raises significant privacy concerns, unlike our dual-level clustering method, which offers enhanced privacy protection. As demonstrated in Tab, 4 directly transferring all local prototypes yields performance comparable to our dual-cluster approach. However, requires transmitting approximately five times many prototypes in each training round.
Table Comparison between w/o and global clustering. w/o means the server distributes all local clustered prototypes to the clients for local training. Avg # of prototypes is the average number of prototypes each client receives from the server during each global round. Details in Sec. 5.2.
Global Clustering
Avg
Avg # Prototypes
Privacy Preservation
Communication Cost
w/o
69.18 + 0.77
100.92
4.76x
w/
69.47 1 0.71
21.20
1x
5.2.2 Impact of a-Sparsity Prototype Loss.
Table 5: Comparison components of a-sparsity prototype loss. Contrast and Correction stand for the contrastive and corrective loss term in the total a- sparsity loss respectively. Avg is the av- erage accuracy result for all clients, De- tails in Sec. 5.2.2
To evaluate the specific impact of our proposed a-sparsity prototype loss, we conduct experiments comparing both contrastive and corrective loss terms. Results in Tab. [5] demonstrates that employing the contrast term led to 1.46% improvement in final average accuracy, while the correction term resulted in 0.99% enhancement. Com- bining all components yields the best performance, show- casing 2.29% improvement. Our investigation also fo- cuses on the sparsity parameter a, depicted in Fig. 4 We find that varying within the range of (0, 1) consistently outperforms the baseline setting with a = 1. After careful consideration of numerical stability, opt for value of 0.250.
Contrast
Correction
Avg
A
w/o w/ w/o
w/o
66.96 + 0.85
w/o
68.42 1 0.95
+1.46
67.95 0.63
+0.99
w/ w/
69.25 + 0.62
+2.29
Comparing method to the prior work FPL. which also integrates its proposed prototype loss with the CE loss, we explore different prototype loss weights, denoted by X. The results depicted in Fig. substantiate the superiority of our approach over FPL across various À settings.

---

72.5
Synth MNIIT-M
FPL
Proposed
Accuracy
67.5
Accuracy
65
65.0
60
62.5
55
60.0
50
57.5
55.0
0.125
0.250
0.500
1.000
1
10
100
1000
Figure 4: Impact of a sparsity and prototype loss weight. The left figure shows the accuracy of two selected datasets and the average accuracy among all clients with different The right figure shows the effects of different for both FPL and our proposed approach. Details Sec. 5.2.2
5.3 Impact of Temperature 7.
To assess the impact of the contrastive temperature (r) on our model's performance, we conduct the following experiment. The results, depicted in Fig. demonstrate that method consistently surpasses the baselines across a range of temperatures (refer to Tab, | and Tab. f details, where FPL exhibits the highest baseline performance 66.88% and 58.55% respectively). Further analysis indicates that an optimal temperature setting for our model on Digit-5 is T 0.070. Another results on Office- 10 identifies an optimal temperature setting for our model at r = 0.045, while we maintain 0.070 for stability in performance comparisons. It is noteworthy that temperatures either significantly higher lower than this value lead training difficulties due numerical instability.
72
70.0
71
Average
67.5
Average
Accuracy
68:45
68.93
69.25
69.16
Accuracy a
62.5
65.0
59.72
60,88
60.65
60.14 59.97
69,7.37
60.0 $7.75
68-
57.5
55.0
52.5
65
50.0
0.025 0.040 0.055 0.070 0.085 0.100
0.025
0.040 0.055 0.070 0.085 0.100 Temperature т
Temperature
r. Figure 5: Impact of r. The left figure shows the impact on the Digital-5 dataset while the right figure shows the Office-10 dataset. Average means the average test accuracy with variance among all clients. Details in Sec. 5.3
Conclusion
In this paper, start by noting the significant difference in domain-specifee representation variance across various datasets within the context of federated learning involving heterogeneous data do- mains. Traditional methods relying on averaged prototypes, calculated the mean values feature representations from samples within the same class, consistently fail to capture this essential local information. Our proposed approach, FedPLVM, addresses this issue by implementing a dual-level prototype generation method. This method leverages first-level local clustering to manage variance information and employs second-level global clustering to streamline communication complexity while ensuring privacy. Additionally, we introduce an a-sparsity prototype loss that prioritizes expanding the inter-class distance, considering the diverse cross-domain variances, and includes correction term to subsequently reduce the intra-class distance. Comprehensive experiments have been conducted validate the effectiveness of FedPI.MM, demonstrating significant accuracy improvement over state-of-the--e federated prototype learning methods.
10

---

Acknowledgements
This work is supported in part by NSF under grants 2033681, 2006630, 2044991, 2319780.
References
[1] Abadi, M.. Chu, A., Goodfellow, McMahan. H.B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. pp. 308-318 (2016)
[2] Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., Saligrama, V:: Federated learning based on dynamic regularization. arXiv preprint arXiv:2111 1112212111263 (2021)
[3] Asoodeh, S., Liao, J., Calmon, F.P., Kosut, O., Sankar, L.: A better bound gives hundred rounds: Enhanced privacy guarantees via f-divergences. In: 2020 IEEE International Symposium on Information Theory (ISIT). pp. 920-925. IEEE (2020)
[4] Canonne, C.L., Kamath, G., Steinke, T.: The discrete gaussian for differential privacy. Advances in Neural Information Processing Systems 33, 15676-15600 15688 (2020)
[5] Dai, Y.. Chen, Z., Li, J., Heinecke, S., Sun, L., Xu, R.: Tackling data heterogeneity in fed- erated learning with class prototypes. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 7314-7322 (2023)
[6] De Boer, P.T., Kroese, D.P., Mannor, S., Rubinstein, R.Y.: A tutorial on the cross-entropy method. Annals of operations research 134, 19-67 (2005)
[7] Deng, L.: The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine 29(6), 141-142 (2012)
[8] Gong, B., Shi, Y., Sha, F., Grauman, K.: Geodesic flow kernel for unsupervised domain adaptation. In: 2012 IEEE conference computer vision and pattern recognition, pp. 2066- 2073. IEEE (2012)
[9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778 (2016)
[10] Huang, W. Ye, M., Shi, Z., Li, H. Du, B.: Rethinking federated learning with domain shift: A prototype view. In: 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16312-16322. IEEE (2023)
[11] Jetley, S., Romera-Pareee Paredes, B., Jayasumana, S., Torr, P.: Prototypical priors: From improving classification to zero-shot learning. arXiv preprint arXiv:1512.01199 (2015)
[12] Jhunjhunwala, D., Sharma, P., Nagarkatti, A., Joshi, G.: Fedvarp: Tackling the variance due to partial client participation in federated learning. In: Uncertainty in Artificial Intelligence, pp. 906-916. PMLR (2022)
[13] Kang. Z., Grauman, K., Sha, F.: Learning with whom to share in multi-task feature learning. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11). pp- 521-528 (2011)
[14] Karimireddy, S.P., Kale, S.. Mohri, M., Reddi, S.. Stich, S., Suresh, A.T.: Scaffold: Stochastic controlled averaging for federated learning. In: International conference on machine learning. pp. 5132 5143. PMLR (2020)
[15] Khosla, P., Teterwak, P., Wang. C., Sarna, A., Tian, Y., Isola, P.. Maschinot, A.. Liu, C.. Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems 33, 18661-18677 (2020)
[16] Li, Q., He, B., Song. D.: Model-contrastive federated learning. In: Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 10713-10722 (2021)
11

---

[17] Li, T., Sahu, A.K., Talwalkar, A., Smith, V. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37(3). 50-60 (2020)
[18] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V:: Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2, 429 450 (2020)
[19] Lu, W., Wang, J., Chen, Y., Qin, X., Xu, R., Dimitriadis, D., Qin, T.: Personalized federated learning with adaptive batchnorm for healthcare, IEEE Transactions on Big Data (2022)
[20] Luo, M., Chen, F., Hu, D., Zhang, Y., Liang, J., Feng, J.: No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems 34, 5972-5984 (2021)
[21] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication- efficcient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273 1282. PMLR (2017)
[22] Mironov, I., Talwar, K., Zhang, L.: R) 'enyi differential privacy of the sampled gaussian mechanism. arXiv preprint atXiv:1908. 100.19908..330000 (2019)
[23] Mu, X., Shen, Y., Cheng, K., Geng, X., Fu, J., Zhang, T., Zhang, Z.: Fedproe: Prototypical contrastive federated learning non-iid data. Future Generation Computer Systems 143, 93-104 (2023)
[24] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y., et al.: Reading digits in natural images with unsupervised feature learning. In: NIPS workshop deep learning and unsupervised feature learning. vol. 2011, p. 4. Granada (2011)
[25] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv: 007748 (2018)
[26] Peng. X., Bai, Q. Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1415-115 (2019)
[27] Quattoni, A., Collins, M., Darrell, T.: Transfer learning for image classification with sparse pro- totype representations. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1-8. IEEE (2008)
[28] Sarfraz, S., Sharma, V., Stiefelhagen, R.: Efficient parameter-free clustering using first neigh- bor relations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8934-8943 (2019)
(29) Sattler, F., Wiedemann, S., Müller, K.R., Samek, W.: Robust and communication--ffici federated learning from non-iid data. IEEE transactions on neural networks and learning systems 31(9), 3400-3413 (2019)
[30] Shoham, N., Avidor, T., Keren, A., Israel, N., Benditkis, D., Mor- Yosef, I.., Zeitak, I.: Over- coming forgetting in federated learning non-iid data. arXiv preprint arXiv:1910.07796 (2019)
[31] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. Advances in neural information processing systems 30 (2017)
[32] Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q. Jiang, J., Zhang, C.: Fedproto: Federated prototype learning across heterogeneous clients. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 8432 8440 (2022)
[33] Tan, Y., Long, G., Ma. J., Liu, L., Zhou, T., Jiang, J.: Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems 35, 19332-19344 (2022)
[34] Tanwisuth, K., Fan, X., Zheng, H., Zhang, S., Zhang, H., Chen, B., Zhou, M.: A prototype- oriented framework for unsupervised domain adaptation. Advances in Neural Information Processing Systems 34, 17194-17208 (2021)
12

---

[35] Wang. F., Liu, H.: Understanding the behaviour of contrastive loss. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495-2504 (2021)
[36] Wang, L., Bian, J. Xu, J.: Federated learning with instance-dependent noisy labels, arXiv preprint arXiv:2312. 10324 (2023)
[37] Xu, J., Chen, Z., Quek, T.Q.. Chong, K.F.E.: Fedcorr: Multi-stage federated learning for label noise correction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern (2022)
recognition, pp.
[38] Yang. S., Park, H., Byun, J., Kim, C.: Robust federated learning with noisy labels. IEEE Intelligent Systems 37(2), 35-43 (2022)
[39] Zhang, F., Kuang, K., Chen, L., You, Z., Shen, T., Xiao, J., Zhang, Y.. Wu, C., Wu, F., Zhuang, Y., et al.: Federated unsupervised representation learning. Frontiers of Information Technology & Electronic Engineering 24(8), 1 1181-1193 (2023)
[40] Zhang. L., Lei, X., Shi, Y., Huang, H., Chen, C.: Federated learning with domain generalization. arXiv preprint arXiv:2111. 10487 (2021)
[41] Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V.: Federated learning with non-iid data. arXiv preprint 180.0000660082222 (2018)
[42] Zhou, K., Yang, Y., Hospedales, T., Xiang, T.: Learning to generate novel domains for domain
generalization. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XVI 16. pp. 561-578. Springer (2020)
[43] Zhou, T. Konukoglu, arXiv:2301. 12995 (2023)
Fedfa: Federated feature augmentation. arXiv preprint
13

---

A Supplementary Material Overview
The supplementary material is organized into the following sections:
B Detailed algorithm of our proposed method, FedPLVM.
C Further results of our experiments.
Dj Additional quantitative results comparing FedPLVM and other baseline methods under the label non-i.idd setting.
Additional quantitative results comparing different clustering algorithms for the prototype cluster- procedure of FedPLVM.
Additional quantitative results comparing FedPLVM and [PO under the unbalanced client domain distribution setting.
G Additional quantitative results employing differential privacy technology in conjunction with FedPLVM.
H: Additional quantitative results comparing different global prototype generation methodologies of MedPLVM.
I Additional quantitative results analyzing the effectiveness of local prototypes clustering on different domains.
Discussion on wider range scenario our method, including limitation and broader impact.
B Detailed Algorithm of FedPLVM
We summarize proposed method, FedPLVM in the Alg.
Algorithm FedPLVM
Input:
Communication rounds T,, local training epochs E, number of classes M, number of clients K, private dataset Dx {xi, w:) M.M.
Output: Global model w
Server Aggregation:
1: for = 1,2, -.., T do
2:
for k = 2... K do
Collect local models and clustered prototypes E {Pr 1M
Local Update
(k, w²
4:
end for
5: Aggregate collected prototypes {pm 1.
mm=1 6: Generate global clustered prototypes & in Eq.
7: Aggregate global model w++1 Σ W
K
8: end for
Local Update(ki, w², 9):
1: wk,1 w²
2: for = 1,2, E do
3: Update w/ from whe using 9 by Eq. 10
4: end for
5: Compute local feature vectors {h(x;) No for Dj
6: Generate local clustered prototypes (pm J.m.MM.. m= in Eq.
7: Return w/ 2 {Pm) M
1m=1
C Further results on DomainNet
Due to page limitation, put some further experimental results in Sec. 5.1
here:
14

---

Table 6: Test accuracy on DomainNet. Details in Sec. 5.1
DomainNet
Methods
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Avg
A
FedAvg
50.62 = 0.58
27.31 = 0.91
42.71 = 2.72
14.11 + 1.26
43.71
3.39
34.67
2.80
35.52
FedProx
52.33 + 1.25
28.22 + 0.11
42.56 + 3.32
13.50 + 0.94
45.85 + 1.19
35.46 = 2.98
36.32
+0.80
FedProto
54.08 = 2.11
30.44 1.70
47.20
3.34
19,40
2.07
44.22 2.61
40,99
+5.47
FedPCL
$3.37 = 1.76
29.53
46.69
3.51
16.32 1.22
51.36
43.32 0.12
40.10
+4.58
FedFA
53.20 = 0.50
29.22
1.50
47.12
17.90
50.14 1.37
45.76 = 1.61
40.56
+5.04
FPL
53.13 = 0.65
27.55 = 1.86
45.40 = 2.29
17.47 - 1.36
50.64 + 1.33
47.64 = 2.45
40.31
+4.79
Ours
54.19 1.73
31.14
1.68
47.22
1.10
22.40
1.91
51.66
48.70
42.55
+7.03
Cipari
Ours
Quickdraw
AVG
80
FPL
Accuracy
40
Accuracy
66.21
84.64
59.88
81.60
66.72
30
80
62.04
80.05
20
10
40
0.125
0.250
0.500
1,000
0.1
0.5
1.0
5.0
10.0
a
Distribution Parameter
Figure 6: Impact of a on the DomainNet Figure 7: Test accuracy on DomainNet with dataset.
different Dirichlet distribution parameters.
D Label Non-I.I.D. Setting
We expanded our method's assessment on Digit-5 and Office-10 10 within a non-i.i.d label setting. Utilizing the Dirichlet method (a: 0.5), shaped the data distribution and generated non-i.i.d. datasets for individual clients. Notably, our approach showcased substantial accuracy improvements compared baseline methods across both datasets. Non-i.i.d. distributions the variation in features across non non-identically distributed data is reflected by the colors of the dots, whereas the non-identical label distribution (a 0.5) is denoted by the sizes of the dots. For the top figure in Fig. [ client index 0, 1, 2, 3, own the datasets MNIST, SVHN, USPS, Synth and MNIST-M respectively; for the bottom figure, client index 0, 1, 2, represent the dataset domains Amazon, Caltech, DSLR and Webcam repectively.
visualized in Fig. g
Client Index
Client Index
Closs Index 4 0
2 Class
Num Samples
Num of Samples
12
24
Client Index
Client Index
Figure 8: Visualization of data distributions under label non-i.i.d. setting. Every point on graph symbolizes collection of samples belonging specific class assigned to client. The left figure is from digit-5 and the right figure from office-10. Details in Sec. D
The outcomes are presented in Tab. and Tab. g Consistent with our findings in the i.i.d. label setting, we observed significant performance boost in domains that pose greater learning challenges. For instance, within the Digit-5 dataset, our method achieved 2. 2.24% accuracy surge in the Synth domain a more complex
showing 0.94% increase in accuracy for the comparatively
15

---

easier USPS dataset. This reiterates our method's s aim: addressing disparate learning difficulties among domains, fostering fairness, and bolstering performance, especially in tougher domains. Similar advancements were evident within the Office-10 10 dataset.
Table 7: Test accuracy Digit-5 under label non-i.i.d setting. Avg means average results among all clients. We apply the Dirichlet method (a = 0.5) to obtain the data distribution and create the non-iid. dataset for each client. Details in Sec. D
Digit-5
Methods
MNIST
SVHN
USPS
Synth
MNIST-M
Avg
A
FedAvg
79.44 + 0.86
24.16 + 0.91
65.46 + 1.86
40.58 = 0.87
55.16 + 1.24
52.96
FedProx
81.40 0.53
24.44
1.06
67.54 + 2.05
41.64 1.02
55.80 0.58
54.16
+1.20
FedProto
83.28 0.80
23.06 0.51
69.98
0.81
40.78 = 0.99
57.66
0.85
$4.95
+1.99
FedPCL.
85.44 2.13
25.82
0.97
69.28+3.77
42.60
1.55
1.46 58.66
56,36
+3.40
FedFA
86.65 + 0.69
29.57 + 1.97
73.48 0.77
47.19
= 0.49
60.24 1.24
59.43
+6.47
FPL
85.18 + 2.66
31.54 + 2.96
72.96
+ 2.41
49.48
2.42
61.08 + 2.10
60.05
+7.09
Ours
86.18 1 0.98
33.24 1 1.31
73.98
1 2.17
51.72 1.08
62.90 + 1.32
61.60
+8.64
Table 8: Test accuracy on Office-10 under label non-i.i.d setting. Avg means average results
among all clients. We apply the Dirichlet method (a 0.5) to obtain the data distribution and create the non-i.i.d. dataset for each client. Details in Sec.
Office-
Methods
Amazon
Caltech
DSLR
Webcam
Avg
FedAvg
47.36 1.60
33.63 + 0.21
49.42 + 1.37
72.95 1.38
50.84
FedProx
48.58 0.76
33.78 : 0.63
52.42 + 5.31
75.10 0.80
52.47
+1.63
FedProto
52.44 1.86
35.56 1.09
51.38 + 0.19
75.62
= 2.11
53.75
+2.91
FedPCL
54.07 0.76
36.56 2.18
57.62 0.11
76.42 0.82
56.17
+5.33
FedFA
53.28 + 1.72
36.67 = 1.49
53.71
73.20
2.52
54.22
+3.38
FPL
53.05 + 0.50
35,41
+ 1.05
58.67
2.95
76.62
+ 1.60
55.94
+5.10
Ours
55.28 0.76
37.48 + 0.21
1.90 59.79
77.36 0.08
57.48
+6.64
E
Different Clustering Algorithms
In our paper, FINCH is employed for our dual-level prototype clustering due to its parameter- free nature. Here select the one with the minimum number of cluster centers among several possible clustering schemes generated by FINCH. This characteristic obviates the need for hyper-paramene parameter tuming typically required in clustering methods, such as selecting the number of centers in K-Mean algorithm. However, this does not imply exclusivity to the FINCH clustering algorithm within our method. Our approach is compatible with various clustering algorithms. including simpler methods like K-Means. The experiments result in Tab. sshow that with a carefully tuned K-Means algorithm (precisely determining the appropriate number of centers), performance comparable to that achieved using FINCH. Conversely, poorly tuned parameters result in weaker performance with the K-Means clustering. This reinforces choice of FINCH, for its parameter-fre free advantage.
method with K-Means achieves
Table 9: Comparison with K-Means Algorithm. Adaptive K means use the number of clustering centers from FINCH as K. Details in Sec.
Local
Global
MNIST
SVHN
USPS
Synth
MNIST-M
Avg
Adaptive K K=2 5=5 FINCH
Adaptive K K=2
89.97
42.10
85.83
60.90
66.03
68.97
89.53
40.67
85.33
59.13
63.53
67.64
K=5
89.72
41.97
85.41
60.13
64.69
68.38
FINCH
90.70
42.08
86.24
60,08
67.16
69.25
F Unbalanced Clients Distribution Setting
We further explore the performance of our FedPLVM in handling unbalanced client distributions, mirroring the scenario demonstrated in FPL. Within pool of 10 clients, the data ownership is
16

---

distributed as follows: 1 client possesses MNIST domain data, 4 clients hold SVHN domain data, 2 clients possess USPS domain data, clients hold Synth domain data, and client owns MNIST-M domain data.
The outcomes, presented in Tab. T0 showcase our approach's equitable performance, notably in enhancing test accuracy on clients dealing with the more challenging datasets. Comparatively, in contrast FPL, our method displays significant 2.94% improvement SVHN dataset performance. It is worth noting that our 'avg' column in Tab. TO represents the average test accuracy across all clients, differing from FPL, which calculates the average test accuracy first among clients with the same dataset and then averages across datasets. This distinction addresses potential unfairness, wherein a higher representation of clients with 'easy' datasets could disproportionatel influence the final accuracy in FPL.
Table 10: Comparison on unbalanced clients distribution. Test accuracy on each dataset domain is the average result among all clients that own the corresponding dataset. Avg means average results among clients. Details in Sec.
Methods
MNIST
SVHN
USPS
Synth
MNIST-M
Avg
FPL
90.44 + 0.67
59.78 + 1.56
85.14 1:44:000 0.93
73.01 + 1.09
69.34 = 0.98
71.52
Ours
90.90 = 0.97
62.72 1.69
85,92 0.75
74.51 + 1.20
71.32 = 0.35
73,40
G Privacy Protection
In our method, the server only receives clustered local prototypes, making it challenging to reconstruct the clients' local datasets. We also employ a differential privacy (DP) technology to validate the impact of FedPLVM. Each client trains local model by DP-SGI) [ to perturb model parameters. The privacy budget and approximate parameter ō
The noise multiplier is determined by 12 are set as 4.0 and 1e- 5 respectively for a (e, 6DD setting. Meanwhile, we also incorporate the clustered local prototypes with the privacy protection technique. We set the scale parameter and the perturbation coefficient p = 0.1 for the Gaussian noise distribution generation of our local clustered prototypes. As shown in Tab. 11 the average accuracy drops at most 0.77% we employ the privacy protection only for the prototypes. The approximate DP causes slightly larger decrease in accuracy, up to 2.45%. Note that even with the privacy protection technologies, our method reaches a comparable performance to the most advanced baseline in Tab.
0.05
Table 11: Impact of differential privacy. Avg means average results among all clients. w/ and w/o represents we incorporate the local model or the local clustered prototypes with the privacy protection technologies or not. Details in Sec. G
Model Prototypes
MNIST
SVHN
USPS
Synth
MNIST-M
Avg
w/
w/
89.74 + 0.35
38.40 + 2.86
84.44
= 0.48
55.98
= 1.27
62.44
1.25
66.20
w/ w/o
89.42
0.85
38.12 + 1.12
83.68
0.43
57.54
1.74
65.26
0.51
66.80
90.64
0.34
40.12
1.95
86.40
0.99
58.80
1.53
66.44
68.48
w/o w/o
90,70 0.39
42.08
1 1.59
86.24
1.37
60.08
1.47
67.16
0.77
69.25
H Global Prototypes Generation
The determination of the number of global prototypes for a class contingent on the associated learn- ing challenges. In instances where class within domain presents significant learning difficulties, it characterized by large variance. Consequently, employing single global prototype, reducing the sufficiency of variance information, could decay the learning performance. Incorporating multiple global prototypes for such a class can introduce additional variance information, thereby enhancing the learning process. We provide averaged experiment results in Tab. T2 (Note in the main paper we only report the single experiment to be consistent with the corresponding visualization.) The results validate the necessity of our dual-level clustering.
17

---

Table 12; Comparison with Local Cluster & Global Average. Details in Sec.
Local
Global
MNIST
SVHN
USPS
Syath
MNIST-M
Avg
Cluster
Avg Cluster
89.43 = 0.55
40.57 = 1.21
85.74 = 0.89
59.77 + 0.88
65.03 + 1.02
68.11
Cluster
90.70 + 0.39
42.08 + 1.59
86.24 + 1.37
60.08 + 1.47
67.16 + 0.77
69.25
3.5
of Clustered Local Prototypes
SVHN
AVG
3.0
2.5
2.0
1.5
# 1.0
5
20
35
50
65
80
95
Global Round
Figure 9: Tendency of average number of local clustered prototypes for different classes in different domains. Details in Sec.
I Local Prototypes Clustering
Local prototype clustering is beneficial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Easy domains tend to show tight clustering within the same category and clear distinctions between different categories, facilitating accurate classification. In contrast, hard domains often exhibit looser clustering, increasing the risk of category boundaries. Therefore, only capturing average
misclassification, especially for samples data suffices for easy domains but falls short for hard domains.
Our proposed method captures more feature distribution variation information by introducing the local prototypes clustering, since clustering provides more prototypes compared to simply averaging especially considering the sparse distribution in hard domains shown in Fig. For easy domains, where the average sufficient, our method generates fewer prototypes.
To demonstrate this, we refer to Fig. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08% in experiment of impact of local prototypes clustering in Tab, 3 also supports this observation.
J Discussion Wider Range of Scenarios
J.1 Limitation.
While FedPLVM introduces innovative strategies address domain heterogeneity in federated learning, it also comes with certain limitations and challenges. For example, noisy labels, or incorrectly labeled data instances, are a common issue in real-world datasets. few previous works has discussed the related impact in FL scenario [37 38 36). This noisy case will bring two problems for prototype learning FL: Prototype Distortion. Noisy labels can lead to distorted prototype representations, the local prototypes clustered for each class may be influenced by mislabeled instances. This distortion can propagate throughout the training process, affecting the model's ability to accurately capture the true underlying data distribution. Prototype Ambiguity. In the presence
18

---

of noisy labels, the distinction between different classes becomes less clear, leading to ambiguity in prototype definitions. Prototypes may longer accurately represent the true characteristics their respective classes, making it difficult for the model to generalize effectively. These make noisy labels in FL. scenario an interesting direction for future extension to our proposed FedPLVM.
J.2 Broader Impact.
FedPLVM primarily addresses the challenges of cross ross-domain federated learning, where data from heterogeneous domains need be collaboratively utilized while preserving privacy. This capability enables the application of FedPLVM in various real-life scenarios, including: Healthcare Data Collaboration. Different hospitals or healthcare institutions often possess diverse patient datasets with varying characteristics and distributions. FedPLVM allows these institutions to collaboratively train machine learning models for tasks such as disease diagnosis, patient outcome prediction, or personalized treatment recommendation while ensuring data privacy and security. Internet of Things (IoT)) Networks. IoT devices deployed different locations or environments generate heterogeneous data streams, including sensor readings, environmental data, and user interactions. FedPLVM enables federated learning across IoT networks, facilitating tasks such as anomaly detection, predictive maintenance, environmental monitoring without centralized data collection, thus preserving user privacy and reducing communication overhead.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Details can be found in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Details can be found in Sec. J
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details be found in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We will provide the code soon in the following github link httts::/ggitub com/TriSttonewaggFaaaaaaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/ public/guides/ howpess/oooouuubsssssoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ioweeeomiisiiiiooooo
//nips. //nips.cc/ppl cc/public/guu
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details be found in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Details can be found in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details can be found in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Details can be found in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Details can be found in Sec.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "GZ9d966QEz", "reviewer": "Reviewer_e95T", "summary": "The domain gap among multiple clients impedes the generalization of federated learning. To mitigate cross-domain feature representation variance, the authors introduce FedPLVM, which establishes variance-aware dual-level prototypes clustering and employs a novel α-sparsity prototype loss. To verify the effectiveness of the proposed method, extensive experiments are conducted.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well organized and written in a way that is easy to understand.\n2. The experimental design is reasonable and a large number of experiments have also proved the effectiveness of the proposed methodology.", "weaknesses": "1. It makes sense to perform global level prototype clustering on the server side, while performing intra-class prototype clustering on the local side makes unclear sense, why it is advantageous to get more prototypes for hard domain samples?\n2. In Eq. (8), what is meaning of C_y?\n3. From Fig. 4, \\alpha has a significant impact on the results, which is not conducive to the robustness of the algorithm.\n4. The latest compared method in the paper is from 2023 and lacks the latest comparison method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. Here is our response to the mentioned weaknesses:\n1.  Local prototype clustering is beneficial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Easy domains tend to show tight clustering within the same category and clear distinctions between different categories, facilitating accurate classification. In contrast, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average data suffices for easy domains but falls short for hard domains. \n\n    Our proposed method captures more feature distribution variation information by introducing the local prototypes clustering, since clustering provides more prototypes compared to simply averaging especially considering the sparse distribution in hard domains as shown in Figure 3 of our paper. For easy domains, where the average is sufficient, our method generates fewer prototypes. \n\n    To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08%  in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation.\n\n2. $C_y$ is the number of clustered prototypes for the class $y$.\n\n3. Firstly, our extensive experiments indicate that maintaining $\\alpha$ within the stable range of 0.125 to 0.25 consistently yields the best performance. We further conduct the ablation study on the large-scale Domain dataset. As shown in Figure 2 of the attached pdf file, we can still observe a performance benefit from the stable range.\n\n    Secondly, it's important to note that $\\alpha$ is a crucial element in our proposed $\\alpha$-sparsity loss. If $\\alpha$ is set to 1, our $\\alpha$-sparsity loss reduces to a standard contrastive loss, although it still includes a specially designed corrective term (and also the proposed dual-level prototypes clustering operation). Therefore, it's natural for $\\alpha$ to impact the results. \n\n    Lastly, in our ablation study (refer to Figure 4 in the main paper), although the results vary with $\\alpha$, our method consistently outperforms all baseline methods, demonstrating its robustness.\n\n4. To the best of our knowledge, there is still no newer work under the same setting. If you are aware of any such studies, please let us know, and we will compare our work with them. Furthermore, there is one recent fair federated learning work, FedHeal [1], which compares with other fair federated learning methods by evaluating whether they can work well with some baseline methods used in our work, such as FedAVG and FedProto and bring any benefits. Although it operates under different setting compared to ours and does not compare with any baseline works in our setting, we conducted experiments to demonstrate that our proposed framework performs better than the SOTA method FPL + FedHeal in [1]. From Table 1 of the attached pdf file, we can observe that while FPL gains a performance boost from FedHeal, our proposed method still outperforms it.\n\n    [1]. Chen Y, Huang W, Ye M. Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 12077-12086.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m85xCGDMWk", "reviewer": "Reviewer_GDmh", "summary": "The paper introduces FedPLVM, a federated learning approach that improves federated prototype learning (FedPL) in heterogeneous data domain setting. Traditional FedPL methods create the same number of prototypes for each client, leading to performance disparities across clients with different data distributions. FedPLVM mitigates this by introducing variance-aware dual-level prototype clustering and an α-sparsity prototype loss. Specifically, each client first clusters the feature vectors of all same-class local samples to get local clustered prototypes. Then,  the server conducts global prototype clustering to reduce the number of prototypes for each class. Experiments show that FedPLVM outperforms the other baselines on the data from multiple domains.", "questions": "1. From Algorithm 1, the goal of FedPLVM is to train a single global model. From the experiments, the model is tested on the local datasets. Given data from different domains, why don’t the parties train personalized local models with personalized federated learning?\n\n2. The paper mentioned that more prototypes are needed for harder domains. For the clustering method, is there a guarantee that the harder domain will also get more local prototypes? The paper may add more analysis or experiments to demonstrate it.\n\n3. Will FedPL have significant higher computation and communication cost than FedAVg?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear. It makes sense to get different prototypes under tasks with different difficulties.\n\n2. The proposed \\alpha-sparsity prototype loss is interesting. The ablation study about the loss and dual-level prototype generation is clear.", "weaknesses": "1. The paper misses important baselines. Personalized federated learning studies are not studied, which should be suitable in the studied setting. \n\n2. The computation and communication cost of FedPLVM compared with other studies is not presented.\n\n3. The clustering method needs more details. It is not clear why clustering can address the heterogeneous domain issue. \n\n4. The work is incremental work based on FedPL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. Here is our response to the mentioned weaknesses:\n\n1. Firstly, we want to highlight that the main objective of this paper is to develop a global model that works across all domain datasets. This approach aligns with the baseline methods, such as FedProto and FPL, which also address variations in client data domains. \n\n    Secondly, training a unified global model, as opposed to personalized models, offers distinct advantages. For instance, a potential use case involves applying the trained model to broader scenarios, such as when a client has data from multiple domains and the domain of each data sample is unknown. In such cases, a global model remains effective, whereas personalized models may not be suitable. \n\n    Lastly, developing a strong global model enhances our ability to implement personalization effectively. While our method can integrate personalization techniques, it is not the current focus of our work. We plan to explore personalization in more depth in our future research.\n\n2. For the computation cost of FedPLVM, the feature representations of data samples can be obtained directly from the model without any additional computation. Our method introduces the dual-level prototypes clustering compared to other baselines, and we test the average running time for local training (0.0296s), local prototypes clustering (0.552s) and global prototypes clustering (0.0214s). The additional local prototypes clustering in dual-level clustering step is negligible, accounting for only about 0.0296 / 0.552 ≈ 5% of the local training time. Meanwhile, the proposed \\alpha-sparsity loss does not change the network structure.\n\n    For the communication cost of FedPLVM, each client uploads local clustered prototypes along with the local model to the server. The client then downloads the global clustered prototypes and the global model for the next round of local training. Our employed ResNet-10 has approximately 4.9 million parameters. Each prototype is a 512-dimension vector, so the maximum number of uploaded parameters for the local clustered prototypes is approximately 3.2 (from the highest average number of local clustered prototypes of the red line in Figure 1 of the attached pdf file) * 10 (number of classes) * 512 = 0.016 million and the estimated value of the downloaded parameters for the global clustered prototypes can be 21.20 (from table 3 in the paper) * 512 = 0.011 million.  Both are negligible compared to the model size. Notably, the previous work FedPCL requires each client to download the prototypes from all other clients via the server, making our clustering prototypes operation more efficient and feasible.\n\n3. Local prototype clustering is beneficial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Easy domains tend to show tight clustering within the same category and clear distinctions between different categories, facilitating accurate classification. In contrast, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average data suffices for easy domains but falls short for hard domains. \n\n    To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08%  in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation.\n\n4. We respectfully disagree. There are a few key distinctions between our work and FedPL. \n\n    Firstly, while both our method and FPL employ prototype clustering, the objectives and implementations are markedly different. FPL performs single-level global clustering, intended to balance the impact of local prototypes from each domain, involving only server-level clustering.  This single-level global clustering is specifically designed to address imbalances in client distribution across domains, aiming to neutralize the skewed influence of domains with more clients on global model training. Conversely, our method integrates dual-level clustering at both the client (local) and server (global) levels. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, enhancing both efficiency and privacy. Our local prototype clustering is crucial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. Hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average local prototype suffices for easy domains but falls short for hard domains. \n\n    Secondly, we introduce an innovative α-sparsity prototype loss, highlighted as “interesting” in the strengths part of your review, featuring a corrective component, which mitigates the potential risk of feature representation overlapping caused by the dual-level clustered prototypes operation. This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.\n\nFor Question 1, 2 and 3, please refer to our response to Weakness 1, 3 and 2 respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzPeXvJRH9", "reviewer": "Reviewer_dkgn", "summary": "This paper aims to investigate the federated prototype learning problem with data heterogeneity. To handle the cross-domain representation variance problem, a new method termed FedPLM is proposed, which includes a dual-level prototype clustering mechanism and an alpha-sparsity prototype loss. Experiments are conducted to show the effectiveness of the proposed method.", "questions": "The authors are expected to address the concerns in the block of \"Weaknesses\". Also, I have one more question: How to balance the two sub-terms (contra and corr) in alpha loss? As they may be on different scales, is that feasible to directly add them together?", "limitations": "Limitations are discussed in the paper. For the negative societal impact, I didn't find any concern from my side.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is well-designed for the specific research problem and has good novelty.\n\n2. The motivation of the paper is solid. \n\n3. The experiments are comprehensive to discuss the properties of the proposed method.\n\n4. The paper is well-written and easy to follow.", "weaknesses": "1. On several datasets (e.g. MNIST and USPS), the performance of the proposed method is not significant enough. Can the author give more explanation for this point?\n\n2. The running efficiency of the proposed method is not given. Complexity analysis and efficiency-related experiments are expected to discuss the efficiency of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. Here is our response to the mentioned weaknesses:\n\n1. Our design principle aims to ensure our method excels particularly in challenging domains, which often have lower baseline accuracy, such as SVHN. This aligns with our motivation to tackle diverse learning challenges across various domains. In more challenging domains, the dual-level clustering method generates multiple clustered local prototypes that capture variance information effectively, thus enhancing focus and fairness, as shown in Figure 1 of the attached pdf file. For easier datasets like MNIST, where feature representations naturally concentrate, our local clustering tends to produce a centralized prototype similar to the traditional average method, leading to a smaller performance gap.\n\n    We focus on hard domains beacuse in real-world settings, difficult domains are common and pose significant challenges that need effective solutions. Previous works often struggle in these hard domains, and our method addresses this gap by ensuring robust performance improvements where they are most needed. \n\n    Contrary to the concern about insignificant results, our method does show considerable improvements in challenging domains. For instance, on the SVHN, Synth and MNIST-M, our approach achieved a notable performance gain of 5.3%, 2.72% and 3.14% respectively compared to SOTA methods. This demonstrates the effectiveness and robustness of our method in handling difficult datasets, which are critical for practical applications. \n\n2. For the computation cost of FedPLVM, the feature representations of data samples can be obtained directly from the model without any additional computation. Our method introduces the dual-level prototypes clustering compared to other baselines, and we test the average running time for local training (0.0296s), local prototypes clustering (0.552s) and global prototypes clustering (0.0214s). Compared to the local training, our additional dual-level clustering step is negligible (approximately 0.0296 / 0.552 ≈ 5% of the local training time). Meanwhile, the proposed \\alpha-sparsity loss does not change the network structure.\n\n    For the communication cost of FedPLVM, we need each client to upload local clustered prototypes with the local model to the server and the client will then download the global clustered prototypes with the global model for the local training of the next round. Our modified lightweight ResNet-10 has approximately 4.9 million parameters. In our network, one prototype is only a 512-dimension vector, so the maximum number of total uploaded parameters for the local clustered prototypes can be approximately 3.2 (from the highest point of the red line in the above picture ) * 10 (number of classes) * 512 = 0.016 million and the estimated value of the downloaded parameters for the global clustered prototypes can be 21.20 (from table 3 in the paper) * 512 = 0.011 million which are both ignorable compared to the size of the model. Note the previous work FedPCL requires each client download the prototypes from all other clients from the server, which makes our clustering prototypes operation more realizable.\n\nQuestion 1: A straightforward approach to balance the two sub-terms is introducing another hyper-parameter to control the weight. However, to maintain the simplicity of our proposed \\alpha loss, we can adjust the temperature $\\tau$ in the contrastive term to balance the two sub-terms. In our experiments, using our default $\\tau$, the average loss value of the first term in the 50th round out of 100 is 0.364, while the second term is 0.105, demonstrating comparable values.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j1JFosfe7m", "reviewer": "Reviewer_8nBN", "summary": "This paper focuses on Federated Prototypes Learning and reveals that existing methods create the same prototypes for different clients, which neglects the distribution diversity. In this work, the authors introduce the variance-aware dual-level prototype clustering and alpha sparsity prototype loss. Various experiments demonstrate the effectiveness of the proposed method.", "questions": "I encourage the author to take careful thinking for the federated prototype learning field. The existing solution shares a high similarity with the FPL (CVPR'23)  and seems like an incremental work.", "limitations": "Yes, the authors have discussed.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Authors conduct comprehensive experiments to demonstrate the effectiveness.", "weaknesses": "This paper has several drawbacks.\n\n1. Motivation for the α-Sparsity Prototype Loss. It is a little strange that \"This multiplicity of prototypes could potentially lead to overlapping feature representations among different classes, especially in challenging client scenarios\". As the authors claim the feature overlapping, the naive solution is to use other ways to cover the local feature behavior rather than multiple local prototypes.  Furthermore, I assume that authors could employ the OT (optimal transport) to require them to concentrate on different parts. I do not think the current operation is a suitable way to deal with the feature overlapping.\n\n2. The paper architecture is not suitable. I wonder why authors spend a lot of space to write meaningless or complicated words and even leave no space for DomainNet and Office-10. Furthermore, I encourage to consider the label skew effect, ie, with different label skew degree. Besides, the experiments on large scale of clients are also important.\n\n3. The Figure 1 is confusing. Why monitor would capture the digitis figure? It is a not soundable problem figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. Here is our response to the mentioned weaknesses:\n\n1. We introduce \\alpha-sparsity loss to mitigate the potential risk of feature representation overlapping caused by the dual-level clustered prototypes operation. This loss focuses on maximizing inter-class distance and putting more attention on expanding the overall feature distribution to a broader range, thereby solving the feature overlapping problem efficiently and simply. While Optimal Transport (OT) may be employed in our framework, it introduces additional significant computational complexity [1]. Given our experimental results, it is effective to use our proposed \\alpha-sparsity loss due to its simplicity.\n\n    [1]. Peyré G, Cuturi M. Computational optimal transport: With applications to data science[J]. Foundations and Trends® in Machine Learning, 2019, 11(5-6): 355-607.\n\n2. We apologize for any confusion caused by our paper's structure. Could you please provide us with a range of 'meaningless' and 'complicated' words so we can revise them accordingly? Moreover, we can move the DomainNet and Officie-10 parts from the Appendix to the main text per your suggestion in our final version. \n\n    Regarding the label skew effect, we conduct label non-IID experiments using the Dirichlet method with a distribution parameter of 0.5, as detailed in Section E. The label distribution becomes more non-IID as the distribution parameter decreases. To illustrate the impact of different distribution parameters (0.1, 0.5, 1, 5, and 10), we compare the average accuracy with the best baseline method FPL from our previous non-IID experiment on Digit-5. Figure 3 of the attached pdf file shows that our method consistently outperforms FPL across all distribution parameters. However, the performance gap narrows as the datasets become more non-IID. This trend is expected, as in non-IID datasets, the number of samples for some classes can be very small, leading to less representative feature representations. Consequently, our clustered local prototypes may become similar to the averaged single prototype used in FPL.\n\n    We also conduct the experiments on a larger scale of 20 clients, the same number in FPL, and the datasets of per four clients come from the same domain. As shown in Table 1 of the attached pdf file, our method still outperforms FPL under such a setting. Note we have already explored the unbalanced clients distribution setting of 10 clients in Section G.\n\nQuestion 1: \n\n  Firstly, while both our method and FPL employ prototype clustering, the objectives and implementations are markedly different. FPL performs single-level global clustering, intended to balance the impact of local prototypes from each domain, involving only server-level clustering.  This single-level global clustering is specifically designed to address imbalances in client distribution across domains, aiming to neutralize the skewed influence of domains with more clients on global model training. Conversely, our method integrates dual-level clustering at both the client (local) and server (global) levels. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, enhancing both efficiency and privacy. Our local prototype clustering is crucial because it captures essential variance information, not just the average data, which is particularly crucial in hard domains. hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Therefore, only capturing average local prototype suffices for easy domains but falls short for hard domains. To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08%  in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation.\n\n  Secondly, we introduce an innovative $\\alpha$-sparsity prototype loss featuring a corrective component, which mitigates the potential risk of feature representation overlapping caused by the dual-level clustered prototypes operation.. This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0RcWg6YS1", "reviewer": "Reviewer_gr7B", "summary": "This paper studies federated learning under heterogeneous data domains and introduces a federated prototype learning strategy, denoted as FedPLVM, to mitigate the problem. A dual-level prototype generation method is proposed to address domain variance between hard and easy domains, reducing the communication burden between clients and the server. Moreover, an \\alpha-sparsity prototype loss is proposed to enhance the sparsity of the prototype representations. The experiment results demonstrate the efficacy of the proposed method and the necessity of each proposed module.", "questions": "This paper is well-organized and the experiment results significantly surpass existing methods. But the level of innovation is questionable: the dual-level prototype generation is similar to FPL, and the only key difference is the \\alpha-sparsity prototype loss. Moreover, I also have some concerns on this paper. For more details, please check the weakness part.", "limitations": "Yes. The authors addressed limitations of the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is overall well-written and easy to follow.\n2. The authors explain why models exhibit varying performance across domains.\n3. The experiment results show the superiority of the proposed method compared to SOTA methods. The ablation results also indicate the effectiveness of each proposed module.", "weaknesses": "1. The proposed dual-level clustering strategy appears similar to FPL. What is the key difference between them? The primary distinction lies in the proposed \\alpha-sparsity loss, which introduces incremental innovation to the paper.\n2. The proposed dual-level prototype strategy utilizes FINCH for clustering. However, the clustering results from FINCH can vary at different steps. Which step's result is selected for use? Do results from different steps vary?\n3. The impact of \\alpha is only evaluated on small-scale datasets. Does the trend consist with big-scale datasets like DomainNet?\n4. In Sec. G, the authors claim that the evaluation protocol used in FPL is unfair. However, the protocol used in FPL is sensitive to both easy and hard datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. Here is our response to the mentioned weaknesses:\n\n1. The key point of our dual-level clustering is local prototypes clustering. Different from easy domains, hard domains often exhibit looser clustering, increasing the risk of misclassification, especially for samples near category boundaries. Our dual-level prototypes clustering captures more feature distribution variation information by introducing the local prototypes clustering operation, since clustering locally provides more prototypes compared to simply averaging especially considering the sparse distribution in hard domains as shown in Figure 3 of our paper. \n\n    To demonstrate this, we refer to Figure 1 of the attached pdf file. The y-axis shows the average number of local prototypes among classes generated at each selected round. Note that the number of prototypes for different classes can vary, resulting in non-integer averages. The easy domain (MNIST) has fewer prototypes generated compared to the hard domain (SVHN), showing that in hard domains, more prototypes are utilized to better capture the variance information. Furthermore, an average performance gain of 3.08%  in experiment of impact of local prototypes clustering in Table 2 of our paper also supports this observation. At the global level, our clustering aims to reduce communication overhead and privacy risks by limiting the prototype variety each client sends, enhancing both efficiency and privacy while FPL intends to balance the impact of local prototypes from each domain, involving only server-level clustering.\n\n    Meanwhile, our proposed α-sparsity prototype loss is also innovated and designed to complement the dual-level clustered prototypes operation. This loss function origins from reducing the potential risk of feature representation overlapping caused by our dual-level clustered prototypes operation. This loss function reasonably utilizes the variance information to reduce feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning. \n\n2. For each clustering operation, the installed FINCH python project provides several clustering results and we select the one with the smallest number of clustering prototypes. During different rounds, the number of local clustered prototypes vary for different classes among different clients, generally decreasing as training progresses. From Figure 1 of the attached pdf file, we observe that the average number of clustered local prototypes consistently decreases over the training process. This indicates that our local prototypes clustering method aligns with the convergence process among different domains. \n\n3. We further conduct the ablation study of \\alpha on the DomainNet dataset as shown in Figure 2 of the attaced pdf file. Our extensive experiments indicate that maintaining $\\alpha$ within the stable range of 0.125 to 0.25 consistently yields the best performance. Although the results vary with different values of $\\alpha$, our method consistently outperforms all baseline methods, demonstrating its robustness. \n\n4. In FPL, the client domain distribution is unbalanced, with data ownership distributed among 20 clients as follows: MNIST (3 clients), USPS (7 clients), SVHN (6 clients), and SYN (4 clients). However, the reported final average accuracy is computed by summing the results from all clients and dividing by 20, which gives more weight to domains with more clients, leading to an unfair representation.\n\n    In Section G, we address this issue by first averaging the results within each domain based on the number of clients in that domain. We then average these domain-specific results based on the number of domains. This weighted averaging ensures that all domains are given equal weight, regardless of the number of clients they have.\n\nFor Question 1, please refer to our response to Weakness 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all reviewers for their insightful feedback and recognition of our work. We particularly appreciate Reviewer GDmh’s comment on the well-designed of our proposed $\alpha$-sparsity prototype loss, Reviewer dkgn's comment on the novelty of our proposed dual-level prototypes clustering method and the other reviewers’ affirmation that our experiments are comprehensive and validate our proposed method effectively. We also value the feedback pointing out areas of weakness and the questions raised about our paper. We prepare the separate respective response to each reviewer about their specific concerns. Below, we first provide a consolidated response addressing the main concerns raised by multiple reviewers:

1. Benefit from Local Prototype Clustering:
    
    Local prototype clustering effectively captures essential variance information, not just averages, which is vital in complex domains. Our method generates more prototypes for challenging domains to better capture this variability, while fewer prototypes are sufficient in simpler domains where averages provide enough detail. This strategy is depicted in Figure 1 of the attached pdf file, which shows fewer prototypes in the easier domain (MNIST) and more in the harder domain (SVHN), highlighting the necessity for more prototypes in complex scenarios to adequately capture variance.

2. Computation and Communication Costs:
     - Computation: Our approach introduces dual-level prototypes clustering, adding a minimal overhead—only about 5% of the total local training time—compared to other baselines.
     - Communication: Each client uploads only the extra local clustered prototypes to the server. Given the size of the large local model (4.9 million parameters), the additional prototypes (approximately 0.01 million parameters) are negligible.

3. Distinction from FPL: 

    Our objectives and implementations differ significantly from FPL. FPL employs single-level global clustering, aiming to harmonize local prototypes’ impact across domains, involving only server-level clustering. In contrast, our approach addresses the varying learning challenges across domains, utilizing dual-level clustering at both the client (local) and server (global) levels. At the global level, our clustering strategy reduces communication overhead and privacy risks by limiting the variety of prototypes each client sends, thus enhancing both efficiency and privacy. Our local prototype clustering is crucial in complex domains, where it captures essential variance information, critical for reducing misclassification risks near category boundaries. This is not necessary in easier domains where capturing average data suffices.

    Additionally, we introduce the innovative $\alpha$-sparsity prototype loss with a corrective component that mitigates the risk of feature representation overlap caused by dual-level clustered prototypes. This loss function effectively utilizes variance information to decrease feature similarity across different classes while increasing it within the same class, promoting more effective and stable learning.

Meta-review: The paper presents a variance-aware dual-level prototype clustering and a $\alpha$-sparsity prototype loss to tackle performance loss on hard domains during federated prototype learning. A majority of the reviewers believe that the paper is very well written and addresses an important shortcoming of the prior work federated prototype learning. 

The authors present strong experimental results to support their methodology and provide appropriate intuitions and visualizations to further improve the readability of the work. The additional results presented by the authors in the rebuttal resolved most of the concerns that were raised by the reviewers regarding (i) space and time complexity, (ii) differences with respect to the prior works, (iii) differences in performance gap among baselines on different datasets, and (iv) details of the FINCH clustering.

I would request the authors to consider these forum discussions and appropriately reorganize the paper before final submission. I believe it would improve the overall quality of the work and help the readers understand the method better.
Decision: Accept (poster)