Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Authors: Hui Zeng
Wenke Huang
Tongqing Zhou
Xinyi Wu
Guancheng Wan
Yingwen Chen
Zhiping Cai
Venue: ICML 2025 poster (2025)
Keywords: One-shot Federated Learning
Prototype Learning
Self-supervised Learning
Heterogeneous Data
Efficiency
Primary area: general_machine_learning
TL;DR: We propose a novel one-shot FL framework to address the model inconsistency in existing methods, thereby circumventing the "garbage in, garbage out" dilemma and achieving performance improvements of 10.86% without requiring auxiliary information.

Abstract: Turning the multi-round vanilla Federated Learning into one-shot FL (OFL) significantly reduces the communication burden and makes a big leap toward practical deployment. However, this work empirically and theoretically unravels that existing OFL falls into a garbage (inconsistent one-shot local models) in and garbage (degraded global model) out pitfall. The inconsistency manifests as divergent feature representations and sample predictions. This work presents a novel OFL framework FAFI that enhances the one-shot training on the client side to essentially overcome inferior local uploading. Specifically, unsupervised feature alignment and category-wise prototype learning are adopted for clients' local training to be consistent in representing local samples. On this basis, FAFI uses informativeness-aware feature fusion and prototype aggregation for global inference. Extensive experiments on three datasets demonstrate the effectiveness of FAFI, which facilitates superior performance compared with 11 OFL baselines (+10.86% accuracy). Code available at https://github.com/zenghui9977/FAFI_ICML25

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Hui Zeng" ! Wenke Huang" 2
Tongqing Zhou'
Xinyi Wu'
Guancheng Wan Yingwen Chen Zhiping Cai
Abstract
et al., 2025). Basically, FL allows clients to train local mod- els independently. collects the locally trained models for aggregation, and broadcasts the aggregated global model for iterative local training. However, such a multi-round client-server interaction process would incur a heavy com- munication burden (e.g., more than 250GB for a simple VGG19 model cumulatively (Zeng et al., 2024; Wu et al., 2020)) and high communication time (e.g.. more than 194 hours for one-round transmission of GPT-3 (Tang al., 2024a)), criticized for being prohibitive in real-world appli- cations (Zhang et al., 2022a; Dai et al., 2023; Chen et al., 2023; Tang et al., 2024a; Liu et al., 2024).
Turning the multi-round vanilla Federated Learn- ing into one-shot FL (OFL) significantly reduces the communication burden and makes a big leap toward practical deployment. However, this work empirically and theoretically unravels that exist- ing OFL falls into garbage (inconsistent one- shot local models) in and garbage (degraded global model) out pitfall. The inconsistency mani- fests as divergent feature representations and sam- ple predictions. This work presents a novel OFL framework FAFI that enhances the one-shot train- ing on the client side to essentially overcome inferior local uploading. Specifically, unsuper- vised feature alignment and category-we -eise proto- type learning are adopted for clients' local train- ing to be consistent in representing local sam- ples. On this basis, FAFI uses informativeness- aware feature fusion and prototype aggregation for global inference. Extensive experiments on three datasets demonstrate the effectiveness of FAFI, which facilitates superior performance com- pared with 11 OFL baselines (+10.86% accu- racy). Code available at https: //gttbb. htmm///enhui99977/88 ICML22
To reduce the communication costs, one-shot Federated Learning (OFL) has emerged recently by compressing the multi-round communications of vanilla FL into just one round (Guha et al., 2019). In OFI., clients perform long- term local training individually based on their private data and upload these well- local models to the server for aggregation. In this manner, OFL is believed to be well-suited for the prevalent model market scenarios (Zhang et al., 2022a; Zeng et al., 2024; Liu al., 2025), where users are willing to trade their models for next-stage knowledge fusion instead of joining into a redundant training process. In the current OFL, clients are supposed to train locally for only one time, gaining one-shot local models, while the server is expected to provide a deliberate aggregation by reforming a global model with one-shot local ones.
1. Introduction
Existing OFL aggregation designs, building on optimization- based methods (McMahan et al., 2017: Zeng et al., 2021; Jhunjhunwala et al., 2024; Liu et al., 2024; Dennis et al., 2021; Su et al., 2023), distillation-basee methods (Li et al., 2021b; Zhou et al., 2020; Yang et al., 2023), generative methods (Yang et al., 2024b; Heinbaugh et al., 2022), and selective ensemble methods (Zhang et al., 2022a; Dai et al., 2023; Zeng al., 2024), focus solely on the server side. Unfortunately, there is no free lunch to reduce the cost of communication. The reported performance of OFL methods has significant gap (over 30%) compared with multi-round FL (Tang et al., 2024a: Zeng et al., 2021; Gao et al., 2022).
As distributed machine learning paradigm featured with Federated Learning (FL) enables mul- tiple clients to collaboratively integrate their knowledge without exposing their local data (McMahan et al., 2017: Zeng et al., 2021; 2025; Karimireddy et al., 2020; Yurochkin al., 2019; Tang et al., 2024b; Huang et al., 2024; Wan
Equull contribution 'College Computer Science and Tech- nology, National University of Defense Technology, Chang- sha, China "National Engineering Research Center for Multi- media Software, School of Computer Science, Wuhan Univer- sity, Wuhan, China. Correspondence to: Tongqing Zhou <zhou- tongqing @ mudt.edu.cn. Zhiping Cai <zpcai nudt.edu.en>..
This work identifies, for the first time, that existing OFL falls in a garbage (inferior one-shot local models) in and garbage (degraded global model) out pitfall. Vanilla FL
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
overcomes such a pitfall by implicitly exchanging local knowledge via multi-round iterations while existing OFL lacks such a measure by reactively gathering the garbage inputs. We further unravel the root cause of such garbage inputs as two aspects of inconsistency in the face of data heterogeneity (8 3). (1) Intra-model inconsistency. A one- shot local model is shown to have divergent predictions for samples under the same semantics. (2) Inter-model inconsis- tency. Different one-shot local models from different clients manifest distinct parameters, causing divergent predictions for even the same sample.
of client-side Self-Alignment Local Training and server- side Informative Feature Fused Inference (FAFI), mitigating inconsistency in OFL for enhanced inference performance without requiring additional sample or local model exchang- ing.
We conduct extensive evaluations on real datasets with various levels of data heterogeneity, accompanied by a set of ablative studies. Experimental results demonstrate the performance superiority of the proposal (10.86% accuracy improvement over 11 baselines on 3 datasets) and the indis- pensability of each module.
Intuitively, the OFL paradigm would be more effective if provided with one-shot but consistent local models. How- ever, dealing with inconsistencies in OFL is non-trivial. On one hand, we find that the heterogeneity of different categories of one client's s samples leads to intra-model in- consistency. How can we construct a model capable of capturing invariant features and achieving stable predic- tions under such heterogeneous conditions (Challenge #1y? On the other hand, we observe that better local performance always accompanied by larger parameter discrepancies. A more tricky challenge is how can we effectively leverage models with parameter discrepancies in a one-shot manner (Challenge #2)?
2. One-shot Federated Learning
One-shot Federated Learning (OFL) is a variant of FL that requires only one round of interaction between the clients and server to reduce the heavy communication cost. Ex- isting OFL methods focus on designing aggregation mech- anisms the server side, which can be categorized into three categories.
(1) Optimization--annn based methods focus on reconstructing better global model with the parameters of local models. Traditional aggregation methods such as FedAvg (McMahan et al., 2017), Median (Yin et al., 2018), Krum (Blanchard et al., 2017), and FedCav (Zeng al., 2021; 2025) can be directly applied in OFL, but achieve low performance. MA- Echo (Su et al., 2023) tries to get the Pareto optimum of the local clients via exploring common harmonized optima. FedFisher (Jhunjhunwala et al., 2024) and FedLPA (Liu et al., 2024) require additional Fisher information matri- ces for reaggregation. However, the nonlinear structure of DNNs makes it difficult to obtain a comparable global model through parameter optimization (Tang et al., 2024a). Besides, directly analyzing the model parameters requires all local models to have the same architecture, whose setting is impractical in real-world heterogeneous scenarios.
In view of these challenges, this work presents novel one- shot Federated Learning framework, named FAFI. For Chal- lenge #1, we design Self- Self-Alligment Local Training (SALT), a dual-step training strategy. which leverages contrastive learning to capture invariant feature representations and em- ploys learnable category-wiss prototype to address the pre- diction inconsistency by establishing semantically aligned decision boundary. Working in an unsupervised manner, SALT could improve the generalization of the one-shot lo- cal training with a tuned feature extractor and category- templates (i.e., prototypes).
For Challenge #2, instead of directly utilizing divergent parameters, FAFI performs informativeness-avee aware feature fusion based on local local-uploaded extractors and aggregate pro- totypes for each category during inference, namely, Infor- mative Feature Fused Inference (IFFI) in the server. Specifi- cally, to reduce the impact of local extractors with uncertain understanding of a sample, the sample's 's representation from each extractor is compared with noise representation of the same extractor. Those with noise-proximal features are assigned with attenuated attention during fusion. Subse- quently, the inference is made by finding the nearest aggre- gated prototype for the fused feature of that sample.
(2) Distillation-base methods try to introduce knowledge distillation (KD) to transfer the massive local knowledge into one global model. The local models (Guha et al., 2019; Li et al., 2021b; Diao al., 2022) locally dis- tilled data (Zhou et al., 2020) are viewed as teachers and the newly constructed global model is the student. (Guha et al., 2019) firstly uses the ensemble prediction of local models as the teachers' output. FedKT (Li et al., 2021 designs a two-tier PATE structure relying on public data to improve the ensemble of local models. To alleviate the label skews, FedOV (Diao al., 2022) adopts open-set voting in OFL to enhance the generalization. F-FED (Dennis et al., 2021) runs a variant of Lloyd's method for k-means clus- tering and obtains an aggregated model through one round iteration of exchanging local cluster means. Some dataset distillation- based one-shot methods, such as FedD3 (Song et al., 2023) and FedMD (Li & Wang, 2019), transmit the
Our main contributions are summarized as follows:
We empirically and theoretically reveal model inconsis- tencies within and across local models, which leads to poor performance for existing model merging designs in OFL.
We present the novel one-shot Federated Learning design
2

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Feature
All Feature Extractors
Extractor
erggral
Fry
Lell- tele
2003
24.95
28
Flip
Inconsislency
+-0.5
+-01
Inconsistency
(b)
(a)
(b)
Figure 1. Empirical demonstration of mntra-model inconsis-
Figure 2. Empirical demonstration of inter-model inconsisteney. (a) The inconsistent features across different one-shot local models. (b) The inconsistent parameters (measured by parameter variance) and averaged training loss under different data heterogeneity. The solid blue lines represent the parameter variance, and the dotted red lines represent the average training loss.
tency. (a) The inconsistent features extracted by the one-shot local model and (b) the inconsistent predictions of the one-shot
local model on the original and flipped samples.
locally distilled dataset rather than models to the server in a one-shot manner. However, these methods require auxiliary public data pre-trained models which may be impractical in privacy-sensitive scenarios, such as biomedical domains.
3.1. Intra-modes Inconsistency
In this part, first investigate the intra- intra-model inconsistency in existing one-shot local models in heterogeneous data, which is common issue in FI., that the categories of one client's training samples are totally different (Luo et al., 2021; Li et al., 2023).
(3) Generative methods use the generative models to synthe- size proxy samples for centralized training on the server side. FedCAVE (Heinbaugh et al., 2022) modifies the local learn ing task into training conditional variation auto-encoder (CAVE) and uses KD to compress the ensemble into pow- erful decoder. The decoder can be used generate training samples for the global model. FedCADO (Yang et al., 2023) adopts the popular diffusion models to get the synthetic data. FedDEO (Yang et al., 2024b) trains local descriptions that serve as the medium for conditional generation with diffusion models. However, these generative data samples may leak the privacy of the local clients.
Empirically Demonstration. As shown in Figure and Figure 1b, the one-shot local model exhibits significant in- consistencies in their extracted feature (visualized by Grad- CAM (Zhou et al., 2016)) and prediction for identical sam- ples when subjected to simple augmented transformations, such as flipping. Besides, with the increased non-IID de- gree (lower Dirichlet Distribution parameter alpha), the intra-model inconsistency (measured by the performance gap between original and flipped samples) becomes more significant. We attribute this the heterogeneity of local data, which hinders the acquisition of a one-shot local model with good generalizability.
(4) Selective ensemble-based methods try to assign pro- portions for each local model and use them for prediction based on ensemble learning. DENSE (Zhang et al., 2022a) and Ensemble (Wang et al., 2023) equally average the pre- dictions of all local models. Co-Boosting (Dai et al., 2023) designs learnable weight for each local model and synthe- sizes data and the ensemble model mutually enhances each other progressively. IntactOFL. (Zeng et al., 2024) trains MoE for dynamic routing. However, all these methods focus on improving performance through server-side design, ignoring that the root cause of low performance in OFI is the inconsistent one-shot local models.
Theoretical Analysis. We represent the m client local model Wm = 0m 2m. where mm, is the classifier and 6,.. is the feature extractor. Existing OFL methods primarily train local models in supervised manner, using the cross-entropy loss as the supervised loss. Considering classification task with C' categories, the loss can be represented as:
Cguu (wm) = Da [eoup(Dm (4m, Omi v)) exp($2 m, OT
(I)
log
3. Motivation
Pm 1=
Σ exp(xx TT 0T
m. m. "3))
Existing OFL aggregation methods focus on better model merging designs on the server side, which easily falls in garbage in garbage out pitfall. In this part, we empirically and theoretically unravel the root cause of such 'garbage input' as two aspects of inconsistency (intra- model and inter- model levels) in the face of data heterogeneity.
where n. is the number of samples in the m-th client. The one-shot local model should have consistent predictions for samples with the same semantics. Thus, we define the intra- model inconsistency as the performance discrepancy of the model on the original samples (I, y) and augmented sam- ples (A(x), Va), where A is the data augmentation function,
3

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
such RotateX/Y and Flip. We assume that the augmented samples have significant differences from the original sam- ples without losing semantics (Cao et al., 2024), that is,
Fe is the feature of the e-th class and w 0 o is the one- shot local model. We denote Ie and = as the average input samples and prediction of all local models, respectively.
[[A(x)) - 2|| > 0, Ya-
Theorem 3.2. (Inter-model inconsistency. See proof in Appendix B). For any two client and with the same quantity of samples nu = Toe the one-step model deviation between the two clients iitter = -W, Vw, can be represented as:
We use the Aintra = [(x, y) - [(x', .)), which is the performance discrepancies between any two samples æ, x' with the same label y. We have the following analysis.
Theorem 3.1. (Intra-model inconsistency. See proof in Appendix A). The intra-modes inconsistency of the one-shot local model on the original samples (a, y) and augmented samples (A(x), v) can be represented as:
|ntter 1|
- [n In.) no,o - Zv,0) Fo,e)
| 1 2 11 (p- Vga VA) (x- A(=)) 0, (2)
- - Σ
Zu
Σ
Re )11
d'E(Cu.Ve 0,
c'E|Colle
is the prediction of w; acti- vated by softmax function with the augmented samples A(r),
where p- (cc
(3)
V is the gradient of the local model wa and VA is the gradient of the data augmentation function.
where 1I is the learning rate, Bu. and Bv.c is the sample quantity of c-th class, c is the negative classes except
Theorem 3.1 indicates that the intra-model inconsistency is inevitable in the one-shot local model when trained through existing OFL paradigms on heterogencous data. Specifically, it is mainly caused by three factors. (1) The performance on augmented samples, which is represented by pV ya- (2) The transformations property of the data augmentation function, which is represented by VA. (3) The discrepancy between the original samples and augmented samples, which is rep- resented by (ar A(x)). However, all these factors are larger than zero, leading to the existence of intra-model inconsistency.
Theorem 3.2 unravels that for any two clients, each local training step would cause the inconsistency of one-shot local models. With more local training steps E, the inter-model inconsistency becomes more significant, which can also be verified in Figure 2b.
In this section, we empirically and theoretically demon- strate the negative effect one-shot local models at both intra-model levels and inter-model levels. We show that the current OFL. local training strategies inevitably cause 'garbage' one-shot local models, making it challenging for server-side aggregation.
3.2. Inter-model Inconsistency
Notably, inconsistency exists not only within individual one-shot local models but also significantly across multiple models trained on different clients.
4. Method
4.1. Overview
Motivated by the analysis in 3, we propose a novel OFL framework, namely FAFI. It consists of two components: self-alignment local training and informative feature fused inference. (1) On the client side, we train the feature ex- tractor to enable the model to learn invariant features that can be generalized to diverse augmented samples. We also design category-wise prototype learning for distinctive pro- totypes, replacing the original classifier, thereby mitigating the negative impact the prediction. (2) On the server side, we aggregate the prototypes from all clients into global prototype. During the inference stage, we informatively fuse the features extracted by the local models to alleviate the inter-model inconsistency. The overview of FAFI is illustrated in Figure 3.
Empirically Demonstration. We visualize the features extracted by different local models for the same sample in Figure 2a. We observe that the features extracted by differ- ent local models are significantly different. Besides, we use parameter variance au = / MM ΣM - . to mea- sure the moder inconsistency of one-shot local models. As demonstrated in Figure 2b, we observe that inter-model inconsistency increases continuously during the training pro- cess, even when the loss is close to convergence. Existing research, such as 'client drift" (Gao et al., 2022; Karim- ireddy et al., 2020; Huang et al., 2023a; 2022), discusses the optimization direction inconsistency in multi-round short- term local training scenarios. Differently. our observation focuses on one-shot long-term local training, where better performance is always accompanied by larger inter-model inconsistency.
4.2. Self-alignment Local Learning
Motivation. According to Theorem 3. 1, we note that the key factor leading to intra-model inconsistency is the model's
where Theoretical Analysis. Let 25 = Ersr exp(w[aa 2x))
czp(u 22))
4

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Self- Self- Ali-nmegmmntt Local Training
Attract
Local Dutaset
Attract
Repel
Repel
Client-side:
Ropel
Augmented Datasct
Feature
Extractor
Category-
Upload
Feature Alignment
Prototype Learning
Upload
Informatve Feature Fused Inference
Noise F Features
Sample Features
Sample
Similarity to prototypes
Global prototype
Server-side:
Random Noise
Local Prototypes
Feature Fusion
Inference
Prototype Aggregation
Figure 3. An overview of the proposed FAFL. All clients perform local training and upload the trained models the server once. (1) Through self-alignment local training, clients could upload generalizable feature extractors and category-wiss prototypes. (2) The server aggregates all local prototypes into global prototype, while in the inference stage, the server informative fuses the features extracted from the local models. is best viewed in color.
inability handle augmented samples with the same se- manties, i.e., when Agu >> 0. Unfortunately, the current supervised learning paradigm only learn fixed semantics based on labels and original input, lacking generalization to diverse augmented samples. the model can perform well on any augmented samples, i.e., Aga = O, then the intra- model inconsistency will be alleviated. Intuitively, seems that directly adopting data augmentation could resolve this problem. However, constrained by the supervised training paradigm, as long as training with labels, p & O, the intra- model inconsistency will inevitably occur. Besides, existing works have shown that solely adopting data augmentation can lead to the problem of complete collapse (Grill et al., 2020). To this end, we introduce self-alignment learning to learn invariant features and an unbiased classifier that can generalize diverse augmented samples.
augmented samples. With similarity function sim, the self- supervised learning loss, whose objective is minimize the discrepancy with the same semantics and different the representation to all other semantics, can be formulated as follows:
sFF.. Fat)
Lest
log
(4)
Thy.
(NN s(Fa,,
where s(Fa,) ezpí cos (F F F/T) denotes the temperature parameter, x; represents the set of samples that have the same label with x4. (3:) denotes the set of sample indexes that are different from yi, and the cos function cosine similarity.
Category-wise Prototype Learning. While contrastive learning can achieve invariant features, the classifier often displays biased behaviors and is sensitive to data hetero- geneity, resulting in inconsistent predictions. Inspired by the success prototype learning in various heterogeneous data scenarios, the learnable contrastive prototypes replace the original classifier, thereby mitigating the negative im- pact of data heterogeneity the classification (Zhang et al., 2024; Wan et al., 2024a). Our goal is to obtain a set of representative and highly discriminative prototypes. (I) Closely align with the features to retain semantic informa- tion. (2) maintain sutegory-wiss distinctions between each prototype. Specifically, we define each client maintains set
Feature Alignment. To enable the model to learn more generalized features and reduce the impact of biased data, we focus on learning invariant features related to themselves rather than features aligned with labels. We introduce a contrastive learning approach. Specifically, consider the th clients with its local dataset D,m. = {x,, ye Him and the augmented dataset De {A(x;), yi "mm where A denotes the augmentation operator. Through sepe--uper learning, we aim to learn invariant features Fz. f(0m: and F F(zz)) = (0,m A(x;)) that can generalize to diverse
5

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
of prototypes Pm (Pm, (Pm.1.Pm.2:: .... :..:::....pp..h Cm, where C is the number of classes. During local training, the prototypes are updated to minimize the contrastive loss between the features extracted by the local model and the prototypes. The contrastive loss for the mh client can be formulated as:
mitigating the negative impact of data heterogeneity and enhancing the inference capability. Additionally, also note that the features extracted by different models exhibit discrepancies during the fusion process. To address this, we design an attention-based feature fusion mechanism.
Rom
exp(Fr. Si..a.a/ EjE(Crai exp(FF, Pmp.///)
Feature Fusion. Specifically, let Fm be the feature ex- tracted by the w-th client's local model. To fuse the features from different clients, we design mechanism to infor- matively aggregate the features. The features with less information, which are similar to noise should be down- weighted, while the features with more information should be up- ap-weeghted. Thus, we define the rescaling factor a, as:
Cproto
log
Tom i=1
where Fs, is the feature extracted by the local model for sample Ti, Pu,m is the prototype corresponding to the ground truth class yr, and 7 is the temperature parameter.
Overall Objective. Thus, the overall local training objective can be:
am 1-11-o0 (F,FF.
(8)
Loaal = Last + Epeoto
(6)
where sim(.,) is the cosine similarity, and Fu M(no) is a feature extracted by a Gaussian distribution with mean μ and standard deviation o. We use the standard Gaussian distribution A( 1) in our implementation.
Notably, the objective of self-supervised local learning loss consistent with alleviating inter-model inconsistency. And the two losses in it are complementary to each other. For learnable category- wwse prototypes Pu., the prediction of the original sample and augmented sample can be formulated as P.. and FAA,,, Pu. respectively. Using the cosine similarity as the distance metrics, the prediction discrepancy, ie., inter-model inconsistency, can be formulated as:
Next, we aggregate the features from different clients using the weighted average:
M Σ
Om
Frused
Fm
(9)
Ev Cpp
M
Aitta = -cos( FI FI.Pm Pm
Fa
Pm
where M is the number of clients.
cxm0
Inference with Global Prototype. After obtaining the fused feature Frused, we use the similarity between the fused feature and the discriminative global prototype for predic- tion. Specifically, let Pa = x prototype aggregated from the clients' learnable prototypes. The prediction ý can be formulated as:
=
cos(FF.. FA(ro))
Pmi
1 Σm= P.n. be the global
we can observe that the optimization objective of Proto in (5) is to learn a discriminative representation, ensuring that the samples with the same semantics have similar pre- dictions, which aligns with the goal of reducing motra-model inconsistency (first line in Eq)( Simultaneously, the ob- jective of Las in (4) is to enable the local model to learn informative and consistent features (second line in (7),,)), which also help to reduce intra-model inconsistency.
i = arg max cos cossFrused, Pg.)) cE(C]
(10)
where cos(:;)) is the cosine similarity. This approach leverages the fused features and the discriminative power of the global prototype to make accurate predictions. The detail process is presented in Appendix C Algorithm 1.
4.3. Informative Feature Fused Inference
Motivation. The aforementioned theoretical analysis Theo- rem 3.2 and recent studies indicate that inter-model incon- sistency caused by data heterogeneity is inevitable. The root cause of the low performance due to inter-model inconsis- tency lies in the inability to reconstruct a well- well-performing global model through parameter-level aggregation from sig- nificantly different local models. Inspired by the Mixture of Experts (MoE) (Zeng et al., 2024; Zhu et al., 2024b), an architecture that enhances the inference capability by leveraging the outputs of expert models. Instead of aggre- gating model parameters, we fuse the features extracted by inconsistent local models to integrate semantics, thereby
4.4. Discussion
Privacy Security. Transmitting prototypes from clients to the server instead of classifier is a common practice in FL (Tan et al., 2022; Mu et al., 2023) that does not com- promise privacy security, as prototypes are statistical-level information of category that does not contain the privacy of individual samples (Huang et al., 2023b; Wan et al., 2024b). Besides, category-wise prototype learning aims to capture the common abstract features of the same category, thereby reducing the exposure risk for individual samples.
Comparison with Analogous Methods.
6

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Prototypes in FL. Existing prototype-based methods, such as FedProto (Tan al., 2022) and FedTGP (Zhang et al., 2024), rely on multi-round interactions to obtain representative global prototype, which limits their appli- cability in one-shot FL. scenarios. In contrast, FAFI can achieve a semantically aligned global prototype with just one-shot aggregation.
Baselines. We compare with several OFL methods, catego- rized into three types:
Optimization-
one-shot FedAvg (0-
FedAvg) (McMahan et al., 2017; Guha et al., 2019)
[arXiv'19], MA-Echo (Su et al., 2023) [NN'23], and FedFisher (Jhunjhunwala et al., 2024) [AISTATS'24].
Model Merging in LLMs. Some studies consider that the existing aggregation process in OFL is akin to model merging and try to improve the performance by either us- ing weighted-based model merging (Tao et al., 2024; Yang et al., 2024a), subspace-based merging (Zhu et al., 2024a; Yadav et al., 2024), or routing- routing-based merging (Zeng et al., 2024). However, (1) all these methods still suffer from the "garbage in, garbage out' pitfall, as they only focus on server-side merging and overlook the negative effect caused by inconsistent pre-trained models; (2) All these methods lack crivacy-preserving properties, as they require source data or additional information that is highly relevant for post-calibration after merging. In contrast, FAFI does not require the source data or any other auxiliary information.
Distillation-based:: DENSE (Zhang et al., 2022a) [NeurIPS 22], FedDF (Lin et al., 2020) [NeurIPS'20], F- ADI (Yin et al., 2020) [CVPR 20), and F-DAFL (Chen et al., 2019) [ICCV*19].
Selective ensemble barning-based:: directly Ensemble, Co- Bo-BBoosting (Dai et al., 2023) [ICLR'23], IntactOFL (Zeng et al., 2024) [MM'24].
Besides, also consider some iterative methods, i.e., FuseFL (Tang et al., 2024a) [NeurIPS which improves the performance through iteratively sharing intermediate features. To ensure fair comparisons, we neglect some meth- ods that require additional information, such as FedKT (Li et al., 202 lb), FedOV (Diao et al., 2022), and FedGen (Zhu et al., 2021).
Limitations. Our framework exhibits limitations domain shift and multi-task scenarios where static class represen- tations fail to adapt conflicting feature distributions or divergent task objectives. This limitation is shared by other prototype-based methods. (Tan et al., 2022; Huang et al., 2023b; Zhang et al., 2024; Mu et al., 2023). One possible solution for this limitation is to cluster and ensemble feature representations across various domains or tasks, allowing unbiased representations encapsulate multi-domain/task knowledge (Huang et al., 2023b).
5.2. Effectiveness
Table I illustrates the effectiveness of our proposed FAFI compared with popular OFL methods in non-IID settings. (1) clearly depicts that our method achieves significant performance improvement over the baselines on all datasets and all settings (10.86% averaged). (2) Notably, in some extreme cases such as a = {0.05, 0.1} on Tiny-Imagenet our method exhibits a more significant performance advan- tage (17.71% % averaged) over the baseline algorithms. (3) Besides, the FuseFL which iteratively shares intermediate features between clients can achieve the second-best perfor- mance in some settings. We attribute this the fact that the FuseFL mitigates the feature inconsistencies by shar- ing intermediate features, leaving intra-model inconsistency unsolved. In summary, the FAFI is effective in various data heterogeneous scenarios and achieves competitive performance over all baselines.
5. Experiments
5.1. Experimental Setup
Datasets. Adhere to the previous work (Tan et al., 2022; Huang et al., 2023b; Zeng et al., 2024; Zhang et al., 2022a; Tang et al., 2024a), we evaluate the efficacy on three widely used benchmarks:
CIFAR-10 contsins 50k, 10k images for training and testing. Images are in size 32x32 with 10 classes.
CIFAR-100 have the same format and size as CIFAR-10, but with 100 classes.
5.3. Scalability
Tiny-Imagenet contains 100k, 10k images for training and testing. Images are in size 64 with 200 classes.
We assess the scalability of FAFI by varying the number of clients m. As presented in Table 2, FAFI consistently achieves the best performance across different client scales. As suggested in (Lian et al., 2017), the server can become major bottleneck while the number of clients increases. We also reach a similar conclusion, with more clients m, the performance decreases, which is consistent with (Zhang et al., 2022a; Dai et al., 2023; Lian et al., 2017; Zeng et al., 2024). In summary, the FAFI is scalable across diverse distributed networks of varying sizes.
Data Heterogeneity. Considering the heterogeneous envi- ronment, we partitioned the dataset through a widely-used non-IID partition method, namely Dirichlet Sampling, in which the coefficient refers to the non-IID degree. A small a represents a biased distribution. Following the set- ting in (Zeng et al., 2024; Zhang et al., 2022a; Zeng et al., 2022), we set a E (0.05, 0.1, 0.5}. respectively. More details are shown in Appendix E.
7

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Table Comparison with the state-of-the-art OFL methods: in CIFAR-IO. CIFAR-I00, and Tiny- ImageNe scenarios with skew
ratio a E (0.05, 0.1, 0.3, 0.5). Underline/bold fonts highlight the best baseline/the proposed FAFI A represents the performance
improvement compared with the best baseline. We report the trials' results in the form t See details in $ 5.2.
CIFAR-10
CIFAR-10 100
Tiny ImageNet
Methods
0.05
0.1
0.3
0.5
= 0.05
0.1
0.3
0.5
0.05
0.1
= 0.3
0.3
MA-Echo
36.77=0..1
51.23+00.2
28
60.1440.21
64.21
19.54+0.45
29.11
37.77+0.24
41.94-0211
15.46+0.06
22.23
23.46=0.10 28.21+0.42
O-FedAvg
12.13 17.4320.51
28.07.0.89
35.4220.77
4.7720.21
6.4520.71
10.67.0.31
12.13_0.05
5.6720.15
2.31
13.61.0.10
13.7120:16
FedFisher
40.03+1.11 47.01+1.81
49.33+1.52
50.34+1.32
16.5642.07
18.98+2.09
27.24+1.92
31.4-1.87
15.65+1.54
17.89+1.48
19.54-1.51
20.77+1.15
FedDF
35.53 41.58+a0.s0 so
44.78+0.80
54,5840.75
15.0740.74
27.17+0..56
31.23+0.79
35.39-0.47
11.45+0.40
16.32+0.53
17.79-0.57
27.55+x0.0
F-ADI
35.93 48.35
52.6611.44
58.7821.67
14.65
28.13
33.18
39.44_1.11
13.9241.00
19.00
26.01
29:98 .4
F-DAFL
38.32+1.40 16.34=1.11
54.03
59.0942.23
16.3140.33
2.
34.89
37.88-1.44
15.12+1.34
19.01
23.78=1.28
27.98=1.10
DENSE
38.3721.0n 50.26..0.24
59.76.0.1s
62.1922.12
18.3742.13
32.03
37.33
38.84_0.30
18.77
22.25
28.14g0.34
32.34
Fnsemble
41.3620.67 45.433.0.3e
62.
18.033
61.61
20.4610.02
26.2310.55
38.01
20.67
41.61_0.11
13.2810.87
15.3820.23
17.53g0.31
28.50
Co- Boosting
39.20+0.81
58.49+1.24
67.21+1.76
70.24+2.34
20.1941.44
27.59+1.96
19.30+1.30
42.67-1.44
19.00+1.46
21.90+1.20
29.24-1.52
30.78+2.01
FuseFL
54.4220.41
73.79 a4
84
29.12 84.34
as
36.86
45.12
20.30
21.15
29.28
33.04
34.34
IntectFFL
48.2220.43
61.1320.68 63
70.21
29.9320.23
27.994..07
39.15 46
20.66
26.78
20.45 34
28.4320.17
20.15
35.09
Ours
71.84 77,83 1.32
84.76 as
88.74.0.11
11.02
4..48
56.65
20.91
61.07
36.96
43.62.0.77
53.32=0.s0
56.48
A
17.42
1.004
0.18
1.40
1.90
6.33
11.53
11.77
14.81
14.34
1 20.28
21.39
70-
* Cuess
Caes
FuselL
Ous
Fraeft InnecOFF.
50-
60-
FuueEL
httettop
Contboottngg
Intoor
20
50-
MA-Echo Eusen dte
Dussmile
40-
Eao
Coettonting
30
20-
Juthis WOSRBOL
Fedrica
Fathis RcSePOEn
Fudivo
Fushis
10
Fedlbr
30
1000
2000
1000
1000
2000
3000
3003
2000
1000
Communssttoo
Communicciion Cost
Communicaiao Coss
(a) (0.00)
(b) (01(
(c) Dir(0.5)
Figure 4. Test Accuracy v.s. Communication Cost on CIFAR-10. FAFI more efficient than all other baselines, while achieving higher performance without large communication overhead. See details in 5.4.
Table 2. Scalability under different number of elients, m = (5, 10, 25, 50, 100} on CIFAR-IO with 0.5. See in 5.3.
Table 3. Ablation Study on Key Components of FAFI on CIFAR-
10 with skew ratio (0.05, 0.1, 0.5). See details in $ 5.S.
Client scales m
SALT
IFFI
0.05
= 0.1
0.3
o = 0.5
Methods
5
10
25
50
100
12.13
17.43
28.07
35.42
MA. Echo
64.21
52.64
48.36
45.35
38.54
53.12
55.76
58.95
61.67
O-FodAve
35.42
32.09
28.03
28.24
27.14
55,23
63.75
68.49
70.44
FedFisher
50.34
45.67
34.66
29.09
28.89
71.84
77.83
84.76
88.74
FedDF
54.58
48.88
35.44
29.91
25.66
F-ADI
59.34
46.33
31.83
27.66
24.89
F-DAFL
58.59
45.45
32.88
29.98
28.91
incurring a lower communication overhead. Notably, with more communication budget, multi-round FL methods can achieve better performance (Figure 4 only presents the first 80 rounds' results of multi-round FL methods). However, as these methods get convergent, the performance improve- ment diminishes, and communication costs become pro- hibitive. In summary, FAFI achieves a better efficiency compared with existing FL methods (both OFL and multi-round FL), making it more suitable for practical deployment.
DENSE
62.19
54.67
48.67
43.34
Ensemble
61.61
60.44
58.44
52.51
45.72
Co Bonding
55.34
51.11
49.32
44.56
42.45
FuseFI.
84.34
78.28
62. 12
42.18
37.11
Intac1OFL
74.93
69.11
64.32
59.45
53.21
Ours
88.74
86.96
85.25
81.32
75.37
5.4. Efficiency
We compare the accuracy and communication cost of FAFI to existing OFL methods and multi-round FL baselines in Figure 4. We select six representative multi-round FL meth- ods, more details are shown in Appendix E. Note that FAFI achieves higher performance than all other baselines, while
5.5. Ablation Study
Key Components. For thoroughly analyzing the efficacy of each module, we perform an ablation study to investigate the
8

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Table Ablation Study on Lest and Eproto of SALT on CIFAR-
Acknowledgements
10 with (ir(0.1). The default loss function is Lee-
Exst
Lproto
Types
CIFAR-10 10
CIFAR 100
Tiny ImugeNNet
This work is supported by the National Natural Science Foundation of China (Nos. 62172155, 62472434, 62102425, and 623B2080), the National Key Research and Develop- ment Program of China (2022YFF1203001) the Sei-Tech Innovation 2030 Agemdda22233220055666000 and the Science and Technology Innovation Program of Hunan Province (Nos. 2022RC3061 and 2023RC3027).
Classifier
17.34
6.45
8.31
Classifier
22.34
12.45
10.44
Protocypes
18.23
7.12
8.92
Prototypes
50.12
22.34
Prototypes
52.34
Prototypes
77.83
100
0:05.0
ar 0.3
100
0.05
e-0.3
Impact Statement
0.1
a= 0.1
a-0.5
90
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
80
80
70
Batchatice
(a) Contrastive parameter
(b) Batch size
References
Figure 5. Analysis on hyper-parametee Performance with hyper- parameter and batch size four data heterogeneity o E (0.05, 0.3, 0,3, 0.5}. See details in $ 5.5.
Blanchard, P., El Mhamdi, E. M., Guerraoui, R., and Stainer, J. Machine learning with adversaries: Byzantine tolerant gradient descent. In Proc. of the Advances in Neural Information Processing Systems (NeurlPS), pp. 118-128. 2017.
effectiveness of Self-alignment Local Training (SALT) and Informative Feature Fused Inference (IFFI). The results in Table 3 show that the two modules contribute significantly the performance improvement of FAFI. The combination of the two modules achieves the best performance, under- scoring the effectiveness of our proposed method.
Cao, C.,, Zhou, F., Dai, Y., Wang, J., and Zhang, K. A survey of mix-based data augmentation: Taxonomy, methods, ap- plications, and explainability. ACM Computing Surveys, 57(2):1-38, 2024.
Key Loss Functions in SALT. We investigate two impor- tant loss functions (Cest and Cproto in SALT. The results in Table 4 present the effectiveness of each loss function. We also find that compared with traditional classifiers, lever- aging prototypes helps mitigate model inconsistencies by providing feature-level alignment anchors.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 9912-9924, 2020.
hyper-parametee We first investigate the т and batch size in self-alignment local learning. For т. we note that its impact on performance when 7 E [0.1, When 5 = 0.01, it would have a certain degree of degradation. For batch size, we note that larger batch size would benefit the performance while requiring more resources. With the minimum batch size, the performance is still competitive.
Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., and Tian, Q. Data-free learning of stu- dent networks. In Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3514-3522, 2019.
Chen, J., Zhu, J., and Zheng, Q. Towards fast and stable federated learning: Confrroning heterogeneity via knowl- edge anchor. In Proc. of ACM International Conference on Multimedia (ACM MM), pp. 8697-8706, 2023.
6. Conclusion
In this paper, we propose novel OFL framework, namely FAFI, to mitigate the model inconsistency. We first empir- ically and theoretically present the inconsistencies at both intra-model and inter-model levels in existing OFL meth- ods. To this end, we propose Self-alignment Local Training for semantically aligned feature extractors and distinctive prototypes, and use Informative Feature Fused Inference to fuse the features for inference. Extensive experiments on three datasets show the superiority of our methods.
Chongjian, G., Wang, J., Tong, Z.,, Chen, S., Song, Y., and Luo, Soft neighbors are positive supporters in contrastive visual representation learning. In Proc. of the International Conference Learning Representa- tions (ICLR), pp. 1-16, 2023.
Dai, R., Zhang, Y., Li, A., Liu, T., Yang, X., and Han, B. Enhancing one-shot federated learning through data and

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
ensemble co-boosting. In Proc. of International Con- ference on Learning Representations (ICLR), pp- 1-21, 2023.
Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 2024.
Jhunjhunwala, D., Wang, S., and Joshi, G. Fedfisher: Lever- aging fisher information for one-shot federated learning. In Proc. the International Conference on Artificial Intel- ligence and Statistics (AISTATS), pp. 1612-1620. PMLR, 2024.
Dennis, D. K., T., and Smith, V. Heterogeneity for the win: One-shot federated clustering. In Proc. of Inter- national Conference on Machine Learning (ICML), pp- 2611-2620. PMLR. 2021.
Diao, Y., Li, Q., and He, B. Towards addressing label skews in one-shot federated learning. In Proc. of International Conference on Learning Representations (ICLR), 2022.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proc. of Interna- tional Conference on Machine Learning (ICML), pp. 5132-5143, PMLR, 2020.
Elbatel, M., Wang, H., Chen, J., Wang, H., and Li, X. Learning unlabeled clients divergence for federated semi- supervised learning via anchor model aggregation. Trans- actions on Machine Learning Research, 2024.
Khosla, P., Teterwak, P., Wang, C.,, Sama, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), pp. 18661- 18673, 2020.
Gao, L.,, Fu, H., Li, L., Chen, Y., Xu, M., and Xu, C.-Z. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), pp. 10112-100 10121,, 2022.
Li D. and Wang. J. Fednd: Heterogenous feder- ated learning via model distillation. arXiv preprint arXiv: 1910:09900003811. 2019.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. Buchatskaya, E., Doersch, C., Avila Pires, B., Guo,
Li, J., Zhou, P., Xiong, C., and Hoi, S. C. Prototypical con- trastive learning of unsupervised representations (2020). Proc. of the International Conference on Learning Representations (ICLR), pp. 4-8, 2021a.
Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. In Proc. of Advances in Neural information Processing Sys- tems (NeurIPS), volume 33, pp. 21271-21284 2020.
Li, Q. He, B., and Song, D. Practical one-shot federated learning for cross-silo setting. In Proc. of the Interna- tional Joint Conference on Artificial Intelligence (IJCAI), 2021b.
Guha, N., Talwalkar, A., and Smith. V. One-shot federated 2019.
learning. arXiv preprint
Heinbaugh, C. E., Luz-Ricca, E., and Shao, H. Data-free one-shot federated learning under very high statistical heterogeneity. In Proc. of International Conference on Learning Representations (ICLR), pp. 1-17, 2022.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proc. of Machine learning and systems (ML- Sys), 2:429 450, 2020.
Huang, W., Ye, M., and Du, B. Learn from others and be yourself in heterogeneous federated learning. In Proc. of the Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2022.
Li, Z.., Shang, X., He, R., Lin, T., and Wu, C. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5319-531151119 5329,, 2023.
Huang, W. Ye, M., Shi, Z., and Du, B. Generalizable heterogeneous federated cross-correlen correlation and instance similarity learning. IEEE Transactions on Pattern Analy- sis and Machine Intelligence (TPAMI), 2023a.
Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. Can decentralized algorithms outperform central- ized algorithms? a case study for decentralized parallel stochastic gradient descent. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Huang, W., Ye, M., Shi, Z., Li, H., and Du, B. Rethink- ing federated learning with domain shift: A prototype view. In Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16312-16322. IEEE, 2023b.
Lin, T., Kong, L., Stich, S. U., and Jaggi, M. Ensemble dis- tillation for robust model fusion in federated learning. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp.
Huang, W., Ye, M., Shi, Z., Wan, G., Li, H., Du, B., and Yang, Q. A federated learning for generalization, ro- bustness, fairness: survey and benchmark. IEEE
2020.
10

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Liu, X., Liu, L., Ye, F,, Shen, Y., Li, X., Jiang, L.,, and Li, J. Fedlpa: One-shot federated learning with layer-wise posterior aggregation. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), pp. 1-39, 2024.
fusion. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), pp. 1-37, 2024a.
Tang, Z., Zhang, Y., Shi, S., Tian, X., Liu, T., Han, B., and Chu, X. Fedimpro: Measuring and improving client update in federated learning. In Proc. of the International Conference on Learning Representations (ICLR), 2024b.
Liu, X., Tang, Z., Li, X., Song, Y., Ji. S., Liu, Z., Han, B., Jiang, L.,, and Li, J. One-shot federated learning methods: A practical guide. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI). 2025.
Tao, Z., Mason, I., Kulkarni, S., and Boix, X. Task arith- metic through the lens of one-shot federated learning. arXiv preprint 1rXiv:2411.1860 2024.
Luo, M., Chen, F., Hu, D., Zhang, Y., Liang, J., and Feng. J. No fear of heterogeneity: Classifier calibration for feder- ated learning with non-iid data. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), vol- ume 34, pp. 5972-5984, 2021.
Wan, G., Huang, W., and Ye, M. Federated graph learning under domain shift with generalizable prototypes. In Proc. of the AAAI Conference on Artificial Intelligence (AAAI), 2024a.
volume 38, pp.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. Communication-effficee learning of deep networks from decentralized data. In Proc. of Interna- tional Conference on Artificial Intelligence and Statis- fics (AISTATS), pp. 1273-1282. PMLR, 2017.
Wan, G., Huang, W., and Ye, M. Federated graph learning under domain shift with generalizable prototypes. In Proc. of the AAAI Conference on Artificial Intelligence (AAAT), volume 38, pp. 15429-15437, 2024b.
Wan, G., Shi, Z.,, Huang, W., Zhang, G., Tao, D., and Ye, M. Energy-based backdoor defense against federated graph learning. In Proc. of the International Conference Learning Representations (ICIR), 2025.
Mu, X., Shen, Y., Cheng, K., Geng, X., Fu, J., Zhang, T., and Zhang, Z. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Generation Computer Systems, 143:93-10 104, 2023.
Wang. N., Feng, W., Duan, M., Liu, F., Ng. S.-K., et al.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding, arXiv preprint mrXiv:1807.03744 2018.
Data-free diversity-basees based ensemble selection for one-shot federated learning. Transactions on Machine Learning Research, 2023.
Song, R., Liu, D., Chen, D. Z., Festag, A., Trinitis, C., Schulz, M., and Knoll, A. Federated learning via decen- tralized dataset distillation in resource-constrained edge environments. In Proc. of the International Joint Con- ference on Neural Networks (IJCNN), pp. 1-10. IEEE, 2023.
Wu, X., Yao, X., and Wang, C.-L. Fedser: Structure-based communication reduction for federated learning. IEEE Transactions on Parallel and Distributed Systems (TPDS), 32(7):1565-1577 2020.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merging models. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2024.
Su. S., Li. B., and Xue, X. One-shot federated learning without server-side training. Neural Networks, 164:203- 215, 2023.
Yang, E., Shen, L., Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408. 07666, 2024a.
Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., and Zhang, C. Fedproto: Federated prototype learning across heterogeneous clients. In Proc. of the AAAI Conference on Artificial Intelligence (AAAI), volume 36, pp. 8432- 2022.
8440, 2022.
Yang, M., Su, S., Li, B., and Xue, X. One-shot federated learning with classifier-guided diffusion models. arXiv preprint S.tiv:2311.088777 2023.
Tang, Z., Zhang, Y., Shi, S., He, X., Han, B., and Chu, X. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In Proc. of International Conference on Machine Learning (ICML), pp. 21111-21132. PMLR, 2022.
Yang, M., Su, S., Li, B., and Xue, X. Feddeo: Description- enhanced one-shot federated learning with diffusion mod- els. In Proc. of the ACM International Conference on Multimedia (ACM MM), pp. 6666-6675, 2024b.
Tang, Z., Zhang, Y., Dong, P., ming Cheung, Y., Zhou, A. C.,, Han, B., and Chu, X. Fusefl: One-shot federated learn- ing through the lens of causality with progressive model
Yin, D., Chen, Y., Kannan, R., and Bartlett, P. Byzantine- robust distributed learning: Towards optimal statistical

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
rates. In Proc. of International Conference on Machine Learning (ICML), pp- 3pp.5655-35 5650-5659. PMLR, 2018.
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Learning deep features for discriminative localization. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2921-2929, 2016.
Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha, N. K., and Kautz, J. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proc. of Computer Vision and Puttern Recognition (CVPR), pp- 8715-8722 2020.
Zhou, T., Zhang, J., and Tsang, D. H. K. Fedfa: Feder- ated learning with feature anchors to align features and classifiers for heterogeneous data. IEEE Transactions on Mobile Computing (TMC), (3(6)6673-66744 2024.
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang. N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In Proc. of Inter- national Conference on Machine Learning (ICML), pp. 7252-7261, 2019.
Zhou, Y., Pu, G., Ma, X., Li, X., and Wu, D. Distilled one- shot federated learning. arXiv preprint arXiv:2009.07990 2020.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In Proc. of the International Conference on Learning Representa- tions (ICLR), 2024a.
Zeng, H., Zhou, T., Guo, Y., Cai, Z., and Liu, F. Fed- cav: contribution-aware model aggregation on distributed heterogeneous data in federated learning. In Proc. of International Conference on Parallel Processing (ICPP), pp. 1-10, 2021.
Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y. Llama-moe: Building mixture-of-experts from Ilama with continual pre-training. In Proc. of the Con- ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 15913-15923, 2024b.
Zeng, H., Zhou, T., Wu, X., and Cai, Z. Never too late: Trac- ing and mitigating backdoor attacks in federated learning- In Proc. of the International Symposium on Reliable Dis- tributed Systems (SRDS), pp. 69-81. IEEE, 2022.
Zhu, Z., Hong. J., and Zhou, Data-free knowledge distil- lation for heterogeneous federated learning. In Proc. of International Conference on Machine Learning (ICML), pp. 12878- 12889. PMLR, 2021.
Zeng, H.. Xu, M.. Zhou, T., Wu, X., Kang. J., Cai, Z., and Niyato, D. bee-shoo-but-not-egraddd federated learn- ing. In Proc. of the ACM International Conference Multimedia (ACM MM), pp. 10070-11079, 2024.
Zeng. H., Zhou, T., Guo, Y., Cai, Z., and Liu, Towards value-sensitive and poisoning- proof model aggregation for federated learning on heterogeneous data. Journal of Parallel and Distributed Computing (JPDC), 196:10499) 2025.
Zhang, J., Chen, C., Li, B., Lyu, L., Wu, S., Ding, S., Shen, C., and Wu, C. Dense: Data-free one-shot federated learning. In Pruc. of the Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 21414- 21428, 2022a.
Zhang, Li, Z., Li, B., Xu, J., Wu, S., Ding, S., and Wu, C. Federated learning with label distribution skew via logits calibration. In Proc. of the International Conference on Machine Learning (ICML), volume 162, pp. 26311- 11 26329, 17-23 Jul 2022b.
Zhang, J., Liu, Y., Hua, Y., and Cao, J. Fedtgp: Train- able global prototypes with adaptive-margnn-enaaeed contrastive learning for data and model heterogeneity in federated learning. In Proc. the AAAI Conference on Ar- rificial Intelligence (AAAI), volume 38, pp. 16768-16776, 2024.
12

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
A. Proof of Theorem 3.1
We first provide a lemma to demonstrate the prediction discrepancy of the one-shot local model on the original sample (æ; y) and augmented samples A(x). y as follows.
Lemma A.1. (Prediction discrepancy). For any sample (x, v) and its augmented sample (A(x), v). the prediction discrepancy can be represented as:
Aintr 2 (p Vga VA
(11)
where p p= Dc
12c (z-- Ve), is the prediction of w; activated by softmax function with the augmented samples A(x), Vga the gradient of the local model Wa, and V the gradient of the data augmentation function.
We provide the proof of Lemma A.1 as follows:
Proof. We derive the performance discrepancy of the local model on the original samples (I, y) and augmented samples (A(x),)) y) as follows.
intra = (wi; (x,,)) - L; (w:; (A(z), ())
(12)
We use the Taylor expansion to approximate the loss function on the original samples (ar, y) as follows:
Ci (wii (I, v)) se ", (wii (A(x), V "e::"((w: (wii ()(x).)))² A(x))
(13)
We can further simplify Equ. 12 as follows:
Dintr V. (((x),)))² A(x)).
(14)
If we adopt cross-entropy loss the supervised loss L(2, v) = EE y/ log 2k, where is the softmax prediction y the ground truth label, we have:
OC Oz 0A T
intra
-(x))))
Oz OA Ox
Σ
C
VA) (x- A(x))- 0A
02
T
(15)
E.c ye)
Note that A is the gradient Vg of local model w; with respect the augmented data A(x), and VA exp(0? o(z,)(z)))
04
is the
transformations property of the data augmentation function A. Since be = Σe= oxp( of A(2,)) is the output of a softmax function, 2. E (0, y is the one-shot encoded vector, We E {0,1] 1). thus, 1ze Mel > 0. Finally, we finish the proof:
A inttra (p Vg. VA)" (x A(x)),
(16)
where p = DE Uc), Vga is the gradient of the local model wy, and V A is the gradient of the data augmentation
function A.
Then we provide the proof the Theorem 3.1 with Lemma A.1 follows:
Proof. Based on Lemma A.1, derive the intra- intra-model inconsistency for any local models trained by existing OFL methods.
When Vaa # 0 and [z)(z)) - æ||| > 0, V A & 0, we have:
|| imtra 2| = (p(p Vga VA) (x A(x))l?
(17)
p2 Vgall
2 ||] [|(x A(+))
Since |p| > 0. 12 0, | > 0, and ||x |l O, we can induce that ||trral > 0.
We finish the proof.
Note that in Theorem 3.1, the Vg. = 0 represents the augmented data A(x) is not sensitive to the local model w,, which is a rare case in practice. the V = 0. means the data augmentation function have fully changed the semantics of the original data x. conclusion, we can infer that the mntr--moee inconsistency inevitable in existing OFL methods.
13

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
B. Proof of Theorem 3.2
Proof. We derive the gradient of cross-entropy loss as
A (V(uu V(De
Mu.e (1 Zu,e) Tu.e
Mos (1 Fv. Tn
(18)
Ul
Tbp
1
"1
Zu Buie"
Yuu
The d'E(Culie
e'E(Culie
where the term of x holds from the Property in (Zhang et al., 2022b). ny = Yus we have
C
Ju,o)T - ny Fo.0) [nu,o
nly
(19)
Thu Cu
Thyy Ro Fe N}
c'
C Co
We consider the || under heterogeneous data distribution, and omit the De for simplicity. When Tue # Fr,e, we have
112 n || -|-51 > 0.
|| >
(20)
When Fn,e = Fo c E [Cn] n [Cn), Zu,e = 2n,e, that is, two clients have same prediction and feature on selected positive class c, then c' E [C.] n [C,],, we have
1/2
| 1
||
n
7/2 MEC.C.M....c...
(21)
n
|l
naye ou
Σ
Dyy To ||
If we assume the equation above equals zero, we have Tuur Z..ct Fn,e', which requires a perfect feature extractor for all clients. It is not practical condition in FL according to (Zhou et al., 2024).
When Fu,c = Zv,es c E U Nu,c = nu,e = 0, we have the same formulation with Equ. 21, we can obtain | m 2 >0.
When Tne Te,es e E [Cu]M[[..] [C.]], Ny,e O, we have
1/2 |n,,c(1 2u,c)
| 2
E
of
Σ Zo e/
n
c'
(22)
112 Du,e Bucll
n
where the equality holds if and only if n1 Za,c)Ru,e = A.. In the training phase, it is challenging to maintain this condition between any two clients, so we use the term ss since E,,c' << Zu.c < 1. We say that the | || | is more likely to be positive. The other cases, c E [C [Cu]} can get the same conclusion.
In summary, we conclude that the | || > under different heterogeneous scenarios. We finish the proof.
14

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
C. Algorithm Details
Here, we present the detailed algorithm FAFI. The FAFI consists two parts. For the client side, the clients perform self-alignment local training for semantically aligned feature extractors and category-wise distinctive prototypes. Then the server aggregates the local prototypes into global prototype. By dynamically fusing the feature with more information, FAFI makes predictions by computing the similarity between the global prototypes and the fused feature.
Algorithm FAFI
Input: Number of clients m, number classes C, number of local epochs E, learning rate 7/ batch size B, data heterogeneity a.
Output: Global prototypes Pg. local models wi prediction of test set Ylest
Client Side:
for each client i E [m]] in parallel do Sample local data Dj with heterogeneity a.
Initialize local feature extractor 0, and local prototypes P..
for each local epoch e E [E] do
Sample a mini-batch B, from Dj
Update local feature extractor 0g and local learnable prototypes P. with self-alignment local training loss:
L (0:))()) (P
end for
Send the local feature extractor 0; and local Iearnable prototypes P: to the server.
end for
Server Side:
Global prototypes aggregation P, M Em=1 Pm Informative Feature Fusion
a. - (Fs F. Nlao))
M
Frused
Σ
Fm
M
Ogp
Inference Yenst = arg maX,g(C] cos(Frusal Friseds Pa Pa.c)
D. More Related Work
D.I. Prototype Learning
Prototype refers to the representative feature vector of the instances belonging to specific class. It is popular and effective method widely used in various tasks, such as supervised classification tasks and unsupervised learning. Since the prototypes can provide abstract knowledge while preserving data privacy, few works introduce prototypes into federated learning. FedProto (Tan et al., 2022) and FedProc (Mu al., 2023) aim achieve a feature-wise alignment with global prototypes. approximated Gaussian Mixture Model (GMM). VHL (Tang al., 2022) builds virtual homogeneous dataset for mitigating data heterogeneity. However, existing federated prototype learning methods rely on multi-round interactions between the clients and server, which is not practical in one-shot federated learning.
CCVR (Luo et al., 2021) generates virtual features based
D.2. Contrastive Learning
Contrastive learning (CL.) is a promising direction in self- seff-supervieed learning. Contrastive learning constructs positive and negative pairs for each training instance and designs various loss functions to contrast positiveness against negativeness, such as InfoNCE (Oord et al., 2018). and SupCon (Khosla et al., 2020). One branch of CL focuses on selecting the informative positive pairs and negative pairs (Chongjian et al., 2023). Another branch investigates the semantic structure and involves clustering methods to construct more representative prototypes (Caron et al., 2020; Li et al., 2021a).
15

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Table Impact different local epochs: in CIFAR-I0. CIFAR-100, and Tiny-ImageNe scenarios with skew ratio a
(0.05, 0.1, 0.3, 0.5).
CIFAR-10 10
CIFAR-100
Tiny ImageNet
Local Epoch
0.05 a 0.1 a = 0.3 a = 0.5
a
0.05 a 0.1 a = 0.3 a 0.5 a 0.05
0.1
0.3 a
0.5
42.34
45.88
54.53
63.76
10.04
13.25
18.28
22.31
10.88
13.68
20.74
23.61
10
52.65
58.25
66.03
73.97
12.73
20.17
27.71
32.43
15.92
21.04
29.88
34.81
50
57.79
63.55
72.21
77.96
15.32
25.83
33.82
39.01
20.36
35,96
36.44
41,16
100
62.08
68.12
76.
82.03
24.35
31.34
41.48
46.77
23.34
36.96
40.91
46.48
150
69.67
72.53
81.64
86.78
29.17
43.19
49.09
59.38
31.06
40.42
46.39
52.13
200
71.84
77.83
84.76
88.74
31,02
45.48
56.65
61.07
36.96
43.62
53.32
56.48
300
72.33
77.92
85.12
89.23
31.67
46.65
57.12
62.11
37.04
43.14
53.65
57.23
Table Impact Model Architecture: ResNet-18, 18, ResNet-50,, ViT-base on mny-ImmaeNet.
Architecture
Accuracy(%)
Memory Cost
Computation Cost
ResNet-18
57.23
11.2 M
3.2 min/epoch
ResNet-50
58.12
25.6 M
4.6 min/epoch
ViT-base
58.09
86 M
5.5 min/epoch
E. Experimental Details
Models and Learning Rate. Following the advanced OFL works (Zeng al., 2024), train ResNet-18 18 all datasets. We use the SGD optimizer with momentum coefficient of 0.9, set the batch size to 256, and make local train E' 200 steps. We report the best results by varying the learning rate in 0.05, 0.01, 0.001. We adopt the temperature of contrastive learning as 7 = 0.5, the noise is the standard Gaussian noise (00, 1). We use the output of the last layer of ResNet- 18 as the feature, and the prototype of each category has the same dimension as the feature. Source code and models are available https: / / thub. com/ zenghut 2377/FFFF 21T///FFTTTMM2222 CMM.25.
Federated Settings. For OFL methods (including FAFI), we adopt the same settings with (Zeng et al., 2024; Zhang et al., 2022a). We set the number of clients 5, and the local training epochs of 200. For multi-round FL methods, we select six representative methods, including FedAvg (McMahan et al., 2017), FedProx (Li et al., 2020), SCAFFOLD (Karimireddy et al., 2020), FedCav (Zeng et al., 2021; 2025), FedProto (Tan et al., 2022), FedDC (Gao et al., 2022). To ensure a fair comparison framework, all multi-round FL methods are evaluated under identical experimental conditions as OFL approaches, including matching client numbers, equivalent heterogeneous data distributions, and consistent local learning rates. We try to adopt the same local training epochs as OFL methods do in multi-round FL.. However, this configuration unexpectedly led performance degradation that persisted levels comparable to O-FedAvg for extended training rounds. To effectively highlight the advantages of FAFI, consequently adopt the conventional local epoch setting (E=5), parameter configuration widely validated in FL literature. Under this optimized configuration, our experimental results demonstrate that conventional multi-round FL methods still require over 80 communication rounds to attain accuracy levels equivalent to those achieved by FAFI
F. More Experimental Results
Original
Flip
Figure 6. Visualization of FAFI with Grad-Cam. The features are consistent with the same semantics.
16

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
Table 7, Applicability to Resource-constrained Scenarios: ResNet-18
MohileNet on CIFAR-I0 with Dir (0.05).
Models
Methods
Batch Size
Memory Cost
CPU
Time
GPU Time
Accuracy
O-FedAvg
512
1942 MB
138
6
10.44
IntactOFL.
512
1942 MB
141
8
32.31
MobileNet
FAFI
32
1086 MB
183
11
55.21
FAFI
256
1838 MB
227 s
14
58.33
FAFI
512
3082 MB
240 s
17
59.64
O-FedAvg
512
4638 MB
197
8
12.13
IntactOFL.
512
4638 MB
204 s
10
48.33
ResNet-18
FAFI
32
2789 MB
307
18
69.73
FAFI
256
8792 MB
319
27
70.24
FAFI
512
14723 MB
428
32
71.84
Table Model Heterogeneity Scenarios: ResNet-18 and MobileNet on CIFAR-10 with Dir(0.05).
Client 0
Client 1
Client
Client 3
Client 4
IntactOFL
Ours
Case #1
LeNet
ResNet-18
VGG
MobileNet
ResNet-50
48.33
71.84
Case #2
VGG
MobileNet
ResNet-50
LeNet
ResNet-18
52.55
72.12
Case #3
ResNet-50
LeNet
ResNet-18
VGG
MobileNet
54.12
72.45
Table 9. Impact of the Feature Fusion Strategy in IFFI: in CIFAR-10 with Dir(0.05).
Feature Fusion Strategy
CIFAR-10
CIFAR-100
Imy-ImmgeNet
Average
72.83
37.48
30.12
(),
71.83
35.4
28.09
N(0, 10)
70.12
33.02
26.34
((0,, 50)
68.23
31.12
24.12
(1,
69.12
32.77
25.12
(O, 1) (Ours)
77.83
45.48
43.62
Impact of Local Epochs. We tested how different numbers of local training rounds affect performance across three datasets. As shown in Table 5, model accuracy keeps improving when increase the number of training rounds. However, the improvements become very small after 200 rounds pushing to 300 rounds only brings less than 0.8% better accuracy compared to 200 rounds, but makes the computation much heavier. After comparing how accuracy improves versus how much computing power is needed, we chose 200 rounds as our standard setting.
Impact of Model Architecture. We used ResNet-18 as the default feature extractor in the previous study. this part, we investigate the impact of different model architectures. We select ResNet-18, ResNet-50, and ViT-base. We compare memory cost (measured by #parameters), computation cost (measured by the computation time per epoch), and accuracy Imag--m in Table 6. Note that the impact of model architecture performance is relatively low, but there significant differences in computational and storage costs. The enhancement in performance is attributed to the designed framework (FAFI), rather than the model.
Applicability to Resource-constrainee Scenarios. We investigate the applicability of FAFI under the resource-constraine scenarios, which present limited computation capability and memory space. Thus, we select a typical light-weight model, i.e., MobileNet, and report the memory cost and computation cost (CPU or GPU time) in Table 7. We use the GPU memory occupied during local training as the metric for memory cost, while computation cost is measured by the time taken per epoch the GPU(RTX 4090) and the CPU(Intel Core i7- 170KK. We believe that FAFI can achieve competitive performance even in resource-conssrained scenarios, requiring only 11 seconds on GPU with less than 2GB of memory or approximately 3 minutes on a CPU.
Model Heterogeneity Scenarios. We test FAFI under model heterogeneity scenarios, where different clients own different model architectures. We list three cases on CIFAR-10 with Dir(0.1) with five different architectures (LeNet, ResNet-1 18, VGG, MobileNet, ResNet-50). As presented by Table we note that FAFI can still achieve better performance under
17

---

Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated Learning
heterogeneous models.
Impact of the Feature Fusion Strategy. We investigate the impact of different feature fusion strategies in IFFI on CIFAR-10 with (ir 1). We notice that the standard Gaussian distribution achieves better performance than other settings. We attribute this to the fact that the distance from standard Gaussian noise better reflecting the informativeness of the extracted features (Elbatel et al., 2024).
Visualization. We visualize the features extracted by the local models trained by FAFI in Figure We select one the local models trained by FAFI and choose some samples for visualization. We found that the features extracted under the same semantics consistent, which proves the effectiveness of the Self-alignment Local Training.
18


Reviews:
[{"review_id": "FBP3TfcQIn", "reviewer": "Reviewer_Chua", "summary": "The paper addresses the critical challenge of model inconsistency in OFL due to heterogeneous data. They identify two key inconsistencies—intra-model and inter-model—and propose a novel framework, FAFI, which combines client-side self-aligned training (SALT) and server-side informative feature fusion (IFFI). Extensive experiments on three classification datasets demonstrate significant performance improvements over 11 baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work. We greatly appreciate your recognition of the proposed method as both technically sound and high-performing. Please find our detailed responses to your concerns below.\n\n**W1 & Q1: Applicability to resource-constrained scenarios.**\n\n**Ans for W1 & Q1:** Thanks for your important comments. We note that **our proposed FAFI can support lightweight models and is applicable to resource-constrained scenarios**. We have complemented statistics on computational and memory cost with MobileNet and ResNet-18 on CIFAR-10.\n\n| Methods            | Memory Cost | Computation Cost (GPU/ CPU)  | Accuracy |\n|-|-|-|-|\n|MobileNet|\n| O-FedAvg | 1942 MB | 6s / 138s | 10.44 |\n| IntactOFL | 1942 MB | 8s / 141s | 32.31 |\n| Ours $b$=32 | 1086 MB | 11s / 183s | 55.21 |\n| Ours $b$=256 | 1838 MB | 14s / 227s | 58.33 |\n| Ours $b$=512 | 3082 MB | 17s / 240s | 59.64 |\n| ResNet-18 |\n| O-FedAvg | 4638 MB | 8s / 197s | 12.13 |\n| IntactOFL | 4638 MB | 10s / 204s | 48.33 |\n| Ours $b$=32 | 2789 MB | 18s/ 307s | 69.73 |\n| Ours $b$=256 | 8792 MB | 27s / 319s | 70.24 |\n| Ours $b$=512 | 14723 MB | 32s / 428s | 71.84 |\n\nWe use the GPU memory occupied during local training as the metric for memory cost, while computation cost is measured by the time taken per epoch on GPU(RTX 4090) and CPU(Intel Core i7-11700K). Here, $b$ represents the batch size. We believe that FAFI can achieve competitive performance even in resource-constrained scenarios, requiring only 11 seconds on a GPU with less than 2GB of memory or approximately 3 minutes on a CPU. Additional details will be included in our revised version.\n\n**W2: Assumption on the number of classes, which may not hold in dynamic or open-set FL scenarios.**\n\n**Ans for W2:** Thanks for your comments. FAFI assumes that the number of classes is known in advance. However, in dynamic or open-set FL scenarios, the number of classes may change over time. We note that FAFI can be easily extended to these scenarios by extracting the invariant features of the new classes and learning a new discriminative prototype for them. We will include this discussion in our revision.\n\n**W3: Some notions are not clear.**\n\n**Ans for W3:** Thanks for your comments. Sorry for the unclear notations. For clarity, 1) the $\\Delta_{intra} = \\vert L(x, y) - L(x', y) \\vert$ is the performance discrepancies between any two samples $x, x'$ with the same label $y$; 2) $A$ refers to the function that can abstract the feature variation among samples. We will clarify these notations in the revision.\n\n**W4 & Q1: Model heterogeneity scenarios.** \n\n**Ans for W4 & Q1:** Thanks for your comments. We note that FAFI can support heterogeneous models. We have conducted new evaluations on heterogeneous models with five different architectures (LeNet, ResNet-18, VGG, MobileNet, ResNet-50) on CIFAR-10 with data heterogeneity $Dir(0.1)$ as follows and added more details in Appendix F. We only report part of them here due to the limited space. \n\n| Client 0 | Client 1 | Client 2 | Client 3 | Client 4 | IntactOFL | Ours |\n| -------- | -------- | ------- | ------- | ------- | --------- | ---- |\n| LeNet | ResNet-18 | VGG | MobileNet | ResNet-50 | 48.33 | 71.84 |\n| VGG | MobileNet | ResNet-50 | LeNet | ResNet-18 | 52.55 | 72.12 |\n| ResNet-50 | LeNet | ResNet-18 | VGG | MobileNet | 54.12 | 72.45 |", "claims_and_evidence": "The claims are clear and convincing.", "methods_and_evaluation_criteria": "FAFI consists of SALT and IFFI for intra- and inter-model inconsistencies, respectively. Evaluations on CIFAR-10/100 and Tiny-ImageNet are appropriate.", "theoretical_claims": "I have checked the theoretical claims and proofs. Theorem 3.1 for intra-model inconsistency, Theorem 3.2 for inter-model inconsistency, and their proofs in the appendix are correct.", "experimental_designs_or_analyses": "I have checked experimental designs or analyses.", "relation_to_broader_scientific_literature": "Unlike approaches that focus on server-side aggregation, this paper emphasizes improving local training strategies to achieve better models and mitigate the ‘garbage in, garbage out’ pitfall.", "essential_references_not_discussed": "References are essential and sufficient for understanding the key contributions.", "other_strengths_and_weaknesses": "Strengths:\n- The proposed method is technically sound. Moreover, leveraging self-supervised methods and prototype learning is novel in OFL scenarios\n- Comprehensive survey on existing OFL methods and good discussion on analogous methods, such as prototype-based FL and model merging approaches.\n- Significant performance improvement and extensive and sufficient evaluations.\nWeaknesses:\n- Computational costs for SALT’s contrastive learning (e.g., batch size=256) seem to be expensive for resource-constrained clients, which could be limited in edge scenarios with restricted computational capabilities. \n- The reliance on class-wise prototypes assumes a fixed and known number of classes, which may not hold in dynamic or open-set FL scenarios.\n- Some notions in theoretical analysis lacks of explanations.\n- Federated learning under heterogeneous model scenarios is a very practical research problem, the model architectures across different clients are different. FAFI only consider the homogeneous model scenarios.", "other_comments_or_suggestions": "No", "questions_for_authors": "- My main concern with this work is its applicability to resource-constrained scenarios, such as edge or mobile environments. In these scenarios, they tend to use more lightweight models~(MobileNet). Can FAFI perform well on some lightweight models? Besides, the diverse computation capabilities across clients restrict the clients from adopting the models with the same architecture, Can FAFI support heterogeneous model architectures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePgMJEWCh1", "reviewer": "Reviewer_WoAM", "summary": "This manuscript tries to solve the 'garbage in garbage out' problem caused by inconsistent models in the one-shot federated learning paradigm. They propose FAFI, a novel OFL framework consisting of two key components: SALT for invariant feature learning and IFFI for server-side feature fusion-based inference. Extensive experiments on CIFAR-10/100 and Tiny-ImageNet demonstrate FAFI’s superiority over 11 baselines, achieving a 10.86% average accuracy improvement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work. We greatly appreciate you find the paper is well-motivated and well-demonstrated. Please find our detailed responses to your concerns below.\n\n**W1 & Q1: Details in IFFI.**\n\n**Ans for W1 & Q1**: Thanks for your important comments. Sorry for missing the explanations about IFFI. We use the Gaussian distribution $\\mathcal{N}(0, 1)$ in Equ.8. For clarity, we will add an ablation study on the hyperparameters. More details can be seen in response to **Reviewer BvHZ W2 & Q2** and will be added in Appendix E in the revision. \n\n\n**W2 & Q2: More settings to verify the effectiveness of FAFI.**\n\n**Ans for W2 & Q2:** Thanks for your important comments. We have conducted new evaluations on CIFAR-10 with data heterogeneity $Dir(0.05)$ and $Dir(0.1)$, and the results are shown below.\n\n| Methods |  Acc. $Dir(0.05)$  | Acc. $Dir(0.1)$ | Comm. Cost |\n| ------------- | - | - | - | \n| MA-Echo | 36.77 | 51.23 | 44.7 MB  |\n| O-FedAvg | 12.13 | 17.43 | 44.7 MB  |\n| FedFisher | 40.03 | 47.01 | 48.2 MB |\n| FedDF | 35.53 | 41.58 | 44.7 MB |\n| F-ADI | 35.93 | 48.35 | 44.7 MB |\n| F-DAFL | 38.32 | 46.34 | 44.7 MB |\n| DENSE | 38.37 | 50.26 | 44.7 MB |\n| Ensemble | 41.36 | 45.43 | 44.7 MB |\n| Co-Boosting | 39.20 | 58.49 | 44.8 MB |\n| FuseFL | 54.42 |  73.79 | 53.32 MB |\n| IntactOFL | 48.22 | 61.13 | 44.7 MB |\n| FedAvg $C=50$ | 23.45 | 27.44 | 2.12 GB |\n| FedProx $C=1$ | 13.53 | 17.58 | 44.7 MB |\n| FedProx $C=50$ | 23.32 | 26.76 | 2.12 GB |\n| SCAFFOLD $C=1$ | 12.45 | 16.23 | 89.4 MB |\n| SCAFFOLD $C=50$ | 27.22 | 30.45 | 4.36 GB |\n| FedCav $C=1$ | 12.49 | 16.77 | 44.8 MB |\n| FedCav $C=50$ | 26.45 | 30.23 | 2.12 GB |\n| FedProto $C=1$ | 12.11 | 16.23 | 44.7 MB |\n| FedProto $C=50$ | 28.31 | 32.55 | 2.12 GB |\n| FedDC $C=1$ | 11.32 | 15.23 | 44.7 MB |\n| FedDC $C=50$ | 30.23 | 44.23 | 2.12 GB |\n| Ours | 71.84 | 77.83 | 44.7 MB |\n\n$C$ is the communication rounds. **We note that FAFI can achieve competitive performance even in extreme heterogeneous scenarios (i.e., $Dir(0.05)$)**, requiring only 44.7 MB of communication cost. Additional details will be included in Appendix F in our revised version.\n\n**C1 & C2: Typos and open source code**\n\n**Ans for C1 & C2**: Thanks for your comments. We will fix the typos in Appendix A, and we promise we will open the source code.", "claims_and_evidence": "The claims made in this manuscript are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are suitable for the problem.", "theoretical_claims": "The theoretical claims in this paper are correct. However, some typos in appendix, line 653 $Vert(x-A(x))||^2>0$ should be $|| x - A(x) ||^2>0$ ?", "experimental_designs_or_analyses": "The experimental parts are well-organized, the demonstrations on main paper and appendix are good. Better presenting more results on efficiency.", "relation_to_broader_scientific_literature": "The paper situates its contributions within the broader federated learning (FL) literature by empirically and theoretically analyzing the limitations of existing one-shot FL (OFL) methods. The paper explicitly differentiates FAFI from related approaches in Sec.4.4, including prototype-based methods and model merging techniques.", "essential_references_not_discussed": "The paper has cited the related works that are essential to understanding the key contribution.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-demonstrated and well-organized. The figures, such as Figure 1(a), Figure 2(a), and Figure 3 are good.\n2. The paper is well-motivated. The analysis of intra- and inter-model inconsistencies is supported by both empirical evidence (Grad-CAM visualizations) and theoretical proofs (Theorems 3.1 and 3.2).\n3. The experimental part is well-organized. Evaluations across diverse non-IID settings and clients scales over 11 baselines validate FAFI's effectiveness, scalability, and efficiency.\n4. The method significantly reduces communication overhead while outperforming multi-round FL baselines in efficiency-accuracy trade-offs (Figure 4). This aligns well with real-world FL deployment needs.\n\nWeaknesses:\n1. Details on the noise in IFFI (eq. 8) are unclear. It is better to provide code and hyperparameter settings.\n2. The efficiency on extreme non-IID settings, such as $\\alpha=0.05$. Figure 4 only provides the evaluations on $\\alpha=0.5$. Please provide more settings to verify its efficiency.", "other_comments_or_suggestions": "1. Some typos in appendix, e.g., line 653 $Vert(x-A(x))||^2>0$ should be $|| x - A(x) ||^2>0$.\n2. It is recommended that open-source code be provided to replicate the experimental results.", "questions_for_authors": "1. Please provide the details about the noise in IFFI.\n2. Please provide more settings to verify the efficiency of FAFI.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHa9CGXUec", "reviewer": "Reviewer_BvHZ", "summary": "Existing OFL methods focus on server-side aggregation, which falls into ‘garbage in garbage out’ pitfall. They unravel the root cause of such garbage inputs as intra-model and inter-model inconsistencies in the face of data heterogeneity. To address these, they design self-alignment local training and informative feature fused inference. Experimental results verify the effectiveness, scalability, and efficiency of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work. We greatly appreciate you find our method is efficient and data-free. Please find our detailed responses to your concerns below.\n\n\n**W1 & Q1: Ablation study on $L_{ssl}$ and $L_{proto}$.**\n\n**Ans for W1 & Q1:** Thanks for your important comments. As suggested, we have conducted new ablation studies on $L_{ssl}$ and $L_{proto}$, and the results are shown below. \n\n\n| Feature Extractor | Prototypes / Classifier | CIFAR-10 | CIFAR-100 | Tiny-ImageNet |\n| ----------------- | ---------------------- | -------- | ---------- | ------------- |\n| $L_{ce}$ | Classifier + $L_{ce}$ |  17.34 | 6.45 | 8.31 |\n| $L_{ce}$ | Prototypes + $L_{ce}$ |  18.23 | 7.12 | 8.92 |\n| $L_{ce}$ | Prototypes + $L_{proto}$ |  52.34 | 33.34 | 23.12 |\n| $L_{ssl}$ | Classifier + $L_{ce}$ |  22.34 | 12.45 | 10.44 |\n| $L_{ssl}$ | Prototypes + $L_{ce}$ |  50.12 | 31.89 | 22.34 |\n| $L_{ssl}$ | Prototypes + $L_{proto}$ |  77.83 | 45.48 | 43.62 |\n\nWe note that the performance of FAFI is significantly improved by using $L_{ssl}$ and $L_{proto}$ for enhanced feature representation and discriminative prototypes, and the combination of $L_{ssl}$ and $L_{proto}$ achieves the best. Additional details will be added in Appendix F in our revision.\n\n\n**W2 & Q2: Impact of the feature fusion strategy.**\n\n**Ans for W2 & Q2:** Thanks for your important comments. As suggested, we have conducted new evaluations on the impact of the feature fusion strategy with $Dir(0.1)$, and the results are shown below. \n\n| Feature Fusion Strategy | CIFAR-10 | CIFAR-100 | Tiny-ImageNet |\n| -------- | -------- | -------- | -------- | \n| Average | 72.83 | 37.48 | 30.12 |\n| $\\mathcal{N}(0,5)$ | 71.83 | 35.40 | 28.09 |\n| $\\mathcal{N}(0,10)$ | 70.12 | 33.02 | 26.34 |\n| $\\mathcal{N}(0,50)$ | 68.23 | 31.12 | 24.12 |\n| $\\mathcal{N}(1,1)$ | 69.12 | 32.77 | 25.12 |\n| $\\mathcal{N}(0,1)$ (Ours) | 77.83 | 45.48 | 43.62 |\n\nWe note that $\\mathcal{N}(0,1)$ facilitates the best performance in the tested cases, as it effectively captures feature information compared to other strategies. Additional details and discussions about this hyperparameter will be included in Appendix F in the revision.", "claims_and_evidence": "Clear and convincing claims.", "methods_and_evaluation_criteria": "The method is suitable for the proposed model inconsistency problem. The evaluation criteria, accuracy on three classification datasets, is suitable.", "theoretical_claims": "Correct theoretical claims. The manuscript contains two theorems, Theorem 3.1 for intra-model inconsistency and Theorem 3.2 for inter-model inconsistency.", "experimental_designs_or_analyses": "I have assessed the soundness and validity of the experimental designs, and the authors have effectively validated the proposed method’s effectiveness.", "relation_to_broader_scientific_literature": "They provide a novel solution for mitigating model inconsistency. Existing OFL methods all focus on server-side aggregation based on inferior local models. This paper proposes to provide a high-quality local model through self-alignment local training, and utilizing the extracted feature for aggregation.", "essential_references_not_discussed": "Almost all essential references have been included and discussed.", "other_strengths_and_weaknesses": "1. Strengths:\n1) This manuscript proposes FAFI which achieves a good performance while requiring only one communication round.\n2) The paper is the first to systematically identify the \"garbage in, garbage out\" pitfall in OFL caused by intra- and inter-model inconsistencies. The visualization and theoretical analysis on the model inconsistencies are interesting. \n3) The authors utilize self-supervised learning methods and prototype learning for better local models and propose a feature fusion-based inference method. The feature fusion is novel in OFL.\n4) FAFI is data-free, which does not require the source data or any other auxiliary information.\n2. Weaknesses:\n1) SALT consists of two parts, feature alignment, and category-wise prototype learning, and how these two parts contribute to mitigating the intra-model inconsistency. Please provide the ablation study on L_ssl and L_proto.\n2) Noise impact in IFFI. How does the noise representation used for attention weighting in feature fusion (§4.3) impact the final performance. Please provide more evaluations to demonstrate the impact of the noise.", "other_comments_or_suggestions": "1) Inconsistent descriptions of existing OFL methods categories (Sec. 2 and Sec. 5)", "questions_for_authors": "1. It is unclear how L_ssl and L_proto contribute to mitigating the intra-model inconsistency\n2. How does the weights in feature fusion impact the performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Oj7SzlVdm", "reviewer": "Reviewer_8kHb", "summary": "The paper investigates one-shot Federated Learning (OFL), which aims to reduce the communication costs associated with traditional multi-round Federated Learning. The authors highlight that existing OFL methods face significant challenges due to \"garbage in, garbage out\" issues, where inconsistent local models lead to a degraded global model. The proposed solution, denoted as FAFI, introduces Self-Alignment Local Training (SALT) and Informative Feature Fused Inference (IFFI) to create consistent features in one-shot local models and enhance global model inference. Through extensive experiments, FAFI outperforms existing OFL methods by a significant margin.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time in reviewing. We appreciate that you consider our presentation is clear, and our method is effective and robust. Please see our detailed feedback for your concerns below.\n\n**W1: Potential solutions and complexities of implementation.**\n\n**Ans for W1:** Thank you for your valuable comments. Indeed, FAFI may have limitations in handling domain shifts/multi-task scenarios. FAFI assumes that invariant feature representations should remain consistent or similar, enabling them to mitigate model inconsistencies. However, in domain shift/multi-task scenarios, feature representations often exhibit significant discrepancies across different domains or tasks, particularly in the isolated training paradigm.\n\nA potential solution is to cluster and ensemble feature representations across various domains or tasks, allowing unbiased representations to encapsulate multi-domain/task knowledge.\n\nRegarding implementation in real-world FL environments, as detailed in Algorithm.1(Appendix C), FAFI requires only two modifications compared to existing OFL methods, making it easily adaptable for both clients and the server:  \n1) On the client side, we train the feature extractor in a contrastive manner and use the prototypes instead of the classifier.\n2) On the server side, fusion is performed at the feature level rather than the parameter or prediction level.  \n\nWe will highlight these in our revised version.\n\n**W2: Comparisons with other methods**\n\n**Ans for W2:** Thanks for your important comments. We have conducted new experiments comparing 3 methods, i.e., FedCompress, FedKD, and FedACG, which reduce the communication cost by gradient compression, gradient fractorization, and transferring gradient momentum. These new results on CIFAR-10 are provided below and added to our revision. Results show that FAFI achieves better efficiency compared with all baselines.\n\n|Methods|Acc.|Comm. Cost|\n|-|-|-|\n|FedCompress(C=1)|15.45|9.86MB|\n|FedKD(C=1)|20.67|34.89MB|\n|FedACG(C=1)|19.77|44.7MB|\n|FedCompress(C=50)|33.34|0.48GB|\n|FedKD(C=50)|60.12|1.70GB|\n|FedACG(C=50)|68.23|2.18GB|\n|Ours|77.83|44.7MB|\n\n**W3: Privacy issue**\n\n**Ans for W3:** Thanks for your very valuable comments. We agree with the statement *there is no free lunch for the privacy-utility trade-off*[1], for that the tenet of FL is to seek balance among efficiency, effectiveness, and privacy. FAFI, as well as most OFL methods, all aim to ensure efficiency for affordable deployment while striving to find the effectiveness boundaries as much as possible and maintain privacy below an acceptable level. As to FAFI, the possible leakage is the prototypes, which have been proven in previous works to compromise no privacy. For stricter privacy requirements, one potential solution is implementing differential privacy or using noised samples with the cost of some effectiveness degradation.\n\n**W4 & Q1: Comparisons with other methods**\n\n**Ans for W4 & Q1:** Thanks for your valuable comments. We present the difference between our proposed FAFI with feature-enhanced and prototype-based methods as follows:\n\nFor feature-enhanced works, we note that FAFI is one-shot and selective enhances the feature at the inference stage for high performance.\n\n|Methods|Paradigm|Stage|Feature Enhancemnet|\n|-|-|-|-|\n|VHL|Mul.|Train|Generative|\n|MOON|Mul.|Train|Constrastive|\n|FedImpro|Mul.|Train|Estimative|\n|FuseFL|One-shot|Train|Adaptor-based|\n|**Ours**|**One-shot**|**Inference**|**Selective**|\n\nFor prototype-based methods, FAFI aligns global prototypes by aggregating learnable local prototypes in one round.\n\n|Methods|Paradigm|Prototypes|\n|-|-|-|\n|VHL|Mul.|Generative|\n|FedProto|Mul.|Statistical|\n|CCVR|Mul.|GMM-based|\n|**Ours**|**One-shot**|**Learnable**|\n\nMul. is short for multi-round. We will add these discussions in our revision.\n\n**W5: More in-depth efficiency analysis**\n\n**Ans for W5:** Thanks for your valuable comments. We have conducted the memory and computation cost analysis of FAFI. Due to limited space, please see the response to **Reviewer Chua W1 & Q1**.\n\n**W6 & W7 & Q2: Writing issues**\n\n**Ans:** Thanks for your careful reading. \n1) The $\\Delta_{intra} = \\vert L(x, y) - L(x', y) \\vert$ is the performance discrepencies between any two samples $x, x'$ with the same label $y$. For any two samples $x, x'$, we can find a function $A$ that satisfies $x' = A(x)$. $A$ can abstract the feature variation among samples. \n2) The 'performance' in L69 represents the performance of the **local model**, we observe that better local performance is always accompanied by large parameter discrepancies(Fig.2-b). The performance in L258 denotes the performance of the **global model**. \n\nSorry for these misunderstandings. We will revise these writing issues and format errors and provide more explanations for clarity.\n\n[1] No free lunch theorem for security and utility in federated learning. ACM TIST, 2022.", "claims_and_evidence": "Claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria (e.g., benchmark datasets) make sense for the problem at hand.", "theoretical_claims": "I checked the correctness of proofs for theoretical claims.", "experimental_designs_or_analyses": "The experimental design is comprehensive. However, the feature fusion part might introduce extra communication or computation, authors can  provide more detailed interpretation of this.", "relation_to_broader_scientific_literature": "This work is related to the one-shot Federated learning. The experiment results reveals general weakness of previous OFL methods.\nThe Figure 4 also provides a holistic view to compare accuracy and communication costs of different OFL and multi-round FL methods.", "essential_references_not_discussed": "For the feature alignment, there are some works also enhance the feature quality including [1,2,3], in which aggregating same-class features and repelling features of different classes are also proposed. For prototype based FL, works [1,4,5] are also related. Authors should consider adding them for discussion, what differences of the proposed feature alignment and prototype learning from them. Moreover, in Section 4.3, what is the difference between the proposed feature fusion and FuseFL?\n\n[1] Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. In ICML 2022.\n[2] Model-Contrastive Federated Learning. In CVPR 2021.\n[3] FedImpro: Measuring and Improving Client Update in Federated Learning. In ICLR 2024.\n[4] Fedproto: Federated prototype learning across heterogeneous clients. In AAAI 2022.\n[5] No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. In NeurIPS 2021.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper thoroughly identifies and analyzes the inconsistencies within and between one-shot local models, providing a theoretical foundation for the proposed method.\n2. The introduction of Self-Alignment Local Training (SALT) effectively addresses intra-model inconsistencies by fostering invariant feature representations.\n3. The use of Informative Feature Fused Inference (IFFI) on the server-side improves the aggregation process, leading to better global model performance.\n4. Extensive empirical validation across multiple datasets demonstrates the effectiveness of the FAFI framework in a variety of scenarios, establishing its robustness.\n5. The proposed solution achieves a notable accuracy improvement than baseline models, showcasing its potential impact on the field.\n\n\nWeaknesses:\n1. The paper does not address the potential limitations and complexities of implementing the FAFI framework in real-world federated learning environments.\n3. There may be a lack of comprehensive comparisons with other state-of-the-art methods that also aim to reduce communication costs, apart from the eleven baselines included.\n4. Although the paper discusses privacy concerns, the utilized label information may still expose data privacy at some degree.\n5. For the feature alignment, there are some works also enhance the feature quality including [1,2,3], in which aggregating same-class features and repelling features of different classes are also proposed. For prototype based FL, works [1,4,5] are also related. Authors should consider adding them for discussion, what differences of the proposed feature alignment and prototype learning from them. Moreover, in Section 4.3, what is the difference between the proposed feature fusion and FuseFL?\n6. The paper could benefit from a more in-depth analysis of the computational efficiency and resource requirements of the FAFI framework compared to existing methods.\n7. Some format errors. Line 216, the formula is out of box. \n8. Some writing issues. Line 164, the \\Delta_intra is not defined in the main texts. The augmentation function A is not clear, why we need an augmentation function A here. I guess the meaning of A in appendix is the global dataset. \n\n\n[1] Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. In ICML 2022.\n[2] Model-Contrastive Federated Learning. In CVPR 2021.\n[3] FedImpro: Measuring and Improving Client Update in Federated Learning. In ICLR 2024.\n[4] Fedproto: Federated prototype learning across heterogeneous clients. In AAAI 2022.\n[5] No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. In NeurIPS 2021.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.\n1. Besides, to highlight the novelty, could you illustrate the detailed differences between the proposed methods with previous works? \n2. In Line 69, ``we observe that better performance is always accompanied by larger parameter discrepancies.''. However, later analysis shows that the large discrepancy is not good so you want to minimize (Line 258). So, could you interpret this with more details?\nI'm willing to increase my score if addressing above weaknesses and questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the OFL’s challenges due to the inconsistent models. Two technical innovations are proposed and achieved over 10% model accuracy improvement.
Decision: Accept (poster)