Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
Authors: Yichen Li
Xiuying Wang
Wenchao Xu
Haozhao Wang
Yining Qi
Jiahua Dong
Ruixuan Li
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Model-Heterogeneity
Primary area: other

Abstract: Model-Heterogeneous Federated Learning (Hetero-FL) has attracted growing attention for its ability to aggregate knowledge from heterogeneous models while keeping private data locally. To better aggregate knowledge from clients, ensemble distillation, as a widely used and effective technique, is often employed after global aggregation to enhance the performance of the global model. However, simply combining Hetero-FL and ensemble distillation does not always yield promising results and can make the training process unstable. The reason is that existing methods primarily focus on logit distillation, which, while being model-agnostic with softmax predictions, fails to compensate for the knowledge bias arising from heterogeneous models.

To tackle this challenge, we propose a stable and efficient Feature Distillation for model-heterogeneous Federated learning, dubbed FedFD, that can incorporate aligned feature information via orthogonal projection to integrate knowledge from heterogeneous models better. Specifically, a new feature-based ensemble federated knowledge distillation paradigm is proposed. The global model on the server needs to maintain a projection layer for each client-side model architecture to align the features separately. Orthogonal techniques are employed to re-parameterize the projection layer to mitigate knowledge bias from heterogeneous models and thus maximize the distilled knowledge. Extensive experiments show that FedFD achieves superior performance compared to state-of-the-art methods.

Feature Distillation is the Better Choice for Modee-Heteoogeneous Federated Learning
Yichen Li Xiuying Wang", Wenchao Xu Haozhao Wang',
Yining Qi Jiahua Dong", Ruixuan Li
School of Computer Science and Technology. Huazhong University of Science and Technology. Wuhan, China Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates International School, Beijing University of Posts and Telecommunication, Beijing, China *Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China {yc1i0204, he_ hz_wang @hust. edu. en, 1eowang980@bupt... eewppptt.ddu..... edu. cn
Abstract
Federated Learning (Hetero-FL.) has attracted growing at- tention for its ability to aggregate knowledge from heterogeneous models while keeping private data locally. To better aggregate knowledge from clients, ensemble distillation, as a widely used and effective technique, is often employed after global aggregation to enhance the performance of the global model. However, simply combining Hetero-FL. and ensemble distillation does not always yield promising results and make the training process unstable. The reason is that existing methods primarily focus on logit distillation, which, while being aggostic with softmmx predictions, fails to compensate for the knowledge bias arising from heterogeneous models. To tackle this challenge, we propose stable and effi- cient Feature Distillation for hodel-hhetroogeneuus Federated learning, dubbed FedFD, that can incorporate aligned feature information via orthogonal projection integrate knowledge from heterogeneous models better. Specifically, new feature-based ensemble federated knowledge distillation paradigm is proposed. The global model the server needs maintain a projection layer for each client- side model architecture to align the features separately. Orthogonal techniques are employed to re- re-parameterize the projection layer to mitigate knowledge bias from heterogeneous models and thus maximize the distilled knowledge. Exten- sive experiments show that FedFD achieves superior performance compared state-of- state-of-the-rr -i-thhe-r art methods.
1 Introduction
Federated Learning (FL) has become a core method for collaboratively training neural networks across distributed clients while ensuring data privacy protected [34] (5)) 29]. Recently, this framework has attracted considerable attention and is being applied in various domains, such as autonomous driving
systems 332 37]. recommender systems [27 26), and intelligent healthcare solutions [7] 40).
Typically, FL has been actively studied with a homogeneous model setting. With the development of loT products, different devices often possess varying computing resources, namely different model training capabilitics [39] 3T. In such scenario, it is essential to explore model-heterogeme federated learning (Hetero-FL) maximize the utilization of distributed computing resources. The key challenge here lies in how to aggregate the shared knowledge from different models.
Ruuixun Li is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address this issue, knowledge distillation [15] has emerged as promising approach, focusing on aggregating the output soft predictions of multiple local models into the global model. The authors in [52] propose aggregating the logit output instead of model parameters and [21] utilize the aggregated class score to regulate the local training process. Building on the partial parameter aggregation proposed in [6] many works use the ensemble distillation to improve the global model like [33] 49].
48 and [55] have focused on knowledge selection with logit distillation to optimize the aggregated model. 43 identifies the accurate and precise knowledge from local and ensemble predictions.
Although these methods achieve per- formance gains by integrating the ensemble distillation technique with Hetero-FL, how this distillation tech- nique works in Hetero- Fl. remains un- certain, especially when it comes to logit-based distillation. While logit distillation can theoretically address the shifts in the class probability dis- tribution caused by data heterogene- ity in FL with homogeneous models perfectly by using publicly available dataset with a similar distribution [46], it is sub-optinn sub-optimal strategy for model heterogeneity where the key challenge lies in different hidden layer represen- tations. During the distillation, each heterogeneous model will map the sample into a distinct feature space, resulting in significant varia- tions in their softmax predictions (logits) [10]. It is widely acknowledged that the logit representation only focuses on the output layer but can not align the representation in different feature spaces of heterogeneous models well, decreasing the distillation effectiveness.
(a) Logit Representation.
(b) Feature Representation.
Figure 1: The t-SNE visualization of ensemble knowl- edge representation by aggregated heterogeneous models on CIFAR-10.
We use t-SNE [45 to visualize the ensemble knowledge representation from client-side models on the distillation dataset in Figure Compared with feature representation in our method, aggregated logit representation has fuzzy classification boundaries, indicating that the performance of the teacher model is not promising enough. Moreover, we empirically find that the logit distillation will cause an unstable federated training process, then directly aggregating logits the teacher's 's prediction is not always effective. The specific experiments will be provided in Section 3.2
To break the limitations of logit distillation for FD with heterogeneous models, we in this paper explore feature distillation to ensemble the distilled knowledge where the feature representation is closely related to the model structure. Using feature representation for ensemble distillation in Hetero-FL poses a novel challenge, as in a centralized environment, feature knowledge is often distilled from a large model (teacher model) to smaller model (student model) with external projection layer [41 35 This differs from the Hetero-FL, where small models on various client sides usually ensemble distill knowledge
large model on the server, and the structures of these client-side models vary. Thus, designing the utilization of the projection layer and aligning feature representations of different client-side models becomes a primary challenge in ensemble distillation for Hetero-FL.
To explore this idea, we propose a stable and effective feature distillation method for Hetero-FL named FedFD that can align feature representations with orthogonal projection to mitigate the knowledge bias aggregated from heterogeneous models. More specifically, for each distillation sample, clients will extract the fcature representation, and the server aggregates the representations of the same client-side model architecture to obtain feature cluster formed by aggregated features from different model architectures. For each representation within the feature cluster, the server needs to train a projection layer to align the extracted feature representation of the server model with it and optimize the parameters of both the projection layer and the feature extractor of the server model. Furthermore, to prevent knowledge bias among aggregated feature representations, we utilize orthogonal projection techniques to maximize the transferred knowledge.
2

---

Server
e²
Mg (Wx)
Projection
Mr
Matrix W
Aggrogate
Feature Cluster S
Teacher
Student Feature
Orthogonalits
F Feature
Update
Gloabl
Model
Update
Feature Cluster S
Teacher
Student Feature
Orthogsnality
Aggregate
Projection
M,(WX)
Layer Mr
w
w
Client
W4
Client 2 W2
Client k
Wk
① Local
Local
① Local
Training
Training
Training
Private
Local
Private
Local
Private
Local
Data Dj
Data
Model Wa
Data
Model
Figure 2: The framework of FedFD. Before knowledge distillation, each client first trains its local dataset and then uploads its local model to the server. The server aggregates these models to obtain global model. During the distillation process, clients perform hierarchical feature alignment. This involves firstly aggregating feature representations from client-side models with consistent architectures. Then, for each aggregated feature representation, projection layer is maintained. This projection layer is obtained by transforming a square matrix to ensure its orthogonality. Finally, feature distillation achieved by aligning the feature representations of different model architectures using KL divergence separately.
Through extensive experiments over three datasets and different settings (various model architectures and data distributions), show that the proposed framework significantly improves the model accuracy as compared to ttate-of-the-ar algorithms. The contributions of this paper are:
We provide an in-depth analysis of model-agnostic federated knowledge distillation, iden- tifying that existing methods primarily rely on logit distillation, which poses significant challenges for misleading distilled knowledge representation with heterogeneous models.
We propose novel framework named FedFD which can be seen as an off offf-the-sheff person- alization add-on for Hetero-FL. FL and it inherits privacy protection and efficiency properties as traditional distillation methods.
We conduct extensive experiments on various datasets and settings. Experimental results illustrate that our proposed model outperforms the state-of-the methods by up to 16.09 % in terms of test accuracy on different tasks.
Related Work
Federated Learning a framework that trains a unified global model by aggregating individual models from multiple clients, each trained on their locally stored datasets [25] 47 28 30), One of the notable FL architectures is FedAvg [34], which strengthens the global model by aggregating parameters from locally trained models on private data. (23) involves incorporating proximal term mitigate the impact of differing data distributions across devices. While these methods are devoted to the homogeneous model across clients, (6) proposes to enable the training of heterogeneous local models with varying computation complexities on different clients. Similar to 6 [5I] integrates flexibility in both width and depth, utilizing skip connections to bypass certain layers and structured pruning to manage width. [17 a strategy that adjusts varying depths by aggregating common layers from clients' networks, such as the VGG network, create global models from different layer

---

groups. This paper focuses on federated learning with heterogeneous models across clients with improved knowledge distillation techniques.
Knowledge Distillation leverages the knowledge of a pre-trained model to supervise a smaller model, facilitating its application and deployment in environments with limited resources [14]. The
domain primarily encompasses two areas: logits distillation [38] 18] I [6] and feature distillation
Logits distillation, centered on classification tasks, introduces an additional goal to minimize the prediction discrepancy between the student and teacher models, initially using KL divergence
44
36
13
5
14 and later extended through spherical normalization [8], label decoupling [57], and probability reweighing [38). Our research focuses on feature distillation due to its versatility across tasks [5] and modalities [42]. The FSP matrix manually designed to capture feature relationships across residual layers [54]. Similarly, other studies proposed transferring knowledge via Gram matrices [24]. Activation boundaries and gradients capturing the loss landscape have also proven effective supervisory signals [44). We in this paper particularly focus on the feature distillation in FD by orthogonal projection and ensemble distillation.
Federated Distillation involves extracting knowledge from multiple teacher models, each trained by different clients, and transferring it to a student model [33
33 introduces the concept of applying knowledge distillation in a server setting, utilizing an unlabeled proxy dataset to transfer knowledge from local models to global model. multiple local models, using weights derived from the Bayesian posterior, to create a series of combined models. These combined models are then distilled into single global model. To break out the limitation of relying the proxy dataset with data generated by generative models, enabling the distillation without the need for actual data. We analyze the challenge of employing existing FD methods in the FL with heterogeneous models and propose to develop feature distillation instead of logit distillation.
further develops this by linearly aggregating
unlabeled auxiliary dataset, [58 56 49 propose suggest replacing
3 Methodology
In this section, we first formulate the standard FL process with both homogeneous and heterogeneous models. Then, we analyze the failure of logit distillation in Hetero-FL with experiments. Last, we introduce our feature distillation-based method, FedFD. The workflow of FedFD is shown in Algorithm I and Segurr/@illutrraaa the FedFD framework.
3.1 Problem Formulation
typical FL problem can be formalized by collaboratively training global model for total clients
in FL. We consider each client & can only access to his local private dataset Dj = { 3k }. where (i) E (1,2, C} is the corresponding label of s with Fk is the i-th input data sample and yj
(9)
(7)
(i)
C' classes. We denote the number of data samples in dataset D& by | The objective of the FL
system is learn global model w that minimizes the total empirical loss over the dataset D:
D DD]
min L(w)
Cu(w),
Dal
where Lx(w)=
[Dal Lce(wr
1
Σ
(I)
where LL(W) is the local loss in the k-th client and LCE is the cross-entropy loss function that measures the difference between the prediction and the ground truth labels. For homogeneous models, the server only needs to average local parameters to obtain the global follow:
K
D
wk.
While for heterogeneous models, we follow the protocol proposed in (6] and assume that all client models can be divided into p different architecture sets {S1, S Sp}. Then, we perform the ..............

---

global aggregation as follows:
W
" p-1
"
W \ up.
Spp
K
S
Wr E
W.ESp
(3) where w denotes the aggregated model of the p-th architecture of client-side models. For nota- tional convenience, we have dropped the training iteration index and simplified the weight for each aggregated client-side model in Eq. (3) which is often weighed by the ratio of sample numbers.
w w U (wz-1 \ wfi)) , \\ w.)
3.2 Logit Distillation Fails Hetero-FL
In existing FL methods, the key technique is to transfer knowledge from the teacher model to the student model by utilizing the model's I' soft predictions (logits) on the distillation dataset. Here, the teacher model prediction is defined with aggregated logits from multiple clients, which can also be referred to as ensemble distillation in FL. This technique works because in Homo-FL. (for homogeneous models), where the client models share the same structure, the logits output by these models do not suffer from model-based biases and can mitigate further data heterogeneity. However, it is inadvisable to directly apply this logit distillation technique to Hetero-FL (for het- erogeneous models), and existing related work has not addressed whether the logits from different client model architectures can be directly aggregated Figure we conduct further experiments. We selected several robust ensemble distillation meth- ods and conducted experiments on CIFAR-10 under both Homo-FL and Hetero-FL settings. The and the learning curves are shown in Figure 3
the teacher model prediction. Based on
details of these methods are described in Section 4.1
Under the Homo-FL setting, all methods con- verge quickly and stably, demonstrating the ef- fectiveness of logit distillation. However, in the Hetero-FL setting, the FL algorithm experiences training instability, where peaks repeatedly ap- pear in the curve, and the methods ultimately fail to converge to the optimal value. Although logit distillation can accelerate model conver- gence in the early stages of training, as the local model gradually converges on local data, the dis- tillation process becomes highly unstable. This contrary to the results observed in Homo-FL. The reason is that logit distillation solely relies on soft predictions without considering differences model architectures. Therefore, we will next introduce our feature-based distillation method to address this issue.
Homtogenencus Mudel
Hetenogenccus Modi
87.5
acuuncy
80.0
27.5
75.0
70
725
70.0
T- Commuricaiio Rounds
7 150
T=200
T-550
T= Communicatio Rounds
T 150 T=200
Figure 3: Learning curves of logit distillation- based methods on CIFAR-10 with different client- side model architectures.
3.3 FedFD: Feature Distillation for Hetero-FL
In this paper, we seek to unlock the potential of feature distillation in Hetero-FL. Despite the generality of feature distillation, it is frequently accompanied by complex design choices and heuristics. These decisions stem from loss functions between intermediate feature representations, leading to extra training costs. To overcome these limitations, we just employ the feature distillation that solely utilizes feature representation before the classifier.
In Hetero-FL, although the teacher model is typically the ensemble output of client-side small models, is still necessary attach projection layer after the global model (student model) to ensure that knowledge can be prooparop through the projection layer into the model parameters during the distillation process. However, this approach poses two challenges: (1) Due to the diversity of client-side model architectures, if the server maintains a personalized projection layer for each client, may lead to difficult training because each client contributes only tiny amount knowledge. Additionally, in FL. scenarios, the number of clients participating in pre-training is usually large, and maintaining an excessive number of projection layers can accumulate and result in significant storage costs for the server. (2) The feature knowledge derived from different cldent-side models may conffict
5

---

Algorithm 1: FedFD
Input :: communication round; K: client number; Dx: local dataset for the client k; global model; Wg: local model; Mę: projection layer; Se: the k-th model architecture; We: matrix for orthogonality.
Output:w, (wi...., J: global and local models.
for
to do
// communicationn round
2
Server randomly selects a subset of devices Sy;
Server send the global model w to devices. for each selected client & E St in parallel do
Train the local model wy with (I: Send the local model W/: back to the server. end
4 ServerAggregationn (es,) with (3):
Get the aggregated feature representation e" with 5 Orthogonalize Wn to obtain projection layer Ma with Distill the feature knowledge to the global model with
10
11
end
within different projection layers, and combining the complex non-linear knowledge together may not optimally train the feature extractor module of the global model.
To address these issues, we propose the reature-based distillation method, FedFD, with two main components: hierarchical feature alignment and parameter orthogonality. To obtain the classification model, in each round, the participating client firstly locally trains the model w2 with (1) and sends it to the server. After receiving uploaded models, the server aggregates multiple local models to get the global model w with (3). Like [33 49
588, the server treats the knowledge of the client model on the distillation dataset as teacher model to distill the global model and enhance its performance. Based on previous experiments, we explore using feature representation as the distilled knowledge. Denote that the local model wy of client & consists of feature extractor w and classifier head 0n. d is the dimension of the output feature. For each distillation sample its feature representation eg over the extractor w can be obtained as follows:
et = f(we: Vk E [].
Then, we divide the features into m groups i Sam... J. where each group S. = {w1. (...... w contains all extractors with the same structure outputting the d-dimensional feature. We use |Sal to represent the number of features in the group Sy. Next, we aggregate all feature representations in the group S, as:
Sa
where Sya (w
W [S
(5)
Sa
1=1
Instead of maintaining the projection layer separately for each client, the server now only needs to train (m - 1) projection layers (M2...., Mm.) This not only reduces training parameters but also ensures that each projection layer has sufficient knowledge for distillation.
However, this does not resolve the knowledge conflict among different model architectures. as the server still needs to integrate knowledge from various projection layers to optimize the global model parameters w. We indicate the knowledge Sy after the projection layer Ma as:
S Ma wx) M&Z, where Z = wx.
While in knowledge distillation, a larger number of distillation samples often leads to better distillation effects, as they can better cover the knowledge within the training data. This results Z being full-rank matrix; consequently, the Sa will also be full-rank matrix that be regarded non-linear mapping process, where the global model fails to discern the knowledge, leading to knowledge conflicts.
To address this issue, since we cannot alter the knowledge distribution of Z, we choose to process the projection layer_Ae Ma that they can map the knowledge Z into separate feature spaces. Here,

---

will investigate orthogonal projection transformations, which can resolve knowledge conflicts and maintain feature shapes due their rotational invariance. It is noteworthy that, due to the differing feature dimensions between Z. and Sg, with Z having a higher dimension than Sa, Ma is no longer an orthogonal square matrix but rather column matrix, specifically a Stiefel matrix manifold with orthogonal column vectors [1]. The favorable topological properties of such matrix can ensure support for gradient descent updates.
To maintain the orthogonality of the column vectors in Ma. we have observed many existing methods method. However, these methods all come
such as Cayley transformation [3] and with relatively high time performance costs. In this paper, we propose to generate the Ma with kkew-symmetric matrix Wa as follows:
exp(Wa) exp(Wa)" (Wa) exp Wa
W = exp(- W + WT) =I.
W
W
W n!
exp W.) = I+
(8)
2
3
where Wa = -W2 The parameters of Wa are randomly initialized. Although this is an infinite series, a reasonably accurate value can usually be obtained by taking the first few terms at a very low computational cost. Ultimately, Ma can be obtained by truncating the column vectors of exp (aa) to match the feature dimension Sa. The client updates W a through back-pru proppaaaatn with (9). By employing orthogonal projection, we ensure linear transformations of features. preventing knowledge conflicts and preserving feature shapes, thereby achieving maximized knowledge distillation.
Through orthogonal projection, Ma(wx) and dd share the same dimensionality. Next, we update the parameters and M to align the feature representations using Kullback-Leible divergence:
m
min
Σ (wx))(wx))e").
(9)
w. (Ma...... Mm) 171
After knowledge distillation, the server will broadcast the global model w the clients participating in the following communication round.
Modularity. FedFD demonstrates key characteristic: modularity. Existing Fl. techniques be seamlessly integrated with FedFD as ready-to-use enhancement with the following several benefits:
Optimization: The proposed framework can accommodate aggregation methods beyond HeteroFL (6] for global model updates, retaining convergence advantages.
Privacy: FedFD maintains the same level of network communication standard FL algorithms, avoiding privacy issues associated with uploading generative models or extra prototype-like information.
Flexibility: Although we search for the additional distillation datasets here, our frame- work can combined with existing data-free distillation techniques, balancing resource constraints and computational overheads in selecting distillation datasets.
4 Experiments
In this section, we evaluate our proposed method using three datasets and various baselines. We investigate the relationship between data heterogeneity and training efficiency. Additionally, we conduct ablation studies to examine each module FedFD. Finally, we conduct a sensitivity analysis to verify the effectiivnees of our method.
4.1 Experiment Setup
Dataset: We conduct our experiments with heterogeneously partitioned datasets over three datasets:
CIFAR-10, CIFAR-100 [19], and Tiny-ImageNet (20 Like [58] 49), we use the Dirichlet distribution Dir(a) on labels simulate the data heterogeneity. We apply all the training samples and distribute them to user models, and we use all the testing samples for the performance evaluation.
Baselines: For fair comparison with other key works, we follow the same protocols proposed by 66 to set up FD tasks with heterogeneous models, which is recorded as "-hetero". We evaluate our
7

---

Table 1: Performance comparison of various methods with the test accuracy.
CIFAR-10
CIFAR-100 100
Tiny ImageNet
Cotegories
Methods
Metniss
e-10.0
am110
n=001
a-10.0
a=1.0
1-10.0
a=1.0
1=0011
HetensF1.
Local
75
63.53g1r6
53.37
40.33
38077115
24.12 29.7
18.54
Global
88 88.45gnes
67.53
78.0
58. 58.51
57.42
g3.98
32.38
20 29.88
23 23.25
Classic FL
Local
50,33
73.87=29
63.98
52.85
50.21:24:
38.21
28.36
24.47az43
17.94
MOON betero
Global
58,05
87.92
79.05
58. 58.39
57.55
55.01
31.28
30.68au
-1.
Local
82.35
75.27
62.20
51.23
=8.2
37.53
33.98:2m0
25.62
20.31
FedFusion hetero
Global
$6,09
85.70aa13
78.47
59,53am11
58.86a112
12.35
15
31.56.122
26.09
Hooo FL
FedGien hetero
82.29.042
7601
62.74
51.99
47.80
39.79
34,04.230
25.37
21.77
Global
87,34
86.81
79
22:100
58.49
57.63
56.05
14.71
32.00
26.84
DaPKDalaterr
Local
83.83a1a1
77.09
64.71
32.76a533
48.99
39.98,539
33.85
26.02a319
22.26
Global
89.268431
87.84
60007
58.97
58.52,019
67.33
35
32.57
26.54.1.5
FedMI
Local
70.33
63.47.43
60.67
42. 42.28
45.64
26.90
28.84
19.37
13,89
Gilobal
75 1
66
46 46.77
40 10:
44
26.37
22, 22.60g
19.42.-17
MSFKD
82.26
27.31
62.50aa3
52.06
49, 29.94
27.93 34
22.59 23.59
Heter--F
Global
87.79
86 26.9
80.00s133
58. 38.88anc
39.09
56 29
36.29ame
31,62auu
26, 70
FedGD
Local
82. 4:400..0 o
76.02:1 x
63.71g446
$1.02ghhn
50.17:1
50 30.17
35.08
26.50ga
23.47g114
Global
87,52
87.22
29.31
59.26ana1
58. 38.03
34
36.86211
30.6621.
27.53
FedFD (ours)
$4.91=a4
78:03:14
65.33:912
54.98:176
57.24:1.0
41.68:412
36.78:511
30,90:522
23.41:4m
Global
90.06:aas
19.64:125
82.74:a25
61.07amm2
60.86
59.24:344
40.27
34.24:145
29.09
method with the following baselines. (1) Representative FL models: HeteroFL 66, MOON-OOONEEE hetero
[22]: (2) FL for homogeneous model: ciFFuusionhhterey [33). FedGen-hetero (58). Detero 49 (3) FL
for heterogeneous model: FedMD (21). MSFKD (48). FedGD (55).
Configurations: Unless otherwise mentioned, we set the number of local training epoch E 10, communication round 7 200, and the client number K = 20 with an active ratio == 0.4. For local training, the batch size is 64 and the weight decay is 1e 4. The learning rate is 0.01 for distillation and 0.001 for training the local model. For the model on the server, we employ ResNet-18 18 [T] as the basic backhone. Like [6), we construct ten different computation complexity levels (a, b, c.....) with the hidden channel decay rate 10%. For example, model "a" has all the model parameters, while models "b" and "c" have the 90% and 80% effective parameters. We use "a-d-g" three different model architectures in our experiments and conduct more experiments with other architectures in Section 422 Similarly to (6, we have conducted statistics based on the performance of the global model for all secondary experiments.
4.2 Performance Overview
shows the test accuracy of various methods with heterogeneous data across three datasets. Each experiment set is run twice, and we take each run's final ten rounds' accuracy and calculate the average value and standard variance. We report both the global model accuracy and the average accuracy of all local models. Firstly, we observe that the performance of the global model significantly outpaces that of the local model. For the Classic FL method, MOON does not achieve substantial advantage with heterogeneous models, suggesting that the contrastive loss hetween different model architectures provides limited improvement to the method. As for the Homo-FD method, while logit distillation introduces performance fluctuations, it offers some enhancement after an increase in training epochs. Both FedGen and DaFKD are data-free distillation methods, and compared to FedFusion, their performance improvements not significant reported in the original papers. possible reason is that the training the generator remains unstable, especially considering we employ relatively complex datasets instead of handwriting-diiit recognition datasets like MNIST. Regarding Hetero-Fi. methods, FedMD performs poorly due to the absence of aggregated parameters. The other two methods follow the logit distillation paradigm and exhibit superior performance. FedFD achieves the best performance in all cases by up to 16.09% in terms of global model accuracy.
Test Accuracy. Isoos
Communication Efficiency. Figure 2 (Table) compares the communication efficiency of various methods by measuring the communication rounds required to achieve the test accuracy and Figure focuses on the learning curves. Although knowledge distillation methods can accelerate model convergence. the training process of logit distillation is unstable, as we have detailed above. With the orthogonal projection technique in fcature distillation, FedFD achieves the fastest convergence with stable training process.
Ablation Study. As shown in Table E] we evaluate the effects of each module in our model via ablation alignnw
studies. -w/o feature alignment and -wo orthogonal projection denote the performance of our model without using hierarchical feature alignment and orthogonal projection. Compared with FedFD, the performance of FedFD -wo orthogonal projection and FedFD -wo orthogonal projection degrades evidently with a range of 0.63%~2..438 43%. While moving both components, the performance will drop significantly. Specifically, the orthogonal projection technique plays much more important role with the obvious improvement in test accuracy.
8

---

Table 2: Evaluation of different baselines three datasets (a = 1.0), in terms of the number of com- munication rounds to reach target test accuracy (ace). The best results are bold.
CIFAR 10 Dataset
CIFAR-100 Dataset
8
Accurscy
85-
Accuraicy
-5
Methods
CIEAR 10
CIRAR-100
25%
Tle ImageNet
80-
85 18
NeseaFt
17cm
15:
70g1
>
13
700
75
MODN
13:4
26.17
2219
5- -200
100.19
188_1
tedlopon-teto
PodGes-tekme
21111
35..5
198.2
> 2000
115.12
17612
70
100T-150T
1507-
T-0
1000- 150T-
44213
Communication Rounds
Communication Rounds
139.8
2
MSFXD
1919
45:17
33:10
171:20
57120
115131
Figure 4: Convergence and efficiency compar-
FolGD
ison of various methods two datasets.
FedED lours)
10:5
20.9
16.4
60.18
67.8
99.m
and the hierarchical feature alignment technique has a relatively small impact on test accuracy, which may be due the scale of the experiment, but it can save storage costs. Experiment results verify that all modules are essential train robust Hetero-FL. model.
Table 3: Ablation study of FedFD with two main components.
Table 4: Evaluation of combination of various client-side models levels for CIFAR- 10 dataset (a 1.0).
CIFAR-10
CIFAR-100
Method
n=1.0
o=0.1
o=l.D
o=0.1
Method
Modal Heterogencity
FedFD
89.64
82.74
60.86
59.24
a-d-2
b-d-f
a-c-d-F-
-wo feature alignment
89.01
81.56
59.77
58.67
HeteroFL
87,53
88. 88.15s
85.01=030
82.53au27
FedFFusion-hetero
85.70
86.85
85.37
84, 84,10
w/o orthogonal projection
87.96
80.92
59.33
57.29
87.96,404
88.47.001
85.23
w/o both components
85.70
78.47
58.86
56.12
FedHH)
89.64:431
89.92.n9
87.47
85.92.021
Data Heterogeneity. Table[[illustraal the variation in test accuracy across different levels of data heterogene- ity. clear trend emerges, with all methods exhibiting improved accuracy as data heterogeneity decreases. Simultaneously, we observed that the data heterogeneity has a greater impact on local models than on global models. Specifically, when a = 0.1, the performance of local models across all baselines on CIFAR- 100 declines significantly, this underscores the urgent need to investigate Hetero-FI., as they are more susceptible to the influence of data distribution. Notably, FedFD) consistently outperforms other methods, achieving the most significant improvements across all settings.
Parameter Sensitivity Analysis. In this section, we first explore the model heterogeneity issue with different models. As shown in Table [ we conduct experiments using four different models and define that a larger number of models represented a higher level of model heterogencity. As the degree of heterogeneity increased, the performance of all methods declined, with FedFusion showing particularly significant drop. This verified the shortcoming of logit distillation in the context of model heterogeneity. In contrast, FedFD consistently retains superior performance.
Table 5: The scalability of FedFD and other base- lines (o=0.01).
Table 6: The different architectures of models (CNN and ResNet).
CIEAR 10
Dataset
63.29
HescroF1.
FediPasion Letero
14
MSFKD 65, 30:30amm
FediFD 67.00:mm
Methaal
CIEAR 10
CIRAR- ICO
Tity- ImageSet
CIFAR-100
a1
45.31a10
44.90
48.05
FelMD
53.10
Thy NmageNett
21.56
22.97a338
23,26
71.24g104
29.81
34.3
3677-11
27.W6aar
Then, examine the scalability of our method. As shown in Table although all methods exhibit significant performance degradation in large-scale experiments, forcing some clients to own very few samples, FedFD outperforms other baselines, demonstrating its robustness and effectiveness.
Model Architecture. Although the above experiments validate the effectiveness of FedFD within the HeteroFL framework, FedFD fundamentally constitutes an advanced feature distillation method independent of model ar- chitectures and parameter aggregation strategies. We selected HeteroFL the framework due to its widespread recognition, which facilitates comprehensive comparisons with other baselines. In Table 0 we conducted experiments using two architectures to mitigate potential impacts arising from the HeteroFL framework. Here, we adopt the FedMD to perform knowledge distillation to fuse client knowledge directly. The experimental results demonstrate the adaptability of FedED with diverse architectures.

---

5
Conclusion
We introduced FedFD, straightforward framework, to tackle feature distillation using heterogeneous models within federated learning. FedFD serves minimal personalization extension for any federated learning algorithms involving global model aggregation, ensuring privacy and communication efficiency. Comprehensive experiments demonstrate that FedFD can enhance test accuracy.
Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62376103, 62302184, 62436003 and 62206102: Major Science and Technology Project of Hubei Province under grant 2024BAA00S Hubei Science and Technology Talent Service Project under grant 2024DIC078; Ant Group through CCF-Ant Research Fund; and Fundamental Research Funds for the Central Universities under grant YCJJ20252319, The computation is completed in the HPC Platform of Huazhong University Science and Technology.
References
[1] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10925-10934, 2022.
[2] Ilai Bistritz, Ariana Mann, and Nicholas Bambos. Distributed distillation for on-device learning. In Proceedings Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, 2020.
[3] Arthur Cayley. The collected matemmttcal papers, volume University, 1894.
[4] Hong-Yop You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia, Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5008-5017, 2021.
[6] Enmao Dian, Ding, and Vahid Tarokh, Heterofl: Computation and communication efficient federated learning for heterogencous clients. arXiv preprint arXiv:2010.01264, 2020.
[7] Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. No one left behind: Real-world federated class-incremente learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2054-2070, 2024.
[8] Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofei He, and Deng Cai. Reducing the teacher-student gap via spherical knowledge disitllation. arXiv preprint arXin:0010.07485, 2020.
[9] Qiushan Guo, Xinjiang Wang. Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 11020- 1000,, 2020.
[10] Zhiwei Hao. Jianyuan Guo, Kai Han, Yehui Tang. Han Hu, Yunhe Wang, and Chang Xu. One-for- all: Bridge the gap between heterogeneous architectures in knowledge distillation, Advances Neural Information Processing Systems, 71:7955777753 2023.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Ping He, Xiaohua Xu, Jie Ding, and Baichuan Fan. Low-rank nonnegative matrix factorization on stiefel manifold. Information Sciences, 514:131-148, 2020.
[13] Bycongho Heo, Jecsoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. comprehen- sive overhaul of feature distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1921-1930, 2019.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 503551 2015. 2015.
10

---

[15] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyue Kim. Communiiaation-efficcenn on-device machine learning: Federated distillation and augmentation under non-iid private data. CoRR, abs/181 1.449 2018.
[16] Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 01277-242285,, 2023.
[17] Honggu Kang, Seohyeon Cha. Jinwoo Shin, Jongmyeong Lee, and Joonhyuk Kang. Nefl: Nested federated learning for heterogeneous clients. arXiv preprint arXiv:2308.0777. 2023.
[18] Youmin Kim, Jinbae Park, YounHo Jang, Muhammad Ali, Tae-Hyun Oh, and Sung-Ho Bae. Distilling global and local logits with densely connected relations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6290-6300, 2021.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[20] Ya Le and Xuan Yang. Tiny imagenct visual recognition challenge. CS 231N, 7(7)3, 2015.
[21] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint / 2019.
[22] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713- 10713-10722, 2021.
[23] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future dircctions. IEEE signal processing magazine, 37(3):50-60, 2020.
[24] Xiaojie Li. Jianlong Wu, Hongyu Fang. Yue Liao, Fei Wang, and Chen Qian. Local correlation consisteney for knowledge distillation. In European Conference on Computer Vision, pages 18 33. Springer. 2020.
[25] Yichen Li. Quawei Li, Haozhao Wang. Ruixuan Li, Wenliang Zhong, and Guannan Zhang. Towards efficient replay in federated incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12820-12829, June 2024.
[26] Yichen Li, Qiyu Qin, Gaoyang Zhu, Wenchao Xu. Haozhao Wang. Yuhua Li, Rui Zhang, and Ruixuan Li. A systematic survey on federated sequential recommendation. arXiv preprint 2/Xiv:2504.05313, 2025.
[27] Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang, Wei Wang. Wang, and Ruixuan Li. Personalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings of the ACM on Web Conference 2025, WWW '25, page 2700 2709, New York, NY, USA, 2025. Association for Computing Machinery.
[28] Yichen Li, Haozhao Wang, Yining Qi, Wei Liu, and Ruixuan Li. Re-fed+: A better replay strategy for federated incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[29] Yichen Li. Haozhao Wang, Wenchao Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang. Xin Yang, Rui Zhang, Shui Yu, Song Guo, and Ruixuan Li. Unleashing the power of continual learning on non-centralized devices: A survey, 2024.
[30] Yichen Li, Yuying Wang, Haozhao Wang. Yining Qi. Tianzhe Xiao, and Ruixuan Li. Fedssi: Rehearsal- free continual federated learning with synergistic synaptic intelligence. Forty-second International Conference on Machine Learning.
[31] Yichen Li, Wenchao Xu, Haozhao Wang, Ruixuan Li, Yining Qi, and Jingcai Guo. Personalized federated innainnincrementlll learning based on adaptive knowledge matching, 2024.
[32] Yijing Li. Xiaofeng Tao. Xuefei Zhang, Junjie Liu, and Jin Xu. Privacy- federated learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23(7):8423 8434, 2021.
[33] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in foderated Icarning Advances in Neural Information Processing Systems, 33:2351-2366 2020.
[34] Brendan McMahan, Eider Moore. Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efliiiee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[35] Roy Miles, Ismail Elezi, and Jiankang Deng. Vkd: Improving knowledge distillation using orthogonal projections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pages
11

---

[36] Roy Miles, Adrian Lopez Rodriguez, and Krystian Mikolajczyk. Information theoretic representation distillation. arXiv preprint arXi::2112.00559, 2021.
[37] Dinh C. Nguyen, Quoe Viet Pham, Pubudu N Pathirana, Ming Ding, Aruna Seneviratne, Zihuai Lin, Octavia Dobre, and Won-Joo Hwang. Federated learning for smart healtheare: A survey. ACM Computing Surveys (CSUR), 55(3):1-37, 2022.
[38] Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang. Respecting transfer gap in knowledge distillation. Advances in Neural Information Processing Systems, 35:21933-21947, 2022.
[39] Kilian Pfeiffer, Martin Rapp. Ramin Khalili, and Jörg Henkel. Federated learning for computationally constrained heterogeneous devices: A survey, ACM Computing Surveys, 1-7,, 2023.
[40] Bjarne Pfitzner, Nico Steckhan, and Bert Armrich. Federated learning in medical context: a systematic literature review. ACM Transactions on Internet Technology (TOIT), (1(2))1331, 2021.
[41] Adriana Romero. Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
[42] V Sanh. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01108, 2019.
[43] Jiawei Shao, Fangzhao Wu, and Jun Zhang. Selective knowledge sharing for privacy-preserving federated distillation without a good teacher. Nature Communications, 15(1):349, 2024.
[44] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint 19Xiv:1910. 10699, 2019.
[45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1 1), 2008
[46] Chumnan Wang, Xiang Chen, Junzhe Wang, and Hongzhi Wang. Atpfl: Automatic trajectory prediction model design under federated learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6563 6572, 2022.
[47] Chunnan Wang, Xiang Chen, Junzhe Wang, and Hongzhi Wang. ATPFL: automatic trajectory prediction model design under federated learning framework. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, New Orleans, LA, Juee 18-24, J8-24,Jume 18-24, pages 3553-6562, 2022.
[48] Dong Wang, Naifu Zhang, Meixia Tao, and Xu Chen. Knowledge selection and local updating optimization for federated knowledge distillation with heterogeneous models. IEEE Journal of Selected Topics in Signal Processing, 17(1)822-1) 97, 2022.
[49] Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain- aware federated knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages (0412-2042) 2023.
[50] Haozhao Wang, Haoran Xu, Yichen Li, Yuan Xu, Ruixuan Li, and Tianwei Zhang. Fededa: Federated learning with cross-rounds divergence-awar aggregation. In The Twelfih International Conference on Learning Representations, 2023.
[51] Kaibin Wang, Qiang Hc, Feifci Chen, Chunyang Chen, Faliang Huang, Hai Jin, and Yun Yang. Flexifed: Personalized federated learning for edge clients with heterogeneous model architectures. In Proceedings of the ACM Web Conference 2023, pages 2979-2990, 2023.
[52] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication--fficiieet federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
[53] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2021.
[54] Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4133-4141, 2017.
[55] Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. Target: Federated consincalttuall learning via exemplar-fre free distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4782 4793, 2023.
12

---

[56] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In IEEE/CVE Conference Computer Vision and Puttern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10164-10173.
[57] Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953-11962 2022.
[58] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou Data free knowledge distillation heterogeneous federated learning. In International Conference on Machine Learning, pages 12878 128889 PMLR, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The summarized contributions provided in the abstract and introduction.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We indicated the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We did not include new theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
14

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We could provide more details and codes to support the reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer:
[Yes]
Justification:
We could provide our code if required.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cclpppss.cc/ppee cctppliic//pc///ccccccccccccc.cc quides/CodeSubmi seioooolcyi for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ :/nniss//.///////////e:::eee cc/public/ qu..dess/ooeeSuubllesssoool..... for more details.
15

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provided the settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The results are accompanied by error bars, confidence intervals, or statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the amount of compute required for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more computing than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Does the research conducted the paper conform. every respect, with NeurlPS Code of Ethics http:///wwrrpp..ccccpplle//cccca
Question:
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code Ethics.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: cited original paper that produced the code package or dataset.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
18

---

Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the rescarch, declaration is not required.
Question:
Answer: [NA]
Justification: did not use L.L for core method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:///wuaiaeccccoooo/eeeeeee/s for what should or should not be described.
19


Reviews:
[{"review_id": "6W011QADWI", "reviewer": "Reviewer_zr5J", "summary": "This paper proposes a model-heterogeneous FL framework, FedFD, that improves information integration between different models through feature distillation. Compared to existing methods that rely on logit distillation, FedFD effectively compensates for knowledge bias from heterogeneous models. Furthermore, FedFD incorporates aligned feature information via orthogonal projection to maximize the distilled knowledge. Extensive experiments demonstrate that FedFD achieves superior performance compared to state-of-the-art methods.", "questions": "1. The paper needs to address the challenge of obtaining high-quality public datasets that correlate well with local data. The authors should elaborate on how this requirement might impact privacy considerations in federated learning settings and propose strategies to mitigate these concerns. Additionally, the authors should discuss whether their method would still be effective in scenarios where no public datasets are available.\n2. While the paper aims to address model heterogeneity in FL, the experiments only consider variations in channel numbers rather than truly heterogeneous model architectures (e.g., ResNet vs. CNN vs. MobileNet). The authors should demonstrate how their method would perform with more significant architectural differences. Additionally, for channel heterogeneity specifically, HeteroFL appears to be a more suitable approach. The authors need to clarify the key distinctions between FedFD and existing approaches like HeteroFL and FedRolex[2].\n3. The paper would benefit from theoretical analysis supporting the effectiveness of FedFD, including theoretical guarantees or convergence analysis. The experiments are limited to relatively simple datasets like CIFAR, and the authors should consider testing on more complex, real-world datasets to demonstrate broader applicability.\n4. The formatting inconsistencies in the paper (e.g., font size variations in lines 270-271) should be addressed in a revised version.\n\n[2] FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction", "limitations": "The authors could further strengthen limitation section by briefly discussing potential privacy or fairness implications that might arise from their approach, especially concerning the requirement for public datasets, and perhaps outline initial thoughts on how they plan to address the LLM challenges in future work.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**：\n\n1. This paper addresses a key challenge in real-world FL deployment - model heterogeneity. Effectively supporting model heterogeneity among participating devices enables broader deployment of federated learning across devices with varying resources.\n2. The paper is well-written with clear logical organization throughout. Additionally, the authors present the proposed algorithm with clear illustrations and well-structured pseudo-code, making it easy for readers to understand the methodology.\n3. The authors perform extensive experiments to demonstrate FedFD effectiveness, including various Non-IID scenarios (α=0.1, 1.0, 10.0), multiple datasets (CIFAR10, CIFAR100, Tiny-ImageNet), and comprehensive ablation studies.\n\n\n\n**Weaknesses**：\n\n1. The primary issue with the method's design is how to obtain high-quality public datasets. Several references demonstrate that the correlation between public datasets and local datasets on participating devices significantly impacts model performance[1], and obtaining such datasets is extremely challenging. Additionally, acquiring these datasets to some extent compromises the privacy of users' local data.\n2. There's a mismatch between the paper's motivation and experiments. The main purpose of model heterogeneity is to enable devices with different computational resources to participate effectively. Typically, models of varying complexity and architecture (e.g., ResNet, CNN) would be deployed on these devices, but FedFD's evaluation shows very low model heterogeneity, with differences only in channel numbers. This fails to effectively validate the proposed method's effectiveness. Furthermore, some existing methods like HeteroFL and FedRolex have already implemented similar approaches, requiring further discussion on the differences.\n3. FedFD lacks theoretical analysis to prove its effectiveness. Additionally, the experimental section uses overly simple models (like ResNet-18) and datasets (like CIFAR). Testing FedFD on a broader range of datasets is necessary.\n4. There are inconsistencies in font sizes, for example in lines 270 and 271.\n\n\n\n[1]FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models", "was_revised": true, "final_justification": "I still think this paper is not good enough to be accepted.", "rebuttal": "Thank you for your careful review and valuable comments. In the following, we give point-by-point responses to each comment.\n\n> **W1&Q1.  Concerns about public dataset.**\n\n**R1:** Thank you for this constructive suggestion. Research on federated knowledge distillation can be approached from two main perspectives: (1) the design of distillation algorithms, and (2) the generation of high-quality public datasets with privacy guarantees. Work in category (1), such as FedFusion [1], typically assumes access to high-quality public datasets and focuses on developing distillation algorithms to better leverage such data for performance improvement. In contrast, research in category (2) explores data-free distillation, aiming to synthesize privacy-preserving distillation data. Representative methods such as FedGen[2], FedFTG [3], and DaFKD [4] adopt generative models (e.g., GANs) to generate synthetic data. **These (2)-type methods can often be viewed as add-on components to (1)-type approaches, as they require additional generative models to support data-free distillation.**\n\n**Our proposed method, FedFD, primarily focuses on distillation algorithm design but is also compatible with data-free distillation techniques.** To validate this, we conduct additional experiments where we employ the three aforementioned data-free methods to synthesize distillation data and evaluate the effectiveness of FedFD. Unlike logit-based distillation, even though the quality of synthetic data may not match that of assumed high-quality public datasets, FedFD leverages feature-level outputs to capture richer knowledge, resulting in no significant performance degradation.\n\n| Methods          | Metrics | CIFAR-10 ($\\alpha=1.0$) | CIFAR-100 ($\\alpha=1.0$) |\n| ---------------- | ------- | ----------------------- | ------------------------ |\n| **FedFD-Public** | Local   | 78.03                   | 52.24                    |\n|                  | Global  | 89.64                   | 60.86                    |\n| FedFD-FedGen     | Local   | 77.12                   | 50.93                    |\n|                  | Global  | 87.99                   | 58.91                    |\n| FedFD-FedFTG     | Local   | 77.04                   | 51.35                    |\n|                  | Global  | 88.11                   | 59.14                    |\n| FedFD-DaFKD      | Local   | 77.58                   | 51.45                    |\n|                  | Global  | 88.97                   | 60.02                    |\n\nFurthermore, to ensure fair comparison in our experiments, the public dataset used for distillation is deliberately chosen to have distributional differences from the training dataset. Specifically, we use {CIFAR-100, Tiny-ImageNet, ImageNet} as the distillation datasets for {CIFAR-10, CIFAR-100, Tiny-ImageNet}, respectively. It is also a common experimental setting in (1)-type works.\n\n> **W2&Q2. Concerns about model heterogeneity setting.**\n\n**R2:**  Thank you for raising this concern. We fully agree with the esteemed reviewer’s observation that the main purpose of model heterogeneity is to enable devices with different computational capabilities to participate effectively. In our experimental setup, we choose HeteroFL as the backbone paradigm for two main reasons:\n\n(1) The channel heterogeneity addressed by HeteroFL is itself a form of model heterogeneity. In this setting, different devices deploy sub-models with similar architectures but varying sizes and computational requirements. Such scenarios are common in practice—for example, **in UAV clusters or joint ventures between subsidiaries with a trusted terminal—where they often follow their own protocol standards that define the set of model architectures to be used, rather than allowing each client to independently choose entirely different model architectures.** Therefore, there is no need to deploy entirely different model architectures that would introduce significant bias and training overhead. In contrast, adapting model based on hardware capabilities is more practical and cost-efficient.\n\n(2) As the first work to explore feature-based federated distillation, we aim to rigorously validate the effectiveness of our method. To do so, it is important to compare against existing state-of-the-art logit-based federated distillation approaches, which typically rely on parameter aggregation to ensure proper convergence. Without such aggregation, these methods tend to struggle since distillation is insufficient to effectively guide the global model aggregation in the early stages of training, it may lead to unstable convergence or even prevent the model from converging properly, resulting in suboptimal performance.. **Therefore, to ensure a fair comparison, we adopt HeteroFL as the backbone paradigm for model aggregation in our experiments.**\n\n**Finally, and most importantly**, in Table 6 of our original paper (We have reused this table from the original content in our rebuttal for your convienience), we have already conducted experiments to address the reviewer's concern regarding models of varying complexity and architecture (e.g., ResNet, CNN). These experiments evaluate the performance of our method alongside the model-agnostic approach FedMD, demonstrating the superiority of feature-based distillation under heterogeneous model settings.\n\n| 50% CNN, 50% ResNet | CIFAR-10     |              | CIFAR-100    |              | Tiny-ImageNet |              |\n| ------------------- | ------------ | ------------ | ------------ | ------------ | ------------- | ------------ |\n| Method              | $\\alpha=1.0$ | $\\alpha=0.1$ | $\\alpha=1.0$ | $\\alpha=0.1$ | $\\alpha=10.0$ | $\\alpha=1.0$ |\n| FedMD               | 67.59        | 53.10        | 30.52        | 24.47        | 21.29         | 19.95        |\n| **FedFD**           | **71.28**    | **59.51**    | **34.93**    | **28.98**    | **30.79**     | **27.96**    |\n\nRegarding the difference between FedFD, HeteroFL, and FedRolex, **federated knowledge distillation can be viewed as an add-on component to parameter aggregation methods, designed to further improve model performance.** Both HeteroFL and FedRolex serve as base parameter aggregation frameworks, while FedFD leverages a public dataset to enhance the performance of the aggregated models. And for logit-based federated distillation methods, they also enhance the global model aggregated in a FedAvg manner.\n\n> **W3&Q3. Concerns about theoratical analysis and simple experimental settings.**\n\n**R3:**  Thank you for raising this concern. Based on our explanation in **R2**:  *federated knowledge distillation can be viewed as an add-on component to parameter aggregation methods, designed to further improve model performance* — we argue that it is not necessary to independently prove the convergence of HeteroFL, which serves as the backbone paradigm in our work. Furthermore, **there is currently no unified or generally applicable theoretical guarantee for the knowledge distillation process**. Prior works also do not provide convergence proofs specifically for the distillation procedure. The theoretical analyses presented in DaFKD [4] and FedGen [2] focus on the convergence of **generative models**, which is beyond the scope of our work, as we do not introduce any new model that requires global aggregation. \n\nRegarding this part of the experimental analysis, we sincerely apologize that our current setup was based on established practices in prior top-tier conference papers, upon which we have already made improvements. Existing works typically use VGG or ResNet family models and conduct evaluations on datasets such as MNIST, Fashion-MNIST, EMNIST, and, in a few cases, CIFAR-10/100. **Building on this, we further extended the evaluation to the Tiny-ImageNet dataset, which is also widely adopted in federated learning research.** We regret that due to the limited rebuttal period, we were unable to collect and analyze more complex, real-world datasets. However, we will expand our experiments in the final version to include larger models and more realistic datasets as suggested.\n\n> **W4&Q4. Concerns about inconsistent font sizes.**\n\n**R4:**  Thank you for this concern. To visually differentiate our method from the backbone paradigm, we **applied the `\\small` command** to the “-hetero” label. **We sincerely apologize if this formatting choice caused confusion or did not align with your presentation standards.** We will polish this distinction in a more effective and visually consistent manner in the revised version.\n\n**PS: We have carefully addressed the concerns you raised in our rebuttal. If there are any remaining questions or clarifications needed, please feel free to point them out during the discussion phase. We sincerely hope that our work will receive your recognition following the rebuttal and discussion. We are looking forward to hearing your feedback soon!**\n\nReferences used in our rebuttal:\n\n[1] Lin T, Kong L, Stich S U, et al. Ensemble distillation for robust model fusion in federated learning[J]. Advances in neural information processing systems, 2020, 33: 2351-2363.\n\n[2] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning[C]//International conference on machine learning. PMLR, 2021: 12878-12889.\n\n[3] Zhang L, Shen L, Ding L, et al. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10174-10183.\n\n[4] Wang H, Li Y, Xu W, et al. Dafkd: Domain-aware federated knowledge distillation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 20412-20421."}, {"review_id": "5HQpqAFNQh", "reviewer": "Reviewer_gifN", "summary": "This paper proposes FedFD, a feature distillation framework for model-heterogeneous federated learning with feature alignment and orthogonal projection to mitigate knowledge bias across clients. Evaluated on CIFAR-10/100 and Tiny-ImageNet under varying data heterogeneity, FedFD outperforms all SOTA methods and demonstrates faster convergence and robustness.", "questions": "1. For data-free deployment, how would FedFD integrate with generator-based methods? Would feature distillation require adjustments?\n\n2. In Figure 3, logit distillation destabilizes Hetero-FL training. Does feature distillation alone explain FedFD’s stability, or is orthogonal projection the key factor?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow. \n\n2. The presentation is quite good and the mathematical formulation is rigorous.\n\n3. The proposed method is a technically sound solution to mitigate knowledge bias across heterogeneous models.\n\n4. Extensive experiments have been done to verify the effectiveness in various settings. \n\nWeaknesses:\n\n1. FedFD relies on a proxy dataset like prior FD works. While mentioned as flexible, experiments only use additional real data but no tests with generated data.\n\n2. Hetero-FL baselines are logit-based. Comparisons to feature-distillation FL methods or personalized FL would strengthen claims.\n\n3. Maintaining projection layers per client architecture could increase server storage or computation budget. I suggest that authors analyze and propose a solution to this issue.", "was_revised": true, "final_justification": "Although the paper has certain limitations in its setting, it still demonstrates considerable potential for broad applicability. Since all raised concerns have been properly addressed, I decide to keep my score.", "rebuttal": "We thank you very much for providing the positive comments. In the following, we give detailed responses to each review. \n\n> **W1&Q1. Concerns about generated proxy data.** \n\n**R1:** Thanks a lot for raising this concern. **Federated knowledge distillation can be viewed as an add-on component to parameter aggregation methods, designed to further improve model performance.** HeteroFL serve as base parameter aggregation frameworks, while FedFD leverages a public dataset to enhance the performance of the aggregated models. And for logit-based federated distillation methods, they also enhance the global model aggregated in a FedAvg manner. To validate this, we conduct additional experiments where we employ the three data-free methods to synthesize distillation data and evaluate the effectiveness of FedFD. Unlike logit-based distillation, even though the quality of synthetic data may not match that of assumed high-quality public datasets, FedFD leverages feature-level outputs to capture richer knowledge, resulting in no significant performance degradation.\n\n| Methods          | Metrics | CIFAR-10 ($\\alpha=1.0$) | CIFAR-100 ($\\alpha=1.0$) |\n| ---------------- | ------- | ----------------------- | ------------------------ |\n| **FedFD-Public** | Local   | 78.03                   | 52.24                    |\n|                  | Global  | 89.64                   | 60.86                    |\n| FedFD-FedGen     | Local   | 77.12                   | 50.93                    |\n|                  | Global  | 87.99                   | 58.91                    |\n| FedFD-FedFTG     | Local   | 77.04                   | 51.35                    |\n|                  | Global  | 88.11                   | 59.14                    |\n| FedFD-DaFKD      | Local   | 77.58                   | 51.45                    |\n|                  | Global  | 88.97                   | 60.02                    |\n\nFurthermore, to ensure fair comparison in our experiments, the public dataset used for distillation is deliberately chosen to have distributional differences from the training dataset. Specifically, we use {CIFAR-100, Tiny-ImageNet, ImageNet} as the distillation datasets for {CIFAR-10, CIFAR-100, Tiny-ImageNet}.\n\n> **W2. Concerns about other feature-distillation or personalized baselines.**\n\n**R2:** Thank you for this helpful comment. We apologize if this was not sufficiently clear. To the best of our knowledge, **we are the first to study feature-based distillation in federated learning**. Therefore, our baselines are selected from two perspectives: state-of-the-art methods in model-heterogeneous federated learning and logit-based federated distillation. As for traditional FL methods, we have included MOON under the HeteroFL paradigm for comparison. Overall, our choice of baselines and datasets is consistent with or even more advanced than those used in existing related works.\n\n> **W3. Concerns about storage or computational cost for projection layers.**\n\n**R3:**  Thank you for your valuable comment. We would like to clarify that the two modules introduced in FedFD will not limit the practical deployment. First, the **hierarchical feature alignment** module is essentially a feature aggregation strategy that does not introduce additional trainable parameters. Although it aggregates multiple feature representations, these are stored and processed on the **server side**, not the clients. In the context of federated learning, it is standard practice to focus on client-side overhead, as servers are generally assumed to have sufficient computational and storage capacity. Second, while the **orthogonal projection** module does introduce new parameters, these computations are also conducted **entirely on the server**. Therefore, this module does not increase the resource requirements for client devices. Overall, both modules are designed to keep the client-side lightweight, and thus do not hinder deployment on edge devices.\n\n> **Q2.Key factor contributing to stable Hetero-FL training.**\n\n**A2:**  Thank you for this question. In Figure 3, the paper demonstrates that logit distillation causes instability in Hetero-FL training, as the aggregation of soft predictions fails to address the misalignment in feature spaces between heterogeneous models. The instability arises because logits reflect only the output layer and are sensitive to variations in model architectures, leading to ineffective and misleading knowledge distillation across clients. FedFD addresses this issue not merely through feature distillation alone, but by combining it with orthogonal projection. The ablation study (Table 3) confirms that removing the orthogonal projection module significantly degrades performance, more so than removing hierarchical feature alignment. T**his suggests that while feature distillation helps align internal representations, the orthogonal projection is the key factor that ensures stability and effectiveness.** It resolves knowledge conflicts by mapping features into orthogonal subspaces, thus maximizing the knowledge transfer and enabling consistent model updates in heterogeneous federated settings."}, {"review_id": "5SbSvRIxXV", "reviewer": "Reviewer_AyEq", "summary": "The proposed framework FedFD focuses on federated learning for edge clients with heterogeneous model architectures. To address the limitations of traditional logits aggregation, FedFD utilizes orthogonal projection to ensure linear transformations of features, preventing knowledge conflicts and preserving feature shapes.", "questions": "Please refer to the Weaknesses section for specific limitations and suggestions for improvement.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n1.The proposed feature distillation technique, FedFD, tackles the challenges of model heterogeneity in federated learning by using orthogonal projection to align feature representations.\n\n2.FedFD is designed to be easily integrated into existing federated learning systems.\n\n3.The experiments are solid.\n\nWeaknesses:\n\n1.A potential weakness is that the HeteroFL setup used in this work is not a typical distillation-based heterogeneous federated learning configuration. This setup leans more towards federated learning based on sub-model partitioning, making it difficult to distinguish whether the final results stem from distillation or model aggregation. The authors discuss this issue in Tab. 6, but a more detailed explanation is still needed for this case study.\n\n2.The author's core contribution to this paper, distillation using orthogonal projections, requires a more detailed explanation to enhance readability.\n\n3.The distillation method proposed by the authors bears similarities to \"VKD: Improving Knowledge Distillation Using Orthogonal Projections.\" VKD also focuses on the computational overhead issues of the Cayley transformation and Gram-Schmidt method and proposes an improved feature distillation approach. The authors need to further clarify the differences between their proposed distillation method and VKD. Additionally, the authors should provide a clearer explanation of why their proposed distillation method is more suitable for distributed environments compared to VKD.", "was_revised": true, "final_justification": "My concerns are mostly addressed. I will raise my original score. But I still concern about the contribution of this work.", "rebuttal": "Thank you very much for this professional review. The critical comments have been addressed carefully, and responses have been given one by one.\n\n> **W1. Concerns about model heterogeneity setting.**\n\n**R1:** Thank you very much for this helpful comment. Regarding the difference between FedFD and aggregation-based methods, **federated knowledge distillation can be viewed as an add-on component to parameter aggregation methods, designed to further improve model performance.** Methods like HeteroFL just serve as base parameter aggregation frameworks, while FedFD leverages a public dataset to enhance the performance of the aggregated models. And for logit-based federated distillation methods, they also enhance the global model aggregated in a FedAvg manner. **Therefore, the performance gap between the distillation-based methods and HeteroFL can be regarded as the performance contribution of the federated distillation technique.** In our experimental setup, we choose HeteroFL as the backbone paradigm for two main reasons:\n\n(1) The channel heterogeneity addressed by HeteroFL is itself a form of model heterogeneity. In this setting, different devices deploy sub-models with similar architectures but varying sizes and computational requirements. Such scenarios are common in practice—for example, **in UAV clusters or joint ventures between subsidiaries with a trusted terminal—where they often follow their own protocol standards that define the set of model architectures to be used, rather than allowing each client to independently choose entirely different model architectures.** Therefore, there is no need to deploy entirely different model architectures that would introduce significant bias and training overhead. In contrast, adapting model based on hardware capabilities is more practical and cost-efficient.\n\n(2) As the first work to explore feature-based federated distillation, we aim to rigorously validate the effectiveness of our method. To do so, it is important to compare against existing state-of-the-art logit-based federated distillation approaches, which typically rely on parameter aggregation to ensure proper convergence. Without such aggregation, these methods tend to struggle since distillation is insufficient to effectively guide the global model aggregation in the early stages of training, it may lead to unstable convergence or even prevent the model from converging properly, resulting in suboptimal performance.. **Therefore, to ensure a fair comparison, we adopt HeteroFL as the backbone paradigm for model aggregation in our experiments.**\n\nWe think that referring to **sub-model partitioning** as a form of **model-heterogeneity** FL is reasonable, as the local models on different clients are indeed different. Currently, most works that adopt the HeteroFL paradigm also use titles referring to it as model-heterogeneous federated learning. However, when the architectural differences between local models are extremely large—for example, completely different networks such as ResNet and CNN—it may be more appropriate to categorize such a setting as a **model-agnostic** FL paradigm. This conceptual distinction currently exists in the field of federated learning but has not yet been strictly defined or clearly separated.\n\n> **W2. Detailed explanation of orthogonal projection technique.**\n\n**R2:** Thank you for this valuable comment. The core contribution of the paper is to firsts introduce a novel feature-based distillation framework, FedFD, for model-heterogeneous federated learning. Instead of relying on traditional logit distillation, which fails to capture the internal representation differences across diverse client models, FedFD distills knowledge at the feature level. It first groups client models by architecture and aggregates their feature representations into clusters. For each cluster, the server maintains a projection layer to align the global model’s features with the client-derived ones. This hierarchical feature alignment enables the server to extract structurally coherent knowledge from clients with varying architectures, enhancing generalization without requiring structural uniformity.\n\nA key innovation in FedFD is the use of orthogonal projections to construct these projection layers. **To ensure effective and conflict-free alignment of feature spaces**, we use the skew-symmetric matrix $\\textbf{W}_d$ to obtain the orthogonal project layer parameters $\\mathcal{M}_d$ by truncating the column vectors of exp($\\textbf{W}_d$). Firstly, the parameters of $\\text{W}_d$ are randomly initialized. Then, the client updates $\\textbf{W}_d$ through back-propagation with Eq.(9). The exponential map of this matrix $\\exp(\\textbf{W}_d)$ produces an orthogonal matrix with desirable mathematical properties, such as preserving vector norms and ensuring rotational transformation without distortion. **Because feature dimensions across clients may differ**, only a subset of columns from $\\exp(\\textbf{W}_d)$ is retained to match the dimensionality of the target space. **This approach ensures each projection layer maps global model features into independent, non-overlapping subspaces, thus avoiding feature conflicts.** \n\nThe server then updates both the global model and the projection layers by minimizing KL divergence between the projected global features and the aggregated client features. This orthogonal projection mechanism is efficient, scalable, and leads to more stable and accurate model training.\n\n> **W3. Differences and specific contributions compared to VKD.**\n\n**R3:** Thank you for raising this concern. We cited VKD [28] in our paper as a related work but did not conduct an in-depth comparison, as the two approaches differ significantly in terms of task setting, technical motivation, and methodological details. Specifically, orthogonal projection is a commonly used technique across various domains, including federated continual learning and machine unlearning. Methods such as the Cayley transform and Gram-Schmidt method are well-studied and are not novel problems now. **Neither VKD nor FedFD aims to innovate on the computational efficiency of these projection methods, as this has already been extensively studied in prior knowledge distillation and mathematical literature.** In FedFD, we simply adopt a computationally convenient orthogonalization technique from the perspective of federated learning to achieve parameter orthogonalization and accelerate the experiments. **Notably, the computational formulations in FedFD and VKD are also different.**\n\nFurthermore, **VKD focuses on preserving intra-batch feature similarity** through orthogonal projection. It analyzes how orthogonal projection can better preserve similar features and further enhances feature quality using standardized normalization. In contrast, FedFD addresses a different challenge: under model heterogeneity, each client should contribute as a teacher model to distill the global model. In knowledge distillation, a larger number of distillation samples typically leads to better effectiveness, as they better represent the knowledge embedded in the training data. This can result in a full-rank knowledge matrix, making the projected knowledge also full-rank, which behaves like a non-linear mapping. Consequently, the global model may fail to properly interpret the knowledge, leading to conflicts. To address this, **FedFD is designed to use orthogonal projection to map knowledge from different projection layers into orthogonal subspaces, thereby preventing knowledge conflict and maximizing knowledge transfer.** **FedFD does not include a normalization module, as normalization is not the primary challenge our work aims to address.**\n\nLastly, in terms of technical implementation: each projection in FedFD is **column-orthonormal** to avoid knowledge conflict, whereas VKD uses **row-orthonormal** projections for orthogonal reparameterization. VKD also users the logit distillation to enhance the performance which is not used in FedFD. FedFD focuses on federated distillation under model heterogeneity, which is a critical yet underexplored direction in distributed learning. Rather than solely aiming for performance improvement as in traditional ML algorithm research, **our work identifies specific challenges brought by heterogeneous models and proposes a novel solution that combines hierarchical feature aggregation and orthogonal projection.** These components not only address practical issues in knowledge alignment and conflict but also contribute methodological innovations to the field.\n\n**Why suitable:** VKD is designed for centralized knowledge distillation involving a single teacher model and a single student model. It cannot be directly applied to distributed settings, where the teacher models are constructed by aggregating outputs from multiple client models. Moreover, VKD does not address the problem of feature knowledge conflicts among different teacher models, which is a key challenge in federated settings. In summary, VKD does not consider the specific challenges present in distributed learning scenarios, and there is no clear motivation to adopt VKD in such contexts. VKD and FedFD focus on fundamentally different problems, with only superficial methodological similarities but entirely different motivations and application goals.\n\n**PS: We have carefully addressed the concerns you raised in our rebuttal. If there are any remaining questions or clarifications needed, please feel free to point them out during the discussion phase. We sincerely hope that our work will receive your recognition following the rebuttal and discussion. We are looking forward to hearing your feedback soon!**"}, {"review_id": "JeBKtUFm9i", "reviewer": "Reviewer_bfr5", "summary": "This paper introduces a feature distillation method to enhance model-heterogeneous federated learning by addressing the limitations of traditional logit distillation. FedFD employs orthogonal projection to align feature representations from diverse client-side models. The proposed method is evaluated on multiple datasets, demonstrating superior performance compared to state-of-the-art baselines.", "questions": "How does FedFD perform in real-world applications, such as healthcare or autonomous driving, where data distributions and model architectures can be highly heterogeneous?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is well-written and easy to follow with open-source code.\n\n2. The paper proposes a novel feature distillation method, which can effectively addresses the challenge of aggregating knowledge from heterogeneous models in FL.\n\n3. The use of orthogonal projection to align feature representations is a unique and theoretically sound technique.\n\n4. The authors conduct extensive experiments on multiple datasets and various model architectures, providing strong empirical evidence for the effectiveness of FedFD.\n\nMajor Weaknesses:\n\n1. Although FedFD maintains privacy by not sharing raw data, the use of feature representations and projection layers could introduce new privacy risks that are not thoroughly discussed.\n\n2. The introduction of orthogonal projection layers and hierarchical feature alignment may increase the complexity of implementation and training, potentially limiting practical deployment.\n\nMinor Weaknesses:\n\n1. The scalability of FedFD to extremely large models, such as those with billions of parameters, is not discussed, which could be a limitation in practical applications.\n\n2. The paper could benefit from a more detailed comparison with data-free distillation methods, which are gaining traction in federated learning due to their ability to work without access to client data.\n\n3. The assumption that feature dimensions can be aligned using orthogonal projections may not hold in all cases, especially when client-side models have vastly different architectures.", "was_revised": true, "final_justification": "The author has addressed all my concerns. I would support this paper", "rebuttal": "Thank you for providing such valuable comments. We have carefully addressed these critical comments, and responses have been given one by one.\n\n> **W1. Concerns about potential privacy risk by using two modules in FedFD.** \n\n**R1:** Thank you for this valuable comment. We would like to clarify that the **projection layers** in FedFD are deployed and stored entirely on the **server side**, and do not require transmission between clients and the server. As a result, they do not introduce any additional privacy risks. Regarding the **feature representations** used for distillation, it is important to note that, similar to traditional logit-based distillation, information such as logits also needs to be transmitted. Both logits and features reflect the model’s behavior on the **public dataset**, which is assumed to carry **no privacy risks**, as it contains no sensitive or user-specific data. Moreover, these features and logits are generated by running the distillation data through client models, which are already uploaded to the server for parameter aggregation as part of standard federated learning procedures. Therefore, **FedFD does not introduce any additional privacy leakage risks compared to FedAvg**.\n\n> **W2. Concerns about training cost of two modules in FedFD.** \n\n**R2:** Thank you for raising this point. We would like to clarify that the two modules introduced in FedFD will not limit the practical deployment. First, the **hierarchical feature alignment** module is essentially a feature aggregation strategy that does not introduce additional trainable parameters. Although it aggregates multiple feature representations, these are stored and processed on the **server side**, not the clients. In the context of federated learning, it is standard practice to focus on client-side overhead, as servers are generally assumed to have sufficient computational and storage capacity. Second, while the **orthogonal projection** module does introduce new parameters, these computations are also conducted **entirely on the server**. Therefore, this module does not increase the resource requirements for client devices. Overall, both modules are designed to keep the client-side lightweight, and thus do not hinder deployment on edge devices.\n\n> **W3. Concerns about scalability of large models.** \n\n**R3:** Thank you for this insightful comment. Deploying models with billions of parameters pose scalability challenges, especially given the resource constraints of edge devices in federated learning scenarios. Addressing such large-scale models is beyond the primary scope of our current work. There is already a growing body of research specifically focusing on combining federated learning with LLMs, which tackles these challenges in depth. Due to the limited rebuttal time and the high computational cost involved, we are unable to conduct large-scale experiments involving billion-parameter models at this stage. We acknowledge the importance of this direction and will consider extending our framework to support such models in the final version of the paper, as well as exploring it more thoroughly as part of our future work.\n\n> **W4. Concerns about generated proxy data.** \n\n**R4:** Thank you for this valuable comment. To validate this, we conduct additional experiments where we employ the three data-free methods to synthesize distillation data and evaluate the effectiveness of FedFD. Unlike logit-based distillation, even though the quality of synthetic data may not match that of assumed high-quality public datasets, FedFD leverages feature-level outputs to capture richer knowledge, resulting in no significant performance degradation.\n\n| Methods          | Metrics | CIFAR-10 ($\\alpha=1.0$) | CIFAR-100 ($\\alpha=1.0$) |\n| ---------------- | ------- | ----------------------- | ------------------------ |\n| **FedFD-Public** | Local   | 78.03                   | 52.24                    |\n|                  | Global  | 89.64                   | 60.86                    |\n| FedFD-FedGen     | Local   | 77.12                   | 50.93                    |\n|                  | Global  | 87.99                   | 58.91                    |\n| FedFD-FedFTG     | Local   | 77.04                   | 51.35                    |\n|                  | Global  | 88.11                   | 59.14                    |\n| FedFD-DaFKD      | Local   | 77.58                   | 51.45                    |\n|                  | Global  | 88.97                   | 60.02                    |\n\nFurthermore, to ensure fair comparison in our experiments, the public dataset used for distillation is deliberately chosen to have distributional differences from the training dataset. Specifically, we use {CIFAR-100, Tiny-ImageNet, ImageNet} as the distillation datasets for {CIFAR-10, CIFAR-100, Tiny-ImageNet}.\n\n> **W5. Concerns about model architectures.**\n\n**R5:** Thank you for your careful review. In Table 6 of our original paper, we have already conducted experiments to address the reviewer's concern regarding models of varying architecture (e.g., ResNet, CNN). These experiments evaluate the performance of our method alongside the model-agnostic approach FedMD, demonstrating the superiority of feature-based distillation under heterogeneous model settings. \n\nNevertheless, we primarily adopt the sub-model partitioning approach because it reflects a common and practical scenario in real-world federated learning deployments. For example, **in UAV clusters or joint ventures between subsidiaries with a trusted terminal—where they often follow their own protocol standards that define the set of model architectures to be used, rather than allowing each client to independently choose entirely different model architectures.** Therefore, there is no need to deploy entirely different model architectures that would introduce significant bias and training overhead. In contrast, adapting model based on hardware capabilities is more practical and cost-efficient.\n\n> **Q1. Real-world applications suitable for FedFD.**\n\n**A1:** Thanks for this question. As we mentioned in **R5**, UAV clusters represent a realistic and well-suited deployment scenario for FedFD. In UAV-based sensing tasks, lightweight models are typically required for on-device deployment, and due to standardized protocols, it is uncommon to deploy models with drastically different architectures across devices. This makes the sub-model partitioning setting particularly applicable. Moreover, data collected in different deployment scenarios often exhibit a certain degree of heterogeneity, while public data can be readily collected on the server side. This combination makes UAV-based applications a representative and practical use case for heterogeneous federated learning."}]

Author rebuttal:

Meta-review: This paper proposes a feature-based ensemble federated knowledge distillation paradigm to compensate for the knowledge bias arising from heterogeneous models in federated learning. The main technique include introducing orthogonal projection technique to re-parameterize the projection layer to mitigate the knowledge bias issue. Overall, the propose technique is solid and proved to be effective via numerous experiments.

The reviewers raised a number of questions regarding motivation, experimental design and results, lacking of theoretical analysis, and practical deployment of the method. The authors address most concerns by providing more clarifications and experimental results in the rebuttal, which most reviewers are satisfied with. There are a few concerns remained after the rebuttal, including the reasonability of using high-quality public data for training and the lack of adopting data-free methods to avoid the use of public data. While I agree these issues constitute the weakness of the paper, they are also partially resolved in the rebuttal. Two strong-supported reviewers think using public data is widely adopted and should not constitute the main weakness of the paper. Also, I believe the authors have also included supplementary validation experiments in the rebuttal to demonstrate applicability of integrating the method with existing data-free techniques, which somehow addresses the lack of adopting data-free methods issue.

Overall, this paper receives 2 strong supports and 2 borderline reviews. Given the solid method and experimental results, I propose accept of the paper, but request the authors to incorporate the additional clarifications and experimental results in the rebuttal to the final revision.
Decision: Accept (poster)