Title: Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Model-Heterogeneity
Abstract: Model-Heterogeneous Federated Learning (Hetero-FL) has attracted growing attention for its ability to aggregate knowledge from heterogeneous models while keeping private data locally. To better aggregate knowledge from clients, ensemble distillation, as a widely used and effective technique, is often employed after global aggregation to enhance the performance of the global model. However, simply combining Hetero-FL and ensemble distillation does not always yield promising results and can make the training process unstable. The reason is that existing methods primarily focus on logit distillation, which, while being model-agnostic with softmax predictions, fails to compensate for the knowledge bias arising from heterogeneous models.

To tackle this challenge, we propose a stable and efficient Feature Distillation for model-heterogeneous Federated learning, dubbed FedFD, that can incorporate aligned feature information via orthogonal projection to integrate knowledge from heterogeneous models better. Specifically, a new feature-based ensemble federated knowledge distillation paradigm is proposed. The global model on the server needs to maintain a projection layer for each client-side model architecture to align the features separately. Orthogonal techniques are employed to re-parameterize the projection layer to mitigate knowledge bias from heterogeneous models and thus maximize the distilled knowledge. Extensive experiments show that FedFD achieves superior performance compared to state-of-the-art methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a feature distillation method to enhance model-heterogeneous federated learning by addressing the limitations of traditional logit distillation. FedFD employs orthogonal projection to align feature representations from diverse client-side models. The proposed method is evaluated on multiple datasets, demonstrating superior performance compared to state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper is well-written and easy to follow with open-source code.\n\n2. The paper proposes a novel feature distillation method, which can effectively addresses the challenge of aggregating knowledge from heterogeneous models in FL.\n\n3. The use of orthogonal projection to align feature representations is a unique and theoretically sound technique.\n\n4. The authors conduct extensive experiments on multiple datasets and various model architectures, providing strong empirical evidence for the effectiveness of FedFD.\n\nMajor Weaknesses:\n\n1. Although FedFD maintains privacy by not sharing raw data, the use of feature representations and projection layers could introduce new privacy risks that are not thoroughly discussed.\n\n2. The introduction of orthogonal projection layers and hierarchical feature alignment may increase the complexity of implementation and training, potentially limiting practical deployment.\n\nMinor Weaknesses:\n\n1. The scalability of FedFD to extremely large models, such as those with billions of parameters, is not discussed, which could be a limitation in practical applications.\n\n2. The paper could benefit from a more detailed comparison with data-free distillation methods, which are gaining traction in federated learning due to their ability to work without access to client data.\n\n3. The assumption that feature dimensions can be aligned using orthogonal projections may not hold in all cases, especially when client-side models have vastly different architectures.\n\nQUESTIONS:\nHow does FedFD perform in real-world applications, such as healthcare or autonomous driving, where data distributions and model architectures can be highly heterogeneous?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed framework FedFD focuses on federated learning for edge clients with heterogeneous model architectures. To address the limitations of traditional logits aggregation, FedFD utilizes orthogonal projection to ensure linear transformations of features, preventing knowledge conflicts and preserving feature shapes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The proposed feature distillation technique, FedFD, tackles the challenges of model heterogeneity in federated learning by using orthogonal projection to align feature representations.\n\n2.FedFD is designed to be easily integrated into existing federated learning systems.\n\n3.The experiments are solid.\n\nWeaknesses:\n\n1.A potential weakness is that the HeteroFL setup used in this work is not a typical distillation-based heterogeneous federated learning configuration. This setup leans more towards federated learning based on sub-model partitioning, making it difficult to distinguish whether the final results stem from distillation or model aggregation. The authors discuss this issue in Tab. 6, but a more detailed explanation is still needed for this case study.\n\n2.The author's core contribution to this paper, distillation using orthogonal projections, requires a more detailed explanation to enhance readability.\n\n3.The distillation method proposed by the authors bears similarities to \"VKD: Improving Knowledge Distillation Using Orthogonal Projections.\" VKD also focuses on the computational overhead issues of the Cayley transformation and Gram-Schmidt method and proposes an improved feature distillation approach. The authors need to further clarify the differences between their proposed distillation method and VKD. Additionally, the authors should provide a clearer explanation of why their proposed distillation method is more suitable for distributed environments compared to VKD.\n\nQUESTIONS:\nPlease refer to the Weaknesses section for specific limitations and suggestions for improvement.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes FedFD, a feature distillation framework for model-heterogeneous federated learning with feature alignment and orthogonal projection to mitigate knowledge bias across clients. Evaluated on CIFAR-10/100 and Tiny-ImageNet under varying data heterogeneity, FedFD outperforms all SOTA methods and demonstrates faster convergence and robustness.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper is well-written and easy to follow. 

2. The presentation is quite good and the mathematical formulation is rigorous.

3. The proposed method is a technically sound solution to mitigate knowledge bias across heterogeneous models.

4. Extensive experiments have been done to verify the effectiveness in various settings. 

Weaknesses:

1. FedFD relies on a proxy dataset like prior FD works. While mentioned as flexible, experiments only use additional real data but no tests with generated data.

2. Hetero-FL baselines are logit-based. Comparisons to feature-distillation FL methods or personalized FL would strengthen claims.

3. Maintaining projection layers per client architecture could increase server storage or computation budget. I suggest that authors analyze and propose a solution to this issue.

QUESTIONS:
1. For data-free deployment, how would FedFD integrate with generator-based methods? Would feature distillation require adjustments?

2. In Figure 3, logit distillation destabilizes Hetero-FL training. Does feature distillation alone explain FedFD’s stability, or is orthogonal projection the key factor?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a model-heterogeneous FL framework, FedFD, that improves information integration between different models through feature distillation. Compared to existing methods that rely on logit distillation, FedFD effectively compensates for knowledge bias from heterogeneous models. Furthermore, FedFD incorporates aligned feature information via orthogonal projection to maximize the distilled knowledge. Extensive experiments demonstrate that FedFD achieves superior performance compared to state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**：\n\n1. This paper addresses a key challenge in real-world FL deployment - model heterogeneity. Effectively supporting model heterogeneity among participating devices enables broader deployment of federated learning across devices with varying resources.\n2. The paper is well-written with clear logical organization throughout. Additionally, the authors present the proposed algorithm with clear illustrations and well-structured pseudo-code, making it easy for readers to understand the methodology.\n3. The authors perform extensive experiments to demonstrate FedFD effectiveness, including various Non-IID scenarios (α=0.1, 1.0, 10.0), multiple datasets (CIFAR10, CIFAR100, Tiny-ImageNet), and comprehensive ablation studies.\n\n\n\n**Weaknesses**：\n\n1. The primary issue with the method's design is how to obtain high-quality public datasets. Several references demonstrate that the correlation between public datasets and local datasets on participating devices significantly impacts model performance[1], and obtaining such datasets is extremely challenging. Additionally, acquiring these datasets to some extent compromises the privacy of users' local data.\n2. There's a mismatch between the paper's motivation and experiments. The main purpose of model heterogeneity is to enable devices with different computational resources to participate effectively. Typically, models of varying complexity and architecture (e.g., ResNet, CNN) would be deployed on these devices, but FedFD's evaluation shows very low model heterogeneity, with differences only in channel numbers. This fails to effectively validate the proposed method's effectiveness. Furthermore, some existing methods like HeteroFL and FedRolex have already implemented similar approaches, requiring further discussion on the differences.\n3. FedFD lacks theoretical analysis to prove its effectiveness. Additionally, the experimental section uses overly simple models (like ResNet-18) and datasets (like CIFAR). Testing FedFD on a broader range of datasets is necessary.\n4. There are inconsistencies in font sizes, for example in lines 270 and 271.\n\n\n\n[1]FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models\n\nQUESTIONS:\n1. The paper needs to address the challenge of obtaining high-quality public datasets that correlate well with local data. The authors should elaborate on how this requirement might impact privacy considerations in federated learning settings and propose strategies to mitigate these concerns. Additionally, the authors should discuss whether their method would still be effective in scenarios where no public datasets are available.\n2. While the paper aims to address model heterogeneity in FL, the experiments only consider variations in channel numbers rather than truly heterogeneous model architectures (e.g., ResNet vs. CNN vs. MobileNet). The authors should demonstrate how their method would perform with more significant architectural differences. Additionally, for channel heterogeneity specifically, HeteroFL appears to be a more suitable approach. The authors need to clarify the key distinctions between FedFD and existing approaches like HeteroFL and FedRolex[2].\n3. The paper would benefit from theoretical analysis supporting the effectiveness of FedFD, including theoretical guarantees or convergence analysis. The experiments are limited to relatively simple datasets like CIFAR, and the authors should consider testing on more complex, real-world datasets to demonstrate broader applicability.\n4. The formatting inconsistencies in the paper (e.g., font size variations in lines 270-271) should be addressed in a revised version.\n\n[2] FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction\n\nLIMITATIONS:\nThe authors could further strengthen limitation section by briefly discussing potential privacy or fairness implications that might arise from their approach, especially concerning the requirement for public datasets, and perhaps outline initial thoughts on how they plan to address the LLM challenges in future work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}