Title: DKDR: Dynamic Knowledge Distillation for Reliability in Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Knowledge Distillation
Abstract: Federated Learning (FL) has demonstrated a promising future in privacy-friendly collaboration but it faces the data heterogeneity problem. Knowledge Distillation (KD) can serve as an effective method to address this issue. However, challenges arise from the unreliability of existing distillation methods in multi-domain scenarios. 
Prevalent distillation solutions primarily aim to fit the distributions of the global model directly by minimizing forward Kullback-Leibler divergence (KLD). This results in significant bias when the outputs of the global model are multi-peaked, which indicates the unreliability of the distillation pathway. Meanwhile, cross-domain update conflicts can notably reduce the accuracy of the global model (teacher model) in certain domains, reflecting the unreliability of the teacher model in these domains. 
In this work, we propose DKDR (Dynamic Knowledge Distillation for Reliability in Federated Learning), which dynamically assigns weights to forward and reverse KLD based on knowledge discrepancies. This enables clients to fit the outputs from the teacher precisely. Moreover, we use knowledge decoupling to identify domain experts, thus clients can acquire reliable domain knowledge from experts. Empirical results from single-domain and multi-domain image classification tasks demonstrate the effectiveness of the proposed method and the efficiency of its key modules. The code is available at https://github.com/YueyangYuan/DKDR.


Peer reviews:
comments: SUMMARY:
To address the issue of unreliability in knowledge distillation within federated learning, this paper proposes DKDR, a federated learning framework with two key innovations: dynamically weighting forward/reverse KLD and knowledge decoupling for domain-experts identification. Extensive experiments across various federated settings demonstrate DKDR’s superior performance over SOTA methods.

STRENGTHS AND WEAKNESSES:
**Strengths:**
- The proposed method of federated KD appears interesting and innovative.
- The paper presents a clear and concise writing style thus easy to follow.
- The dynamic weighting of forward and reverse KLD is a novel approach to mitigate distillation bias with both theoretical and empirical validation.
- The analysis of forward vs. reverse KLD behaviors provides a clear foundation for the method’s design.

**Weaknesses:**
- SVD and clustering add computational overhead, potentially limiting scalability on resource-constrained devices.
- The method assumes distinct domains, which may falter with significant overlap.
- The work lacks analysis of model performance with different local training epochs for each communication round.

QUESTIONS:
The authors mention that prevalent distillation solutions primarily minimize forward KLD to fit global model. Can you elaborate on why it results in significant distillation bias?

LIMITATIONS:
See weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-16

comments: SUMMARY:
The paper presents DKDR. It dynamically weights forward and reverse KLD for reliable distillation and decouples knowledge to get reliable teachers. Experiments on image classification tasks show its superiority.

STRENGTHS AND WEAKNESSES:
Strengths:
S1.	Innovative Approach: The dynamic weighting of forward and reverse KLD based on knowledge discrepancies is a creative and effective solution to mitigate distillation bias in multi-domain FL, as supported by theoretical and empirical analysis.
S2.	Easy to follow: The paper is well-written and easy to follow. Moreover, the precise description of modules with detailed exquisite framework visualization makes the overall work clear.
S3.	Well-motivated: The proposed method appears well-motivated and effectively addresses the identified challenges in federated learning.
Weaknesses:
W1.	Computational Overhead: The use of SVD and clustering introduces additional computational complexity, which may be burdensome for resource-constrained clients, though this is acknowledged in the limitations.
W2.	Limited Heterogeneity Extremes: The paper does not explore performance under extreme data heterogeneity, which could further demonstrate robustness.
W3.	Scalability Concerns: Experiments are limited to 20 clients; scalability with hundreds or thousands of clients, common in real-world FL, is not addressed.

QUESTIONS:
Q1.	Extreme Heterogeneity: How does DKDR perform under more severe data heterogeneity? Could the authors provide additional experiments to explore this?
Q2.	Scalability: How does the method scale with a larger number of clients? Are there optimizations to handle such scenarios efficiently?

LIMITATIONS:
See weaknesses above.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed the DKDR (Dynamic Knowledge Distillation for Reliability) framework to address the reliability issue of knowledge distillation in federated learning. The author pointed out two key issues in existing federated knowledge distillation methods, namely the unreliability of the distillation path and the unreliability of the teacher model. To address the two issues, the DKDR framework contains two core components. The first is the dynamic knowledge distillation (DKD) module, which divides knowledge into dominant knowledge components (DKCs) and auxiliary knowledge components (AKCs). At the same time, the weights of the forward and reverse KL divergences are dynamically adjusted according to the knowledge differences in different knowledge modules, thereby achieving more accurate knowledge fitting. The second is the knowledge decoupling (KDP) module, which decomposes knowledge into shared knowledge and unique knowledge. Then, SVD filtering and clustering techniques are used to identify domain experts, enabling clients to acquire knowledge from reliable domain experts rather than global models. The authors conducted comprehensive experimental verification on multiple datasets to demonstrate the effectiveness of the method. In addition, ablation studies confirmed the effectiveness of the two core components, and hyperparameter sensitivity analysis verified the stability of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\na)\tThe paper proposes a technically sound two-component framework, which are the Dynamic Knowledge Distillation (DKD) and Knowledge Decoupling (KDP) modules. The experimental verification is comprehensive, covering single-domain and multi-domain scenarios, detailed ablation studies and hyperparameter sensitivity analysis.\nb)\tA dynamic KLD weight adjustment mechanism based on knowledge differences is proposed. This adaptive method is different from the existing static knowledge distillation method. Although the individual technical components (SVD, clustering, KL divergence) are not new, their integrated application to the federated KD reliability problem has technical contributions.\nc)\tThe motivation of this paper is clearly stated, and Figure 1 effectively illustrates the two core issues of path unreliability and teacher unreliability. The method description is highly structured and logically fluent, from the basic definition to the DKD theoretical analysis and then the KDP algorithm steps.\n\nWeaknesses\na)\tThe method is mainly a combination of existing technologies rather than a fundamental innovation. The dynamic weight mechanism is essentially a combination of heuristic forward/backward KLD based on a simple knowledge difference measure, SVD denoising and clustering to identify domain experts are all standard machine learning techniques.\nb)\tThere is a lack of convergence guarantee for the dynamic weight mechanism, and the superiority of the proposed weight strategy over the fixed weight strategy has not been proven. At the same time, there is a lack of necessary theoretical analysis of convergence and convergence rates.\nc)\tThere are major flaws in the baseline of experimental design. The most recent method is from 2022 and lacks comparison with SOTA methods.\nd)\tThe number of clients in the experiment (N=20) is relatively small, therefore, the effectiveness of large-scale scenarios has not been fully verified.\ne)\tThere is a complete lack of computational overhead evaluation, which is particularly important for FL methods based on knowledge distillation. SVD decomposition, clustering operations, and dynamic weight calculations introduce significant computational overhead, which may make the method infeasible on resource-constrained devices.\n\nQUESTIONS:\na)\tCan the convergence and effectiveness of DKDR be theoretically proven?\nb)\tCan you compare DKDR with the SOTA method in 2024 to further prove the effectiveness of your method?\nc)\tCan you demonstrate the effectiveness of your approach to a larger and more varied number of clients?\nd)\tYou only used the EMNIST dataset in the Comparisons between Condition Options section. Can you verify this on a more complex dataset?\ne)\tCan we compare the proposed method with the baseline method in terms of computational cost and communication time?\nf)\tThe paper only uses a simple set of experimental data to prove that FINCH is more suitable than Kmeans and DBSCAN. Can this problem be proved theoretically and in details?\n\nLIMITATIONS:\nThe authors discussed some limitations in the \"Limitations\" section. However, the main limitation of the paper is the weak theoretical foundation and the limitations of the experimental setting.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the critical issue of data heterogeneity in FL, particularly in multi-domain settings. It proposes DKDR, a novel framework that enhances distillation reliability through two methods: dynamic knowledge distillation, which adaptively weights forward and reverse KLD to reduce distillation bias, and knowledge decoupling, which identifies domain experts to provide reliable guidance. The approach is rigorously evaluated on single-domain and multi-domain image classification tasks to validate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- S1: The motivation is clear and good. The paper clearly identifies two underexplored issues in federated KD: pathway unreliability and teacher unreliability. The analysis of forward and reverse KLD in federated learning is novel.\n\n- S2: The dynamic KLD weighting adapts to knowledge discrepancies, offering a fresh take on mitigating distillation bias. And identifying domain experts via knowledge decoupling to enhance reliability across diverse domains is a practical innovation for FL (Sec. 3.3).\n\n- S3: The figures of problem and framework illustration are clear and detailed, and the equations and explanations are reasonable. \n\n- S4: Comprehensive experiments across diverse datasets and settings (Sec. 4) showcase the robustness and practical utility of DKDR.\n\nWeaknesses:\n\n- W1: The paper uses extensive mathematical notation without a summary table, which could hinder readability for some audiences.\n\n- W2: While FINCH clustering is justified (Sec. 3.4), hyperparameter details for baseline clustering methods (e.g., DBSCAN) are omitted, limiting reproducibility of the ablation study (Tab. 1).\n\nQUESTIONS:\n- Q1: In the KDP module, why was clustering applied to SVD-filtered differences rather than directly clustering client updates (e.g., updated parameters)? Could the authors clarify the motivation linking this design to improved domain expert identification?\n\n- Q2: The dynamic weighting in DKD relies on hyperparameter μ (Sec. 3.2). Could the authors elaborate on how μ=0.5 was determined and whether it generalizes across all datasets?\n\n- Q3: How does DKDR compare to prototype-based methods (e.g., FedProto) in terms of conceptual differences, given the apparent similarity in leveraging domain-specific knowledge?\n\n- Q4: Why do single-domain tasks and multi-domain tasks use different backbones? Can further ablation studies be provided?\n\nLIMITATIONS:\nThe paper acknowledges computational overhead from SVD and clustering in DKDR, which may pose challenges in resource-constrained settings. Additionally, the assumption of distinct domains could falter with significant domain overlap, potentially impacting clustering efficacy.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFirst, when the global model's output exhibits a multi-modal distribution, minimizing forward KL divergence leads to significant deviation. Second, the global model's accuracy drops sharply in certain domains, failing to provide high-quality guidance for clients. To address these issues, the authors propose the DKDR method.\nDKDR dynamically allocates weights between forward and reverse KLD based on knowledge discrepancies, thereby reducing distillation bias. Furthermore, it decomposes knowledge into shared and unique components, employing SVD and clustering to identify domain experts, enabling clients to learn from these experts rather than the global model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper conducts an in-depth theoretical analysis of the behaviors of forward and reverse KL divergence, explaining their advantages and disadvantages in different scenarios, and proposes a dynamic weighting method to balance these two divergences, thereby reducing distillation bias. It introduces Knowledge Decoupling (KDP) technology, which uses SVD and clustering to identify domain experts, enabling clients to learn from these experts rather than the global model, thereby improving the model's generalization capability.\n\n**Weaknesses**\n\n1、Will the computational overhead or training latency caused by dynamic weighted balancing and SVD become a bottleneck?\n\n2、How does DKDR perform on non-image datasets?\n\nQUESTIONS:\nQ1. The paper mentions that the DKDR method involves SVD and clustering techniques, which increase computational overhead and may significantly prolong training time, especially on resource-constrained clients. Therefore, it would be beneficial to provide theoretical or experimental evidence in the paper to demonstrate the training overhead and latency.\n\nQ2. While the experimental section covers multiple datasets and scenarios, it primarily focuses on image classification tasks. If applied to other tasks such as natural language processing, would the dynamic divergence balancing and knowledge decoupling still offer advantages?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}