DKDR: Dynamic Knowledge Distillation for Reliability in Federated Learning
Authors: Yueyang Yuan
Wenke Huang
Guancheng Wan
Kaiqi Guan
He Li
Mang Ye
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Knowledge Distillation
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Federated Learning (FL) has demonstrated a promising future in privacy-friendly collaboration but it faces the data heterogeneity problem. Knowledge Distillation (KD) can serve as an effective method to address this issue. However, challenges arise from the unreliability of existing distillation methods in multi-domain scenarios. 
Prevalent distillation solutions primarily aim to fit the distributions of the global model directly by minimizing forward Kullback-Leibler divergence (KLD). This results in significant bias when the outputs of the global model are multi-peaked, which indicates the unreliability of the distillation pathway. Meanwhile, cross-domain update conflicts can notably reduce the accuracy of the global model (teacher model) in certain domains, reflecting the unreliability of the teacher model in these domains. 
In this work, we propose DKDR (Dynamic Knowledge Distillation for Reliability in Federated Learning), which dynamically assigns weights to forward and reverse KLD based on knowledge discrepancies. This enables clients to fit the outputs from the teacher precisely. Moreover, we use knowledge decoupling to identify domain experts, thus clients can acquire reliable domain knowledge from experts. Empirical results from single-domain and multi-domain image classification tasks demonstrate the effectiveness of the proposed method and the efficiency of its key modules. The code is available at https://github.com/YueyangYuan/DKDR.

DKDR: Dynamic Knowledge Distillation for Reliability in Federated Learning
Yueyang Yuan 1.2f Wenke Huang 122, Guancheng Wan Kaiqi Guan', He Li', Mang Yel
1 National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence,
Hubei Key Laboratory of Multimedia and Network Communication Engineering.
School of Computer Science, Wuhan University, Wuhan, China.
2 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ.
{yueyangyuan, wenkehuang, guanchengwan, yemang) &whhu, edu.
Abstract
Federated Learning (FL) has demonstrated a promising future in privacy-friendly collaboration but it faces the data heterogeneity problem. Knowledge Distillation (KD) can serve as an effective method to address this issue. However, challenges arise from the unreliability of existing distillation methods in multi-domain sce- narios. Prevalent distillation solutions primarily aim to fit the distributions the global model directly by minimizing forward Kullback-Leibler divergence (KLD). This results in significant bias when the outputs the global model are multi-peaked, which indicates the unreliability of distillation pathway. Mean- while, cross-domain update conflicts can notably reduce the accuracy of the global model (teacher model) in certain domains, reflecting the unreliability of the teacher model in these domains. In this work, we propose DKDR (Dynamic Knowledge Distillation for Reliability in Federated Learning). which dynamically assigns weights to forward and reverse KLD based on knowledge discrepancies. This enables clients fit the outputs from the teacher precisely. Moreover, we use knowledge decoupling to identify domain experts, thus clients can acquire reliable domain knowledge from experts. Empirical results from single-domain and nolti-domain image classification tasks demonstrate the effectiveness of the proposed method and the efficiency of its key modules. The code is available btttps://ithai com www.//o///a//eeeeeee DKDR
Introduction
Federated learning a collaborative paradigm 21 60
27
14 62 enabling multiple clients to
jointly train shared global model [39
while ensuring privacy protection (52). However, the distributed data is collected from different sources with diverse preferences and brings the non- independent and identically distributed (non-lII) characteristics. Knowledge distillation [I3] 4 addresses this challenge effectively by aligning the outputs of local models with the global model. brings the optimization objectives of each client closer together thus resolving the problem.
28 15
5 typically use forward KL. to fit the distributions of the global model. We argue that this approach is unreliable in multi-domain scenarios. Given the global model distribution (z)(z) and the local model distribution Zn (y|2z parameterized by w, standard knowledge distillation objectives aim to minimize the forward KLD between them, denoted as K L J- This approach compels Z to encompass all modes of Z. However, we
However, existing distillation methods 25
Equal Contribution.
Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Typical Distillation Methods
(b) Passway Unreliability
dNLD
Aggregation
Entropy
Clokil Mote
Mull peoked
(Tachhe Modan
Webcean
0.41
A17
Wabcars
0.331
Sigiiicam Saas
Amrsso
0:45
73.0
0:77
B11
(c) Teacher Unrellability
Ddlr
Acturacy
Rital Mode
Cabal modalia
LOw
Entropy
Figure I: Problem illustration. (a) Typical Distillation Methods exist two unreliability problems as follows. (b) Pathway Unreliability: in multi-domain scenarios, the aggregated global model shows a significantly higher entropy compared to local models in the distributions over private datasets, exhibiting a multi-peat structure. In such case, minimizing forward KLD leads to significant bias; (c) Teacher Unreliability: after aggregation, the global model experiences catastrophic accuracy drops in certain domains. Thus in these areas, the global model is unable to serve an effective teacher to provide high-quality guidance for clients.
notice that the global model will produce multi- multi-peaked peaked outputs in domains with limited data, as shown in Fig I In such situations, minimizing the forward KLD causes Zw to assign unreasonably high probabilities to regions of low density in Z [38]. Therefore, distillation on these domains will introduce significant bias, which is the unreliability of the distillation pathway. This context naturally raises the following critical question: 1) how can we establish a reliable distillation pathway in federation? ? Meanwhile, conflicts in update directions across different domains can significantly reduce the accuracy of the global model on some domains. Thus, the global model is inherently less effective on these domains, which is the unreliability of the teacher model. This situation prompts another intriguing question: II) how can we get reliable teacher for distillation?
To address these challenges, we propose DKDR (Dynamic Knowledge Distillation for Reliability in Federated Learning). Concerning the issue of pathway unreliability mentioned in D we initially conduct a theoretical analysis of federated knowledge distillation (see Sec The reverse KLD, denoted as L |Z]. is widely used in knowledge distillation. Reverse KLD promotes a mode- seeking behavior, leading Zu to focus on singular mode of Z B. 53 22 while forward KLD induces mean- mean-seeking seeking behavior, encouraging Zn capture the overall distribution of Z. Therefore, when distilling the multi-peaker distributions, forward KLD will assign high probabilities to low- density regions. In contrast, the reverse KI.D prioritizes confidence intervals. Both methods exhibit different types of bias. An intuitive idea is utilizing their characteristics to design a dynamic weighting method, aiming to minimize the bias introduced by distillation pathway as much as possible. We demonstrate from both experimental and theoretical perspectives that the forward KLD prioritizes fitting the dominant regions of the global distribution, while the reverse KLD prioritizes fitting the lower-probability segments. Based on their characteristics, we introduce Dynamic Distillation: dynamically allocating weights to forward and reverse KLD based on the knowledge discrepancies between the Dominant Knowledge Components (DKCs) and Ancillary Knowledge Components (AKCs) (see Sec B.2). Thus clients can precisely fit the distributions of the teacher model.
In the second place, to get reliable teacher mentioned in IT) we propose Knowledge Decoupling to get domain experts: we first modularize knowledge into shared and unique components and then use SVD to extract the main components of unique knowledge. Subsequently, clustering techniques are employed on the refined components to identify domain experts that specialize in distinct domains. In federation, clients learn from domain experts rather than the global model, thus gaining reliable domain knowledge. Experimental results reveal that our method consistently achieves better performance than others. The main contributions summarized
Re-examining KD in FL from a Reliability Perspective. Our findings indicate that existing federated distillation methods unreliable in multi-domain scenarios, which results in significant distillation bias and less effective guidance for clients in domains with limited data.
Novel Dynamic Multi-experts Distillation Framework for Reliability. Building on the phenomenon of unreliable KD in FL, we effectively mitigate distillation bias and comprehensively improve the performance across domains by addressing the pathway unreliability and teacher unreliability. Theoretical Guarantees and Experimental Validation. We provide theoretical guarantees for our framework, and further demonstrate the effectiveness of it through comprehensive experiments.

---

2 Related Work
2.1 Heterogeneous Federated Learning
A pioneering work proposed the currently most widely used algorithm, FedAvg 39 However, it suffers from performance deterioration when applied to non-i.i.d data (data heterogenetty. Shortly thereafter, substantial body of research [29
emerged, focusing non-i.i.d data. These methods primarily address label distribution skew, where non-i.i.d data [8 is created by
46
48
28
partitioning existing data based on label space with limited domain shift. FedProx (29] FedCurv [47), PFedME [48], and FedDyn [2] calculate global parameter stiffness to control discrepancies. Besides, MOON 28), FedUFO [64]. FedProto [4)), and FedProc (42) maximize feature-level alignment of local model and global model. Moreover, SCAFFOLD 19 and FedDC [9] leverage global gradient calibration to control local drift. Nevertheless, when private data is sampled from different data domains, these works do not consider diter-domain performance. concentrating instead on learning an internal model. Recent studies have explored related issues in unsupervised domain adaptation for target domains 4 30 and domain generalization on unseen domains [4). However, collecting data in the target domain can be time-consuming and impractical, while considering performance on unknown domains represents an idealistic scenario. In more realistic settings, participants are likely to be more concerned with performance across other domains, as this could directly enhance economic benefits. Our method leverage the Knowledge Decoupling to capture domain-specific signals and identify domain experts. focuses on improving performance outer domains during distillation, learning a generalizable and stable global model during the federated learning process.
2.2 Federated Knowledge Distillation
Knowledge Distillation (KD) [13 is a technique that has been extensively studied and applied in various areas of machine learning. Currently, KD has found widespread applications in FL, which can be broadly categorized into three main areas: addressing data heterogeneity, enhancing generalization
capabilities, and mitigating catastrophic forgetting [54] 66, 58 26 61 [5 17). In terms of addressing data heterogeneity, FedFTG [65] employs data-free knowledge distillation method fine-tune the global model, while FedDKD [1 introduces a decentralized knowledge distillation module to distill knowledge from local models. Moreover, FedUSI. (6] employs self-label reassigning method to rectify the global model predictions. Regarding the enhancement of generalization capabilities, FedX [II] utilizes a two-sided knowledge distillation approach with contrastive learning as core component, enabling the federated system to operate without requiring clients to share any data features. Furthermore, FedMEKT [24] develops distillation
multimodal embedding knowledge transfer mechanism, which allows the server and clients to exchange joint multimodal embedding knowledge extracted from a multimodal proxy dataset. Finally, to address the issue of catastrophic forgetting, FedNTD (25) proposes novel and effective algorithm, Federated Not- Not-True Distillation, which preserves the global perspective on locally available data exclusively for the not-true classes. Additionally, CFeD [36] performs knowledge distillation both the clients and the server to mitigate forgetting. And DFRD 35 maintains an exponential moving average copy of the generator the server overcome the catastrophic forgetting, using dynamic weighting and label sampling to accurately extract knowledge. is worth noting that all of these methods distill knowledge directly by utilizing forward KLD, resulting in significant bias in multi-domain scenarios. In our work, we firstly introduce Dynamic Distillation, which dynamically weights forward and reverse KLD for different knowledge modules and establishes precise distillation.
Methodology
3.1 Preliminary
Generic Federated Learning. In general federated learning settings 29 41 59 15 there are K clients (indexed by k) each possessing its respective private data, denoted as Dj- = (zi, We J.N, where N₁ represents the number of data points held by the ttth client. The global model parameters at the beginning of the tth communication epoch are denoted as w². The server broadcasts these parameters to each client, assigning them as w t w'. Each client conducts local optimization and
3

---

uploads the updated parameters back to the server for weighted parameter aggregation:
wi
<- wi
v
1(wi. Et), ie
Σ
w
Σ Cariy
(1)
here, the B5 denotes the mini-batch sampled from the private data DE. & represents the query instance, and 1j indicates the local learning rate. The optimization objective to secure well-performing global model through the federated learning process.
Domain shift. There exists domain shift among private data. Specifically, for the same label space, distinctive feature distributions exists among different participants, which can be defined as:
Pa(xl))) Py Pr(u)= PP)))-
(2)
Federated Knowledge Distillation. In typical federated knowledge distillation settings 25
14]
clients use forward KLD distill knowledge from the global model. Specifically, the global model w/-1 at the end of the (t - 1)th round involves the knowledge learned from other participants. We calculate the distribution through the htth client model and global model of the (4 - 1)th round on private data: 7 = fwf,, m.) and 7 f(w""1, xx;) for private data 24 w.r.t its ground truth
label ys. The standard KD loss function of tt client can be formulated as:
OZ
( (z - o oZ) log(
(Zi,,)
where denotes softmax function. The optimization objective is mitigate the issues such as catastrophic forgetting and data heterogeneity problem.
KD Based reverse KLD. KD based on minimizing reverse KLD has been widely applied in
specific scenarios due to its mode-seeking characteristics 10 20 56). Unlike standard KD, its
distillation function can be expressed as:
Zin or (ZZ,,) ((.x))log log( (Zi,))
(4)
(Z!-1)
Research 10] suggests that due its mode-seeking characteristics, this distillation method is more suitable for complex tasks compared to standard distillation methods.
3.2 Dynamic Knowledge Distillation (DKD)
Definition 3.1. (Knowledge Modules) We take digits in 2 in descending order and then cumulatively summed until the number of selected values surpasses where is hyperparameter and typically defined as 0.5. The selected values are defined as Dominant Knowledge Components (DKCs), while the remaining values are termed Ancillary Knowledge Components (AKCs), formulate
if2 E AKCs
a(i)
(5a)
if DKCs
min
a(j)
s.t. Eabisf 2p.
(5b)
f=0
3=0
Definition 3.2. (Knowledge Diserepancy) Knowledge discrepancy reflects the distance between two distributions. The knowledge discrepancy Is within AKCs and nu within DKCs defined
7
a(i))=))- 2/,ab
(6a)
20 a())2 2/,a1.
(6b)
Theoretical Analysis. Forward KLD's mean-seeking characteristics result in unreliable distillation when the global model has multi-peaked distributions. Conversely, reverse KLD will also lead to distinct bias due to its mode-seeking nature. Forward KLD and reverse KLD are adept at fitting different regions of the distribution. Therefore, how to balance forward and reverse KLD to minimize distilation bias becomes key issue. This naturally leads us to reflect on the fundamental reasons
behind the different behaviors of forward and reverse KLD. Let Z = (zf, 2 =f{x) and z =21-1 .222 1-1
..... 21-1), where denotes the size of Z. The Lokd and Cysa can be denoted as:

---

Likd Σ 2/-
log
(7a)
j
Lika Σ
23 log(
(7b)
it
The gradient for ej,, under forward and reverse KLD can be calculated by the chain rule as follows:
0Cakd 0zj,R
=2 8/j
(8a)
üCrkd
=2j a log(
Lrkd.
(8b)
Rj
Considering the converge condition of forward and reverse KLD:
Vj 21..,,.., n).
(9)
d
we can infer that for both two methods, the sufficient and necessary condition for converge is:
= = Vjj E (1, 2,.,,....11
(10)
According to Eq.(TO) both the forward and reverse KLD have the same optimal projective. Thus, the fundamental reason for their differing behaviors the optimization process. Considering Eq. 7ap.
larger 2) means larger weight in total loss and also more likely generate larger log( (2)
Hence, fitting the area with larger e the priority of forward KLD. What's more, when (2)
goes to +00 the forward KLD goes to +oc. Therefore, 2/,, would try cover many peaks of as possible, leading to the mean-seeking behavior of forward KLD. Similarly, considering Eq.(7b) (2),a/23-) is easier to be +oo when 23 gets smaller, leading to a larger loss. Therefore, fitting the area with smaller 2/-1 the priority of reverse KLD. It avoids (2)-//)),) go to 0+, which means
= shouldn't 't be too large when 1-1 is small, leading to mode-seeking behavior of reverse KLD.
Empirical Analysis. We per- form distillation using forward and reverse KLD separately, cal- culating the average knowledge discrepancies between the client models and the global model across the two knowledge mod- ules (DKCs and AKCs, defined in (5) on Cifar- with 10 cilents for 100 communication epochs, as illustrated in Fig The experimental results indi- cate that when using forward KLD, the differences in the DKCs are lower, while when using reverse KLD, the differ- ences in the AKCs are lower.
0.350
Forward KLD
0.350
Reverse KLD
0.325
0.325
0.300
0.300
0.275
0.275
0.250
0.250
0.225
0.225
0.200
0.200
0.175
0.175
0.150
0.150
10
20
30 40
50 60
70
80
90
Communication Epoch
Figure 2: Empirical Analysis. The dashed line represents distillation based on reverse KLD, while the solid line denotes distillation based on forward KLD. For detailed information, please refer to the left main text.
intuitive assign weight to the forward and the reverse KLD according to the knowledge discrepancies between the client model and the teacher model. In cases of disparity in DKCs, prioritize the forward KLD; Conversely, when there is a significant difference in AKCs, prioritize the reverse KLD. Combined with Eq. and Eq.44) the dynamic knowledge distillation loss function Like follows:
Method. Based on these observations and analysis,
Laca (Zi Zz-1)
%
Lava L -
7
Land (2 C. -1-1)
(11)
No 7
7a +
3.3 Knowledge Decoupling (KDP)
In prior knowledge distillation approaches, the aggregated global model has catastrophic accuracy drops, resulting in poor performance in some domains. Therefore, global model can not provide reli- able guidance in these domains. We address this issue through Knowledge Decoupling: Decoupling
5

---

Client
Expert
Dynamic Knowledge Distilationn (DKO)
Knouledge Discrppncc
D
D-11
Sort Based
BMcs
on Teacher
Expert
Allgnment
L
AKCOi
Descending Order
Expert
Clien
Ciient
Client
160
Expert 20%
Agreegation
Knowlledge Decoupling (KDP)
Carresponding Client Modal
Chastering
Finch
Similanity
SVD Fibering
Model
Clobal
Eo.
Samsrr
Globaal
Figure 3: Architecture illustration of DKDR. DKDR consists of two core components: The top right box refers Dynamic Knowledge Distillation (DKD), which adaptively weights forward and reverse KLD based on knowledge discrepancies DKCs and AKCs (See 3.2). @ The bottom left represents Knowledge Decoupling (KDP), where we separate shared and unique knowledge by SVD filtering and clustering to identify domain experts (Sec 3.3 Clients will distill knowledge dynamically and equally from each expert.
knowledge into shared knowledge and unique knowledge globally, and extracting domain signals from unique knowledge by SVD. Then we use Finch Clustering to get domain experts. Thus, Clients distill knowledge from these experts more efficiently.
Specifically, to get domain experts, we first examine the knowledge distillation process finer- grained knowledge perspective. We identify two types of critical knowledge: (1) Shared knowledge, which benefits multiple domains, and (2) Unique knowledge, which is useful only for specific domain. In federated knowledge distillation, the mixing of shared knowledge and unique knowledge obscures domain-specific signals coming from unique knowledge. A natural idea is separate shared knowledge and unique knowledge clarify domain-specific signals. Therefore, during the zth communication epoch, we consider the global model from the (t - 1)th communication epoch as a natural placeholder encapsulate the shared knowledge (denoted as w w² Then we calculate the difference vector for each client w of wf w This subtraction vector preserves domain- specific signals while diminishing the interference of shared knowledge. For practical use, we apply SVD [ filter redundant noise and clarify domain-specifer signals within unique knowledge:
(vi, vl.
vk) UΣV
(12a)
L. 1.8
12
................................
"x 1.
V.
(12b)
We apply truncated SVD (ui, ut ......, vkk, retaining the top singular values extract domain- specific signals. Then we use Finch Clustering on (v).S) v..5. ...., ux.,,xx 1.S. based on cosine similarity
to capture dppeii- signals in unique knowledge. Then, we replace each u in each cluster with the corresponding client & to obtain client clusters 8r, where each St 3 corresponds domain and contains clients belonging to this domain. Next aggregate the clients within each 8 to identify domain experts f.oom The process can be formulated as:
v = [1,, 12, 13, u4, vs, M6]
I Cluster
Un, D22
Us,
Us. Vo
Domain
(13a)
I
Map to client models
W1,
Pa
W
e-1
W
dom
E.
(13b)
ONk

---

Thus clients can get rich domain knowledge from these domain experts. We calculate the logits oulput through h client model and each domain expert wadomm on private data n, w.r.t its ground truth label y:: - = fw H3) and Zz:dom = i,u
f( dom zz). By inserting 7 and zi.dom into the Eq.(I1 We get the final defined knowledge distillation loss function Lidd:
Lyke Σ 1 Li, Z(.dom))
(14)
Eq. [4) assigns each expert the same weight thus mitigating the issue of domain skew. This dynamic knowledge distillation method reliably and efficiently distills knowledge for each domain. Combined with the cross-entropy loss LCE, the local loss of client k is now defined as:
Lx (0) ( (Cox + eL
(15)
where represents the knowledge distillation intensity of the method.
3.4 Discussion and Limitation
Clustering Technical. A variety of clustering techniques
Table 1: Ablation on popular clustering
have been proposed to discover natural grouping [55] 7 5, 33 45). The well-known methods, K- K-Means [37] and DBSCAN iteratively assign points to a fixed group number. However, they are sensitive to hyper- parameter selection under different scenarios. Thus shift the gaze towards FINCH [45), which is parameter-free and thus
methods. Please refer to See 3.4 for details.
Office5
Methods
A
W
D
AVG
K-means
66.89 44.31 28.24 46.48
DBSCAN
65.28 36.82 28.76 43.62
FINCH (ours) 67.06 54.82 29.84 50.57
suitable for heterogeneous federated learning. Specifically, we leverage the cosine similarity metric to evaluate the distance between any two client weights and view the weight with minimum distance as its "neighbor". sorted into the same set. After clustering, we aggregate all clients in the same cluster as they have related domain knowledge in order to get domain experts. We compare FINCH 45 with the
clustering methods, K-Means 37 and DBSCAN g The results are shown in Tab
Conceptual Difference. Unlike conventional federated KD methods that depend solely on forward KLD and single teacher model, DKDR introduces two distinctive innovations: dynamic KLD weighting and the use of multiple domain experts. The dynamic weighting reduces distillation bias by adapting knowledge discrepancies, which is departure from the static approaches of methods like FedNTD (25) or FedX [1]. It ensures precise alignment with the target distribution across diverse domains. Furthermore, by identifying domain experts, DKDR provides tailored guidance clients, overcoming the limitation that the single teacher model may underperform in specific domains. This multi-expert paradigm enhances reliability and performance in multi-domain FL scenarios.
Limitation. While DKDR effectively enhances the reliability of distillation pathways and teacher models in multi-domain federated learning, it is not without drawbacks. The dynamic weighting mechanism involves SVD and clustering techniques. adding computational overhead that may significantly prolong training, particularly on resource-limited clients. Additionally, DKDR assumes that domains are sufficiently distinct for clustering to accurately identify experts; domains overlap significantly, this assumption falters, potentially degrading overall performance.
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate DKDR on two single-domain scenarios and two multi-domain scenarios.
Cifar-10 2X
contains 50k training images and 10k test images with 32 x2 for 10 classes.
Cifar-100 23
contains 50k and 10k images with 32 x 32 for 100 classes.
consists of three domains: Amazon (A), Webcam (W) and DSLR (D). In total, the dataset contains 4, 1 10 images with 256 x 256 across 31 categories, shared among the three domains.
Office31 44
Office Home [51] consists of four domains: Art (A), Clipart (C), Product (P), and Real World (R).
It contains 15,500 images with 256x x 256 across 65 categories, shared among the four domains.
Data Heterogeneity. As for the data heterogeneity simulation, we utilize the Dirichlet distribution, Dir(C) to simulate the label skew, as previous methods 29 28 63 imbalanced the local distribution is.
The smaller Ç is, the more
7

---

Counterparts. We compare our method with state-of-the-art (SOTA) federated knowledge distillation
and federated learning approaches: FedAvg [39. FedProx (29 FedDyn [2]. Scaffold [9], FedProto
49
MOON 28 FedNTD FedDf
32
Implement Details. We conduct communication epoch for F: 200 and local updating round T where all federated learning approaches have little or no accuracy gain with more communications. We use the SGD optimizer with the learning rate Ir le 3. The corresponding weight decay is le - 5 and momentum is 0.9. The training batch size is 64 for single-domain tasks and 16 for multi-domain tasks. The client number K is 20 for different datasets. We conduct experiments with ResNet- 10 [2] on single-domain scenarios and ResNet- 18 [12] on molti-domain scenarios. We fix the random seed ensure reproduction and conduct experiments on the NVIDIA 3090Ti.
Table 2: Comparison with the state-of-the-ard method in the Office31 1 and Office Home with domain skew. Best in bold and second with underline. Please refer to Sec 4.2 for further explanations.
Methods
Office31
Othec Home
AVG
AVG
FedAvg
48.04
32,91
22.45
34,47
39,24
61,29
74.58
38.45
58.39
FedProx
45.55, 2.4
26.58 6:33
19.39
30.51
39. u44
74.52, cmm
59
58, (8:74
FedDyn
56.58-am
20.25
23.47,
33.43;
39.26
60.21 21,
73.84_07
59.08
58. 10; 0.39
Scaffold
46.80, 1:44
14
23,47,
34.82,0a1 0:35
39 0.4
61.24-o05
74.75-017
18.8
68.6888
FedProto
51.60-
36.711
31.63
39.98
40.29
63.65 1.3
75.31
60.34:
59.90+
MOON
49.47 1.4
32.28
26.53 4.1
36.09 122
38.78, u4t
62.09-uu ume
74.27
58.90 u.45
58.51 1.
FeINTD
48.22
35,44,2.1
22.45
35.37-
39.28
63.07-177
75.20-ax
59.20,
89.18;
FedDf
27
32.78
24.72 2:37
34.25,
37.93
60.53
6.76
71.94; am
57.83
$7.06 1:3
DKDR
67.06 1903
54.82
20.01
29.84
50.57
42.68
65.99
4.70
78.22
63.03
62.48 4.0
4.2 Comparison to State-of-the-Arts
Performance Comparison
Tresent The Tab Tab and Tab present the final accuracy metric by the end
of the federated learning process with popular SOTA methods. It depicts that our method outperforms all other baselines in seven out of the eight settings, which confirms that the knowledge distillation of DKDR reliable, efficient, and possesses superior domain generalization capabilities.
Table 3: Comparison with the state-of-the-art method in the Cifar-10 with skew ratio Ç E
Table 4: Comparison with the state-of-the-ar method in the Cifar-100 with skew ratio e {0.1, 0.3, 0.5}. Please refer to S.2 for details.
{0.1, 0.3, 0.53. Please refer to Sec 4.2 for details.
Methods
Cifar 10
Methods
Cifar 100
ç 0.1
0.3
Ç 0.5
FedAvg
76.91
79.86
80.34
FedAvg
43.76
46.66
48.57
FedProx
70.43 6.48
g.72
75.11
FedProx
33.92
37.81
39.79
FedDyn
78.77 1.36
80.79 0.93
81.26 11.266.0
FedDyn
46.21 2.45
48.82
2.16
50.23 1.66
Scaffold
79.62 79.62,2.77
80.99
1.13
81.40 1.06
Scaffold
46.32 2.56
50.33, 3.67
51.76- 3.19
FedProto
78.45 1.54
80.13 0.27
81.49 1.15
FedProto
44.21 0.45
49.88 3.22
51.34- -.77
MOON
75.24 1.07
79.83 0.05
80.71 0.33
MOON
42,96 0.80
45.73 0.93
48.58 0.01
FeaNTD
77.18 0.27
80,36- 0.50
80.94 +
FedNTD
44.12
47.10
48.99
FedDF
78.53
1.62
80.24- 0.38
81.37
FedDr
45.
46.62
48.97 0.40
DKDR
79.46 2.55
81.93 2,07
82.63
2.29
DKDR
46.80 3.00
51.84
5.18
53.18 4.61
Convergence Analysis Fig/sshows the curves of the average test accuracy during the training process across three random runs of three datasets (Cifar-100 Office31. 1, Office Home) representing single- domain and multi-domain scenarios, including the results of various baselines. Traditional FL methods such as FedAvg performs poorly in heterogeneous scenarios while methods designed specifically for heterogeneous problem such Scaffold and FedProto achieve much better performance.
Cifar-
Fallyys
Communication
Communication
Communiaation
Figure Visualization of training curves of the average test accuracy of DKDR and various baselines on three datasets (Cifar- 100, Office31, Office Home). Please refer to Sec 4.2 for further explanations.

---

4.3 Sensitivity
Hyper Parameters and In the single-domain task Cifar-100 and the multi-domain task Office31, the optimal values of с and μ remain stable at 1.25 and 0.5 across different scenarios. This aligns with our theory (Sec[3.2): when l 0.5, DKD gradually degenerates into forward KLD, while when > 0.5, it gradually degenerates into reverse KLD. Both cases introduce different biases.
12.01
1890
200
R2z
380
(a) Office31 for y and e (b) Cifar- 100 for j and c
(a) Office31 for
(b) Office Home for
Figure 5: Sensitivity analysis for j and on Of-
Figure 6: Sensitivity analysis for
two multi-
fice31 100. The z-axis represents the perfor- mance improvement relative to the baselines. Please accuracy. The horizontal line represents the baseline. refer to Sec 4.3 for further detailed explanations.
domain tasks by Top-1 test Accuracy and Clustering
Please refer to S.c for further explanations.
Hyper Parameter r. In assessing the filtering strength r of SVD, we utilize the average accuracy of clustering within each domain additionally. If cluster includes clients from more than one domain, is considered clustering failure for all clients in that cluster. As shown in Fig and Fig 6b] the optimal setting of remains stable at 1 for both two datasets. It is worth noting that when too few singular values are retained, domain signals tend converge towards a common low-dimensional subspace, which can lead to the failure of clustering, just when = 0.05.
4.4 Effectiveness.
Effects of Key Components Mechanism of DKDR. To substantiate its robustness and stability, we meticulously evaluate the performance across both single-domain and multi-domain scenarios. As illustrated in Tab[5] compared to FedAVG, DKD achieves a consistent performance enhancement in both tasks, whereas KDP demonstrates more pronounced effectiveness in complex multi-domain tasks, as domain-specific experts provide reliable guidance for clients in each domain.
Ablation Study DKD. We compare DKD with KD using forward and reverse KLD separately to validate the effectiveness of DKD. As shown in Tab DKD is more suitable for federated learning tasks both single-domain and multi-domain tasks compared to using forward or reverse KLD alone.
Table 5: Ablation study of the key components Table 6: Ablation study of the DKD of four datasets (Office31, Office Home, Cifar-10, Cifar-100), Please refer to Sec|4.4 for further detailed explanations.
DKDR on four datasets (Office31, Office Home,
Cifar-10, 10, Cifar- 100). Please see Sec 4.4 for details.
DKD
KDP
Office31
Office Home
Cifar 10
Cifar 100
FKD
RKD
DKD
Office31
Office Home Cifar Cifar
x
34.47
58.39
79.86
X
34.47
58.39
79.85
46.66
/
x
36.83
59.52
80.72
48.79
/
X
x
34.88
58.70
80.59
47.52
35.67
/
50.57
62.48
81.23
$1.84
x
/
36.83
59.52
80.72
48.79
5
Conclusion
In this paper, we address two significant problems in existing federated knowledge distillation methods: the unreliability of distilling pathway and teacher model. We empirically and theoretically analyze the fundamental differences between forward and reverse KLD, which leads us to propose dynamic distillation approach that minimize distillation bias. To get reliable guidance, we employed knowledge decoupling to identify domain experts. Based on these insights, we propose the DKDR framework, which is strategically designed to achieve robust performance across diverse tasks. The effectiveness of DKDR has been validated with many methods over various classification tasks.

---

Acknowledgement
This work is supported by National Natural Science Foundation of China under Grant (62361 (62361166629, 166629, 623B2080, 62506269), the Major Project of Science and Technology Innovation of Hubei Province (2024BCA003, 2025BEA002), and the Innovative Research Group Project of Hubei Province under Grants 2024AFA017. The supercomputing system the Supercomputing Center of Wuhan University supported the numerical calculations in this paper. This research was financially supported by the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), under Grant GML-KF-24-10.
References
[1] Hervé Abdi. Singular value decomposition (svd) and generalized singular value decomposition. Encyclopedia of measurement and statistics, 907(912):44, 2007.
[2] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In ICLR, 2021.
[3] Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23(253):1-79. 2022.
[4] Huajun Chen. Large knowledge model: Perspectives and challenges. arXiv preprint arXiv:2312.02706, 2023.
[5] Yiqiang Chen, Wang Lu, Xin Jindong Wang, and Xing Xie. Metafed: Federated learn- ing among federations with cyclic knowledge distillation for personalized healtheare. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[6] Zongyi Chen, Sanchuan Guo, Liyan Shen, Xi Zhang, and Zhuonan Chang. Improving knowledge distillation for federated learning on non-iid data. In 2023 IEEE International Conference on Big Data (BigData), pages 998-607. IEEE, 2023.
[7] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. JEEE TIT, pages 21-27, 1967.
[8] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, al. A density-based algorithm for discovering clusters in large spatial databases with noise. ACM SIGKDD, pages 226-231, 1996.
[9] Liang Gao, Huazhu Fu, Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Federated learning with non-iid iid data via local drift decoupling and correction. In CVPR, 2022.
[10] Yuxian Gu, Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference Learning Representations, 2024.
[11] Sungwon Han, Sungwon Park, Fangzhao Wu, Sundong Kim, Chuhan Wu, Xing Xie, and Meeyoung Cha. Fedx: Unsupervised federated learning with cross knowledge distillation. In ECCV, 2022.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint ftth::5503025.44
[14] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. In CVPR, 2022.
[15] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In CVPR, pages 163 16312-16322, 2023.
10

---

[16] Wenke Huang, Mang Ye. Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. A federated learning for generalization, robustness, fairness: A survey and benchmark. TPAMI, 2024.
[17] Hai Jin, Dongshan Bai, Dezhong Yao, Yutong Dai, Lin Gu, Chen Yu, and Lichao Sun. Personal- ized edge intelligence via federated self-knowledge distillation. IEEE Transactions on Parallel and Distributed Systems, 34(2)5667-580, 2022.
[18] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.0483 1912.04977, 2019.
[19] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learning. In ICML, 2020.
[20] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402. 03388, 2024.
[21] Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimiza- tion: Distributed machine learning for on-device intelligence. arXiv preprint arXiv: 1610.02527, 2016.
[22] Zhiqiang Kou, Si Qin, Hailin Wang, Mingkun Xie, Shuo Chen, Yuheng Jia, Tongliang Liu, Masashi Sugiyama, and Xin Geng. Label distribution learning with biased annotations by learning multi-label representation, 2025. Main Track.
[23] A. Krizhevsky and Hinton. Learning multiple layers features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[24] Huy Le, Minh NH Nguyen, Chu Myaet Thwal, Yu Qiao, Chaoning Zhang, and Choong Seon Hong. Fedmekt: Distillation based embedding knowledge transfer for multimodal federated learning. Neural Networks, 183:107017, 2025.
[25] Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se- Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. In NeurIPS, 2022.
[26] Daliang and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. In NeurlPS Workshop. 2019.
[27] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In ICDE, pages 965-978, 2022.
[28] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR, pages 10713-10722, 2021.
[29] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys, 2020.
[30] Xiaoxiao Li, Yufeng Gu, Nicha Dvorek, Lawrence H Staib, Pamela Ventola, and James S Duncan. Multi-site fmri analysis using srivacy-preserving federated learning and domain adaptation: Abide results. MedIA, page 101765, 2020.
[31] Xinjia Li, Boyu Chen, and Wenlian Lu. Feddkd: Federated learning with decentralized knowledge distillation. Applied Intelligence, 53(15):18547-1856) 2023.
[32] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NewrlPS, pages 2351 2351-2363, 2020.
[33] Fuchang Liu, Yu Wang, Zheng Li, and Zhigeng Pan. Geikd: Self-knowledge distillation based on gated ensemble networks and influences-based label noise removal. Computer Vision and Image Understanding, 235:103771. 2023.

---

[34] Quande Liu, Cheng Chen. Jing Qin, Qi Dou, and Pheng-Ar Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In CVPR, pages 1013-1023, 2021.
[35] Kangyang Luo, Shuai Wang, Yexuan Fu, Xiang Li, Yunshi Lan, and Ming Gao. Dfrd: data-free robustness distillation for heterogeneous federated learning. arXiv preprint arXiv:2309.13:: 2023.
[36] Yuhang Ma, Zhongle Xie, Jue Wang, Ke Chen, and Lidan Shou. Continual federated learning based on knowledge distillation. In IJCAI, pages 2182-211 2188, 2022.
[37] James MacQueen et al. Some methods for classification and analysis of multivariate observations. In BSMSP, pages 281-297, 1967.
[38] Gales M Malinin A. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. In NeurIPS, 2019.
[39] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efinn -fffiiccenntt learning of deep networks from decentralized data. In AISTA7SS pages 1282, 2017.
[40] Matias Mendieta, Taojiannan Yang, Pu Wang. Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In CVPR, pages 8397-8406, 2022.
[41] Jiaxu Miao, Zongxin Yang, Leilei Fan, and Yi Yang. Fedseg: Class- htaso-heterogeneouss federated learning for semantic segmentation. In CVPR, pages 8042-8052, 2023.
[42] Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. Fedproc: Prototypical contrastive federated learning on non-iid data. arXiv preprint arXiv:2109. 12273, 2021.
[43] Xingchao Peng. Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. In ICLR, 2020.
[44] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, pages 213-226, 2010.
[45] M. Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neighbor relations. In CVPR, pages 8934-8943, 2019.
[46] Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor- Mor-Yosef, and Itai Zeitak. Overcoming forgetting federated learning on non-iid data. In NeurlPS Workshop. 2019.
[47] Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor- Yosef, and Itai Zeitak. Overcoming forgetting federated learning on non-iid data. NeurlPS Workshop, 2019.
[48] Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In NeurlPS, pages 21394-21405, 2020.
[49] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fedproto: Federated prototype learning across heterogeneous clients. In AAAI, 2022.
[50] Grant Van Horn, Rui Qian, Kimberly Wilber, Hartwig Adam, Oisin Mac Aodha, and Serge Belongie. Exploring fine-grained audiovisual categorization with the ssw60 dataset. In ECCV, 2022.
[51] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 5018-5027, 2017.
[52] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). Practical Guide, 1st Ed., Cham: Springer International Publishing, page 3152676, 2017.
12

---

[53] Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023.
[54] Haozhao Wang. Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain- -amarn--wwareaae federated knowledge distillation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages
2023.
[55] Joe H Ward Jr. Hierarchical grouping optimize an objective function. JASA, pages 236-244, 1963.
[56] Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. arXiv preprint arXiv:2307.15190, 2023,
[57] Chen Wu, Sencun Zhu, and Prasenjit Mitra. Federated unlearning with knowledge distillation. arXiv preprint wwxiv:2201.044444 2022.
[58] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication- efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
[59] Yuan-Y Yi Xu, Ci-Siang Lin, and Yu-Chiang Frank Wang. Bias-eliminating augmentation learning for debiased federated learning. In CVPR, pages 20442-20452, 2023.
[60] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM TIST, pages 1-19,2019 2019.
[61] Dezhong Yao, Wanning Pan, Yutong Dai, Yao Wan, Xiaofeng Ding, Chen Yu, Hai Jin, Zheng Xu, and Lichao Sun. Fedgkd: Toward heterogeneous federated learning via global knowledge distillation. IEEE Transactions on Computers, 73(1):3-17, 2023.
[62] Mang Ye, Wei Shen, Bo Du, Eduard Snezhko, Vassili Kovalev, and Pong C Yuen. Vertical federated learning for effectiveness, security, applicability: A survey, ACM Computing Surveys, 17(9):1-32, 2025.
[63] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In ICMI., pages 2631 1-26329, 2022.
[64] Lin Zhang, Yong Luo, Yan Bai, Bo Du, and Ling- Yu Duan. Federated learning for non-iid data via unified feature learning and optimization objective alignment. In ICCV, pages 4420 4428, 2021.
[65] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In CVPR, pages 10174-10183, 2022.
[66] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning. In International conference machine learning, pages PMLR, 2021.
13

---

A Convergence Proof of DKDR
A.1 Assumptions
L-smoothness: For all k. fw) is L-smooth: [[V vif(w2))l - L|[w w2ll.
(Dynamic weights preserve smoothness; 7x: "i changes Lipschitz: (7,(wr) - (aw2)) S 1:llw1-011 w2ll. yielding L' I L + r max(G Grka). Quasi-static: y per round.)
Bounded gradients:
[[ G
G?.
Bounded variance: E& V / (u) VS(w)] S a²
Bounded distillation gradients: V (w)) - D2 & max(Grad Grea) (logits clipped).
Non-convex: f(w) lower-bounded by f*
Parameters: 17 > 0, rounds T, local steps E, ll S 1/LLE))
A.2
Theorem (Convergence of DKD)
1 T-1 T
[ 4(f(w") - f
d² 7TE +6L'n(E-1) +8(L (o+
A(2) (+++A) K +0(221
Σ t=0
where A(7) fvll Vf(w))
Dynamic weights yield A(7) Ao (static). For 1 OlVK/(TE")), RHS OLLVVKTH converges 0 as 7' co. Compared to FedAvg (A a²) or statie KLD, DKD tighter.
Key Lemma. Reduction in A(7) Forward KLD prioritizes 71 (mean-seeking; OLekd/ /Lekk/Ozj.. /2zz, 2j.k =)-1); reverse prioritizes "s (mode-seeking; 0Trka/ @TTzk/O2z,k = 2j.k log(2;,x / Tyykd). Weights = 7/ (7/ + n) balance via gradient norm.
V Laakd=
T V Terke
71 V Ligkd- V/ + 71
7ss + 71
By Pinsker's (KL 2 (1/2) T
/ V max Lsska), R2 (((Trka) - path
7/700
E (2
min
(Y, + 71)2²
5 > 0 from complementarity. Specifically,
Laka Vaxda VLsudd (VLrndd | B1l Loakd
Lrrkall (7/ )2)
767
(8 ((1). Thus,
(V Ao
7
(7r + )2)
A(r) < Ao - <) = 1/2 when balanced) Ag-
A.3 Proof Sketch
Step 1: Local:
w
1eet 1.e
19 0k (w
de 1e
Aggregate:
w = =(I/K) Σ wf
Define
whe = (1/K) Σ wx ghe = (1/K) Σ g
14

---

Step 2: Descent:
E[f(w'+1)] & f(w²) + (Vi(w"), w w²) + (I/ w'l,
E-1
w 1+1
w¹ -1 Σ gB
yielding:
(2/(2))
Σ
+ (L')
Step Bias:
Elighe [f(w))11 (o² + +
Step Sum t=0 to T-1, divide by /2 Ee S /2/2, constants to 8, yields bound.
B
Notations Table
Symbol
Meaning
Symbol
Meaning
Z (y x))
Local model distribution
( x))
Global model distribution
De
Private data of k-th client
Nx
Number of data points
w
Global model parameters at round
w
Local model parameters of k-th client Global model output distribution Standard KD loss
Client model output distribution
2
Softmax function
Cad
Cykd
Reverse KD loss
a(i)
Indicator function for DKCs/AKCs
Threshold to define DKCs/AKCs Knowledge discrepancy in DKCs
"
Knowledge discrepancy in AKCs
Lakd
Dynamic KD loss
Difference vector
U, E, V
SVD decomposition matrices Domain expert model
g
Client clusters
!.dom
Likd
Final KD loss
L*
Local loss of k-th client
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Sec 3,4
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to Sec 3.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Sec 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code is accessible in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guudeeee http://oeeubiiissioooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: Please refer Sec 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Sec 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Sec 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics lattps://neuripa http://www.p....ou......ccccccl.
Answer: [Yes]
Justification: Please see the supplementary file.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Justification: The research presented in this paper is foundational. It is not directly tied any specific applications or deployments.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
20

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer 4.1
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
21

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: None of the core methods in this paper rely on LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httess///eerrrppcccc///oterrrrrp for what should or should not be described.
22


Reviews:
[{"review_id": "wIOd5ptoX6", "reviewer": "Reviewer_Wn6p", "summary": "First, when the global model's output exhibits a multi-modal distribution, minimizing forward KL divergence leads to significant deviation. Second, the global model's accuracy drops sharply in certain domains, failing to provide high-quality guidance for clients. To address these issues, the authors propose the DKDR method.\nDKDR dynamically allocates weights between forward and reverse KLD based on knowledge discrepancies, thereby reducing distillation bias. Furthermore, it decomposes knowledge into shared and unique components, employing SVD and clustering to identify domain experts, enabling clients to learn from these experts rather than the global model.", "questions": "Q1. The paper mentions that the DKDR method involves SVD and clustering techniques, which increase computational overhead and may significantly prolong training time, especially on resource-constrained clients. Therefore, it would be beneficial to provide theoretical or experimental evidence in the paper to demonstrate the training overhead and latency.\n\nQ2. While the experimental section covers multiple datasets and scenarios, it primarily focuses on image classification tasks. If applied to other tasks such as natural language processing, would the dynamic divergence balancing and knowledge decoupling still offer advantages?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper conducts an in-depth theoretical analysis of the behaviors of forward and reverse KL divergence, explaining their advantages and disadvantages in different scenarios, and proposes a dynamic weighting method to balance these two divergences, thereby reducing distillation bias. It introduces Knowledge Decoupling (KDP) technology, which uses SVD and clustering to identify domain experts, enabling clients to learn from these experts rather than the global model, thereby improving the model's generalization capability.\n\n**Weaknesses**\n\n1、Will the computational overhead or training latency caused by dynamic weighted balancing and SVD become a bottleneck?\n\n2、How does DKDR perform on non-image datasets?", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer Wn6p\n\nDear Reviewer Wn6p,\n\nWe sincerely thank you for your comprehensive review and positive feedback on our DKDR framework. We appreciate your recognition of the theoretical analysis of forward and reverse KL divergence. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.\n\n---\n\n### Weaknesses & Questions\n\n>**`W1 & Q1: Training Overhead and Latency of DKDR.`**\n\n**A1:** Thank you for your advice!  We conduct comprehensive experiments evaluating both computational and communication costs using the OfficeCaltech dataset. Compared with baselines, the computational and communication expenses of DKDR are acceptable and the performance gains are worthwhile.\n\n*Table. **Comparison of Time and Memory Usage** in OfficeCaltech dataset.*\n\n| Method   | Client Time (s) | Server Time (s) | Communication Cost (mb) | Accuracy (%) |\n| -------- | --------------- | --------------- | ----------------------- | ------------ |\n| FedAVG   | 2.70            | 0.02            | 2.83                    | 77.94        |\n| FedProto | 14.13           | 0.01            | 2.83                    | 79.26        |\n| Scaffold | 9.23            | 0.02            | 8.48                    | 75.32        |\n| **DKDR** | 6.28            | 0.11            | 8.25                    | **81.14**    |\n\nWe also conduct scalability experiments with varying client numbers (K ∈ {20, 60, 100})， as shown in following table. For K=100 clients, server overhead remains under 3 seconds per round.\n\n*Table. **The relationship of client Number $K$ and computation cost** in OfficeCaltech dataset.*  \n\n| Client Number $K$ | Server Processing Time (s) | Improvement (Compared to FedAVG) |\n| ----------------- | -------------------------- | -------------------------------- |\n| 20                | 0.11                       | +3.20%                           |\n| 60                | 0.42                       | +2.06%                           |\n| 100               | 2.94                       | +2.54%                           |\n\n\n\n>**`W2 & Q2: Non-image Dataset Supplementation.`**\n\n**A2:** Thank you for this valuable suggestion! We acknowledge that our current evaluation focuses primarily on image classification tasks, which limits the generalizability of our findings. To address this concern, we conduct additional experiments on federated recommendation task to demonstrate the effectiveness of DKDR across different modalities. We evaluate DKDR on federated recommendation using the Adressa dataset. Adressa is publicly released by Adresseavisen, a local newspaper company in Norway. we use the 6-th day click to build training dataset and construct historical clicks from the first 5 days samples. We randomly sample 20% clicks from the last day clicks for validation and the rest clicks for testing. The historical clicks of validation and testing dataset are constructed from the first 6 days samples. Following many previous news recommendation works ([1,2,3]), we use AUC, MRR, nDCG@5 and nDCG@10 as evaluation metrics. AUC measures a model's ability to distinguish positive and negative classes. MRR averages the reciprocal rank of the first relevant item in a ranked list. nDCG@5 and nDCG@10 evaluate ranking quality for the top 5 and 10 items, respectively, weighting relevant items higher when they appear earlier, normalized for comparison. The results are shown in the following table. Our analysis reveals that core principles in DKDR (pathway reliability and teacher reliability) are indeed modality-agnostic. The multi-peaked distribution problem occurs in both vision and recommendation domains when aggregating heterogeneous client models, validating our theoretical foundation. We will include comprehensive cross-modal evaluation in our final version to demonstrate the broad applicability of DKDR beyond computer vision tasks.\n\n*Table. **Comparison with the SOTA in the non-image dataset Adressa.*** \n\n| Method | AUC       | MRR   | nDCG@5    | nDCG@10   |\n| ------ | --------- | ----- | --------- | --------- |\n| FedRec | 70.83     | **41.39** | 42.58     | 46.34     |\n| DKDR   | **72.17** | 40.82 | **43.68** | **47.12** |\n\n[1] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long- and short-term user representations. In ACL, pages 336–345.\n\n[2] Tao Qi, Fangzhao Wu, Chuhan Wu, Yongfeng Huang, and Xing Xie. 2020. Privacy-preserving news recommendation model learning. In EMNLP 2020, pages 1423–1432.\n\n[3] Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020a. User modeling with click preference and reading satisfaction for news recommendation. In IJCAI, pages 3023–3029."}, {"review_id": "yGtCANUuSM", "reviewer": "Reviewer_CP1N", "summary": "The paper tackles the critical issue of data heterogeneity in FL, particularly in multi-domain settings. It proposes DKDR, a novel framework that enhances distillation reliability through two methods: dynamic knowledge distillation, which adaptively weights forward and reverse KLD to reduce distillation bias, and knowledge decoupling, which identifies domain experts to provide reliable guidance. The approach is rigorously evaluated on single-domain and multi-domain image classification tasks to validate the effectiveness of the proposed approach.", "questions": "- Q1: In the KDP module, why was clustering applied to SVD-filtered differences rather than directly clustering client updates (e.g., updated parameters)? Could the authors clarify the motivation linking this design to improved domain expert identification?\n\n- Q2: The dynamic weighting in DKD relies on hyperparameter μ (Sec. 3.2). Could the authors elaborate on how μ=0.5 was determined and whether it generalizes across all datasets?\n\n- Q3: How does DKDR compare to prototype-based methods (e.g., FedProto) in terms of conceptual differences, given the apparent similarity in leveraging domain-specific knowledge?\n\n- Q4: Why do single-domain tasks and multi-domain tasks use different backbones? Can further ablation studies be provided?", "limitations": "The paper acknowledges computational overhead from SVD and clustering in DKDR, which may pose challenges in resource-constrained settings. Additionally, the assumption of distinct domains could falter with significant domain overlap, potentially impacting clustering efficacy.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- S1: The motivation is clear and good. The paper clearly identifies two underexplored issues in federated KD: pathway unreliability and teacher unreliability. The analysis of forward and reverse KLD in federated learning is novel.\n\n- S2: The dynamic KLD weighting adapts to knowledge discrepancies, offering a fresh take on mitigating distillation bias. And identifying domain experts via knowledge decoupling to enhance reliability across diverse domains is a practical innovation for FL (Sec. 3.3).\n\n- S3: The figures of problem and framework illustration are clear and detailed, and the equations and explanations are reasonable. \n\n- S4: Comprehensive experiments across diverse datasets and settings (Sec. 4) showcase the robustness and practical utility of DKDR.\n\nWeaknesses:\n\n- W1: The paper uses extensive mathematical notation without a summary table, which could hinder readability for some audiences.\n\n- W2: While FINCH clustering is justified (Sec. 3.4), hyperparameter details for baseline clustering methods (e.g., DBSCAN) are omitted, limiting reproducibility of the ablation study (Tab. 1).", "was_revised": true, "final_justification": "I have reviewed the rebuttal and checked the feedback from other reviewers carefully. The SVD-filtered opponent is well discussed and the rationale for selecting hyperparameter is well explained. The additional experiments also further validate the effectiveness of the proposed method and address my key concerns. I will raise my score to accept.", "rebuttal": "## Response to Reviewer CP1N\n\nDear Reviewer CP1N,\n\nThank you very much for your detailed and constructive review. We are encouraged by your positive feedback on our motivation, technical contributions, and experimental validation. Your recognition of our novel analysis of forward/reverse KLD in federated learning and the practical innovations of our framework is greatly appreciated. We hope our responses below will address your concerns and strengthen your confidence in our work.\n\n---\n\n### Weaknesses\n\n>**`W1: Notation Table of DKDR.`**\n\n**A1:** We appreciate this feedback on improving readability. You are absolutely right that a notation summary would enhance accessibility. We will add a comprehensive notation table in the final version that clearly defines all mathematical symbols used throughout the paper, including:\n\n*Table: **Notation table for DKDR.**\n\n| Symbol              | Meaning                              | Symbol                                      | Meaning                                 |\n| ------------------- | ------------------------------------ | ------------------------------------------- | --------------------------------------- |\n| $Z\\_w(y \\mid x)$     | Local model distribution             | $Z(y \\mid x)$                               | Global model distribution               |\n| $D\\_k$               | Private data of $k$-th client        | $N\\_k$                                       | Number of data points                   |\n| $w^t$               | Global model parameters at round $t$ | $w\\_k^t$                                     | Local model parameters of $k$-th client |\n| $Z\\_{i, k}^t$        | Client model output distribution     | $Z\\_i^{t-1}$                                 | Global model output distribution        |\n| $\\sigma$            | Softmax function                     | $\\mathcal{L}\\_{\\text{skd}}$                  | Standard KD loss                        |\n| $\\mathcal{L}\\_{rkd}$ | Reverse KD loss                      | $a(j)$                                      | Indicator function for DKCs/AKCs        |\n| $\\mu$               | Thresold to define DKCs/AKCs         | $\\gamma\\_s$                                  | Knowledge discrepancy in AKCs           |\n| $\\gamma\\_l$          | Knowledge discrepancy in DKCs        | $\\mathcal{L}\\_{dkd}$                         | Dynamic KD loss                         |\n| $v\\_k^t$             | Difference vector                    | $\\mathbf{U}, \\mathbf{\\Sigma}, \\mathbf{V}^T$ | SVD decomposition matrices              |\n| $\\beta^t$           | Client clusters                      | $w\\_v^{t, \\text{dom}}$                       | Domain expert model                     |\n| $\\mathcal{L}\\_{fkd}$ | Final KD loss                        | $\\mathcal{L}\\_k$                             | Local loss of $k$-th client             |\n\n\n\n>**`W2: Hyperparameter Details for Clustering Methods.`**\n\n**A2:** For the ablation study in Table 1, we used the following hyperparameters:\n\n- **K-means**: optimal k=3 (corresponding to the 3 domains in Office31)\n- **DBSCAN**: appropriate $ε=0.05$ and $MinPts=1$\n- **FINCH**: No hyperparameters required (parameter-free)\n\n---\n\n### Questions\n\n>**`Q1: How SVD-filtered improves domain expert identification?`**\n\n**A1:** Excellent question! The key insight is that **raw client parameters contain both shared knowledge and domain-specific signals mixed together**. By clustering raw parameters, we would group clients based on their overall model similarity, which is dominated by shared knowledge rather than domain-specific characteristics.\n\nOur approach specifically isolates domain-specific signals through two steps:\n1. **Shared knowledge removal**: $v^t\\_k = w^t\\_k - w^{t-1}$ removes the shared knowledge\n2. **SVD filtering**: Retains the top-r singular values to extract the most significant domain-specific patterns while filtering noise\n\nThis design ensures that clustering focuses purely on domain-specific knowledge differences, leading to more accurate domain expert identification (see details in Sec.4.3, Page 9).\n\n\n\n>**`Q2: How μ=0.5 was determined and whether it generalizes across all datasets?`**\n\n**A2:** The choice of μ=0.5 is both theoretically motivated and empirically validated:\n\n**Theoretical Justification**: μ represents the threshold for separating Dominant Knowledge Components (DKCs) from Ancillary Knowledge Components (AKCs). When μ=0.5, we capture exactly half of the cumulative probability mass in the DKCs, providing a natural balance between the two components. This creates an optimal trade-off where:\n- μ < 0.5: DKD degenerates toward reverse KLD (mode-seeking bias)\n- μ > 0.5: DKD degenerates toward forward KLD (mean-seeking bias)\n\n**Empirical Validation**: Our sensitivity analysis (Figure 5) shows that μ=0.5 consistently achieves optimal performance across both Cifar-100 and Office31 datasets. The stability across different dataset types (single-domain vs. multi-domain) demonstrates its generalizability.\n\n\n\n>**`Q3: Conceptual Difference with Prototype-based Methods.`**\n\n**A3:** While both approaches leverage domain-specific knowledge, they differ fundamentally in methodology and granularity:\n\n**FedProto**: Uses class prototypes to align feature representations across clients. It focuses on **feature-level** alignment by matching class centroids, assuming similar classes should have similar representations across domains.\n\n**DKDR**: Creates specialized teacher models for different domains at the **knowledge level** leveraging domain-specific knowledge.\n\nOur experimental results show DKDR (50.57%) outperforms FedProto (39.98%) on Office31, demonstrating the effectiveness of our knowledge-centric approach.\n\n\n\n>**`Q4: Ablation Studie of Backbones.`**\n\n**A4:** The choice of backbone is based on the complexity of the datasets. Single-domain tasks, such as Cifar-10 and Cifar-100, are relatively simpler, so we use ResNet-10 to reduce computational cost while maintaining performance. Multi-domain tasks, such as Office31 and Office Home, are more complex due to domain shifts, requiring the more powerful ResNet-18 to capture intricate patterns effectively. This is detailed in Sec.4.1 (Page 8). We also conduct experiments with ResNet-18 on Cifar-100 to validate consistency:\n\n*Table. **Ablation study of backbone** on Cifar-100 dataset ($\\zeta=0.5$).*  DKDR achieves consistent better performance compared to three baselines (FedAVG, FedProto, Scaffold).\n\n| Method       | ResNet-10 (Original) | ResNet-18 |\n| ------------ | -------------------- | --------- |\n| FedAvg   | 48.57                | 68.40      |\n| FedProto | 51.34                | 69.86     |\n| Scaffold     | 51.76                | 56.80     |\n| **DKDR**     | **53.18**            | **72.44** |\n\nThe consistent improvement demonstrates that the effectiveness of DKDR is backbone-agnostic. We will include this ablation in the final version.\n\n\n\n---\n\n### Limitations\n\n>**`L1: Computational Overhead Discussion.`**\n\n**A1:** Sorry for the misunderstanding. We want to clarify that while SVD and clustering add computational cost, this occurs only **once per communication round on the server side**, not on resource-constrained clients. The server-side computation is typically acceptable in federated learning scenarios. Additionally, our analysis shows the overhead is reasonable compared to the significant performance gains achieved. We conduct experiments in OfficeCaltech to further analyze it, as shown in below table.\n\n*Table. **Comparison of Time and Memory Usage** in OfficeCaltech dataset.*\n\n| Method   | Client Time (s) | Server Time (s) | Communication Cost (mb) | Accuracy (%) |\n| -------- | --------------- | --------------- | ----------------------- | ------------ |\n| FedAVG   | 2.70            | 0.02            | 2.83                    | 77.94        |\n| FedProto | 14.13           | 0.01            | 2.83                    | 79.26        |\n| Scaffold | 9.23            | 0.02            | 8.48                    | 75.32        |\n| **DKDR** | 6.28            | 0.11            | 8.25                    | **81.14**    |\n\n\n\n>**`L2: Discussion on Domain Overlap Scenarios.`**\n\n**A2:** You are correct that the knowledge decoupling module relies on the assumption that domains are sufficiently distinct for clustering to accurately identify domain experts. When domains have significant overlap, this assumption may not hold, potentially affecting the quality of domain expert identification. When domain boundaries are unclear, our method doesn't fail catastrophically. Instead, it exhibits graceful degradation:\n\n- **Scenario 1**: If clustering fails to separate domains correctly, some clusters may contain clients from multiple domains,\n- **Scenario 2**: The dynamic distillation (DKD) component continues to provide benefits even when domain experts are suboptimal,\n\n **Result**: DKDR reduces to a form closer to standard federated distillation but still benefits from the DKD. It can still perform well in such situation (see details in Sec.4.4, Page 9)."}, {"review_id": "8EFISGMjyn", "reviewer": "Reviewer_bzDB", "summary": "This paper proposed the DKDR (Dynamic Knowledge Distillation for Reliability) framework to address the reliability issue of knowledge distillation in federated learning. The author pointed out two key issues in existing federated knowledge distillation methods, namely the unreliability of the distillation path and the unreliability of the teacher model. To address the two issues, the DKDR framework contains two core components. The first is the dynamic knowledge distillation (DKD) module, which divides knowledge into dominant knowledge components (DKCs) and auxiliary knowledge components (AKCs). At the same time, the weights of the forward and reverse KL divergences are dynamically adjusted according to the knowledge differences in different knowledge modules, thereby achieving more accurate knowledge fitting. The second is the knowledge decoupling (KDP) module, which decomposes knowledge into shared knowledge and unique knowledge. Then, SVD filtering and clustering techniques are used to identify domain experts, enabling clients to acquire knowledge from reliable domain experts rather than global models. The authors conducted comprehensive experimental verification on multiple datasets to demonstrate the effectiveness of the method. In addition, ablation studies confirmed the effectiveness of the two core components, and hyperparameter sensitivity analysis verified the stability of the method.", "questions": "a)\tCan the convergence and effectiveness of DKDR be theoretically proven?\nb)\tCan you compare DKDR with the SOTA method in 2024 to further prove the effectiveness of your method?\nc)\tCan you demonstrate the effectiveness of your approach to a larger and more varied number of clients?\nd)\tYou only used the EMNIST dataset in the Comparisons between Condition Options section. Can you verify this on a more complex dataset?\ne)\tCan we compare the proposed method with the baseline method in terms of computational cost and communication time?\nf)\tThe paper only uses a simple set of experimental data to prove that FINCH is more suitable than Kmeans and DBSCAN. Can this problem be proved theoretically and in details?", "limitations": "The authors discussed some limitations in the \"Limitations\" section. However, the main limitation of the paper is the weak theoretical foundation and the limitations of the experimental setting.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\na)\tThe paper proposes a technically sound two-component framework, which are the Dynamic Knowledge Distillation (DKD) and Knowledge Decoupling (KDP) modules. The experimental verification is comprehensive, covering single-domain and multi-domain scenarios, detailed ablation studies and hyperparameter sensitivity analysis.\nb)\tA dynamic KLD weight adjustment mechanism based on knowledge differences is proposed. This adaptive method is different from the existing static knowledge distillation method. Although the individual technical components (SVD, clustering, KL divergence) are not new, their integrated application to the federated KD reliability problem has technical contributions.\nc)\tThe motivation of this paper is clearly stated, and Figure 1 effectively illustrates the two core issues of path unreliability and teacher unreliability. The method description is highly structured and logically fluent, from the basic definition to the DKD theoretical analysis and then the KDP algorithm steps.\n\nWeaknesses\na)\tThe method is mainly a combination of existing technologies rather than a fundamental innovation. The dynamic weight mechanism is essentially a combination of heuristic forward/backward KLD based on a simple knowledge difference measure, SVD denoising and clustering to identify domain experts are all standard machine learning techniques.\nb)\tThere is a lack of convergence guarantee for the dynamic weight mechanism, and the superiority of the proposed weight strategy over the fixed weight strategy has not been proven. At the same time, there is a lack of necessary theoretical analysis of convergence and convergence rates.\nc)\tThere are major flaws in the baseline of experimental design. The most recent method is from 2022 and lacks comparison with SOTA methods.\nd)\tThe number of clients in the experiment (N=20) is relatively small, therefore, the effectiveness of large-scale scenarios has not been fully verified.\ne)\tThere is a complete lack of computational overhead evaluation, which is particularly important for FL methods based on knowledge distillation. SVD decomposition, clustering operations, and dynamic weight calculations introduce significant computational overhead, which may make the method infeasible on resource-constrained devices.", "was_revised": true, "final_justification": "The authors' rebuttal (to me and other reviewers) have addressed my concerns, so I decided to raise my score.", "rebuttal": "## Response to Reviewer bzDB\n\nDear Reviewer bzDB,\n\nThank you very much for taking the time and effort to review our paper. Your positive feedback on the contribution of our approach, the clarity of our writing, and the comprehensiveness of our experiments is very encouraging. We hope that our responses below will address your concerns and update the score.\n\n---\n\n### Weaknesses & Questions\n\n>**`W a): Novelty Discussion of DKDR.`**\n\nA1: Thank you for the feedback. We would like to emphasize that the core contribution of DKDR lies in **being the first to conduct systematic observational analysis of distillation behavior in federated learning**. We discover that using different distillation methods leads to distinct knowledge discrepancies between local and global models in federated learning. Specifically, when using forward KLD, local models better fit the Dominant Knowledge Components (DKCs) of the global model, while when using reverse KLD, they better fit the Ancillary Knowledge Components (AKCs). Based on this significant finding, we propose a dynamic weighting mechanism that effectively addresses the bias introduced by different distillation methods, thereby solving the unreliability problem of distillation pathways. **Our systematic analysis of federated distillation and the consequent revelation of the essential behaviors of different distillation methods constitute the core innovation.** It fundamentally shifts how researchers should approach federated KD by providing a theoretical framework for understanding KLD behaviors. We will emphasize this key point in the final version.\n\n\n\n>**`W b).1 & Q a): Convergence Theory.`** \n\n**A2:**  Thank you for the advice. We extend analysis in Sec.3.2 (Page 4) to prove DKD convergence under non-convex federated settings. Under L-smooth losses with bounded gradients, our main result shows:\n$$ \\frac{1}{T}\\sum\\_{t=0}^{T-1}\\mathbb{E}|\\nabla f(w\\_t)|^2 \\leq \\frac{4(f(w\\_0) - f^*)}{\\eta T E} + 6L'\\eta(E-1)G^2 + \\frac{\\sigma^2}{K} + 8(L')^2 E\\eta\\left(G^2 + \\frac{\\Lambda(\\gamma)}{K}\\right) + 2\\kappa^2 \\eta^2 E G^2, $$\nwhere\n$$ \\Lambda(\\gamma) = \\mathbb{E}\\_k\\left[|\\nabla f\\_k(w) - \\nabla f(w)|^2\\right] \\leq \\Lambda\\_0 - \\beta \\frac{\\gamma\\_s \\gamma\\_l}{(\\gamma\\_s + \\gamma\\_l)^2}\\mathbb{E}\\left[|\\nabla L\\_{skd} - \\nabla L\\_{rkd}|^2\\right], $$\n**Key Innovation:** The gradient diversity term satisfies:  \n$$\\Lambda(\\gamma) \\leq \\Lambda\\_0 - \\beta \\frac{\\gamma\\_s \\gamma\\_l}{(\\gamma\\_s + \\gamma\\_l)^2}\\mathbb{E}\\left[|\\nabla L\\_{skd} - \\nabla L\\_{rkd}|^2\\right]$$\nThis proves the dynamic weighting mechanism reduces effective variance compared to standard federated averaging and static KLD approaches ($\\Lambda\\_0$), leading to faster convergence. **The reduction occurs because forward KLD prioritizes DKCs while reverse KLD focuses on AKCs, and their dynamic combination minimizes gradient conflicts.** It validates our empirical observations (Table. 6, Page 9) that DKD outperforms static KLD approaches by adapting to knowledge discrepancies.\n\n**\\_Due to space limitation, the complete proof with detailed lemmas and technical derivations will be provided in the appendix in the final version.**\n\n\n\n>**`W b).2: Dynamic Weight Mechanism (DKD) Discussion.`** \n\n**A3:** Let $L\\_{dynamic} = α(t)L\\_{forward} + (1-α(t))L\\_{reverse}$ where $α(t) = γ\\_l/(γ\\_s+γ\\_l)$. As knowledge discrepancies decrease, $α(t)$ adapts to minimize dominant bias sources, thus outperforming fixed weights. We conduct experiments with some fixed weight strategies to confirm it, in which DKD outperforms all fixed weight strategies, as shown in the following table.\n\n*Table. **Ablation study of the proposed weight strategy DKD on Cifar-100.***\n\n| Method  | $α(t)$ = 0 | $α(t)$ = 0.5 | $α(t)$ = 1 | **DKD (ours)** |\n| ------- | ---------- | ------------ | ---------- | -------------- |\n| **acc** | 48.03      | 47.69        | 47.52      | **48.79**      |\n\n\n\n>**`W c) & Q b): SOTA Method Comparison (Later 2024).`** \n\n**A4:** We have incorporated two SOTA methods (FedTGP (AAAI 2024), FedAA (AAAI 2025)) for comparison with our method. The results are shown in the table below. Our method consistently outperforms these SOTA methods across four datasets. We will include these two baselines in the final version.\n\n*Table. **Comparison with the SOTA methods in four datasets.*** \n\n| Method   | Cifar-10  | Cifar-100 | Office31  | OfficeHome |\n| -------- | --------- | --------- | --------- | ---------- |\n| FedTGP   | 80.64     | 50.12     | 41.23     | 60.04      |\n| FedAA    | 79.88     | 50.61     | 36.12     | 59.23      |\n| **DKDR** | **81.93** | **51.84** | **50.57** | **62.48**  |\n\n\n\n\n>**`W d) & Q c): Discussion on Large-scale Client Scenarios.`** \n\n**A5:** We conduct additional ablation experiments on client numbers N, as shown in the table below. Our method consistently outperforms baselines, demonstrating the robustness of DKDR in large-scale client scenarios. We will include this ablation study in the final version with detailed discussion.\n\n*Table. **Ablation study of client Number $N$** on OfficeCaltech dataset. DKDR achieves consistent better performance compared to three baselines in three settings.*\n\n| Client Number $N$ | 20        | 60        | 100       |\n| ----------------- | --------- | --------- | --------- |\n| FedAvg            | 77.94     | 72.33     | 61.93     |\n| FedProto          | 79.26     | 68.77     | 49.32     |\n| FedProx           | 78.28     | 72.46     | 63.96     |\n| **DKDR**          | **81.14** | **74.39** | **64.47** |\n\n\n\n>**`Q d): Dataset Supplementation.`**\n\n**A6:** To further demonstrate the superiority and robustness of DKDR, we introduce the OfficeCaltech dataset for additional comparison with baseline methods, as shown in the table below. The results show that our method consistently outperforms all the baselines.\n\n*Table. **Comparison with the SOTA methods in OfficeCaltech dataset.***  \n\n| Method            | FedAvg | FedProx | FedDyn | Scaffold | FedProto | MOON  | FedNTD | DKDR      |\n| ----------------- | ------ | ------- | ------ | -------- | -------- | ----- | ------ | --------- |\n| **OfficeCaltech** | 77.94  | 78.28   | 77.90  | 75.32    | 79.26    | 78.51 | 79.43  | **81.14** |\n\n\n\n>**`W e) & Q e): Overhead Comparison with SOTA Methods.`** \n\n**A7:** We conduct experiments on computational and communication costs, comparing our method with baselines. We calculate the average communication and computational cost per communication epoch for each method, as shown in the table below. The computational and communication costs are acceptable and the resulting performance improvements are worthwhile. \n\n*Table. **Comparison of computational and communication costs** in OfficeCaltech dataset.*\n\n| Method   | Client Time (s) | Server Time (s) | Communication Cost (mb) | Accuracy (%) |\n| -------- | --------------- | --------------- | ----------------------- | ------------ |\n| FedAVG   | 2.70 | 0.02            | 2.83                    | 77.94        |\n| FedProto | 14.13      | 0.01            | 2.83                    | 79.26        |\n| Scaffold | 9.23            | 0.02            | 8.48                    | 75.32        |\n| **DKDR** | 6.28            | 0.11            | 8.25                    | **81.14**    |\n\n\n\n>**`Q f): Clustering Method Discussion.`**  \n\n**A8:** We provide theoretical justification for choosing FINCH over K-means and DBSCAN based on their fundamental formulations and hyperparameter requirements:\n\n**K-means** requires pre-specification of cluster number $k$ and aims to minimize:  \n$J=∑\\_{i=1}^k∑\\_{x∈Ci}∣∣x−μi∣∣^2$\nwhere $\\mu\\_i$ is the centroid of cluster $C\\_i$. **The hyperparameter $k$ must be determined a priori, which is challenging in federated settings where the number of domains is unknown.**\n\n**DBSCAN** requires two hyperparameters:\n- $ε$ (epsilon): neighborhood radius\n- $MinPts$: minimum number of points to form a dense regio\nA point p is a core point if: $|N\\_ε(p)| \\geq MinPts$, where $N\\_ε(p) = {q \\in D : dist(p,q) \\leq ε}$\n**Its performance is highly sensitive to these parameters, and optimal values vary significantly across different data distributions.**\n\n**FINCH** is a **parameter-free** clustering algorithm. Given the integer indices of the first neighbor of each data point, FINCH directly defines an adjacency link matrix. The elements of the adjacency matrix are defined as follows:\n\n$$a\\_{ij} = \\begin{cases} 1 & \\text{if } j = \\kappa\\_1^i \\text{ or } \\kappa\\_1^j = i \\text{ or } \\kappa\\_1^i = \\kappa\\_1^j \\\\\\ 0 & \\text{otherwise} \\end{cases},​$$\n\nwhere $\\kappa\\_1^i$ symbolizes the first neighbor of point $i$. For a dataset with $N$ points, this produces an $N \\times N$ sparse adjacency matrix:\n\n$$A = \\begin{bmatrix} a\\_{11} & a\\_{12} & \\cdots & a\\_{1N} \\\\\\ a\\_{21} & a\\_{22} & \\cdots & a\\_{2N} \\\\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\\\ a\\_{N1} & a\\_{N2} & \\cdots & a\\_{NN} \\end{bmatrix}.$$\n\nThe clustering equation directly delivers clusters as connected components of the graph $G = (V,E)$ where $V = {1,2,...,N}$ and $E = {(i,j) : A(i,j) = 1}$, without requiring graph partitioning or distance thresholds. \n\nIn general, FINCH is more suitable for our federated domain expert identification task because:\n\n• **Parameter-Free Operation**: Unlike K-means (requires pre-specifying k clusters) and DBSCAN (requires ε and MinPts hyperparameters), FINCH automatically determines optimal clustering without any hyperparameter tuning, making it robust across diverse federated scenarios.\n\n• **Adaptive to Unknown Domain Numbers**: In federated settings, the number of domains is typically unknown and varies across deployments. FINCH naturally adapts to this uncertainty, while K-means fails without knowing k and DBSCAN requires extensive parameter search.\n\n• **Empirical Validation**: Our experiments in Table.1 (Page 7) also validate FINCH is more suitable: 50.57% avg accuracy vs. K-means (46.48%) and DBSCAN (43.62%) on Office31, confirming its effectiveness for domain expert identification."}, {"review_id": "u2jveBF6mu", "reviewer": "Reviewer_rkk9", "summary": "The paper presents DKDR. It dynamically weights forward and reverse KLD for reliable distillation and decouples knowledge to get reliable teachers. Experiments on image classification tasks show its superiority.", "questions": "Q1.\tExtreme Heterogeneity: How does DKDR perform under more severe data heterogeneity? Could the authors provide additional experiments to explore this?\nQ2.\tScalability: How does the method scale with a larger number of clients? Are there optimizations to handle such scenarios efficiently?", "limitations": "See weaknesses above.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nS1.\tInnovative Approach: The dynamic weighting of forward and reverse KLD based on knowledge discrepancies is a creative and effective solution to mitigate distillation bias in multi-domain FL, as supported by theoretical and empirical analysis.\nS2.\tEasy to follow: The paper is well-written and easy to follow. Moreover, the precise description of modules with detailed exquisite framework visualization makes the overall work clear.\nS3.\tWell-motivated: The proposed method appears well-motivated and effectively addresses the identified challenges in federated learning.\nWeaknesses:\nW1.\tComputational Overhead: The use of SVD and clustering introduces additional computational complexity, which may be burdensome for resource-constrained clients, though this is acknowledged in the limitations.\nW2.\tLimited Heterogeneity Extremes: The paper does not explore performance under extreme data heterogeneity, which could further demonstrate robustness.\nW3.\tScalability Concerns: Experiments are limited to 20 clients; scalability with hundreds or thousands of clients, common in real-world FL, is not addressed.", "was_revised": true, "final_justification": "The authors has addressed most of my previous concerns. By considering with other reviewers' comments, I decide to raise my score to 6.", "rebuttal": "## Response to Reviewer rkk9\n\nDear Reviewer rkk9,\n\nThank you very much for your positive and constructive review. Your acknowledgment of our technical soundness and high impact are deeply appreciated. We hope our responses below will further strengthen your confidence in our work.\n\n---\n\n### Weaknesses\n\n>**`W1: Computational Overhead of DKDR.`**\n\n**A1:** Sorry for the misunderstanding. We want to clarify that the **computational overhead occurs primarily on the server side, not on resource-constrained clients**. We will fix it in the final version. Here is the detailed breakdown:\n\n**Client-side Operations**: Clients only perform standard local training and knowledge distillation using the dynamic weighting mechanism. The dynamic weight calculation ($γ\\_s/(γ\\_s+γ\\_l)$ and $γ\\_l/(γ\\_s+γ\\_l)$) involves simple arithmetic operations with negligible overhead.\n\n**Server-side Operations**: SVD decomposition and FINCH clustering are performed once per communication round on the server, which typically has sufficient computational resources in federated learning scenarios.\n\n**Quantitative Analysis**: We conduct overhead analysis on the OfficeCaltech dataset, as shown in the following table.\n\n*Table. **Comparison of Time and Memory Usage** in OfficeCaltech dataset.*\n\n| Method   | Client Time (s) | Server Time (s) | Communication Cost (mb) | Accuracy (%) |\n| -------- | --------------- | --------------- | ----------------------- | ------------ |\n| FedAVG   | 2.70            | 0.02            | 2.83                    | 77.94        |\n| FedProto | 14.13           | 0.01            | 2.83                    | 79.26        |\n| Scaffold | 9.23            | 0.02            | 8.48                    | 75.32        |\n| **DKDR** | 6.28            | 0.11            | 8.25                    | **81.14**    |\n\n\n\n\n>**`W2 & Q1: Data Heterogeneity Ablation.`**\n\n**A2:** Excellent point! We conduct additional experiments under extreme heterogeneity settings. We use Dirichlet parameter $ζ=0.05$ (much smaller than our original $ζ∈{0.1,0.3,0.5}$) to create extremely non-IID scenarios in Cifar-10:\n\n*Table. **Ablation study of ζ in Cifar-10.** The smaller ζ is, the more imbalanced the local distribution is.*\n\n| Method       | ζ=0.5 (Original) | ζ=0.05 (Extreme) |\n| ------------ | ---------------- | ---------------- |\n| FedAvg   | 80.34            | 64.22            |\n| FedProto | 81.49            | 60.84            |\n| Scaffold | 81.40            | 44.69            |\n| **DKDR**     | **82.63**        | **66.73**        |\n\nDKDR shows consistent improvements under extreme heterogeneity, demonstrating that our dynamic distillation mechanism is effective when dealing with severe data heterogeneity. This validates the robustness of our approach.\n\n\n\n>**`W3 & Q2: Client Scalability Discussion.`**\n\n**A3:** Thank you for this crucial scalability question. We conduct extensive scalability experiments on the OfficeCaltech dataset with varying client numbers:\n\n*Table. **Ablation study of client Number $K$** on OfficeCaltech dataset.*  DKDR achieves consistent better performance compared to three baselines in three settings.\n\n| Client Number $K$ | 20        | 60        | 100       |\n| ----------------- | --------- | --------- | --------- |\n| FedAvg            | 77.94     | 72.33     | 61.93     |\n| FedProto          | 79.26     | 68.77     | 49.32     |\n| FedProx           | 78.28     | 72.46     | 63.96     |\n| **DKDR**          | **81.14** | **74.39** | **64.47** |\n\nFor large-scale deployment, we can perform knowledge decoupling every few communication rounds rather than every round, reducing server-side overhead."}, {"review_id": "mMlocHgfCJ", "reviewer": "Reviewer_YmgD", "summary": "To address the issue of unreliability in knowledge distillation within federated learning, this paper proposes DKDR, a federated learning framework with two key innovations: dynamically weighting forward/reverse KLD and knowledge decoupling for domain-experts identification. Extensive experiments across various federated settings demonstrate DKDR’s superior performance over SOTA methods.", "questions": "The authors mention that prevalent distillation solutions primarily minimize forward KLD to fit global model. Can you elaborate on why it results in significant distillation bias?", "limitations": "See weakness.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The proposed method of federated KD appears interesting and innovative.\n- The paper presents a clear and concise writing style thus easy to follow.\n- The dynamic weighting of forward and reverse KLD is a novel approach to mitigate distillation bias with both theoretical and empirical validation.\n- The analysis of forward vs. reverse KLD behaviors provides a clear foundation for the method’s design.\n\n**Weaknesses:**\n- SVD and clustering add computational overhead, potentially limiting scalability on resource-constrained devices.\n- The method assumes distinct domains, which may falter with significant overlap.\n- The work lacks analysis of model performance with different local training epochs for each communication round.", "was_revised": true, "final_justification": "Thank you for your comprehensive and thoughtful response to my concerns. I appreciate the effort you’ve put into addressing each point with detailed explanations, quantitative analyses, and additional experimental results.\n\nYour clarification on computational overhead effectively addresses my initial worries about practical feasibility. The quantitative evidence showing that computational and communication costs remain within acceptable ranges, even with varying client numbers, is particularly convincing.\n\nI also find your analysis of domain overlap scenarios compelling. The explanation of how hierarchical clustering and dynamic distillation mechanisms enable graceful degradation, combined with ablation results demonstrating the resilience of the DKD module, helps alleviate concerns about the method’s robustness in real-world heterogeneous settings.\n\nFinally, your theoretical and empirical explanation of why forward KLD introduces distillation bias in multi-domain scenarios clarifies a key motivation for your work, enhancing the depth of your contribution.\n\nBased on these thorough responses and the additional evidence provided, I am pleased to increase my score. I encourage you to incorporate these clarifications and experimental details into the final version of the paper to strengthen its overall impact.", "rebuttal": "## Response to Reviewer YmgD\n\nDear Reviewer YmgD,\n\nThank you very much for your thoughtful and thorough review. We are encouraged by your recognition of our innovative approach to federated knowledge distillation. We appreciate your careful examination of our work and hope our detailed responses below will address your concerns.\n\n---\n\n### Weaknesses\n\n>**`W1: Computational Overhead Discussion.`**\n\n**A1:** Thank you for this important practical concern. We want to clarify the computational distribution and provide quantitative analysis:\n\n**Computational Distribution:**\n- **Client-side overhead**: \n    - Only involves simple arithmetic for dynamic weight calculation: $\\frac{\\gamma\\_s}{\\gamma\\_s+\\gamma\\_l}$ and $\\frac{\\gamma\\_l}{\\gamma\\_s+\\gamma\\_l}$\n    - No SVD or clustering operations on client devices\n- **Server-side operations**: \n\t- SVD and FINCH clustering perform once per communication round\n    - Typical federated learning servers have sufficient computational resources\n\n**Further Analysis:**  \nWe conduct experiments in OfficeCaltech to further analyze it, as shown in the below table.\n\n*Table. **Comparison of computational and communication costs** in OfficeCaltech dataset.*\n\n| Method   | Client Time (s) | Server Time (s) | Communication Cost (mb) | Accuracy (%) |\n| -------- | --------------- | --------------- | ----------------------- | ------------ |\n| FedAVG   | 2.70            | 0.02            | 2.83                    | 77.94        |\n| FedProto | 14.13           | 0.01            | 2.83                    | 79.26        |\n| Scaffold | 9.23            | 0.02            | 8.48                    | 75.32        |\n| **DKDR** | 6.28            | 0.11            | 8.25                    | **81.14**    |\n\nWe also conduct experiments on varying client numbers, as shown in the following table. The computational and communication overheads are within an acceptable range, and the performance improvements are worthwhile.\n\n*Table. **The relationship of client Number $K$ and computation cost** in OfficeCaltech dataset.*  \n\n| Client Number $K$ | Server Processing Time (s) | Improvement (Compared to FedAVG) |\n| ----------------- | -------------------------- | -------------------------------- |\n| 20                | 0.11                       | +3.20%                           |\n| 60                | 0.42                       | +2.06%                           |\n| 100               | 2.94                       | +2.54%                           |\n\n\n\n>**`W2: Discussion on Domain Overlap Scenarios.`**\n\n**A2:** Thank you for highlighting this important assumption. We acknowledge that significant domain overlap poses challenges to our knowledge decoupling approach. However, we have conduct analysis showing that DKDR exhibits **graceful degradation** rather than catastrophic failure in such scenarios:\n\n**Robustness Under Domain Overlap:**  \nWhen domain boundaries become unclear due to significant overlap, our method handles this situation through multiple mechanisms:\n1. **Hierarchical Clustering Resilience**: FINCH clustering naturally accommodates overlapping structures by creating hierarchical representations, allowing it to identify meaningful groupings even when domains are not perfectly distinct.\n2. **Fallback to Dynamic Distillation**: Even when knowledge decoupling produces suboptimal domain experts due to overlap, the **Dynamic Knowledge Distillation (DKD) component continues to provide substantial benefits**. The dynamic weighting mechanism adapts to knowledge discrepancies regardless of domain expert quality.\n3. **Graceful Performance Degradation**: In worst-case scenarios where clustering completely fails to separate domains, DKDR essentially reduces to an enhanced federated distillation method that still outperforms standard approaches due to the DKD component.\n\n**Empirical Validation:**  \nOur ablation studies (Sec.4.4, Page 9) demonstrate that even when the KDP module is disabled (✗ KDP), the DKD module alone (✓ DKD) still provides consistent improvements across all datasets, confirming the method's resilience to domain overlap issues.\n\n\n\n>**`W3: Abalation Study of Local Epoch.`**\n\n**A3:** Excellent point! We conduct additional experiments analyzing the sensitivity of DKDR to local training epochs:\n\n*Table. **Ablation study of Local Epochs T** on Digits dataset.*  DKDR achieves consistent better performance compared to three baselines (FedAVG, FedProto, Scaffold) in 4 of 5 settings.\n\n| Local Epochs (T) | FedAvg | FedProto | Scaffold  | DKDR      |\n| ---------------- | ------ | -------- | --------- | --------- |\n| T=1              | 63.99  | 63.56    | 53.43     | **64.53** |\n| T=3              | 64.32  | 63.24    | 61.13     | **65.36** |\n| T=5              | 62.85  | 62.84    | **67.04** | 66.83     |\n| T=10             | 60.13  | 61.90    | 65.20      | **67.24** |\n| T=20             | 58.56  | 60.90    | 65.62     | **65.73** |\n\n\n\n---\n### Questions\n\n>**`Q1: Why minimizing forward KLD results in significant distillation bias?`**\n\n**A1:** Excellent question that goes to the heart of our motivation! The significant distillation bias from forward KLD arises from its **mean-seeking behavior** in multi-domain federated scenarios. Let us elaborate with both theoretical and empirical perspectives:\n\n**Theoretical Foundation:**  \nForward KLD aims to minimize: $KL[Z^{t-1}||Z^t\\_{k}] = \\sum\\_j z^{t-1}\\_j \\log\\frac{z^{t-1}\\_j}{z^t\\_{j,k}}$\nFrom the gradient analysis (Eq. 8a): $\\frac{\\partial L\\_{fkd}}{\\partial z^t\\_{j,k}} = z^t\\_{j,k} - z^{t-1}\\_j$\n\nThis shows that **larger $z^{t-1}\\_j$ values receive proportionally higher weights** in the loss function. When the global model produces multi-peaked distributions (common in heterogeneous FL), the mean-seeking behaviour of forward KLD forces the local model $z^t\\_{j,k}$ to cover all peaks of the global distribution $z^{t-1}\\_j$, including low-density regions between peaks.\n\n**Concrete Example:**  \nConsider a scenario where:\n- Client A specializes in cats (peak at class 3)\n- Client B specializes in dogs (peak at class 7)\n- Global model after aggregation shows peaks at both classes 3 and 7\n\nWhen Client A uses forward KLD to distill from this global model, forward KLD forces Client A to assign probability to both cats and dogs. This creates **unreasonable high probabilities in low-density regions** (the valley between peaks), which results in bias. \n\n**Empirical Evidence:**  \nOur experiments (Figure 2) demonstrate this phenomenon. Forward KLD shows higher knowledge discrepancies in AKCs (lower-probability regions) while reverse KLD shows higher discrepancies in DKCs (dominant regions), which validates that forward KLD struggles with multi-peaked distributions."}]

Author rebuttal:

Meta-review: The paper propose a novel method to address reliability issues of knowledge distillation (KD) in FL. In particular, the paper highlights two key issues in federated KD: unreliability of the distillation path and unreliability of the teacher model. To address these, the proposed method incorporates two novel components: (1) a dynamic knowledge distillation (DKD) module, which divides knowledge into dominant knowledge components (DKCs) and auxiliary knowledge components (AKCs); (2) a knowledge decoupling (KDP) module, which decomposes knowledge into shared knowledge and unique knowledge. The latter is achieved via using SVD filtering and clustering techniques to identify domain experts, enabling clients to acquire knowledge from reliable domain experts rather than global models. The proposed method is evaluated comprehensively on multiple datasets with additional ablation studies confirming the effectiveness of the two core components, and providing hyper-parameter sensitivity analysis.

The original reviews of this paper are mostly positive. The reviewing panel recognizes the following key strengths of the paper: (1) novel theoretical analysis of forward/reverse KLD behaviors in federated learning, providing systematic insights into distillation pathway reliability (CP1N, YmgD, rkk9); (2) innovative dynamic weighting mechanism adapting to knowledge discrepancies, effectively mitigating distillation bias with theoretical and empirical validation (Wn6p, bzDB, YmgD); (3) comprehensive experimental validation across multiple datasets demonstrating consistent improvements (bzDB, CP1N).

The reviewing panel also raised several concerns/questions on (1) computational overhead, (2) theoretical foundation, (3) cross-modal generalizability. Most of these questions/concerns are addressed sufficiently by the rebuttal and/or during reviewer-author discussion. This leads to a significant increase in rating, strengthening the acceptance consensus. As there is no further concern, I would like to recommend acceptance for this paper.
Decision: Accept (poster)