Rethinking the Temperature for Federated Heterogeneous Distillation
Authors: Fan Qi
Daxu Shi
Chuokun Xu
Shuai Li
Changsheng Xu
Venue: ICML 2025 poster (2025)
Keywords: Heterogeneous Federated Learning
Logits Distillation
Primary area: social_aspects->privacy
TL;DR: Dynamic Temperature Design for Federated Heterogeneous Logits Distillation.

Abstract: Federated Distillation (FedKD) relies on lightweight knowledge carriers like logits for efficient client-server communication. 
Although logit-based methods have demonstrated promise in addressing statistical and architectural heterogeneity in federated learning (FL), current approaches remain constrained by suboptimal temperature calibration during knowledge fusion.
To address these limitations, we propose ReT-FHD, a framework featuring: 1) Multi-level Elastic Temperature, which dynamically adjusts distillation intensities across model layers, achieving optimized knowledge transfer between heterogeneous local models; 2) Category-Aware Global Temperature Scaling that implements class-specific temperature calibration based on confidence distributions in global logits, enabling personalized distillation policies; 3) Z-Score Guard, a blockchain-verified validation mechanism mitigating 44\% of label-flipping and model poisoning attacks.  Evaluations across diverse benchmarks with varying model/data heterogeneity demonstrate that the ReT-FHD achieves significant accuracy improvements over baseline methods while substantially reducing communication costs compared to existing approaches. Our work establishes that properly calibrated logits can serve as self-sufficient carriers for building scalable and secure heterogeneous FL systems.

Rethinking the Temperature for Federated Heterogeneous Distillation
Fan Qi Daxu Shi
Chuokun Xu
Shuai Li'
Changsheng Xu"
Abstract
across devices), statistical heterogeneity (non-IID data dis- tributions across clients), prohibitive communication costs, and privacy risks due to exposure of sensitive visual data. Integrating knowledge distillation (KD) into FL frameworks addresses these issues by enabling architecture-gnnssti knowledge transfer from heterogeneous local models to a lightweight global model, minimizing direct data transmis- sion, and preserving data privacy through indirect gradient- free knowledge aggregation.
Federated Distillation (FedKD) relies on lightweight knowledge carriers like logits for efficient client-server communication. Although logit-based methods have demonstrated promise in addressing statistical and architectural het- erogeneity in federated learning (FL), current approaches remain constrained by suboptimal temperature calibration during knowledge fusion. To address these limitations, we propose ReT-FHD, framework featuring: 1) Multi-level Elastic Temperature, which dynamically adjusts distillation intensities across model layers, achieving optimized knowledge transfer between heterogeneous local models; 2) Category-Aware Global Temperature Scaling that implements class-specific temperature calibration based on confidence distributions in global logits, enabling personalized distillation policies; 3) Z-Score Guard, blockchain-verified validation mechanism mitigating 44% of label-flipping and model poisoning attacks. Evaluations across diverse benchmarks with varying model/data heterogeneity demonstrate that the ReT-FHD achieves significant accuracy improvements over baseline methods while substantially reducing communication costs compared existing approaches. Our work establishes that properly calibrated logits can serve as self- sufficient carriers for building scalable and secure heterogeneous FL. systems.
Early federated knowledge distillation frameworks, for ex- ample, the FedKD (Jcong et al., 2018) transmit mean logit vectors from clients to a server. These vectors are averaged into global consensus outputs and redistributed to clients, where they serve as teacher signals for distillation regular- ization during local training. Addressing the model hetero- geneity, FedMD (Li & Wang. 2019) proposes the clients communicate based their output class scores on samples from the public dataset. Later, additional knowledge carri- ers are transferred for resolving model heterogeneity, such as features (He et al., 2020; Wang et al., 2024) and proxy models (Lee et al., 2024). However, transferring additional information, whether as proxy models (Wang et al., 2025) synthetic data (Huang et al., 2024; Liang al., 2024), introduces both security risks and significant computational and communication overhead.
In this paper, we investigate whether the minimal carrier of knowledge-logitt loaitse-- can effectively address model het- erogeneity in FL without public datasets and additional information. Recently, multi-level distillation, i.e., apply- ing knowledge distillation at intermediate layers, has been demonstrated to enhance both the comprehensiveness and coherence of knowledge transfer (Khan et al., 2024; Hao et al., 2024; Huang et al., 2025b). However, the temperature, which is a critical hyperparameter distillation, is typically simplified static or monotonically decreasing (Hao et al., 2024; Huang al., 2025b). This temperature strategy fails to adequately address the intricacies of data and model het- erogeneity in FL, significantly limiting the efficiency and flexibility of distillation across heterogeneous models.
Introduction
In cross-device Federated Learning (FL), critical challenges arise from model heterogeneity (c.g., diverse architectures
'Scoool of Computer Science and Engineering, Tianjin Univer- sity of Technology, Tianjin, China "School of Computer Science and Engineering, Southeast University. Nanjing, China IInstitute of Automation, Chinese Academy of Sciences, Beijing, China. Correspondence to: Chuokun Xu <chuokunxu @wu.. en..
Therefore, we rethink the temperature for distillation and propose ReT-FHD framework federated heterogeneous learning from theoretically and empirically: 1) Model Het- erogeneity: We incorporate the elastic temperature that as-
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Rethinking the Temperature for Federated Heterogeneous Distillation
signs flexible temperatures to multi-level distillation. Theo- retically, temperature scheduling balances staged logit en- tropy, enhancing distillation precision and inter-stage knowl- edge transfer. 2) Data Heterogeneity: We present global temperature scaling, which assigns category-specific temper- atures, aligning with gradient disparity dynamics to mitigate class-wise update imbalances. Analysis shows this stabi- lizes convergence by harmonizing gradient variations. 3) Security: Integrated with Mlocchhin-eemooeeeed FL (BC- FL), our framework employs Z-score verification to validate logit distributions against dynamic boundaries, enabling au- tomated reward/punishment protocols that deter malicious behaviors. 4) Experiments across four benchmarks demon- strate ReT-FHD's s superiority over centralized and decentral- ized (DeL) baselines.
eter communication. Zhang et al. (2024a) propose FedTGP, framework using Adaptive Boundary Contrastive Learn- ing (ACL) to learn trainable global prototypes (TGP) on the server, improving prototype separabilityy while main- taining semantic meaning. On the other hand, the prexy model method focuses on training a compact model at each participant as a "proxy" for the primary model. Shen et al. (2020) propose FML, a federated learning framework en- abling joint training of universal model and personalized models, allowing custom designs for diverse applications beyond standard FI. limitations. Wu et al. (2022) propose FedKD, a federated learning framework enabling mutual learning of client suudett-teclle models and student model sharing for knowledge transfer, alongside a dynamic gradi- ent approximation method to reduce communication costs. Moreover, Wang et al. (2025) propose FedTTpe, a bidirec- tional knowledge distillation strategy to optimize private and proxy models jointly, along with an uncertainty- -basd behavioral imitation method to improve proxy model guid- ance.
2 Preliminaries
2.1 Federated Distillation
Federated Distillation (FD) research addresses three core challenges: 1) Data Heterogeneity: Orthogonal efforts fo- cus on refining knowledge carriers for non-llD data. Itahara et al. (2021) mitigate statistical heterogeneity via entropy reduction aggregation, while Zhang et al. (2022) synthe- size hard samples to amplify KD effectiveness in data-free FL. 2) Model Heterogeneity: Foundational works include FedMD (Li & Wang. 2019), which aligns logits from public datasets for consensus, and FedDF(Lin al., 2020), which ensembles soft labels from unlabeled data. FedGKT (He et al., 2020) further bridges model capacity gaps by distill- ing compact client features into large server model. 3) Security: Chang et al. (2019) minimizes leakage via robust mean estimation on public predictions, Gong et al. (2022) employs quantized noise to perturb shared logits.
While the aforementioned methods address the challenges of Heterogeneous Federated Learning, the resulting increase in communication and computational demands poses sig- nificant limitations to the practical deployment of federated systems.
2.3 Dynamic Temperature Knowledge Distillation
Dynamic temperature strategies in knowledge distillation (KD) can be broadly divided into two categories: Global scheduling methods and Local adaptive methods. Global scheduling methods adjust a single temperature to control overall distillation difficulty. Liu et al. (2022) propose to learn the temperature by performing meta-learning on the extra validation set. Li et al. (2022b) dynamically adjust a shared temperature via curriculum learning to progressively increase distillation difficulty from easy to hard. Local adaptive methods assign temperatures at finer granularity. Guo et al. (2023) adjust sample-wise temperatures based on teacher confidence, while Wei & Bai (2024) minimize sharpness differences between teacher and student logits to derive separate temperatures for each.
To defend model poisoning attacks, Khan et al. (2024) pro- pose HYDRA-FL by offloading some of the KD loss shallow layer via an auxiliary classifier for local-to-local dis- tillation. In contrast, our framework transfers logits instead of model parameters, introducing additional challenges due to compounded heterogeneity in both data distributions and model architectures.
Although these methods emphasize the effect of temper- ature on the distillation process, they are not suitable for knowledge distillation of heterogeneous models in FL by multi-client non-iid data.
2.2 Model Modell-Heteoogeneuur Federated Learning
For generalized scenarios of heterogeneous models, two mainstream methods are Prototype-based methods and Proxy model methods. The prototype-basee framework primarily facilitates knowledge transfer through client col- laboration in constructing unified class prototype. Tan et al. (2022) propose FedProto, a prototype- based method that addresses heterogeneity challenges in FL. Clients com- municate local abstract class prototypes, which the server aggregates into a global prototype, eliminating model param-
3 Proposed Method
3.1 Background
Multi-level Knowledge Distillation: Compared to the orig- inal Federated Distillation approach (Li Wang, 2019), Hy- brid Knowledge Distillation (Khan al., 2024; Hao et al.,
2

---

Rethinking the Temperature for Federated Heterogeneous Distillation
2024) has been shown significantly enhance robustness and accuracy in Federated Learning. This improvement is achieved by integrating an auxiliary knowledge distillation constraint within shallow layer of the local model.Thr hy- brid loss function for FL. clients is formulated as following:
and model heterogeneity necessitate finer temperature reg- ulation enhance the utilization effectiveness of global logits.
To address model het- erogeneity, we dynamically adjust the temperature during knowledge distillation based on the hierarchical charac- teristics of the models, effectively bridging representation discrepancies across architectures different levels. Our multi-level distillation framework employs rrge-adaptivve temperature scaling: each level's output features feed into the subsequent level, enabling dynamic temperature calibra- tion based on the statistical divergence between successive stages. Specifically, if the Z-score difference between the logits of the next stage and the logits of the teacher's 's model is large, the temperature of the current stage should be in- creased to reduce the amount of knowledge in the logits of the current stage, and thus promote learning more infor- mation from the logits of the teacher's 's model. Conversely, when the difference in logits is small, lower the temperature so that the model focuses more on learning in other phases.
Multi-level Elastic Temperature: :
C LOE (Ne, y) + Σ KL (o (2(" 28 (T) (2
(1)
Here, a denotes the softmax function, 2 denotes the logits of the student model in layer 1, ( denotes the logits of the teacher model in layer 1. T is called temperature and serve as the offsets of the softmax function, a and effectively adjusts the sharpness or softness of the probability distri- bution derived from the teacher's s logits. The loss function contains two main elements:
LCe Scross-Entropy Loss) is used to optimize the clas- sification ability of the client model for good task per- formance.
We rewrite 7i in level as Ti = Gi(AZ), AZ ZT Zi+I Il. ZT is the Z(-) distribution of the global logits. AZ represents the difference in Z-Scores between the global logits and the local logits, which is crucial for adjusting the temperature of the distillation process. The modulation of the effect of AZ is achieved through the function G. defined as a logarithmic function, as follows:
Eie ci. (Multi-level Distillation Loss) is used to calculate the KL divergence between the teacher logits and the student logits at each level I to ensure that knowledge at different levels is adequately transferred.
Sun et al. (2024) propose a logit Z- Score standardization as a pre-process to enable student to focus on essential logit relations from teacher rather than requiring a magnitude match. To bridge the capacity gap of heterogeneous clients, we rewrite softmax based on the mean v and variance Var (+) as follows:
Z-Score Distillation:
Gi(-) E 7 7log(1 (1 + AZi)
log (1 + A7
where y is scaling factor that determines the sensitivity of the function, & is the lower bound of the temperature ad- justment represented by the initial value of the temperature, and A 7 is used as a normalisation factor to determine the sensitivity of the function.
exp 3 (z; (z)
(2;2, Var (z))
(2)
ΣW exp 2 (z: (2i))
Category-Awars Global Temperature Scaling: To ad- dress data heterogeneity, we calculate distinct temperatures for each category in the global logits, enabling flexible adap- tation to diverse client data characteristics, enhancing robust- ness and inter-category fairness while mitigating challenges from heterogeneous data distributions. Here, we give the formula for adjusting the temperature of the e-th category of global logits:
2(0) -
s.1. 2 (2)
Var (z)T
where z(o) represents the e-th term of z, which is the e-th class specific logits value, and Z is the Z-score function.
3.2 Rethink the Temperature in Heterogenous Federated Distillation
CCB(f(E)) - Ei, Ccr(f(E))
f. = F. - B. max (Cco(((Te)) - / DE LoE(s (Z.))
Despite the effectiveness against poisoning attacks (Kundu et al., 2021; Khan et al., 2024), the temperatures 7 in Eq.1 are fixed, which neglects the inherent data and model hetero- geneity present in the distillation process. Typically, higher T makes the distribution sharper (more confident), while lower T makes it softer (less confident), thus influencing the gradient and the learning process of the student model. However, in heterogeneous federated distillation, both data
(4)
where 3 controls the weight of the temperature update, maæ(I-I) represents the absolute maximum value taken to control the stability of the update. Since cross entropy di- rectly reflects the robustness of the category, it is chosen to regulate the temperature update.
3

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Algorithm 1 Algorithm of our ReT-FHH
3.3 Different FL Modes of Our ReT-FHD
Input: Dataseti D., D2...., Dx ). T b., Maximum communication rounds 1: Initialization: W <- Worker, 1/ <- Validater, M M-ineer Global logits =9,
The ReT-FHD framework supports distributed training through multilevel elastic distillation and statistical logit analysis, enabling intrinsic security and scalable FL deploy- ment.
Blacklist 13, Client neighhors set N, Honest server S. Client reward set R.
2: for
T do
3: each & K Randomly(W, M,
each & K. (W)
4: for each [W] do
5: or downlonds =f- from blockchain - do local update with Eq..6
FL and DeL. mode: In the Algorithm 1, the situation will not be so complicated for federated or decentralized learn- ing. Each node belongs to the E W. For FL mode, each client computes a multilayer distillation loss function with the server-side global logits, highlighted in yellow. For Del. mode, multilayer distillation training is performed after veri- fying the legitimacy of neighboring node logits, highlighted in pink.
6: ->sends x;" to E V
7: downleads =f from & and update with Egg55++ sends =;" to S
8: gathers +;_; from N
9:
end for
10:
for each V w W or S do
evallates W with = by Eq.7 11:
12:
- Auto set r for each W by Eq.8
13:
sonds
M
BC-FL mode: Following the blockchain FL et al., 2021; Ying et al., 2024), we apply the POS voting- based consensus mechanism, highlighted in purple of Algorithm 1. In each round t, each & E K. is randomly assigned role, including w E WW E V.n E .,, where [W|+[V|||M] Kg. w is responsible for performing computations related to f." Each w download global logtis 2; from the latest block and train with private local data D,. At this point the w node performs the F. round local update and broadcasts the logits x corresponding the number of dimensions of the dataset categories, to the E V in the network. See the Appendix for more details.
14:
S evaluates
Fq.7
15:
+ set 1%2 each w by Eq.8 - gather Re4 by Tia E R
16:
ovaluates si" by Eq.7 set rw for each n E Nu. by Eq.8
17:
+ gather by r E R
18:
do lncal update Fq.6-> sends 2i" to v. E Ni..
19:
end for
20:
aggregates 2, according to the weight R. makes by Σi_.
21:
mines block + uploads B, R. blockchain
22:
end for
Standardized Finding of Malicious Logits: ( plays a crucial role in evaluating logits for hetero- geneous client models. This property likewise suggests that ((;-) can also play an important role in blockchain authentication sessions. In the verification process, validators V (verifiers) train locally and upload logits, while examinees {(validators) evaluate V's logits. Under data heterogeneity, label sets across nodes may partially overlap or diverge. We define the labeling relationship among nodes as: \V n H| = n,, IE n H| = ø. and IV n El = v. Here, H. represents the complete set of labels for a distributed classification task.
Z-Score Guard:
At the beginning of each training round, the client receives the global logits and then first adjusts the global logits ac- cording to the temperature i calculated at the end of the previous round.
Local update: Ultimately, the FL optimization function can be expressed as:
min
D
E
LCE + ΣΣ
(1) KL (o (1) (0))))()))))))))))))))))))))(( ()) c=1
D
S.I. a a-() a (24; Ee, Var (z+), G(AZ), fc),
o² (²) (Zxi E. Var (z.). G(AZ),
The importance of v. 11 and & arises particularly if E acts as a lazy node by re-uploading 2(2g. -), which impedes the verification process by V. To facilitate this verification, we suggest plotting both 2(2y, ()() and 2(zv, r)(w) within a coordinate system:
(5)
where ig is the isolated temperature for category e in the global logits.
The DeL optimization function can be expressed as:
(VZZ exp ( (AAZ
exp
Dg
H Σ exp 1=1
((ZZ),)
0111 = 0 Σ
kL (0
2 2(2g1 (v)
+ Z (2 7) (0)
JEN(() 1=1
(6)
2 (26,,7) (1)
)
s.t. d o()) ~ o (Zt: &, Var (zz), F),
> 2
exp
2
1=1
(-) ~ o (Zai Var (z.). G(AZ).)
s.f. A Az = 2 (24,7) - 2 (sb,T))
0.8

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Here is a fixed temperature. This approach allows us to quantify the criteria necessary for verified individual to be considered valid or to be subject to voting, based on their positional relationship within the coordinate system.
knowledge scale, and the more local logits can learn from the global logits that can be learned.
In order to alleviate the problem of uneven distribution of category information due to data heterogeneity, we further introduce Global Logits Scaling, but before that, we need to When -2), equilibrium between ((vv:) and (zg.)) :) is reached, v will evaluates highest scores and vote bad.
discuss the effect of the previous Multi-level Elastic Tem- perature on categories.
Logits-based Secure Validation: Logits, as crucial medium for information transmission, can also indicate the confidence level of a sample, in particular, the confidence level can demonstrate the knowledge of the model and also reveal information about the security of the model, which can be solved by looking at the logits relationship in the distributed system. We dynamically assign rewards based on this confidence. Where Rd is the base reward, || represents the number of samples per client, is the preset confidence range, provide a base algorithm for measuring reward values as follows:
Corollary 4.2. For a model with random distribution I in the initial state, for a deep neural network (DNN) model f with random distribution I in the initial state, the model has preference for any sample. Thus for the model's predic- rive distribution f(z; 01) = (pp.. P2..P2...... ........ the following relationship exists:
Epp' (x)) = E(pl(x)) = E(pá(x)) = =... =..==E(p''aa))
(10)
RB --ill [a = x) max( > Rs ||]]]] (1 max ( Ply i x))),, otherwise
We assume that the model is optimised under the guidance of a soft label distribution with smoothing degree λ1 which is satisfied:
R-
4 Theoretical Analysis
(21)
p2 >1 (222 = =...= =p (Ix) 1 C
In this section, we systematically demonstrate the effective- ness of the proposed Multi-level Elastic Temperature and Global Temperature Scaling mechanism from the perspec- tives of information entropy and gradient updating.
p (Bef1) = 122 (ZeØ2) = ................................ ... (IEtpc)
(11)
The following relationship exists for the partial derivatives of the knowledge scale for different classes of logits:
Referring to Zhao et al. (2024), we introduce the knowledge scale to quantify the level of knowledge of the predictive distribution, which can measure the scale of knowledge learnt from the probability distribution by the model with randomly initialized weights:
(K; (z)
(K (12)
(z,(F)
Corollary 4.2 shows that the overall change in knowledge scale has balanced impact on different categories. The proof of Corollary 4.2 can be found in Appendix A.2.
KT = KL(T(>), T(x)) = log C H(P")
(9)
where KL(-) stands for Heilback-Leibllee divergence, which calculates the cost of information required travel from one distribution to another, C' is the total number of categories and is constant for the same dataset, and (-) is the information entropy.
Now we can discuss the theory of Global Logits Scaling, Here, we give Theorem 4.3 about the relationship between the category temperature of global logits and the category gradient update of local models.
From the definition of knowledge scale, it can be found that the information entropy can be directly used to mea- sure the knowledge level of teachers. Therefore, we obtain the following theorem by studying the partial derivative of information entropy with respect to temperature.
Theorem 4.3. Following the setup in Corollary 4.2, the lemperature t. of category e is positively correlated with the gradient update Vi-a of the local model category c.
The proof of Theorem 4.3 can be found in Appendix A.3. Theorem 4.3 states that different temperatures affect the gradient update. Increasing the temperature of category will cause the gradient update Vý, of that category to increase, and vice versa will cause the gradient update J~~ of the category to decrease, and by regulating the temperature of different categories, it will cause the gradient update of different categories to be more balanced.
Theorem 4.1. Temperature is positively correlated with the information entropy of the predictive distribution of logits.
The proof of Theorem 4.1 can be found in Appendix A.1. Therefore, combining Theorem 4.1 and the definition of knowledge scale, the larger the temperature, the smaller the
5

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Table 1. Testing accuracy (%) in FI. model heterogeneous (a.) and homogeneous (b.) settings. Dir-ser: Dirichlet distribution; Pat-ser: Pathological distribution.
Tiny ImageNet
Flower
Cifar- 100
Cifar- 10
Methods
Dir-set
Pat- xet
Dir set
Pat-set
Dir-set
Pat-set
Dir set
Par sel
FedProto
14.68 12.61
24.91
21.68
55.12
21.71:3.00
32.00s2
60. 48
25.8
Proto-based
FedTGP
14.94
10.56 10.56
27. 15
08.7621.42
26, 26.24=0.72
62. 08
68. 08. 000
89,00+1.25
RethinkFL.
15.21
38 38.4710.0
30.9611.1
65.24
28.37 Lo.33
63.11 1
68.21
88.9619.21
FML
14.59-0.54
35.74+1.a
21.63+1..7
52.27 +1.31
21.20-0.ca
19. 19.55+1.2
61.57+1.0r
82. 22.41
(a.)
Proxy-models
FedKD
14.57 0.66
32. 32.07
20.76 20.76
52.71.1.43
10.4
51.58 33
63.32 oe
13.73
FodType
16.05-0.44
38.8 81
23. Se
60.89+1.24
21.21=0.su
52. 52.51+1.34
66. 110.13+...0 as
81.288+0.52
HYDRA-FI.
30.87a1.00 14.
55.18 18
20.82.1.a ai
48.5010
61.74127
79.37
FedDistil
15.20
37. 37.67=1.
27.47+1.a1
02.27
21.04-0.57
63.72+1.15
63.51+1 os
83.08 10.64
Logets-based
FCCL
15.49
38.0110.4
25.20 03
65.61.11.17
22.72.10.6
58.621.0
64.29k1 37
86.58
ReT-FHD(Ours, FL)
17.24
42 35.35
33. 13.16
72.91
30.77
68. 08..66...
72. 70..77
91. 23
FedProto
31.48-1.64
37. 77
53. 13.14
16.35
48.01
13.49
86.32
85.04.0.45
Proto-based
FedTGP
31,08=1.0
39 20.92+
58 27
72.66at.12
19.32=0.83
64. 11.16+0
88. 18.38+112 21
89.10
RethinkFI.
35.04_a.7
37.5011.1
67.9521.55 .s
48.34_0.m
63.9011
89.3311 06
38.35.11.91
FML
31.13
37 37.50+0.77
53.58+1..4
66.21+1.04
45.61-oo.e
60.68+0.4
87.56+1114
87.26
(b.) mroxy-models
FedKD
33.99 1.50
37 37.9720.77
53.82
66.06
48.62.20.z
63. 13.9810046
88. 88.38 1o
19.05
FodType
31.299.0.a
10.71+1.0
60. 18
75.27+2.05
17.38=0.4 es
64. 73
89.77+1.22
88, 88.91
FYDRA-FI.
34.45_o.ss
44.621077
58.2771..5
70.6321111
51.18....
59.71
87.0410 as
85.9221.00
FedDisull
33.71-154
37.73:..00
53.67
65.37 $1.05
49.39
63.51
88.666+ 13
88. 1S.1990.0
based
FCCL
33.95
38.3611.aa
10.4710.49
69.75
51.52.0.7
64.4210.57
88.29
87.9810.15
ReT-FHD(Ours, FL)
35,13+1.22
50.99
61. 31.66e0.3
89.66+1.1
51.97+0.72
68. 38.49=0.40
90.541=1.0
89.85
FL setting: We select eight baselines, excluding Fed-
Table 2. Compared to decentralized methods in four datasets.
Proto (Tan et al., 2022), FedTGG (Zhang et al., 2024a),
Method
Cif-10
Cir-100
Flow.
Tiny.
RethinkFL (Huang et al., 2023), FML (Shen et al., 2020),
Local SGD
87 ar
55.17+2.00
62. 13+1
05.26=22.0
DPSGD
83. a
40.56a1 24
50.1 1a) se
23.57 36
FedKD (Wu et al., 2022), FedType (Wang et al., 2025),
CGA
35.6512237
35.6212.4 30.81
18.8
FedDistill (Jeong et al., 2023), HYDRA-FL. (Khan et al.,
Coll-FL
87. as
50.65
64.21
37.45
L2C
90.14+0.34
49.00+0.42
65.92+0.71
37 97.95-
2024) and FCCL (Huang et al., 2022). DeL setting: We add DPSGD (Lian et al., 2017), CGA (Esfandiari et al., 2021), Coll-FL (Zhu et al., 2023), L2C (Li et al., 2022a) and DeSA (Huang et al., 2025a) in a DL experiment. We evaluate the superiority of the algorithm by evaluating the av- erage of all client accuracy and standard deviations through- out the training process. All experiments are implemented in Pytorch (i.e. Intel(R) Xeon(R) Platinum 8176 CPU @ 2. I 10GHz with NVIDIA GeForce RTX 3090 GPU).
DeSA
84. 54.1410.1
60.1210..
66.68
38.00 28.05
RCT-FHIROORS,, Del)
07
64.1115:1
07.4815:16
38.21
Experiment
5.1
Experimental Setup
Datasets and Heterogeneity Setting
Datasets:
Cifar (Krizhevsky & Hinton, 2009),
Cifar100 (Krizhevsky & Hinton, 2009). Tiny-
ImageNet (Chrabaszez et al., 2017), and Flower102 (Nils- back& & Zisserman, 2008). All datasets are taken as 75%
Security Settings
To test the reliability of validation in a blockchain based FL setting, we simulate malicious nodes through three methods: label reversal attacks, model noising to emulate malicious models (Qin et al., 2024), and logits attacks (Khan et al., 2024), evaluating robustness under these adversarial condi- tions.
as training set and 25% as test set. Data Heterogeneûty: For each dataset, we apply two main types of Non-IID setting (Zhang et al., 2024b): 1) Dirichlet distribution, we set a= = (0.1, 0.5), it's worth noting that a is the concentra- tion parameter and smaller a corresponds to stronger data heterogeneity; 2) Pathological Setting, each client contains a fixed number of sample categories, but each client has a different sample category. Model Heterogeneity: We adopt the heterogeneous model setup from (Zhang et al., 2021), including AlexNet (Krizhevsky et al., 2017), ShuffleNetV2 (Ma et al., 2018), ResNet18 (He et al., 2016) and GoogleNet (Szegedy et al., 2015), which are randomly and evenly assigned to clients. For the DeL experimental setup, we use only 2-layer CNNs and apply two-stage elastic distillation for homogencous models (Li et al., 2022a).
5.2 Comparison with State-of-the-Art Methods
Heterogeneous model. We conduct experiments across four datasets and two data partitioning schemes, with re- sults detailed in Tab. Our method consistently outper- forms others by at least 2% across all experimental settings, with a 6.01% accuracy gain over FedTGP on Flower102 (Dirichlet 0.5). This highlights our temperature strategy's effectiveness in optimizing client gradient descent and en- hancing knowledge transfer across heterogeneous models. Moreover, as shown in Fig. la, our method achieves optimal
Baselines and Implementation
6

---

Rethinking the Temperature for Federated Heterogeneous Distillation
17.
40
150
30
125
30
50
10.0
40
20
73
30
50
10
20
25
110
HOALA
30
Pelt
0.0
200
150
200
100
350
(a) Accuracy curves for Dir (0.1) distribution tested on Tiny-
(b) Accuracy curves for PAT distribution tested on Tiny-Image ImageNet (right), Flower102 (left).
ImageNet (left), Flower102 (right).
Figure I. Accuracy curves for PAT distribution tested on Tiny-ImageNet, Flower102.
Table The testing accuracy (%) of heterogeneous model combination on Cifar-10, 10. AA for AlexNet, for ResNet18, G. for googleNet. S. for ShuffleNetv2
A.
R.
G.
S.
A. + R.
G.+ R.
S. + 87.84%
A. R. + G..
A. R. + S.
4(All).
Accuracy
89.28%
89.04%
89.38%
86,65 86.69%
88.94%
89. 19.16%
88.64%
87.49%
88. 79%
formance. The experiment removes these key components sequentially and observes their impact on the final perfor- mance. Specifically, we start from the full model (Ours: Multi-level + Elastic Temperature T + Global Logits Scal- ing) and sequentially remove Global Logits Scaling, Elastic Temperature т, and ultimately simplify it to a single-level fixed temperature (Single-level Fixed 7) of the simplest version. As shown in Tab.4 and Fig.2(leff), the full model performs the best out of all the setups, achieving perfor- mance of 72.75%. At the same time, it shows that Multi- level Distillation can initially improve the quality of dis- tillation, while Elastic Temperature т and Gloabl Logits Scaling can provide more knowledge in distributed learning compared with Fixed
Table 4. Ablation studies of our method on four datasets.
Method
Cifar10
Cifar100
Flower102
Tiny.
Single-level& & Fixed r
66.85
27.52
26.97
13.20
Multi-level & Fixed r
69.93
29.12
30.26
14.56
Multi-level & Elastic 70.82
29.45
32.05
15.14
ReT-FHD(OOrs, FI.)
72.76
30.77
33.16
17.24
results with fewer communication rounds, reaching near- peak performance on Tiny- ImageNet in under 80 rounds. In Fig. lb, while our method exhibits slower initial growth Flower102, it ultimately surpasses other baselines in later rounds.
Homogeneous model. 1b) shows our method out- performs leading FL baselines across four ResNet-based datasets, achieving 19.03% accuracy gain over HYDRA- FL on Flower102 under PAT's high-variability data. This highlights our framework's 's strength in handling feature am- biguity via elastic temperature scaling.
Evaluation of Distillation Layers Number L. To in- vestigate the impact of the number of distillation layers on model performance, we conduct experiment using the ResNet-19 18 architecture, varying the number of layers from I to 5. Fig. shows that increasing distillation layers from 1 to 4 enhances accuracy: CIFAR- 10 boosts from 67. % to 72.75% (+5.54), CIFAR-100 from 28.01 % to 30.77% (+2.76), and Tiny-ImageNet from 14.21% to 17.24% (+3.03). These findings reflect diminishing returns with excessive layers, highlighting the need to match distil- lation layers to model complexity for optimal performance.
DeL Experiments. Our method employs blockchain for de- centralized federated training, eliminating the need for a cen- tral server. We compare it with state-of-the-an decentralized methods in Tab.2. On CIFAR-10, our method and DeSA fall short of L2C due to its efficient convergence on sim- pler datasets. However, the more complex CIFAR-100, our method achieves the highest accuracy, demonstrating its strength in extracting meaningful patterns. Addition- ally, blockchain integration enhances privacy protection in distributed communication.
Effect of different heterogeneous models on accuracy. Tab. 3 shows the performance with heterogeneous model combinations on CIFAR-10. Multi-branch distillation achieves optimal accuracy (89.16%) using complemen- tary architectures like GoogleNet (G) ResNet18 (R). In contrast, combining all models (2() degrades performance (88.79%), underperforming standalone G. Results empha- size strategic model complementarity over quantity for ef- fective FL.
5.3 In-depth Analysis
Ablation studies. We design a series of ablation experi- ments to verify the effects of Multi-level distillation, Elastic Temperature T and Global Logits Scaling on model per-
7

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Atttoon Studies
Accuracy fumber of Delfiition Layers
Table 5. Comparison of the mean communication cost and amount of computation per round for each approach. "MB" are short for megabyte.
2.6
225
Methods
Comm.(MB)
Computation((llop])
Up.
Down.
lnve boste Oene tearp
FedProto
0.00256
0.00256
11340
FedTGP
0.00256
0.00256
11680
10
Bounds
of Layers
FedKDD
FML
10.45
10.45
10.45
10:45
21310
4620
FedType
5160
Figure 2. Ablation study (left) and Evaluation of Distillation Lay- ers Number I. (right) on Cifar-10. 10.
redDDistill
0.003
0.083
11760
DeSA
0.6
0.6
24680)
ReT-FHD((Ours)
0.0003
0,0000
3780
0.35
25.30
0.55
Table 6. Security Evaluation of Logits and Z-scores under Mali- cious Node and Label Reversal Attacks on CIFAR-1O. This table presents the logits and their corresponding 7-scores under two ad- versarial attack scenarios: (1) malicious nodes injecting anomalous data for the "Airplane" and "Bird" classes, and (2) label reversal attacks flipping the labels between "Deer" and "Dog" classes.
6.25
30.00
60.13
4.30
0.8
hrchkag Maloooo to
Nyierlago PiBike Artictorts
a) Cifar10
b) Cifar100
Malicious model
attack
Air
Auto
Bind
Cat
Deer
Dog
Figure 3. Security testing in blockchain framework
C/C
Airplane
Bird
0.0477
681.74
-1.0924
248
0,0016
1805.4
Deer
Dog
1,7653
1.7862
3.3860
0,90266
0.9474
3,4350
Airplane
0.7710
0.1045
2.5008
Deer
L.48]
0.5623
2.2111
Zm
Bird
0.5957
-0.1628
2,6090
Dog
1,441
2.2295
0.5963
AZZ,
4,5235
3.6608
A2%
288462
10.203
Analysis of Complexity and Communication Volume. As shown in Tab.5, we analyze the communication upload and download data volumes, as well as the training complexity of all baseline methods. The experiments are conducted on the CIFAR- dataset, encompassing four different modali- ties in a heterogeneous setting (with the proxy model being ResNer-18) Overall, methods based on proxy models gen- erate the highest communication volume due to the need to transmit model parameters. In contrast, our logits distilla- tion method results in the least communication overhead. Regarding computational complexity, proxy model methods incur a significant computational burden during the local knowledge transfer between the teacher and student. In comparison, the prototype-be tups exhibit relatively lower computational demands during local training. In summary, our method outperforms other approaches in terms of both communication volume and computational complexity.
Airplane
0.7203
-0.6765
.0.595
Deer
0,7833
1.6304
1,0069
Bird
-0.7223
Deg
0.7682
0,8572
1.8016
AZ.
2.3946
4.0729
47114
AZ5
288665
7.6342
7,5652
A22
4,4746
3.0139)
5.6661
A22
2.0812
2.9867
5.2146
Table 7. Testing accuracy($) under attack and no attack.
Methods
n10 attack
attack Cif-100
Flow.
Cif-100
Cif-10
Flow.
Cif-10
FedDistil
20.47
21.04
63.51
10.31
9.28
20.75
FCCL
25.20
22.72
64.29
12.88
9.06
31.18
DeSA
30.18
26.84
65.68
12.89
9.82
29.14
HYDRA-FL
22.27
20.82
61.74
9.65
11.04
36.94
ReT-FHD(Ours, FL)
33.16
30.77
72.76
20.11
19.28
38.75
and logits distillation se-
weights, we examine its impact on logits and Z-scores for randomly selected clients. For the first three CIFAR-10 cat- egories, the malicious node's logits show irregular values, such as abnormally high scores for "Airplane." " Similarly, its Z-scores Zm exhibit unusual patterns. To quantify devia- tions, use A and AZ (calculated via Eq.7), measur- ing differences between global, verification, and malicious Z-scores. While verification and global Z-scores remain nor- mal, the malicious node's 's scores are significantly elevated, with A Z% and AZT confirming its aberrant behavior.
5.4 Security and Robustness Performance
To establish a reliable global logits benchmark, we configure 20 clients, each responsible for independent training on the CIFAR-10 dataset. We aggregate all logits of the same class using a weighted average to obtain the Z-scores for each category. These Z-scores are then visualized, with diagonal lines indicating predictions for the correct classes, demonstrating that the Z-scores for correct predictions are generally the highest.
Label Reversal Attack. In Tab.6, we simulate label inver- sion by flipping "Deer" and "Dog" labels on a CIFAR- 10 client. The malicious node shows skewed logits, with ab- normally high confidence for "Dog" on deer samples, while the verification node's Z-score deviation from the global average remains minimal. This divergence in logit distribu- tions and Z-score alignment confirms our system's 's efficacy in detecting label-flipping attacks.
Malicious Model. The experimental results in Tab.6 ana- lyze the behavior of a malicious node in a simulated dis- tributed network. By injecting noise into a ResNet model's
8

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Logits Attack. Clients under anomalous attacks produce logits conflicting with the optimization goal, disrupting fed- erated learning distillation. Tab.7 analyzes test accuracy un- der client anomaly attacks on state-of-the the--rt logits distilla- tion methods across three datasets. Our method outperforms others, with its multi-level distillation effectively mitigating logits attacks, as supported by prior research (Huang et al., 2025a). While HYDRA-FL also uses Multi-level distilla- tion, it focuses on model homogeneity and underperforms in heterogeneous settings. This confirms that our dynamic temperature strategies enhance robustness and security in federated distillation.
References
Chang, H., Shejwalkar, V., Shokri, R., and Houmansadr, A. Cronus: Robust and heterogeneous collaborative learn- ing with black-box knowledge transfer. arXiv preprint arXiv: /912. 1279,, 2019.
Chen, H., Asif, S. A., Park, J., Shen, C.-C., and Bennis, M. Robust blockchained federated learning with model validation and proof-of-stake inspired consensus. arXiv preprint arXiv:2101.0300 2021.
Chrabaszez, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of imagenet as an alternative to the cifar datasets, 2017.
Accuracy and Poisoning Tolerance. Model accuracy de- crease as the percentage of malicious participants increased, with CIFAR-I0K 100 showing sharper decline due to its com- plexity. In Fig. 3.a (CIFAR-10), accuracy dropped from 0.35 to below 0.1 with 60% malicious participants, while Fig. 3.b (CIFAR- 100) shows a decline from 0.65 to under 0.2. These results demonstrate that server trust alone cannot mitigate adversarial impacts in federated learning, necessi- tating robust detection mechanisms and adaptive defenses to preserve performance under attack.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer,
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: / //arxiv. abs abs/bbs/2000 20100.221111111 11929.
Esfandiari, Y., Tan, S. Y., Jiang, Z., Balu, A., Herron, E., Hegde, C., and Sarkar, S. Cross-gradien aggregation for decentralized learning from non-iid data. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th Interna- tional Conference on Machine Learning, 1021, 18- 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3036-3046. PMLR, 2021. URL httt:://proceedee wtt.///faanaaarrrraa 21a. .tmml.
Conclusion
In this paper, we propose a novel dynamic temperature con- trol strategy for federated distillation (ReT-FHDD) and imple- ment in all FL, Del and blockchain-driven FL frameworks. Through comprehensive experiments, our approach demon- strates substantial improvements across three critical dimen- sions: model performance, communication efficiency, and security robustness. While our method addresses data and model heterogeneity, the current framework does not evalu- ate compatibility with attention-basee architectures such as Vision Transformers (VIT) (Dosovitskiy et al., 2021). Future efforts will extend validation to Transformer frameworks given their prevalence in visual tasks.
mwr. press/
Gong, X., Sharma, A., Karanam. S., Wu, Z., Chen, T., Doer- mann, D., and Innanje, A. Preserving privacy in federated learning with ensemble cross-domain knowledge distilla- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1 11891-11899 2022.
Guo, Z., Zhang, C.,, Fan, Y., Tian, Y., Zhang. C., and Chawla, N. Boosting graph neural networks via adaptive knowl- edge distillation, 2023. URL https: aaw.iv..gg.//////..... a abs/ 221 0. 05920.
Acknowledgements
This work was supported by NSFC (No.62206200, 62206137, 62036012, 62376196 and U23A20387).
Hao, Z. Guo, J., Han, K., Tang, Y., Hu, H., Wang, Y., and Xu. C. One-for-all: Bridge the gap between heteroge- neous architectures in knowledge distillation. Advances in Neural Information Processing Systems, 36, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
He, C., Annavaram, M., and Avestimehr, S. Group knowl- edge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33: 14068-14080, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep resid- ual learning for image recognition. In 2016 JEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2016, Las Vegas, NV, USA, June 27-30.

---

Rethinking the Temperature for Federated Heterogeneous Distillation
2016, pp. 770-778. IEEE Computer Sociery, 2016. doi: (0.1109/CVRR22016999 URL httph:ttp///c..... org/ 10.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun, ACM, 60(6):84-90, may 2017. ISSN 0001- 0782. doi: 10. 1 145/3366368 URL https://doi. /doi.
Huang, C.-Y., Srinivas, K., Zhang, X., and Li, X. Over- coming data and model heterogeneities in decentralized federated learning via synthetic anchors. arXiv preprint arXiv:2405. 11525, 2024.
org/10 10. 1 104.111655338888
Kundu, S., Sun, Q., FU, Y., Pedram, M., and Beerel, P. A. Analyzing the confidentiality of undistillable teachers in knowledge distillation. In 35th Neural Information Processing Systems, 2021.
Huang, C.-Y,, Srinivas, K., Zhang, X., and Li, X. Over- coming data and model heterogeneities decentralized federated learning via synthetic anchors. In Proceedings of the 41st International Conference on Machine Learn- ing, ICML'24. JMLR.org, 2025a.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (00ML,000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Huang, W., Ye, M., and Du, B. Learn from others and be yourself in heterogeneous federated learning. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10133-10143, 2022. doi: 10.
Lee, R. FernandezMaarques,
Hu, S. X., Li, D., Laskaridis, S., Dudziak. E., Hospedales, T., Huszár, F. and Lane, N. D. Recurrent early exits for feder- ated learning with heterogeneous clients. arXiv preprint arXiv:2405. 14791, 2024.
1109/CVPR52688.20222
Huang, W., Ye, M., Shi, Z., Li, H., and Du, B. Rethinking federated learning with domain shift: A prototype view. In 2023 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 16312-16322. IEEE, 2023.
Li, D. and Wang, J. Fedmd: Heterogenous feder- ated learning via model distillation. arXiv preprint arXiv: 1910.03581, 2019.
Li, S., Zhou, T. Tian, X., and Tao, D. Learning col- laborate in decentralized learning of personalized mod- els. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 9756-9765. IEEE, 2022a. doi: 10. 10.1109CCPPR222882222200093 URL htths:/:: doi wttp://...1//////22/225501 2022 00954.
Huang, Y., Yan, Z.,, Shen, C., Fang, F., and Zhang, G. Har- monizing knowledge transfer in neural network with uni- fied distillation. In European Conference Computer Vision, pp. 58-74. Springer, 2025b.
Itahara, S., Nishio, T., Koda, Y., Morikura, M., and Ya- mamoto, K. Distillation-based semi-supervised federated collaborative train-
Li, Z., Li, X., Yang. L., Zhao, B., Song, R., Luo, L., Li, J., and Yang. J. Curriculum temperature for knowledge dis- tillation, 2022b. URL https: btttaps://arxv.ccom /at/aaaxvvvorr.e... abs/ 2211. 16231.
learning for communication--ffiiol ing with non-iid private data. IEEE Transactions on Mobile Computing, 22(1):191-205 2021.
Jeong, E., Oh, S., Kim, H., Park, J., Bennis, M., and Kim, S.- L. Communication--fficiin on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv: 1.1 1479, 2018.
Lian, X., Zhang, C., Zhang, H., Hsieh, C., Zhang, W. and Liu, J. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, M., Fergus, R., Vishwanathan. S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5330-5340, 2017. URL https: httpppppcceeee..../////////
Jeong, E., Oh, S., Kim, H., Park, J., Bennis, M., and Kim, S.- on-device machine learning: Federated distillation and augmentation under non-iid private data, 2023.
L. Communication--effile
Khan, M. A., Chandio, Y., and Anwar, F. M. Hydra-fl: Hybrid knowledge distillation for robust and accurate federated learning. arXiv preprint 2rXiv:2409.1990 2024.
courips.cc//aaee /20papeeer211// F7552665933104 cb www.ccb11333444 Abstract. html.
Liang, P., Chen, J., Wu, Y., Pu, B., Huang, H., Chang, Q. and Ran, G. Data free knowledge distillation with fea- ture synthesis and spatial consistency for image analysis. Scientific Reports, 14(1):27557, 2024.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
10

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Lin, T., Kong, L.,, Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion in federated learning- Advances in neural information processing systems, 33: 2351-2363, 2020.
Wang, J., Zhao, C., Lyu, L., You, Q.. Huai, M., and Ma, F. Bridging model heterogeneity in federated learning via mncertainty-basee asymmetrical reciprocity learning. arXiv preprint arXiv2240..02477, 2024.
Liu, J., Liu, B., Li, H., and Liu, Y. Meta knowledge distilla- tion, 2022. URL https:/ / www.w..ww...ab///000.0 07940.
Wang. J., Zhao, C., Lyu, L., You, Q. Huai, M., and Ma, F. Bridging model heterogeneity in federated learning via uncertainty- -based asymmetrical reciprocity learning. In Proceedings of the st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
Ma, N., Zhang, X., Zheng, H., and Sun, J. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 1 11218 of Lecture Notes in Computer Science, pp. 122-138. Springer, 2018. doi: 8. URL https:://d/ //doi. org/10. 1000000.00009000 9-0-9778---00000 01264-9
Wei, Y. and Bai, Y. Dynamic temperature knowledge distilla- tion, 2024. URL https: /awiiv.vvvvv.///////////////avv. org/ has///b///// 2404. 12711.
Wu, C., Wu, F., Lyu, L.,, Huang, Y., and Xie, X. Communication -ffficienn federated learning via knowl- edge distillation. Nature Communications, 13(1), April 2022. ISSN 2041-1723. doi: 10.1038/
10.1007/978.3-000-012200
Nilsback, M.-E. and Zisserman, A. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graph- ics and Image Processing, pp. 722-729, 2008. doi: 10.1 10.100//CVG))2200..4777
441467-022-29763--1 URL http: //dp: //dxxdoo..org 10.
Ying, C., Xia, F., Wei, D. S., Yu, X., Xu, Y., Zhang, W., Jiang, X., Jin, H., Luo, Y., Zhang, T., et al. Bit-fl: Blockchain-enabled incentivized and secure federated learning framework. IEEE Transactions on Mobile Com- puting, 2024.
Qin, Z., Yan, X., Zhou, M., and Deng, S. Blockdfl: A blockchain-based fully decentralized peer-to-peer feder- ated learning framework. In Proceedings of the ACM on Web Conference 2024, pp. 2914 2125,292,,222 2024.
Zhang, J., Guo, S., Ma, X., Wang, H., Xu, W., and Wu, F. Parameterized knowledge transfer for personalized feder- ated learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Con- ference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 10092- 10104, 2021.
Shen, T., Zhang. J., Jia, X., Zhang. F., Huang, G., Zhou, P., Kuang, K., Wu, F,, and Wu, C. Federated mutual learning, 2020.
Sun, S., Ren, W., Li, J., Wang, R., and Cao, X. Logit standardization in knowledge distillation. arXiv preprint arXiv:2403.01427, 2024.
URL https: httpp:/:s/p/ooeednnng//
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. E., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9, IEEE Com- puter Society, 2015. doi: 10. 10.11099CVPR.22055228899 URL https: http///i..oooog////////o 1109....11190.... 2015. 7298594
5383c731 8a html.
Zhang, J., Liu, Y., Hua, Y., and Cao, J. Fedtgp: Trainable global prototypes with trastive learning for data and model heterogeneity in fed- erated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16768-16776, 2024a.
con-
Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Jiang, J., and Zhang, C. Fedproto: Federated prototype learning across heterogeneous clients. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Four Fourth Con- ference on Innovative Applications of Artificial Intelli- Edu-
Zhang, J., Liu, Y., Hua, Y., and Cao, J. An upload-efficient scheme for transferring knowledge from a server-side pre- trained generator to clients in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024b.
gence, IAAI 2022, The Twelveth Symposium cational Advances in Artificial Intelligence, FAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 8432- 8440. AAAI Press, 2022. doi: 10.1609/AAALLV366..
Zhang, I.., Shen, L., Ding. I., Tao, D., and Duan, L.-Y. Fine- tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10174-10183, 2022.
20819. URL ht /opp://ddue
1609 aaai.
v3688. 20819.

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Zhao, S., Wang. X., and Wei, X. Mitigating accuracy- robustness trade-off via balanced multi-teacher adversar- ial distillation. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zhu, T., He, F., Chen, K., Song, M., and Tao, D. De- centralized SGD and average-directio SAM are asymp- totically equivalent. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 43005-43036. PMLR, 2023. URL https: / /pppoceedingg.....//eee mlr mar..press v202/zhu23e. .hmmll.
12

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Appendix
A.1 The proof of Theorem 4.1 in Sec.4
For discrete random variable X with probability distribution P(x), the information entropy is defined as:
-(X)=-
Σ
log P(z)
(15)
xE
where P(x) obtained from the softmax function. The softmax function is defined as follows:
softmax(p:)
(16)
Σラ 6
Firstly, we can calculate the derivative of the probability distribution ps with respect to the temperature T:
3p.
0
exp (z//T)
(17)
0T
-ー=1 exp 12//TT
By calculation, is possible to obtain:
Op: 07
exp
exp
2:
(
-
- exp (
(18)
2
n
exp
3
2j
Σ
exp
T
7
j=1
Simplifying and organizing gives:
θp: 0T
Pi
(19)
Sccondll.informmtiion entropy is fundamental concept used to measure the uncertainty of random variable. For a discrete random X with probability distribution P(z), the information entropy is defined as:
H(X)
Σ P(z) P(x)
(20)
ZEX
The information entropy is derived with respect to temperature as follows:
OH
n
(pp;
a
Σ
log Ps + Pi aT (log pi)
(21)
0T
OT
Simplifying and organizing gives:
0H
Opp ØT
Op,
=
log +
0T
0T
1=1
(22)
Op:
(log P; + 1)
OT
By substituting the expression for dp. we get:
Σ j=
Pjžj
Zi
=
Σ
(log + 1)
(23)
aT
T
iml
13

---

Rethinking the Temperature for Federated Heterogencous Distillation
Based the above derivation, the equation for the derivative of the entropy H with respect the temperature 7 is known. Next, it is necessary replace the conventional temperature 7 with predefined elastic temperature in Eq. (6)TTo accurately determine the relationship between entropy and the elastic temperature it is necessary to understand how 0r
varies with AZ, namely 8A2
By the definition of G, we can derive:
0T OAZ
? log AZ)
dAz
log(1 + A Mmax
(24)
log(1 + A 7maa)
1+ AZ
Ultimately, the derivative of the entropy H with respect to T can be expressed as:
0H
0H
0
OAz
m ji=1
PjRj
(logP: +1)
Ph
(25)
i=1
1
log(1 + A max
1 + AZ
The condition 0H implies that increase in the elastic temperature leads to decrease in the system's entropy. This 85 reduction in entropy is reflected increase in information, known as information gain. Consequently, appropriately increasing the elastic temperature can enhance information gain, allowing the model to learn more effectively from the training data.
Then the Theorem 4.1 is proved.
A.2
The proof of Corollary 4.2 in Sec. 3
For a model f with random distribution T in the initial state, for a deep neural network (DNN) model f with random distribution I in the initial state, the model has no preference for any sample. Thus for the model's predictive distribution f1 fpl ph. .... p. the following relationship exists:
B(p' (x)) = (x)())) (x)5 = (x))
Assumption A.1. We assume that the model is optimised under the guidance of a soft label distribution with smoothing degree X is satisfied:
A (xr1 = =D²'(20 (x2) = =... ....= (xx) 1
p (Xep1) ) = 122 (Xez2) = ... (I(pp))
(10)
For simplicity, only consider dataset that includes categories, for which the categories defined follows.
Definition A.2. Let the cross-entropy loss of sample be denoted as LCE, and the average cross-entropy loss over all samples be LCE. The class ID is defined as:
class ID 11, if LcB > LCE,
otherwise.
Referring to the study of (Zhao et al., 2024), we introduce the knowledge scale to quantify the level of knowledge of the predictive distribution, which can measure the scale of knowledge learnt from the probability distribution by the model with randomly initialised weights:
KT <L(I(2),T(E)) log C C )
(1))
14

---

Rethinking the Temperature for Federated Heterogencous Distillation
where K L()) stands for Kullback-Leible divergence, which calculates the cost of information required to travel from distribution to another, C' is the total number of categories and is constant for the same dataset, and (.))) is the information entropy.
First we calculate the derivative of the knowledge scale with respect to logits =c(x):
[KT( (1), Px.)
(2)
Öpi (1)
(z)(x)
Opi
p (x) - pc (x)
= (x) p. (x) (x)
(12)
For simplicity, we will subsequently develop the proof using C 2 example. So can easily obtain:
(KL( PAt) E
(1)
021(11)
(22(22)
= = - xi (x1)
(o - 1/2 (52)
(13)
OKI(f(r1;0)), Px:) 022(x1)
PA.)) (82L(f(12;;1). 01), Pxj) 021 (x2)
E (p((ec) 12 (21) (p)( - p (12)
(14)
Combined with the previous assumptions, we can obtain that the partial derivatives of knowledge scale changes are symmetric for all categories, which suggests that the overall change in knowledge scale has balanced impact on different categories. Therefore, our elastic temperature focus on the adjustment of knowledge scale different stages of the model and treat different categories equally, which helps to bridge the structural differences between heterogeneous models of different clients.
Then the Corollary 4.2 is proved.
A.3 The proof of Theorem 4.3 in Sec.4
We explore the relationship between changes in the category gradient and temperature, using category 1 as an example. The gradient for category 1 is:
dzi 00
0KL
(fd(
aze
ØKL (f)(21;11),P)
f~ =E
+
(2)
Ōzi
a0
Oz2
We increase the temperature of category 1 to F
AF to obtain the global logits distribution Px.. We can then obtain:
~~
021
ØKL (fa(11; Pxa)
02g
0KL (fa(21;01) 01), PA.)
+
00
Oz1
30
Oz2
(3)
102
ØKL (11; 01), Px.)
022
ØKL (x1 01), PA.)
E
+
00
0z1
00
022
0z where 20 can be regarded as the impact of the class itself on model weight optimization, in a practical sense, this part reflects the inner relationship between DNN and sample, more specifically, can reflect the difficulty of the sample itself for
15

---

Rethinking the Temperature for Federated Heterogeneous Distillation
the model and not relate the optimization object. Therefore, the above equation can be simplified as:
V
f~
Ōz1
(01) - (m1)
azg 00
(1 (12) 23 12 (on))
(4)
E
Based on the effect of temperature on the probability distribution: pi (x1) 2ी (2)) > 0. (z1) - -2 (x1) <
At the
same time, since category is the main category, its gradient changes dominate. Therefore we can get:
ymmaa
V/
(5)
The same can be obtained when we reduce A for the category temperature to get the global logits distribution Pag
V SNNAS - V/NAs
<0
66)
Then the Theorem 4.3 is proved.
A.4 Blockchain-based FL Details
In the blockchain network, the role of the e node is exclusively dedicated to the voting processes. This node evaluates transactions based on the established Equation x, which serves as a decision makiin criterion. Depending on the results derived from applying this equation, the verification node can make its decisions, opting either for P (Positive) or N (Negative). This dichotomy allows the node actively participate in the consensus mechanism by approving or rejecting transactions, thereby ensuring the reliability and security of the blockchain.
Subsequently, the node will take on the responsibility of broadcasting all the votes it has received for the node w to all m nodes within the network. This dissemination ensures that every m node in our simulated network receives the set of values z belonging to the set 2 for each w. This comprehensive distribution of vote information across the network not only enhances transparency but also enables each node to have a holistic view of the network's ' consensus on each w, facilitating more informed decision decision-making and maintaining the integrity of the overall system.
Each node m in the network meticulously processes all data extracted from the transactions sent by node v. Upon processing, each node synthesizes this information into series of candidate blocks. A typical candidate block, denoted as block?" encapsulates not only the global logits 2; but also includes the result R and data pertaining to the blacklist B. This blacklist is dynamically generated based on the collective voting outcomes, reflecting the network's consensus on disreputable nodes or transactions.
Both types of nodes, v from set V and m from set M, receive rewards in aligmment with their contributions the network's security and integrity. The reward system is designed to incentivize activities that uphold the blockchain's 's operational standards and trustworthiness. Furthermore, the blockchain protocol is engineered to select the candidate block from the most affluent nodes-the ones with the highest accrued rewards-as the preferred choice for linkage to the blockchain. This selection criterion ensures that the most reliable and trustworthy nodes, as determined by their wealth accumulation through valid and honest contributions, have their candidate blocks prioritized in the blockchain extension process.
To effectively simulate a real blockchain-based Federated Learning (FL. environment, our methodology includes the dissemination of pertinent information through the network. This is achieved by encapsulating the information within a transaction, which is then securely signed using the private key of the designated role. This approach ensures the integrity and authenticity of the data being broadcast across the network.
A.5 Additional Experimental Results
Effeets of different and K. In our experiments, we specifically assess the performance of our method across different setups, involving 50 and 100 clients. The results, detailed in Table 8, indicate that while there is slight decline in accuracy as the number of clients increases, our method demonstrates relatively stable outcomes over multiple training cycles. For instance, with 50 clients, our method achieve an accuracy of 42.46% at E = 20, and with 100 clients, it maintains a respectable 37.57%, despite the increased complexity and potential for data discrepancies among larger number of clients. In contrast to the general trend observed with other models, our proposed High-Dimensie (HDFL.) approach managed to maintain performance across an extended number of client training cycles. This resilience
Federated Learning
16

---

Rethinking the Temperature for Federated Heterogeneous Distillation
Table TTee accuracy (%) C0 in the practical setting using the heterogenenus model group with different number of client training epochs (E) and different number of clients(K).
E
K 50
K' 100
FML
39.63 + 0.83
39.81 1 0.82
39.45 1 0.83
36.65 1: 1.62
33.78
1.66
FedProto
37.61 0.91
37.31 = 1.01
35.71 1.31
33.24
2.14
31.79
1.89
FedDistill
40.20 0.98
39.82 = 0.97
39.89 1.01
37.77
1.60
35.36
1. 16
FedKD
37.01 + 1.11
36.96 4 1.12
37.77 + 1.07
32.64
1.85
32.34
FedTGP
42.87 = 1.67
41.69 + 1.84
40.94 # 1.80
39.81 + 2.03
36.75
1.84
48.38
0.64
0.63
42.46
1.62
37.57
Atplam
085
134
3.08
032
6.59
442
-.72
8.50
4.34
3.5
Ca2
027
1224
6.1
131
131
430
4.8
432
4.20
4U59
0.9
430
931
4.33
Dgg
P0g
1.20
4:21
6.6
038
63
1.12
CA2
1.20
4.4
SNPP
+6.2
6.1
4.22
-333
0.5
48
241
0.9
3.04
029
0.14
-.14
4.55
Figure 4. Global Logits Z-Score Visualization for Cifar- 10 with 20 Clients
can be attributed to our model's 's ability to effectively manage and synchronize heterogeneous data and learning strategies across diverse client environments. The capability to sustain high accuracy levels under varying client counts and training conditions highlights the robustness of our approach, making it particularly suitable for practical applications where client variability and training dynamies are challenging factors.
Global Logits Z-Score Visualization. Fig.4 visualizes the Z-Score of the global logits under the Cifar-10 dataset. It can be found that the data distribution of logits is more concentrated through the z-score processing.
Supplement of Tab.6. Tab.9 and Tab.10 extends the results of Tab.7 by providing a comprehensive display of the complete logits and Z-scores, offering deeper insights into the behavior of the models under the evaluated scenarios. These supplementary results aim to enhance the understanding and robustness of the conclusions drawn from the main experiments.
Complete Experimental Data on Malicious Model. Tab.9 shows that the complete logits and Z-scores obtained by injecting noise the weights of the malicious node's category airplane and category bird. The comprehensive analysis of logits and Z-scores across all categories in the supplementary data reveals distinct behavioral patterns between malicious and normal nodes. Specifically, the malicious node exhibits significant anomalies in the "Airplane" and "Bird" categories, with logits and Z-scores deviating markedly from the normal range (c.g., specific values in Z-scores for category airplane corresponding to category airplane and category bird: [0.7710, 2,5008] + [0.7203, -6595J). In contrast, other categories such as "Car" and "Trucks" remain stable, with minimal deviations (e.g., [-0.6385, -0.8651] vs. [-0.7638, -0.7107J, A [0. 1253, -0. 1544)). The differential measures A 22, and AN effectively quantify these deviations, with values for anomalous categories (c.g.. Bird: [22.485, 61.4704J significantly exceeding those of normal categories. These findings underscore the targeted nature of the attack and the robustness of the proposed detection metrics in identifying malicious activities.
Complete Experimental Data on Label Reversal Attack. In Tab. 10, we simulate a label inversion attack by flipping the labels of "Deer" and "Dog" on a selected CIFAR- 10 client. The malicious node exhibits significant anomalies in its logits, particularly displaying abnormally high confidence for the "Dog" class when the true label is "Deer." For instance, the Z-Score for "Dog" on deer samples rise to 2. 2.2411, compared to the normal range of [-0.6547, 1. 1.1481]. In contrast, the
17

---

Rethinking the Temperature for Federated Heterogencous Distillation
Table 9, Supplement Table 6.Mallicious model)
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Trucks
Airplane
681.74
248.60
1805.43
-233.92
-289.58
-13.881
117.65
105.17
-111.56
-381.13
c/c
Bird
0.0477
-1.0924
0.00163
-0.03615
-0.6927
-0.0313
-0.78591
0.11322
0.17006
-0,0338
Airplane
0.7710
0.1043
2.5008
-0.6385
-0.7242
-0.4844
-0.0973
-0.1165
-0.4501
-0.8651
Tm
Bird
0.5957
-0.1628
2.6090
-0.7720
-0.2781
-0.6905
-0.2938
-0.1418
-0.1322
-0.7333
4.8235
3.6608
22.485
7.0336
14.503
5.1918
8.1249
4.8195
5.7027
3.1847
Airplane
0.7203
-0.6765
-0,6595
-0.7638
0.4510
1.0750
-0.6278
-0.8247
2.0166
-0.7107
Bird
0.3182
-0.7181
-0.7223
-0.5224
0.4126
1.6685
-0.6531
-0.8371
1.7418
-0.6880
Zv
A A
2.3946
4.0729
61.4704
2.4421
5.5224
17.1949
3.2096
4.1907
20.675
2.2256
AZ3
4.4746
3.0139
5.6661
7.3089
6.3297
6.99128
5.1713
2.40193
8.0513
3.3053
Table 10.
Supplement to Table (Label reversal attack)
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Trucks
Deer
008718
-0.3399
-0,8518
1.7862
0.9026
3,4350
-0.9332
-0.9158
-0.9916
-0.8755
C/C
Dog
-0.9006
-0.3496
-0.8857
1.7653
3.3860
0.9474
-0.9386
-0.9114
-0.7518
-0.7910
Deer
0.6140
-0.2614
-0,6008
1.1481
0.5623
2.2411
-0.6547
-0.6432
-0.6688
-0.6165
Zm
Dog
-0,6413
-0,2723
-0.6313
1.1441
2.2295
0.5963
-0.6667
-0.6485
-0.5416
-0.5679
AZz
2.2690
2.1762
3,3980
2.8462
10.203
18.7151
3.2746
4.6532
2.4289
3.3192
Deer
0.3912
-0.7767
-0.6786
0.7833
1.6304
1.0069
-0.6776
-0.8615
0.3785
-0.6293
Dog
0,3617
-0.7883
-0.6498
0.7682
0.8572
1.8016
-0.7156
-0.8255
0.3113
-0.5843
Zv
AZV
5,6487
4.0295
2.4296
2.8765
7.6342
7.5652
2.4054
2.9043
5.02901
2.3278
AZ5
6,4078
4.3835
3,3734
2.0812
2.9867
5.2146
2.7634
6.7946
4.140
3.8648
verification node's s Z-scores remain closely aligned with the global average, with smaller deviations (c.g., AZY values within [2.7634, 6.4078)). This stark divergence logit distributions and Z-score alignment between the malicious and verification nodes highlights the effectiveness of our system detecting label-flipping attacks. Furthermore, the differential measures A A and A quantify the malicious node's deviations, with values for the flipped categories (e.g., [7.6342, 7.5652) significantly exceeding those of normal categories. These results demonstrate that our approach only identifies the presence of label- ffippiii attacks but also provides robust metrics for quantifying the extent of the adversarial manipulation.
18


Reviews:
[{"review_id": "xFMsCGnhxk", "reviewer": "Reviewer_JGAK", "summary": "The paper introduces ReT-FHD, which aims to improve the efficiency of federated learning by addressing model and data heterogeneity. The paper’s key motivation comes from the common weakness of current federated distillation methods, i.e., suboptimal temperature calibration during knowledge fusion. Therefore, the paper proposes several innovative mechanisms to tackle this challenge. Firstly, Multi-level Elastic Temperature is dynamic temperature adjustment mechanism to adjust the distillation intensity across model layers, which can optimize knowledge transfer between heterogeneous local models; Category-Aware Global Temperature Scaling introduces temperature calibration to individual categories, which is based on the confidence distribution in global logits and it can ensure a more personalized distillation process; and Z-Score Guard is a blockchain-verified validation mechanism to mitigate attacks Through various experiments on various datasets, the proposed framework significantly outperforms existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer JGAK for the constructive comments: \"…reducing communication costs and enhancing security …\", \"… provides clear experiments …\". To thoroughly address your concerns, we will answer the questions one by one:\n\n**Questions:**\n\n**Q1: Differences from existing dynamic temperature knowledge distillation methods**\n>Existing dynamic temperature adjustment strategies—BGNN[1] (network-learned temperature for GNN nodes), CTKD[2] (course-learning-based scheduling), and DTKD[3] (sharpness approximation for temperature optimization)—are designed for **centralized learning** with **homogeneous architectures and IID data**, adjusting temperature **across training rounds (e.g., early vs. late epochs)**. Our approach 1) employs class-specific calibration for global logits to mitigate **non-IID label skew** and 2) assigns independent temperature coefficients to **distinct model stages (e.g., shallow vs. deep layers) within a single training round** to address **model heterogeneity**.\n\n**Q2: Which one contributes more to solving model heterogeneity in FL**\n>As detailed in **Line 32** and **Line 78**, the multi-level distillation is foundational to our heterogeneous distillation framework. As in the **Ablation Study (Tab.4)**(removing multi-level settings reduces performance by 3.08%), we validate the effectiveness in heterogeneous federated learning scenarios. Meanwhile, dynamic temperature( in **Theorem 4.1 and 4.3**) allows for more knowledge about global logits and improves performance by **2.82% (Tab.4)**.  We therefore respectfully assert that these components form an indivisible contribution core.  \n\n**Weaknesses:**\n\n**W1: Not much discussion of dynamic temperature work**\n>Our motivation is to explore whether **logits** distillation can be adapted to federated heterogeneous learning. According to the answer for **Q1**, temperatures are adjusted between training rounds. This **coarse-grained adaptation** relies on global training progress and fails to localize dynamic changes within a single round. In FL, due to clients' non-IID data, **models across clients may exhibit significantly divergent optimization states within a single round**. However, cross-round temperature updates[1][2][3] require synchronization in the next round, leading to slowed convergence and even suboptimal solutions caused by delayed adaptation. \n\n**W2: The novelty is not so sufficient**\n>Our work is not a \"simple\" combination of multi-level distillation and dynamic temperature, as detailed in **Q1** and **W1**. We clarify our contributions as follows:\n>\n>>1. **Incorporating dynamic temperature distillation into heterogeneous FL**, enhancing knowledge transfer across diverse models;\n>>2. **Designing a Z-score-based malicious node verification mechanism**, effectively detecting and filtering abnormal logits to improve model robustness;\n>>3. **Providing a theoretical justification** for ReT-FHD’s effectiveness from the perspective of information entropy and gradient updates, offering solid theoretical support for our method;\n>>4. **Extensive experiments** on centralized FL (Tab.1 and Tab.3), decentralized FL (Tab.2), and blockchain FL (Tab.6, Tab.7, and Fig.3) validate our robustness and flexibility.\n>>\n>We respectfully request reconsideration of this core contribution.\n\n**W3: Blockchain deployment is not suitable as a main contribution**\n>We do not clarify that blockchain deployment is our core contribution.  In our contribution 3, we clarify that \"our framework employs **Z-score verification** to validate logit distributions against dynamic boundaries, enabling automated reward/punishment protocols that deter malicious behaviors.\"We address its unique challenge in logits-sharing scenarios:  \n>\n>>- **Motivation**: Traditional blockchain FL [4][5][6] validates via *model parameters*, but this fails for *logits-based distillation*.  \n>>\n>>- **Contribution**: Our **Z-score validation** (Eq.7) ensures compatibility without parameter exposure, achieving:  **20.11% post-attack accuracy** (Tab.7 vs. baseline 11.43%)\n>\n>This mechanism directly enables our main contribution—**Z-Score Guard:** (Eq.7) for Standardized Finding of Malicious Logits. \n\n**References:**\n>[1]Boosting Graph Neural Networks via Adaptive Knowledge Distillation\n>\n>[2]Curriculum Temperature for Knowledge Distillation\n>\n>[3]Dynamic Temperature Knowledge Distillation\n>\n>[4]Robust blockchained federated learning with model validation and proof-of-stake inspired consensus\n>\n>[5]Blockdfl: A blockchain-based fully decentralized peer-to-peer federated learning framework\n>\n>[6]Bit-fl: Blockchain-enabled incentivized and secure federated learning framework\n\nPlease let me know if I have addressed your concerns and if not, we welcome any further questions you might have. Finally, thank you for your time and effort in reviewing our work.", "claims_and_evidence": "Yes, the authors provided the corresponding analysis and experiments for the claims.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, I have checked the proofs of theorem 4.1 and corollary 4.2.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The key contribution of the paper is multi-level elastic temperature scaling.", "essential_references_not_discussed": "Some knowledge distillation methods with dynamic temperature are not discussed.", "other_strengths_and_weaknesses": "Strengths:\n1.\tThe paper combines multi-level elastic temperature scaling, category-aware global temperature scaling, and blockchain-based security mechanisms into a single framework for federated knowledge distillation.\n2.\tThe framework focus on reducing communication costs and enhancing security without introducing significant computational. Therefore, the application range of the framework is wider.\n3.\tThe paper provides clear experiments across multiple benchmark datasets compared with several methods. The use of several datasets(CIFAR-10, CIFAR-100, Tiny-ImageNet, Flower102) and non-IID data distribution settings shows the comparing performance of the proposed method.  The ablation studies help prove the contribution of each component in the proposed framework.\n\nWeaknesses\n1. Dynamic temperature has been discussed by several works on knowledge distillation, but the work does not discuss much about these works, while dynamic temperature is a core contribution of the work.\n2. The work seems to be a simple combination of multi-level knowledge distillation and dynamic temperature, while both multi-level knowledge distillation and dynamic temperature have been already proposed. Though the authors made several contributions by designing a Z-score based temperature adjustment and introducing it into federated learning, which indicates much novelty, I personally consider the novelty is not so sufficient.\n3. The different deployment scenarios make only a little sense. Overall, the federated learning framework does not require the server to conduct much computation with privacy preservation, thus it is obvious that it can be easily to be extended to different scenarios. The deployment method on blockchain might not be proper to be considered as a main contribution.", "other_comments_or_suggestions": "1. Not all the notations are provided with the introduction.\n2. Some key references are not introduced.\n3. The adjustment strategy for temperature requires more detailed justifications.", "questions_for_authors": "1. What is the main difference in the strategy for temperature when compared existing dynamic temperature knowledge distillation methods? Is there any specific requirements or challenges?\n\n2. Which one contributes more to enabling the logit-based knowledge distillation to address the model heterogeneity in federated learning?  The dynamic temperature or the multi-level setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jEg1My5eDp", "reviewer": "Reviewer_TEBy", "summary": "The paper introduces ReT-FHD, a framework for heterogeneous federated knowledge distillation that contributes three core ideas. First, it proposes Multi-level Elastic Temperature to adaptively regulate how much knowledge is distilled at each layer, enhancing cross-architecture consistency. Second, it implements Category-Aware Global Temperature Scaling, assigning class-specific temperatures to better handle non-IID data. Analysis shows this stabilizes convergence by harmonizing gradient variations. Finally, it integrates a Z-Score–based Guard to detect and deter malicious behaviors like label flipping. Experiments across benchmarks show significant accuracy improvements and communication efficiency.\n\n\n**Update after rebuttal:**\n\n Thanks to the authors for their response. After reading it, I would like to keep my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer TEBy for the constructive comments: \"…enhance accuracy under model/data heterogeneity\", \"…reduces communication costs compared to proxy-based methods\", \"… represents a novel synthesis of ideas from robust FL, KD, and malicious detection … \", \"… address a variety of practical deployment scenarios\". To thoroughly address your concerns, we will answer the questions one by one:\n\n**Questions:**\n\n**Q1: Potential Interaction**\n>The interaction of category temperature tuning (Eq.4) with the multi-level distillation (Eq.3) can have a **positive impact** on mitigating model heterogeneity. We facilitate knowledge transfer by applying category scaling to global logits, which simultaneously improves multi-level distillation. The ablation experiments in Tab.4 show that the combination of the two significantly improves performance, demonstrating that they are **complementary** in dealing with heterogeneity.\n\n**Q2: Differences with Existing Dynamic Temperature Methods**\n>Please refer to Reviewer JGAK's Q1 for the answer to this question.\n\n**Weaknesses:**\n\n**W1: Extension to Transformer**\n>Logits-based knowledge distillation is also **applicable to the Transformer architecture**, and similar work [1] has validated the effectiveness of hierarchical distillation on Transformers. Since we deliver logits information and eliminate architecture specificity through branching exits, our approach is more general and can be widely adapted to different architectures.\n\n**W2: More Detailed Clarity**\n>The exact model distribution for heterogeneous clients: the four models AlexNet, ShuffleNetV2, ResNet18, and GoogleNet are randomly and evenly assigned to clients.\nIn distillation learning, logits serve as high-level guidance. Our elastic temperature calculation uses **next-stage logits** (vs. current-stage) to **avoid local stage overfitting and enhance stability**. Besides, the experiments on Cifar-10 can verify this:\n| Cifar-10 | next-stage | current-stage |\n| --- | --- | --- |\n| accuracy | 72.76% | 71.54% |\n\n**Responses to Questions about Theoretical Analysis**\n\n>**Theorem 4.1:** Theorem 4.1 **does not** require specific assumptions and it is valid in the framework of both multilevel elastic temperature and category-aware global temperature scaling. According to the information entropy formula, the information entropy of logits is only affected by the softmax temperature and the model output and is **only related to the temperature coefficient** when the model output **cannot be changed**. In Eq.3, we set the **initial temperature ξ** and limit its variation by a **scaling factor γ and a log function** to avoid the boundary case. In addition, in lines 237-238 of the code file client.py, we set the temperature range to **0.5 ≤ τ ≤ 5**.\n>\n>**Corollary 4.2:** We refer to **the assumptions of [4]** to simplify the theoretical proof process. In addition, Corollary 4.2 is proposed to provide theoretical support for category temperature setting. Regardless of the initial state of the model, we dynamically adjust the category temperatures of the global logits based on the local learning of different categories. Since temperature scaling is done at the softmax level, it is applied evenly to all classes in the same round, thus ensuring that the scaling effect remains consistent. In addition, we focus mainly on model heterogeneity and non-iid data distributions in FL, whereas **changes in the category distribution are not our concern**.\n>\n>**Theorem 4.3:** Although Theorem 4.3 assumes category equilibrium for analytical simplicity, temperature scaling **remains effective in heterogeneous settings**, enhancing gradient updates for difficult categories. Tab.1(a) and Fig.1 validate this in non-IID scenarios (e.g., α = 0.1, 0.5, pat), showing **significant performance gains**. Fig.2 further demonstrates that increasing distillation layers from 1 to 4 improves accuracy, plateauing at 5, confirming the validity of our multi-level mechanism.\n\n**Supplementary experiments**\n**Experiments on model heterogeneity in DeL**\n>These decentralized baselines except DeSA[3] basically **do not consider model heterogeneity**, and it is unfair to forcefully change them to baselines for heterogeneous models. Due to time constraints, we supplemented our experiments with model heterogeneity on only one dataset:\n|Flower102|DeSA|ReT-FHD|\n|---|---|---|\n|accuracy|37.17%|40.82%|\n\n**References:**\n>[1] One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation\n>\n>[2] Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation\n>\n>[3] Overcoming data and model heterogeneities in decentralized federated learning via synthetic anchors\n\nDue to space constraints, I apologize for not being able to answer some of the questions, but if there are still questions, we can respond to them in the second discussion. Finally, thank you for your time and effort in reviewing our work.", "claims_and_evidence": "**Well-Supported Claims:**\n\n**1. Adaptive Temperature Calibration Improves Accuracy.**\n\nClaim: Multi-level elastic temperature and category-aware scaling enhance accuracy under model/data heterogeneity.\n\nEvidence: Experiments on CIFAR-10/100, Tiny-ImageNet, and Flower102 show consistent accuracy gains. Ablation studies confirm the necessity of both components.\n\n**2. Communication Efficiency.**\n\nClaim: ReT-FHD reduces communication costs compared to proxy-based methods.\n\nEvidence: Quantified communication overhead aligns with the claim.\n\n**Problematic Claims:**\n\n**1. Theoretical Guarantees (Theorem 4.1) on Temperature-Efficacy.**\n\nClaim: Theorem 4.1 is presented as a theoretical foundation showing how temperature adjustments reduce entropy or gradient variance, thereby enhancing heterogeneous distillation.\n\nIssue: The paper does not clarify under what specific assumptions (e.g., model randomness, uniform client participation, or particular distributions of logits) this theorem holds. It is also unclear whether the same conditions remain valid in the multi-level knowledge distillation or Category-Aware Global Temperature Scaling framework. From the proof in the appendix, it appears that the theorem’s validity depends on the specific algorithm proposed by the authors.\n\n**2. Applicability of Lemma 4.2 in Realistic Scenarios.**\n\nClaim: Lemma 4.2 assumes a random initialization where the model is indifferent to all classes, thereby implying balanced effects on each class under soft label distributions.\n\nIssue: This assumption may be too strong for real-world settings, where data can be highly skewed and models might be partially pre-trained or not truly “randomly” initialized. As a result, the claimed uniform impact on different categories could fail to hold, and the lemma does not clarify how quickly or under what conditions the model transitions from this idealized initialization into a state where nontrivial data heterogeneity and model structure are present.", "methods_and_evaluation_criteria": "**1. Design Rationale for Key Equations:**\n\nFor Equation (3), the paper bases its temperature adaptation on the Z-score difference between the next-stage logits and teacher logits, yet does not explain why the difference involving the current-stage logits was not considered. If the goal is to measure the discrepancy between local and teacher knowledge, one might expect the current stage to be more directly relevant. For Equation (4), it applies a negative sign before \\beta, effectively lowering the temperature for classes with higher loss. This design choice seems at odds with the intuition that “hard” classes (i.e., larger loss) should receive more guidance. A detailed rationale or additional experiments could clarify why reducing temperature for higher-loss classes aids learning.\n\n**2. Selective Scaling of Only Global Logits:** \n\nThe paper confines category-aware temperature scaling to global logits, without mentioning the possibility of partially or fully scaling local logits. Since local distributions may be highly skewed, applying a similar scaling locally could potentially yield further improvements—or at least warrants discussion to justify why the authors limit scaling to the global side.\n\n**3. Completeness of the Update Formula:** \n\nEquation (6) is missing the intermediate update and lacks an explicit explanation of how the parameters are iteratively updated. Clarifying that step would better illustrate how multi-level distillation is integrated into the local optimization process.\n\n**4. Motivation and Implementation Details:** \n\nWhile the paper’s overarching goal—enhancing model performance via adaptive temperature—is clear, some formulaic details require further justification. Providing explicit motivations for these design choices (e.g., how each approach helps balance knowledge across architectures or handles non-IID distributions) would help readers understand why these decisions are suited to the specific problem of heterogeneous federated learning.", "theoretical_claims": "I checked the proofs for Theorem 4.1, Corollary 4.2, and Theorem 4.3 in the Appendix, and here are some observations:\n\n**1. Boundary Conditions and Dynamic Distributions:** \n\nTheorem 4.1 relies on taking the partial derivative of information entropy with respect to temperature under a fixed, single softmax distribution. In a multi-round FL process, however, both teachers’ and students’ logits evolve. If \\tau approaches 0 or becomes very large, or if the model distribution becomes nearly deterministic at intermediate steps, the partial derivative behavior may shift drastically. The paper does not clarify how Theorem 4.1 extends to these boundary cases, nor does it show whether the distributional assumptions remain valid across rounds and layers, given that entropy typically changes along with training progress.\n\n**2. Uniform Class Impact in Partial Derivatives:** \n\nThe proof of Corollary 4.2 concludes that all classes experience the same shift in knowledge scale by showing the partial derivatives for each class are identical. However, it does not fully justify why this equivalence holds once training proceeds and the network starts updating logits in potentially different ways for each class (especially if class distributions shift or vary across training steps). While the algebraic steps appear correct for an initial, uniformly random setup, the proof does not clarify how or whether the partial derivatives remain identical in subsequent iterations.\n\n**3. Class-Specific Temperature and Gradient Updates:** \n\nFor Theorem 4.3, the logic that “increasing a class’s temperature increases its gradient update” is demonstrated under the corollary’s assumption of balanced classes and random initialization. This might not strictly hold in more skewed data settings or in deeper, multi-stage distillation pipelines unless further assumptions are introduced.", "experimental_designs_or_analyses": "**1. Heterogeneous Model Configuration:** \n\nThe paper states that four models (AlexNet, ShuffleNetV2, ResNet18, GoogleNet) are used in a heterogeneous setup. However, it does not detail exactly how these models are distributed among clients, nor whether some clients share the same architecture. Adding a clear baseline (e.g., no heterogeneity) for comparison would help highlight how well the proposed method handles real-world model diversity.\n\n**2. Range of Dirichlet Parameters:** \n\nThe authors mainly present results for \\alpha=0.5, with limited discussion of the more extreme alpha=0.1. It would be helpful to visualize or table out how performance changes as \\alpha varies (e.g., 0.1, 0.3, 0.5) to confirm the method’s sensitivity to stronger non-IID data.\n\n**3. Homogeneous vs. Heterogeneous in DeL:** \n\nThe decentralized setting (Table 2) uses only 2-layer CNNs, implying no model heterogeneity. It remains unclear how the method would perform under simultaneous data/model heterogeneity in a truly decentralized scenario. Presenting such an experiment would better demonstrate ReT-FHD’s robustness in the most challenging setups.\n\n**4. Ablation Studies vs. Main Table Results:** \n\nThe ablation experiments in Table 4 report a final CIFAR-10 accuracy of 72.75%, which is close to the 72.76% in Table 1 under Dir(\\alpha=0.5). However, it is unclear if those ablations are indeed run under the same Dirichlet parameters and the same heterogeneous model assignment. Moreover, the “Single-level & Fixed \\tau” condition seems to combine multi-level knowledge distillation and Z-score distillation, but the paper does not indicate whether that configuration matches any baseline in Table 1. Clarifying these experimental conditions would help readers match ablation performance to main table comparisons.\n\n**5. Data/Model Heterogeneity Parameters (Tables 5-7):**\n\nThe paper does not specify the exact Dirichlet settings or model architectures used for the experiments in Tables 5, 6, and 7 (e.g., how many clients used each architecture or the distribution of labels). Since these tables measure communication cost, malicious attack detection, and robustness, readers need a clear understanding of the data/model heterogeneity to interpret the results accurately; Table 5 omits a direct comparison with FCCL, another logit-based approach, leaving it uncertain how the proposed method stands against the most relevant baseline in terms of communication and computation. Also, the authors claim a notably lower cost than all other methods, but an explanation of why ReT-FHD, despite using additional temperature scaling steps, still results in reduced overhead would be beneficial.", "relation_to_broader_scientific_literature": "The paper builds on and extends multiple lines of work within federated knowledge distillation and heterogeneous FL:\n\n1. Heterogeneous FL. It aligns with prior methods such as FedMD and FedGKT in enabling cross-architecture knowledge transfer. However, whereas many of these methods rely on proxy models, public datasets, or large feature transfers, the authors focus on lightweight logit exchanges alone, making it more communication-efficient.\n\n2. Multi-level Distillation. Inspired by hybrid distillation approaches, the paper introduces a multi-stage temperature calibration that had not previously been emphasized in the federated setting. This multi-level tactic leverages work showing intermediate-layer KD can improve model robustness and accuracy, but extends it with a dynamic temperature schedule designed for non-IID data and heterogeneity.\n\n3. Security Extensions. The Z-score–based guard echoes recent interests in detecting malicious updates or Byzantine nodes. By combining blockchain-inspired validation with logit-based screening, ReT-FHD attempts to formalize trust boundaries and reduce risks of label-flipping or adversarial logits.", "essential_references_not_discussed": "One area the paper could further contextualize is robust aggregators and Byzantine-resilient FL. Although the authors mention label-flipping and some blockchain-based checks, they do not engage extensively with the broader literature on strong Byzantine defense mechanisms. For example, seminal works like Krum, Bulyan, or other robust-mean estimators have laid out various theoretical guarantees. Since the authors’ proposal relies partly on logit-verification to detect anomalies, connecting their method to known robust aggregation strategies would help clarify whether and how ReT-FHD stands on par with or improves upon existing Byzantine-robust solutions.\n\nAdditionally, there is a growing body of work on dynamic or adaptive temperature in knowledge distillation (outside of FL contexts), such as “Adaptive Temperature for Distillation” or “Dynamic Soft Labels,” which the paper might draw upon to strengthen its theoretical rationale. Including those references would make the paper’s unique application of adaptive temperature to heterogeneous FL even clearer.", "other_strengths_and_weaknesses": "**Other Strengths**\n\n1. Creative Combination of Techniques: The paper’s integration of elastic temperatures (multi-level and category-aware) and a Z-score–based blockchain security guard represents a novel synthesis of ideas from robust FL, KD, and malicious detection. Even if each concept (dynamic temperature, logit sharing, blockchain validation) has appeared in some form, combining them systematically to tackle heterogeneous FL highlights originality.\n\n2. Applicability to Diverse FL Modes: By explicitly supporting centralized, decentralized, and blockchain-based topologies, the authors address a variety of practical deployment scenarios. This flexibility extends the potential reach of ReT-FHD to real-world federated systems that might not always rely on a single server or a purely decentralized approach.\n\n**Other Weaknesses**\n\n1. Limited Discussion of Other Cases: While the paper mentions Transformers or attention-based models as future work, it does not evaluate or discuss how easily ReT-FHD might extend to such architectures. This leaves open the question of whether the proposed solution is limited to some specific models or can generalize broadly.\n\n2. Clarity Gaps: Although the presentation is overall coherent, a more detailed discussion of certain design choices (e.g., the exact model distributions for heterogeneous clients, the rationale behind measuring Z-score difference with the next stage rather than the current stage) would improve transparency and reproducibility.", "other_comments_or_suggestions": "There are some typos:\n\n1. In the paper, Algorithm 1 references color coding (e.g., “highlighted in orange/pink/purple”) to distinguish between FL, DeL, or blockchain steps, but there is no explicit legend or explanation for the orange portion. This can be confusing to readers trying to follow the workflow.\n\n2. In line 262, the phrase “We assume that the model is optimised under the guidance of a soft label distribution with smoothing degree \\lambda_1 is satisfied:” seems grammatically wrong. \n\n3. Line 220 incorrectly refers to Corollary 1 instead of Corollary 4.1.\n\n4. In Section 3.2, the text introduces $z_{t}^{(l)}$ (logits of the teacher) but never explicitly defines $z_{s}^{(l)}$.", "questions_for_authors": "1. Potential Interaction Between Model and Data Heterogeneity: In Corollary 4.2, you focus on the effect of multi-level elastic temperature for mitigating model heterogeneity, concluding that class-specific differences are not impacted. Yet you later propose category-aware scaling to address data heterogeneity. Would class-wise temperature adjustments (Equation 4) interact with the multi-level scheme (Equation 3) in ways that actually affect model heterogeneity as well? Do you think it is necessary to analyze whether category scaling at each level might improve—or possibly interfere with—multi-level temperature adjustments?\n\n\n2. Related Work on Dynamic Temperature Tuning: The authors introduce Multi-level Elastic Temperature and Category-Aware Global Temperature Scaling in the context of FL. However, there is other research in KD or general deep learning about adaptive/dynamic temperature (even if not specifically for federated settings). Could you clarify whether there are existing algorithms that similarly adjust temperature across multiple layers or categories, and if so, how your proposed approach differs? A comparison or reference to this broader literature (outside federated learning) would help situate your contributions more clearly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apfAajbNCh", "reviewer": "Reviewer_e9EW", "summary": "This paper highlights the suboptimal temperature calibration issue in existing federated distillation (FD) methods. To address this, the authors propose ReT-FHD, which introduces multi-level elastic temperature to dynamically adjust distillation intensities across different model layers and category-aware global temperature scaling for class-specific calibration. Additionally, they integrate a Z-score Guard blockchain verification mechanism to defend against label-flipping and poisoning attacks. Extensive evaluations across multiple benchmarks validate the effectiveness of ReT-FHD, demonstrating its superiority over existing FD approaches.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer e9EW for the constructive comments: \"…dynamically adjust distillation intensities across different model layers...\", \"…validate the effectiveness of ReT-FHD...\". To thoroughly address your concerns, we will answer the questions one by one:\n\n**Questions:**\n\n**Q1: Connection between elastic temperature scaling and the security components, and the rationale for incorporating security**\n>Our motivation is to **explore whether logits can be applied to complex heterogeneous FL scenarios**. To this end, we adopt Z-Score to normalize the knowledge distillation process and further introduce multi-level distillation, elastic temperature scaling, and secure validation mechanisms for optimizing cross-architecture knowledge distillation and enhancing model robustness, respectively. In addition, the verification mechanism of the model-based blockchain FL approach is difficult to apply to the logits sharing scenario. Therefore, we design a Z-Score-based verification mechanism, which is customized for the logits sharing FL method, which ensures efficient logits-based communication and effectively defends against malicious attacks.\n\n**Q2: Lack of annotation in equations**\n>- **Eq.3** describes the multistage elastic temperature mechanism, where each stage of the model is assigned an independent temperature adjustment. The temperature of each stage of the model is adjusted according to the logarithmic difference of the next stage. The temperature is bounded by $\\xi$ to maintain stability and prevent degradation. The scaling factor $\\gamma$ controls sensitivity. $\\Delta Z_l$ represents the difference between current and next stage logits, guiding the temperature adjustment. $\\Delta Z_{\\max}$ normalizes the temperature across stages for consistency. \n>\n>- **Eq.4** describes the category-aware temperature scaling mechanism, which dynamically adjusts each category’s temperature based on its cumulative loss relative to the average loss. The weighting factor $\\beta$ controls the sensitivity of temperature updates. $\\mathcal{L}_{\\mathrm{CE}}(f(\\tilde{x}_c))$ represents the cumulative loss for category c, and the average loss across all categories serves as a global reference. The maximum absolute difference standardizes the temperature update across categories, ensuring consistency. \n>\n>- **Eq.5** defines the optimization objective of ReT-FHD. Here, $|\\mathcal{D}|$ and $|\\mathcal{D_i}|$ denote the total and local data sizes, respectively.\n$\\mathcal{L}_{\\mathrm{CE}}$ is the cross-entropy loss. The model applies KL divergence for knowledge distillation at each layer. $\\sigma^t\\left(\\cdot\\right)$ and $\\sigma^s(\\cdot)$ represent the softmax functions of the global and local logits, respectively. $\\mathbf{z_t}$ denotes the global logits, $\\bar{z_t}$ local logits' mean, and $\\operatorname{Var}\\left(\\mathbf{z}_t\\right)$ local logits' variance. $G(\\Delta Z)$ and $\\tilde{\\tau}_c$ represent the temperature coefficients for model stages and categories, calculated using Eq.3 and 4, respectively. Unlike global logits, local logits are unaffected by category temperature adjustments.\n\n**Experimental Designs Or Analyses:**\n>Our validation mechanism, designed for logits-based scenarios, standardizes malicious logits detection and is mainly applicable to logits sharing, rather than model parameter-sharing FL frameworks. In security experiments, we demonstrate its effectiveness compared to other FL methods. \nOn CIFAR-10, each validation (Eq.7) requires **~100-200 FLOPs**, while a ResNet18 forward pass takes **~1.8 GFLOPs**, making the blockchain overhead negligible. Moreover, since the blockchain mechanism only verifies logits without altering the FL communication process, ReT-FHD maintains the same communication efficiency as standard FL, as confirmed in Tab.5.\n\n**Essential References Not Discussed:**\n>Thanks for pointing out these references. Due to time constraints, we add additional experiments on **Cifar10** to more fully assess the effectiveness of ReT-FHD in dealing with model heterogeneity and safety. While **FPL[1]** mitigates domain shift, it may introduce information redundancy in our setting, whereas **SDEA[2]** is designed for homogeneous models and relies on high-quality public datasets.\n|Hetero. & No attack|RethinkFL|ReT-FHD|\n|---|---|---|\n|accuracy | 68.21% | 72.76% |\n|**Attack & Homo.**|**SDEA**|**ReT-FHD**|\n|accuracy|50.89%|51.75%|\n\n**Other Strengths And Weaknesses:**\n>If you have any doubts about specific equations, please refer to our response to Q2. As for Fig.1, the y-axis represents accuracy, and the x-axis indicates the number of communication rounds.\n\n**References:**\n>[1] Rethinking federated learning with domain shift: A prototype view\n>\n>[2] Self-driven entropy aggregation for byzantine-robust heterogeneous federated learning\n\nThank you for dedicating your time and effort to reviewing our work. Please let me know if I have addressed your concerns and if not, we welcome any further questions you might have.", "claims_and_evidence": "The security protection related content in this article suffers from an inadequately articulated rationale and demonstrates weak contextual integration with the remainder of the content, ultimately coming across as a superfluous addition that compromises the paper's structural coherence.", "methods_and_evaluation_criteria": "1. The proposed multi-level elastic temperature is based on layer-level distillation, which looks incremental but interesting.\n2. The category-aware global temperature scaling component is difficult to understand, primarily due to the lack of annotations in the equations and an unclear description of the training strategy.\n3. Section 3.3 is quite confusing, as it presents a large amount of information with weak contextual integration.", "theoretical_claims": "I am not an expert in theoretical analysis. From my point of view, the theoretical claims in the paper are solid.", "experimental_designs_or_analyses": "1. The authors show abundant experiments to demonstrate the effectiveness of the proposed methods on heterogeneous data & models.\n2. The experimental section lacks a comparative analysis between the proposed method and established attack-defense FL frameworks, which significantly undermines the empirical validation of ReT-FHD's effectiveness. \n2. While the authors present a comparative analysis of communication efficiency, the blockchain overhead inherent to ReT-FHD remains notably absent from their evaluation framework.", "relation_to_broader_scientific_literature": "The core idea of this paper builds on previous findings that layer-level distillation enhances performance. The authors investigate the impact of temperature at each layer and introduce elastic temperature scaling to further refine distillation quality.", "essential_references_not_discussed": "Some key competitors are not discussed in the paper.\n\nFor proto-based method: Rethinking federated learning with domain shift: A prototype view[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 16312-16322.\n\nFor attack defense: Self-driven entropy aggregation for byzantine-robust heterogeneous federated learning[C]//Forty-first International Conference on Machine Learning. 2024.", "other_strengths_and_weaknesses": "1. The notations in equations are not clearly described.\n2. The axis titles are missing in Fig. 1.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. The whole structure of the paper is massive. There is no clear connection between elastic temperature scaling and the security components, and the rationale for incorporating security remains unclear.\n2. Lack of annotation in equations makes the paper hard to understand.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers are in agreement that this paper makes a valuable contribution and should be accepted. While the reviewers have made some suggestions for improvement, these are generally minor and aimed at further strengthening the final version. We encourage the authors to carefully address the reviewers’ comments when preparing the camera-ready revision.
Decision: Accept (poster)