Transforming Gaps into Gains: Bridging Model and Data Heterogeneity in Federated Learning via Knowledge Weak-Aware Zones
Authors: Ke Li
Yan Ding
Zhiqin Zhu
Shenhai Zheng
Venue: NeurIPS 2025 poster ()
Keywords: Heterogeneous Federated Learning
Knowledge Distillation
Cognitive Discrepancy
Primary area: deep_learning
TL;DR: This paper proposes FedKWAZ, a framework that enhances heterogeneous federated learning by bridging semantic and decision gaps via Knowledge Weak-Aware Zones.

Abstract: Heterogeneous federated learning enables collaborative training across clients under dual heterogeneity of models and data, posing challenges for effective knowledge transfer. Federated mutual learning employs proxy models to bridge cross-model knowledge exchange; however, existing methods remain limited to direct alignment between the outputs of private and proxy models, ignoring the deep discrepancies in representation and decision spaces between them. Such cognitive biases cause knowledge to be transferred only at shallow levels and trigger performance bottlenecks. To address this, this paper proposes FedKWAZ to identify and exploit Knowledge Weak-Aware Zones (KWAZ)—spatial zones of deep knowledge misalignment between private and proxy models, further refined into Semantic Weak-Aware Zones and Decision Weak-Aware Zones, which characterize cognitive misalignments in representation and decision spaces as focal targets for enhanced bidirectional distillation. FedKWAZ designs a Hierarchical Adaptive Patch Mixing (HAPM) mechanism to generate multiple mixed samples and employs a Knowledge Discrepancy Perceptron (KDP) to select the samples exhibiting the largest representation and decision discrepancies, thereby mining critical KWAZ. These modules are integrated into a two-stage mutual learning framework, achieving global class-level representation-decision consistency alignment and local KWAZ-guided refinement, structurally bridging cognitive biases across heterogeneous mutual learning models. Experimental results on multiple datasets and model configurations demonstrate the superior performance of FedKWAZ.

Transforming Gaps into Gains: Bridging Model and Data Heterogeneity in Federated Learning via Knowledge Weak-Aware Zones
Ke Li", Yan Ding', Zhiqin Zhu Shenhai Zheng**
1 Chonggiin University Posts and Telecommunications (s230301058, #230301006) cqupt. edupt.edu..... en {Zhiqin Zhu, Shenhai @cqupt. edu. cn
Abstract
Heterogeneous federated learning enables collaborative training across clients under dual heterogeneity of models and data, posing challenges for effective knowledge transfer. Federated mutual learning employs proxy models to bridge cross-model knowledge exchange; however, existing methods remain limited to direct alignment between the outputs of private and proxy models, ignoring the deep discrepan- cies in representation and decision spaces between them. Such cognitive biases cause knowledge to be transferred only at shallow levels and trigger performance bottlenecks. To address this, this paper proposes FedK WAZ to identify and ex- ploit Knowledge Weak-Aware Zones (K (KWAZ)-Spanl misalignment between private and proxy models, further refined into Semantic Weak-Aware Zones and Decision Weak-Aware Zones, which characterize cognitive misalignments in representation and decision spaces focal targets for enhanced bidirectional distillation. FedKWAZ designs Hierarchical Adaptive Patch Mixing (HAPM) mechanism generate multiple mixed samples and employs Knowl- edge Discrepancy Perceptron (KDP) to select the samples exhibiting the largest representation and decision discrepancies, thereby mining critical KWAZ. These modules are integrated into a two-stage mutual learning framework, achieving global class-level representation-deciite consistency alignment and local KWAZ- guided refinement, structurally bridging cognitive biases across heterogeneous mutual learning models. Experimental results multiple datasets and model configurations demonstrate the superior performance of FedKWAZ.
zones of deep knowledge
Introduction
37 39 19 has emerged as a privacy-preserving distributed training paradigm, enabling devices collaboratively learn global model without sharing raw data. However, FL faces two major challenges in practical deployments: clients often hold non-IID data, and necessitate training personalized models that fit device-specific resource constraints.
Federated Learning (FL.) 21
The dual heterogeneity in data and model disrupts traditional parameter aggregation and makes difficult for single global model effectively serve all clients. Consequently. Heterogeneous Federated Learning (HtFL) 44]. aiming to learn personalized models for each client, has become central research focus.
To tackle model and data heterogeneity in HtFL., federated mutual learning 28 has been proposed, where clients train nerformance-driven private models and resource-friendly proxy models in parallel, facilitating knowledge transfer via bidirectional distillation. However, existing methods [28 36 15
*Corresponingg authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

26 primarily focus on direct alignment in the feature and decision space across heterogeneous models, overlooking deeper mismatches in their learned representations and decision behaviors. As illustrated in Figure] T (left), for the same input, the private and proxy models on given client often produce significantly different feature distributions and classification a phenomenon termed semantic-decision dual drift. This dual drift implies that naive output matching yields only coarse and superficial knowledge transfer, leaving critical representation- and decision-level discrepancies unresolved. Without pinpointing and addressing these misaligned "weak zones," knowledge transfer remains inefficient and ultimately constrains the client's performance.
HAPM
Connuction
Princtr Shake
Mated
Semante Hasteg
Diarge
KDP
Canodiion
Knowledge Mismatch in Semantic and Decision
Mutual Learning FedKWWAZ
Figure 1: Knowledge mismatch between private and proxy model and mutual learning in FedKWAZ.
Motivated by the necessity to address this cemmantcc-ecisionn dual drift, this paper proposes Knowl- edge Weak-Aware Zones (K WAZ), characterizing zones where heterogeneous models exhibit the strongest cognitive discrepancies. KWAZ is further decomposed into Semantic Weak-Aware Zones (SWAZ) and Decision Weak-Aware Zones (DWAZ), focusing respectively on representation and decision discrepancies that indicate key zones requiring targeted knowledge enhancement. Building on this insight, the FedKWAZ framework is designed to mine and align complementary knowledge within KWAZ through two-stage mutual learning scheme integrating local and global collaboration.
At the local stage, as shown in Figure T (right), HAPM generates multiple mixed samples via diverse parameter pairs (a,,, 9:) (where i = 1,.... K), where controls the mixing ratio between two images and g specifies patch granularity to divide each image into v x vg blocks for local mixing. KDP then evaluates representation and decision discrepancies between private and proxy models across these samples and selects (a",g") inducing the maximal divergence, thereby defining the corresponding samples KWAZ, which are leveraged via SWAZ and DWAZ for path-specific bidirectional distillation, mitigating semantic and decision-level discrepancies between private and proxy models. At the global stage, the server aggregates class-level semantic prototypes and decision distributions uploaded by clients to construct unified representation-dccisin anchors. This approach eliminates the need for frequent proxy model synchronization, significantly reduces communication overhead, and provides representation-deciiie
consistency alignment from a global perspective.
The main contributions of this work are summarized as follows:
KWAZ is proposed to characterize zones where heterogeneous models exhibit strong cogni- tive divergence, offering fine ine-igrined perspectives beyond conventional output alignment.
FedK is developed federated mutual learning framework integrating HAPM and KDP to dynamically discover and exploit SWAZ and DWAZ, enabling complementary learning between private and proxy models in their zones of disagreement.
A communication-effiiiee global knowledge alignment mechanism is designed using aggre- gated class-level semantic and decision anchors, achieving strong cross-client knowledge alignment with significantly reduced communication costs.
2 Related Work
HtFL focuses on addressing collaboration challenges arising from variations in clients' computational capabilities, model architectures, and data distributions. Based on the degree of heterogeneity and structural constraints, existing studies can be grouped into the following three categories:

---

Resource-Heterogeeeen FL Methods. A number of approaches handle device heterogeneity by allowing clients train smaller or pruned versions of global model that fit their local resource constraints. For example, FedRolex L. FjORD 12]. FLASH [4], FedResCuF, 44 and HeteroFI. 7 enable each client to work with sub-model or width-reduced network tailored to its hardware capabilities. These methods focus on resource adaptation e.g., dynamically extracting training sub-networks from a larger global model - so that both high-end and low-end devices can participate in federated training. However, their emphasis is resource allocation rather than maintaining cross-client knowledge consistency. As a result, essource-heterogeneee align the semantic representations or decision distributions learned by different client models. The absence of such alignment means that knowledge gaps can persist between clients' sub-models, potentially limiting the overall knowledge transfer efficiency across the federation.
solutions do not explicitly
Hoddular-Hetrogeneeee FL. Methods Another line of work addresses model heterogeneity by splitting models into shared global modules and private local modules. FedRep [6] and FedPer [ share and aggregate a global feature extractor across clients (aiming for aligned semantic represen- tations), whereas LG-FedAvg [20] and FedGH [41] share a global classifier across clients (aiming for aligned decision output distributions). By partitioning the model, these approaches attempt to transfer knowledge either at the representation level (feature extractor) or at the decision level (classifier) among clients. However, since the "global" module in such frameworks is obtained by aggregating components trained on each client's non-iid local data, forcing all clients to adopt the aggregated module can undermine personalization and lead to suboptimal local performance. In practice. these methods face trade-off between achieving global alignment and preserving local adaptation. Merely sharing or exchanging modules may prove inadequate in addressing fine-grained, data-dependent misalignments in the learned representations decision boundaries of client models, potentially leaving certain semantic or decision-level discrepancies unresolved.
Autonomy-heceerggneonn methods allow clients to construct local models without structural limits. Distillation-based approaches, such as FCCL 13 and FedGen 48]. leverage public or synthetic data to facilitate cross-model knowledge transfer. yet their effectiveness is highly dependent on data fidelity and distributional consistency. Other methods [30 31 logits knowledge carriers. However, since prototypes and logits are cittormattionoompresst units, they limit the richness of knowledge transfer and hinder deep, comprehensive consistency alignment.
[4] transmit class prototypes or
In contrast, federated mutual learning introduces proxy models bridges for knowledge transfer, thereby accommodating seoonomy-heterogeeeee bidirectional distillation between private and proxy models to establish interactive transfer pathways. However, they focus on direct output matching, resulting in shallow distillation that fails to capture critical semantic and decision divergence. FedMRL [42 mitigates this paradigm by incorporating multi-granularity features, yet lacks explicit mechanisms for correcting semantic or structural gaps, introducing noise when concatenating diverse features in its MRL. Moreover, frequent transmissions of proxy models in these methods make hard to balance communication cost and training gains.
setups. FML (8 and FedKD (6 perform
Insights. Driven by the limitations identified dchove-especii and decision discrepancies between heterogeneous models this work introduces FedKWAZ. a cognitively-aware mutual learning framework that explicitly targets the problem of dual drift. It formalizes KWAZ as spatial zones where heterogeneous models display significant knowledge divergence, and further decomposes them into SWAZ and DWAZ, enabling targeted distillation the semantic and decision discrepancy zones, respectively. In addition, a lightweight class-level global knowledge anchoring mechanism is designed to achieve new balance among architectural flexibility, communication efficiency, and transfer quality. By directly confronting the -eeentin-decisoon dual drift, FedKWAZ bridges overlooked knowledge gaps and enables more precise and efficient cross-model collaboration.
the neglect of deeper semantic
3 The Proposed Fed WAZ Approach
3.1 Problem Setting
A typical HtFL system is considered, consisting of a central server and a set of clients CCo. 1N- Each client C, holds a private local dataset D,, sampled from distribution Pi. satisfying P. # P, for any i * j. each communication round, K N . clients are selected according to the participation rate p. Each client Cs trains private model M, and a proxy model Qx
3

---

Semantic Weak-Aware Zonee
Decision Weak-Aware Zones
(x;
(a (1
(M:
Mi Teacher
Q. Student
Jois Studdent
Sx
Mr
Teacher
M
SEEAR
My
DWAAZ
2 M,
DMYA
Gradient
Gradiien Descant
SUplood Local
MSE-Losa
Server
Legits
Local Protos
Local Logits
Ma
Flzcor "0
H(22 MA
Cossss
Client
Input
Co-Loss
Knowlodge
MSE-oss
Global Protos
Global Logits
Figure 2: The workflow of FedKWAZ.
on its local data DR. parameterized by th and Øk, respectively. The feature extractor and predictor are defined as w and w represented as th = {wg w/ H.Wh and Φ {ur , . H.... Unlike conventional mutual learning, global model is maintained at the server; instead, global guidance is constructed from feature- and decision-level knowledge uploaded by the clients. Fed aims to minimize the aggregated local losses across all joint models. The joint model each client denoted by W& (tha, Φx), and the optimization objective is:
N-1 Σ
min
ED (W)) Dx)]
where E(-) denotes the empirical loss of the joint model on the client's private dataset. After training, each client retains its private model Mr for final deployment and inference.
3.2 Modeling of KWAZ and Dual-stage Deep Mutual Learning
To investigate cognitive gaps between heterogeneous mutual learning models, KWAZ is charac- terized the knowledge-deficien zones of the student model S and the teacher model T in both representation and decision spaces. To detect these zones, the following modules are introduced:
HAPM: Every k rounds, HAPM generates multiple mixed inputs by partitioning into vg x vø patches (granularity g) and applying patch- wise interpolation with mixing ratio a under varied (a. g) pairs, enabling discrepancy
selection (details in Appendix (Ap.
KDP: For each mixed input generated by HAPM under specific (a, g). KDP quantifies the knowledge gap between and on it, and selects (a".9") inducing the largest discrepancy to guide targeted KWAZ mining and refinement over the following k training rounds.
Based on this mechanism, dual- mutual learning framework is constructed: coarse- coarse-grained grained cross-client knowledge alignment achieved at the first stage via global semantic prototypes and decision anchors, while fine-gra mmn-igrined distillation is conducted at the second stage under KWAZ guidance. Figure iillustrates the workflow, and Appendix B details the algorithm.
3.2.1 Stage Global Knowledge Mutual Learning
Global semantic and decision anchors are constructed by aggregating class-level features and predic- tions from private and proxy models on the server, enabling M and Q+ to mutually align with each other's global semantic and decision knowledge. Given class-wise dataset Dx,e from client C₁, for each & E Dr.co features and predictions from
Mg and Q₁ are:
Mr. = F(z; F.t) T-10,, D. Hl2 :. (.V) ), Fla;w ).000 =H(22 w w H.s. 2x
Mr Mc; H.V.
(2)

---

Class-wise features and logits are averaged as local knowledge. In the case of Mg;
P.Me
Mr Fx
Te
Vx
[Di,el
REDk.c
DEDk.c
and the same formulation applies to Qk yielding Pe= and Le+, All uploaded local knowledge aggregated to obtain global knowledge, where No denotes clients with data containing class
PM
Σ PMM. KEN.
LM
Σ L. Ne
1
We
REN.
with pe and 0 aggregated analogously from 2x With the global knowledge above, global mutual alignment between M₁ and Qs is achieved, where "Mx + Qx" indicates that Mg learns from Qs. Le denotes the cross-entropy loss, and the softmax function:
k
Ma
Σ
Σ
(ee ,,) + lee (J (((?))
MM
p2 [1
(5)
k
R
Σ
Σ
[00 ,2) + lee (D (L
22
P.
De.
Model parameters updated based on the above loss by performing one step of gradient descent:
v
1/ 1---11 ThU Ve 0i ox 176 VeRC-M
(6)
3.2.2 Stage II: Local KWAZ Mutual Learning
Following global coarse-grained alignment, FedKWAZ performs local fine- fine-grained mutual learning using KWAZ guidance. This stage incorporates three components: (1) a basic mutual alignment loss, (2) semantic refinement loss based on SWAZ, and (3) decision refinement loss based on DWAZ. These objectives are optimized through a second local gradient descent step. Prior to KWAZ- specific distillation, basic bidirectional loss computed to align intermediate features and output distributions between M₁ and Qk Here, KL(p q) = - Pe log Es denotes the Kullback- divergence [I], and is a temperature parameter used to smooth the probability outputs:
( Ma 2x KL (T /T) /T) 72 1|a 2 112
base
O. KL (o ( 7² + 1|
Me
base
Symmetric Semantic Enhancement. SWAZ characterizes input zones where student model S severely mismatches teacher model T in feature space. To identify such zones, of mix ratios A {0n,,... , and patch sizes G {91...., ., 9m.) are defined for HAPM. For each (a, g) E A x G, mixed input X is generated by blending the original sample x with ratio and patch size y through HAPM. This mixed sample is fed into both S and T, producing feature vectors S 8x and =T T The feature gap between S and T on the mixed sample is measured by
SWAZ Mutual Learning:
Mean Square Error (MSE) 33
S (0,g; EnND
[|
x
T 112
(8)
Swnz
The KDP conducts search over (a, E A to find the pair that maximizes this feature gap:
(a", = arg
max Axx)
S (a, gi D)
(9)
5

---

2 = HAPM(I; a", ,g"), Here, Using the optimal pair (a" (g",g"). the SWAZ is defined as SWAZ "Mę + Qx" indicates that the same imput is symmetrically used both directions, since the optimal parameters are selected by maximizing the MSE-based feature gap between the two models, and MSE is symmetric with respect its inputs. As a result, exchanging student and teacher roles yields the same mixing parameters and the same SWAZ. Let MM
X + and Xx denote the feature Xswiz vectors of Ma and Qx on this optimal mixed input. Then the SWAZ loss for both directions is:
M MMMae SWAZ
Ms ||
1 = ||
2e M
Me.
(10)
M
M,
+ SWAZ
Swiz
which highlights the symmetric nature of the loss. Minimizing this term encourages M₁ and Qk to align their representations in zones where their feature discrepancies most pronounced.
DWAZ Mutual Learning: Asymmetric Decision Enhancement. DWAZ characterizes zones where the predictive distributions of the student model S and teacher model T diverge most in decision space. expose such zones, a set of mix ratios = {Bi,.... ,......,,,,. 3m} and patch sizes G [91...... :9m are defined for HAPM. For each (3, g) E B x G, mixed sample DDw (aPM( B, g) is constructed and passed through S and TT to obtain logits i V predictive gap measured using KL divergence:
and 1/ x The
Lowal D) EEnD KL / (K
-2]
(11)
The KDP explores (B,g) E B x G to identify the pair that maximizes the prediction divergence:
(8*,g") = arg max COWAZ (3, D)
(12)
(3.g)EBXG
Unlike SWAZ, DWAZ direction-dependent due the asymmetry of KL divergence, where KL(p || q) + KL(4 || p). Accordingly, two distinct mixing setups are derived based on the student- teacher role assignment between M2 and Qs: (5) when Mr learns from Qx (i.e.. S = M₁, T Q+), and (52 92) when the roles are reversed. These configurations respectively localize the zones where M2 and Qx diverge most from each other's predictions.
When S MM and T Qs, the corresponding mixed sample is X-Ma-tu and the DWAZ loss for the direction Mr + Qx is:
HAPM( (2: 91.91)
Mx = KKL (a /T) o DWAZ.
( / 772 p
(13)
Similarly, for the reversed roles where S HAPM(2; B3, v²),, and the corresponding DWAZ loss for the direction Qi + Mr is:
Qn and T Mr. the mixed input becomes X M DDWA
KL Pe-Mi -T, DWAZ
(o X
lm/e/)) /T) a (i
Mr s.
( 72
(14)
puizz
Minimizing ÍDwAz in each direction forces Mr and Q2 to mutually align their predictions, focusing on zones exhibiting the largest disparity in their decision outputs.
Based on the combination of base mutual loss, SWAZ loss, and DWAZ loss, second-stage lo- cal gradient descent is conducted to refine both representation and decision knowledge between heterogeneous models. The model parameters are updated as follows:
wit
no V
basc
+ pM- 22 SWWZ
+
MM 2 DWAZ
(15)
0 di no
Me
Me-Mi SWAZ
Ma
DWAZ
By combining global coarse alignment and local KWAZ-guided refinement, FedKWAZ constructs two-stage closed-loop distillation process. Heterogeneous models mutually compensate for semantic and decision-level gaps by emphasizing training on KWAZ

---

4 Experimental Evaluation
4.1 Experiment Setup
Baselines and Datasets. Ten mainstream heterogeneous federated learning methods are selected for comparison, including LG-FedAvg (20), FedGen (48), FML (28). FedKD [36]. FedDistill[14]]
FedProto [30]. FedGH [41. FedMRL 42 FedTGP
44 and FedKTL 45]. Local Training is
adopted as the baseline for the non-federated scenario where training is conducted entirely locally.
Five public
datasets, covering tasks from natural image medical image, are used. The
datasets are Cifar10, 10, Cifar100 [17]. Skin-Lesions- 14 (merged from HAM10000 32 and MSLDv2.0
22, Flowers 102 (23, and Tmny-ImageNet [5].
Training Configuration. The experimental setup involves 20 clients, all participating in each communication round (p- 1). Local training is performed once per round using a learning rate of 0.01 and a batch size of 10, over total of 1000 rounds. Each client splits its local dataset into 75% for training and 25% for testing. The reported result corresponds the optimal averaged test accuracy across all clients at each round. All experiments are repeated three times, and both the mean and standard deviation are reported.
43 two non-llI partitioning schemes are used to simulate data heterogeneity: Pathological Split: For Cifar 10, Skin- 14, Cifar100, Flowers and Tiny-ImageNet (with 10/14/1 00/102/0000 classes, respectively), each client receives 2/2/10/10/20 disjoint classes induce class-wise isolation. Practical Split: For each class c and client i, the sample proportion is drawn from a Dirichlet distribution qe, ~ Dir(3), with B = 0.1, introducing heterogeneity in both class distribution and data volume.
Data Heterogeneity Modeling. Follow FedALA
Modeling heterogeneity scenarios. Two types of heterogeneous setups are constructed for feature extractors and classifiers. Feature extractors follow the HtFEx scheme, where client i is assigned
the (i mod X)-th model. An average pooling layer with output dimension K = 512 is applied for feature alignment. The main test setup, HtFEg, including 4-layer CNN (21). GoogleNet [29], MobileNet_ __ 27). and ResNet variants [T]. For classifiers, together with HtFEs two Transformer architectures Viniins---V--TEEEEE VIT- and ViT- B/32 8 with different classifier architectures of ResNets, form the 1oM setup. Furthermore, the HrFEs-HHCA introduces dual heterogeneity by pairing the eight extractors with four distinct fully connected classifiers. Methods requiring homogeneous classifier structures (LG--FeAAvg, FedGen, FedGH) are excluded from this comparison. All architectural details and extended results are provided in Appendix D and Appendix
4.2 Performance under Two Data Heterogeneity Settings
As shown in Table 1 FedKWAZ yields accuracy improvements 4.54% and 3.80% on Cifar100 and Flowers102 under the Dirichlet split. For the Pathological scenario, further gains of 4.66% and 3.87% are reported on C0 and Imag--m These benefits arise from KWAZ-driven knowledge refinement, by adaptively targeting and supervising cognitively misaligned samples via SWAZ and DWAZ, FedKWAZ effectively addresses heterogencous label assignment and sample imbalance. Performance trends in Figure confirm its stability across tasks.
Table 1: Accuracy across datasets with top three results highlighted as first second and third
Kettings
Seiting
Perholagicn Senting
Lifart0
Skin-Lekzze::4
Citari00
Fimert00
Tar Imags
Skin-Lenmee14
Cdietco
Riesen102
Timy-maaeeN
Taze
54-21-00.1
19:10 01-10-1111
49:00-11
1510111
9161-0.18
50
3187101
36-01-011
57.80-0:31
85001-011
51.10-0.1 0.18
41:13-01:2
45.15.10:2
21.00 0:35
911901133
05115:1 0..31
11.58-0.31
56.57-00 16.57-0..5
13.41-0.15
Trakten
29:21/0113
30-17 0.11
40/09 (1.20
40:19101.5
23.00 0.15
60 10.30
9101 0..55
51.51 0:53
4216 (1.70
12.0100.0
1.111
0.30
40:17
454611115
0/18
64021001
91.24 0.15
$1.51.0.30 0.00
56/65 (176
21
52.000011
FML
06:44:2119
56384:0.16
3944:5020
45.354017
13:5-0018
56.26:0225
31,08011
FeOGH
83 73.77-3445
83 25
35 36.83.0 11
45
22/02012
53.59 236900 15
03.55 13250.11 19
55.25-0133
$9.41:036
31 11.7500 m3
FeOMRL
96.76--011
47.22-027
45.47:41166
53.77-022
51301-021 21
14.33-0.24
23.55-0..21
02.41=0.00
33.836011
PedTTPPP
5810750042
0700=001
73:003
FeaKWAZ
90.39:8.12
8912
5169:8014
57.5220131
303558009
9L1759.11
14.27:8.88
06.39:8.17
71.28:0.29

---

Skin Lestons---
Citar100
Flowers102
ImageNet
Puthological Setiing
Patholopical Setting
Pathlogical Setting
Fartelopical Serting
Accuroty (N)
Accundy (N)
60
16
2
Feeti pertigge
Prooein to reoto
reeeTs reermee
Feathent paatin
PuNiTL resi
Fecttem Les
Figure 3: Performance comparison under pathological and dirichlet data heterogeneity settings.
4.3 Performance under Various Model Heterogeneity Settings
As shown in Table 2 most methods experience noticeable performance degradation as model hetero- geneity increases, particularly during the transition from the homogeneous HIFE, the heteroge- neous HtFE, Figure (left one) highlights the sensitivity of cross-client collaboration to architectural variation. In contrast, FedKWAZ performs architecture-aware distillation by decoupling semantic and decision discrepancies, enabling path-specific alignment across heterogeneous models. It yields a 2.45% average gain and outperforms FedTGP by 3. 3.26% under the HiFEg-HIU setting.where where both extractors and classifiers are heterogeneous, demonstrating high adaptability to model diversity.
Table 2: Effects of model heterogeneity and client scale on Cifar 100 test performance.
Settings
Different Degrees Model Heterrggneety
Large Client Amount
HiFE,
HIFE,
HIPE,
HIFE,
HIM
HIFE,-HIC,
Clicats
Clissts
280
Local
31,59100
26
110.21
32.61
LG-FooAAg
15.4911.14
38.54:11.11
36.264-015
30.34=00.1
53.19:0.116
4470011117
43.26:10.15
38.42::111
30.85::0.1
FedKD
53.09:0.10
26.1720112
34.391:0.33
30.341:.15
44:33:0.45
16.1010.11
34.82-1112
32.243:015
40.41.1..16
29.27
FoITCPP
5048:00344
02:00
PNKTT
53.53:0.15
FerKWAZ
5633::017
52.05:0.12
52.42/19.33
48.38-0.22
43.7016013
55
Aauenae Ph
Phi
G Ph
Peltyy
Pull
Pults
25
HPS
Model Entrogeeiilyyy Degres
HPP
HMMp
H4P HCC
FOOT Feelees PaRKTL PackWOL
POOTOP
FOOMeEL PARKTL PodkRou
POITOP FOURRNG PARKTL POOKANU
Figure 4: Test accuracy across various model settings (left one) and client scales (right three).
4.4 Performance in Multi-Client Setup
Scalability is evaluated with 50, 100, and 200 clients under partial participation (p 0.5) simulate intermittent communication. As shown in Table 4 most methods exhibit notable performance degradation as client count increases and data becomes more fragmented. In contrast, FedK maintains margins of 4.58%, 2.80%, and 3.89% across the three scales, benefiting from its capacity to extract transferable cues from HAAM--generate inputs and enhance cognitively divergent zones, thus alleviating data iparsity-induced degradation. The per-client accuracy distributions in Figure
8

---

(right three) further support this trend, with FedKWAZ presenting higher medians, smaller variances, and more concentrated distribution, indicating improved utility under large-scale scale deployment.
4.5 Communication and Computation Cost
Table repoots the theoretical and empirical communication cost per round. In FedGen, the auxiliary generator is denoted as θ; in FML,, FedKD and FedMRI., Og and wy represent the proxy feature extractor and classifier, with singular value decomposition (SVD) applied to 0g at compression ratio r, where 10,1 > K X C' and C, is the class count for client i. In FedKTL, the generator latent size is KG.
Table 3: Communication cost per iteration of HtFEs model group in theory and practice for C0far100.
Iotal
Uyloed
Dowsload
Upkool
[
Dowekkoo
Egg Cx
x
0.200
0.2M
Pani Isa]
1833
1.00M
21MM
18, [all
Portec Lowal [a]]
1633
7.604
5.004
PML
187 [8. bag
24 25x bey)
13,304
15.50M
57.00M
and the generated image has C₁ channels and spatial size S. "M" denotes "million".
Deatt
RaMRL
(8) b6g
K
13,004
18.50M4
180M
57.00M
1,775
beatur
Fl K X.
20 x
16463
1.0M
1.4M
PeRKIL
00993
7.31M
7.20M
Substantial transmission overhead is intro- duced by FMI., FedKD, and FedMRL. due to the need for proxy model exchange. FedGen further amplifies downstream bandwidth via generator transmission. Although FedDistill maintains low communication volume, its accuracy degrades due to limited information content in transmitted logits. FedKTL also exhibits increased downstream load during generator- based knowledge transfer.
FORWWA
22 + 02+2
1.6M
2.4M
3.50M
Compute
Fosthern 93
1.20
LG-PedAogill 122
PM
1..0
Fedto
reatroto Miae
FeeKD
FedMD
FodKD
FNL
Fedtae 44
Pacmon
PodWAL 2049
Pester
PodMML
4:
FaskTL 0.3
PodTOP
FecTOP
1254
Foutor
FodrWAz
Peatvoo Coenn
13
206
500
2003
4000
Cest (Hocess Reansu
Figure 5: Comparison of communication and computation costs across methods.
As shown in Figure 5 [a), owing to the use of class-level semantic representations and prediction outputs as compact cross-model carriers, FedK WAZ achieves substantial reductions in both upload and download communication volume. Figure 5 (b) reports the number of communication rounds required by FedTGP, FMI,, FedKD, and FedMRL. to reach peak performance, compared to the rounds needed by FedK WAZ to match these baselines. Total FL.OPs per local forward-backward pass are also evaluated. Benefiting from its two-stage mutual learning scheme, FedK attains and surpasses baseline performance with fewer rounds. As illustrated in (d),, both communication and computation costs are reduced, improving adaptability under figheffre interactions.
4.6 Ablation Study
Three ablation variants are constructed to isolate the contributions of FedKWAZ components: (1) Local Mutual Learning (LML), which omits the
Table 4: Ablation study on four datasets in the practical setting using HiFEg
global alignment stage; (2) One-way KWAZ learning from proxy model to private model Sroxy-to-Private Learning. P2P-L); (3) One- way KWAZ learning in reverse from private model to proxy model (Private-to-Proxy Reverse Learning. P2P-R). As reported in Table 4 Fed-
LMI.
P2P-1.
P2P-R
FodkWWA
Cifar10
89.4210.10
19.6010.15
89.9510.13
90.391:0112
Skin- Lint-Lesooo-tt 14
88.7011.118
19.12.022
89.3010.16
0.19
Cifar 100
50.36:00116
5091-0012
51.1310.19
5..690.144
Flower1102
56.4420.25
568772.17
57.0810.23
57.52::021
KWAZ consistently outperforms LMI. by 0.97%, 1.12%, 1 1.33%, and 1.08% on C0 Skin-
Lesions- 14, Cifar100, and Flowers102, respectively. Figure 6 (left) shows that one-way transfer pro-

---

vides limited gains over LML, suggesting that cross- model knowledge gaps remain under-addressed. In contrast, the full FedKWAZ with bidirectional KWAZ learning and global alignment, achieves markedly superior results, demonstrating its efficacy in focused and deeper knowledge transfer.
51.4
H4=0, 97
1y-0
H4
Hyn0
Accuracy (5)
Accuracy
$1.0
12-0
01 D3
$1.34
50.6
1g
89.4
OFAR-10
GIAAR100
Methods
Search Interval (k)
Figure 6: Ablation study results (left two) and hyperparameter study results (right one).
4.7 Hyperparameter Study
As shown in Figure 6 (right one), the impact of updating HAPM mixing configurations via KDP every la rounds is evaluated with (10, 20, 30, 50). Frequent updates (k 10 or 20) lead to performance degradation, as the model fails to fully exploit the knowledge learned from KWAZ. Conversely, infrequent updates (A: 40 or 50) hinder the timely detection and bridging of emerging knowledge gaps, limiting the effectiveness of KWAZ-guided learning. Accuracy consistently peaks 30 across datasets, suggesting that a balanced schedule between knowledge absorption and dynamic gap correction is essential. Overall, FedK WAZ sustains strong performance across different k, validating the effectiveness of its dynamic update strategy.
Conclusions
This paper proposes FedKWAZ, cognitively structured mutual learning framework that captures semantic and decision discrepancies across heterogeneous models by decomposing KWAZ into SWAZ and DWAZ. Through HAPM-based sample structuring and KDP-guided discrepancy local- ization, KWAZ is dynamically identified and reinforced, offering a fine-grained perspective beyond conventional output alignment. Comprehensive experiments across diverse datasets and architectures validate its effectiveness under both model and data heterogeneity in federated learning.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grants 62576062, 6233 1008 and 62276039, in part by the Science and Technology Research Program of Chongqing Municipal Education Commission under Grants KJZD-M20250060 and KIZDDKK02200606, in part by the New Chongqing Youth Innovation Talent under Grant CSTB02ANSCOOCXMXX00
10

---

References
[1] Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Modell-heteoogeneou federated learning with rolling sub-model extraction. Advances in Neural Information Processing Systems, 35:29677-29699 2022.
[2] Shams Nafisa Ali, Md Tazuddin Ahmed, Tasnim Jahan, Joydip Paul, SM Sakeef Sani, Nawsabah Noor, Anzirun Nahar Asma, and Taufiq Hasan. A web-based mpox skin lesion detection system using state-of-the-ae deep learning models considering racial diversity. Biomedical Signal Processing and Control, 98: 106742,,, 2024.
[3] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXtv: 1912.00818, 2019.
[4] Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, and Salman Avestimehr. Revisiting sparsity hunting in federated learning: Why does sparsity consensus matter? Transactions on Machine Learning Research, 2023.
[5] Patryk Chrabaszez. Ilya Loshchilov, and Frank Hutter. A downsampled variant imagenet as an alternative to the cifar datasets, arXiv preprint arXiv: 170.00888,, 2017.
[6] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pages 2089-2099. PMI.R, 2021.
[7] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, arXiv preprint arXiv:2010.01264, 2020.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[9] Peng Han, Han Xiao, Shenhai Zheng, Yuanyuan Li, Guanqiu Qi, and Zhiqin Zhu. Fedmkd: Hybrid feature guided multilayer fusion knowledge distillation in heterogeneous federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2025.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[11] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0255: 2015.
[12] Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis. Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated lcarning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34:12876-1220 12289, 2021.
[13] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10143-10153, 2022.
[14] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyur Kim. Commuiicatoon-ellcciettt on-device machine learning: Federated distillation and augmentation under non- iid private data. Advances in Neural Information Processing Systems, 2018.
[15] Shivam Kalra, Junfeng Wen, Jesse C Cresswell, and Maksims Volkovs. Decentralized federated learning through proxy model sharing. Nature communications, 14(1):2899, 2023.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X6v 2014.
[17] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[18] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper. broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pages 5542-5550. 2017.
[19] Haoyuan Liang, Xinyu Zhang. Shilei Cao, Guowen Li, and Juepeng Zheng. Tta-feddg: Leveraging test-time adaptation to address federated domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages
2025.

---

[20] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen. Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint urXiv:2001.0122,, 2020.
[21] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication eettiiiiiaatoo-efiieenttttt learning of deep networks from decentralized data. In Artificial intelligence and PMLR, 2017.
statistics, pages
[22] Lei Meng, Zhuang Qi, Lei Wu, Xiaoyu Du, Zhaochuan Li, Lizhen Cui, and Xiangxu Meng. Improving global generalization and local personalization for federated learning. IEEE Transactions Neural Networks and Learning Systems, 36, 2024.
[23] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[24] Zhuang Qi. Lei Meng, Zitan Chen, Han Hu, Hui Lin, and Xiangxu Meng. Cross-silo prototypical calibration for federated learning with non-iid data. In Proceedings the 31st ACM International Conference on Multimedia, pages 3099-3107, 2023.
[25] Zhuang Qi, Lei Meng, Zhaochuan Li, Han Hu, and Xiangxu Meng. Cross-silo feature space alignment for federated learning on clients with imbalanced data. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pages 19986-19994 2025.
[26] Zhen Qin, Shuiguang Deng, Mingyu Zhao, and Xueqiang Yan. Fedapen: Personalized cross-silo federated learning with adaptability to statistical heterogeneity. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pages 1954-1964, 2023.
[27] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4510-4520. 2018.
[28] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Zheqi Lv, Kun Kuang, Chao Wu, and Fei Wu. Federated mutual learning: collaborative machine learning method for heterogeneous data, models, and objectives. Frontiers of Information Technology & Electronic Engineering, 24(10):130-1102, 2023.
[29] Christian Szegedy, Wei I Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1-9, 2015.
[30] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chenggi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings the AAAI conference on artificial intelligence, volume 36, pages 8432-8440, 2022.
[31] Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and ling Jinng Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems, 11119993994444 2022.
[32] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[33] Michael Tuchler, Andrew C. Singer, and Ralf Koetter. Minimum mean squared error equalization using priori information. IEFE Transactions on Signal processing, 50(3)673-683, 2002.
[34] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 911), I), 2008
[35] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[36] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication-cffiicnnn federated learning via knowledge distillation. Nature communications, 13(1) (3)322 2022.
[37] Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang. Guogang Zhu, and Hao Su. Decou- pling general and personalized knowledge in federated learning via additive and low-rank decomposition. In Proceedings of the 32nd ACM International Conference Multimedia, pages 7172-7188 2024.
12

---

[38] Xinghao Wu, Xuefeng Liu, Jianwei Niu, Guogang Zhu, and Shaojie Tang, Bold but cautious: Unlocking the potential of personalized federated learning through cautiously aggressive collaboration. In Proceedings the IEEE/CVE international conference on computer vision, pages 19375-19384, 2023.
[39] Xinghao Wu, Jianwei Niu. Xuefeng Liu, Guogang Zhu, Shaojie Tang. Wanyu Lin, and Jiannong Cao. The diversity bonus: Learning from dissimilar clients personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2025.
[40] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmn mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07771 2017.
[41] Liping Yi. Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. Fedgh: Heterogeneous federated learning with generalized global header. In Proceedings of the 31st ACM International Conference on Multimedia, pages 8686-8696, 2023.
[42] Liping Yi, Han Yu, Chao Ren, Gang Wang, Xiaoxiao Li, et al. Federated model heterogeneous matryoshka representation learning. Advances in Neural Information Processing Systems, 37:66431-666 2024.
[43] Jianqing Zhang, Yang Hua, Hao Wang. Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11237 11244, 2023.
[44] Jianqing Zhang. Yang Liu, Yang Hua, and Jian Cao. Fedtgp: Trainable global prototypes with adaptive- margin-enhaneca contrastive learning for data and model heterogeneity in federated learning. In Proceedings of the AAAI conference on artificial intelligence, volume pages 16768-16776, 2024.
[45] Jianqing Zhang. Yang Liu, Yang Hua, and Jian Cao. An upload-efficient scheme for transferring knowledge from server-side side pre-trained generator to clients heterogeneous federated learning. Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition, pages 12109-12119, 2024.
[46] Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. Advances in Neural Information Processing Systems, 28, 2015.
[47] Zhuangdi Zhu, Junyuan Hong, Steve Drew, and Jiayu Zhou. Resilient and communication efficient learning for heterogencous federated systems. Proceedings of machine learning research, 162:27504, 2022.
[48] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free free knowledge distillation heterogeneous federated learning. In International conference on machine learning, pages 12878 12889. PMLR, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions are clearly and accurately articulated in the Abstract and Introduction (Section [] including the concept of Knowledge Weak-Aware Zones (KWAZ), explicitly subdivided into Semantic Weak-Aware Zones (SWAZ) and Decision Weak-Aware Zones (DWAZ). Furthermore, a dual-stage deep mutual learning mechanism integrated with the proposed HAPM and KDP methods effectively enables knowledge transfer among heterogeneous federated models. These stated claims align consistently and comprchensively with subsequent theoretical analyses and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Potential limitations of this work are explicitly addressed in Section G includ- ing the use of fixed temperature parameter during prediction distillation and the uniform setting of feature dimensionality K across datasets. The potential for dynamically adapting these configurations to better accommodate model complexity and data characteristics is also discussed.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
14

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: A comprehensive set of theoretical assumptions and corresponding detailed
proofs are provided explicitly the Theoretical Proofs (Section(C) (C. This section thoroughly
covers Lemma I 2 and Theorem [] [2] including clearly stated assumptions (Assumptions
3 and as well complete proofs (Proof of of
and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All critical components required to reproduce the experimental results have been thoroughly disclosed. The experimental protocols, dataset configurations, training routines, and modeling heterogeneity strategies are detailed in Section 4.1 with supplemen- tary pseudo-code provided in Appendix B Furthermore, implementation-level specifics provided in Appendix DD including hy typerparameter settings (Appendix D.3). experimental environments (Appendix D.I.. and architectural design choices (Appendix D.4). These collectively ensure full reproducibility the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
15

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Open-source implementations have been made publicly accessible via the GitHub repository linked in Appendix D.I All datasets used in the experiments are publicly available, and their access links are clearly listed in Appendix D.2 Together with the
algorithm description and hyperparameter configurations, the provided resources support faithful replication of the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Sublic/guides httee//uideee/eeebiieeeeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.ccppubbl httpssscccbbblccguiiciiie http:///guddeee/eeee htt.a///////gddee///eeuuuiiiaaia ssi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baseline. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Complete training and evaluation details are provided in Section 4. including data partition strategies (pathological and Dirichlet), number of clients, training/testing splits, learning rate, batch size, and local/global update frequency. Appendix D.3 expands
16

---

on baseline- baseline-specifie hyperparameter settings, optimizer types, distillation parameters, and HAPM configurations. These ensure the setup transparent and comprehensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experimental results are reported with mean and standard deviation over three independent trials, stated in Section[4.] This allows evaluation of result consistency and robustness. Error ranges are numerically indicated in all tables and figure legends, providing a reliable measure of statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on the computational environment are clearly stated in Appendix D.I where it is noted that all experiments were executed on four NVIDIA GeForce RTX 4090
GPUs (24GB) using the PyTorch framework. This allows readers to understand the scale of resources required and estimate the expected runtime and memory consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuripss httppppbcc//iiiiccsuuuddoooooedd
Answer: [Yes]
Justification: The NeurlPS Code of Ethics has been carefully reviewed, and the research presented in this paper fully conforms to all stipulated ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts explicitly discussed in Section H Positive societal impacts are addressed, including enhanced adaptability of heterogeneous federated models, reduced communication
with improved privacy protection, accelerated convergence and improved system energy efficiency, and the expanded potential for practical applications across domains such as healthcare, smart manufacturing. and smart cities. No direct negative societal impacts have been identified; however, careful monitoring is encouraged when deploying the technology in critical domains.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
18

---

Answer: [NA]
Justification: No significant risk of misuse or dual-use associated with the released code or models has been identified in this work; therefore, explicit safeguard mechanisms not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All utilized datasets and related assets have been properly credited in Sec- tion D.2] The licenses and official sources for the datasets, including Cifar10, 1ifar100, Flowers102 Tiny-ImageNet, and Skin-Lesions- have been explicitly mentioned, and their terms of use have been properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpppssiithhoeeeooo thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The newly developed code for FedKWAZ has been publicly released and prop- erly documented, with access information provided Appendix Detailed descriptions of hyperparameters, architectures, and experimental setups have been provided to facilitate reproducibility and reusability.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing or human subject experiments have been involved in the research presented in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human participants, and therefore IRB approvals or equivalent institutional review procedures are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

Appendices
An overview the Appendix is provided below, covering algorithmic components, theoretical founda- tions, implementation specifics, extended empirical analysis, and broader contextual considerations.
Hierarchical Adaptive Patch Mixup
22
A.1 Hierarchical Random Partitioning of Spatial Patches
22
A.2 Patch-wise Independent Interpolation and Fusion
22
Pseudo Codes Fedk WAZ
23
Theoretical Proofs
24
Proof of Lemmal
25
Proof of Lemmal
26
3
Proof of Theorem
26
4 Proof of Theorem
26
Additional Experimental Details
27
D.I
Experimental Environment
27
D.2
Datasets
27
D.3
Hyperparameter Settings
27
D.4
Detailed Setting of Model Heterogeneity
27
Additional Experimental Results
28
Impact of Feature Dimensions
28
Homogeneous Models Setting
28
Performance Fashion-MNIST under Pathological and Dirichlet Data Settings
29
Performance under Low Client Participation Rates and High Client Drop Rates
29
E.5
Impact of SWAZ and DWAZ
31
E,6
Impact of Feature Shift
31
7
Effectiveness of HAPM vs. Hard Sample Selection
32
Correlation between HAPM Parameters and Client Properties
33
E.9
Variance and Stability Analysis of Dynamically Selected HAPM Parameters
34
Visualizations Data Distributions
35
Limitations
37
Broader Impacts
37
21

---

A Hierarchical Adaptive Patch Mixup
To explicitly capture knowledge discrepancies between heterogeneous models, novel Hierarchical Adaptive Patch Mixup HAPM mechanism is introduced in FedKWAZ. High-quality samples with local diversity and hierarchical structure are generated through spatially granular partitioning and stochastic perturbation, supporting the identification and enhancement of weak knowledge-aware zones. The input HAPM includes the images x, a mixing strength parameter a, and a spatial granularity parameter G. Two key procedures are involved in HAPM: hierarchical random partitioning of spatial patches and independent interpolation of partitioned samples.
A.1 Hierarchical Random Partitioning of Spatial Patches
To reflect knowledge differences across heterogeneous models at a local granularity, a spatially hierarchical random perturbation mechanism is incorporated into HAPM. The original input images X BxCx H xW (where B, C, H, and W denote the batch size, channels, height, and width) is partitioned into VG x VG local patches. Fach patch is then subjected to spatial perturbations to improve diversity and sensitivity to knowledge. Specifically, the base height and width of each patch are defined as:
H
W
hbaas
LVG]
Wbase
LVG
(16)
Next, the initial boundaries of each patch are constructed based on grid indices:
(i.f) haan /thase (Ga))) (i + Whuse:
Wstart
(4.5)
j
Whase:
Wend
(j + 1) Whase (17)
Here, the grid indices i,j = 0, 1..... VG - are used to specify patch positions. To increase the
diversity of local perturbations. jitter factors o/ ( and g(1-3) are introduced and sampled as integer variables from uniform distributions:
((155) u
(-1
/hass
hbsse
((5-5 1l
(- W.bss
Wbase
(18)
4
To ensure validity, the perturbed positions are clipped within image boundaries using the clip function:
((,)) = clip h + g(2.5) ,,,, 11 Aind) = clip
( (b-f)
Rnn +
(i-5)
,,,000,00 H,HH)
(19)
(i.j) @gaatt clip start + g(1d), 0, W), ), (t.j]) clip w (and
(f.j)
((,)) W) 6
These designs jointly form the hierarchical structure of HAPM, where local-granularity patches are first divided in space and then perturbed at the patch level to enhance the diversity of the mixed samples.
A.2 Patch-wise Independent Interpolation and Fusion
Based on the above spatial partitioning, the mixing coefficient (b.J) is independently sampled for each patch zone from Beta distribution:
(i-)) Beta(a, a)) >>>>>
(20)
Given randomly shuffled batch index the interpolation fusion is independently performed in each patch zone:
x i, (((1)) T,15.) Gstart afttd)
X (b.) T.(i.) Titdd) (h.))
Watart ا(hاا
(21)
+ (1 (x
Aetar
(1,5) (G3)) ((..)
Staatt ((tا)
Through this mixing process, each local zone of the image is independently adjusted based on the patch-specific mixing strength, enabling HAPM to generate more diverse and moowledge-sensitrve samples, thereby enhancing the model's recognition capacity in knowledge-weak zones.
22

---

B Pseudo Codes of FedKWAZ
Algorithm I: FedKWAZ
Input: N, total number of clients; P. participation of clients in round; T, total number of rounds; 1) learning rate of private and proxy models; EB,,BB, training epoch of the first and second stage; o ,91 88 g" mixing parameters HAPM; r, temperature of distillation; Dx. the local dataset of the k-th client. Output: Private and proxy models' PcMk, Pe and Le Le Randomly initialize client local heterogeneous models [Mi((1)) (N. MNNI and local proxy homogeneous small models (01)) @N-1(oN-1)] for each round t=I...,7-1 do
// Server Side:
St +- -Randomly sample K clients from N clients; Broadcast P.M, PC, LEM, LE them; for each client & = L..... ......... K do CliemUppate(( P.M. P. T.M T.2) end
|
Aggregate global semantic representation and decision output. I/ ClientUpdate:
Receive P.M, P L.M. Le from the server; for k E St do
I/ Stage I: Global Knowledge Mutual Learning
for epoch e 1..... Ea do
for batch (x, y) Dr do
Obtain representation and logits:
zt 0/ +- Ma th.,, 2x y Ox
+ -+(=;
Compute global alignment loss:
t MM,+ lorly My v) + EcE
+ 2/ M, Pe 11:
c tox(52" 2h v)) foe ye o(LC))
+ 2. P.M 11::
Update: wi vit no Ve VeMa
+e 01 0/ No Vel Ma
end end
// Stage II: Local KWAZ Mutual Learning for epoch == 1...., FB do for batch (x, y) De do Compute base mutual learning losses:
MM Ux KL
(o() /T) ay Qt /T))
7² + || Mr 2
(O KL
(y 2+ /r) a /T)
72 20 2
Generate SWAZ mixed samples: Xswaz
HAPM(E; a²,g"):
Compute semantic weak-awareness loss:
MM2 Rx
<SWAZ
= M2
Ce x-Mg =13=
SWAZ
Me
Generate DWAZ mixed samples:
X Y-2--2 HAPM(x; gi)) X X HAPM(T;B2.00)
DWAZ
Compute decision weak-awareness losses:
4M2-+
KL
o(j Mi /T) 00
2x
/T)
.22
10-+Mr KL DWAZ
aty Ok Cs MM, /T)
(1 2,
/T)
.72
x DRizz
Dwizz
M
Update models again:
W +- 1/ no MaMe
PAAAz 2b |;
GSwAZ 12 Mr
ok <- di no
+
e 3M ME
J:
end end
Aggregate and upload updated Me pp M,MM L0 to the server.
end
end
23

---

C Theoretical Proofs
The following notations and expressions are defined. E 0, T 1 is used to denote the t-th round of federated communication. The local loss function Ls(wx) is defined over the parameter set wg of the local model Mr and proxy model Qs on client k. In each communication round, a total of E = EA + Eg local steps are conducted on client k in two stages. The start of the local update at communication round is denoted as tE + O. where the first local iteration of Stage begins after the global semantic and decision anchors are received from the server. Stage I involves Ε steps of local update, with parameter sequence denoted as { (f Ex at the end of this stage. Starting from w EA, Stage II proceeds with Eg local update steps, with EA +
Je=0 Je=0 and wi +E++EA EA indicating the parameters
Eg represents the final parameters after the t-th local training round. After Stage II, the local representations and decision outputs are uploaded to the server for global aggregation and update, initiating the next communication round. The learning rates on client k are unified into a scalar n, comprising The and T/6-
parameters denoted by {w (w EA Eg where W le=E,
Assumption 1. Lipschitz Smoothness. The local loss function on any client k is assumed to satisfy the I I-Lipschitz smoothness condition. Specifically, for any parameter vectors w w it holds that:
V (wif ;z, v) V V (w x. 3)1 S Lillu w 11. Vt1.t2 > O, (x. E Dr
(22)
Moreover, this can be further expressed as:
ch c S ( (w - w + L/2 || w
Ia
(23)
Assumption Unbiased Gradient and Bounded Variance. On client k, during the t-th local update, the batch gradient sampled at w is denoted as g'm = V V (wff BB)). and it is assumed to satisfy unbiasedness and bounded variance:
Unbiasedness:
E bu. = (.((
(24)
Bounded variance:
EBECD ( BB) VL (mxt) ||] S 0²
(25)
Assumption 3. Bounded Gradient in Global Alignment. Inspired by the theoretical bound on semantic alignment loss variation per round in FedProto, the first-stage local update iterations in FedKWAZ are defined, during which global semantic and decision alignment is performed on each client's 's local model. The resulting gradient shift from each single round is uniformly upper-bounded as:
V Eit+ k. Globbalaaign 11255 S 82, Ve E Eo
L.. - 1}.
(26)
Assumption Bounded Gradient in KWAZ Alignment in the second stage of WAZ (the KWAZ learning stage), the knowledge alignment operations (including SWAZ and DWAZ) performed between the local heterogeneous model and the proxy model on the client side are assumed to exhibit a unified bound on per-step gradient variation:
V C.C.E.
E. K,KWAZuign 112 < 22,,200 Ve E (EA, + +,...,, EA + EB 1], E NO. - 13
(27)
Lemma 1. Stage-I Bias. Given Assumptions 7 aandd 3 After the completion of the first stage, the following inequality is satisfied for any client:
(1
Lnn
Ex
ELCx (u [E+E S Cu(W
|| (w 112
+ 16
2
2
(28)
Lemma 2. Stage-Il Bias. Given Assumptions ! the following inequality is satisfied for any client:
2 and 4
After the completion of the second stage,
Tnn²
Ea+Eg-1
E (w EA <E (c((bbEEA))
(VL
-
2
(29)
L LIEn'0002
+
+E
2
24

---

Theorem 1. One Complete Round of FL. Given the above lemma, for any client, after the two-stage mutual learning has been completed, the following inequality holds:
((10)-1()))) Lin2
Ba + En
E (ILtttEtEAAttB S (x(
Σ
V
(uck t l12
(30)
Ia F.B)TPO²
+ (E + EBA?)
2
Theorem Non-convex Convergence Rate of FedKWAZ. Given Theorem For any client and any constant 0, the following inequality holds:
EB-
V LL (wtttt
+ =
Cu
n1(EE8 EBT²)
2
<e
Lny²
2(e (EĄ02 EB7)))
s.1. 0 0<リ<
Li(e + (EA + En)o²)
(31)
C.1 Proof of Lemmal
1
Proof. In the e-th local update of the first stage (e 0, 1,.1.... E 1), any client is is considered, and the result can be derived based on the Lipschitz smoothness in [
(w (wt+)+ I E[ [[C + (VLx ul
wB + [lll
11
Ca E [ 1R -1996 Butt La lonk +
Lu 7i E (V Bt0) g [btte]] + L [| [
Lu (c) m/ V La
12 Lu
(119
2F FF 1
(d)
I Le(wt (1)) n| C EE
"اا [ 12
LA (u
- 11 أبي [u
- ||2 + Liq'o"
Step (a): Based on Assumption] 1
L1 is approximated near w LE using a first-order Taylor expansion. iE+e is applied, where g :E+e denotes the stochastic
Step (b): The update rule w 1E++++ w lE+e - -99 gradient over the current batch.
Step (c): Based on Assumption (unbiased gradients).
Step (d): From Assumption (bounded variance of the gradient), it holds that
E|ll
9
EE (u
EE+e 1 < <²
(32)
From this, the following upper bound derived:
Elll
gk E+e |
|
(w (Bt)) 11 to"
(33)
By summing the single-step inequalities from 0 EA 1, the following result is obtained:
ElCk (u (wt4+0)) (1) L
E
LIEATO
(w (Et)
(34)
Furthermore, based Assumption 3 (bounded global alignment gradient in the first stage), a per- round gradient variation upper bound 82 is introduced, leading to an extra term 1 the total sum. Finally, Lemma Tis fully expressed as Eq. 28
D
25

---

C.2 Proof of Lemma 2
Proof. Similarly, based on Assumptions I and 4 after the second stage of FedkwAZ (KWAZ
knowledge alignment stage), the expected local loss at any client lo is shown to satisfy Eq. 29
local
C.3 Proof of Theorem
Proof. By substituting Lemma into the right-hand side of the inequality in Lemma 2 obtained.
Eq.
30
be
C.4 Proof of Theorem
Proof.
The left and right sides of Theorem are swapped, and the gradient terms are rearranged:
Ex+ Bn
VL (wttte))) 11
(35)
Lu(ub
(u ElLa(wk E + Li(EA-Ealy's + (E + EBT
Laz2 2
the total T rounds of federated communication training, the expectation the above inequality taken for from 0 to T and summed, yielding:
T-1
EA+EB -1
Σ
VCN 1
1=0
e=0
T-1
-B (BEtttt +
BlCx(wk
+ nnAAd² + EB?²)
7
2
Lyd 11 2
(36)
The difference between the loss at the initial time and the optimal loss is defined as:
Lo L
(37)
The expected loss function over T rounds is expressed as:
[u(w (BC $
A
(38)
T
Hence, the original inequality can be further simplified as:
EA+EE-1
A
Li(BA+En)n''
+ (E + EBA
V 1135
T
Lin² 2
(39)
0=
2
Let the expected norm of the modulus in the above equation be expected to converge to constant
+ En) n + EBT?) <e
(40)
17 L,7²
Since the number of training iterations T > 0, and > 0, the denominator must satisfy:
Lnn²
L1
nEEAd² + -BB?) > 0
(41)
2
2
Thus, the upper bound of is obtained as:
2(e (Frg02 + F))?))
0<ク< Li(e + (EA + EB)o²)
(42)
26

---

Since all of the quantities L1, o², 82, and 72 are positive finite constants, the constraint on the learning rate 77 exists in non-empty solution set. When the learning rate nj satisfies the above conditions, the expected norm of the local loss for any client can converge to the constant e. According to the right-hand side of the above equation, except for the term divided by 7. the remaining terms are constants. Therefore, the non-convex convergence rate of FedK WAZ is achieved as: ((1/T).
D Additional Experimental Details
D.1
Experimental Environment
All experiments are conducted on the PyTorch platform. The experiments are executed on four NVIDIA GeForce 4090 GPUs (24GB memory) across five supervised image classification datasets
D.2
Datasets
The sources of the datasets are detailed. The experiments are conducted based on five public multi- class datasets, covering natural and medical image recognition tasks, including: Cifar10
CIFAR h0.mlll).
Cifar100 hhtps://pttorch.or///sooooo/////
Flowers 102 http://ppw...poooooooo/////oo//s CIFARIO0. Liny-ImageNer
Flowers 1iarss00222 102.htll)
and Skin- Skin-Lesions- http://wwww..aggee...oooooooceon are utilized.
D.3 Hyperparameter Settings
In addition to the hyperparameter settings provided in the main text, the hyperparameter configurations for each baseline method are also followed according to their original publications. Specifically, L.G-FeAAv configured with additional hyperparameters: for FedGen, the noise dimension is set to 32, the generator learning rate is set to 0.1, and the hidden dimension is aligned with the feature dimension K, with 100 training rounds the server. The distillation hyperparameters in FML are set as = 0.5 and 3 = 0.5; in FedKD, the proxy model's learning rate is set to 0,01 to match the client; the temperature range for distillation is set to Tta 0.95 and Tend 0.95; for FedDistill, is applied. FedProto is configured with À = 0.1; in FedGH, the server learning rate is set to 0.01 to match the client; the representation dimension For Fed A is set to 0.1, the distillation margin threshold f to 100, and the server training rounds to 100; in FedKTL,
the proxy model in FedMRL is set to 256.
is set to C. μ 50, 1, with server learning rate n7 0.01, batch size B. = 100, and server training rounds F. = 100. Except for FedGen and FedKTI. where server-side training performed using the Adam [6] optimizer, SGDD 44) applied for client- and server-side training in all other methods. In FedK WAZ, the HAPM module is employed to automatically search for optimal mixing strength parameters (a or (3) using KDP within the range 0.1, 0.5, 1.0. The spatial granularity G used to divide each input image into VG VG local patches. Specifically, for CIFAR10 and CIFAR 100 (input size 3 x 32 32), as well as Flowers 102 and Tiny-ImageNet (input size 3 x 64 X 64), G E 64, 16, 4; for Skin-Lesions-1 14 (input size 3 28 x 28), G E 49, 16, 4. In FedKWAZ, the KWAZ update frequency (i.e., the interval & for KDP-based HAPM parameter search) is fixed to once every 30 rounds to balance the ability to absorb prior knowledge and explore novel knowledge; the distillation temperature 7 is set to 4. The training epoch EA and Eg for the first and second stages are uniformly set to 1.
D.4 Detailed Setting Model Heterogeneity
The primary test configuration for feature extractor heterogeneity is defined as HtFEg, consisting of eight representative network architectures: 4-layer CNN 21), GoogleNet 29) MobileNet_ v2 [27]. ResNet18, 8, ResNet34, ResNet50, ResNet101, and ResNet152 deeper are classified under the ResNet series and are used to construct multi-level heterogeneous configurations, including: HtFE, (containing only ResNet4, used to simulate fully homogeneous scenario), HtFB, (composed of CNN and ResNet18, indicating slight heterogeneity). HiFE, (inte- grating CNN, GoogleNet, MobileNet_v2 v2, and ResNet18 8 to simulate moderate heterogeneity). and
10 Networks of ResNet18 and
"Code is available at: http:/thhhw.ccccmccmmm6e6//////
27

---

HiFEg (configured as extremely heterogeneous by combining ResNet4, ResNetó, ResNet8, ResNet10, ResNet18, ResNet34, ResNet50, ResNet101, and ResNet152).
Regarding classifier heterogeneity, the HtC, scenario is constructed to include four types of classifiers composed only of fully connected (FC) layers, simulate structural heterogeneity in decision-making.
The four architectures are defined as: (1) single-layer FC (100-d), (2) two-layer FC (512-d + 100-d) with 512-d hidden layer, (3) two-layer FC (256-d + 100-d), and (4) two-layer FC (128-d 100-d). In the HtFEs-HIC, scenario, the eight types of feature extractors and four types of classifiers are cross-combined according to client indices, thereby forming test environment with simultaneous heterogeneity in both feature extraction and decision modules.
In mutual learning schemes, the proxy model structure is defined simple 4-layer CNN suit methods such as FedKD, FMI,, and FedMRL. that periodically upload parameters the proxy models the server, aiming reduce communication overhead. In contrast, FedKWAZ avoids proxy model transmission entirely and instead constructs global semantic prototypes and decision anchors by aggregating local class-wise features and logits, the proxy model is still uniformly set 4-layer CNN to ensure consistency and fairness in experimental comparisons, and to eliminate the influence of model complexity confounding variable.
E
Additional Experimental Results
E.1 Impact of Feature Dimensions
As shown in Table 5 most algorithms exhibit accuracy gains the feature dimension increases from K 64 to K 256. At 256, FedKWAZ attains an accuracy of 50.95%, surpassing FedKTL by 5. 16% (Figure 7}. indicating its strong capability in leveraging high-dimensional representations for effective knowledge interaction. When the feature dimension is further increased to K 1024, performance declines are observed in several methods, which can be attributed to elevated sparsity and reduced discriminative efficiency in overly expanded feature spaces. contrast, FedKWAZ maintains competitive performance with 50.87% accuracy, supported by its KWAZ-guided knowledge coordination.
In addition, several federated learning baselines are outperformed by the Local method under certain configurations, as negative transfer under dual heterogeneity in data and model structures compro- mises collaborative training. FedKWAZ, by incorporating K KWAZ-guided modeling of representation and decision discrepancies, effectively alleviates cross-model knowledge transfer bottlenecks and maintains consistent performance across diverse tasks and configurations.
Table 5: Impact of feature dimensions (K) on HtFF model group performance Cifar100
20.55
K
256
1024
Local 39.24+0.14
FedDistill
38.93-1..20
44.10=0.15
40.92+0.11
42.56-10.33
40.25++0.18
Accuracy (%)
89
3.44
LG-FedAvg
39.66=40.22
40.15+0.144
41.25=#0.19
FedGen
38.75.10.15
40.2310.19
40.43:10.11
FedKD
40.54=10.16
40.26:20.17
41.08=0.110
FedProto
31.84
35.63.10.12
34.1410..14
200
107
FML.
38.40=+0.18
40.80+0.111
40.59++0.19
FedGH
38.19++0.18
40.01+0.155
38.48+00.17
FedTOP
FedKWAz
FedMRL
39.06:10.14
41.90-0.16
42.95=0011
45
FeeKTL
FedTGP
47.05=0.117
47.87+0.27
47.43=+0.24
64
256
1024
FedKTL
45.98.1.155
45.7910.14
46.76.10.17
Feature Dimension (K)
Figure 7: Accuracy comparison among
FelKWAZ
50.04:20.13
50.95:10.16
50.87=00.12
Fedk WAZ. FedTGP, and FedLTL.
F.2 Homogeneous Models Setting
To further investigate the role of data heterogeneity in isolation, all clients are configured with identical model architectures across three homogeneous settings: ResNet10, ResNet18, and ResNet34, in addition to the original HtFE, (ResNet4) baseline listed in Table 2 By removing model-level
28

---

heterogeneity, these experiments focus solely on the impact of non-llD data distributions on federated knowledge transfer. As shown Table FedKWAZ consistently outperforms all baseline methods under each homogeneous configuration. These results demonstrate that, even without structural differences across models, the proposed dual-stage mutual learning framework remains effective by explicitly aligning both representation semantics and decision behaviors across clients. This joint alignment enables FedKWAZ to bridge knowledge gaps induced purely by data distribution shifts, thereby ensuring stable performance improvements under non-IID conditions.
E.3 Performance Fashion-MNIST under Pathological and Dirichlet Data Settings
On the Fashion-MNIST 440] dataset, the HtCNNg model group is adopted to accommodate the grayscale single-channel input, with partition details listed in Table g A comprehensive evaluation of all methods is conducted under both Pathological and Practical non-IID settings. As shown in Table 7 FedKWAZ consistently yields the highest accuracy in both scenarios. Although FMNIST presents relatively modest classification complexity and most methods perform well, Fedk WAZ maintains consistent performance lead. In addition, Figure presents the t-SNE (4] visualization of learned feature representations under the pathological setting for representative mutual learning baselines (FML,, FedKD, FedMRL) and FedKWAZ Compared these baselines, the features produced by FedKWAZ exhibit stronger intra-class compactness and inter-class separability, reflecting enhanced semantic consistency under model heterogeneity and data distribution shift, and supporting improved cross-model knowledge transfer.
Table 6: Performance using homogeneous mod- els on Cifar100 in the practical setting.
Table 7: Performance FMNIST using the HiCNNs.
Architectures
ResNettO
ResNet18
ReaNe334
Settings
Puthological Setting
Practical Setting
Local
45.38:0.16
42.57-0.12
41.62:10.14
Local
99.38:10.05
97.22:10.09
FedDisill
44.78.00118
44.12-0.21
43.62:10.23
FedDistill
99.42:10.04
97.42++0.03
LG-FedAvg
47.11=0.17
44.33=0.15
44.04:2..00
LG-FedAvg
99.37:10.05
97.22:10.04
FedGen
47.02=0.22
44.53=0.13
44.2740.11
FedGen
99.34+0.006
97.34:0.00
FedKD
45.13-0.11
41.32-0.15
40,26++0.12
FedKD
99.40:10.07
97.36:10.03
FMI.
46.37-0.16
40.23110.18
43.07-0.17
40.25+00.12
38.02.10.15
FedProto
99.39:10.03
97.35:10.02
FedGH
45.30.10.12
43.2910.11
41.84.10.10
FML
99.41110.05
97.3410.04
FedMRL.
47.36:20.09
45.67-0.11
45.40+00.12
FedGH
99.38110.04
97.36:10.03
FedTGP
47.0510.38
45.79.0.35
47,43.10.33
FedMRL
97.08:10.04
FedKTT.
51.20:20.15
50.10:0014
48.17:10.12
FedTGP
99.52110.66
97.33110.05
FedKWAZ
82.35-0.11
52.47 0.14
51,75-00112
FedKWAZ
99.60110.03
97.6111.04
Trouser
Bag
(a) FML
(b) FedKD
(c) FedMRL.
(d) FedKWAZ
Figure 8: T-SNE visualization of features extracted by FML, FedKD, FedMRL and FedKWAZ.on on FMNIST under pathological partition.
E.4 Performance under Low Client Participation Rates and High Client Drop Rates
On the Ci dataset (Dirichlet distribution, 100 clients), client participation is systematically restricted to simulate federated scenarios with limited and unstable communication. Participation rates of 5% and 10% are adopted, corresponding to only 5 or 10 randomly selected clients contributing to each round, while aggregation uses updates only from the clients participating in that round. For FML., FedKD, and FedMRI,, global proxy models are constructed by aggregating local proxy models
29

---

Table 8: The model architectures in the HtCNNg group. Convolutional layers represented as "I5 x 5,32]" indicating a convolution with kernel size 5 x 5 and 32 output channels, and "2 x max pooling" refers to max pooling layer with kernel size x 2.
Sequentially Connected Feature Extractors
Classifiers
CNN1
Conv (5 x 6. 32). 2 x 2 max pool. 512-d fc
10-d fc
CNN2
Conv (5 x 5, 32), 2 x 2 max pool, Conv (5 5, 64), 2 x 2 max pool, 512-d fc
10-d fc
CNN3
Conv (5 x 5, 32). 2 x 2 max pool, 2 x 512-d fc
10-d fc
CNN4
Conv (5 x 5, 32), 2 x 2 max pool, Conv (5 x 5, 64), 2 x 2 max pool, 2 x 512-d fc
10-d fc
CNN5
Conv (5 x 5, 32). 2 x 2 max pool. 1024-d fc, 512-d fc
10-d fc
CNN6
Conv (5 x 5, 32), 2 x 2 max pool, Conv (5 x 5, 64), 2 x 2 max pool, 1024-d fc, 512-d fo
10-d fo
CNN7
Conv (5 x 5, 32), 2 max pool. 1024-d fo x2, 512-d fc
10-d fo
CNN8
Conv x 5, 32), 2 max pool. Conv (5 5, 64), 2 x 2 max pool. 1024-d fc, 512-d x2
10-d
from all clients. In contrast, WAZ performs global aggregation using class- wise prototypes and logits from all clients to form semantic representations and decision anchors. As reported in Table 9 overall performance degrades as participation rate decreases. Nonetheless, FedK WAZ consistently achieves the highest accuracy at both 5% and 10% settings, demonstrating strong resilience client sparsity. Figure further illustrates the prediction matrix at 10% participation, where FedK WAZ exhibits sharper diagonal confidence compared to other methods, indicating improved inter-class separation and stronger decision reliability under sparse client engagement.
Table 9: Performance of FML, FedKD, FedMRL and FedK under low client participation rates and high client drop rates.
Client Participation Rate
Client Drop Rate
5%
10%
90%
95%
FML FedKD
0.05
80.40=00.12
01.55
81.12:t..11
79.1
79.67
80.66+0.116
00.1
FedMRL
00.2 0.09
81.08+0.15
81.65+0.13
81.39+H..10
FedKWAZ
83.71
84.37
03.9
sae con cas cil san sEc Koscos
vale porconss sat ant 602-00
juaee cansas pillez
scecor
SALECOCa1
caeial sai
vates ap cdenal eie sar sccco)
(a) FML
(b) FedKD
(c) FedMRL
(d) FedKWAZ
Figure 9: Confusion matrices of FML, FedKD, FedMRL, and FedK on Ci under dirichlet partition with client participation rate of 0. showing the prediction probability for each class.
To further evaluate communication disruptions, a Client Drop Rate experiment is conducted. In this setting, all 100 clients participate in local training, but only a portion are able to upload their knowledge due to simulated dropout. Dropout rates of 90% and 95% are used, meaning that only 10 or 5 clients successfully contribute to global aggregation. As shown in Table these settings yield better accuracy than the 10% and 5% participation scenarios, suggesting that consistent local participation even under partial upload failure failure-helps preserve training efficacy and partially compensates for reduced aggregation scale.
These results collectively confirm that FedKWAZ, empowered by its dual-stage knowledge transfer mechanism and lightweight semantic anchoring, maintains robust knowledge integration even under extreme communication sparsity and unstable client availability, effectively addressing practical challenges in real-world federated deployments.
30

---

E,5 Impact of SWAZ and DWAZ
The individual contributions SWAZ and DWAZ to the second-stage mutual learning FedKWAZ are further investigated. Specifically, experiments are conducted by selectively enabling either SWAZ or DWAZ while disabling the other. and performance is assessed on the Skin-Lesions 14 and Flowers 102 datasets under both Dirichlet and Pathological data partitions. As reported in Figure T0 DWAZ consistently achieves higher accuracy than SWAZ. indicating that lecision-level alignment exerts more immediate influence on cross-model knowledge transfer. This observation highlights the critical role of decision space consistency in federated mutual learning. Furthermore, the full FedKWAZ-comle
the highest performance across all scenarios, confirming the complementary nature of semantic and decision-leve discrepancy modeling. These findings underscore the necessity of jointly capturing both representational and predictive misalignments to optimize knowledge transfer between heterogeneous models.
both SWAZ and DWAZ-achie
Lesiomm-1440i
Lesiom--1-PPtt
0.90
0.57
(h))
9.89
6.90
0.50
0.85
FeeSWAZ
6:34
FedSNAAZ FeduWAz
FedSWWkZ
0.00
FessWAZ
FoSEHNAZ
FedDWAZ
FoSDWAZ
FestONA
FeakWAZ
FooKWWAZ
0.31
0.93
200
900
1600
200
1003
200
1058
1800
Communication
Communication
Communication
Figure 10: The contributions of SWAZ and DWAZ to performance.
E.6 Impact of Feature Shift
To further assess the robustness of each method under feature shift scenarios, a heterogeneous data experiment is constructed in which each client is provided with a complete set of class labels, but the visual characteristics of the across domains. Two widely adopted domain generalization benchmarks are selected: PACS (18). comprising 9,991 images from 7 categories across 4 domains (Art, Cartoon, Photo, Sketch), and OfficeHome [5], containing 15,588 images from 65 categories over 4 domains (Art, Clipart, Product, Real). For each dataset, samples are split into training and test sets with a 3:1 ratio, and each domain is treated as distinct client, simulating federated feature heterogeneity.
as style, texture, and viewpoint- differ substantially
In model allocation, the HIFE, architecture configuration adopted, where heterogeneous back- bones-including a 4-layer CNN, GoogleNet, MobileNet_v2, and ResNet18 --ae deployed across clients to emulate the hardware and model diversity typical in real-world federated environments. Figure IT illustrates the fomain-induces feature shift across different domains in PACS and Of- ficeHome for a representative category, highlighting the substantial variation visual statistics. Table summarizes the per-domain test accuracy at the final communication round and the overall average accuracy achieved at the optimal round for each method. Notably, the average accuracy not obtained via direct averaging over individual client scores but is instead computed based on the total number of correct predictions across all test samples, yielding a more comprehensive and balanced measure of global performance.
As presented in Table TO and Figure 12 FedKWAZ consistently demonstrates faster convergence, improved communication efficiency, and higher per-domain accuracies. In addition, it achieves the highest aggregated performance, significantly surpassing all baselines. These results suggest that FedKWAZ effectively addresses the representational and decision-level mismatches induced by feature shift through its dynamic KWAZ-based localization and targeted enhancement strategy, thereby enabling more reliable knowledge transfer across heterogeneous clients.
31

---

Table 10: Test accuracy (%) comparison of various federated learning methods PACS and OfficeHome datasets. Each column represents the accuracy on one domain, while "Avg" indicates the mean accuracy across all domains.
PACS
Officttome
Methods
Art
Cantoon
Photo
Steoch
Asg
Art
Cliput
Product
Real
Arg
7866+00.1
+0.07
51.06-0.00
FedDieil
42.06:10.20
78.00:0.11 14
60:7760.21
71:2102
08.1110.2 34
16.14:0.15
49.7329020
8847-0.13
208110.1
36, 26.66:0.11 16.16
LG-FddAyg
7662::0.25
15.79.00.15
14.00:0.19
46434.0.1
37.84-0.10
19.05.0.12
31.57
FedGGen
80.43:10.18
78861:0
57. 18.0000
69.38:0011 16
12.85-0.21
47.2510211
33.56.00.00
10100.244
31,08:10.27
FodKD
07.40:10.0
28.50:0211
62:44:0.35
76.16101
07.01:0.1
16.31
05.0010
0676-0016
180210.1
90 18-10.15
65.31:10:24
33:06:0:225
IMLL
22.21:0.18
05
59.57.0.10
7192:0022
05.11:00
15.17:0.15
43951022
33.34:0.19
17.4210.23
30.3:0014
50.5++0.5
90.33-0.26
15.49-0.15
47.53+0116
10.0000000
PedMRL
19.5510.17
78.00:00155
00.29:0.13
76. 76.001:0.00
67.39.0.19 19
15.82:0.23
51.7/1014
37.39.0.135
17.8110.20
33.11:0.11
PofTGP
S847+4033
76.45-0.3
63. 16.10.10
76 94
68:7140.5
14:38-0.22
47.6011.25
13.77-0.34
1822610.36
9073-0,32
85.50_00.18
Dog Guitar House Person
Chair
Fork
Mouse Pen
Art
Art
Cartoon
Clipart
Photo
Product
Sketch
Real
(a) Domain shift in PACS across four domains.
(b) Domain in OfficeHoome across four domains.
Figure 11: Visualization of feature shift across different domains in PACS and OfficeHome datasets.
0.08
Accunccy (4)
0.59
0.56
200
400
600
800
1003
1003
(a) Performance on PACS
(b) Performance OfficeHome
Figure 12: Comparison of test accuracy across various methods on PACS and OfficeHome datasets.
E.7 Effectiveness of HAPM vs. Hard Sample Selection
To further verify the effectiveness of the proposed Hierarchical Adaptive Patch Mixing (HAPM) module, we conducted additional comparative experiments to evaluate its advantage over naive image distillation and local hard-sample selection strategies in facilitating cross-model knowledge transfer. The experiments were performed on two representative datasets, CIFAR-10 and Flowers 102.
The compared methods include:
32

---

FedKD (Original): The private and proxy models align their features and logits using only original local images.
FedKD + Local Hard Sample Enhancement: The top 10%, 30%, and 50% local samples with the highest feature MSE and logits KI. divergence were selected for enhanced distillation.
FedKWAZ (Full Framework): HAPM generates perturbed mixed samples, and KDP identifies semantic-decision weak zones (SWAZ and DWAZ) for fine-grained mutual learning.
Table 11:
Performance comparison between FedKD, hard sample-enhanced FedKD (top 10%, 30%, 50% samples), and FedKWAZ on CIFAR- 10 and Flowers 102 datasets.
Model
CIFAR-10
Flowers102
FedKD (Original)
86.31
46.67
FedKD top 10% hard sample
86.91
50.02
FedKD + top 30% hard sample
86.72 36.72
50.95
FedKD + top 50% hard sample
86.55
49.05
FedKWAZ.
90.39
57.52
(1) As shown in Table 11 the performance gain from hard-sample enhancement exhibits dataset- dependent trends. On CIFAR- selecting the top 10% of hard samples yields the best result, while expanding the selection ratio slightly decreases performance. This indicates that the most challenging samples are highly informative, but excessive inclusion introduces redundancy and noise.
(2) On Flowers 102, the best performance is achieved with the top 30% hard samples. Since this dataset has fewer training samples, overly strict selection (e.g.. 10%) limits generalization, whereas a moderate 30% ratio provides a better balance between infornativeness and coverage.
(3) Overall, although selecting high-divergence samples can improve distillation, its effect is sensitive to the chosen ratio and lacks robustness. In contrast, FedK WAZ, equipped with HAPM for generating structurally diverse and perturbed samples and KDP for dynamically identifying discrepancy zones, consistently outperforms all sarp--amlle-baee variants aciievi--chieving +4.08% gain on CIFAR- and +10.85% on Flowers 102.
These results demonstrate that HAPM provides a more generalizable and stable mechanism for discrepancy exposure, eliminating the need for fixed selection thresholds and enabling models to better identify and assimilate knowledge weak-aware zones (KWAZ). Consequently, FedK WAZ supports more effective and fine- mme-grained mutual distillation under heterogeneous federated learning settings.
E,8 Correlation between HAPM Parameters and Client Properties
We investigate how the HAPM mixing parameters correlate with client properties. Eight clients (ID 0-7) are assigned private models in ascending capacity order: 4-layer CNN MobileNet V2 < GoogLeNet < ResNet18 < ResNet34 ResNet50 ResNet101 ResNet152. All clients use the same 4-layer CNN as the proxy model. On the Flowers102 dataset (102 classes), we set the mixing strengths a, 3 E 10.1, 0.5, 1.0), the spatial granularity g E {4, 16, 64), and the update frequency k = 30. Over 1000 rounds, 33 updates yield 8 x 33 = 264 HAPM configurations per client, each comprising (SWAZ. (a", ,g"}}} and (DWAZ: (9t: 9i (8)))))
Because KWAZ-guided mutual learning operates solely on local data, cross-client data heterogeneity has limited influence on local parameter selection; we therefore focus on model architectural differences. Based on the capacity gap between private and proxy models, clients are grouped into Group (small gap, IDs 0-3: private models from 4- CNN to ResNet18) and Group B (large gap, IDs 4-7; private models from ResNet34 to ResNet152).
Mixing strength (ar, 9). Smaller a or 33 induces stronger information mixing, amplifying inter- model discrepancies. Empirically, a", 3 and 33 most frequently take 0.1 (244, 233, and 220 times, respectively), indicating that KDP often prefers smaller mixing strengths to expose cross-model gaps. Smoother settings (0.5 or 1.0) occur 26 times in Group A and 69 times in Group B, showing that
33

---

when private and proxy models differ substantially, even smoother mixing can still form distinct discrepancy regions that benefit transfer and alignment.
Spatial granularity (g). A smaller produces fewer (larger) patches- patches-coarser coarser fusion-whil larger yields finer local mixing. The most frequent choice for g², 9i. and 9 16 (238, 225, and 217 times, respectively), suggesting medium granularity that balances discrepancy revelation and image recognizability. When g = 4, Group A appears 15 times vs. Group 38 times; when = 64, Group A appears 41 times Group B 18 times. Thus, larger structural gaps favor coarser mixing (larger patch areas) to construct discrepancy regions, whereas more similar models benefit from finer mixing explicitly stimulate subtle mismatches.
Overall, these results indicate that FedKWAZ. exhibits structure-aware behavior: HAPM/KDP adap- tively select mixing strengths and granularities according to the degree of architectural heterogeneity across clients.
E.9 Variance and Stability Analysis Dynamically Selected HAPM Parameters
To examine the dynamic behavior and stability of the HAPM parameters, recorded the values of six key parameters (a", 10, analyzed three representative rounds- the 30th 510th, and 990th as summarized in Table
/ 9", Ie (i.9g) 9z) selected by each client at every communication round and
Table 12: Selected HAPM parameters across clients at rounds 30, 510, and 990.
Round
Client ID
a
B
3
g
9i
9
30
0
0.1
0.1
0.1
16
16
16
30
1
0.1
0.1
0.1
16
64
16
30
2
0.1
0.1
0.1
16
16
16
30
3
0.1
0.5
0.1
16
4
30
4
0.1
0.1
0.1
16
16
16
30
5
0.1
0.1
0.5
16
30
6
0.1
1.0
0.1
16
16
16
30
7
0.5
0.1
0.1
510
0
0.1
0.1
0.1
16
16
64
510
1
0.1
0.1
0.1
16
16
16
510
2
0.1
0.1
0.1
16
64
16
510
3
0.1
0.1
0.5
16
16
510
4
0.1
0.1
0.1
16
16
16
510
5
0.1
0.1
0.5
64
16
510
6
1.0
0.1
0.1
16
4
16
510
7
0.1
0.5
0.1
16
990
0
0.1
0.1
0.1
16
16
16
990
1
0.1
0.1
0.1
16
16
16
990
2
0.1
0.1
0.1
16
16
16
990
3
0.1
1.0
0.1
4
16
16
990
4
0.1
0.1
0.1
16
16
16
990
5
0.1
0.1
0.1
16
16
990
6
0.5
0.1
0.1
16
4
16
990
7
0.1
0.1
0.5
16
16
16
Inter-client variance. In the th round, the variance of a" reached 0.089, mainly because Client (ResNet101, significantly more complex than the CNN proxy model) selected a = 1.0. In such cases, clients with higher model capacities tend to prefer smoother mixing strategies to generate samples with stronger transferability. In contrast, in the 30th round, the variance of gî reached 285.75, primarily due to Client (MobileNet_ V2, a lightweight model) selecting şf 64, which considerably higher than others. This observation suggests that when the private model is structurally close to the proxy model, the system adapts by employing finer-grained mixing strategies to better expose cross cross-model discrepancies.
34

---

Intra-client variance. Client 4 consistently selected identical values for all six parameters across rounds 30, 510, and 990, indicating strong internal stability. Similarly, most clients (Clients 0, and 5) maintained stable parameter selections across rounds, demonstrating that the KDP mechanism enforces a high degree of strategic consistency within each client.
Overall, FedKWAZ exhibits low intra-client variance (strong stability within clients) and rela- tively high inter-client variance (clear differentiation across clients). These findings confirm that FedK WAZ possesses structure-aware and adaptive KDP mechanism, capable of dynamically adjusting mixing strategies according to model heterogeneity, thereby enhancing both robustness and transfer effectiveness in federated knowledge distillation.
Visualizations of Data Distributions
Class labels
Class labels
Class labels
8 16
0 7 9 11111177777
13:11 10
Client IDs
Client IDs
Client IDs
(a) CIFAR-10
(b) CIFAR-100
(c) Flowers
Class labels
Class labels
Class labels
R 1011 14
IDs
(d) FMNIST
Skin-Lesions-L
Tiny- ImageNet
Figure 13: Top row (a-c) illustrates the practical non-lID scenario (8 = 0.1) CIFAR-10, CIFAR- 100, and nowerrs12.. Bottom row (d-f) shows the pathological non-lID setting (s = 2/2/200 on FMNIST, Skin-Lesions-L 14, and Inn-lmmgeNeet. In each plot, each circle represents the class-wish data distribution client, and the size of the circle corresponds to the number of samples.
70
Class labels
Class labels
50
40
20
2
Client IDs
Client IDs
(a) Distributions of PACS
(b) Distributions of OfficeHomme
Figure 14: The data distributions of clients on PACS and OfficeHome, where each domain assigned to an individual client. The size of each circle reflects the number of samples.
35

---

Class labels
40
100
Client IDs
100
80
Class labels
60
40
20
0
40
80 Client IDs
120
160
200
Figure 15: Data distributions of clients on Cifar 10 and Cifar100 under practical settings (B 0. The number of clients is 100 for Cifar10 and 200 for Cifar 100, respectively. The size of cach circle indicates the number samples held by each client.
36

---

G
Limitations
During the prediction distillation stage, a fixed temperature hyperparameter 4.0) employed. While the empirical results demonstrate consistent performance across heterogeneous model structures and data distributions, potential remains for optimizing distillation effectiveness by dynamically adjusting the temperature in response to model complexity or input data characteristics, which may further enhance system-level efficiency.
In the default experimental configuration, the feature dimension is uniformly set to K = 512. To assess the impact of dimensionality, additional experiments conducted with K 64, 256, and 1024 on the Cifar100 dataset. As datasets differ in their demands for feature expressiveness, determining feature dimensionality in accordance with data-specific properties warrants further investigation to improve representational capacity and adaptation across tasks.
H Broader Impacts
In the context of heterogeneous federated learning, fine-grained mutual learning framework based on Knowledge Weak-Aware Zones (KWAZ) is proposed, yielding several broader impacts:
Enhanced adaptability of heterogeneous models. By explicitly modeling Semantic Weak-Aware Zones (SWAZ) and Decision Weak-Aware Zones (DWAZ), the proposed framework enables effective crosttacc knowledge transfer. This design improves generalization under dual heterogeneity, supporting deployment in real-world scenarios with varying data distributions and model configura- tions.
Reduced communication overhead and improved privacy. By transmitting only class-level feature prototypes and prediction distributions, rather than full model parameters, the approach substantially reduces communication bandwidth consumption. Furthermore, as neither model structures nor weights are exposed, the system exhibits enhanced resistance to privacy leakage.
Improved system efficiency via faster convergence. The targeted refinement KWAZ-guided distil- lation accelerates alignment in feature and decision spaces, enabling faster model convergence. This yields improved energy efficiency compared to conventional output-aligned distillation, contributing to the sustainability of large-scale distributed learning.
Broadened application potential. The architecture-agnosiic design accommodates diverse client models and devices, promoting practical deployment in heterogeneous environments such as health- care, smart manufacturing, and urban sensing systems. This broadens the real-world applicability of federated learning across domains.
37


Reviews:
[{"review_id": "7hzYDu06ZV", "reviewer": "Reviewer_nXns", "summary": "The paper proposes FedKWAZ, a novel federated learning framework designed to address dual heterogeneity in both non-IID data and diverse client models through key innovations including: Knowledge Weak-Aware Zones (KWAZ) that identify regions of maximal discrepancy between client-private and proxy models by decomposing them into Semantic Weak-Aware Zones (SWAZ) for feature-space misalignment, and Decision Weak-Aware Zones (DWAZ) for output-space misalignment. Hierarchical Adaptive Patch Mixing (HAPM) that generates mixed samples to probe model discrepancies. A Knowledge Discrepancy Perceptron (KDP) that selects samples with largest SWAZ/DWAZ gaps for targeted distillation, and a two-stage training approach where Stage I establishes global class-level alignment using aggregated prototypes while Stage II implements local KWAZ-guided bidirectional distillation. Experimental validation across five datasets (CIFAR-10/100, Flowers102, etc.) demonstrates that FedKWAZ outperforms ten baselines by 2.45–4.66% in accuracy under high heterogeneity conditions, scales effectively to 200 clients, and reduces communication costs by 3–10× compared to proxy-sharing methods.", "questions": "See weaknesses for details.", "limitations": "None.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. KWAZ provides a granular framework to pinpoint and rectify cross-model cognitive gaps, advancing beyond crude output alignment.\n2. Technical Innovation: HAPM and KDP enable dynamic discovery of hard-to-align samples, while the two-stage design decouples global consistency from local refinement.\n3. Communication overhead is slashed by avoiding proxy model transmission, using lightweight prototypes instead.\n4. Convergence proofs (Appendix C) guarantee non-convex optimization at rate O(1/T).\n5. Code/dataset links, hyperparameters, and ablation studies (e.g., SWAZ/DWAZ impact) are thoroughly documented.\n\nWeaknesses\n1. Temperature (τ=4) and feature dimension (K=512) are fixed. Exploring adaptive schemes (e.g., τ tuned per model complexity) could boost performance.\n2. Experiments use CNNs/ViTs but lack extreme architectural gaps (e.g., transformers vs. MLPs). Testing cross-backbone distillation (e.g., CNN→ViT) would strengthen claims.\n3. While upload costs are reduced, the overhead of frequent HAPM/KDP operations (every k=30 rounds) is not quantified.\n4. Privacy/fairness risks of shared prototypes (e.g., membership inference) are noted in Appendix H but lack mitigation strategies.", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: We sincerely thank the reviewer for the insightful suggestion. The idea of dynamically adjusting the distillation temperature ($\\tau$) and the feature alignment dimension ($K$) based on model complexity or data characteristics offers a highly promising direction for performance enhancement.In our current experiments, we adopt a fixed $\\tau = 4$ and $K = 512$ to ensure consistent experimental settings. However, the reviewer’s observation has made us more aware of the potential benefits of adaptive strategies, especially under heterogeneous settings. Recent studies in knowledge distillation, such as [1], [2], and [3], have explored adaptive temperature scheduling and logit scaling, further validating the feasibility and value of the reviewer’s suggestion.\n\nTable 1: Accuracy on CIFAR-100 in Practical Settings with Different Values of $\\tau$.\n\n|               | $\\tau=2$     | $\\tau=3$      | $\\tau=4$      | $\\tau=5$        | $\\tau=6$        \n|---------------|------------|------------|------------|------------|------------|\n| Accuracy(%)     | 50.23 | 50.97 | 51.69 | 51.44 | 51.38 |\n\nTo further investigate, we conducted additional experiments evaluating the effect of different $\\tau$ values in FedKWAZ. As shown in **Table 1**, on the CIFAR-100 dataset, FedKWAZ achieves accuracy of 50.23%, 50.97%, 51.44%, and 51.38% for $\\tau = 2, 3, 5, 6$, respectively, confirming $\\tau = 4$ as the optimal setting. This trend suggests that small $\\tau$ may produce overly sharp distillation signals, while large $\\tau$ may introduce noise, thus weakening the transfer effectiveness. These results highlight the importance of properly tuned temperature in heterogeneous knowledge transfer.\n\nAs a next step, we plan to incorporate learnable or schedule-driven temperature adjustment mechanisms, and explore strategies to dynamically set the feature alignment dimension $K$ based on model capacity or data distribution. This direction will serve as a key extension of our framework. We truly appreciate the reviewer’s constructive feedback.\n\n**W2**: We thank the reviewer for the valuable suggestion and have strengthened our claim by demonstrating the effectiveness of our method under scenarios involving greater architectural heterogeneity. In the $HtM_{10}$ heterogeneous setup, we explore a cross-backbone distillation setting from CNN to ViT. Specifically, in $HtM_{10}$, each client maintains a 4-layer CNN as its proxy model for knowledge exchange, while some clients adopt ViT architectures (including ViT-B/16 and ViT-B/32) as their private models. This setup enables bidirectional knowledge transfer across different backbones, such as CNN-to-ViT and ViT-to-CNN, where our method performs notably well.\n\nTo further assess the robustness of our method under even more extreme architectural disparities, we conducted an additional experiment **where each client uses a multi-layer perceptron (MLP) as its proxy model**. The MLP consists of three hidden layers with dimensions 1000, 500, and 200, followed by a projection layer mapping to a unified 512-dimensional feature space, and a final fully connected classification layer—representing a standard MLP structure. The private client models span a wide range of architectures, including 4-layer CNNs, GoogleNet, MobileNetV2, ResNet variants (18/34/50/101/152), and ViT-B/16 & ViT-B/32.\n\nTable 2: Test accuracy (%) on Cifar100 and Flowers102 datasets using MLP as the proxy model.\n\n| Method    | Cifar100 | Flowers102 |\n|----------|:--------:|:----------:|\n| FedMRL   |  41.98   |   48.22    |\n| FedTGP   |  47.15   |   53.72    |\n| FedKTL   |  47.09   |   52.85    |\n| FedKWAZ  | **51.24** | **57.31** |\n\nThis distillation setting, involving Transformer-to-MLP transfer, presents a significant challenge for cross-backbone knowledge alignment. Experimental results on both CIFAR-100 and Flowers102 datasets demonstrate the strong performance of FedKWAZ, which achieves 51.24% and 57.31% accuracy, respectively. Compared to baseline methods such as FedMRL (41.98%, 48.22%) and FedKTL (47.09%, 52.85%), FedKWAZ delivers substantial improvements. These results validate the robustness of our approach in handling architectural heterogeneity between teacher and student models. We sincerely thank the reviewer for the constructive suggestion, which has enabled us to further strengthen our experimental design.\n\n**W3**: We thank the reviewer for the opportunity to further clarify the computational cost of HAPM and KDP. These modules are not executed in every round but are instead triggered periodically (e.g., every $k = 30$ rounds by default). When a HAPM/KDP cycle is activated, each client performs a lightweight search over a predefined set of candidate parameter configurations to identify the one that maximizes the knowledge discrepancy between the local and proxy models. Crucially, this search involves only forward propagation, without any backward propagation or weight updates, and thus incurs limited additional overhead.\n\nBased on our measurements on CIFAR-100, inserting a KWAZ  search step every 30 rounds introduces an extra cost of $4.75 \\times 10^{4}$ GFlops, which averages to $1.58 \\times 10^{3}$ GFlops per round—only **8.93%** of the total per-round computation. This moderate overhead is a reasonable trade-off given the significant accuracy gains it enables.\n\nMoreover, $k$ (i.e., the search frequency) is a tunable hyperparameter, allowing users to reduce the overhead by increasing $k$ (i.e., triggering HAPM/KDP less frequently), with only marginal impact on performance. In our hyperparameter study, FedKWAZ maintained strong performance even at $k=40$ or $k=50$. For example, at $k=50$, FedKWAZ achieved 51.38% accuracy on CIFAR-100, outperforming the FedTGP baseline by 4.23%. Increasing $k$ to 40 and 50 reduces the amortized per-round KDP-related computation by 25% and 40%, respectively, compared to the default setting of $k=30$. This shows that users can flexibly adjust $k$ according to device capabilities, while still enjoying efficient and effective knowledge transfer.\n\nWe appreciate the reviewer’s insightful question, which allowed us to better articulate the practicality of our method.\n\n**W4**:We sincerely thank the reviewer for highlighting the importance of ensuring data privacy and fairness in federated learning. Our method has been carefully designed with several measures to mitigate privacy risks. Specifically, we only transmit aggregated class-level prototypes and prediction distributions, rather than raw data or complete model weights. This strategy avoids directly exposing client training samples or model parameters and aligns with a number of privacy-conscious FL methods such as [4,5], which also opt to share intermediate statistics rather than full model updates.\n\nRegarding fairness, we deliberately avoid relying on client-reported local dataset sizes for weighted aggregation. Traditional methods like FedAvg often require the number of samples per client to compute weighted averages, which may inadvertently reveal data-related information and grant undue influence to data-rich clients, potentially leading to global knowledge bias. In contrast, our aggregation of prototypes and logits treats each client’s contribution to a class $c$ as equal, without requiring the server to access any information about client data volume:\n\n$$\n\\\\bar{P}\\_c\\^{\\\\mathcal{M}} = \\\\frac{1}{|\\\\mathcal{N}\\_c|} \\\\sum\\_{k \\\\in \\\\mathcal{N}\\_c} P\\_c\\^{\\\\mathcal{M}\\_k}, \n\\\\bar{L}\\_c\\^{\\\\mathcal{M}} = \\\\frac{1}{|\\\\mathcal{N}\\_c|} \\\\sum\\_{k \\\\in \\mathcal{N}\\_c} L\\_c\\^{\\\\mathcal{M}\\_k}\n$$\n\n\nThis design both protects the privacy of data quantity and distribution and avoids the systemic bias introduced by unbalanced data. Furthermore, by not transmitting full model structures or parameters, our approach is inherently more resilient to certain types of privacy attacks compared to conventional FL frameworks.\n\nWe also acknowledge the reviewer’s valid concern that even aggregated class-level prototypes or logits might be subject to potential privacy risks such as membership inference. To address this, we plan to incorporate differential privacy mechanisms in future work (e.g., injecting calibrated noise into the shared prototypes or logits), or explore other perturbation-based strategies to further reduce the risk of information leakage while maintaining distillation effectiveness. We will expand the discussion in the final version of the paper and clearly articulate these planned enhancements.\n\nOnce again, we thank the reviewer for bringing attention to these critical concerns. We remain committed to improving the design of our framework to proactively tackle privacy and fairness challenges in federated settings.\n\n[1] Curriculum Temperature for Knowledge Distillation. AAAI 2023\n\n[2] Logit Standardization in Knowledge Distillation. CVPR 2024\n\n[3] Rethinking the Temperature for Federated Heterogeneous Distillation. ICML 2025\n\n[4] Federated Learning from Pre-Trained Models: A Contrastive Learning Approach. NeurIPS 2022\n\n[5] FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning. AAAI 2024"}, {"review_id": "t76DkPQfUi", "reviewer": "Reviewer_pAYd", "summary": "This paper proposes FedKWAZ, a cognitively-aware mutual learning framework for heterogeneous federated learning (HtFL). The authors identify a key limitation in existing federated mutual learning methods—namely, their reliance on direct output alignment between private and proxy models, which overlooks semantic and decision-level divergences. FedKWAZ introduces Knowledge Weak-Aware Zones (KWAZ), further divided into Semantic Weak-Aware Zones (SWAZ) and Decision Weak-Aware Zones (DWAZ), as the focal points of knowledge misalignment. The method consists of a dual-stage learning framework: (1) global class-level alignment via aggregated prototypes/logits, and (2) local KWAZ-guided distillation using Hierarchical Adaptive Patch Mixup (HAPM) and Knowledge Discrepancy Perceptron (KDP). Extensive experiments across multiple datasets and model heterogeneity levels demonstrate state-of-the-art results.", "questions": "See weakness", "limitations": "The paper includes a dedicated limitations section (Appendix G), which acknowledges the potential cost in client-side computation and limited exploration of dynamic client behavior. These are reasonable and honestly addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper addresses an under-explored but important challenge in federated mutual learning: misalignment between private and proxy models under heterogeneity.\n\n- The concept of KWAZ is novel in the FL context and offers a structured way to locate and exploit knowledge discrepancies.\n\n- The proposed HAPM and KDP modules are well-motivated and enhance the mutual learning process by focusing on semantic/decision-level disagreement zones.\n\n- The experimental section is comprehensive, covering five datasets and three types of heterogeneity (data/model/client), with solid gains over strong baselines like FedTGP and FedKD.\n\n- The paper includes theoretical convergence analysis, good ablation studies, and clear algorithmic structure.\n\n**Weaknesses**:\n\n- The conceptual novelty of KWAZ would be stronger if compared and contrasted more explicitly with existing saliency- or region-aware distillation methods in CV or KD literature.\n- The paper lacks discussion and analysis of scalability to realistic FL scenarios involving thousands of clients or low-resource edge devices.\n- The impact of key hyperparameters (patch granularity \\(g\\), mixing ratios \\(\\alpha, \\beta\\)) in HAPM is not analyzed or visualized.\n- Some figures (especially Figure 1) are visually crowded and hard to parse; a cleaner design could improve clarity.\n- Notations like \\(\\alpha^*, \\beta^*, g^*\\) are introduced abruptly and could be defined more clearly at first appearance.", "was_revised": true, "final_justification": "Thank for authors' rebuttal, which addressed my concerns. I will maintain my positive attitude and score.", "rebuttal": "**W1**: We sincerely thank the reviewer for the valuable suggestion and agree that a more explicit comparison with existing saliency/region-aware distillation methods can further highlight the conceptual novelty of KWAZ.\n\nIn knowledge distillation and computer vision tasks, the core idea of saliency or region-aware distillation [1,2,3] is to focus on the local regions of an image that contribute most to model prediction, thereby improving the efficiency of knowledge transfer. Such methods typically rely on attention maps or saliency responses generated inside the teacher model as guiding cues for student learning. The basic assumption is that the regions attended by the teacher represent the most discriminative semantic information, and aligning features or attention in those regions helps the student model acquire “effective knowledge” under limited capacity. However, these methods are essentially **“one-way guidance” — that is, the teacher statically determines which regions the student should learn from**.\n\nKWAZ fundamentally differs from the above methods in that it does not rely on the saliency judgment of a single model, but instead dynamically identifies the regions of cognitive discrepancy between the private model and the proxy model as alignment targets. This design has three innovations: \n\nFirst, traditional methods assume that the teacher's salient regions are optimal learning targets for the student, while KWAZ identifies regions with the largest representational or decision-level discrepancy between the two models. These regions often reflect knowledge that the student has not yet mastered. Distilling on these regions helps close knowledge gaps while **avoiding the neglect of student-specific errors, thereby making alignment more targeted**.\n\nSecond, KWAZ automatically generates and evaluates mixed samples via HAPM+KDP to locate these discrepancy regions, without requiring saliency networks or hand-crafted rules. This is suitable for real-world federated heterogeneous mutual learning settings. While traditional saliency-based distillation relies on the **fixed authority of the teacher**, KWAZ operates in a **peer mutual-learning context** and focuses on bidirectional cognitive mismatch, making it more adaptable.\n\nFinally, KWAZ conceptually proposes a new perspective: **transforming inter-model cognitive discrepancies into learning opportunities**. Traditional methods focus on the “teacher's optimal regions,” whereas KWAZ focuses on the “most misaligned regions between student and teacher.” This shift in perspective enables more direct identification of performance bottlenecks and improves the quality of knowledge transfer, demonstrating both theoretical uniqueness and practical value.\n\nWe hope the above comparison effectively addresses the reviewer’s concern regarding the novelty of KWAZ.\n\n**W2**: We sincerely thank the reviewer for the valuable suggestion and fully agree that discussing the scalability of FedKWAZ to ultra-large client populations and resource-constrained devices is essential for validating its practical applicability.\n\nTable 1: Test accuracy (%) on Tiny-ImageNet with 1000 clients under low client participation rates (ρ = 0.1).\n\n| Method    | $\\rho=0.1$ |\n|----------|:----------:|\n| FedMRL   |   13.42    |\n| FedTGP   |   10.77    |\n| FedKTL   |   11.25    |\n| FedKWAZ  | **15.74** |\n\nTo evaluate scalability, our main experiments included settings with 50, 100, and 200 clients, verifying the effectiveness of our method at moderate scales. In response to the reviewer’s concern regarding scenarios with thousands of clients, we conducted additional experiments on Tiny-ImageNet with 1000 clients and very low client participation rates (ρ = 0.1). As shown in **Table 1**, under this extremely sparse regime, FedKWAZ achieved 15.74% accuracy, significantly outperforming FedKTL (11.25%) and FedMRL (13.42%) at the low participation rates. These results suggest that **FedKWAZ can generate more transferable input cues via HAPM and effectively reinforce distillation in high-divergence regions**, thereby mitigating performance degradation even in large-scale, low-resource federated environments.\n\nRegarding low-resource devices, our main paper already demonstrates that FedKWAZ can **reduce both communication and computation costs by accelerating convergence**. In each round, only aggregated prototypes and logits are exchanged to align semantics and decisions, leading to low communication overhead—suitable for deployment on limited-resource devices. Furthermore, the HAPM and KDP modules are not triggered every round, but rather at fixed intervals (default $k=30$). When triggered, the client performs the search using only forward propagation over candidate configurations to identify those that maximize the divergence between models, without invoking backward propagation or weight updates, thus significantly reducing extra computation.\n\nOn CIFAR-100, for instance, performing a KWAZ search once every 30 rounds leads to a per-round amortized overhead that accounts for only 8.93% of the local training computation, which is acceptable given the corresponding accuracy gain. Moreover, the trigger interval $k$ is tunable based on device capacity—larger $k$ values (i.e., less frequent searches) can further reduce computational cost, while still maintaining strong performance. For example, with $k=50$, FedKWAZ achieves 51.38% accuracy on CIFAR-100, outperforming the FedTGP baseline by 4.23%. Further optimization can also be achieved by reducing the size of the candidate search space.\n\nWe thank the reviewer again for the detailed suggestion and will further refine the related discussion to strengthen the practical deployability of our method.\n\n**W3**: We thank the reviewer for their attention to the hyperparameter settings of HAPM. In our current design, $g$ controls the patch granularity (i.e., dividing an image into $\\sqrt{g} \\times \\sqrt{g}$ patches), while $\\alpha$ (chosen from $\\\\{0.1,\\ 0.5,\\ 1.0\\\\}$, following $\\lambda^{(i,j)} \\sim \\text{Beta}(\\alpha, \\alpha)$) determines the mixing ratio between two images within each region. A larger $g$ leads to finer perturbations and stronger diversity, whereas a smaller $g$ helps preserve global semantics. A smaller $\\alpha$ emphasizes inter-image differences, facilitating the induction of cognitive discrepancies, while a moderate value better preserves semantic consistency and training stability.\n\n**Our current configuration balances perturbation strength, semantic fidelity, and alignment efficiency**. As detailed in Appendix D.3, we set dataset-specific values of $g$ and $\\alpha$ based on dataset characteristics. In response to the reviewer’s suggestion, **we will add visualizations of samples generated under different $g$ and $\\alpha$ values in the revised version**, clearly illustrating how these hyperparameters modulate HAPM’s behavior and supporting the rationale of our parameter choices.\n\n**W4**: We sincerely thank the reviewer for the valuable suggestions regarding the figure design. We fully agree that the current Figure 1 contains a high density of information, which may hinder intuitive understanding. The reviewer’s suggestions are highly insightful for improving the clarity of the illustration. In the revised version, we will enhance the **color scheme** (to improve contrast between modules), adjust the **layout** (by simplifying arrow paths and emphasizing structural hierarchy), and enlarge the **label fonts** (to improve readability), aiming to present a more intuitive and concise visual representation. We believe these adjustments will substantially improve the reader’s experience and more clearly convey the problem setting and the core mechanism of FedKWAZ.\n\n**W5**: We thank the reviewer for pointing out the unclear definition of the symbols. We noticed that $(\\alpha, g)$ first appears in the Introduction, and we have revised the corresponding description as follows:\n\n**“HAPM generates multiple mixed samples via diverse parameter pairs $(\\alpha_i, g_i)$ (where $i=1, ..., K$), where $\\alpha$ controls the mixing ratio between two images and $g$ specifies patch granularity to divide each image into $\\sqrt{g} \\times \\sqrt{g}$ blocks for local mixing. KDP then evaluates representation and decision discrepancies between private and proxy models across these samples and selects $(\\alpha\\^\\*, g\\^\\*)$ inducing the maximal divergence.”**\n\nThis revision clarifies: (1) that $(\\alpha_i, g_i)$ denotes a set of candidate mixing parameters; (2) the functional roles of each parameter; and (3) that $(\\alpha\\^\\*, g\\^*)$ refers to the optimal pair automatically selected by KDP as the target configuration for subsequent distillation. We thank the reviewer again for highlighting this important detail.\n\n[1] Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. ICLR 2017\n\n[2] Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition. IEEE Transactions on Multimedia 2024\n\n[3] Region-aware knowledge distillation between monocular camera-based 3D object detectors. ICT Express 2025"}, {"review_id": "JE9HEEoYBr", "reviewer": "Reviewer_c2QV", "summary": "This paper proposes FedKWAZ, a new framework for tackling heterogeneous federated learning (HtFL) problems where both data distributions and model architectures vary across clients. The key idea is to explicitly identify and exploit Knowledge Weak-Aware Zones (KWAZ). The authors introduce HAPM (Hierarchical Adaptive Patch Mixing) to generate diverse local perturbations and KDP (Knowledge Discrepancy Perceptron) to quantify feature and decision discrepancies, decomposing them into SWAZ (semantic) and DWAZ (decision) zones. The approach further combines a two-stage learning procedure with global prototype and decision anchor aggregation. Extensive experiments across multiple datasets, non-IID partitions, and heterogeneous model settings show significant improvements over prior SOTA methods.", "questions": "In the appendix, the authors analyze the convergence rate of the proposed algorithm as $O(1/T)$. However, at line 517, it appears that the authors directly assume the gradient norm is bounded by a constant $\\epsilon$. This assumption is rather strong and not commonly justified in distributed training literature. Generally, existing works derive an inequality showing the gradient norm converges to zero as $T$ increases, from which the convergence rate is established. For example, the convergence rate of FedAvg is usually shown to be $O(1/\\sqrt{T})$ [1]. \n\n[1] Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. NIPS 2020.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "##### Strengths\n\n1. The paper directly targets the dual challenge of data and architectural heterogeneity in federated learning, which is highly relevant for practical deployments.\n2. By maintaining private and proxy models on each client and using class-level prototypes and decision anchors for global alignment, the method avoids structural constraints of FedAvg and reduces communication.\n3. Experiments cover diverse datasets (including medical images), multiple non-IID splits, and different architectural combinations, with ablations that isolate the roles of local mutual learning, KWAZ directionality, and SWAZ/DWAZ component, demonstrating the superior performance of the proposed method.\n\n##### Weaknesses\n\n1. Related work discussion is somewhat unbalanced. While the paper organizes related work into *Resource-, Modular-, and Autonomy-heterogeneous methods*, the discussion of Resource and Modular heterogeneity is relatively shallow compared to the extensive treatment of Autonomy-heterogeneous and mutual learning methods. It would strengthen the positioning to analyze existing works from the perspectives of *semantic representation alignment* and *decision distribution alignment*, which directly relate to the core contributions.\n2. Core motivation could be clearer. The main novelty — using HAPM to find maximal representation/decision discrepancies (KWAZ), and then enforcing targeted bidirectional distillation — is sound, but the writing does not highlight this flow sufficiently. As a result, the paper sometimes feels like it lacks a clear driving question, especially in the related work section, which can confuse the reader about how exactly this method addresses gaps in prior approaches.", "was_revised": true, "final_justification": "The authors addressed my question and concern. I'll keep my positive rating.", "rebuttal": "**W1**: We thank the reviewer for suggesting the perspective of analyzing existing works through the lens of semantic representation alignment and decision distribution alignment. In the original submission, we categorized related works into three types: resource heterogeneity, modular heterogeneity, and autonomous heterogeneity, aiming to systematically summarize heterogeneity issues in federated learning and corresponding solutions. However, as the reviewer rightly pointed out, our analysis of the first two categories was relatively brief. We fully agree with this assessment and have accordingly enhanced the revised version by systematically **strengthening the technical analysis of resource- and modular-heterogeneous methods from both the semantic and decision alignment dimensions**.\n\n**W2**: We also thank the reviewer for highlighting the need to present the core motivation and conceptual pipeline more clearly. In the revised version, we have further strengthened the explanation of our methodological innovations to help readers better understand the driving problem addressed in this work and our proposed solution. Specifically, we explicitly formulate the core challenge as **\"semantic–decision dual drift,\"** which characterizes the systematic mismatch between heterogeneous models in both representation and decision space. This challenge is now clearly stated in the introduction as the central motivation of the paper.\nTo address it, we clarify our complete strategy, which involves using HAPM + KDP to identify KWAZ regions and performing targeted bidirectional distillation over these critical areas. With these revisions, the research motivation and conceptual design are now more clearly presented, highlighting both the novel contribution of our method to deep knowledge alignment and its targeted response to limitations of prior work.\n\nBased on the reviewer's insightful suggestions and our analyses in W1 and W2, we have rewritten the second and third paragraphs of the Introduction. Additionally, we have revised the discussions on resource heterogeneity and architectural heterogeneity in the Related Work section, and restructured the concluding paragraph of that section accordingly.\n\n**Introduction**:\n\n(Paragraph 2): To tackle model and data heterogeneity in HtFL, federated mutual learning has been proposed, where clients train performance-driven private models and resource-friendly proxy models in parallel, facilitating knowledge transfer via bidirectional distillation. However, existing methods primarily focus on **direct alignment in the feature and decision space across heterogeneous models, overlooking deeper mismatches in their learned representations and decision behaviors**. As illustrated in Figure 1 (left), for the same input, the private and proxy models on a given client often produce significantly different feature distributions and classification outputs—a phenomenon termed **semantic–decision dual drift**. This dual drift implies that naive output matching yields only coarse and superficial knowledge transfer, leaving critical representation- and decision-level discrepancies unresolved. Without pinpointing and addressing these misaligned “weak zones,” knowledge transfer remains inefficient and ultimately constrains the client’s performance.\n\n(Paragraph 3): Motivated by the necessity to precisely address this semantic–decision dual drift，this paper proposes Knowledge Weak-Aware Zones (KWAZ), characterizing zones where heterogeneous models exhibit the strongest cognitive discrepancies. KWAZ is further decomposed into Semantic Weak-Aware Zones (SWAZ) and Decision Weak-Aware Zones (DWAZ), focusing respectively on representation and decision discrepancies that indicate key zones requiring targeted knowledge enhancement. Building on this insight, the FedKWAZ framework is designed to mine and align complementary knowledge within KWAZ through a two-stage mutual learning scheme integrating local and global collaboration.\n\n**Related Work**:\n\n**Resource-Heterogeneous FL Methods**. A number of approaches handle device heterogeneity by allowing clients to train smaller or pruned versions of a global model that fit their local resource constraints. For example, FedRolex, FjORD,FLASH,FedResCuE and HeteroFL enable each client to work with a sub-model or width-reduced network tailored to its hardware capabilities. These methods **focus on resource adaptation** — e.g., dynamically extracting or training sub-networks from a larger global model — so that both high-end and low-end devices can participate in federated training. However, their emphasis is on resource allocation **rather than on maintaining cross-client knowledge consistency**. As a result, resource-heterogeneous solutions do not explicitly align the semantic representations or decision distributions learned by different client models. The absence of such alignment means that knowledge gaps can persist between clients’ sub-models, **potentially limiting the overall knowledge transfer efficiency across the federation**.\n\n**Modular-Heterogeneous FL Methods**. Another line of work addresses model heterogeneity by splitting models into shared global modules and private local modules. FedRep and FedPer share and aggregates a global feature extractor across clients (aiming for aligned semantic representations), whereas LG-FedAvg and FedGH share a global classifier across clients (aiming for aligned decision output distributions). By partitioning the model, these approaches attempt to transfer knowledge either at the representation level (feature extractor) or at the decision level (classifier) among clients. However, since the **“global” module in such frameworks is obtained by aggregating components trained on each client’s non-iid local data**, forcing all clients to adopt the aggregated module can undermine personalization and lead to suboptimal local performance. In practice, these methods face **a trade-off between achieving global alignment and preserving local adaptation**. Merely sharing or exchanging modules may prove inadequate in addressing fine-grained, data-dependent misalignments in the learned representations or decision boundaries of client models, potentially leaving certain semantic or decision-level discrepancies unresolved.\n\n(The revised final paragraph of Related Work). **Insights**. Driven by the limitations identified above—**especially the neglect of deeper semantic and decision discrepancies between heterogeneous models**—this work introduces FedKWAZ, a cognitively-aware mutual learning framework that explicitly targets the problem of semantic–decision dual drift. It formalizes KWAZ as spatial zones where heterogeneous models display significant knowledge divergence, and further decomposes them into SWAZ and DWAZ, enabling targeted distillation in the semantic and decision discrepancy zones, respectively. In addition, a lightweight class-level global knowledge anchoring mechanism is designed to achieve a new balance among architectural flexibility, communication efficiency, and transfer quality.By directly confronting the **semantic–decision dual drift**, FedKWAZ bridges overlooked knowledge gaps and enables more precise and efficient cross-model collaboration.\n\n**Q1**: We appreciate the reviewer’s attention to the definition of $\\epsilon$ in our convergence analysis and understand the concern regarding the potential ambiguity of “directly assuming the gradient norm is bounded by $\\epsilon$.” To avoid misunderstanding, we would like to further clarify the logic of our analysis and the role of $\\epsilon$.\n\nThe derivation we adopt follows the **“target-driven”** analysis paradigm, which has been widely used in recent heterogeneous federated learning literature. The key idea is not to treat $\\epsilon$ as a priori bound on the gradient norm, but rather to **first specify an acceptable convergence precision target $\\epsilon$, and then derive the communication rounds $T$, learning rate $\\eta$, and other hyperparameter conditions required to reach this target**. This approach has been adopted by several works on heterogeneous learning, such as FedProto [1].\n\nSpecifically, we first establish the theoretical framework in Appendix Section C. Under a set of reasonable assumptions, Lemma 1 and Lemma 2 respectively provide the upper bounds for local training errors in the two stages: global semantic-decision alignment and KWAZ-guided mutual learning. Then, Theorem 1 combines them to derive the overall loss descent in a full communication round. Finally, in Theorem 2, we set an arbitrarily small precision target $\\epsilon$, and derive the constraints on $\\eta$ and $T$ such that the expected average squared gradient norm is below $\\epsilon$, ensuring convergence to the desired accuracy.\n\n**It is important to emphasize that $\\epsilon$ serves as a convergence target, not a prior assumption. It is introduced as a controllable error threshold to characterize the convergence behavior under a finite number of training rounds**.\n\nThis target-driven analysis contrasts with the asymptotic approach, such as in FedAvg, which focuses on whether the gradient norm vanishes as $T \\to \\infty$. In contrast, our approach emphasizes whether there exists a feasible parameter configuration that allows the model to achieve a specified accuracy within limited communication cost, aligning better with the real-world deployment demands of federated learning.\n\nWe fully agree with the reviewer on the importance of asymptotic analysis and plan to explore incorporating the two-stage non-convex optimization process of FedKWAZ into such a framework in future work. In the current revision, we will further clarify the role of $\\epsilon$ and supplement our explanation by showing its consistency with the derivation paradigms used in works like FedProto.\n\n[1]FedProto: Federated Prototype Learning across Heterogeneous Clients. AAAI 2022"}, {"review_id": "Hrsvok8cya", "reviewer": "Reviewer_GNua", "summary": "This paper introduces FedKWAZ, a novel mutual learning framework designed to overcome the shallow knowledge transfer common in heterogeneous federated learning (HtFL). The core contribution is the concept of \"Knowledge Weak-Aware Zones\" (KWAZ)—input regions where heterogeneous models disagree most, which are decomposed into Semantic (SWAZ) and Decision (DWAZ) zones to address discrepancies at both the representation and prediction levels. Using a Hierarchical Adaptive Patch Mixing (HAPM) module to create challenging samples and a Knowledge Discrepancy Perceptron (KDP) to dynamically identify these zones, FedKWAZ performs a targeted, KWAZ-guided local distillation. This is integrated into a two-stage process that starts with a communication-efficient global alignment using aggregated class-level anchors, and extensive experiments show that this approach significantly outperforms state-of-the-art HtFL methods.", "questions": "- Local Computational Cost: Could the authors provide a more detailed analysis of the local computational overhead introduced by the KDP search mechanism in Stage II? Specifically, is the search for the optimal pair exhaustive over the predefined sets A and G for every local training batch? A comparison of the wall-clock training time or FLOPS per local round against key baselines like FedTGP or FML would be very insightful to better understand the practical trade-offs between communication and computation.\n\n- Can you show the actual value $\\alpha$ that is selected per update round? I wonder if $\\alpha$ is too high, the perturbed images do not represent class semantics but rather represent smoothed or generalized feature representation of the specific models.\n\n- Regarding the distillation temperature (τ=4), this value appears relatively high compared to what is often used in standard knowledge distillation. Could the authors clarify the rationale behind this choice? It would be helpful to understand if this is a common practice for model-heterogeneous FL, or if there's a specific aspect of the FedKWAZ mechanism that benefits from a more pronounced softening of the probability distributions.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- Focusing on discrepancy-focused alignment is technically solid and sound.\n\n- The decomposition of KWAZ into semantic (SWAZ) and decision (DWAZ) components is logical, allowing for a fine-grained approach to correcting model mismatches at different levels of abstraction.\n\n- The paper's experimental validation is thorough and rigorous. The evaluation spans multiple datasets (Cifar10/100, Flowers102, Skin-Lesions, Tiny-ImageNet) and also includes challenging domain-shift scenarios (PACS, OfficeHome). The methodical testing under various degrees of data (pathological, Dirichlet) and model (HtFE/HtC series) heterogeneity provides compelling evidence for the method's robustness and superiority. \n\n- This paper provides a theoretical guarantee for the convergence of the proposed algorithm\n\n\n### Weaknesses\n\n- Clarity: The paper lacks comprehensive observations, motivations, and justification for the design choices of the proposed methods. \nSpecifically, 1) while the motivation to align the feature representations between the proxy model and the private model, unlike previous algorithms that focus on aligning only the logits, is clear, it lacks observations or comprehensive analysis of why the logit alignment in mutual learning is not enough for model heterogeneous FL. The lack of such analysis causes a lack of motivation for using HAPM, which uses a form of mixed, perturbed synthetic images. 2) Authors should discuss the design choice of using 2-stage local training, not using all the proposed regularization losses jointly. 3) The paper lacks the justification or motivation for making smoothed, perturbed synthetic images, not just selecting local hard images that incur a large information mismatch between the proxy model and the private model. The paper also lacks an extensive analysis of how HAPM resolves the information misalignment. Since there would be a trade-off because this hard regularization can prevent effective local optimization of the private model, which has a more complex structure, aligning feature representations between the proxy model and the private model does not always improve the performance. Therefore more detailed and comprehensive discussion of the proposed method will improve the quality of the paper.\n\n\n- Ablation study of more local epochs and lower participation rates would validate the robustness of the proposed methods to the data and model heterogeneity in FL.\n\n- Ablation study of using mutual learning losses without HAPM (i.e., using naive local images or selecting hard examples) will validate the effectiveness of using HAPM.\n\nIf the authors address by concerns, particularly regarding the clarity, I would be inclined to increase my rating.", "was_revised": true, "final_justification": "The authors addressed my concerns thoroughly with extensive empirical evidence.", "rebuttal": "**W1**: We sincerely thank the reviewers for the suggestions on writing clarity.\n\n**On the limitation of “logits-only” alignment in existing methods**: the proxy and private models often differ in both their feature extractors and classifiers. Such heterogeneity leads not only to decision divergence at the output (logits) level, but also to semantic mismatch in the hidden feature space. As shown in Fig. 1, even with the same input, their embeddings differ significantly due to varied capacity in feature modeling. When passed through their own classifiers, this difference further **amplifies into prediction gaps**. Thus, methods that align only logits (e.g., FML) are difficult to establish a stable **feature-decision mapping** relationship due to the **accumulation of biases during the semantic representation stage**, thus limiting the effectiveness and consistency of knowledge transfer between models.\n\nIn our results, FedKD—which aligns both features and logits—usually outperforms FML on complex datasets, showing the importance of feature alignment in addressing model-structure gaps.\n\nHowever, these methods still align features or logits directly on raw images, without identifying the key areas that drive cross-model gaps—resulting in shallow alignment. FedKWAZ tackles this by modeling the **“semantic–decision dual shift.”** It uses HAPM to generate local, perturbed samples with diverse semantics and KDP to locate KWAZ. Dual-path distillation is then applied on SWAZ and DWAZ to enable adaptive and targeted alignment, effectively going beyond prior FL distillation schemes.\n\n**Design Choice of the Two-Stage Local Training Scheme**: The decision to employ a two-stage local training strategy stems from the need to **balance global consistency and local discrepancy alignment effectively**.\n\n    Stage I-Global Anchoring: During this stage, the proxy and private models align each other's global class prototypes and logits after completing one round of training. This coarse-grained alignment helps establish a shared semantic space and basic decision boundaries, ensuring a stable foundation for subsequent alignment. \n\n    Stage II-KWAZ-Guided Distillation: Building on the preliminary alignment from Stage I, the second stage shifts its focus to learning the discrepancies within KWAZ. These regions highlight the most significant areas where the models differ. The goal in this stage is to address fine-grained differences in both semantic and decision-level spaces that were not fully resolved in the first stage.\n\nThis design is motivated by the observation that combining all regularization losses simultaneously would **force the models to balance both global and local knowledge alignment at once**, risking model confusion and suboptimal learning. The two-stage approach ensures that each phase targets alignment at the appropriate granularity, optimizing the training process.\n\n**Regarding the use of HAPM-generated synthetic samples instead of directly selecting local hard samples**: Our adoption of the HAPM-based synthesis strategy is motivated by two main considerations.\n\nFirst, relying solely on local hard samples—i.e., those with large prediction gaps—cannot fully capture the diverse knowledge gap between private and proxy models. These samples are often few and **biased by the local data, making it hard to reveal broad differences in representations**. In contrast, HAPM introduces block-wise spatial mixing and local perturbation to produce diverse samples with mixed semantics. This leads to more informative contrastive signals and better exposure of representation or decision gaps between models.\n\nSecond, the locally perturbed samples from HAPM bring mild **adversarial effects**, which help improve model robustness and generalization, while reducing overfitting risks from a few extreme samples.\n\nTo address the reviewer’s concern about lacking analysis of how HAPM mitigates information mismatch, we clarify: the key role of HAPM is to break the agreement between proxy and private models on original samples by injecting localized noise and semantic mixing. This magnifies their representation or prediction differences, which are then modeled through contrastive loss. As a result, model alignment becomes more targeted and effective, systematically reducing information inconsistency caused by misaligned knowledge. Put differently, **HAPM is not only a generator, but also a divergence amplifier that enhances KDP’s ability to detect and correct gaps**.\n\nWe also acknowledge the reviewer’s concern that overly strong alignment may harm local optima. To mitigate this, our design does not apply KWAZ-based distillation at every round. Instead, KDP is triggered periodically to dynamically locate areas of divergence, striking **a balance between consolidating existing knowledge and correcting emerging discrepancies**. In addition, **the mixing strength and patch granularity in HAPM are carefully controlled** to preserve semantic consistency and ensure the generated samples remain task-relevant. We also reorganized the introduction and related work to better present comprehensive observations, motivations, and justification for our design choices, as suggested by Reviewer c2QV.\n\n**W2**: We appreciate the reviewer’s suggestion regarding the choice of local training epochs and client participation rates. These factors indeed intensify bias and instability in federated learning, and we find the feedback highly valuable. \n\nTable 2: Test accuracy (%) on Cifar100 in the practical setting with large E.\n\n| Method    | E=5 | E=10 | E=20 \n|----------|:--------:|:----------:|:----------:\n| FML   |  39.33   |   38.25    |   35.87\n| FedKD   |  41.28   |   40.25    |   38.75\n| FedMRL   |  42.41   |   42.97    |   43.65\n| FedTGP   |  46.82   |   46.36    |   47.03    \n| FedKTL   |  46.87   |   46.62    |   45.98    \n| FedKWAZ  | **51.76** | **51.97** |   **52.13**    \n\nAs shown in **Table 1**, we extended the number of local training epochs to 5, 10, and 20. Benefiting from the local **perturbation signals provided by HAPM and the mutual enhancement enabled by KWAZ**, FedKWAZ consistently improves as the number of local steps increases. Under low participation scenarios, FedKWAZ also maintains strong performance. \n\nTable 2: Test accuracy (%) on Tiny-ImageNet with 1000 clients under low client participation rates (ρ = 0.05).\n\n| Method    | $\\rho=0.05$ |\n|----------|:----------:|\n| FedMRL   |   12.87    |\n| FedTGP   |   9.86    |\n| FedKTL   |   10.73    |\n| FedKWAZ  | **14.92** |\n\n Furthermore, as shown in **Table 2**, we scaled the evaluation to 1000 clients on Tiny-ImageNet with only 5% participation, a setting characterized by extreme data sparsity. FedKWAZ achieved 14.92% accuracy under this condition, significantly outperforming FedKTL (10.73%) and FedMRL (13.42%). These results collectively demonstrate the robustness of FedKWAZ in handling both data and model heterogeneity. We will incorporate the full results and discussion into the revised manuscript.\n\n**W3**: We thank the reviewer for highlighting the need to validate the effect of HAPM. FedKD, which aligns features and logits using only original images, serves as a baseline. It underperforms FedKWAZ on multiple datasets, **showing that lacking a discrepancy-aware design limits its distillation power**.\n\nWe further compared with a \"local hard sample\" strategy: selecting the top 10%, 30%, and 50% samples by logits gap. This method shows clear limits, as it uses few samples, is sensitive to bias, and tends to focus on narrow semantic areas. **It fails to reflect full model differences and may lead to overfitting**.\n\nBy contrast, HAPM creates diverse samples via spatial perturbation and layered mixing. KDP then finds regions of high model disagreement. This setup better exposes semantic–decision shifts and allows for more adaptive and fine-grained distillation.\n\n**Q1**: We thank the reviewer for asking about the cost of HAPM and KDP. These steps are not done every round, but once per fixed interval (30 rounds by default). Each client runs forward propagation over a few mixing settings to find the one that maximizes model divergence. No backward propagation or weight updates are used, so the cost stays low.\n\nOn CIFAR-100, KWAZ search adds only $1.58 \\times 10^{3}$ GFlops per round on average, which is **8.93%** of total cost per round. Although FedKWAZ uses more GFlops per round ($1.77 \\times 10^{4}$) than FML ($1.01 \\times 10^{4}$) and FedTGP ($1.19 \\times 10^{4}$), it trains **faster and reduces total compute and communication**. It also boosts accuracy by 12.22% over FML and 4.54% over FedTGP. We believe this trade-off is both fair and worthwhile.\n\n**Q2**: We thank the reviewer for pointing out the setting of the mixing factor $\\alpha$. To balance semantic fidelity and perturbation strength, we restrict $\\alpha$ to ${0.1, 0.5, 1.0}$. Lower values (e.g., 0.1, 0.5) yield stronger contrast in mixed areas, which helps expose local model differences. A larger $\\alpha=1.0$ gives smoother blends that improve generalization.\n\nWe observe that on complex datasets like CIFAR-100 and Tiny-ImageNet, KDP tends to pick smaller $\\alpha$, showing that it prefers preserving semantic structure. Also, since HAPM mixes locally in image space, **even a large $\\alpha$ only affects part of the input, not the global view**. This ensures a good trade-off between diversity and semantic retention.\n\n**Q3**: We also thank the reviewer for highlighting the choice of distillation temperature $\\tau=4$. In heterogeneous FL, logits across models often vary sharply. In FedKWAZ, we target KWAZ areas with strong model disagreement, where predictions are more spread. If $\\tau$ is too low, alignment may bias toward the dominant class and ignore minor classes. **A higher $\\tau=4$ smooths the output and better balances all class signals, supporting fine-grained transfer**. This setting has shown consistent benefits in our experiments."}]

Author rebuttal:

Meta-review: This paper proposes a novel framework for heterogeneous federated learning. It integrates four key mechanisms into a two-stage mutual learning framework to address both data and architectural heterogeneity.  This paper received four borderline acceptances, and most concerns about experiment details were adequately addressed during the rebuttal. Therefore, it is recommended for acceptance.
Decision: Accept (poster)