FedDAG: Clustered Federated Learning via Global Data and Gradient Integration for Heterogeneous Environments
Authors: Anik Pramanik
Murat Kantarcioglu
Vincent Oria
Shantanu Sharma
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Clustering
Distributed Machine Learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We introduce FEDDAG, a clustered FL approach that tackles data heterogeneity by combining data and gradient similarity for improved client clustering, and employs a dual-encoder architecture to enable representation sharing across clusters.

Abstract: Federated Learning (FL) enables a group of clients to collaboratively train a model without sharing individual data, but its performance drops when client data are heterogeneous. Clustered FL tackles this by grouping similar clients. However, existing clustered FL approaches rely solely on either data similarity or gradient similarity; however, this results in an incomplete assessment of client similarities. Prior clustered FL approaches also restrict knowledge and representation sharing to clients within the same cluster. This prevents cluster models from benefiting from the diverse client population across clusters.
To address these limitations, FEDDAG introduces a clustered FL framework, FEDDAG, that employs a weighted, class-wise similarity metric that integrates both data and gradient information, providing a more holistic measure of similarity during clustering. In addition, FEDDAG adopts a dual-encoder architecture for cluster models, comprising a primary encoder trained on its own clients' data and a secondary encoder refined using gradients from complementary clusters. This enables cross-cluster feature transfer while preserving cluster-specific specialization.
Experiments on diverse benchmarks and data heterogeneity settings show that FEDDAG consistently outperforms state-of-the-art clustered FL baselines in accuracy.

Published as a conference paper at ICLR 2026
FEDDAG: CLUSTERED FEDERATED LEARNING VIA GLOBAL DATA AND GRADIENT INTEGRATION FOR HETEROGENEOUS ENVIRONMENTS
Anik Pramanik', Murat Kantarcioglu", Vincent Oria', Shantanu Sharma' New Jersey Institute of Technology, USA Virginia Tech, USA
ABSTRACT
Federated Learning (FL) enables a group of clients to collaboratively train model without sharing individual data, but its performance drops when client data are heterogencous. Clustered FL tackles this by grouping similar clients. However, existing clustered FL. approaches rely solely on either data similarity or gradient similarity; however, this results in an incomplete assessment of client similari- ties. Prior clustered FL approaches also restrict knowledge and representation sharing to clients within the same cluster. This prevents cluster models from ben- efiting from the diverse client population across clusters. To address these limi- tations, FEDDAG introduces a clustered FL framework, FEDDAG, that employs a weighted, class-wise similarity metric that integrates both data and gradient in- formation, providing a more holistic measure of similarity during clustering. In addition, FEDDAG adopts dual-encoder architecture for cluster models, com- prising primary encoder trained on its own clients' data and secondary encoder refined using gradients from complementary clusters. This enables cross-cluster feature transfer while preserving cluster-specific specialization. Experiments diverse benchmarks and data heterogeneity settings show that FEDDAG consis- tently outperforms state-of-the-art clustered FL baselines in accuracy.
INTRODUCTION
Federated Learning (FL.) enables users/clients to collahoratively train a model on their data with- out sharing it with other clients or a central entity (McMahan et al. 2017). However, diversity in user behavior results in heterogencous data distributions, known as non-identically independently distributed (non-IID) data, across clients. This heterogeneity can lead to slower convergence and suboptimal accuracy of the global model (Kairouz et al., 2021). More specifically, non-lID data arise due to various factors, including class/label skew, feature skew, quantity shift, concept shift, and concept drift - common types of data heterogeneity. Class/labe! skew refers to the non- identical distribution of labels/classes different clients, e.g., the absence of label one client while the same label is present at other clients (Zhang et al., (022a)). Feature skew occurs when distributions vary due different personalization nuances, e.g., an alphabet letter can be written in different ways (Li et al.] 20211). Quantity shift happens when different clients have different amounts of data (Wang et al. (021). e.g., an online retailer with millions transaction records is compared to local store with only few hundred records. Concept shift happens when different clients assign the same label to fundamentally different data samples due to variations local data distributions or labeling criteria (Kang et al. 2024).
Clustered FL handles non-IID data effectively, especially when distinct groups of clients display substantial variations in their data distributions (Ghosh al. 2020 Guo al.) 2024 Vahidian al. 2023). In clustered FL, clients are grouped into clusters based on their similarities their data distributions, and each cluster trains its model tailored to its specific data. However, despite their advantages, existing clustered FL approaches suffer from the following limitations:
1. Improper Similarity Method. Cluster FL approaches use either data gradient alone to com- pute similarity for clustering. Cluster FL approaches (Sattler et al.) 2020; Long et al., 2023 Ghosh et al. 2020) that use gradients or loss values cluster clients can group clients incorrectly due the

---

Published as a conference paper at ICLR 2026
high dimensionality of data the presence of various skews in client data (Vahidian et al. 2023 Other drawbacks of these approaches include: requiring each client to evaluate multiple global mod- els every round (Ghosh et al. 2020 Licciardi et al. 2025), delaying cluster formation until many training iterations, and requiring clients to upload full model updates (Sattler et al. 2020).
On the other hand, the data-based approach, such PACFL (Vahidian et al., 2023), only considers label skew and does not account for skew issues like concept shift. Moreover, PACFL defines inter- client similarity as the minimum cosine angle between the clients' feature subspaces. However, by relying on the smallest angle across the subspaces, PACFL may yield high similarity even when only a small portion of the clients' data is similar. while the remaining subspaces are vastly dissimilar.
2. Global Representation Sharing. Existing Clustered FL approaches restrict knowledge sharing to clients within the same cluster. This prohibits clients across clusters to benefit from low-level latent representations. One way FedSoft (Ruan & Joe- Wong 2022) and FedRC (Guo et al., 2024) address this issue by incorporating multiple cluster models through soft clustering with learnable cluster importance weights. However, in these approaches, client's model becomes a noisy blend of several cluster models. While this blending may occasionally benefit data that aligns with sev- eral clusters, the added noise from unrelated clusters may degrade the performance on the client's s primary dataset, since the model is no longer explicitly optimized for its own data.
3. Limited Consideration of Distribution Skews. Clustered FL techniques (Sattler et al. 2020 Ghosh et al. 20200 Vahidian et al. 2023) Licciardi et al.) (025)) primarily address label skew. How- ever, these approaches do not account for concept shift or quantity shift.
4. Predefined Cluster Numbers. Existing clustered FL approaches lack adaptive mechanisms for automatically adjusting the number of clusters. For example. IFCA (Ghosh et al., 2020) requires the optimal number of clusters to be specified in advance. [Sattler et al. (2020) adopts a recursive strategy to split clusters when gradients converge to stationary point but cannot merge clusters when needed, such as upon the arrival of new clients. [Zeng et al. (2023) supports merging clusters but not splitting them. Li al. (2024a) evaluates candidate clustering using traditional clustering metries that do not account for the unique characteristics of FL setting. These limitations raise the following crucial question:
How can we overcome the above challenges posed by various skews in heterogeneous data distributions by utilizing both data and gradient information to dynamically cluster clients and enabling representation sharing among clusters in FL?
Our contribution. This work proposes novel algorithm, entitled clustered Federated Learning via global DatA and Gradient integration (FEDDAG). FEDDAG introduces a novel method to compute similarities among clients and an innovative approach that combines data and gradient information for improved client grouping. To combine data- and gradient-based similarity to achieve a more accurate similarity matrix, FEDDAG assigns each client weight that indicates how much emphasis to place on data versus gradient information. FEDDAG optimizes these weights using an entropy- based loss that sharpens the final adjacency matrix. To further improve client similarity estimation, FEDDAG extends the data-based approach PACFL (Vahidian et al.] 2023) by performing class-wise comparisons rather than comparing entire data subspaces restricting comparisons to subspaces corresponding to the same class across clients. This approach yields a more accurate similarity metric and naturally accounts for concept shift. In addition, FEDDAG assigns weights to the class- wise similarity values to address quantity shift. FEDDAG also improves upon the existing gradient- based similarity so that client computes gradients for at most one model per round and transmits only a compressed gradient.
These above mechanisms improve similarity computation and lead to better client clustering. We further enhance FEDDAG by employing dual-encoder architecture to enable effective representa- tion sharing across clusters. During the training phase, each cluster model consists of: (i) primary encoder, optimized using the cluster's own client data, and (īi secondary encoder, designed to learn complementary features from other clusters. The outputs of the two encoders are concatenated along the feature dimension, and classifier is trained the combined representation. This design facilitates cross-cluster knowledge transfer while preserving cluster-specific specialization.
Compared to prior works, of data heterogeneity: label skew, feature skew, concept shift, and quantity shift. FEDDAG accounts for concept shift by performing class-wise comparisons when computing similarity between clients'
knowledge, FEDDAG is the only work that addresses all four types
2

---

Published as a conference paper at ICLR 2026
data. Additionally, FEDDAG introduces adaptive clustering mechanism that automatically deter- mines the optimal number of clusters through a novel evaluation metric. Specifically. it generates a range of candidate clusterings using hierarchical clustering (HC) (Day & Edelsbrunner 1984) and evaluates them with a novel federated-aware metric that rewards compact cluster formation while penalizing over-spliting I
In summary, the contributions of this paper are follows:
1. A new clustered FL algorithm, FEDDAG, that combines both data and gradient similarity for bet- ter client clustering and improves data similarity estimation with class-wise weighted method.
2. FEDDAG introduces a novel method for knowledge and representation sharing across clusters by employing dual-encoder architecture.
3. This work introduces a novel federated-aware metric to evaluate candidate clusterings and auto- matically determine the optimal number of clusters.
4. We evaluate FEDDAG under non-Ill) data, having class skew, feature skew, concept shift, and quantity shift, and across different degrees of heterogeneity (e.g., high vs. low). Table [ the accuracy of FEDDAG in comparison to existing clustered FL methods. Detailed experimen- tal results are provided in $
The full version of the paper and code is available at https://tinyurl.co http:///inyyrl.mmmm///bbbbbb2b
2 LITERATURE REVIEW
Table 1: Accuracy (%) of FEDDAG vs. cluster-
ing baselines under non-IID label skew (20%) and quantity shift (Dirichlet o' 11=
There exists an extensive body of work on improving the performance of FL in data- heterogeneous environments via clustered FL., knowledge distillation, meta-learning, data augmentation, and related techniques. Below, we summarize the approaches most relevant to our work; additional related directions are dis- cussed in Appendix A.I
Algorithm
Technique
CIFAR-10
FMNIST
PACEL
Data (D)
90.45=0.30
04.41=0.31
CFL
Gradient (G)
72.00=0.66
0.
TFCA
Gradient (G)
89.68=0.17
94.03
FEDDAG
D + G Global Feature
94.53=0.12
0.82
(Ours)
Sharing
Clustered FL techniques address distribution shift by grouping clients based on their data distribu- tions. PACFL (Vahidian al. 2023) clusters clients by analyzing principal angles between client data subspaces, but it ignores label information, making it prone to incorrect clustering under con- cept shift. Ding & Wang (2022) constructs K shared models based on each client's dataset contribu- tion. Another line of work (Ghosh et al., 2020 Licciardi et al., (205)) uses loss values on gradients iteratively cluster clients each training round. Other methods group clients via gradient or param- eter similarity (Sattler et al. |020 Zhang et al.
2024). while soft clustering enables clients to join multiple clusters (Ruan & Joe- Wong [022]] Guo et al. [024). Additional methods, such as Long et al. y2033;; Marroq et al. (2021); Wu et al. (2023), rely on maximizing log-likelihood functions or modeling joint distributions. Compared to these methods, FEDDAG combines data and gradient information for better clustering and enables knowledge sharing across clusters.
Knowledge distillation (KD) approaches such as Lin et al. (2020) Li & Wang 2019) use a global dataset to transfer knowledge from local teacher models to global student model. FedFTG (Zhang ct al. 2022b) trains a generator to approximate the input space of local models and uses it to generate patudo-data. Another line of work, data-free KD, generates pseudo-data directly from pretrained teacher model to perform knowledge distillation (Guo et al.; 2023] Chen et al.] 2019). DeepImpres- sion (Nayak et al. (2019) recovers approximate real data by modeling the output space of the teacher model, while DeepInversion (Yin et al.] (020) further refines pseudo-data by regularizing the dis- tribution of intermediate feature maps. Instead of relying on a public/pseudo dataset our proposed FEDDAG's global parameters are updated directly using data from complementary source clusters, enabling cross-cluster knowledge sharing.
FEDDAG ALGORITHM
FEDDAG, framework for clustered FL, can be formulated empirical risk minimization (ERM) problem over N clients, each holding a local dataset D,=(X,, Y.), where X, and Y, de-
Over spliting is a common issue in ICC for FL that can violate key principles of FL by producing degenerate clusters with very few clients Lccaaadi et 2025).
3

---

Published as a conference paper at ICLR 2026
note the input samples and labels, respectively. The data can be non-iid and may exhibit various skews (as discussed in ş7). The server partitions the clients into Z clusters Ci...., Cz. The ob- jective is to minimize the local loss C(Y,, Fa(o) (X,)) for each client ie where a(t) is the cluster assignment determined by FEDDAG. Simplified FEDDAG cluster-level model is defined as:
(-) = e[); (9)
(I)
Here, @ is the feature encoder and d is the classifier head. FEDDAG also supports more expressive dual-encoder architecture, where the outputs of two encoders are jointly processed by the classifier
head, represented below:
(:: F2(-) = (( ( (1) (;:):; 0%)
We describe FEDDAG (see Algorithm ! in two parts. First, we introduce the weighted class- wise approach (Algorithm 2 in Appendix A.3 for computing data similarity among clients and combine both data and gradient to improve clustering (A (Algorithm in Appendix (A.3). The improved clustering can be directly used for traditional clustered FL, resulting in higher accuracy (see $ We then further enhance FEDDAG with dual-encoder mechanism (described in $4) that enables inter- cluster representation sharing during FL training. which further increases FEDDAG's performance. An illustration of FEDDAG is shown in Figure 1 and its components are described below.
3.1 GRADIENT-BASEI SIMILARITY
Cheriaad
Survar-adee
Adgannniyy
Cradion
High-level idea. FEDDAG introduces a lightweight method for computing gradient similarity. Prior approaches such as |Satteer et al. (2020) and Kim et al. (2024b) period- ically send gradient updates to the server to measure client similarity. In contrast, our ap- proach has each client first train locally on its own data (without federation) for few rounds to partially converge the gradients. We observed that two such rounds (10 lo- cal steps each) are sufficient to achieve par- tial convergence, making inter-client similar- ity more distinguishable (see experiments on Local Steps (ty) in Appendix {B.2). To fur- ther reduce communication, FEDDAG trans- mits a h-sparse version of the gradients (re- taining only k coordinates) to the server for similarity computation (Wangni et al. 2018).
Cllnts
Dira
Clurter
Corphranhely
Clira Curniing
Triining
Claker Secondary Eacoter Traiiing
Churert
Sacenbbry
Server
Staling
Ulitee SPerery breed Inrey
Piivary
903
Lipttaee
Figure Overview of FEDDAG. Clients compute prin- cipal vectors and gradients to build an adjacency matrix and a graph indicating which clusters can supply fea- tures for cross-cluster sharing. Training proceeds in two phases: (1) the primary encoder and classifier are trained on each cluster's local data; (2) the secondary encoder of a requesting cluster is trained on source cluster's data
Details of the method. Each client: E N is initialized with random parameters 0 and performs local training (without federation) on D; for = 2 rounds (see Appendix (B.2) to obtain a gradient update A The update is k-sparsified retaiini only a small random subset of entries (typically 1-2%) (Wangni et al. 2018). The sparsified update A is then sent to the server, which constructs a pairwise similarity matrix. The similarity Gi.j between clients and j is computed as:
(Ā. )')
180
GL cos
E.
(3)
A |
3.2 WEIGHTED CISE DATA- BASSED SIMILARITY
High-level idea. Our goal is to construct a data-based similarity matrix that will be fused with the gradient matrix for clustering. Unlike the existing data-based approach, PACFL (Vahidian et al., 2023). which compares the entire data subspaces of two clients, we measure similarity in a class- wise manner and assign weights to the class-level similarities compute the final client similarity.
Details of the method. Let C be the total number of classes, and Da. the data of client i E N for class E C. Each client applies truncated SVD (Klema & Laub] 1980) on the transpose of Dr.e to compute p principal vectors per class, denoted U. [U,,..., Up]. These vectors are then sent to
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: FEDDAG Algorithm
Input:
Number of clients sampling rale E (0, 1]. classes. Initialize client E N with random !??
Oulput: Updated global model parameters
for each round
do +- max(R N, Sm ( Am // Sampling rate for each eltent $
of sampled elients parallel do
if S. ty then
Local training 6., client local leederatio))
if
then
Client i sends sparsified local model update to server
10
Client i performs SVD and extracts principal vecturs U. Ve E C and sends to server
Server forms
Proximityye www.iiittattiiii............mmmt (0" " Å*) (Algorithm)) // Adjacency matrix
Server compates optimal Clustering (Ca..... ..... Cz) <- Opt i ma www.eeaeeeggg.naerreo
(Algorithm B Find beat clusterin
13
Server computes the CC-Graph H as per Bo. 12
14
Server initiates @ as in Eq. 22 and 62/ and Ø randomly // cluster ercoder initial ization
15
else
sends {el) 2
2(33 s(e)- (ics) and 2 -)
0f client
17
Client sets (8 05 (61 @ and trains them via SGD Bq. 17 primary training phase
Client 025 22 updates SGD in Eq secondary training phase
19
Client broadcasts (0, (0)) and 025" to server
20
2 fy for each clister & Z do
21
22
Update 6 and @ in Eq.
18
Update learner cluster 2
-
the server to compute the data similarity matrix For each class c, the computes the principal angle (Jain et al. 2013) between U and U3. indicating the similarity between clients and j as:
V' Liee
min xEU
cos
||v||
[|x[|
Vi,j E.
(4)
class e is present in only one of the clients, V" v'..j. 90º; if in neither, Vi..cc 09. Next, the server assigns weights Wij.c to each class-wise similarity V Vis..c to reflect class frequency dif- ferences (i.e., quantity skew) between clients and j. This weighting scheme ensures that larger differences in class frequency lead to higher dissimilarity values. The weights computed
max(In() Di.c + <), In) +))
Wase
min(In(|D..) e), In (D,l +))
then min-mat max normalized bounded range 61 1+8), where
controls the server's toler-
ance to frequency imbalance. The final similarity between clients i and is:
V Σ Vise 1...0 W Wester Liie:
W 1,ee normalized Waj.c-
3.3 COMBINING DATA & GRADIENT ALGORITHM
High-level idea. After constructing the data and gradient similarity matrices, FEDDAG applies min-max normalization and then combines them into a single proximity/ajjacency matrix.
Details of the method. Given the normalized Vi.y and G.,,, FEDDAG learns a weight vector w (Wi...., w [0, 1I where each w, is assigned to client to control the relative importance
of gradient versus data similarity. FEDDAG then fuses the normalized matrices to construct the proximity matrix as follows:
Aij = wi G..5 + (1 (1-w;)De Di ISi<iSN, Ajú = Ai.
FRDDAG optimizes w by minimizing the entropy loss:
N
Ans
Cen
Ans log Ar, Aog
ΣN A
11 FEDDAG, clients small principal vectors Irequency information principal vectors not actual client data, but a linear combination of them. Moreuver, the number of principul vecturs shared with the server less
server compute similarity.
per client. This approach aligns with peior works, such Vahidiaa

---

Published as a conference paper at ICLR 2026
where A., is the row-wise softmax normalization of A.,- In Eq the loss Leu sharpens each of the fused matrix A.,, encouraging each client to retain only its strongest neighbors (Ghasedi Dizaji et al. 2017). This, in turn, guides to favor the view (i.e., data gradient) that leads to more clusterable affinity structure. FEDDAG learns the weight vector w using a lightweight multi-layer perceptron (MLP) (Almeida (020) trained via gradient descent to minimize the entropy loss Len- Finally, FEDDAG constructs the proximity matrix using the learned w as shown in Eq.
3.4 OPTIMAL CLUSTERING ALGORITIM 3
High-level idea. FEDDAG introduces adaptive clustering mechanism that automatically identi- fies the optimal number of clusters. This mechanism incorporates a novel federated-awan metric to evaluate clustering quality.
Details of the method. Given the proximity matrix A.j. the server applies agglomerative hierar- chical clustering (HC). In HC, the clustering threshold (0, 1] controls merges: clusters with pairwise distances below a are merged. Smaller a yields more clusters; larger a merges more broadly. The server iterates over different values generate candidate clusterings (Ci,...; Cz), each with a distinct number of clusters Z. Each clustering is evaluated using two metrics. Compact- ness loss Li promotes tight clusters, while degeneracy penalty L2 discourages small clusters:
L1 Σ
1
AnD
L2 =
1
2
exp
max (0, E - Yac |C+|]
(9)
where = N/Z and oe denote the mean and standard deviation of cluster sizes. A cluster C2 is penalized if size |C,| < C -CC--00 yoc, with >> 0 controlling sharpness. The total loss is
C(c = + XL2,
(10)
where x > 0 balances the two terms. Lower L1 (tighter clusters) and Le (less over-splitting) indicate better partitions. FEDDAG selects the clustering with the lowest loss and relatively few clusters.
4 GLOBAL REPRESENTATION SHARING (GRS)
High-level idea. In the previous section, we have combined data and gradient information to im- prove clustering. This section introduces global representation sharing across clusters during training phase via dual-encoder mechanism to further enhance FEDDAG's ability to learn comple- mentary representations. The process for determining which clusters should complement each other and how training is carried out is described below:
Building Cluster Complementarity Graph (CC-Graph). We first construct a directed graph that identifies, for each cluster, which other clusters supply the class representations it lacks. Intu- itively, a cluster has a demand for class that class is underrepresented among its clients, and a supply if the class is well represented.
For each client and each class e E C, let m. denote the number of distinct classes present on client i and let ri.c E {0, .. my 1} be the rarity rank of class that client, where ri.c 0 means that is the rarest class on client i. For single client i and each class we define the client-level demand score (m, - rt.c) and the supply score (ri, + 1), so that rarer classes induce higher demand while more frequent classes induce higher supply. To obtain cluster-level scores, we aggregate the client-level values. For a requesting cluster Cp and a source cluster Cy. the demand and supply for class can be computed as shown below:
dp.c Σ (m re.e), 8g.c
Σ (re, + +1, iEC,
(11)
[C
LEC,
where dp.e captures how strongly Cp lacks class c, and 8g.r. measures how abundantly C, represents class average. Combining demand and supply yields the complementarity score between requesting cluster p and a source cluster q::
H'p. = Σ dp..c
H' -00
(12)
cEC
A large value of Hy, indicates that Cp has high demand for exactly those classes for which C, has high supply. However, Hy, only accounts for the relative quantity of each class and does not capture

---

Published as a conference paper at ICLR 2026
the quality alignment of the data between the two clusters. To make the CC-Graph sensitive to alignment, we incorporate the per-class principal-angle information V between client subspaces (see Section (3.2) into the complementarity score. For each client pair (i.j) and class c, first clip the class-wise angle i.c. to the range [0*, ,900 and then map it to [0, 1] as follows:
Ti
Visie 90
Σ ΣΣ Tugie.
(13)
Col[Col
VECp JE
Here, the mapped value Tejj. is close to 1 when the class-c feature subspaces of clients i and j are well aligned and close to when they are poorly aligned. To obtain a cluster-level alignment score I p.q.ec we average over all client pairs across clusters p and q. Finally, we incorporate the alignment score into the demand-supply term Hip. to compute a refined complementarity score as:
H p.9 > dp.c Sq.e I Lipcs
Hp. oc.
(14)
CEC
Here, a high value of Hp.g indicates that C. is a strong complementary source for Cp: the terms dzee and Sq.e capture relative quantity, while I p.q.. ensures that complementarity also reflects how well the corresponding class-c representations are aligned between the two clusters.
Finally, we sparsify this score matrix into a directed adjaceney matrix. For each row p. we keep only the top-k largest values Hp. to build the CC-Graph. An edge + q in this CC-Graph indicates that cluster C will receive class representations from cluster Cy.
Training using dual encoders. For each client E Cz, the prediction model can be described as:
F. (x.) vb (6))) (6) (Xr; 0K),, (0)) (x.: O
(15)
FEDDAG optimizes the parameters ! 0. 0g) minimize the weighted empirical loss across N clients. This is achieved through parallel training phases of the primary and secondary encoders. During the primary phase for each cluster, the primary encoder O and the classifier Oç are optimized using data from clients i Cz, enabling the model to learn its own cluster-specific features. During the secondary phase, cluster C2 enriches the secondary encoders of clusters that seek to learn from it, as directed by the CC-Graph H. The procedures for both phases and their unified training strategy are detailed below.
ii) Primary encoder training. For each cluster, we optimize the primary encoder 6 and the classifier O via gradient descent, while keeping the secondary encoder e²e fixed. To approximate this, each client i E CD initializes its local parameters as (0!!/ 09) + (0! @g) and keeps the secondary encoder @? frozen. The local loss is then defined as:
((0),, 08) (o. (X,; 8.') (2) (X 6 ( (E)
(16)
Using the client loss defined in Eq. 16 each client performs SGD training update (0//, 65) as: (otf. .eç) (.(0 ,00), Vi t C.
(0V/ 0 (0 {8)
V
(17)
FEDDAG weighted aggregates the local primary encoder and classifier updates (0! - 01/) and (0f -) from each client i C2 to update (0!/, Og) as:
e.r - !!! Σ
IEC. LREc. Du (0" - e!/), e t -
D.l
Ec Exec, [r (Ø - +.)
[D,
(18)
(ii) Secondary encoder training. Given the CC-Graph H, an edge - means that learner cluster Cj asks source cluster C. to refine its secondary encoder 6 using C, 's data. To achieve this, all learner clusters {C; H(j. =) 1} of C₂ first aggregate their current secondary encoders into a single combined encoder and send this combined encoder to C.. Clients in C₂ then jointly train this received secondary encoder on their local data (with the primary encoder and classifier frozen), and the resulting gradients are aggregated and sent back to the learner clusters so that each of them can update its own secondary encoder. The process is described in detail below.
For each source cluster C., we optimize the secondary encoders (0)/) of the clusters that seek to learn from C.. First, given the CC-Graph H, FEDDAG first aggregate the secondary encoders of all learner clusters into a single combined encoder: 6 = Ej H(j=))=1 @ and sends it to C.. Then, each client i E C₂ initializes its local instance of the secondary encoder with the received encoder
7

---

Published as a conference paper at ICLR 2026
0 -- O while keeping the primary 0/ and the classifier Ø fixed; and then minimizes the following loss:
"(OP")) )x v( (X; 20)2) (Xx: (x==)());))))
(19)
Using this loss, each client performs SGD to update its local secondary-encoder parameters 03 as:
07r 02
V 02F' C((P5))
(20)
FEDDAG then weighted aggregates the local gradients (0 "25") for secondary encoder from each client i E C, and broadcasts the aggregated gradient back to the learner clusters. FEDDAG then updates the secondary encoder of each learner cluster Cj (where H(j 2) 1) using the aggre- gated gradient as follows:
[Dj] SECs EKec, [D&] (0-5 -
0 6
Σ
(21)
Unifying Primary and Secondary Training. Since the primary and secondary encoder updates are independent (Eq. 17 20 they can be trained in parallel. However, because the primary 0! and secondary 6 encoders are intended to capture complementary information, initializing them both randomly may lead redundant features. To avoid this, we ensure the primary encoder is partially converged before joint training starts. Specifically, during gradient-based similarity computation in $3.1 each client trains local model to partial convergence. We reuse the resulting feature extractors 8 initialize the global primary encoder 0!, thereby avoiding extra training rounds:
0
Σ SEC. LEEC
[D,]
V2 E Z,
(22)
[
FEDDAG structure summary. During the initial rounds, FEDDAG determines the optimal clus- tering configuration (see Algorithm [] Lines 1-14). Once the clustering is established, FEDDAG parallelly executes two phases: a primary training phase and secondary global feature-sharing phase (Algorithm[] [ Lines 15-23). Additional mechanisms for incorporating new clients and adapt- ing to distribution shifts without interrupting training are provided in Appendix A
EXPERIMENTS
This section experimentally evaluates FEDDAG, com-
Table 2: Exp S: Performance comparison for
pares it against existing works, and investigates: (i) FED-
concept shift across datasets.
DAG accuracy, (ii) Finding optimal clustering, (iii) Ab-
Algorithm
CIFAR-10
FMNIST
SVHN
lation studies, (iv) During evaluation, we report two vari- ants of our method: FEDDAG", which is restricted to the approach in combining data and gradient in- formation to form clusters and then training standard clustered FL model (single encoder and classifier) with- global representation sharing-and FEDDAG, which is the full algorithm that additionally incorporates dual- encoder inter-cluster sharing described $4
FedAvg
42.8710.36
42.68
37.93:10.39
FedBR
62.4110.288
8281
80.12.1.222
FedSoft
64.34:10.38
05.89=0.15
76.35+:0.40
PACEL
59.82:+0.22
78.42=0.35
78.8240.12
CFL
61.48:20.15
82.73=00233
79.15:0033
CFL-GP
66.74:10.28
84.71=0.13
82.38:00.33
FedGWC
65.91:10.19
03.85=0.21
81.63:0.28
FedRC
65.48.10.33
79.87
77.8610..9
64.58-10.39
0.26
81.56.10.14
FEDDAG
67.79-10.27
86.03=0.21
83.73+0.199
FEDDAG
69.00:10.20
0.19
85.34+0221
Baselines. We compare FEDDAG against SOTA methods:
single-model FL: Fe-
Table 3: Exp 3: Ablation study of cross-cluster rep-
dAvg (McMahan et al., 2017). FedProx (Li
resentation sharing under 20% label skew (Dirichlet
al.] 2020). (ii) personalized FL methods:
a' = 0.25), comparing FEDDAG, FEDDAG (dual en- coder w/o GRS), and FEDDAG (single encoder).
PerFedAvg (Fallah et al.) 2020). (iii) non-
clustered non-IID Fl.: FedMix (oon et al.]
Algorithum
CIFAR-10
FMNIST
SVHN
CIFAR-100
2021) FedBR (Guo et al.)
2023). )
(iv) clus-
FEDDAG
92.61-0.31
91.95 70.28+0.38
FEDDAG
88.67-0.18 90.76=20.12
92.75 93.22-0.20
00.2 70,37:20.33
tered FL data-based: PACFL (ahidiaa
FEdDAG
93.91 -023 72.84+0.30
et al.
2023).
(v) clustered
FL gradient-
based: IFCA Ghosh et al.
2020
CFL (Sattler et al. 2020 FedSoft Ruan & Joe- Wong 2022
FedRC (Guo et 2004)), FedGWC (Licciardi al. 2025). CFL-GP (Kim et al. 2024a).
Experimental Setup. We consider 100 clients, with 20% randomly selected per round. Unless stated otherwise, all experiments run for 200 rounds with each selected client performing 10 local
8

---

Published as a conference paper at ICLR 2026
Table 4: Exp 1: Performance comparison for Data Distribution with high degree of quantity shift (Dirichlet a' 0.25)
20%
Label Skew
30%
Label Skew
Algorithm
CIFAR-10
FMNIST
SVIN
CIEAR-100
CIFAR-10
FMNIST
SVIIN
CIFAR-I00
FedAvg
42.02 = 117
53.11 0.31
69.79
47.16100.1
54.24 0.0.5
72.86 I 0.40
61.15 I 0.64
50.99 = 1.33
RedProx
43.98 = 0.17
53.61
= 0.20
74.75 + 0.27
50.56
0.70
54.99
0.20
68.22
0.16
64.80 +: 0.25
48.66 0.80
PerFedAyg
81.09 0.35
86.51
00000..0 0.19
89:20
0.05
65.59 + 0.02
77.45
+ 0.24
89.77
0.155
88.23
+ 0.31
57.38 = 0.10
FodMix
83.55
83.12
0.33
81.9
0.27
82.21
FedBR
81.62 = 0.28
85.32
= 0.23
84.05
+ 0.30
81
+ 0.37
84.12 2 0.25
84.78
+: 0.38
56.32 0.33
FodSoft
76.44 0.18 0.18
84.58
0,14
83.75
0.33
62.54
0.41
72.48
1.48::0.11 0.17
85.15
0.17
$2.43
2 0.40
55.24
0.43
PACEL
86.93 = 0.40
91
89
0.25
66.11
0.29
84,66
91.96
0.25
90.48
0.23
58.30
0.56
CFL
68.67 0.76
81.90 0.10
79.83
1 0.38
$7.38
- 0.95
67.57
2 0.69
80.64
2: 0.21
75.21
+ 0.09
49.63
1.29
CFL-GP
09.1
0.22
67.89
90.01
FedGWC
85.97 -: 0.13
91.02
0 0.17
89.35
1: 0.10
69.19
1: 0.48
83.38
0.21
91.45 1: 0.12
88.94 1: 0.15
56.52
0.40
FodRC
75.12 = 0.28
88.32
88.05
+ 0.30
63.25 = 0.37
76.48
+ 0.37
88.12 0.25
85.78 - 0.38
54.32 0.33
IFCA
90.93
0.17
89.51
91.50
88.81
FEDDAG
88.67 0.18
92.75
0.22
91.87
4 0.26
70.37
- 0.33
86.95
0.21
92.18
0.15
90.97 + 0.13
60.84 0.65
FEDDAG
93.82
0.20
0.23
72.84
92.72
0.13
92.65
63.21
epochs (batch size 10, SGD). The principal vector U transmitted per class is roughly 1% the size of Diel- For gradient similarity Gi.j. each client trains locally for 22 rounds. To construct the CC-Graph, we select the top-k-2 source clusters.
Datasets. We use four popular datasets for the image classification task in FL setting, i.e., CIFAR- 10 (Krizhevsky et al. 2009). FMNIST (Xiao et al. 2017). SVHN (Netzer et al. 2011 and CIFAR- 100 (Krizhevsky et al. 2009).
Non-IID Data. We use multiple data distributions to simulate traditional and complex data skews:
Data Distribution I: This distribution evaluates FEDDAG under combined label skew and quan- tity shift. To simulate label skew, we randomly select p% of labels and assign them random client groups, repeating the process until all clients are assigned simillr -iimilar to PACFL. For quantity shift, we allocate samples the assigned labels using the Dirichlet factor (Ng et al. 2011]. A real-world example is predictive text input, where users may discuss similar topics, but word distributions vary due to individual preferences and typing habits.
This distribution evaluates FEDDAG under concept shift. Following prior work (Jothimurugesan et al. 2023, Guo et al. 2024), we simulate concept shift by modifying the labels of subset of clients. For example, label y is changed to (C- y) (v+ 1)%C, where C is the total number of classes. We perform three such transformations to simulate three distinct concepts.
Data Distribution II:
Similar modifications are applied the test set.
Number eiaaatre
Nurter clster
5
Nurber (MAters
Dolance Sroppladi 9o1
Dilance Shreano 12)
Distamo thrsshoodd
Dasance Uroahoold (g)
(a) CIFAR-10
(b) FMNIST
(C) SVHN
(d) CIFAR-100
Figure 2: Exp 2: Clustering score vs cluster a and number of clusters for finding optimal clustering.
This distribution evaluates FEDDAG under a different form of label skew. We adopt the Latent Dirichlet Allocation (LDA) method from Hsu al. (2019). using Dirichlet concentru concentration factors = 0.25 and a' 1.0.
Data Distribution III:
Additional experiments (e.g., performance evaluation, communication rounds) on the above and new (feature skew) distributions, hyperparameter tuning, implementation details, abla- tion studies are provided Appendix R Algorithm theoretical issues, such as convergence, complexity, and privacy analysis; distribution and client shifts discussed in Appendix A
Experiments on Data Distribution
Exp 1: Performance evaluation. We consider class skew p 20% and 30%, with the Dirichlet concentration parameter a' set to for low and 0.25 for high quantity shift. Table 4 shows the re- sults for a' 0.25, while the results for a' 1 are included in Appendix B.3 We observe that

---

Published as a conference paper at ICLR 2026
Table Exp 5: Performance comparison under LDA skew (a' 0.25 and a' 1.0).
0.25
1.0
Algorithm
CIFAR-10
FMNIST
SVHN
CIFAR-10
FMNIST
SVHN
FodAvg
66,48 0.21
47.26 + 0.28
46.13 = 0.48
41.78
0.73
85.48 + 0.36
81.89 = 0.31
FedSoft
71.08 1 0.26
83.75 0.26
85.67 1 0.19
73.83
0.42
87.85 1 0.31
85.92
PACEL
73.91 1 0.43
85.93 1.132
87.23 0.20
80.52
0.15
93.31 0.228
92.17
CFI.
67.46 0.12
8..1
85.19 0.25
78.94 = 0.18
83.16
82.75 - 0.28
CFL-GP
73.84 0.28
0.14
88.04 = 0.19
83.57 0.15
92.21 0.23
91.67 - 0.19
FedRC
70.19 0.42
8.224
87,91 = 0.26
81.76
0.16
88.27
+ 0.22
86.29 - 0,42
IFCA
74.43 1 0.32
87.53
88.81 1 0.13
82.27 1 0.19
92.79
1 0.33
92.12 1 0.15
FEDDAG"
75.52 + 0.27
89.65
1 0.16
91.27 = 0.22
85.03 0.21
93.95
+ 0.20
93.08 + 0.18
FEDDAG
77.84 + 0.23
91.88 + 0.10
93.17 = 0.18
87.62
0.14
94.68
1 0.13
94.15 Σ 0.11
single global FL baselines (e.g., FedAvg, FedProx) perform poorly under heterogeneity due model drift (Zhao et al. 2018), while clustered FI. methods yield stronger performance. Both variants of FEDDAG outperform state-of-the-ard mnsclines-iicluding data-based methods (c.g.. PACFL) and gradient-based methods (e.g., IFCA, FedGWC). The lighter variant, FEDDAG", achieves strong performance by combining data and gradient information to yield improved clustering. The full FEDDAG further enhances accuracy by enabling complementary representation sharing across clus- ters, allowing them to learn richer feature spaces.
Exp 2: Finding Optimal Cluster Formation. The server iterates over the clustering threshold a in Agglomerative HC at regular intervals (e.g., 0.05) to generate candidate clusterings. For each, the clustering loss (() (see $3.4) is computed. In Figure 2 the x-axis shows or; the red curve indicates loss, and blue bars denote the number of clusters. Unlike traditional metries (c.g., inertia) where loss decreases with more clusters, we observe abrupt increases in loss even as the number of clusters decreases for certain a values. This is due to FEDDAG's federated- aware clustering loss penalizing over-splitting into small clusters. The optimal is selected as the point with low clustering loss and a relatively small number of clusters (e.g., for Figure@b> a = 0.65).
Exp 3: Ablation Studies. We examine whether accuracy gains from inter-cluster global representa- tion sharing (GRS) via the dual-encoder architecture (see $4) arise from genuine feature enrichment simply from increased model parameters. To isolate this effect, variant with GRS disabled: during secondary-encoder training, instead of receiving representations from other clusters, each client trains its secondary encoder only on its own data and aggregates within its cluster. We denote this variant FEDDAGf; it is distinct from FEDDAG, which uses a sin- gle encoder. As shown Table 3] full FEDDAG (with GRS) achieves the highest accuracy. while FEDDAG performs comparably to FEDDAG, confirming that the gains of FEDDAG stem from cross-cluster representation sharing rather than model size alone.
implement dual-encoder
Experiment on Data Distribution II
compares the performance of SOTA algorithms and FEDDAG on different datasets under concept shift and shows that FEDDAG achieves higher ac- curacy than the baselines. This improvement stems from FEDDAG's class-wise wiis comparison mecha- nism, which provides more accurate similarity estimation under concept shift than existing methods.
Exp 4: Performance under concept shift. Table 2
Experiment on Data Distribution
Exp 5: Performance under varying LDA skew. Table Shows shows accuracy under LDA-based skew with = 0.25 and a' 1.0. FEDDAG consistently outperforms SOTA methods by leveraging cross-cluster feature sharing and integrating data and gradient information for clustering, leading to robust performance under LDA-base based partition.
CONCLUSION
We develop a novel algorithm, FEDDAG, that addresses the limitations of existing clustered FL techniques and effectively tackles data heterogeneity challenges in FL by developing novel method that combines both data and gradient information to cluster clients more effectively. Furthermore, FEDDAG utilizes representation sharing across clusters and incorporates automatically determine the optimal number of clusters. Experiments on various heterogeneous data distributions demonstrate that FEDDAG outperforms existing approaches in terms of accuracy.
efficient mechanism to
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Author M. K. was supported in part by NSF awards DMS-2204795, OAC-2115094, CNS-2331424, ITE-2452833, ARL/Army Research Office awards W911NF-24.1-0222 and W911NF-24-2-014 and Virginia Commonwealth Cyber Initiative grants. Author V. O. was supported in part by the National Science Foundation (NSF) under Grant DGE 2043104. Author S. S. was supported by NSF Grant 2245374. We thank NJIT HPC facility for providing GPU support for our experiments.
REFERENCES
Luis B Almeida. Multilayer perceptrons. In Handbook of Neural Computation, pp. C1-2. CRC Press, 2020.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint arXiv:1912.008/8, 2019.
Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning via submodular maximization. In International Conference Learning Representations, 2022.
Rahul Atul Bhope, KR Jayaram, Nalini Venkatasubramannian, Ashish Verma, and Gegi Thomas. Flips: Federated learning using intelligent participant selection. In Proceedings of the 24th Inter- national Middleware Conference, pp. 301-315, 2023.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF international conference computer vision, pp- 3514-3522, 2019.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. arXiv preprint wrXivv2107.00778, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre-
sentations for personalized federated learning. In International conference machine learning, pp. 2099. PMLR, 2021.
William HE Day and Herbert Edelsbrunner. Efficient algorithms for agglomerative hicrarchical clustering methods. Journal of classification, 1(1):7-24, 1984.
Shu Ding and Wei Wang. Collaborative learning by detecting collaboration partners. Advances in Neural Information Processing Systems, 35: 15619-11566411 2022.
Moming Duan, Duo Liu, Xianzhang Chen, Yujuan Tan, Jinting Ren, Lei Qiao, and Liang Liang. Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. In 2019 IEEE 37th International Conference on Computer Design (ICCD), pp. 246-254. JEEE, 2019.
Moming Duan, Duo Liu, Xinyuan Ji, Yu Wu, Liang Liang. Xianzhang Chen, Yujuan Tan, and Ao Ren. Flexible clustered federated learning for client-level data distribution shift. IEEE Trans- actions on Parallel and Distributed Systems, 33(1 1):2661-2674, 2021.
Cynthia Dwork. Differential privacy. International colloquium on automata, languages, and programming, pp. 1-12. Springer, 2006.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in neural information processing systems, 33:3557-3568, 2020.
Craig Gentry. A fully homomorphic encryption scheme. Stanford university, 2009.

---

Published as a conference paper at ICLR 2026
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference computer vision, pp. 5736-5745, 2017.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586 19597, 2020.
Boqing Gong, Kristen Grauman, and Fei Sha. Learning kernels for unsupervised domain adaptation with applications visual object recognition. International Journal of Computer Vision, 109(1): 3-27, 2014.
Yongxin Guo, Xiaoying Tang, and Tao Lin. Fedbr: Improving federated learning on heterogeneous data via local learning bias reduction. In International conference on machine learning, pp- 12034-12054 PMLR, 2023.
Yongxin Guo, Xiaoying Tang, and Tao Lin. Fedre: tackling diverse distribution shifts challenge in federated learning by robust clustering. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Nathan Halko, Per-Gunnar Martinsson, and Joel Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53 (2):217-288, 2011.
Jialuo He, Wei Chen, and Xiaojin Zhang. Fedaa: reinforcement learning perspective on adaptive aggregation for fair and robust federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17085-17093 17093, 2025.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint 1903. /2261, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.063:: 2019.
Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alter- nating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pp. 665-674, 2013.
Ellango Jothimurugesan, Kevin Hsieh, Jianyu Wang, Gauri Joshi, and Phillip B Gibbons. Federated learning under distributed concept drift. In International Conference on Artificial Intelligence and Statistics, pp. 5834-5853, PMLR, 2023.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and trends in machine learning, 14(1-2):1 2021.
Myeongkyun Kang, Soopil Kim, Kyong Hwan Jin, Ehsan Adeli, Kilian Pohl, and Sang Hyun Park. Fednn: Federated learning on concept drift data using weight and adaptive group normal- izations. Pattern Recognition, 149: 10230, 2024.
Sai Pranceth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132-5143. PMLR, 2020.
Heasung Kim, Hyeji Kim, and Gustavo De Veciana. Clustered federated learning via gradient-bart basedd partitioning. In Forty-first International Conference on Machine Learning. 2024a.
Heasung Kim, Hyeji Kim, and Gustavo De Veciana. Clustered federated learning via gradient-based partitioning. In Forty-first International Conference on Machine Learning, 2024b.
Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some appli-
cations. IEEE Transactions automatic control, 25(2):1441776, 1980.
12

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2 2324, 2002.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.0533 2019.
Minghao Li, Dmitrii Avdiukhin. Rana Shahout, Nikita Ivkin, Vladimir Braverman, and Min- lan Yu. Federated learning clients clustering with adaptation data drifts. arXiv preprint wrXiv:2411.015800 2024a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and sys- lems,
2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357- 6368. PMLR, 2021a.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.076633 2021b.
Zhilong Li, Xiaohu Wu, Xiaoli Tang, Tiantian He, Yew-Soon Ong, Mengmeng Chen, Qiqi Liu, Qicheng Lao, and Han Yu. Benchmarking data heterogeneity evaluation approaches for person- alized federated learning. In International Workshop on Trustworthy Federated Learning, pp- 77-92. Springer, 2024b.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis Shii--Pil Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001. 01553, 2020.
Alessandro Licciardi, Davide Leo, Eros Faní, Barbara Caputo, and Marco Ciccone. Interaction- aware gaussian weighting for clustered federated learning. arXiv preprini arXiv:2502.03340, 2025.
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems, 33:2351-2366 2020.
Qiqi Liu, Jiaqiang Li, Yuchen Liu, Yaochu Jin, Lingjuan Lyu, Xiaohu Wu, and Han Yu. Personal- ized federated learning under local supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4069-4079, 2025.
Guodong Long, Ming Xie, Tao Shen, Tianyi Zhou, Xianzhi Wang, and Jing Jiang. Multi-center federated learning: clients clustering for better personalization. World Wide Web, 26(1):4815500 2023.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34:1 14:15434-15447, 2021.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-cfficiina learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International con- ference machine learning, pp. 4743 4751. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2026
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Kai Wang Ng, Guo-Liang Tian, and Man-Lai Tang. Dirichlet and related distributions: methods and applications. 2011.
Theory,
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In International Conference on Machine Learning, pp. 17716-17758. PMLR, 2022.
Anik Pramanik, Amlan Sarker, Zabirul Islam, and MMA Hashem. Public sector corruption analysis with modified k-means algorithm using perception data. In 2020 1 11th International Conference on Electrical and Computer Engineering (ICECE), pp. 198-201. IEEE, 2020.
Anik Pramanik, Pan Xu, and Yifan Xu. Equity promotion in public transportation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11890-11898, 2023.
Yichen Ruan and Carlee Joe- nee-ong. Fedsoft: Soft clustered federated learning with proximal local updating. In Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 8124 2022.
Felix Sattler, Klaus-Rober Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neu- ral networks and learning systems, 32(3):371033722,22 2020.
Satu Elisa Schaeffer. Graph clustering. Computer science review, 1(1):27-64, 2007.
Changlong Shi, He Zhao, Bingjie Zhang, Mingyuan Zhou, Dandan Guo, and Yi Chang. Fedawa: adaptive optimization of aggregation weights in federated learning using client vectors. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 30651- 30660, 2025.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in neural information processing systems, 13:21394--21055 2020.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual ho- mogeneity learning: Defending against data heterogeneity federated learning. In International Conference Machine Learning, pp. 21 11111-22 11-21132. PMLR, 2022.
Tiffany Tuor, Shiqiang Wang. Bong Jun Ko, Changchang Liu, and Kin LLeun. Overcoming noisy and irrelevant data in federated learning. In 2020 25th International Conference Pattern Recognition (ICPR), pp. 5020-5027. IEEE, 2021.
Saced Vahidian, Mahdi Morafah, Weijia Wang, Vyacheslav Kungurtsev, Chen Chen, Mubarak Shah, and Bill Lin. Efficient distribution similarity identification in clustered federated learning via prin- cipal angles between client data subspaces. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 10043-10053 2023.
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10165-10173, 2021.
Jiangiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication- efficient distributed optimization. Advances in Neural Information Processing Systems, 31, 2018.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2575-2584, 2020.
Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu. Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. In International Conference Machine Learning, pp. 37860-37879. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnin a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 708.077778.77777,,000 2017.
Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang, and Liusheng Huang, Pri- vate fl-gan: Differential privacy synthetic data generation based federated learning. In Icassp 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2927-29311 IEEE, 2020.
Yang Xu, Zhida Jiang, Hongli Xu, Zhiyuan Wang, Chen Qian, and Chunming Qiao. Federated learning with client selection and gradient compression in heterogeneous edge systems. IEEE Transactions on Mobile Computing, 236))5446-5461, 2024a.
Yi Xu, Ying Li, Haoyu Luo, Xiaoliang Fan, and Xiao Liu. Fblg: A local graph based approach for handling dual skewed non-iid data in federated learning. In Proceedings the Thirty-Thine International Joint Conference on Artificial Intelligence, IJCAI-24, K. Larson, Ed. International Joint Conferences on Artificial Intelligence Organization, volume 8, pp. 5289-5297, 2024b.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8715- 8724, 2020.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. arXiv preprint arXiv:2107.00233 2021.
Dun Zeng, Xiangjing Hu, Shiyu Liu, Yue Yu, Qifan Wang, and Zenglin Xu. Stochastic clustered federated learning. arXiv preprint arXiv:2303.00 00897, 2023.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learn- ing with label distribution skew via logits calibration. In International Conference Machine Learning, pp. 26311-26329 PMLR, 2022a.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10174-10183, 2022b.
Yuxin Zhang, Haoyu Chen, Zheng Lin, Zhe Chen, and Jin Zhao Fedac: A adaptive clustered federated learning framework for heterogeneous data. arXiv preprint arXiv:2403. 16460, 2024.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-idd data. arXiv preprint 3xxx:/800.005530 2018.
15

---

Published as a conference paper at ICLR 2026
A TECHNICAL DISCUSSION AND ANALYSIS
This section presents a detailed discussion and breakdown of FEDDAG, covering key design ele- ments, communication and privacy considerations, adaptability to new clients and shifting distribu- tions, and practical implementation details.
A.1 ADDITIONAL RELATED WORK
Data Augmentation based FL techniques propose sharing a small global dataset among clients and combining it with their local datasets to mitigate heterogeneity (Tuor al. 2021 Xin et al., 2020). Approaches such as Xin et al. 22000;; Yoon et al. (202T); Guo et al. (2023) employ generative adver- sarial networks (GANs) or averaged local data (Random Sample Mean) to create prreecyprrservingg
pseudo-data that is used to reduce bias in client models. Astraea (Duan et al. 2019) constructs a
globally balanced data distribution by performing local augmentation across participating clients. VHI. (Tang et al. 2022) creates virtual samples from noise shared across clients to regularize train- ing by aligning local feature representations with these virtual data.
Meta-learning approaches include personalized FL (Arivazhagan et al. 2019; Liang et al. 2020 Li al.| 2024b) and model regularization methods (Li et al.) 2021a Dinh al.] 2020: Karim- ireddy et al. 2020). Per-FedAvg (Fallah et al., 2020) is a personalized variant of FedAvg based on the Model-Aggostie Meta- -eta-Learning (MAML) framework. FedPer (Arivazhagan et al. 2019) and
FedRep (Collins et al. (02I)) split the backbone into a feature extractor and a head to share feature in- formation, while FedRoD (Chen & Chao (02T) maintains a shared feature extractor and two heads. FedSimSup (Liu et al. (025) uses a local supervisor and hataa-smilaritt-weerr
inter-learning model to better align global knowledge with heterogeneous local data. Our approach, FEDDAG, maintains global-persondl decoupling via dual-encoder architecture, where the global parameters are updated only by selected subset of clients instead of being influenced by all clients.
Client Weighting and Selection Another line of work tackles non-IID FL by selecting participants or assigning aggregation weights based on client properties such as data quality, label distribution, model alignment, and historical contribution. Representative strategies for assigning weights lecting clients include linear programming (Pramanik et al.] (023). submodular optimization, rein- forcement learning, bandit-based optimization, and adaptive wnrreggttionweighhe optimization (Bal-
akrishnan et al. 20222] Bhope et al. 2023 Xu et al. 2024a He et al. 2025 Shi et al. (025)..
A.2 PRELIMINARIESS
Principal Angles Between Two Subspaces. Consider two subspaces, V span (onn(vi--v vv.....,y, vp) and X' = .p.. ..yh where V and X' are p-dimensional and q-dimensional subspaces of R" respectively. The sets (VI.... vp} and {x]...., ., x,y are orthonormal, with S p S q. A sequence of p principal angles, 0 < +1 I 2b2 .... I op S 2 is defined to measure the similarity between the subspaces. These angles are calculated as:
|V |||||||||
+ (V, = min cOs
(23)
vEV
where || is the norm. The smallest of these angles is ( xI), with the vectors vi and xı as the corresponding principal vectors. The principal angle distance serves as a metric to quantify the separation between subspaces Jain et al. (2013).
Agglomerative hierarchical clustering (HC). (Day & Edelsbrunner 1984 is a popular method in machine learning for grouping similar objects based on an adjacency (proximity) matrix. We found HC to be the best fit for FEDDAG. We also experimented with other clustering algorithms, e.g., K-means (Pramanik et al. 2020) and graph clustering (Schaeffer) (007). but we observed that the clustering algorithm does not make much difference in cluster formation. HC begins by treating each data point as its own cluster. During each iteration, HC identifies two clusters that are most similar and merges them. The criterion for selecting which clusters to merge depends on linkage method: e.g.. in single linkage, the L2 (Euclidean) distance between two clusters is defined as the smallest distance between any pair of points from the two clusters. As merging criterion, FEDDAG defines a clustering threshold a E (0, 1]. such that any two clusters with a distance less than a are merged.; e.g., a=1 results in all clients being grouped into single cluster.
16

---

Published as a conference paper at ICLR 2026
A.3 FEDDAG TECHNICAL COMPONENTS
An illustration of the FEDDAG algorithm is shown in Figure | weighted data-based similarity computation is shown Algorithm And, the algorithm for com- bining both data and gradient information to improve clustering is shown in Algorithm
The algorithm for class-wise
A.4 CONVERGENCE ANALYSIS
Following Pillutla et al. 2022 that works on partial model personalization, we consider the Slared-erssonaizee objective:
n
min F(u, V) 8,
Σ v(,,
(24)
where u denotes shared parameters and V = {v:)=_ personalized parameters. In our dual-encoder model (Eq. equation [3), for each cluster z we map the secondary encoder as the shared block and the primary encoder (optionally together with the classifier) as the personalized block:
02 (shared: secondary encoder),
u2
V
! 99) (personalized: primary encoder + classifier).
Given a fixed clustering {C₂ ) (one-shot data and gradient combined similarity; see $ the cluster-level empirical risk can be written in the shared-pert personnazee form of Pillutl al. (002
Z
D
min F((uz, V21) Vo
Σ
Fu V2),
Enec.
D
IEC.
(u,, V₂) = C(v,. (X, 0/'), (x;u,,,;
Thus, for each cluster Pillutla et al. (2022)* 's analysis applies to the pair (u,, V.), and the full objective is a weighted sum over clusters. So, based on this, we will define notations, assumptions, and the convergence analysis below:
Block notation and participation model. For each cluster {1..... Z} in the fixed partition (C.)2_1 we decompose the parameters
u2 = O
(cluster-global / secondary encoder),
(25)
:= (o ) (cluster-personal: primary encoder classifier). Let m be the total number of clients and m, == |C2 the number of clients in cluster 2; define the cluster weights
(26)
2 Tx 1.
m
(27)
#=1
In each communication round, cluster samples 8, clients (without replacement) and runs E local steps. The average per-round participation fraction
n mz
E (0,1).
(28)
Loss and per-client objective. For client E C. with data D, (X,, Y.), define Flu,,, V.) = dr., (X;: (X;; 09))
(29)
and the cluster-weighted empirical risk
D
F(uu,, v2)=
Fi(ue, Vz).
(30)
Exec, Dr
iEC,
17

---

Published as a conference paper at ICLR 2026
Scope of the analysis. The full FEDDAG algorithm includes three dynamic components: (i) an initial clustering phase that combines data- and gradient-basse similarity (Algorithm T Lines 1- 14), (ii) mechanism to attach newcomers to existing clusters (Appendix, Algorithm and (iii) an optional re-clustering step under significant distribution shift. this subsection, we analyze the stationary training regime between such events: we condition partition { and study the convergence behavior of the shared-personalized objective Eq. 30 under this partition. Newcomers mild distribution changes that do not trigger re-clustering can be viewed as small perturbations of the variance and heterogeneity constants in our bound, whereas re-clustering event corresponds to switching
fixed client set and fixed
new objective with new partition and new constants. The theorem below should therefore be interpreted as per-phuse guarantee for any interval between two re-clustering events.
Assumptions used in the theorem. We state the standard conditions in our block notation; expec- tations are w.r.t. the algorithm's sampling and stochasticity. Assumption A.1 (Smoothness). Each client loss in equation 29 is L-smooth in (uz, V2). For all (Ug, V2) and (u',, V(),
\ (v..v.) Pi(z. V.)- V....v.) Fol V
S (1,, Vz) (u/. VOll-
(31)
Equivalently, Ft is L-smooth in each sub- bub-block 04, 0. and 0 Assumption A.2 (Unbiased stochastic gradients with bounded variance). For any sampled client i E Cz.
E[V a. F] = V., F, E F[V v. F] Vv, F, where Fv.F (Voy o F. Ves F). Define the cluster-weighted variances
EiIV., F V., FI]]] S oz - FL Vv. FI S So:..
(32)
(33)
Σ T2 M,21
o. =
Σ
T2 V..
(34)
Assumption A.3 (Gradient diversity heterogeneity).
Let Eu V2) There exist finite constants
Enec, [Pe] Eiec, [D,] (u,, Vz) be the average loss in cluster 2.
02 2 0 and dant
2 such that
2
z
Σ Te
V Vo
Σ Te Vv. 2=1
-or.F--vv.lle vv. 8
(35)
in
and the cross-cluster mismatch (relevant to the sharing step) is bounded by 8out-
Assumption A.4 (Phase-wise stable clustering). For the training phase under consideration. the partition {C₂ )² obtained at initialization (t=0 of this phase) remains fixed for the analysis hori- zon 1, 1....,7. no clients reassigned, and clusters do not split or merge within this phuse. Potential re-clustering events (triggered by significant distribution shift) are modeled as the start of new phase with its own objective and constants and are therefore outside the scope of the current per-pie per-phase guarantee.
Assumption A.5 (Cross-cluster sharing noise). The cross-cluster representation sharing (via the CC-Graph) is either deterministic (no additional noise), or introduces an additive variance bounded by 22 Share in the updates of the u-blocks.
Initial suboptimality.
We denote the initial gap by
A = F( vo)2 FF,
(36)
where Fx is the optimal value of equation 30
Theorem A.1 (Convergence of FEDDAG (per-cluster globals, dual encoders)). Let the assumptions above hold. Choose learning rutes 1 = T/(LE) and Tashare ((1/L), for a constant t depending on Tt, the variance terms, heterogeneity, and participation. Then, ignoring absolute constants and provided clustering is stable,
2
F
mL Σ B Vv Vecy FI]?
<
(Ae a.m. 1/2
(47 0m..2)
1/3
T1/2
T2/3
T
(37)
18

---

Published as a conference paper at ICLR 2026
where the effective variance terms
2
2
sim,
L
in
T2
1 -
m
8z
+
0u
L
+
Σ
m
8z
0v.2 + oaharc)
(38)
==1
==1
0
sim,2
2
(0 + out
+ 0, σV
sharo
1 (1-1) D-
1
(39)
Remark 1 (Clustering stability). The bound relies on fixed partition; oscillations due to re- clustering invalidate the descent decomposition. In practice, stability is supported empirically by (i) one-shot blended (data+gradient clustering at t-0 and (ii) the fact that training is conducted within the fixed clusters thereafter.
Remark (What differs vs. single-global frameworks). In FEDDAG, we aggregate both the per- cluster global blocks V1:2 (secondary encoders, coupled via the CC-Graph) and the cluster-persen personal blocks 122 (primary encoder + classifier). Consequently, 2am,, and sim,2 expose: (i) per-cluster sampling s=/m= (larger 8N improves the first term), (ii) local steps E (fewer local steps reduce the drift factor 1/E), E), and (iii) cross-cluster sharing noise
a share (zero for deterministic Laplacian smoothing; small but positive for stochastic distillation). The asymptotic 7' rate observed once all devices are seen on average at least once; convenient sufficient condition (up to constants)
Ae
(1-9)B)
E
8=
T
max
2
n
(40)
sim,
q
m2
A.5 COMMUNICATION AND COMPUTATION COMPLEXITY
FRDDAG minimizes communication and computation overhead, aligning with the scalability quirements of federated learning systems. Before dual-encoder joint training begins, each client locally trains for i rounds without federation (see Algorithm D At the end of this phase, each client uploads: (i) a kisparse gradient F, of dimension & < Dil. and (ii) class-wise p principal vectors U E Raxr for c = I,...,C. C. The number of principal vectors is kept small (typically 1-2% of the class size). Hence, the combined communication cost of Å; and U is negligible rel- ative to the size of the model parameter space (+). The computation of principal vectors via SVD incurs cost of OFFN2) per client, assuming local dataset of N samples and F features with N F.
Once the proximity matrix and clustering are finalized, FEDDAG maintains the per-round communication cost as FedAvg in terms of transmitting model parameters. However, due to its dual-encoder architecture, additionally transmits a secondary encoder (02f) alongside the primary encoder (f)f), both of equal size. In each training round, selected clients perform two local SGD phases:
Primary phase standard local update on (f 8C).
Secondary phase - additional local update on 02f", which has the same size as 0'f
both phases are executed in the same round, the local computation cost approximately 2 that of FedAvg. However, the two phases can be alternated when computation constrained- example, updating the primary encoder for several rounds (e.g.. 5 rounds) followed by a single round updating the secondary encoder. We also perform an experiment evaluating the trade-offs of these alternating schedules, showing how they affect convergence speed; see $B.9
Since updates the primary and secondary encoders are independent, the correctness and conver- gence of the final model are preserved under this alternating schedule.
A.6 CLUSTERING OVERHEAD OF FEDDAG
While ÇA.5 analyzes the overall complexity of FEDDAG, here we focus specifically on the clus- tering component and compare it to other clustered FL approaches. Conceptually, clustering in
19

---

Published as a conference paper at ICLR 2026
FEDDAG can be separated into two parts: (i) one-time warm-up and initial clustering phase and (ii) a re-clustering phase to handle distribution shift.
(i) Initial clustering phase. In this part, discuss the computation and communication overhead of the warm-up and initial clustering phase:
Computation overhead. In the initial phase, each client runs fy rounds of local warm-up epochs. As detailed in Appendix B.2 (Hyperparamee Tuning), a small number of rounds, such as fg = 2, is enough to obtain the client gradient signatures for similarity computation. Another important aspect is that these warm-up updates are not discarded. While the clients simply do not participate in FL aggregation during this short period, once clustering is assigned, training continues directly from the warmed-up model. The same warmed-up parameters are also used to initialize the secondary encoder training during our dual-encoder-basee clustered FL, so the warm-up not an additional or wasted cost.
principal vectors via truncated SVD for client with N samples and feature dimension F, which costs O(FN2). This one-time SVD, together with lg S 2 warm-up rounds, constitutes the main clustering-side computation over- head in FEDDAG. The SVD cost is incurred only once during initialization and is amortized over all subsequent communication rounds. This computation overhead of FEDDAG is compared with other clustered FL approaches follows:
Another aspect of FEDDAG's s initial clustering is computing
PACFL also performs an SVD/PCA-type step per client to extract principal components, so the clustering- -ttge complexity is of the increase is the ig rounds of warm-up, but FEDDAG mainly adds the reuse of the ig warm-up for the model during the clustering phase, rather than treating clustering separate, discarded pre- processing step.
same order. In FEDDAG, one addition that can create a slight
In contrast, iterative clustering methods (IFCA, CFL, CFL-GP, FedSoft, FedRC) re-evaluate clus- tering at every communication round: at round f, the server sends K cluster models to each partic- ipating client, and each client must evaluate all K models locally decide its cluster assignment. Over T rounds, this leads to roughly KT model evaluations per client, whereas FEDDAG pays the SVD + warm-up cost only once and then trains on a fixed cluster assignment (unless a rare re-clustering event is triggered; see below).
Communication overhead. After warm-up, each client uploads k-sparse gradient Å (only 1-2% of coordinates) and a small number of per-class principal vectors U (about 1-2% of the class data size). The resulting communication is negligible compared to sending the full model parameters JO). Both the sparsified gradient and the principal vectors are communicated only once during the clustering phase, so their cost is also amortized over the entire training run. The communication overhead is compared to other clustered FL approaches as follows:
PACFL has similar one-shot communication pattern for principal components during clustering, as it also sends principal vectors. FEDDAG incurs a slightly higher cost because it additionally sends the sparsified gradient, but this is still single extra message of size 1-2% of the model.
Iterative clustered FL methods such as IFCA, CFL, CFL-GP, FedRC, and FedSoft must send all K cluster models (each of size M) every participating client in every round, so each client receives on the order of KM parameters per round and returns updates for its chosen cluster. In contrast, FEDDAG only sends 1-2% of the model parameters M and principal vectors whose total size is about 1-2% of the client data.
(ii) Re-clustering overhead. We discussed that FEDDAG performs re-clustering under severe dis- tribution shift (Appendix A.8) as an infrequent, deployment-time procedure: it is triggered only after a major shift in the client distribution and not used in our main experiments (e.g., when the Wasserstein distance changes by more than 20%). When activated, it simply re-runs the same warm-up - principal vector computation and clustering procedure, again without resetting the mod- els, so the cost is similar to the initial clustering phase as above. During the FL training period (apart from the re-clustering mechanism), FEDDAG does not require further clustering, as it trains on the assigned clusters.
To complement the qualitative discussion of computation and communication overhead above, we next provide simple numerical complexity analysis that quantifies how the clustering cost of one-
20

---

Published as a conference paper at ICLR 2026
time SVD and partial gradient convergence in FEDDAG compares to the per-round clustering cost of iterative clustered FL methods. Secondly, we also provide an alternative approach to compute principal vectors that reduces the overall clustering overhead.
A.6.1 NUMERICAL ANALYSIS OF CLUSTERING OVERHEAD
The discussion above compares FEDDAG qualitatively iterative clustered FL methods, We now provide a simple numerical complexity analysis to quantify the relative overhead.
For each client with local sample size N and feature dimension F, let X E RNX denote its feature matrix. A full SVD of X would cost on the order of:
O(min F2, FN²)):
but FEDDAG only requires the top < min(N, F) singular vectors per class. Using efficient truncated or randomized SVD, as shown here Halko et al. (2011). the per-client cost reduces to:
OppNF),
which is linear in both N and F, and this cost is incurred only once during the warm-up initial clustering phase.
By contrast, iterative clustered FL methods (e.g., IFCA, CFL, CFL-GP, FedGWC, FedRC, FedSoft) maintain K cluster models and, in each communication round, send all K models every partic- ipating client. To update cluster assignments, each client evaluates all models on its local sample. Then
samples. Let Med denote the cost of single forward pass of the full model the per-round and total clustering costs of these methods are
iterative per-round cost: O(KN Miwa).
iterative total cost over 7 rounds: O(KTN
Comparing the clustering cost of FEDDAG and iterative clustered FL methods for a single client, first account for the components of FEDDAG's clustering overhead. This consists of the one- time truncated SVD used for initial clustering and the lg 2 warm-up rounds needed to obtain stable gradient signatures. Even though these warm-up updates are reused for subsequent training (and hence are not wasted), we conservatively count them toward the clustering overhead. Thus, the per-client clustering cost of FEDDAG is
FedDAG clustering cost: OpNFF) + M1..)
where p is the number of principal vectors, N is the local sample size, the feature dimension, and Miva denotes the cost of a single forward pass of the full model on one sample.
By contrast, iterative clustered FL methods require evaluating all K cluster models on the client's N local samples in every communication round and repeating this over T rounds. The resulting per-client clustering cost
iterative clustering cost: DKTNNMrul
The ratio between the total iterative clustering cost and the (conservatively defined) clustering cost of FEDDAG is therefore
iterative clustering cost
KTN
KTM
R FEDDAG clustering cost
pNF + 2N Miwd pF + 2M
To obtain a concrete sense of scale, consider CIFAR-10 setup with 100 clients. The training set has 50,000 examples, so each client holds about network with three convolutional layers followed by two fully connected layers, whose encoder outputs F = 256-dimensional features and whose total parameter count is approximately 5.4 x 105. We approximate the per-sample forward cost by this parameter count, ie.. Mas 5.4 x 10S Following our earlier assumption, we take the number of principal vectors to be a small fraction of the local data, p 0.02N 2 10.
500 samples. Assume a standard convolutional
Substituting these values into the denominator,
pF + 2Madd 10 256 + 2 5.4 x 10a 2,560 + 1.08 10 1.08 10
so the term 2.M dominates. Consequently,
KT Mraa
KT
R ~d
2M
2
21

---

Published as a conference paper at ICLR 2026
For representative clustered FL setting with K = 5 clusters and T 200 communication rounds, we obtain
5 200
R 2 = 500,
meaning that the cumulative iterative clustering overhead is roughly 500x larger than FEDDAG'S one-time truncated SVD plus warm-up cost per client.
These estimates indicate that, in the regimes we study (small p relative N. moderate K, tens to hundreds of rounds, and a realistic convolutional backbone such as SimpleCNN), the additional clustering overhead of iterative K- K-model evaluation in clustered FL baselines dominates FEDDAG's one-time truncated SVD plus warm-up cost.
A.6.2 ALTERNATIVE PRINCIPAL-VECTE COMPUTATION VIA CLASS-WISE SUBSAMPLING
Although the analysis above shows that the one-shot clustering overhead of FEDDAG is significantly smaller than the per-round clustering cost of iterative clustered FL methods, we can further reduce the overall cost and make FEDDAG's clustering overhead comparable to other non-clustering FL approaches by modifying how we compute principal vectors.
Subsampled principal vectors. In our original class-wise weighted data-similarity mechanism (Eq. 6 each client uses the full of samples Di.e for class e compute p principal vectors U via truncated SVD. In the alternative variant, each client instead constructs a class wise subsample D.. S Di.e of size Si,e. chosen according to its local computation and memory budget (e.g.. fixed per-class cap or fixed fraction of Dr.c)), and applies truncated SVD (Klema & Laub 1980) DT,,
to obtain p principal vectors Ú = [ūI..... upl. which replace U in the subsequent pipeline. The server then proceeds exactly in Eq. [6 computing principal angles and applying class-frequency weighting using U that the final similarity Vi, is obtained from subsampled data.
Extension to CC-graph construction. The same sampling idea can be applied when construct- ing the cluster complementarity graph (CC-graph) in $ Instead of using all available samples to compute the alignment scores T p.g.. between clusters p and q for class c, we compute p.g.e from class-wise subsamples drawn from the participating clusters. This further reduces the overall com- putation and communication overhead of FEDDAG.
A.7 PRIVACY CONSIDERATIONS
Privacy is a foundational aspect of federated learning, which aims enable collaborative model training while protecting the sensitive data of individual clients. In the context of FEDDAG, we ex- amine the privacy implications of both the similarity estimation and representation--saring phases. During client clustering, FEDDAG constructs weighted, class-wise data similarity matrix using a small set of raass-rpresenttttie principal vectors and per-class sample counts provided by each client. Crucially, the shared principal vectors are reduced linear combinations of local data and do not expose any raw samples or labels. Moreover, each client contributes fewer than 1% of such vectors per class, ensuring minimal data exposure. This approach aligns with prior privacy-aware clustering methods Vahidian et al. (202)), which also transmit low-dimensional representative vec- tors the server. In more privacy-sensttive deployments, additional protection mechanisms can be integrated into FEDDAG. For instance, secure aggregation protocols (Bonawitz et al.) 2017), encryption techniques, or differential privacy can be used to protect the shared principal vectors.
Privacy mechanisms can also be used to prevent leakage of the class-frequeet use when weighting similarity values. For example, FLIPS (Bhope et al.) (023) employs a trusted execution environment (TEE) to protect label distributions, which are used to select a diverse set of clients during FL training. Similarly, FEDDAG could employ differential privacy (DP) (Dwork) 2006) by perturbing class counts with carefully calibrated noise so that the contribution of any single example is statistically hidden. As another privacy-based option, FEDDAG could employ homomorphic encryption (HE) (Gentry 2009). allowing the server compute weights directly on ciphertexts without learning the raw values. However, designing and implementing such privacy mechanisms is beyond the scope of this work, so we do not pursue them further.
information that we
22

---

Published as a conference paper at ICLR 2026
Algorithm 2: Proximity Matrix Computation
Input: Principal vectors sparsified gradients Å elients
Output: A. proximity matrix between all elient pains
Function: Proximitytee mexiimittyattiia (U' 3))
for client 1, .... N and for client 1,
for class e = 1,..., -.....CC. C. do
Compute V" 3idy using Eq
Compute W 4.j.e using Fu
Compute Vi,, using E4
6 and
normalize(
Compute f using Eq. 5 and 9 (9r
Initialize weight vector
(WI...... WN)" E 1." randomly
while not comverged
10
Compute entropy loss f. using
11 w 4- w v w Cusi and w 4- clrp(w, O, 13// MLP-based update
12
Compute A as in Eq. 7 and return A..j
Algorithm 3: Clustering Threshold Search in FL
Input: Proximity matrix A..,, thresholdd Sa
Output: Optimal clustering (C1, .C...... Cz]
Function optinmalllll iiaaiiistteeeeeeeee
(A,, 8.):
Initialize emptyy list zecords
E S.
Generate candidate clustering C using bicrarchical clustering (HC) on A with threshold
Compute fi and Ca (Fq. 9 for C
Total clustering score (C ..)
ACa
tuple (o, fiup....Cg .Cg) records
with low score relatively small from records
return Optimal clustering [Cj...., Cz] ga
To further mitigate information leakage during gradient- similarity estimation, FEDDAG can adopt encryption strategies similar to those proposed in [Satterr et al. (2020). During cross-cluster feature sharing (see $4). when a cluster requests representations from a source cluster, only the aggregated gradients computed from the source cluster's clients are shared. No individual client's gradient information is exposed at any point.
A.8 GENERALIZATION TO NEWCOMERS - AL.OORITHM 4
High-level idea. In real-world FL systems, clients may join after the initial clustering and model training have already begun. Moreover, clients may not always remain continuously available. To handle such cases, we extend FEDDAG with lightweight mechanism that allows new clients to seamlessly join existing clusters without disrupting ongoing training. Specifically, each new client computes its data and gradient information, which are used extend the proximity matrix to include similarity values for the new client. This updated matrix is then used by the clustering algorithm to determine the appropriate cluster assignment. Once assigned, the client is integrated into the designated cluster without re-evaluating the optimal clustering retraining any previously learned weights.
Details of the method. The process for integrating new client knew is similar to that used for initial clients (as $3). FEDDAG first performs local training on inew 's data for lg rounds to reach partial convergence. Afterwards, client fnew computes its sparsified gradient update Afe and class-wiss wise principal vectors Uhen and sends them to the server. The server updates the existing data similarity matrix Vi,, and gradient similarity matrix Gi to their extended forms V.new and Gnew incorporating information from the new client. To combine the data and gradient, FEDDAG initially learns weight vector w (see $3.3). To integrate the new client, FEDDAG extends this process by assigning a weight W < [0, 1] and learning it using the same entropy loss (Eq. () used during initial training, but optimizing only for wi... without modifying existing weights. Using the extended weights Wnew> the proximity matrix A is computed based on nee Grvew and Wige. defined in Eq. in $3.3
Finally, the server reuses the previously selected clustering threshold a" (from Optimal Clustering in $3.4 and performs single hierarchical clustering (HC) pass Anw to assign knew to cluster Cz( After assignment, client inew initializes model from the corresponding cluster's global parameters and directly joins the existing FEDDAG training flow (i.e., the else branch at line 15 in
23

---

Published as a conference paper at ICLR 2026
Algorithm 4: Generalization to Newcomers
Input:
New client clustering threshold current clusters (Ci...., Cx current proxinity matrix A.. data malrix Vi.
gradient matrix Si.j
Outpot: Updated elient fazw models
Function Newcomer Integzat pattion ion (era a²))
Initialize client fwo with random 0
Set local counter Fizey 0
Tracks local wazm: up rounds
for cach global round E do
ifar <.
Local trnining 60 using local data (no federation)
bieww
if Sare: = then
Client huss sends i and U/eew to server
ExLend proximily
Server extends View
fanu.; GtcH to include the new elient
Server initializes Winn E (0, 1] and learns it using Eq
$ keeping existing weights fixed
Server extends proximity matrix AT, using Eig
with Viy and G:
/1 --- cluster aas gnment
Server exccutes hicrarchical clustering Client hes sets 0) or from (6')
Ai, to assign iew cluster Cx(fwe)
"(tnex)" 9 s(teee)
aggregate secondary encoders
related clusters Ivia
Client bas sets 025" - ((H(()(((()))))) 62/
else
// Standard training
(same else branch in Algorithm
Train (6) (E...)) via SGD using 17
Pelmarry training
Train 62 SGD using 20
secondary training
Breadeast updated (0tf 0r 025") server
This extension enables efficient onboarding of new clients by reusing the established clustering threshold and global models, avoiding disruption to ongoing training. The complete pro- cess is summarized in Algorithm [ Also, evaluate the generalization capability of FEDDAG to unseen elients through experiments reported in Appendix (B.6
Algorithm [].
A.9 HANDLING DATA-DISTRIBUTION SHIFT
High-level idea. After FEDDAG has converged, the data of already-clustered clients may still evolve over time (e.g., new sensor drifts, changes in user behavior). If the local distribution of client drifts too far from what its current cluster represents, the global model quality may degrade. We, therefore, add a mechanism that decides whether a client needs be re-evaluated for clus- ter assignment. In addition, to accommodate growing client population, FEDDAG periodically re-assesses the clustering to ensure the configuration remains consistent with the evolving client landscape. Specifically, the Wasserstein distance (Duan et al. 2021) is employed to track shifts in the class distribution of each client's local data over time; when a significant shift is detected, the system recomputes that client's 's data and gradient representations and re-evaluates its proximity to other clients using the same similarity fusion mechanism described in $3 This enables re-clustering of the client without disrupting other participants restarting global training.
Client re-evaluation. Let p!()) denote the empirical class histogram of client round 4. Every 8' rounds, we compute the W-Wasserstein distance" between the current and previous histograms as Wi(P(!) pp(l=")) Client i is marked as shifted if:
(P(P Pl
0.2
=
LabelSize Tii
(41)
where is the number of new samples processed by client i since round t-8'. 8. Eq. flags a shift when roughly 20% of local data has changed.
For image classification, treat classes discrete points the line O,..., C-1; the 1-Wasserstein distance then has closed form based cumulative histograms.
24

---

Published as a conference paper at ICLR 2026
A shifted client does not immediately trigger global re-clustering. Instead, its cluster assignment is re-evaluated through a procedure that re-computes data and gradient information for the server (sim- ilar to generalizing to newcomer clients in $A.8). For the next Eg rounds, the shifted client i trains its primary encoder and classifier on local data without federation so that the resulting gradients reflect its own distribution rather than the global model. After local training, the client computes its gradient update A', applies k-sparsification to obtain Á', re-computes class- principal vectors U from local data, and sends U and Á' to the server. These components update the data similar- ity matrix Vi.j. the gradient similarity matrix G.g. and the proximity matrix A.,j- The server then re-evaluates clients' cluster assignments by performing single hierarchical-cluseene updated A., using the fixed optimal threshold a (derived $3.4). If reassignment oceurs, the client initializes its model from the corresponding global model and continues training.
pass on the
Accommodating growing population. To support an expanding set of participants, FEDDAG periodically re-evaluates the clustering after a specified number of new clients have joined. This reassessment determines whether the updated client distribution warrants a change in the cluster structure. Concretely, FEDDAG re-runs the optimal clustering selection procedure by sweeping over candidate threshold values a (as in $34).. If a new clustering configuration is chosen, the al- gorithm updates the necessary components (e.g., the cluster complementarity graph, re-initializes the global model from client models) and resumes training, ensuring consistency with the evolving client landscape.
A. REASSESSING THE NOVELTY OF FEDDAG AND THE GRS MECHANISM
A core novelty of FEDDAG is its dual-encoder design for knowledge sharing among clusters via the proposed Global Representation Sharing (GRS) mechanism. At first glance, this architecture may appear similar to (i) personalized/shared head designs in personalized FI. and (ii) personal- ized/shared feature-extractor designs in feature-skewed FI.. Moreover, one may view FEDDAG's GRS mechanism as unnecessary overhead compared to the personalized//haared head baselines, thus requiring a clear justification for adopting this architecture. In this section, we reassess the nov- elty of FEDDAG relative to prior personalized FL approaches and explain why FEDDAG's GRS is fundamentally different from simply sharing single global encoder.
Comparison to traditional FL. Classical personalized FL often assumes that a single feature ex- tractor can be globally shared and only the classifier head needs to be personalized, which works well when client feature distributions are roughly aligned. heavily non-IID settings, however, clients are grouped into clusters with severe label skew and feature shift; in those cases, a shared encoder/feature extractor cannot satisfy both global and local needs, as different clients can have different learning objectives based on their data distribution. FedBR (Guo al., 2023) explicitly shows via visualization that feature extraction is not a purely global, universally shared job, as in, even for the same input, mocal-model and global-model features can differ significantly.
To tackle this generalization vs. personalization issue, a separate line of FL work maintains both local and global feature extractors adds supervisors/auxiliary encoders to relate them. For exam- ple, FedBR (Guo et al. 2023) uses a local and a global feature extractor to reduce classifier bias while preserving client-speciffe features, and FedSimSup (Liu al. 2025) lets each client hold two full models- a local supervisor and an inter-lcarning model where the supervisor aligns the inter-learning model with heterogeneous local data.
Comparison to other personalized/global FL methods. In summary, FEDDAG can be included in the family of methods that add an extra encoder to balance generalization and personalization. However, it leverages the clustered FL framework improve upon existing designs. While the idea of decoupling a model into two components is not new by itself, how these encoders are trained and what they represent in FEDDAG is fundamentally different. Most prior global/local feature schemes maintain: (i) Global part: shared global feature extractor, (ii) Personalized part: a per-client local feature extractor. How FEDDAG differs compared to above setitng is dfined below.
1. Global feature selective complementary features. In the above-mentioned pFL approach, sharing a single global feature extractor can be suboptimal. The global feature space is influenced by all clients, including:
25

---

Published as a conference paper at ICLR 2026
Clients with very different learning objectives data distributions, and
Clients with poor unstable features (e.g.. very few samples for certain classes).
This mixture can lead to negative and noisy feature transfer: aligning representation with all clients often harmful, especially when only subset of clients truly relevant complementary.
In contrast, FEDDAG does not simply blend all clients into one global representation. Instead, it:
Uses the CC-Graph to identify which clusters are actually beneficial, and
Imports features only from those complementary clusters via the secondary encoder.
In this sense, FEDDAG factorizes what would otherwise be monolithic global feature space into: (cluster-specific personal features) + (selected complementary features from other clusters), thereby avoiding contamination from unrelated comparatively less relevant client sources.
2. Secondary encoder trained directly on other clusters' data. In FEDDAG, the secondary encoder of the learner cluster is explicitly trained on the datasets of clients from source clusters:
The secondary encoder parameters sent source cluster identified by the CC-Graph.
At the source cluster, that encoder is locally trained on the clients' data, and the resulting gradients or updates are then sent back to the learner clusters.
Thus, the secondary encoder acts as a parameter carrier that travels across clusters, learns on other clusters' data, and then returns with updated gradient.
This fundamentally different from a single global feature extractor feature-skew FL. Those approaches typically rely on knraseege- signals such pseudo-data, distilled logits, feature embedding, or prototype features to gather global representation; to our knowledge, they do not perform explicit remote training of model component other clients' local data. This limits their ability to fully exploit the complementary feature structure present in other domains.
In summary, by leveraging the structure and training workflow of clustered federated learning, FED- DAG's secondary encoder performs targeted cross-clusted feature extraction guided by the CC- Graph, rather than functioning as just another shared personalized feature extractor.
A.11 DUAL-ENCODER ALTERNATIVE INITIALIZATION AND TRAINING
High-level idea. In our dual-encoder architecture, the goal is for the primary and secondary en- coders to capture complementary information. the main FEDDAG pipeline, this facilitated by initializing the primary encoder using partially converged gradients obtained during gradient-based similarity estimation. rather than initializing both encoders at random. Here, we investigate an al- ternative strategy to initialize encoders. regularizer. This variant modifies FEDDAG dual-encoder initialization and architecture as follows:
Both encoders in each cluster initialized randomly
The mrimary-encoder objective is augmented with regularization term that penalizes excessive alignment between the feature representations of the encoders.
All other components of FEDDAG (cluster-wise FedAvg, CC-Graph-guided enrichment, secondary-encoder updates, etc.) remain unchanged. This variant is described in detail below:
Details of the method. At first, FEDDAG initializes both the primary-encoder parameters ! and the secondary-encoder parameters @ randomly, as described above. Then, during the primary- encoder training phase (see Eq. 16), to explicitly encourage (:;) 01 and (:: to learn complementary representations rather than collapse to similar feature directions, we introduce a diversity regularizer Raiv that penalizes excessive alignment between their outputs, follows:
Rai (0 0%() E.ND, [Coos( (C (x:; 0,'), (l2) 02( )))
(42)
where cos(:.) :) denotes cosine similarity. Large cosine similarity indicates alignment between the encoders; minimizing cos therefore pushes the encoders toward complementary feature directions.
26

---

Published as a conference paper at ICLR 2026
The local objective, as shown in Eq. 16 after modification becomes:
(43) where Adiv > controls the strength of the diversity term. During local SGD, client i updates (0/5, of) using e ( 0f), while 6%/1 remains fixed and is updated later via the secondary- encoder enrichment step. The secondary-encodes training and all other components of FEDDAG remain unchanged. We also evaluate the effectiveness of this alternative encoder initialization em- pirically as shown in |B.8
( (f) = - (X,
(X,;
02(6); (x;;60000 09) + >av (a 02))
B ADDITIONAL EXPERIMENTS
In this section, we show implementation details, ablation studies, additional experiments regarding hyperparameter selection and sensitivity, and FEDDAG performance on different data distributions.
B.1 IMPLEMENTATION DETAILS
We now describe the implementation details used in our experiments, including model architectures and training hyperparameters. For datasets such as CIFAR-10 and SVHN, we adopt convolutional neural network (LeCun et al. 2002) composed three convolutional layers followed by two fully connected layers. For FMNIST, we use a simpler architecture with two convolutional layers and a single dense layer. Local training on each client is performed using stochastic gradient descent (SGD) with learning rate of 0.01, momentum of 0.5, weight decay of 1 x 10-4, and batch size of 64. Each client trains locally for 10 epochs per round. Unless stated otherwise, we run total of 200 global communication rounds, with 20% of clients sampled per round. We report classification performance using balanced accuracy, averaged across clients to account for non-lID data distributions.
B.2 HYPERPARAMETER TUNING
In the context FEDDAG, hyperparameters play crucial role in determining the model's perfor- mance, stability, and robustness. To better understand the effectiveness of FEDDAG, we investigate how sensitive the algorithm is to variations different hyperparameters.
Local Steps (tg). The parameter ty controls the number of local training epochs each client per- forms on its own data before sending gradient information the server. This step crucial for estimating each client's 's gradient direction, which is used to compute the gradient similarity matrix. Since this training done without any federation, the resulting gradients reflect only the client's local data. The choice of ly affects the trade-off between computation efficiency and the quality of similarity estimation. Ideally, we want ly to be as small as possible, while still enabling the gradients to converge enough to produce meaningful similarity measurements. Table shows how accuracy varies with different values of ty across datasets. these experiments, the gradient similarity ma- trix alone (instead of combining data and gradient) effectively gradient information captures client similarity. We also switch off dual-encoder and only use single encoder FEDDAG" (see $ The setup follows Data Distribution I (see $5) with a' 1, 30%, and consistent hyperparameter settings. Each communication round included 10 local training steps. As shown, increasing 'g improves accuracy initially, as longer local training leads to more stable and comparable gradients. However, accuracy plateaus around tg 2 for most datasets, indicating that the gradients have sufficiently converged for reliable similarity estimation. Beyond this point, additional local steps yield diminishing returns. Therefore, lg 2 provides good trade-off between accuracy and efficiency.
used as the proximity matrix to assess how
Weight Range for Data Similarity Matrix (8). The parameter 5 controls the sensitivity of the data similarity matrix to dataset size imbalance when comparing clients. Specifically, this weight- ing mechanism penalizes similarity scores between clients with large differences in dataset sizes, thereby reflecting the quantity shift more accurately. The impact of these size-based penalties is governed by the value of 6: smaller values result in minimal influence, while larger values increase the penalty's 's effect. Each computed similarity value reweighted and normalized into the range 6, + 8). with 5 [o, 1), allowing the final similarity score scale by most factor of two.
27

---

Published as a conference paper at ICLR 2026
Table 6: Test accuracy for different values local training rounds by (with 10 local steps per round) across datasets, evaluated under Data Distribution I with 30% class skew and Dirichlet o' 1.
t
CIFAR-10
SVHN
FMNIST
80.81
84.82:10.24
93.1810.11
90.05=:.116 03.34=0.522
93.1810.11
83.54
90.05:0.116
93.1810.11
Table report test accuracy for various values of 8 across multiple datasets. Since the weighting mechanism primarily addresses size disparity and quantity shift, examine its effect under the Dirichlet concentration factor: a' = 0.25 (severe shift). To isolate the effect different values of on accuracy, we use only the data similarity matrix (instead of combining data and gradient similar- ity) when computing the proximity matrix for clustering. These experiments follow the Data Dis- tribution I described in $5 using identical hyperparameters. From Table 7 we observe that higher 5 values (e.g., 0.6) can lead to improved clustering and accuracy. This suggests that the weight- ing scheme particularly beneficial in highly heterogeneous environments, where accounting for dataset size differences enhances similarity estimation.
Table 7: Accuracy metrics for various values of weight range å under Data Distribution (Dirichlet a' 0.25, 20% class skew).
8
CIFAR-10
SVHN
FMNIST
0.2
87.51=00.18
91.74+00.08
91.79-+0.08
0.4
87.5110.18
91.74:10.08
92.2110.09
0.6
87.9510.113
91.91+0.13
92,21
0.8
87.9510.113
91.91-10.13
92.2110.099
1.0
87.95+0.113
91.91
92.21
Top-k values in CC-Graph. The parameter k determines how many top-ranked clusters are selected as knowledge sources for each target cluster in the complementarity graph H. This graph guides which clusters will supply feature representations to others during the secondary encoder training phase. For every row in the complementarity score matrix (Eq. [2). only the top-k highest scor- ing entries retained to form directed edges. A smaller k limits each cluster to fewer sources, possibly reducing noise but also restricting diversity. In contrast, a larger A increases the opportu- nities for learning from complementary clusters but may include low-quality connections that dilute representation quality.
Table & how varying the number of source clusters k in the complementarity graph H affects the performance of FEDDAG. Experiments are conducted under Data Distribution with Dirichlet concentration parameter a' and 30% label skew, while keeping all other hyperparameters fixed. We observe that performance generally improves when k: 2 2, benefiting from knowledge transfer across multiple relevant clusters. In some cases, increasing k beyond 2 continues help (e.g., FMNIST), while in others (e.g., SVHN), it leads to marginal drops in accuracy. This suggests that the optimal value of k depends on the dataset characteristics and the number of clusters in the current formation. We adopt h = 2 as balanced choice to ensure diversity while maintaining relevance.
Table 8: Accuracy for different values of top-k retained in the complementarity graph H under Data Distribution (Dirichlet o' 30% class skew).
CIFAR-10
SVHN
EMNIST
90.85+H0.13
97.09+0000
97.71+00.05
91.02+-0.12
97.19+0.004
98.36+0.10
90.9510.116
97.0110.07
98.5710.08
90.96:10.21
96.7810.11
98.4110.11
Sparsified Gradient Á From Section 3.1 that, after short local warm-up, each client i computes gradient update A on its local data and transmits k-ssarsified version À' to the server. Here, À retains only a small random subset of coordinates of A', and the gradient sparsification ratio refers to the fraction of coordinates that are kept in A when constructing the gradient similarity matrix in Eq. 3 In this subsection, we study how the final test accuracy varies change this sparsification
28

---

Published as a conference paper at ICLR 2026
ratio. To isolate the effect of sparsification, we use only gradient- based similarity (no data-based similarity) when forming the client similarity matrix. We also switch off dual-encoder and only use single encoder FEDDAG* (see S The experimental setup matches
main configuration for Data Distribution we use 30% label-skew with Dirichlet a' = 1, fix the local warm-up to lg = rounds with 10 local steps per round, and vary the fraction of coordinates retained in each Á
Table 9: Test accuracy (%) as function of the gradient sparsification ratio with only gradient similarity and singel encoder (fraction of coordinates retained in ! under Data Distribution I (30% label skew, a' 1, tg 2).
Sparsity (entries kept)
CIFAR-10
SVHN
FMNIST
0.1%
82.52 0.60
89.23 0.25
92.81 + 0.14
0.5%
83.34 0.52
90.05 +.15
93.18
1%
83.34 0.52
93.18 0016
2%
83.34 0.52
0.16
0
20%
83.34 + 0.52
90.05
93.18 0.11
Table 9 reports the test accuracy as function of the sparsification ratio (percentage of coordinates kept in Á'). We observe that keeping little as 0.5% of the gradient coordinates in Á' is already sufficient to obtain essentially the same accuracy much denser settings. For sparsification ratios at or above 0.5%, the resulting gradient- similarities are very similar, leading to almost identical clustering structure and final performance. In contrast, at 0.1% sparsity, the similarities become noisier, slightly degrading clustering quality and accuracy. Based on these results, in our main experiments we choose sparsification ratios in the range of 1-2% for À', which provides a good trade-off between communication efficiency and clustering quality.
B.3 EXPERIMENTS ON DATA DISTRIBUTION
Exp I: Performance Evaluation. This section presents the additional results referenced in the main paper for a' = 1, under class skew p = 20% and 30,, following the setup described in $5 The results, shown in Table IO, further validate the effectiveness of FEDDAG on Data Distribution under moderate quantity shift. The same set of baselines used, and results are reported across all four datasets.
Table 10: Exp I: Performance comparison for Data Distribution I with 20% and 30% non-lID label skew under low quantity shift (Dirichlet a' 1).
Skew
Algorithm
CIFAR-10
FMNIST
SVIN
CHEAR-I0
CIFAR-I0
PMNIST
SVTIN
CIFAR-I00
FedAvg
52.12
0.30
14.61
0.36
51.34
71.17
53.13
BodProx
46.77 = 0.14
5681
= 0.16
71.23
2: 0.45
53.38 - 0.86
57,80
1 0.23
73.87 = 0.25
69.65 - 0.19
53.97 = 0.85
PerFedAvg
84.68 0.19
91.18
0.21
92.34
0.13
69.43
+:
82,83 + 0.14
94.74 0.11
91.48 +: 0.29
60.70
FedSoft
77.42 0.21
87.64
90.48
0.24
65.98 0.37
76.94
0.38
89.56 0.37
84.86
0.45
56.61 0031
PACFL
90.45 0.30
94.41=031
0311
94.96000
0.12
70.35 0.36
0.38 87.01
91.28
0.22
94.36 0.19
63.1 0.76
0.66
71.85
85.67
CFL-GP
87.83 0.19
91.45
1155-022 0.27
90.38
0.16
69:73 0.20
85.67
0.25
96.82 0.24
92.29 + 0.09
61.24 = 0.73
FedGWC
93.56
0.09
93.67
0.13
72.75
+
86.18
0.25
92.94
0.19
61.35 0.43
FedRC
76.12 - 0.28
86.45 0.42
89.22
0.31
64.78 = 0.33
75.12 0.31
91.02 - 0.44
83.67
0.38
57,89
IRCA
89.68 + 0.17
94.02
0.09
93.28
0.13
72.86
+ 0.29
86.42 + 0.25
96.61 4 0.14
92.86 + 0.19
61.34 0.43
FEDDAG
Exp 6: Convergence Under Limited Communication rounds. We compare the performance of FEDDAG against SOTA baselines under constrained communication budget of 80 rounds. Fig- ure reports the final local test accuracy versus the number of communication rounds for four datasets. The results demonstrate that FEDDAG consistently converges within 20 to 30 communi- cation rounds, outperforming all other methods in both convergence speed and final accuracy.
B.4 DATA DISTRIBUTION IV
This distribution evaluates FEDDAG under a combination of feature skew and label skew. To sim- ulate feature skew, we follow an approach similar to FedRC (Guo al. 2024), which leverages
29

---

Published as a conference paper at ICLR 2026
100
80
60
20
20 40 66
70
92
Commurication Roundd
(a) CIFAR-10
(b) FMNIST
(c) SVHN
(d) CIFAR-100
Figure 3: Exp Accuracy vs. number of comm. rounds, Data Distribution non-III (30%), a/=1.
datasets (c.g., CIFAR-IO-C) that apply diverse image corruptions or use domains, thereby introduc- ing different feature styles. To simulate label skew, adopt the LDA method (Hsu et al. 2019). Specifically, each client is assigned one of the available corruption types (e.g., fog, contrast, etc. for CIFAR-10-C) domains (e.g., cartoon, photo, etc. for PACS) to create feature skew, and the samples are distributed using the Dirichlet factor a' 1.
Table 11: Exp 8: Performance comparison of various SOTA algorithms and FEDDAG under com- bined feature skew and label skew (Data Distribution IV). Feature skew arises from corruption types on CIFAR-10-C and TINY IMAGENET-C, and from domains PACS and Office-Caltech-10
Algorithm
CIFAR-10-C
TINY IMAGENET-C
PACS
Office-CaltcchII
FedAvg
30.73 + 0.36
18.43 = 0.43
38.47 + 0.28
46.12 0.31
PerFedAvg
60.39 0.13
25.54 = 0.31
70.82 + 0.22
67.34 0.23
FBLG
60.25 + 0.30
29.91 = 0.23
72.38 1 0.37
68.44 0.14
FedBR
61.09 + 0.32
30.83 = 0.25
73.83 + 0.32
71.37 + 0.18
FedMix
60.11 + 0.14
30.71 = 0.24
73.25 + 0.41
70.56 0.23
PACFL
63.62 + 0.22
33.53 = 0.38
76.63 + 0.19
73.31 0.14
CFL
59.48 0.15
28.97 = 0.26
72.14 1 0.25
68.02 0.19
FedRC
61.82 + 0.21
32.14 = 0.19
76,28 + 0.17
73.54 0.22
IFCA
62.52 + 0.39
32.33 = 0.19
75.12 + 0.21
72.48 + 0.29
FEDDAG
65.62 + 0.31
36.27 = 0.32
80.34 + 0.27
76.28 0.20
Dataset. We use four datasets for this task in the FI. setting: CIFAR-10-C, TINY IMAGENET- (Hendrycks & Dietterich] 2219), PACS et al. (2017), and Seatee-Caattch-11GGGG et al. 2014 thereby covering both corruption- based and domain-leve feature shift.
Exp 8: Performance under Feature Skew. We evaluate the performance of SOTA algorithms and FEDDAG on different datasets under combination of feature skew and label skew. Each client is randomly assigned one of the 20 available corruption types (For CIFAR-IO-C and TINY IMA- GENET-C),
of the four available domain types (for PACS and Office-Caltech-100 cases, the samples are distributed using a Dirichlet concentration factor a' 1. We randomly select 80% of clients for training and keep the remaining 20% unseen clients, reporting test accuracy on these held-out clients using the final trained models. To more directly compare against approaches specialized for quantity also include the dedicated baseline FBL. Gu et al. (2024b), which employs client-selection strat- egy that prioritizes clients with larger local datasets while grouping clients with similar sizes. The results in Table [1 show that FEDDAG consistently achieves higher accuracy than the base- line methods. This improvement is attributed to FEDDAG's 's data-based similarity metric, which provides more accurate feature similarity estimation compared to existing approaches. Across all four feature-sker skew benchmarks, FED achieves the best performance, providing direct empirical evidence of its robustness feature distribution shift combined with label skew.
In both
naturally arises this Dirichlet-based sample allocation we
30

---

Published as a conference paper at ICLR 2026
B.5 EXPERIMENT ON LARGE-SCALL WERLD DATASET
To evaluate FEDDAG in a large-scale, real-world setting, we additionally evaluate it on the Google Landmarks dataset Weyand et al. (2020), following the setup of Licciardi al. (2025). Specifically, we consider the I Landmarks- Uees- T60K partition, where the dataset is partitioned into 1,000 clients based on the landmark dataset's authorship information. All other aspects of the experimental setup are kept the same as in our Data Distribution I experiments in Section 5. We compare FEDDAG against a group of established FL. baselines, and the results are reported in Table T2
Table 12: Performance on the large-scale real-world Google Landmarks dataset.
Dataset
FedDAG Google Landmarks 36.53 E 0.21 4..7 45.29 0.28 51.51 E 0.31 51.97 1.16 58.23 1 0.15
FedArg
PACEL
CFL
FedGWC
IFCA
B.6 EXPERIMENT ON GENERALIZATION TO NEWCOMERS
To assess the ability FEDDAG to generalize to unseen clients (see Appendix (A8)). we simulate a dynamic FL environment using Data Distribution I with 30% label skew and Dirichlet concentration factor = Initially, training is performed on 80 out 100 clients for 80 communication rounds, following the standard FEDDAG procedure. At the end of this phase, the remaining 20 clients join the system as newcomers. Each newcomer executes steps (1-15) of Algorithm and is assigned cluster. Once assigned, the client receives the current global model from its designated cluster and personalizes it for round (10 local epochs). To evaluate model quality, we report the average final test accuracy of the 20 newcomers across different datasets. As shown in Table [3] FEDDAG achieves better generalization to newcomers than competing methods. This improvement attributed its robust cluster assignment and generalization strategy for new clients.
Table 13: Test accuracy of newcomer clients, Data Distribution with 30% label skew and a'
Algorithm
CIFAR-10
FMNIST
SVHN
FedAvg
55.38:10.15
74.93110.22
66.86110.28
PerFedAvg
80.92-10.10
92.62-10.17
90.00:10.15
FedSoft
74.98+10.27
87.45-0.22
83.59:1.12
PACFL
85.3310.15
95.1710.24
92.76110.08
CFL
69.97:0.11
83.6410.13
78.94
FedGWC
84.3010.13
94.5310.10
91.5611006
FedRC
73.36+00.26
88.91+-0.21
82.36+0.112
IFCA
84.55+00.2
94.61+0.30
91.58-0.22
FEDDAG
88.23+L0.18
96.84+0.23
95.74+-013
B.7 ADDITIONAL ABLATION STUDIES
To observe the contribution of different components and their behavior under different non-IID set- tings, we perform a set of targeted ablations that isolate each module (combining data and gradient, adaptive optimal clustering, and dual-encoder representation sharing).
B.7.1 COMBINING DATA- AND GRADIENT-BASER SIMILARITY
To isolate the contribution of combining data- and gradient- based information for similarity compu- tation, we perform an ablation study where, instead of using the combined similarity, we construct the client similarity matrix using either gradient-only or data-only similarity. We run this ablation on the FEDDAG single-encoder variant (no dual-encoder sharing), so that performance differences can be attributed directly to the similarity design without interference from other techniques. Con- cretely, we compare: FEDDAG* -rrd (gradient-only similarity), F larity), and FEDDAG* -DoA**--aa-te -Data+Grad (our combined similarity). We evaluate on CIFAR- 10 and SVHN under two different non-IID data distributions: (i) Data Distribution with 30% label skew and high quantity shift (Dirichlet& a' = 0.25), and (ii) Data Distribution with concept shift (see Section 5). The results for these two settings are reported in Table 14 and Table 15 respectively. These results
(data-only simi-
31

---

Published as a conference paper at ICLR 2026
consistently show that weighted data-based similarity (FEDDAG"-Datl outperforms gradient-oniy similarity, and that combining data- and gradient-basee similarity (FEDDAG* BEEDDAG--Dttt-Grral -Dat++Grad) yields the best performance in both label+quantity-kkee skee and concept-shift settings.
Table 14: Ablation on combining data and gradient under Data Distribution I (label skew 30%, Dirichlet a' = 0.25).
Method
CIFAR-I0 83.79 + 0.50
FMNIST
SVHN 89.05 + 0.19
FEDDAG -Grad (gradient-only)
91.26 + 0.38
FEDDAG - -Data (data-only)
85.03 + 0.33
91.82 0.25
89.52
FEDDAG -Data+Grad (combined)
86.95 - 0.21
92.18 0.15
90.97 0.13
Table 13: Ablation on similarity design under Data Distribution (concept shift).
Method
CIFAR-10
FMNIST
SVHN
FEDDAG -Grad (gradient-only)
64.52 + 0.45
83.88 + 0.32
81.17 0.21
FEDDAG" -Dta (data-only)
67.10 + 0.23
85.74 0.28
FEDDAG' -Data+Grad (combined)
67.79 = 0.27
86.03 + 0.21
83.73
B.7.2 DUAL-ENCODER REPRESENTATION SHARING
Here, we examine whether the accuracy gains from inter-cluster global representation sharing (GRS) via the dual-encoder architecture (see $ arise from genuine feature enrichment or simply from increased model capacity. We already performed this ablation on Data Distribution (20% label skew, Dirichlet a' = 0.25; see Table 3 in $ To further observe the behavior of dual-encoder sharing under a different non-IID regime, we repeat this ablation the concept-shiff setting (see Data Distribution II in S Using the same three variants of FEDDAG as in Table 3 we obtain the following results as shown in Table [6] Similar to the experiment under Data Distribution (label- skew setting), full FEDDAG achieves clear accuracy gains over both the single-encoder (FEDDAG") and the no-sharing dual-encoder (FEDDAGt) variants across all three datasets under concept shift. This indicates that cross-cluster representation sharing remains beneficial even when the primary challenge a mismatch in local decision boundaries rather than pure label skew.
Table 16: Ablation of cross-cluster representation sharing under concept shift (Data Distribution II in $ comparing the single-encoder baseline (FEDDAG"), dual-encoder variant without cross- cluster sharing (FEDDAG)), and full FEDDAG.
Algorithm
CIFAR-10
FMNIST
SVHN
FEDDAG" (single encoder)
67.79110.27
86.033:0.21
83.73
FEDDAG (dual encoder, no sharing)
67.51+0.22
85.91+0.155 69.13110.3
83.65:0..24 88.7911.199
FEDDAG (dual encoder + sharing)
85.06:10.26
B.8 EVALUATION OF THE ALTERNATIVE DUAL-ENCODER INITIALIZATION
We empirically evaluate the alternative dual-encoder initialization and egy described in $A.11 In this variant, both encoder parameter sets are initialized randomly, and the local primary-encoder objective is augmented with the diversity regularizer to encourage com- plementary feature extraction. To assess its effectiveness, we compare this variant against the warm- start initialization used in the main FEDDAG pipeline.
strat-
Experiment setup and results. We evaluate on CIFAR-10 and SVHN under the same 20% label- skew configuration with Dirichlet parameter a' 1.0 used Data Distribution (Section 5). Each experiment is repeated across three random seeds, and report the mean and standard deviation of the final test accuracy. The (warm-start) version of FEDDAG on both datasets, with slight improvements in certain cases. These results suggest that while warm-start initialization is effective, the random- init + regularization ap- proach also provides competitive- and in some settings slightly stronger alternative.
variant achieves accuracy comparable to the main
32

---

Published as a conference paper at ICLR 2026
10
Nur Number of clusters
Numbel of duises
humbod chisters
Curte tog loo
92
900 10.00 50
Distance Bnreehold FM
023322034400 02833465653700
Datence Bhrenood
65633708610
00010025 04000566003333
(a) CIFAR 10, loss and cluster count vs. a using only.
(b) CIFAR-10, loss
(c) SVHN, loss and cluster count vs. a using L1 only.
(d) SVHN, loss and cluster count vs. a using L1+AC2.
cluster count vs. using
+12
Figure 4: Effect the federated-awant clustering loss on the adaptive clustering mechanism under Data Distribution (30% label skew, Dirichlet a' 0.25) for CIFAR-10 and SVHN
Table 17: Comparison of dual-encoder initialization strategies in FRDDAG (Data Distribution I).
Encoder Initialization Method
CIFAR-10
FMNIST
SVHN
Main wrarm-start) (see $4
90.76 0.12
93.82 0.20
93.91 + 0.23
Random init-rreuularizer A.I1
90.93 0.16
93.76 0.17
93.86 + 0.10
B.9 SECONDARY- ENCODEE SCHEDULING UNDER RESOURCE CONSTRAINTS
The dual-encoder design in FEDDAG doubles the number of encoder parameters compared to single-encoder model and introduces extra communication for secondary-encoder updates. While ablation on FEDDAG' (single encoder) (see 5 shows that the dual-encoder architecture is ben- eficial for robustness, is important to understand the trade-off between accuracy and the additional computation/comit computatio/coomunicattioo overhead, especially for conorre-conntraineed edge devices.
To this end, we evaluate lighter- weight training schedules that throttle eecondary-encodeee updates to reduce computation overhead.. Let K:11 schedule denote pattern where perform K' rounds of standard FEDDAG primary training (updating the primary encoder and classifier) followed by one secondary-encoder enrichment round (updating only the secondary encoder). The original FEDDAG corresponds to updating both encoders every round, i.c., no throttling of secondary updates.
We evaluate four primaryssecondary scheduling patterns-1:: 1:1, 5:1, 10:1, and computeecoommunication budget of 80 communication rounds. All experiments are conducted on CIFAR-10 and SVHN under the same heterogeneous setting Data Distribution (30% label skew, Dirichlet =025.. The results are summarized in Table I8
fixed
Table 18: Accuracy of FEDDAG under different primary:secondary training scheduling patterns (Data Distribution 80 comm. rounds). "Full FEDDAG" updates both encoders every round.
Schedule (Primary: Secondary)
CIFAR-10
SVHN
Full FEDDAG (both)
89.87 1 0.19
92.65 + 0.11
1:1
80.34 1 0.44
86.26 1 0.32
5:
87.65 0.28
91.31 0.22
10:
87.25 0.31
91.46 0.20
15:
86.54 + 0.35
90.87 + 0.27
The 1:1 schedule significantly underperforms because the primary encoder is updated only every other round and thus remains under-trained. In contrast, the 5:1 schedule provides the best trade-off:: reduces the number secondary-encoder updates by 80% while maintaining accuracy close to the full FEDDAG model on both datasets. The 10:1 and 15:1 schedules further reduce the number of secondary updates but incur slightly larger accuracy drop. Overall, these results show that modest throttling of secondary-encoder updates (e.g., 5:1) can substantially reduce the effective compute and communication devoted to the secondary encoder while preserving most of the dual- encoder performance gains.
33

---

Published as a conference paper at ICLR 2026
B.10 EVALUATING THE OPTIMAL CLUSTERING MECHANISM
To understand the efficacy of the optimal clustering mechanism (see $3.4) in FEDDAG, we examine its behavior when the number of true underlying data distributions large. To do this, design an experiment where the inherent number of clusters is intentionally high. The experiment is set up and performed follows:
Dataset setup. We use the SVHN and CIFAR-I0 datasets to construct federated setting with an inherendl large number of ground-truth clusters. Each dataset has 10 classes; any pair of classes defines possible two-class distribution, yielding total of (22) 45 distinct two-class combina- tions. From these 45 possibilities, we select // distinct two-class combinations to create a controlled setting with 11 ground-truth clusters (e.g., class pairs (1, 3), (2, ctc.).
Each client is assigned exactly one of these 11 two-class combinations. For each chosen pair of classes, we collect all corresponding samples and distribute them across the assigned clients using Dirichlet sampler, which introduces within-cluster heterogeneity while preserving the underlying two-class structure. As a result, each client contains data from exactly two classes, while the over- all population spans 11 distinct underlying distributions. This setting allows to test whether FEDDAG can automatically discover a relatively large number of true clusters.
Experiment on optimal clustering procedure. After FEDDAG computes adjacency matrix, we hierarchical clustering from & 1.0 and decrease a in steps of 0.05. For each value of a, we record (i) the resulting number of clusters and (ii) the corresponding clustering loss Lc. The results for optimal clustering on CIFAR-10 and SVHN are summarized jointly in Table 19 and visualized in F
grid of distance and evaluate clustering loss Lc. We start
Table 19: Number of clusters and clustering loss Le as a function of the distance threshold for CIFAR-10 and SVHN in the inherent high-cluster (i.e., 11) distribution setting.
CIFAR-10
SVHN
CIFAR-10
SVHN
#clusters
Cc
#clusters
Le
#clusters
Le
#clusters
Le
1.000
0.770
0.675
0.500
0.149
5
1.000
0.950
0.770
0.675
0.450
0.149
0.896
0.900
2
1.000
0.675
0,400
11
0.149
0.688 9
0.850
3
1.000
0.675
0.350
11
0.149
0.343 11
0.800
3
1,000
0.866
0.300
0.149
0.343 11
0.750
1,000
0,866
0.250
0.149
0.343 11
0.700
4
1,000
3
1.000
0,200
12
0.251
0,343 11
0.650
4
1,000
4
1.000
0.150
13
0.161
0.315 14
0,600
5
1,000
1.000
0.100
14
0.117
18
0.205
0.550
1.000
1.000
0.050
0.117
24
0.141
14
1.0
25
13
Number of clusters
clusters
Clsstrrigg loss
13
Clustering loss
0.8
0.8
10
0.2
2.0.10.2.33.40050..00 00..5000......9.11111.
0.0.0....00..........0.... Distance threshold (x)
Distance threshold (a)
(a) CIFAR-10: #clusters and Le vs. a
(b) SVHN: #clusters and Le vs. a
Figure 5: Behavior of FEDDAG's adaptive clustering mechanism a setting with inherent high number of clusters (e.g.. > 10) ground-truth distributions.
For CIFAR-10, as a decreases from 1.0, the clustering initially remains extremely coarse (between and clusters), and the loss stays high and saturated 1.0, reflecting severe under-clustering, where
34

---

Published as a conference paper at ICLR 2026
many heterogeneous clients are incorrectly merged. A clear transition occurs around 0.50: the number clusters jumps to 11, and the loss drops sharply from 1.0 to approximately 0.15. Impor- tantly, this !1-cluster solution forms stable plateau across a wide threshold range E [0.25, 0.50), with both the cluster count and the loss remaining effectively unchanged. Lowering a below 0.25 further fragments clusters into 12-14 smaller groups, but the loss only improves marginally (from a 0.15 to 2 0.12). We interpret this as over-segmentattion rather than revealing additional mean- ingful structure. In contrast, the 11-cluster configuration dominates across broad a interval and matches the true number of underlying distributions. For SVHN, we observe a similar trend. At large (between 0.85 and 1.0) the solution clearly under-clustered (1-2 clusters) with high loss. As decreases, an 11-cluster configuration emerges and remains stable for E [0.20, 0.35] with loss around 0.34. Pushing a below 0.20 further splits clusters (14-24 clusters) and only slightly reduces the loss (down to x 0.14), again indicating over-segmentation rather than meaningful clusters.
Overall, these experiments show that FEDDAG's adaptive clustering mechanism scales effectively to scenarios with more than 10 true distributions on both CIFAR- 10 and SVHN, and can recover the correct number of underlying clusters (here, K* 11) without ever hard-coding into FEDDAG.
35


Reviews:
[{"review_id": "RyFKpjnwMo", "reviewer": "Reviewer_Lm4p", "summary": "FedDAG proposed a new clustered FL algorithm, designed to tackle data heterogeneity. Specifically, first, to improve clustering, FedDAG introduces a hybrid similarity evaluation by combining both data and gradient similarity, and then obtains a novel federated-aware metric to evaluate candidate clusters, enabling the framework to automatically determine the optimal number of clusters. Second, to enhance global representation sharing, FedDAG employs a dual-encoder architecture. This design allows the model to learn both intra-cluster and inter-cluster representation simultaneously during training, which further increases FedDAG’s performance.", "questions": "see the above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. FedDAG can address label skew, feature skew, concept shift, and quantity shift simultaneously. This comprehensive approach is rare in Federated Learning. \n2. FedDAG can adaptively determine the optimal number of clusters. This mechanism effectively solves a long-standing practical issues where most methods require setting the number of clusters in advance. It is practical to handle dynamic scenarios, such as the arrival of new clients (Appendix A.7) and data distribution shifts over time (Appendix A.8) .\n3. The proposed global representation sharing which is implemented via a dual-encoder architecture and a novel CC-Graph is a key innovation. By considering the \"supply\" and \"demand\" of class representations based on data rarity, this mechanism breaks the traditional information silos of clustered FL, allowing clusters to learn from complementary data sources across the entire network for the first time.", "weaknesses": "1. FedDAG has large computational overhead due to its complex design, involving an intricate multi-step process (e.g., SVD, clustering, CC-Graph, dual-encoder training, MLP optimization,). Specifically, for computational cost, server-side operations are at least O(N^2) for computing the similarity matrix and clustering, making scalability beyond the 100 clients tested questionable. For communication overhead, the dual-encoder training phase appears to double the communication cost compared to FedAvg, and the proposed mitigation (alternating training) is not clearly presented in the main algorithm.\n\n2. The \"category frequency weighting\" in formulas (5) and (6) is specifically designed to address the issue of \"quantity shift\". Directly uploading category frequency is a serious privacy issue, as it completely exposes the label distribution of the client. Although the author acknowledges these issues in Appendix A.6 and proposes that “uniform weighting can be employed in place of class-frequency-based weighting of similarity values to prevent leakage of class distribution information.”, this solution logically completely denies one of the core contributions of the paper.\n\n3. The concept of CC-Graph is novel, but its metric function is overly simplistic. It merely reflects differences in the ranking of numbers of samples belong to the same class among clients, but cannot capture disparities in quantity or quality of samples.\n\n4. The design of the GRS mechanism is also open to debate. Traditional personalized federated learning treats the feature extraction layer as an aggregatable part of the whole parameters while preserving the personalization of the classifier layer. This design assumes feature extraction is globally similar across different datasets and different tasks. The dual-encoder architecture proposed in this paper introduces an independent global feature extractor. It concatenates representations from global extractor($\\theta^{2f}$) with those from the cluster feature extractor($\\theta^{1f}$) before feeding them to the downstream classification layer. This design lacks compelling justification and instead incurs significant communication overhead. More comparative experimental results and analysis should be included.\n\n5. There is a significant disconnect between the paper's theoretical guarantees and the algorithm's practical operation. The convergence analysis in Appendix A.4 critically relies on Assumption A.4 (Stable clustering). This assumption is directly contradicted by the FedDAG algorithm's own design, which includes a dynamic phase to determine clusters (Algorithm 1, Lines 1-14) and further mechanisms to change cluster assignments in response to newcomers (Appendix A.7) or distribution shifts (Appendix A.8). Consequently, the provided theory only applies to a static scenario and fails to model the algorithm's core dynamic behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bTjfP9lZI7", "reviewer": "Reviewer_uk8Q", "summary": "This paper proposes a clustered federated learning framework, termed FedDAG, which is designed to address performance degradation under heterogeneous (non-IID) client data. Focusing on different non-IID scenarios, FedDAG applied a learnable weight combination for data and gradient similarities integration.  It also includes an adaptive clustering metric to automatically determine the optimal number of clusters. In addition, FedDAG employs a dual-encoder architecture for cross-cluster representation sharing. Experiments across four types of non-IID scenarios (label skew, feature skew, concept shift, and quantity shift) show consistent improvements over several baselines, including PACFL, IFCA, and FedRC", "questions": "1. Can the authors provide theoretical convergence or complexity analysis for FedDAG in different non-IID cases?\n\n2. How do the different modules (data weighting, gradient sparsification, adaptive clustering) behave under various types of non-IID settings?\n\n3. What is the computational and communication overhead compared to PACFL or IFCA?\n\n4. Can the authors test FedDAG on other modalities or large datasets?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.  The paper clearly defines different types of non-IID heterogeneity and systematically designs corresponding modules to treat each.\n\n2. The framework integrates clustering, similarity measurement, and inter-cluster knowledge transfer into a cohesive system.\n\n3. FedDAG performs well across datasets and heterogeneity settings, demonstrating robustness in practice.\n\n4. Presentation is clear and easy to follow.", "weaknesses": "1. The novelty of this work is limited. Most components extend existing approaches with minor tweaks. The framework lacks a central new idea or theoretical insight.\n\n2. There is no convergence, optimality, or complexity analysis; the method’s robustness under different heterogeneity settings is justified only empirically.\n\n3. The ablation studies are incomplete.\n\n- The sensitivity of the gradient similarity module to the number of local optimization steps and the sparsification ratio is not examined, leaving uncertainty about the balance between communication efficiency and clustering accuracy.\n\n- The data similarity module includes both class-frequency weighting and entropy-based weighting for fusion, yet their individual effects are not isolated or discussed.\n\n- The paper does not verify whether each designed module actually mitigates the corresponding non-IID type (e.g., whether the class-wise weighting improves label skew, or whether the entropy weighting helps with concept shift). This makes it difficult to assess whether the proposed design behaves as intended.\n\n4. All core experiments are conducted on relatively small and clean benchmark datasets (CIFAR-10/100, FMNIST, and SVHN). Although the appendix includes an additional comparison on Tiny-ImageNet, this test remains limited in scope and scale and is insufficient to demonstrate robustness or scalability to more realistic, large-scale federated settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XdMMcPaKLu", "reviewer": "Reviewer_VAQF", "summary": "This paper proposes FedDAG, a novel clustered Federated Learning (FL) algorithm designed to handle diverse types of data heterogeneity, including label skew, feature skew, concept shift, and quantity shift. The core contributions are threefold: (1) a client similarity metric that fuses both data-based (via a weighted, class-wise principal angle comparison) and gradient-based information, with client-specific weights learned via an entropy loss; (2) a dual-encoder architecture that enables cross-cluster representation sharing to enrich feature learning without sacrificing cluster specialization; and (3) an adaptive clustering mechanism that automatically determines the optimal number of clusters using a novel federated-aware metric that penalizes over-splitting. Extensive experiments on multiple datasets and non-IID settings demonstrate that FedDAG consistently outperforms a wide range of state-of-the-art clustered and personalized FL baselines.", "questions": "- How does the performance of FedDAG scale with an increasing number of clusters? The current experiments show a comparison against methods with a pre-set or automatically determined K. It would be insightful to see how FedDAG's adaptive mechanism performs when the inherent number of true data distributions in the client population is large (e.g., >10).\n\n- In the dual-encoder training, the primary encoder is initialized from the locally pre-trained models to avoid redundancy. Was any other initialization strategy explored? Could randomly initializing both and using a regularization term to encourage diversity be a viable alternative?\n\n- The authors need introduce some more related works based on clustering or data similarity:\n\n   - Collaborative learning by detecting collaboration partners\n   - Personalized Federated Learning under Local Supervision\n   - Benchmarking data heterogeneity evaluation approaches for personalized federated learning,", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive Problem Formulation: The paper does an excellent job of identifying and articulating the key limitations of existing clustered FL methods, such as reliance on a single similarity modality (data or gradients), restricted intra-cluster knowledge sharing, and inability to handle all forms of data skew. The motivation is clear and well-justified.\n\n- Novelty and Technical Sophistication: The proposed method is technically sound and introduces several novel ideas. The fusion of data and gradient similarities is not merely a weighted average but involves a learned, client-specific weighting scheme. The class-wise, weighted data similarity is a clear and meaningful improvement over prior work like PACFL. The dual-encoder architecture for cross-cluster sharing is a creative and well-motivated solution to the problem of isolated clusters.\n\n- Thorough Empirical Evaluation: The experimental section is a major strength. The authors evaluate FedDAG across four different non-IID data distributions, encompassing all the heterogeneity types they claim to address. The comparison against a large set of strong baselines is comprehensive. The inclusion of ablation studies (FedDAG*, FedDAG+) effectively isolates the contribution of different components (clustering vs. sharing, and sharing vs. parameter increase).\n\n- Practicality and Completeness: The paper thoughtfully addresses practical concerns such as communication/computation complexity, privacy implications, convergence (with a provided theorem), and dynamic scenarios like newcomer integration and data distribution shift. This makes the work feel mature and applicable to real-world FL systems.", "weaknesses": "- Computational and Architectural Overhead: The dual-encoder architecture inherently doubles the parameter count for the feature extractor compared to a single-model approach. While the ablation study (FedDAG†) convincingly shows that the gains are due to sharing and not just more parameters, this overhead is non-trivial for resource-constrained edge devices. The paper mentions the possibility of alternating training phases to mitigate this, but a more detailed discussion on the trade-offs (e.g., how much it slows convergence) would be beneficial.\n\n- Clustering Stability and Cost: The clustering mechanism, while adaptive, is performed initially and then assumed to be stable (Assumption A.4 in the convergence analysis). The method for handling distribution shift (Appendix A.8) is reactive and requires clients to locally re-train, which incurs additional communication and computation cost. The overall one-time cost of the initial clustering phase (local warm-up, SVD, gradient sparsification) is non-negligible, though the paper argues it's a small fraction of total training. A more explicit comparison of this \"clustering overhead\" against other methods would be helpful.\n\n\n- Clarity of the CC-Graph and Sharing Mechanism: The process for building the Cluster Complementarity Graph (CC-Graph) and the subsequent secondary encoder training is complex. The description in Section 4, while detailed, is challenging to follow on a first read. A more intuitive explanation of the \"demand\" and \"supply\" scores, and a clearer step-by-step walkthrough of the training phases, would improve accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2425f7ra8S", "reviewer": "Reviewer_8htn", "summary": "This work proposes a novel clustering method for federated learning, FedDAG. FedDAG combines data and gradient information to cluster clients more effectively. FEDDAG leverages cross-cluster representation sharing and incorporates an efficient mechanism to automatically determine the optimal number of clusters. Experiments are conducted to evaluate the performance of the proposed mechanism.", "questions": "See weaknesses. I would like to change my score if the authors could address the concerns.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The structure of this paper is clear and reasonable.\n\n2. The paper is easy to follow.\n\n3. Experiments show that the performance of the proposed method is significantly better than existing methods in different scenarios.", "weaknesses": "1. The literature review section should be expanded to include more complete related work and discuss several related subtopics.\n\n2. Directly reporting the class sample count will leak sensitive sample distribution information.\n\n3. The novelty of the dual encoder structure is limited, similar to the personalized/shared head of pFL works and the personalized/shared feature extractor of feature-skewed FL works.\n\n4. Although the author claims that FEDDAG is the only model that can solve all four data heterogeneity problems, the paper lacks experiments on feature distribution shifts. Commonly used datasets like Office-Caltech-10, PACS, and DomainNet should be studied.\n\n5. The baseline is mainly compared with clustered FL methods. It should also be compared with FL methods that address different heterogeneity issues.\n\n6. The layout of the paper can be improved. For example, the experimental results in Table 2 and Figure 1 are too far away from the experimental section; the method diagram in Figure 2 could be placed in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)