Differentially Private Clustered Federated Learning
Authors: Saber Malekmohammadi
Afaf Taik
Golnoosh Farnadi
Venue: Accepted by TMLR ()

Abstract: Federated learning (FL), which is a decentralized machine learning (ML) approach, often incorporates differential privacy (DP) to provide rigorous data privacy guarantees to clients. Previous works attempted to address high structured data heterogeneity in vanilla FL settings through clustering clients (a.k.a clustered FL), but these methods remain sensitive and prone to errors, further exacerbated by the DP noise. This vulnerability makes the previous methods inappropriate for differentially private FL (DPFL) under structured data heterogeneity. To address this gap, we propose an algorithm for differentially private clustered FL, which is robust to the DP noise in the system and identifies the underlying clients’ clusters correctly. To this end, we propose to cluster clients based on both their model updates and training loss values. Furthermore, for clustering clients’ model updates at the end of the first round, our proposed approach addresses the server’s uncertainties by employing large batch sizes as well as Gaussian Mixture Models (GMM) to reduce the impact of DP and stochastic noise and avoid potential clustering errors. We provide theoretical analysis to justify our approach and evaluate it across diverse data distributions and privacy budgets. Our experimental results show the approach’s effectiveness in addressing high structured data heterogeneity in DPFL.

Published in Transactions on Machine Learning Research (09/2025)
Differentially Private Clustered Federated Learning
Saber Malekmohammadi
Mila Quebec Al institute, Montreal, Canada
School of Computer Science, University of Waterloo, Waterloo, Canada
Afaf Taik Mila Quebec Al institute, Montreal, Canada
Golnoosh Farnadi Mile - Quebee AI institute, Montreal, Canada
School of Computer Science, McGill University, Montreal, Canada Université de Montréal, Montreal, Canada
Reviewed OpenReview: https://openrevvee net/ per/forum? id= JSsko0a4yr
Abstract
Federated learning (FL), which is a decentralized machine learning (ML) approach, often incorporates differential privacy (DP) to provide rigorous data privacy guarantees to clients. Previous works attempted to address high structured data heterogeneity in vanilla FL settings through clustering clients (a.k.a clustered FL), but these mothods remain sensitive and prone to errors, further exacerbated by the noise, making them inappropriate for differentially private FL (DPFL) under structured data. heterogeneity. To address this gap, propose an algorithm for differentially private clustered FL, which is robust to the DP noise in the system and identifies the underlying clients' clusters correctly. To this end, we propose to cluster clients based on both their model updates and training loss values. Furthermore, for clustering clients' model updates at the end of the first round, our proposed approach addresses the server's uncertainties by employing large batch sizes as well as Gaussian Mixture Models (GMM) to reduce the impact of DP and stochastic noise and avoid potential clustering errors. We provide theoretical analysis to justify our approach and evaluate it across diverse data distributions and privacy budgets. Our experimental results (codes: here show effectiveness of the approach in addressing high struclured data heterogeneity in DPFL.
1 Introduction
Federated learning (FL)
McMMhan et al.,
2017) collaborative paradigm, which allows multiple clients
to train shared global model without sharing their data. However, in order for FL algorithms to ensure
rigorous privacy guarantees against data privacy attacks (Hitaj et al. 2017 Rigaki & García 2020 Wang
et al.
2019 Zhu et al.
2019 Geiping et al. 2020), they are reinforced with DP
(worrk et al.] 2006b Dwork
2011
Dwork & Roth 2014).
This is done in the presence of a trusted server (McMahan et al. 2018 Geyer
ctt al.
2020 Duchi et al. 2013 2018 In the latter case and for sample-level DP, each client runs DPSGD AAbadi et al. 2016 locally and shares its noisy model updates with the server.
2017) and its absence [Zhao et al.
A key challenge in FL settings ensuring an acceptable performance across clients under heterogeneous data distributions. Several existing works focus accuracy parity across clients with single common model by agnostic FL (Mohri et al. 2019) and client reweighting (Li et al. 2020c b Zhang et al. 2023 However, single global model often fails to adapt to the data heterogeneity across clients (Dwork et al. 2012). especially when a high data heterogeneity exists. Furthermore, when using a single model and augmenting FL with DP, different subgroups of clients are unevenly affected even with loose privacy guarantees (Farrand et al.) 2020
Fiorctto et al., 2022 Bagdasaryan Shmatikov 2019). In fact, subgroups with minority clients experience a larger drop in model utility, due the inequitable gradient clipping DPSGD (Abadi et al. 20166] Bagdasaryan & Shmatikkv
2019 Xu al.] 2021 Esipova et. al. 2023). Accordingly, some works proposed to model

---

Published in Transactions on Machine Learning Research (09/2025)
personalization by multi-task learning (Smith et al.
2017 Li et al. 2021 Marfoq et al.
Wu et al. Mansour Briggs et al. 2020 The latter
2021
2023) transfer learning (Li & Wang
2019; Liu et al.
2020 and clustered FL (Ghosh et al.
2020
et al. 2020; Ruan & Joe-Wong! 2021 Sattler et al. 2019 Werner et al. 2023
has been proposed for vanilla FL and is suitable when "structured data heterogeneity" cxists across clusters of clients (as in this work): subsets of clients can be naturally grouped logether based on their data distributions and one model is learned for each group (cluster). However, as discussed by Wernerr et al. (023), the existing non-private clustered FL approaches are vulnerable to errors in clustering due to their sensitivity to: 1. model initialization 2. randomness in clients' model updates due to stochastic noise. The DP noise existing in DPFL systems' training mechanism exacerbates this vulnerability by injecting more randomness.
To address the aforementioned gap, we propose a sample-eveel differentially private clustered FL algorithm (Algorithm [], which uses both clients' model updates and loss values for clustering clients, making it more robust to DP p/sstochasticc noise: 1) Justified by our theoretical analysis (Lemma 4. and 4.2) and in order to cluster clients correctly, our proposed algorithm uses a full batch size in the first FL round and a small batch size in the subsequent rounds, to reduce the noise in clients' model updates at the end of the first. round. 2) Then, the server soft clusters clients based on these less noisy model updates using Gaussian Mixture Model (GMM). Depending on the "confidence" " of the learned GMM, the server keeps using it to soft cluster clients during the next few rounds (Section 4.3) 3) Finally, the server switches the clustering strategy to local clustering of clients based on their train loss/accuragy values in the remaining rounds. These altogether make our DP clustered FL algorithm effective and robust. The highlights of our contributions are as follows:
We propose a sample-level DP clustered FL algorithm (R-DPCFL), which combines information from both clients' model updates and their loss values. The algorithm is robust and achieves high-quality clustering of clients, even in the presence of DP noise in the system (Algorithm D
We theoretically prove that increasing clients' batch sizes in the first round (and decreasing them in the subsequent rounds) improves the server's ability to cluster clients based on their model updates at the end of the first round (Lemma 4.2).
We show that utilizing sufficiently large batch sizes in the first round (and sufficiently small batch sizes in the next rounds) enables super-linear convergence rate for learning a GMM by the server on the clients' model updates at the end of the first round. This leads to soft clustcring of clients using a GMM with a low computational overhead (Theorem 4.3)
We extensively evaluate the proposed algorithu across diverse datasets and scenarios and demonstrate the effectiveness of our robust DP clustered FL algorithm in detecting the underlying cluster structure of clients, which leads to an overall utility improvement for the system (Section
2 Related work
Model personalization is technique for improving utility under data heterogeneity (Li et al. 2021 Liu et al. 2022a), which usually leverages extra computations, e.g., extra local iterations (Li et al.) 2021 On The other hand, clustered FL has been proposed for personalized FL under a high structured data heterogeneity, where clients can be naturally partitioned into clusters: clients in the same cluster have similar data distributions, while there is a significant heterogeneity across various clusters. Existing clustered FL algorithms group clients based on their loss values GGhoh et al. 2020 Mansour et al. 2020 Ruan & Joe- 2021 Dwork et al. 2012 Liu et al. 2022b) or their model updates (based on e.g., their euclidean distance (Werner et al. 2023 Briggs et al. 2020) or cosine similarity (Sattler et al. 2019)) ). As shown by Werner et al. (2023), the
algorithms are prone to clustering errors in the early rounds of FL training -due to gradient stochasticity, model initialization or the form of loss functions far from their optima- which can even propagate in the subsequent rounds. This vulnerability is exacerbated in DPFL systems, due to the existing extra DP noise. Without addressing this vulnerability, Luo et al. (2024) proposed a DP clustered FL algorithm with a limited applicability, which clusters clients based on the labels that they do not have their local data. In contrast, our DP clustered FL algorithm is applicable to any setting characterized by number of clients, where each client holds many data samples and needs sample-level privacy protection. Cross-silo FL systems can be
2

---

Published in Transactions on Machine Learning Research (09/2025)
considered an instance, where there are fewer clients but each hold many data subjects that require protection. For example, when sespittal/baans//cchoooo wish to federate is the people owning those records rather than the participating silos that should be protected. The closest study to this setting has been done by Liu et al. (2022)), which considers silo-specific sample-level DP for cross-silo FL and studies the interplay between privacy and data heterogeneity. More specifically, they show that when clients have large datasets and "moderate" " data heterogeneity erits across them: participation in FL by clients is encouraged over local training, as the FL model averaging on the server diminishes the effect of DP noise 2. under the same total privacy budget, model personalization through mean regularized multi-task learning (MR-MTL) leads to better performance compared to training single global model or local training by clients (see Appendix C.4 about MR-MTL formulation). Complementing the work, we show that model personalization (with MR-MTL), local training and even for DPFL with structured data heterogeneity. Accordingly, we propose our new DP clustered FL algorithm.
records, it
clustered FL are not efficient.
3 Definitions, notations and assumptions
There are multiple definitions of DP. We adopt the following definition to be satisfied by every client:
Definition 3.1 ((e, 6)-DP (Dwork et al., 2006at). A randomized mechanism M A + R with domain A and range R satisfies (c. 8DP if for any two adjacent imputs D, D' E A, which differ by only a single record (by replacement), and for any measurable subsel of oulputs S R it holds that
PH(M(D) ES] S (HMPDDD) E ES] ES]+0 5.
The gaussian mechanism randomizes the output of query f as M(D) A f(D)+ + o0) The randomized output satisfies (e, 0)-DP for a continuum of pairs (e, 6): for all e, E (0, and a > 2/1m((1.5//6) A2ʃ, where A2f A m || ((D) - F(D) lla is the lz-sensitivity of the query f with respect to its imput. Also, the and 8 privacy parameters resulting from rumning Gaussian mechanism depend on the quantity 2 A.7 (called "noise scale"). We consider a DPFL system (see Figure 2 left), where there are n clients running DPSGD with the same "sample-level" privacy paramcters (c, the set of information (including model updates and cluster selections) sent by client i to the server salisfies (e, 6)D for all adjacent dalasets Dj and D' of the client i differing in one sample (by replacement).
Let x E X S Rd and e E y'= [1....,CC C} denote an input data point and its target label. Client holds datasct D; with N₂ samples from distribution Pa(x,, y) = (((x)) Let h : X x 0 - RC be the predictor function, which is parameterized by 0 IRP, Also, let é: RC x y + R. be the used loss function (cross-entropy loss). Client i in the system has empirical train loss fc(0) - EaaweD, [e(h(z,))) ( 0), v)], with minimum value fi There are E communication rounds indexed by and K local epochs with learning rate m during each round. There are M clusters of clients indexed by m, and the server holds M cluster models {0e, JM m=1 for them at the beginning of round e. The value of .M maybe unknown at the beginning. Clients i and j belonging to the same cluster have the same data distributions Pz 8) = Pj (a. y), while there is a high data heterogencity across clusters. si) denotes the true cluster of client and (i) denotes the cluster assigned to it at the beginning of round e. Let us assume the batch size used by client i in the first round e ==1 is b/, which may differ from the batch size bi¹ that it uses in the rest of the rounds e 1. At the I-th gradient update during the round e, client uses batch Bedd with size by, and computes the following DP noisy batch gradient:
の. 0²² = 1 by
E JEBS
0o)(0) + +N(0, ((0000.0pTT))
(1)
where (0) clip(((((((( 0), c), and is a clipping threshold to clip sample gradients: for given vector v, clip(v,c) = min M Also, N is the Gaussian noise distribution with variance of where d, = c (e, 6,, 6,1 NN, K, E), and 2; is the noise scale needed for achieving (c,6)-DP by client i, which can be determincd with a privacy accountant, c.g., Rényi-DP accountant (Mironov et al.] 2019) used in this work, which is capable of accounting composition of heterogeneous DP mechanisms (Mironov 2017). The privacy parameter ó is fixed to 10-1 in this work with 5 < N for every client For random
3

---

Published in Transactions on Machine Learning Research (09/2025)
== (UI,.... pp) E Rxx1 we define Var(v) := Eド Bi E[u;)) i.e., the sum of the variances of all its elements. Table in the appendix sunmarizes the used notations. Finally, we have the following assumption: Assumption 3.2. The stochastic gradient (0) पे Ejeo (0) is an unbiased estimator of f with bounded variance: v0 RP: Varlg,"(()) S o only on the used batch size bf: the laryer Ug, the smaller oigg(1))
oig(0))) (b.)) The light bound og.g (g)) is constant depending
4 Methodology and proposed algorithm
As discussed by Werner el al. (002)), clustered FL algorithms which cluster clients based on their loss values (Mansour et al. 2020; Ghosh al.) 2020; Ruan & Joe-Wong, 2021), ie., assign client to cluster Re (i) arg minm (1)0 at the beginming of round e, are prone clustering errors in the first few rounds, mainly due to random initialization of cluster models {0 1m On the other hand, clustering clients based on their model updates (gradients) (Werner et al. 2023 Briggs et al. 2020 Sattler et al. (219) makes sense only when the updates are obtained on the same model initialization. Additionally, even if we assume these algorithms can initially cluster clients perfectly in each round e, the clients' model updates (gradients) will approach zero as the clusters' model parameters converge. Hence, clients from different clusters may appear to belong to the same cluster, which again results in clustering mistakes.
1200
f1
1000
F4
800
ess
600
400
200
0-
100
18
18
-10
-5
10
model parameteers
Figure 1: Loss- Loss-based clustering algorithms miscluster in the initial rounds, due to model initialization. Also, even with the assumption of perfect clustering of clients in the first rounds, clustering algorithms based on gradients (model updates) leads to clustering errors in the last rounds, due to the gradients approaching zero.
We now elaborate on why clustering clients based on their losses (model updates) is prone to errors in the first (last) rounds with an example. Consider Figure [ where there are M = 2 clusters (red and blue) and n = clients. The clients in the red cluster have loss functions (0) = 4(0 + 6)2 and f2(() = 4(0 + 5)2 with optimum cluster parameter 0 -5.5. Also, the the clients in the blue cluster have loss functions fa( = 4(0 5)2 and f4) = 4(0 6)² with optimum cluster parameter 0.00 5.5. Clustering algorithms, which cluster clients based on their loss values clusters' models, are vulnerable to model initialization. For example, if we initialize the clusters' parameters with 09- = -11 and 09 = O (shown in the figure), all four clients will initially select cluster 2, since they have smaller losses on its parameter. At 09 = O, the average of clients' gradients (model updates) is zero, so all clients will remain stuck at 89 and will always select cluster 2.
On the other hand, clustering clients based on their model updates (gradients) (Wenner et al. 2023 Briggs et al. 2020: Saatler al. 2019) have clearly issues. One of these issues appears after some rounds of training. For instance, even if assume these algorithms can initially cluster clients "perfectly" in each round e, the clients' model updates (gradients) will approach zero as the clusters' models converge to their optimum parameters. Hence, clients from different clusters may appear to belong to the same cluster, which results in clustering mistakes. For example, as shown in Figure [ let us assume after T rounds of "correct" clustering of clients, the clusters' parameters get to 0T = -4.5 and 63 = 5.5. At these parameters, clients 1 and 2 (which have been "correetly" assigned to cluster so far) will have gradients ((T) = 12 and (/( = 4. Similarly, clients 3 and (which have been "correctly" assigned to cluster so far) will have f((05) = 4 and (100)) -4. We see that f2 is closer to S/2 and S than to fi, and in the next round it will be wrongly
4

---

Published in Transactions on Machine Learning Research (09/2025)
1stt siager leam GNM with M components
Server
gin
bashh sizo fork epochs
E,
25e
stage: soft clustaring
2) DPS3D wit amall epochs
Sample- DF
Uso GMBM for disttiiig
Cariass NIVEE
stage: "loaal"
Broadoast cussor models
Run looal dustoring
Clases (E.5)
Uidaato duster modill
31 Trikk using small batchee
(e.0)
Figure 2: Left: Considered threat model in this work, where client i has local train data Dj and "sample-level" DP privacy parameters (c, and does not trust any external party. Right: Three main stages of R-DPCFL.
assigned to cluster (with clients 3 and 4). This happens while the clients are clearly distinguishable based their lossos, some progress in training has been made after T rounds: f2(0f) 1, while f2(05) 21 which clearly means that client 2 correctly belongs to cluster 1. Therefore, after making some progress in training the clusters' models, it makes more sense to use a loss-based clustering strategy than using a strategy based on clients' gradients (model updates).
Motivated by this vulnerability, which will get exacerbated by DP noise, we next propose a DP clustered FL algorithm which starts with clustering clients based on their model updates for the first several rounds and then switches its strategy cluster clients based on their loss values. We augment this idea with some other non-obvious techniques to enhance the clustering accuracy.
4.1 R-DPCFL algorithm
Our proposed R-DPCFL algorithm has three main steps (also see Figure 2 right and Algorithm D
In the first round, clients train the initial model ganit locally. They use full batch sizes in this round to make their model updates {A0] Fi less noisy. Note that even when clients have a limited memory budget, they can still perform DPSGD with full batch size and no computational overhead by using gradient accumulation technique (see Appendix G). Then, the server soft clusters them by learning GMM on their model updates. The number of clusters (M) is cither given or can be found by maximizing the confidence of the learned GMM (Section 4.3
2. During the subsequent rounds e E E.....,,... the server uses the learned GMM to client uses a small batch size b,1 and contributes to the training of cach cluster (m) model proportional to the probability of its assignment to that cluster (Ts,m). The number of rounds Eg is set based on "confidence level" of the learned GMM (Section 4.3)
clients:
After some progress during the first E, rounds, clients' train loss/ aaccuracy values on cluster models (0 Jm= are meaningful. Accordingly, clients use them locally during the remaining rounds to privately select cluster using exponential mechanism (with parameter Cyneett Equation 8
In Sections
4.2 and
we provide theoretical justifications for the usage of full batch size in the first round.
4.2 Reducing GMM uncertainty via using full (small) batch sizes in the first (next) round(s)
It the DP noise in (^ that makes hard for the server to cluster clients by learning a GMM on the model updates. Lemma 4.1 extends a similar result in (Malekmohammadi al., (024)) and shows that the amount of noise in A the end of each round rapidly declines when by increases.
5

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: R-DPCFL
Input: Initial parameter 0inat dataset sizes (N N. batch sizes {bi ..... b, clip bound local epochs K, global round E, number of clusters M (optional) Output: cluster models {0m m//m=
for each client i (I....,n) n) do
by N
full batch size
2, - (e, ee, 64 N₁ K)
for e E EI. do
if e then for each client i [l.....n] in parallel do
L A0 DPSGD (@init b). N₁, K, 2,
7
on server:
if M is unknown then
M arg max M. MSS GMM A A0 M²)
// set M (Section 4.30
11
{II....., Tn, MPO} GMM A0 set (MPO)
A0,;M) M):
// 1st stage: GMM
12
// set Eg (Section 4.3
Initialize cluster models uniformly: 0f 0% 0 inst
14
continue;
// go to round 2
15
else if E (2....,Bo Ee) then for each client i (1....,n) do with probability mm[m]]
L (1)
// 2nd stage: soft clustering
18
else
19
on server: broadcast cluster models {mm inJ== all clients run clustering locally for each client i (1....,m) do
21
L (1) arg minm )i(0%); // 3rd stage: private loss/ loss/accuracy-basee clustering for each client i (1,.. n in parallel do A DPSGD (0): by N₁, K, 2,,
22
// batch size b21
23
24
server:
25
for each client i E [I. do T
26
Ej=1 1R=()-
27
for m E (I,...,M MM do
1c+1 t 0% + 12 wt A0;
28
Lemma 4.1. Let us assume 0:000 is the starting model parameter for client i at the beginming of round e. After local epochs with step size n, the client generates the noisy DP model update A (1t) at the end of the round. The amount of noise in the resulting model update can be quantified as:
(of :- ( se No oi pe 27 (E,,(,,,,)) K,
(2)
(b5)3
The first conclusion from the lemma that the noise level A rapidly declines as of increases: See Figure 3 and the effect of batch size on Var (on the left);; and the effect of batch size 6711 on Var (ar (0) (e > 1) (on the right). Let us consider = 1 especially: If all clients use full batch sizes in the first round (i.c., bj = N; for every client i), it becomes much casier for the server cluster them at the end of the first round by learning GMM on {AO. 1 as their updates become more separable. An illustration of 1n this is shown in Figure 5 As the second key takeaway, Figure 3 left shows that in order to make (AO) )

---

Published in Transactions on Machine Learning Research (09/2025)
150
16
40
Var
256
Vorl (oor (00))m)))
200
314.77
33
30
200
230
100
50
256
320
38,0
Figure 3: Plot of (ar Var(Ao: (b))) (left) and Var (b;) 10000 (e > 1) (right) v.s. both by and b There are two clear takeaways: 1) for all e {1.... E), Var A0; (b. 100 decreases with if steeply (from (left figure) is considerable (see Figure 4 for the plot of (, A...bb1... N, K. E) v.s. by and b2²). The results are obtained on CIFAR10 from Rényi-DP accountant (Mironov et al. 2019) a setting with N, = 6600, e 5, 10_1,
Lemma 4.1p. 2) The effect of 6,21 on Var(A0)
1, 200, p-
11, 181, 642, 1/ 10
28
2.4
22
1.8
1.6
16
1.2
256
0.929
64 326
128
6)
36
Figure 4: Plot of 2,(e, 6, 6,, N₁, E) v.s. 6) and 6.¹1 obtained from Rényi-DP Accountant (Mironov et al, 2019) in a setting with N₁ 6600, e = 5, 10 K E, E 200. As observed, 2; is a sublincarly increasing function of both by and b:21. Also, the effect 6,'1 is much more than the effect of bj. The reason is that bi11 is used in E --1 rounds, while 67 is used only in the first round. So the value of 6,11 affects z the most.
less noisy, we have to make {6] as large as possible and also keep (b M smal In the next section, we will provide a thcoretical justification for the observation in Figure
4.2.1 Effect of batch sizes (b) on the separability of clusters
In order to theoretically understand the reason behind the observation in Figure 5 lel. assume clients have the same dataset sizes and first batch sizes for simplicity: Vi N; N,
N, b = 6¹. Also, remember that 07.00 ginit
Having uniform privacy parameters (e, 8), have: Vi: (b²)) = Var (b)) (Oint) (b²))) Hence, we can consider the model updates {AO) (b as the samples from a mixture of M Gaussians with mean, covariance matrix, prior probability parameters: (bb))) = (uim (b²), Σx, (b'), o mm= where Vm. or > 0 and p" (b2) 4 p" (b)) (m & m') (due to data heterogeneity across clusters):
M
There close relation between the result of Lemma
the law large numbers. See Appendix for details
7

---

Published in Transactions on Machine Learning Research (09/2025)
0.0004
second prinippa component
prinppa componerr
0.0002
cluster
Mg (b**
cluster
(b*)
cluster
M4 (b*)
cluster
Mg (b*)
000002
-0.0007
0.0000
0.0004
first priccppl component
principal component
Figure 5: 2D PCA visualization of updates {40 Left: 6, = 10, b = 32 for all i and e. Right: G; = 10; by = = 6600, i.e, full batch sizes (assuming N₁ = N = 6600 for all clients), and 6,² = 32 for all The empty markers show the centers of the Gaussian components. The model updates are obtained from clients running DPSGD for K epochs locally on CIFAR10 with covariate shift (rotation) across clusters.
1. (b) E (b! ginit Poot, b m
(3)
Em (b)==)=== = (A0) - pmn ( - (mn((11))) (b¹)) init .b. = bb', = m (o (b)))
Ip:
(4)
where the last equality is from Var [A] Oinit ,b, = 6-1 = BiIlA! - ( - and that the noises existing in each of the p elements A0; are i.i.d (hence, 2,, (b1) is a diagonal covariance matrix with equal diagonal elements). Intuitively, we expect more separation between the true Gaussian components (14, (b²), ,, Sm (b²)) > from which clients' updates (AO: ) sampled, to make the model updates more distinguishable for the server. Next, show that the overlap between the Gaussian components (N (b+)), Ex () decreases fast with b²:
Lemma 4.2. Let A (b-) = ( (b-)( when Vi : 63 = 62. The overlap between com- ponents N (P (b'), En,(")) (b))) and (( Eb (b')) is Om,m' 2Q 20- VarlA0: Oinit by = b*] and Q(-) is the Q function. Furthermore, if we increase b 6² to bj = kb² S N (for kppm
pmm...10 where (a (b'))²
m/(b))) where 1 O(1) is a small constant.
all i), we have Om,m' < 2Q( 2b1)(bb)
The terms Am.m (b*) and a² (b²) represent the "data heterogeneity level across clusters and m² and "privacy sensitivity of their clients", respectively. We define their "scparation score" as (m, m')
20 20-
(b') 20 (6 The larger S5(m, m'), the smaller overlap Om,m² 2S((s(m, m')). Based on the form of the Q function, an SS(m, m') above 3 can be considered as complete separation of the components.
4.3 Tuning hyper- parameters of R-DPCFL
As we observed in Lemma 4.2 the separation score S5,m, m') (the overlap Om.m') increases (decreases) as b Am. (b²)
increases. Remember that SS(m, m') 2o (6')/pp and note that (a- (b²))2/2ppp /p is the value of diagonal elements of covariance matrices of Gaussian components, which the GMM aims to learn (see Equation 4 Therefore, when the CMM is learned, we can use its parameters to get an estimate score SS(m,, m') for every pair clusters m and m'. Then, we can define the "minimum pairwise separation score" as MSS 5, (b] (6? [1 } = minm,m' SS m, m') E [0, +00) as measure of confidence of the learned GMM in its identified chusters. The larger the MSS of learned GMM, the more "confident" it is in its clustering decisions. For instance, if we learn
8

---

Published in Transactions on Machine Learning Research (09/2025)
GMM Figure leftt, it will have much smaller MSS than when we learn GMM on Figure 5 right. We can similarly define the estimated "maximum pairwise overlap" for a learned GMM as MPO = measure of uncertainty of the learned GMM. As we will explain, we can use MSS and MPO of the learned GMM to set the hyper-parametors of R-DPCFL, namely b,², Ec and M (when the number of clusters is unknown).
E [0,1), as a
4.4 Convergence rate of EM for learning GMM
Let us define the maximum pairwise overlap between the components in (=*))) (b"), Em (b*), am J Imm11 as 0 (vb (b') = maXm.m² (
According to Lemma 4.2 when b' is large enough, O (b')) decreases (like in Figure @ right) and we can expect EM to converge to the true GMM parameters 1/" (b'). Next, we analyze the local convergence rate of EM around 10" (b*)..
Theorem 4.3. Ma et al. 2000) Given model updates
samples from true mixture of Gaussians v5 (b") [M (b') (b'))) om JM if O (bb()) is small enough, then:
I 20 (b')l
(0
lim 500 | - - (b*)1l
(5)
as n increases, v²' is the GMM porameters returned by EM after r iterations. y is an arbitrary small positive number, and o(z) means it is a higher order infinitesimal as + 0: lim [(x) = O.
This means that convergence rate of EM around the true solution 30 (b") ) is faster than how [O (v² (b²) ()8)))
decreases with 6 (from Lemma 4.2p. Hence, as an important consequence, the computational complexity of learning the GMM in the first round also decreases fast as b' increases.
5 Formal privacy guarantee of R-DPCFL
The privacy guarantee of R-DPCFL for each client in the system comes from the fact that the client runs DPSGD with fixed DP noise variance di c² 27 (c, 8, b,, ,,,,,,,,,,,,, ,N,,K, In the following theorem, we provide formal privacy guarantec for the algorithm to show the sample-level privacy guarantees provided to each client i. We defer the proof to Appendix E
E. E) for all of its batch gradient computations.
Theorem 5.1. The set of model updates { 1 which are uploaded to the server by each client i E
{1....,, ,n] during the training time, as well as their private local model cluster selections satisfy (c, 6) DP,
where the parameters e and depend on the DP noise variance o2 used by the client for DPSGD (Equation
and the porometer Eselect used for its privale cluster selections using exponential mechanism (Equation 8
6 Evaluation
Datasets, models and baseline algorithms: We evaluate proposed method three benchamark datasets, including: MNIST (Deng) 2012), FMNIST (Xiao et al. 2017) and CIFARIO (Krizhessky 2009). with heterogeneous data distributions from covariate shift (rotation; (z( varies across clusters) (Kairouz
et al. 2021 Werner et al. 2023 and concept shift (label flip; ()) varies across clusters) (Werner et al. 2023). which are the commonly used data splits for clustered FL (see Appendir C) We consider four clusters of clients indexed by m E 2, with {3, 6, 6, 6} clients, where the smallest cluster considered as the minority cluster. We compare our method with most recent related DPFL algorithms under an equal total sample-level privacy budget e: 1. Global (Noble et al. 2021): clients run DPSGD locally and send their model updates to the server for aggregation and learning one global model 2. Local (Liu et al. 2022a): clients do not participate FL and learn a local model by running DPSGD on their local data 3. A DP extension of IFCA (Ghosh et al. 2020 Liu et al. 2022a :: local loss/accuraccy-aseel clustering performed by clients on existing cluster models 4. MR-MTL (Liu et al. 2022a): : uses model personalization to learn one model for each client 5. O-DPCFL: an oracle algorithm which has the knowledge of the true clusters from the first round. For R-DPCFL and IFCA, we use exponential mechanism (Rogers & Steinke 2021), which satisfies zero concentrated DP (x-CDP) (Bun & Steinke 2016), to privatize clients' local cluster selections. See also Appendix B.3 for details.
9

---

Published in Transactions on Machine Learning Research (09/2025)
PNAIS:: covaditte 672
CRASSD conanate St.
CRAR30 corlegt OV1
use
ploxs piinccto G wil Urrt
phuis pronded vif be to"" CIPRRO, conrept thn
WhiST, conomate still
PANST covar ane shift
CITARLD covariate shif
Dumber eonnet ruppet
Fambe AEronee Aaatterree
priveyy (wnseetery, altid 10-+
10"
Figure 6: Top: Average test accuracy across all clients for different total privacy budgets Results are from four different runs. 10% means performing local clustering by clients only in 10% of the total number of rounds; i.e., rounds E. S E. + L Lo for R-DPCFL and rounds 1 S e S. 1+ L for IFCA (see Appendix C.6 Figure 11 in the appendix includes the "Global" baseline too. Bottom: Number of times that R-DPCFL and IFCA successfully detect the underlying cluster structure of all existing clients (out runs).
E
6.1 Results
Liu et al. (2022a) observed that under sample-level differential privacy (as this work) and "mild" data heterogeneity, federation is more beneficial than local training, because, despite the data heterogencity across clients, the model aggregation (averaging) on the server diminishes the effect of the DP noise in clients' model updates. However, when there is a "structured" data heterogeneity across clients, the level of heterogeneity is remarkable. Hence, learning one global model through FL is not beneficial, as one single model can barcly adapt to the high level of data heterogeneity across the clusters. Therefore, in DP clustered FL systems, local training and model personalization can be better options than global training, as they diminish the adverse effect of the high data heterogeneity. Furthermore, if one can detect the underlying clusters, one can perform in them in parallel and will simultancously benefit from 1. eliminating the effect of data heterogeneity across clusters; 2. diminishing the effect of DP noise by running FL aggregation cluster (as observed by (Liu et al. 2022ah). Hence, if the clustering task is done correctly, we can expect further improvement over local training and model personalization. This is exactly what R-DPCFL aims to do.
the server within each
RQ1: How does R-DPCFL perform in practice? S the average test accuracy across clients for four datascts. As can be observed, R-DPCFL outperforms the bascline algorithms (thanks to its more efficient clustering method: Figure bottom row). While R-DPCFL performs close the oracle algorithm, IFCA has a lower performance due to its clustering errors. For instance, IFCA has a clearly low clustering accuracy on MNIST and FMNIST, which Icads it to perform even worse than Local and MR-MTL. In contrast, it has better clustering performance on CIFAR10 (covariate and concept shifts) and outperforms the two baselines. On the other hand, the reason behind the low performance of MR-MTL is that it performs personalization on a global model, which in turn has a low quality due to being obtained from federation across "all" clients (hence adversely affected by the high data heterogeneity). Similarly, Local, which performs close to MR-MTL, cannot outperform R-DPCFL, as it does not benefit. from diminishing the effect of DP noise by FL aggregation.
RQ2: How does the minority cluster benefit from R-DPCFL? Figure 7 compares different algorithms based on the average test accuracy of the clients belonging to the minority cluster. R-DPCFL leads to a better overall performance for the minority clients, by virtue of its correct and robust cluster detection. Correct detection of the minority cluster prevents it from getting mixed with other majority clusters and lcads to a utility improvement for its clients. In contrast, IFCA has a lower success rale in detecting the minority cluster (Figure 7 bottom row) and provides a lower overall performance for them. Similarly, Local and MR-MTL lead to low performance for the minority, as they are conditioned on a global model that is learned from federation across "all" clients and provides a low performance for the minority. Correct detection of minority clusters is important, failure in detecting them correctly leads to a low performance for the smaller clusters.
10

---

Published in Transactions on Machine Learning Research (09/2025)
MRBI: owwr100 $10
HAR.5 Covarite SV
MMAIO comsriate s7
CIPARO concept J11
perieettr wit MNST covarate thir
phaty ARt 10
Urea 0dtecin menery chater
PHNIST, covariate shift
Shre Firccyy clante
CHARRO,, covariate thift
0000 mreeet Cluper
CINKES concent stift
privecy parametary, 15
with 5- 10100
Figure 7: Top: Average test accuracy across clients belonging to the minority cluster for different total privacy budgets and four different runs, Bottom: Number of times that R-DPCFL and IFCA successfully detect the minority cluster as a separate cluster (out of 4 runs).
rrin coparation
250. min parlwke 2.35
Figure 8: The effect of batch sizes 1,_ on the model updates { at the end of the first round with full batch sizes in the first round (Vi: = N.). For a fixed e = 10, the model updates scalter further in space as 622 increases and different clusters get less separable. This leads to a decline in the confidence level (MSS score) of the resulting GMM (written on top). Results are on CIFAR10 with covariate shift.
RQ3: How can we set the hyper-parameters of R-DPCFL?
Batch size b:'l: The batch size used by R-DPCFL during rounds e > has to be set to a small value, as observed in Figure 3 right. R-DPCFL is not sensitive to this parameter, as long as a small value is chosen for it. For the results reported so far, we used 6,21 =2 for all experiments with R-DPCFL. As we observed in
Lemma 4.1 and Figure 3 left, Var(Aá) (b))) (b)))t)))t is an increasing function of 6? More generally, the effect of increasing 6,²1 is threefold: 1) increasing noise variance (ar(a)) (b)) [0init) (as shown in Figure 3 left) 2) decreasing noise variance Var(A④) (bf) () (e 1, as shown in Figure 3 right) 3) decreasing mumber of gradient steps during each round for >> 1. While the first one is only limited to the first round e = 1, the last two affect the remaining F rounds and have conflicting effects on the final accuracy. However, an important point is that finding the true structure of clusters in the first round is a prerequisite for making progress in the next rounds. Therefore, increment in noise variance Var (b!)))init) (the first effect) the most important one. We have demonstrated this effect in Figure g which shows that how increasing 6,-1 adversely affects the clustering done at the end of the first round. Note how MSS score of the Iearned GMM increases as b²¹ increases. Therefore, in order to have a reliable client clustering at the end of the first round, need to keep the value of 6²² small: the smaller total privacy budget e, the smaller value should be used for 1 Following this observation, we have fixed b?21 to 32 in all our experiments with R-DPCFL.
The strategy switching time E.: The strategy switching time E. can also be set by using the uncertainty metric MPO E [0,1). Intuitively, the learned GMM is not certain about its clustering decisions, R-DPCFL should not rely on its docisions for large Eg, and vice versa. Hence, we can set E, as a decreasing function of MPO. For instance, E. (1 - MPO) E. which used in this work, means that if GMM completely confident about its clusterings, e.g., what happens in Figure 5 right, the server changes the clustering strategy to loss-based
11

---

Published in Transactions on Machine Learning Research (09/2025)
CIFAR10, covariste shift, M 6,N, 10
6. 2erffoall for al)
foral wo a
Number of EM steps
foral LB)- MM--------- 66 6
60
Mssie. 5. (oF)).
40-
privacy parameter c, with 6 10"4
Figure 9: Left: The number of EM iterations needed for learning the GMM decreases as bj increases. Especially, at full batch size (Vi : b! N.). very few iterations of EM are needed. The results are obtained on CIFAR10 with covariate shift across M = 4 clusters. Using a large enough batch size by in the first round makes the underlying clusters in {A0] Y more distinguishable, and convergence rate of EM algorithm for learning the GMM increases with 64. Right: MSS score v.s. e for two different local dataset sizes. A small local dataset size can be compensated for by using smaller batch sizes {b," 1b Ji=1 to get a larger MSS score.
after the first half of rounds. As the uncertainty increases, this change happens earlier (e.g., when is small), and R-DPCFL slowly gets close to the existing loss-based clustering methods like IFCA (Ghosh et al., 2020
Number of clusters M: Finally, can use MSS score of the learned GMM to set the number of underlying clusters M when it is unknown at the beginning. More specifically, we use the value of M for which the uncertainty of the learned GMM is minimized. We refer to Appendix CC. for further details. We remind that the DP extension of IFCA needs to know the true number of clusters M beforehand.
RQ4: How does 67 affect the convergence rate of EM for learning the GMM at round 1? Note that the only computational overhead of R-DPCFL compared to the DP extension of IFCA that it learns a GMM at the end of the very first round. Importantly, by simulating large batch sizes in the very first round (by using the gradient accumulation technique with zero overheads). The results in Figure 9 (left) supports Theorem 4.3 by showing that if 67 is large enough, the convergence rate of EM algorithm for learning the GMM at the end of the first round increases with b). Hence, using large batch sizes in the first round reduces the computational complexity of learning the GMM.
proved in Theorem 4.3 this overhead drops very quickly
RQ5: What if clients have small local datasets? While we envision the proposed approach being more applicable cross-silo FL, where elients' datasets are large, it is still worth exploring how beneficial it can be under scarce local data. As learned in the previous sections, the MSS score of the learned GMM can strongly predict whether the underlying true clusters will be detected: an MSS above 2 almost always yields to correct detection of the underlying clusters (see Figure 13 for experimental results). For a fixed (c, 6), larger (b and smaller (b;-1 J increase MSS (Lemma 4.1) When by = N₁ (i.e. full batch sizes for all in the first round), smaller local datasets result in lower confidence in the learned GMM. Nevertheless, this can 1 Figure 9 (right) compares two different dataset sizes under varying e. As observed, redueing [0 can compensate for smaller local dataset sizes by leading to less noisy model updates {A01 1m Jz=1- This improves the MSS score of the learned GMM and the clustering accuracy. Finally, we refer to Appendix for some hints about using data augmentation in the first round.
be compensated for by using even smaller (b, 1/-
7 Conclusion
We proposed sample-level DP clustered FL algorithm addressing systems with structured data heterogeneity. By clustering clients based on both their model updates and training loss/accuraa values, and using large initial batch sizes, our approach enhances clustering accuracy and miligates the disparate impact of DP on utility, all with minimal computational overhead. The proposed approach is robust to DP noise and has easy parameter selection. While envisioned for DPFL systems with large clients' datasets, the method is capable of compensating for moderate dataset sizes by using smaller batch sizes after the first round. A future research direction is addressing this limitation and extending the approach to be suitable for scarce data scenarios.
12

---

Published in Transactions on Machine Learning Research (09/2025)
8 Acknowledgment
Funding support for project activities has been partially provided by Canada CIFAR Al Chair, Facebook award, Google award, FRQNT scholarship and MEI award. We also express our gratitude to Compute Canada for their support in providing facilities for our evaluations.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security, 2016. URL https://doi.org/// 1141////1..1455/7777779999 191149 2978318
David Arthur and Sergei Vassilvitskii. k-means+4: the advantages of careful seeding. In ACM-SIAM Symposium on Discrete Algorithms, 2007. URL
1283494
Eugene Bagdasaryan and Vitaly Shmatikov. Differential privacy has disparate iwtpaaattttttimppaaa model accuracy. In Neural Information Processing Systems, 2019. URL httptttp:///rra..ooo//bbb////// 12101
Dimitris Bertsimas, Vivek F Farias, and Nikolaos Trichakis. The price of fairness. Operations research, 59(1):17-31, 2011. URL https://seb. mit. httpt///tw.atta...///oop/o/o//ta 20Fairness. pdf]
Christophe Biernacki, Gilles Celeux, and Gérard Govaert. Choosing starting values for the algorithm for
getting the highest likelihood in multivariate gaussian mixture models. Comput. Stot. Data Anal. 41:561
575, 2003. URL https://wwr: ott ott.:///c....ooocoooooo/oaaaa/aa
Patrick Billingsley. Probability and Measure. John Wiley Sons, Inc., 1995. ISBN 0471007102. URL
.pdf
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local
updates to improve training on non-iid data, 2020. URL
Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds, 2016. URL https:///rxii.. wwtp:/og///aaoeeeeeo 02065
Clément Canonne, Gautam Kamath, and Thomas Steinke. The discrete gaussian for differential privacy. In Advances in Neural Information Processing Systems (NewrIPS), 2020. URL httpss://proceedinga neurips.cc/paper htt.://pww.....eeoooooooooooaaa.
Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlocking high-accuracy
differentially private image classification through scale, 2022. URL
13650
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEFE Signal Processing Magazine, 2012. URL https::/teeexxploreeeee wtw../ie...g///ommmteeeeeeeeeeee
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with morean envelopes, 2022. URL
John C. Duchi, Michael Jordan, and Martin J. Wainwright. Local privacy and statistical minimax rates. 2013
5ist Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1592-1592, 2013. URI. ottpa///g////////..aaaaa.aaaaaoa http://eeattt.eeeeo..o..oorrraaa
John C. Duchi, Martin J.
Wainwright, and Michael I. Jordan. Minimax optimal procedures for locally
private estimation. Journal of the American Statistical Association, 113:182 201, 2018. URL https:
Cynthia Dwork. firm foundation for private data analysis. Commun. ACM, 2011. URL https: https://doi.org
10. 1144//16666699... 1866758
13

---

Published in Transactions on Machine Learning Research (09/2025)
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Compul. Sci., 2014. URL https:///l/ acm httg//a61/10.1511440000000000000
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, llya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Proceedings of the 24th Annual International Conference on The Theory and Applications of Cryptographic Techniques, 2006a. URL http::://ddo.oor//1/// 11761679 29.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryplography. Springer- Verlag, 2006b. URL https://doi. org/ 10. 1077/11611878
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214-226. Association for Computing Machinery, 2012. URL https://doi. org/ 10000.115/2000 1145/2090236 2090255
Maria S. Esipova, Atiyeh Ashari Ghomi, Yagiao Luo, and Jesse C. Cresswell. Disparate impact in differential privacy from gradient misalignment, 2023. URL https: orgg
07737
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi task learning. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2004. CRL https://doi..ooo wttp///...//////....... 1145//1111//12..1 10140677
Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. Neither private nor fair: Impact. of data imbalance on utility and fairness in differential privacy. In Proceedings of the 2020 Workshop on 04.1145/3411501.00 3419419. URL https://doi. sttps://doi.orr/100 10. 1145/1411111111111111 3419419
Machine Learning in Practice. Association for Computing Machinery, 2020. doi:
Ferdinando Fioretto, Cuong Tran, Pascal Van Hentenryck, and Keyu Zhu. Differential privacy and fairness in decisions and learning tasks: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, jul 2022.
doi: 10.24963 101224663/jjaii22000 ijcaai.0022/TT /766. URL https::/ org/10 10. 24963X2Fijcai 200.:922FF1..1..222227777666 2022%2F766
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how casy is it to break privacy in foderated lcarning? In Advances in Neural Information
wewriiiinnnnnnnnnnnneee.mrr. neurips. cc/paper files/ htlls//paper2220//f1eee. Processing Systems, 2020. URL Pdf.
Robin C. Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. ArXiv, 2017. URL attps://arxiv.org/ httg/pdf/111..075777
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In H. Tarochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19586 19597. Cur- ran Associates, Inc., 2020. URL
pdf
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models, 2021. URL https://arxiv.orr wwwpxxivv.or//aaao http://awwivv..rg/b////......./
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized lederated learning, 2020. URL https::/// http:///xxvvvvoov orgg/ass//////./// 02372
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognátion (CVPR), 2015.
URL httpp:://wwwt fo-founnaaion.. http://ppaacccss/oottttttoo http////eppppeeee_DaPS Btcp:/HHeeDeD____e @boep_Residual. Learning CVPR_ 201,__2166___1___11_ 1r66-_pperpppp__ .pdf
14

---

Published in Transactions on Machine Learning Research (09/2025)
Briland Hitaj, Giuseppe Ateniese, and Fernando Pérez-Cruz. Deep models under the gan: Information leakage from collaborative deep learning. Proceedings of the 2017 ACM SIGSAC Conference on Compuler and Communications Security, 2017. URL http:://ppp...enaticcch...i
semanticscholarr org/CorpusID:5051285
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hocfler, and Daniel Soudry. Augment your batch: better training with larger batches, 2019.
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: De- fonding against black-box membership inference attacks via adversarial examples. Proceedings of the 2019 ACM SIGSAC Conference on Compuler and Communications Securily, 2019. URL https:
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends in machine learning, 14(1-2):1-21 2021. URL
https://arxiv http://aaxxvv..o//bbbbbbaa/a /taabb////9..... 1912.04977
Yigitcan Kaya and Tudor Dumitras. When does data augmentation help with membership inference attacks? In Proceedings of the 38th International Conference on Machine Learning, 2021. URL https:
//proceedings.. warpaag...l......... Stl...prees//vv3//ee https///vv////aaaaaaaaa htmm
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. URL attps://www wttp:/www..ooo.....oooo TR..
Tian Lan, David Kao, Mung Chiang, and Ashutosh Sabharwal. An axiomatic theory of fairness in network resource allocation. In 2010 Proceedings IEEE INFOCOM, 2010. URL 0557
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. ArXio, abs/1910.0355 2019. URL https://api. oet.ammmm..cccc.....ooooorrraaaa ( 203951869
Jiacheng Li, Ninghui Li, and Bruno Ribeiro. Membership inference attacks and defenses in supervised learning via generalization gap. ArXiv, 2020a.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. In International Conference on Learning Representations, 2020b. UBL/attppe///rrrv.oo/////////000
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in International Conference on Learning Representations, 2020c. URL http:/pp///aar...ogg////////
federated ind learning. Tn
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learming, volume 139 of Proceedings of Machine Learning Research, pp. 1357-6360 PMLR, 18-24 Jul 2021. URL
mlr. press//v39/1110 html wttp://s....r...ee////////////
Jingcheng Liu and Kunal Talwar. Privale selection from private candidates, 2018. URL https: bttps://arxxr //rxiiv
org/ ha//a/bs/1110 /aaab////....... 0tt////81.009999 07971
Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, and Qiang Yang. A secure federated transfer learning framework. IEEE Intelligent Systems, 35:70 82, 2020. URL httpps//app..emmmnicccchlolar
semanticscholaar
:tg/CorpusID::190000
Ziyu Läu, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. On privacy and personalization cross-silo federated learning, 2022a. URL https://arxiv.. wwwgg//a///./..0rr//r/
Ziyu Liu, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. On privacy and personalization in cross-silo federated learning, 2022b. URL http://wwww.aaarrra/////////////
15

---

Published in Transactions on Machine Learning Research (09/2025)
Guixun Luo, Naiyue Chen, Jiahuan He, Bingwei Jin, Zhiyuan Zhang, and Yidong Li. clustering lederated learning for non-iid data. Future Generation Computer Systems, 154:384-388 -993844-5,,, 2024.
httpp::/wwwy / occeaccccc..oossseeetataa////roo
Jinwen Ma, Lei Xu, and Michael I. Jordan. Asymplotic convergence rate of the em algorithm for gaussian
mixtures. Neural Computation, 2000. URL httpss//ppp..eennttccccoo....
:000P=660:00073022 10273602
Saber Malekmohammadi, Yaoliang Yu, and Yang Cao. Noise-aware algorithm for heterogeneous differentially private federated learning, 2024. URL
03519
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Three approaches for personalization with applications to federated learning, 2020. URL httpp::/arxiv..rrgg/// http://www..r/aab///./....... 10619
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Taetitia Kameni, and Richard Vidal. Federated multi- task learning under a mixture of distributions. In Neural Information Processing Systems, 2021. URL
Brendan McMahan, Eider Moore, Daniel Ramage, Scth Hampson, and Blaisc Aguera y Arcas. Commuiication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017. URL https://arxiv..rrgr awtpaaass.aaeeeeeaee. 1602. 05629
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In ICLR, 2018. URL htttp:::/aaxii..rrggeg.eee/ wwtp///g///oeo/////oo//////e http://f//711..00000 pdf
H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, llya Mironov, Nicolas Papernot, and Peter Kairouz. A general approach to adding differential privacy to iterative training procedures, 2019. URL http::://arxiv.orrt
1812.06210
Nya Mironov. Rényi differential privacy. In 2017 IEEF 30th Computer Security Foundations Symposium (CSF), pp. 263 275. IEEE, August 2017. doi: 10.1109//sf.2017..1. URL http:://ct..oi..org//1.... CSF 2017.11.
Iya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism, 2019. URL httpp::://aaxi.oo///bbb////// 10530
Mchryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In International Conference on Machine Learning, pp. 00146
PMLR, 2019. URL https://arxiv.. www.arwwiw...oreeee. aa//abba///5 1902
Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut. Differentially private federated learning heterogencous data. In International Conference on Artificial Intelligence and Statistics, 2021. URL
https://proceeddngs mmr. http://cwwg....tt....ee////////a pttp:e//s.///oboce2/////le2a2aa pdf
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy, 2022. URL https://arxivv.rr//aaa 2160.03620
Sikha. Pentyala, Nicola Neophytou, Anderson Nascimento, Martine De Cock, and Golnoosh Farnadi. Privfairfl: Privacy-preserving group fairness in federated learning, 2022. URL ftttp:://arxxv.oor/abaaa 2200 11584
Maria Rigaki and Sebastián García. A survey of privacy attacks in machine learning. ArXiv, 2020. URL
220022550
Ryan Rogers and Thomas Steinke. A better privacy analysis of the exponential mechanism, 2021. URL
Sounded-range.
Yichen Ruan and Carlee Joc- Wong. Fedsoft: Soft clustered foderated lcarning with proximal local updating. CoRR, abs/2112.06053, 2021. URL http:://aarrvvvorr//bb////////// /ttpa/ab/b/221...3333
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs blx:k-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, 2019. https: httt::://axxx..orrppp///////0
16

---

Published in Transactions on Machine Learning Research (09/2025)
A. Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes. MI-leaks: Model and data inddpen- dent. membership inference attacks and defenses on machine learning models. Networks and Distrumuedd Sys- tems Security (NDSS) Symposium, 2019. URL https: http:///wwww.ddsssy/ee wwwpa///sii.ooooaoooooo/oo// 2019/02/ndss2019 2attaea22110001111111
03A-1 1 Salem_paper
Felix Sattler, Robert Müller, and Wojeiech Samek. Clustered federated learning: Model-agnostic distributed mult optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 52:3710-3722 3722, 2019. URL
http:://ppp....mtaaicccccccc...a wwwphhoolar..roo/ooooooooooo org/Corpus ID: 203736521
R. Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against. machine learning models. 2017 IEEE Symposium Security and Privacy (SP), 2017. URL https:
//api seaatticmat ottpa//iiccccccoooooooororrrra/a http//orpuID::1048880:
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Fodcrated multi-task learning.
In Newral Information Processing Systems, 2017. URL httapa:/aap..semaniicccoooo wttp:/aarrrrrrooossoolooooa
3586416
Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. Differentially private and fair deep learning: A lagrangian dual approach. ArXie, abs//0099.18 12562, 2020. URL https://api.
http://pp...emaniicccchhol..r Utt.ppps://pp...eenniieeees
org/
Corpus:D:221990859
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Beyond inferring class reprosentatives: User-level privacy leakage from federated learning. IEEE INFOCOM, 2019. URL
54436587
Mariel Werner, Lie He, Sai Praneeth Karimireddy, Michael Jordan, and Martin Jaggi. Provably personalized and robust federated learning, 2023. URL
Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. ArXiv, abs/2300 abs/2305.01068, 2023. URL
https://api. wwwanticschollatttt.teoeo org/CorpusID 258436670
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image datasct for benchmarking machine learning algorithms. CoRR, 2017. URL httt::://arxi..og/bbbb/////
07747
Depeng Xu, Wei Du, and Xintao Wu. Removing disparate impact on model accuracy in differentially private stochastic gradient descent. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021. URL
org/CorpusID: org/CorpusID:22668816. 2369801066
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning:
Analyzing the connection to overfitting. 2018 IEEE 31st Computer Security Foundations Symposium
(CSF), 2017. URL httph:://///t //api. wwwpppsssseaaaicccaaa...ooom wrt.chooooaaoorg//ruruurllllll 12:56444 2656445
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Yan Liu. How does data augmentation affect privacy in machine learning?, 2021. URL
10567.
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks, 2024.
URL httpp:://aaxxvoorr//bb////22
Guojun Zhang, Saber Malekmmhhammadi, Xi Chen, and Yaoliang Yu. Proportional fairness in federated learning. Transactions on Machine Learning Research, 2023. URL http::::/openreveeeeeeeet/eeee id= ryUHgEaWCQ
Yang Zhao, Jun Zhao, Mengmeng Yang, Teng Wang, Ning Wang, Lingjuan Lyu, Dusit Tao Niyato, and Kwok Yan Lam. Local differential privacy- based federated learning for internet of things. IEEE Internet of Things Journal, 8:8836 8853, 2020. URL httppss//app..emmmtiicccoooooo
swwantcsccooollrrr... wttpattiiicccooaaooorrrrrrrrroee :218228540
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Neural Information Processing Systems, 2019. URL htt.h:ttp///// www.aattiiccooaa...oorroooaoouoa
17


Reviews:
[{"review_id": "eZmUyNCL7e", "reviewer": "Reviewer_4b8Q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The two-phase strategy (GMM soft clustering based on model update first, then loss-based clustering) addresses the limitations of existing loss-based or gradient-based methods.\n2. Solid theoretical foundations link batch size selection with noise reduction, cluster separability, and GMM convergence, enhancing the effectiveness of the algorithm.\n3. Experiments on different datasets, privacy budgets, and data partitions demonstrate its generalizability, with an explicit focus on performance on minority class clustering.\n\nWeaknesses:\n1. R-DPCFL relies on tuning batch sizes $\\left(b_i^{>1}\\right)$, strategy switching time ( $E_c$ ), and cluster count ( $M$ ). While guidelines exist, their optimal selection remains non-trivial for new datasets.\n2. Performance degrades with small local datasets, requiring very small $b_i^{>1}$ for compensation, which may increase computational cost.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Existing clustered Federated Learning (FL) methods in differentially private FL (DPFL) under high structured data heterogeneity are sensitive to DP noise. This paper proposes a novel algorithm for differentially private clustered FL, which is robust to DP noise and accurately identifies underlying client clusters. Key contributions include:\n\n1. Clustering clients based on both their model updates and training loss values. Mitigating server uncertainties when clustering clients' model updates at the first round's end by using large batch sizes and Gaussian Mixture Models (GMM), reducing the impact of DP and stochastic noise to avoid clustering errors.\n2. Providing theoretical analysis to justify the approach and evaluating it across diverse data distributions and privacy budgets, with experimental results demonstrating its effectiveness in handling highly structured data heterogeneity in DPFL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explains gradient accumulation for full batch training in plain language, and provides GMM-based clustering code snippets for reproducibility.\n2. Add experiments on smaller datasets to quantify the trade-off between $b_i^{>1}$ reduction and computational overhead.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "38WgSb3i9s", "reviewer": "Reviewer_B8k9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The paper address an important problem in Federated Learning ie Data Heterogeneity.\n2. The proposed algorithm uses both model updates and training loss/accuracy values of clients, and mitigating noise impacts with large initial batch sizes to enhance the clustering accuracy.\n3. The paper presents theoretical results indicating the positive impact of  large batch sizes on the clustering accuracy.\n4. Experimental results presented on 3 different (small) datasets show that the proposed method performs better than the baselines.\n\nWeaknesses/questions:\n1. The paper uses sample-level DP. However, for FL use-cases user-level DP is more realistic. \n2. The paper doesn't mention anything about participation rate. Did the paper assume full participation rate? If yes, then can the authors talk about the generalization properties of the algorithm to a generic case with partial client participation?\n3. The third stage is very communication heavy as the server needs to communicate M models to all the clients. \n4. The paper doesn't present any theoretical analysis on the convergence rate of the proposed algorithm. \n5. Bar plots in Figure 6 and 7 are not clear to me. What does 10% for IFCA mean? Is full batch-size the main reason why R-DPCFL always determines the right clusters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes DP clustered FL algorithm for heterogeneous data distributions referred to as R-DPCFL. The proposed method aims to address two major vulnerabilities in the clustering process of existing DP clustered FL algorithms namely 1) sensitivity to model initialization, and 2) sensitivity to randomness in clients’ model updates due to stochastic noise. R-DPCFL has three stages of training:\n1. GMM with M components: In this stage, all the clients run one step of DP-SGD with full batch which results in less noisy (stochastic noise). These model updates are then used to learn the cluster centers of a GMM model with M components. The number of clusters (M) is either given or can be found by maximizing the confidence of the learned GMM.\n2. Soft clustering: This stage is run for $E_c$ number of epochs. The server determines the cluster of each client based on $\\pi_i[m]$ determined in the previous stage where $\\pi_i[m]$ is the probability of client $i$ belonging to cluster $m$. Each client contributes to the chosen cluster via DP-SGD model update. Server updates each cluster's model via weighted aggregation of the model updates from the clients belonging to the given cluster.\n3. local hard clustering: In this stage ($E_c to E$), server sends all the models to each client and the client picks a cluster/model that results in minimum train loss/accuracy. The clients select the cluster centers in a private way using exponential mechanism.\n\nThe  paper presents theoretical results showing the noise level in the DP-SGD model updates decreases with increase in batch-size. The paper also establishes that the computational complexity of learning the GMM in the first round also decreases fast as batch-size increases. \n\nThe experimental results on MNIST, FMNIST and CIFAR-10 with a setup of four clusters of clients indexed by m ∈ {0, 1, 2, 3} with {3, 6, 6, 6} clients show that the proposed algorithm performs better than three chosen baselines (IFCA: clustering based on local train loss/accuracy, local training: clients do not participate FL and learn a local model by running DPSGD on their local data, and MT-MRL: learns a personalized model for each client while regularizing the distance from the averaged model across all clients). The experiments focus on covariate and concept shifts to simulate  heterogeneous data distributions. simple 2-layer CNN is used for MNIST/FMNIST training and ResNet-18 for CIFAR-10.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. What is the compute and communication overhead of the proposed R-DPCFL as compared to DP extension of IFCA?\n2. What are the limitations of the proposed method? Please add a section on limitation either in the main paper or appendix. \n3. What is the impact on performance (test loss/accuracy) when there is no stage 3 ie $E_c=E$?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SfYZB9Ha1u", "reviewer": "Reviewer_SLhf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The algorithmic design is well-motivated. The full-batch first round combined with hybrid update clustering directly addresses the brittleness of previous loss-only or gradient-only clustered FL under DP noise.\n- Sound theory that links batch size, DP noise, and EM convergence, providing intuition that is immediately actionable for practitioners.\n- Thorough evaluation with four datasets, multiple heterogeneity types, strong baselines, per-cluster analysis, and ablations on batch size, $\\varepsilon$, dataset size, and hyperparameters.\n\n\nWeaknesses:\n\n\n- All experiments use image classification; evaluation on tabular tasks would test generality.\n- Privacy budgets. Main results use rather loose $\\varepsilon \\in \\{5, 10\\}$; additional runs at tighter $\\varepsilon \\leq 2$ would contextualize utility–privacy trade-offs.\n- The theory might not hold if the number of cluster is mis-specified. \n- Gaussian-mixture separability and homogeneous variance may not hold in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the problem of sample-level differentially private federated learning under structured data heterogeneity. It introduces R-DPCFL, a three-stage clustered FL algorithm that (i) uses full-batch local training in the first round to obtain low-noise model updates, (ii) performs soft clustering of clients with a GMM whose confidence is quantified by a minimum separation score, and (iii) gradually switches to loss-based private clustering via the exponential mechanism.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it would be beneficial if some of the following points are addressed:\n- Include an ablation where some clients opt out of the full-batch round to test robustness.\n- Report results on at least one non-vision dataset to demonstrate broader applicability.\n- Compare with recent personalized DP-FL methods that incorporate clustering implicitly (e.g., FedMix/FedNova variants). \n- Comment on the correctness of theory given an mis-specified $M$.", "broader_impact_concerns": "No concerns. This is a theoretical work that may have societal impacts in the future.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)