Adaptive Clipping for Differential Private Federated Learning in Interpolation Regimes
Authors: Takumi Fukami
Tomoya Murata
Kenta Niwa
Venue: Accepted by TMLR ()

Abstract: We investigate improving the utility of standard differential private optimization algorithms by adaptively determining the clipping radius in federated learning. Our adaptive clipping radius is based on the root-mean-square of the gradient norms, motivated by the interpolation property and smoothness of the objectives. In addition to Renyi Differential Privacy (RDP) analysis, we conduct theoretical utility analysis of the proposed algorithm, showing that our method enhances utility compared to DP-SGD for smooth and non-strongly convex objectives. Numerical experiments confirm the superiority of our adaptive clipping algorithm over standard DP optimization with fixed clipping radius in federated learning settings.

Published in Transactions on Machine Learning Research (06/2025)
Adaptive Clipping for Differential Private Federated Learn- ing in Interpolation Regimes
Takumi Fukami
takumi.fukomi@ntt.com
NTT Social Information Laboratories
Tomoya Murata
muruta
NTT DATA Mathematical Systems Inc.
Kenta Niwa NTT Communication Science Laboratories
kenta. niwa @ntt. com
Reviewed OpenReview:
Abstract
We investigate improving the utility of standard differential private optimization algorithms by adaptively determining the clipping radius in federated learning. Our adaptive clipping radius based on the rool-mean-square of the gradient norms, motivated by the interpo- lation property and smoothness of the objectives. In addition Renyi Differential Privacy (RDP) analysis, ing that our method enhances utility compared to DP-SGD for smooth and non-strongly convex objectives. Numerical experiments confirm the superiority of our adaptive clipping algorithm over standard DP optimization with fixed clipping radius in federated learning settings.
conduct thcoretical utility analysis of the proposed algorithm, show-
1 Introduction
Federated Learning (FL) is an emerging distributed learning paradigm in machine learning. In this frame- work, each client possesses a local dataset, and these clients work collaboratively to train shared machine learning model by exchanging information through communication (Konečný et al., 2016; McMahan et al., 2017). To preserve the privacy of client information, federated learning prohibits direct sharing of local datasets. Instead of sharing the raw dalasets, federated learning algorithms try to learn model only by exchanging the local model parameters or the gradients computed on the local datasets. However, this approach does not fully protect the privacy of the local datascts. In fact, it is possible to extract private information from the communicated local models gradients. Major attacks designed to steal private information include membership inference attack (Shokri et al., 2017; Yeom et al., 2018; Nasr et al., 2019) and reconstruction attack (Fredrikson et al., 2015; Zhu et al., 2019; Yang et al., 2019; Wang et al., 2019b; Geiping et al., 2020; Zhang et al., 2020).
Differential Privacy (DP) has recently received much attention to protect client data privacy against the aforementioned attacks. DP provides framework that measures the extent to which privacy informa- tion leaks when using given random mechanism (Dwork et al., 2006; Dwork & Naor, 2010; Dwork et al., 2010; Dwork, 2011; Dwork et al,, 2014). To ensure the DP guarantees of federated learning algorithm, cxisting works typically rely on the gradient perturbation technique(Song et al., 2013; Abadi et al., 2016; McMahan el al., 2018; Geyer et al., 2017; Triasteyn & Faltings, 2019). For example, Differential Private Stochastic Gradient Descent (DP-SGD) a workhorse DP optimization. In DP-SGD, independent Gaus- sian noise with mean zero added the computed gradient, resulting noisy gradient that then used to update the parameters at each iteration. The algorithm determines the standard deviation of this noise, referred to as the Do-noise size, based the desired privacy level.

---

Published in Transactions on Machine Learning Research (06/2025)
In DP optimization, fundamental trade-off exists between the optimization accuracy (often referred to utility) and the privacy level. Thus, it is crucial to compare the utility of the algorithms at given privacy level in order to evaluate the performance of the DP optimization algorithms. For instance, DP-SGD is known to achieve a utility of O( (dlog(1/))/mm)) with a central (z, 6)-DP guarantce, where d represents the problem dimension and n is the total number of samples. This rade is indeed minimax optimal in non-strongly convex scttings (Bassily et al., 2014). Given certain privacy levels, the development of DP optimization algorithm with improved utility attracts much interest in the field due its fundamental importance.
To enhance utility, a promising approach is to use adaptive clipping (Pichapati et al., 2019; Asi et al., 2021; Andrew et al., 2021; Yang et al., 2022; Bu et al., 2022; Xia et al., 2023). Standard DP optimization algo- rithiss often use gradient clipping to bound the L2-sensitivity of communicated parameters, with a fixed clipping radius (Abadi et al., 2016). In contrast, adaptive clipping dynamically adjusts the clipping radius basod on observed gradient values, leading to improved utility. In particular, iteration-level adaptive clip- ping aims to reduce the DP noise size by estimating the iteration-dependent L2 sensitivity from the gradient information. A key implicit assumption underlying the iteration-level adaptive clipping is the interpolation property of the objectives, meaning that per-sample gradient norms converge to zero the model parameter approaches the optimum. This property is often observed in high-dimensional problems, such as training deep neural networks. With the growing prevalence of large-scale machine learning models, the interpolation property has become much more realistic, further highlighting the importance of iteration-level adaptive clipping.
However, many iteration-level adaptive clipping algorithms are primarily heuristic, lacking rigorous utility analysis and theoretical improvements over non-adaptive clipping algorithms. While empirical effectiveness has been reported, a solid theoretical foundation is often missing. The objective of this study is to develop novel iteration-level adaptive clipping algorithm for federated learning and provide a theoretical utility analysis. Additionally, we conduct numerical experiments to demonstrate the superiority of the proposed method over non-adaptive clipping approaches.
Main Contributions
We propose a new adaptive clipping algorithm for local record-level DP federated learning, which is particn- larly relevant in cross-silo FL. The proposed per-round adaplive clipping radius C, is determined based on at the current global model zy-1, which
the coot-mean-square gradient norm (T//))E<ED. [VV[))>>-]][(((>>>>--) becomes smaller as the optimization proceeds, and reduces the amount of DP noise added to the model parameter in local optimization by exploiting the interpolation property of the objective function, where 7 is the predefined parameter controlling the clipping bias.
We provide not only DP guarantees of the algorithm based on RDP technique (Mironov, 2017), but also a utility analysis. In our utility analysis, the main technical difficulty is to show the utility improvement over DP-SGD by carefully quantifying the trade-off between the reduction of the adaptive clipping radius C, (and the DP noise size) and the clipping bias depending on the choice of r. Furthermore, we analyze the impact of local optimization bias and the stochastic gradient noise, as our algorithm is based on FedAvg.
For smooth convex objectives under interpolation regimes, we show that our algorithm with (c, 8)-D guaran- (ees achieves a utility of ō(e util DP-)GD) with a factor depending on 7 and Typ where DP-SGD = θ ( VP))) is the best-known utility of DP-SGD under local record-level DP settings, and Ty is the mean-quantile ratio as defined in Definition 1. Our result indicates a thcoretical utility improvement over the best-known one when the mean-quantile ratio Tq is not too large, which depends on the model and dataset. To validate this finding, we conduct numerical experiments to demonstrate empirical superiority over DP-FedAvg, which is a natural counterpart in federated learning of DP-SGD.
utill
Related Works
Here, we briefly discuss the relationship between existing studies on adaptive clipping and this study.
One primary motivation for adaptive clipping is to enhance utility compared to fixed clipping methods such DP-SGD and Dv- Fu et al. (2022) studied iteration-level adaptive clipping method for
2

---

Published in Transactions on Machine Learning Research (06/2025)
DP federated learning, utilizing the mean of the (non-squared) gradient norm with heuristic DP noise size scheduling. Despite its empirical efficacy, this approach lacks utility analysis and fails to achieve theoretical improvements over the non-adaptive clipping methods. In contrast, we delve into the theoretical utility of the proposed algorithm and demonstrate both empirical and theorctical enhancements over the non-adaptive clipping methods. Pichapati et al. (2019) introduced a coordinate-level adaptive clipping method based on gradient normalization and de-normalization techniques. Asi al. (2021) proposed PAGAN, which is a combination of coordinate-level adaptive clipping with adaptive gradient methods (Duchi et al., 2011). These studies provided theoretical utility improvements over DP-SGD under the assumption of gradient magnitude imbalance. While coordinate-level adaptive clipping is conceptually different from iteration-level adaptive clipping, it is feasible to merge these approaches. However, our study primarily focuses on iteration- level adapLive clipping.
Another motivation for adaptive clipping is to reduce the cost of tuning fixed clipping radius (Andrew et al., 2021; Yang al., 2022; Bu et al., 2022; Xia et al., 2023). Andrew et al. (2021) studied quantile estimation method to estimate the L. sensitivity of the loss gradient and empirically demonstrated that the hyperpa- rameter of the estimation algorithm was much more robust than using the fixed clipping radius. Yang et al. (2022); Bu et al. (2022): Xia et al. (2023) explored per-sample normalized gradient methods to climinate the need for a predefined clipping radius and reported that these methods achieved nearly the same accuracy as DP-SGD.
Another related study investigated DP optimization under interpolation regimes. (Asi et al., 2022) proposed a DP optimization based on their so-called Lipschitz extension with a decreasing Lipschitz constant under interpolation regimes for non-federated learning settings. While the theoretical result is quite strong, their theory essentially relies on quadratic growth condition, which is much stronger than our assumptions, and it requires huge computational cost due to the need to solve a per-sample optimization sub-problem to obtain the Lipschitz extension at each iteration. In addition, (Béthune et al., 2024) recently proposed a novel algorithm called Clipless DP-SGD, which removes the clipping procedure and its resulting bias in standard DP-SGD by analytically accounting the per-layer Lipschitzness of the neural network during training.
2 Notation and Problem Settings
This section first introduces the notation used in this paper. Then, the problem settings are explained.
Notation. ||. || denotes the Fuclidean Lo norm || lla: ||=||| = E x.. for vector x. For a natural number m, [m]] denotes the set (1. For any number a, b, a v b means maxfa, b) and a b does minfar (a, 6). Clip(x, C)= == min(1, C/||==|>> for x Rd and C O. For any q E [o, 1], (X) denotes the q-quantile of X. R> denotes the set of non- nog-negative real numbers.
In this paper, we consider the minimization of objective functions that typically arise in federated learning under differential privacy constraints.
Objective Function. The objective function is defined as f(-) = (1/P)EnE(P) So()) fp(+) on Rd.. Here, fz(:) is the local objective associated with client and is defined by (1/mp) EIED, (a(-), where Dp is the local dataset of client and no is its size, and lg is the loss function associated with sample z. Let n and np be [D] and [Dp] respectively.
Differential Privacy Constraint. We focus on local record-level differential privacy (also known as Inter- Silo Record-Level (ISRI.)-D (Lowy et al., 2023)), which plays a crucial role in private cross-silo federated learning. Local record-level privacy assumes that both a central server and clients are honest-but-curious, ie., the server is not trustworthy. Then, given a mechanism M, it is required to guarantee record level (c, diffferrrlt privacy² for all the outputs of client p with respect to each local dataset D, for every E [P], that is P(M(D,) E S) < {PP((D))) E S) + 6, for every S and every adjacent datasets Dp
I Although we focus on record-level differential privacy, it is straightforward to extend our analysis to the case client-level differential privacy.
3

---

Published in Transactions on Machine Learning Research (06/2025)
and Dz. where > 0 denotes the distinguishable bound of all outputs on two adjacent data subsets and 8 E 10 represents the probability of information leakage.
We introduce an important notion referred to as the q-- ratio, which plays an important role in our theoretical analysis. In the following, let Qq denote the q-quantile of the input values. Note that we always have the loose bound (x) S n.
Definition 1 (Ta-quatille-mann ratio). For given q E [0, 1], we define Tq : Rd R20 as Ty(2) == ((((IIV (())) [V1(())]?)
To analyze the utility of our proposed algorithm, the following thcoretical assumptions are required. With the exception of Assumption 2, our assumptions are fairly standard. Importantly, in our DP guarantees analysis, Assumptions 1-4 are not necessary.
A Assumption (Convexity of (z). & conver on Rdd for every 2 E D.
Assumption (Interpolation Regimes (Vaswani et al., 2019)). There erists an optimal solution x. of f, i.e., F. E arg minzezs (x) with ||x.|| < B for some B > 0. Moreover, it holds that Eg 0 and l (z(x.) = 0 for every E D.
This assumption is a key to our utility improvement. It is well-known that interpolation often occurs in high-dimensional problems including (approximated) kernel methods and over-parameterrzeel deep learning (Jacot et al., 2018; Allen-Zhu et al., 2019). Here, we do not assume the strong growth condition or its variants, which are much stronger than Assumption 2.
Assumption 3 (G-Lipschitzness of (z). le is G-Lipschitz, i.e., (=(x))((()) S G|| yll, for every I, y E Rd and 2 D.
Assumption 4 (L-smoothness of l2). le is L-smooth, i.., [V)2(z) (+,(1) s L| wl1, for every 2. y E Id and 2 E D.
3 Approach and Proposed Algorithm
In this section, we present our strategy for incorporating adaptive DP noise in federated learning.
3.1 Review of DP-GD and DP-SGD
First, we briefly review the algorithm of DP-GD and DP-SGD (Abadi el al., 2016), which are the gold standard methods in DP optimization based on non-adaptive clipping, to clarify our approach.
In DP optimization, the rumning algorithm is required to be (e,6)-DP, and a gradient perturbation tochnique is typically used to satisfy this constraint in federated learning, where privacy preservation is required even during training time communications. Gradient perturbation adds some random noise to the aggregated gradient to satisfy the DP constraint for each iteration. For example, DP-(S)GD has the update rule of 2, 2r-1 - T(gr + Er), where 9r is the full gradient (f(=-11) for DP-GD and is a minibatch stochastic gradient for DP-SGD, r) is a learning rate, and & is Gaussian noise with mean zero and variance o², which is called DP noise.
For DP-GD, the noise size a for (e, 8)-DP depends on the number of iterations R, dataset size n, privacy parameters e, 5, and L2 sensitivity A of gr- Specifically, it is known that the DP noise size to satisfy (c, 8D is = OOA ( The L2 sensitivity of 9r is formally defined as A = suPAn(D. (D) g. (D')), where the supremum is taken over every pair of adjacent datasets (D, D') and (D) denotes the gradient computed on dataset D.
A standard sensitivity analysis requires the uniform boundedness of the norm of the per-sample gradients, i.e., (x,-)i & C for every sample z E D and every parameter -1 for some C 2 To satisfy this requirement, a typical approach to use gradient clipping; gr == (1/n) predefined C. Then, can get simple upper bound A < 2C/n.
C) for some
We say that datasets D= = {= and D' = [=02 are adjacent if dD D') 1, where da is the Hamming distance between D and D' defined by dn( D') Σi 2./7/
4

---

Published in Transactions on Machine Learning Research (06/2025)
3.2 Our Approach
Gradient clipping requires the clipping radius to execute the algorithm. Most the previous methods focus on using a constant C. However, this may degrade the utility of the algorithm; clipping bias occurs when (2r-(1111)) >> C for large polion of D, and the privacy budgel is wasted in vain when [[V(2(xr-1)) < C for every 2 e D. Thus, it is desirable to adaptively determine C based the observed per-sample gradients (1 for cach round to obtain better utility.
Since (()){a( depends on fr-1 and changes across rounds, it is desirable to make the clipping radius depend on the parameter 2,-1. This type of clipping referred to as iteration-leva adaptive clip- ping in this paper. A basic approach to rcalize itcration-level adaptive clipping to use the q-quantile Q(((IVV((())=))))>>>>) with some q E [0, 1] for constructing the clipping radius for each iteration. How- ever, direct use of the quantile is difficult because the quantile information is much more sensitive than the standard aggregation results such as the mean, which results in large DP noise. To mitigate this problem, several estimators have been proposed in the previous study (Fu et al., 2022). However, these are heuristic and do not provide theoretical improvements.
We propose an adaptive clipping radius C, based on the key quantity v (T/n) > LEED VV given hyper-parameter T 1, which can be an upper bound of Q2(1I for some q [0, 1] with high probability. To guarantee (s, OEP of this procedure, we need to add DP noise g(P).C, because (T/n) EeED ([2((r=))) already contains private information. The behind reason for using this quantity is that (1/n) EIED (x, < (I holds in general for L-smooth convex f in interpolation regimes. This means that C, can adaptively decrease depending on the current model Iy-1 as f(z goes to fzz)) in optimization. Then, the L2 sensitivity of C, and thus the DP noise size also decreases, and we expect better utility than non-adaptive clipping methods.
for a ())) with adequate T
3.3 Proposed Algorithm
Algorithm AdeptDP-FedArg
1: for Prppe E [P] in parallel do 2: Compute of and op based on (1).
end for
Ty = Tint-
for r to R do
for p [P] in parallel do
C.P).C ftt Exeroonc
where 1.27.0 Dr with size bC.
8:
end for
C. C- ---120 /27 2T Ep + ( + +t) G, where clo).c (000009)80
10:
for p E [P] in parallel do
(p)
11:
-o Ir
12:
for k 1 to K do
13:
xkr--1 1/ E.errh)
(p)
Clip (l (p) (x (=) Cr), where T(p) kr--1 ~ Dp with size b
(p)
(p)
n + (p) <(p>_)), where <lp)_, (C..))
14:
I - -
15:
end for
-(()) R k== "k-1,r-1"
K (p)
16:
17:
end for
18:
I, =, Clipp + Σ pE 2 (p)
Krr-ii B).
IT (out) = EpE [(p)
19:
20: end for
21: (out)
Erein L (ouE)
22: Return: FR x (out),
5

---

Published in Transactions on Machine Learning Research (06/2025)
The concrete procedures of our proposed method are given in Algorithm 1. In each round, we first compute Cr. Each client p computes GfP) based on local minibatch gradients with size bC, which is the average of the squared per-sample gradient norm with & constant clipping radius G Then, the central server aggregates GfP)C +g(n)C and constructs C, as in line 9. Here, c.P).CC (O, (of)²) is Gaussian noise with mean zero and variance (o DP noise to protect the privacy of G(P).C. The constant i is also added to account for the stochastic noise due to the minibatch sampling and the gradient deviation [V because we use a common C, for each client p and each local iteration within round r. Reusing the clipping radius in a round is desirable as it reduces both the number of privacy information releases and the computational cost. After calculating Cr, the server sends C, to each client, and each client. runs local optimization based on the adaptive clipping radius C.. The local optimization follows standard DP-SGD with minibatch size b and DP noise < - A key distinction is that o, depends linearly the adaptive clipping radius C, instead of an estimated Lipchitzness parameter G. The resulting parameters {<R---1)PE(P) are aggregated by the server to form the next round global parameter ry. The specific values of {op DPE(P) and {Tp]pE[P) to guarantee (e, ()P will be provided in the next section.
Remark 1 (Additional computational and communication costs). Compared to DP-FedAvg, constructing C, requires additional computational and communiiaatin costs. However, these cosls are oflen negligible becouse (i) C, is computed only once per round; (ti) Cr allows relying on minibatch sampling, which further reduces the computational cost; and (iti) G(P).C is scalar and the communication cost is only ((P) instead of (Pd)), which is quile small.
Remark 2 (Hyperparameters).
The hyperparameters of Algorithm 1 are 11, T and G. Since the hyperporam- eters of DP-SGD are n and fired clipping radius C, our algorithm requires only one more hyperporameter than DP-SGD. Note that i' can be theoretically determined by Proposition 2 with u = 0.01, for example. The parameter clipping radius B never affects the DP guarantees of Algorithm 1 and the clipping procedure is only necessary for lechnical reasons related to ulility analysis. Therefore, in practice, the clipping in line 18 can be ignored.
4 Theoretical Analysis
In this section, we provide theoretical analysis of our proposed algorithm.
4.1 DP Guarantee Analysis
We investigate the DP noise size Φ and a to guarantee (e, (DP of Algorithm based on RDP theory (Mironov, 2017) and incorporates the subsampling amplification technique (Wang al., 2019a).
Given any oo] + (0, oo] and 7 (0, we define S(E,7) : o1) + (0, oo] as
S((():))(o) a a-1 log 1 + (1+00(0 min (1(6212) 1), (2) minmin - -1)2))
()=0) min (cs1x0)111 - 117
3=3
which is defined in (Wang et al., 2019a). Tn the following analysis, we assume that the minibatch sampling is based on uniform sampling without replacement.
Proposition 1 (One-round RDP analysis). Suppose that b S np for p [P]. Let AC = 2/1C, g/ == 60 //pp, A = 2/6 and 7op == b/ng- We define e, ([(a)= == al /2(0((((00)))) C, is (a, ()) (0),)))) ()))) and given x--1 and Cr, (fx k,r - IRE[K), a(?)) is (a, KS((p()) (a)) for any integer a 2 2.
Given Iy-1,
6

---

Published in Transactions on Machine Learning Research (06/2025)
Theorem (DP noise size). Let a, == 1 + (2)og(1/))/s) Then, the output information of client p [P] in Algoritlom 1 is (s,6)-DP, if
c == min m fo 2 0 (), _ Etarget I
(1)
Op min RK (sp((s S Etarget
with Starget = e/4. In particular, the DP noise size of and op in (1) satisfy
V/llog Apt
Vlog
op
0
DEVE
(2)
VKRlogt Upt
V Vlogg by/5
Op o
under 8 < Op
A a.(o;))
Ang
b
Rp
4np a.
and 0llog(16))
Zea
Zea.
Remark 3. When the computation of Cr involves full-batch gradients rather than minábatch ones, replacing AC = 2/(b)) and (8(e(() S Etarget in (1) with A := -2/up and Rey (a.) S Étarget respeclively
gives (s, ()- guarantees, and (2) still holds.
4.2 Utility Analysis
In this subsection, we derive convergence rate and utility of Algorithm 1 with (s, 6)-DP. In the convergence analysis, the minibatch sampling is assumed to be uniform sampling with replacement for simplicity. Also, to simplify the arguments, we assume that the gradients in the computation of C, are deterministic, i.e. we focus on the full batch cases, and np = ny holds.
Notation used in utility analysis. (1/np) EXED, (x) and (x) = (1/P)EpEPP) fe())) Also, (E)(2), (x) and Vfc de- note min (I (1/mp) EMED, (x) and (1/P) Eper V (x)) respectively. We de-
Let (x) min (,) f
fine (0 = (1/P) (09)), σ² = Ep/ oz E = (1/P) Ep ا (p).C and E = (1/P)
(p)
-k Tlooe denotes ni Tl'
The following proposition ensures that for at least a (= fraction of the per-sample gradients, our adaptive gradient clipping with radius C, only reduces to 1/ //VVV (Tg(#+ of the original size. Also, it provides theoretical suggestion for parameter V.
Proposition (Simplified version of Proposition 4). Let E 10, be sufficiently small, q E [0, and > 0. Suppose that Assumptions 3 and 4 hold. With G > G and appropriate n S. Ō(1/(KL) A vP/(v Ko²dL)), with probability at least 1 - u, for every k E [K], r [R], it holds that when we set Di = / //10/(2P//)))
VIV (T,(T((,,)))//
To derive convergence rate, we first derive a single round bound. Unlike the standard analysis, we develop a bound for fCr rather than the original f. Also, our analysis relies on the arguments from (Woodworth et al., 2020) to bound the localization error due the use of multiple local updates (K 1).
Proposition 3 (One-round Analysis (Simplified version of Proposition 5)). Suppose that Assumptions 1, 2, 3 and 4 hold. Let 10 be sufficiently small and q [0, With probability at least u, Algoritho satisfies
K Σ E Cr (T)- ( E Ō
- | - 12 22(m) -2
k=
Ve (2()) (p)
Stoc. Error.
Tip PElP
min 2
zED,

---

Published in Transactions on Machine Learning Research (06/2025)
for every r E [R] under T $ Ti and & ((L/(v /T /TKL) A 11/11111T //T /TV Ko²dL), where squix :=
c Tq 1r)). Here, A K + K Lo + (n) KoPd/P) + (Pb) and Stoc. Error.
2m Σ #k-1 Zx. 9k B (1 pelp [ 1
Then, we incorporate the adaptive clipping radius Cr, which includes another DP noise & into consideration in the inequality derived in Proposition 3. Finally, applying this inequality recursively leads to our main result.
Theorem 2 (Convergence Rate (Simplified version of Theorem 3)). Suppose that Assumptions 1, 2, 3 and 4
hold. Let u E (0, 1) be sufficiently small. Then, with probability at least 1 - u, Algorithm with i as defined in Proposition 2 and (oC,0) ,)) as defined in Theorem 4, satisfies
cr (Fk-110-1)
KR
<ō
Zini
Ixll B²
TVRG?
(222L++ + nKRd
"
+ (nKL+ ))
nd big
)
njKR
Moo v Pg
nin Pg. pb
under & 2 G with G = - T Timax and
S δ
Mloe
2 PE
Pb (TT)
[[[TKL A
KGB ^
Φ(T) RdL ((T)
where ((T) == V Tiiax /T (TImax) (b(T)= v T T and T is defined in Proposition 3.
Note that Theorem only guarantees the convergence of fcr rather than f. To obtain meaning- ful results, we use a fact that there exists De C D such that |D9] Z qn and S fDa(Tx-1,-1)) s under suitable conditions, where S = (1/191))een (x))) Note
I Ol that when g = 1, D = D and thus we can always have a lower bound of fcr based on f. Using this fact, from Theorem 2, choosing appropriate n, K and R, which minimize the convergence error yields the best achievable utility of Algorithm with (e, ()PD)-D guarantees, summarized in the following corollary.
Ol
(CT
Corollary 1 (Utility bound (Simplified version of Corollary 2)). It is assumed that || xx-ll B, G and L are @(1). Then, for any q E [0.5, 1.0], there erists Dy C D such that DDg 2 g and Algorithm with (P guarantees satisfies
fDr (z(ou))
rinax
7 DP-SGD 10 (6())TVP)
DR-SGDD VT (CuP + (CDp
Tito + d
1v
(cuR
(DD-SGDD
d3
dè
dh
+ (
(T() (Eutil
DP-SCD
(cutit
DP-San
(d(T)d) 1 (5
DP-SGD
(c mnnsans))
dbb
db
under the conditions in Theorem 2, where fxa(x) == (1/1011) ESEDi (x(x), -D = and ((T), (T()) are defined in Theorem 2.
PP))
Remark 4 (Interpretation of Corollary 1). Corollary 1 gives utility bound of fDs, which represents the averaged risk on some dataset D C D with [Da] 2 qn. At first, setting 1 gives a utility bound of the original objective J, since D = D always holds. In this case, selling T = ((Tmmax) minimizes the utility bound. However, setting 9 < 1 can yield a much better bound because mmax may be much smaller than -imux, at the expense of the replacement of f with fD.. This observation suggests that using T & Timax can be beneficial in practical situations. This point will be further explored in Section Note that our algorithm does not explicilly depend on the choice of q, the bound given in Corollary 1 holds every q E [0.5, 1.0) simullaneously.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Remark 5 (Comparison with the best known utility bound). The best known utility under local record- level (e, (E is Catll DDDSGD == (tvd/(muuVPPP for non-strongly conver objectives, which is achieved by DPD Compared with this rale, we can see that all the terms of our bound have o(DD) thanks to the interpolation property and our adaptive clipping scheme. Thus, our method can achieve better theoretical utility than the best-known one of DP-(S)GD in our problem settings. Although each term has factor depending on Tq Ty and Time, it also has a factor of 1/d for some a > 0 except for the last term, problems. In Section 5, we will empirically confirm that T is not
which is favorable in significantly large.
5
Numerical Experiments
5.1
Experimental Setups
To empirically demoustrate the elficiency of the proposed AdapiDP-FedAvg, as detailed in Algorithm 1, several experiments were conducted.
Comparing Algorithms For comparison, we use De-FedAvg (Abadi et al., 2016) with fixed noise size as the baseline. However, the original amalysis of De-FedAvg did not incorporate the subsampling amplification technique, which we include in our analysis in Section 4. To ensure a fair comparison, we conduct a new study to determine the appropriate noise size for De-FFedvvg, along with its update rules, as presented in Algorithm 2 in Appendix D.
Task Scttings To compare the performance of Algoithmss-NapiiP--Foal -eed.Avg using adaptíve noise size versus Avv employing fixed noise size-we we prepared three privacy levels for each datasel, as shown in Table 1. Based on our understanding, conventional studies on model training using DP typically follow one of two evaluation approaches: i) setting the target privacy level (c,5) first and then computing the corresponding noise size 0, e-g., (Pichapati et al., 2019; Geyer al., 2017; Triastcyn & Faltings, 2019); or ii) setting the noise size in advance, which implicitly determines the resulting (c, e-g., (Abadi et al., 2016; Bu et al., 2020; Fu et al., 2022). While the number of data samples and rounds needed for model convergence vary across datasets, the latter approach is more practical for testing purposes, and we therefore adopted it. As shown in Table 1, we first set three fixed noise sizes for DP-FedAvg on each dataset. For a fair comparison, the corresponding privacy level e was computed while fixing = 10-4, using (3) in Appendix D. Then, (p,,o for DdedtDP-FedAvg was determined, as in line 2 in Algorithm 1. Note that noise sizes differ between AdaptDP-Fed. Avd and De-FedAvg as in Table 1, the same privacy level (s, is theoretically ensured.
In total, we prepared four benchmark tests, denoted T1) T4). For T1), which uses an artificial dataset, we evaluated the function 1=(2nuatu)( - (2 Z) - b), where represents the reference parameters, x the model parameters, and Z the data samples. The dataset was randomly generated with Z E R In addition, we utilized real-world datascts, including T2) Bank Markoting (Murain = 45, 211), T3) MNIST (Nurain = 3, 000), and T4) MNSH (Merain = 3, 000) (Xiao et al., 2017). Separate test datasets were prepared for each task.
For the convex model, we employed two-layer Multi-Layer Perceptron (MLP) with fixed (i.e., untrainable) first-layer weights to ensure convexity. This design choice is motivated by our utility analysis in Corollary 1, which suggests that a larger model parameter size d can lead to a smaller upper bound on the utility for convex objectives. In our sctup, the hidden layer dimension was set to 128 units for T2) and 512 units for T3) T4). The training dataset, consisting of Purain samples, was partitioned across P = 2 clients such that each client holds Amin = [ samples. Test accuracy was recorded at the center server at the end of each round. As additional experiments, we conducted evaluations with p = 4, 6 clients, as well as experiments using a non-convex model. These results are provided in Appendix E.2 and Appendix E.3, respectively.
We configured (R, K) for each dataset to ensure that the model parameters approached convergence using minibatch size of b 100 and 1. Under this condition, we performed
Hyperparameter Tuning
3nttps://archive.. www.eiiviiiieee.... www...ci..e./eeeeeee. www.eeed/aattttt///22 httpee//e//e/..annaaaeeeiinngngn

---

Published in Transactions on Machine Learning Research (06/2025)
Comparison of noise sizes for each privacy level and tested hyperparameters used for tuning, across each method and datasct. Noise sizes in AdaptDP-FedAyy and DP-FedAvg chosen to ensure the same privacy level (e, 8), as described in Section 5.1.
Table 1:
Datasct
Privacy level
DP-FedAvg
AdaptDP-FedAwg
Tested hyperparameterrs (C,
T1) Artificial dataset
Lv.1
= 0.003
(dp.ae) (0.004),0.00097))
Lv.2
dp 0.006
(5p. g (0.0078, 0.0021)
(0.1,0.3.0.5)
{0.5, 1,3,5
Lv.3
dp 0.01
(Gp. Pp ()) (0.012, 0.0043)
T2) Bank Marketing
Lv.1
= 0.006
(Op, gg (0.0077, 0.0025)
Lv.2
Pp 0.01
(Op, og C) (0.0103, 0.0078)
{0.01, 0.03,,03,0..1
{0.05,0. 0.1, 0.5)
Lv.3
Pp 0.03
(0624,0.010) (op, ()
T3) MNIST
Lv.1
dy 0.003
(Op, o (0.093, 0.0021)
Lv.2
= 0.006
(dp, o
{0.1,0.3
00.05,0.1,0.5
Lv.3
dp 0.01
(Ap. p (0.013, 0.0050)
T4) MNIST
Lv.1
Op 0.003
(Op, op (0.0041, 0,0011)
Lv.2
0.006
(Op. o (0.093, 0.0021)
(0.03,0.1,033)
{0.05, 0.1,
I..3
0.01
(0.. (0.013, 0.0050)
Table 2: The lowest training loss and the highest test accuracy R rounds with selected hyperparameters. For T1), test accuracy is omitted since it is not classification task.
Dataset
Privacy
De-FedAvg
AdaptDP-FedAAg
level
Trainlass TeainlosssTestAce AestAAc
Selected (C, m)
Trainlass/ TrainlossTesstAc Aoe
Selected (G. v)
TI) Artificial dataset (R, ()50k)==11000))))) 20)
Lv.1
6.73e 20.29 /-
(0.5,
3.34e 20:43
(1,0.1)
Lv.2
2.69e 20.21
(0.5,0.3)
3.81e
(3,0.1)
Lv.3
7.62c 20.24
(0.5,0.3)
1.82e -
(3,0.1)
T2) Bank Marketing
3.13e
3.130==40.00///
(0.05, 0.03)
3.12e
=0.02 /89.43 #0.14
(0.05,0.03)
(R, (1)000, 226)
3.12c +0.17/8 26
(0.5,
3.12c 3.126 +0.04/ /8.7888.7 73 +0.42
00.1,
Tw.3
3.13e 10 3..3e--1o.110 11/.......999999
20.40
(0.05, 0.03)
3.13e g0. 13/.13//9..0 89.155...16
(0.1,
T3) MNIST
Lv.1
4.84e 4.81e-4+o.10 +0.10 /61.50 +6..00
(0.05,0.3)
4.25e +0.41, /56.00 +0.50
(0.05,0.3)
(R, K) = (1000, 15)
Lv.2
5. 10 30.31 /64.75
27. 25
(0.05,0.3)
4.38e
1 10. ma/59.00 11.00
(0.05.0.3)
Lv.3
4.06c 41.44/57.35
40.
(0.1,0.1)
4.02e
10.00/60.30 /60.30 21.20
10.00/
(0.05,0.3)
T4) FashionMNIST
Lv.1
2.02e 2 +1.07/57.00 /57.50 17.50
(0.05.0.3)
5.96e
14.01/ 22.1/68.50 68.50 27.30
(0.1,0.3)
(R, = 15)
Lv.2
3.39c 2 11.10/60.00
(0.5,0.03)
2.94c
#2.30
/6/ 67.20
(0.1,0.3)
Lv.3
1.28c +0.22/29.50
(0.1,0.03)
1.27e
+0.00/32..0 /3.80 40.70
+0.00
(0.05, 0.3)
experiments with several hyperparameter settings, as shown in Table 1. The results obtained with the optimal combination of (C,G) and to achiove the lowest training loss are summarized in Subsoc. 5.2. Further details are provided Appendix E.
5.2 Experimental Results
(a)
(b)
1x10
Ixt0
Extol
Ivi0]
Training loss
1x1d
Training loss
Ixin
1xto-
1xt0
2x10
1x10
1x1073
1xt0
1x10-4
Li MMM
5
5
100
Round
Round
(a,b) AdaptDP-FedAvg - DP-FedAvg
Figure Training loss curves with sclected/tuned hyperparameters. (a) corresponds to the T1) artificial dataset, while (b) corresponds to T3) MNIST under privacy level Lv.3.
10

---

Published in Transactions on Machine Learning Research (06/2025)
(a1)
(b1)
3.5
0.05
Clip radius
Clip radius
0.04
2.0
0.02
0.5
w/o. Mm A An
0.01
0.00
750
1000
Round
Round
(a2)
(b2)
quantile- quantlie-mean meee ratio To
quantile-mean ratio To
5
4
2
500
750
1000
Round
Round
(2)
Q=1.0
Q=0.9
q=0.8
Q=0.7
q=0.6
q=U.5
Figure (a1) Transition of clipping radius C, and (a2) transition of Tg for various
(0.5(median))...... ..., 1.0 (max)} in AdaptDP-FedAAe with well-tuned hyperparamers on T1) artificial dataset under privacy level Lv.3. (bl) and (b2) present the corresponding results for T3) MNIST under privacy level Lv.3.
Table presents the experimental results obtained using the selected hyperparameter settings that achieve the lowest training loss for each method. A comprehensive version of the experimental results, including various hyperparameter scttings (c.g., lcarning rate 7,, noise size d, clip radius C, is summarized in Table 3 in Appendix E.1. From Table 2, we observe that AdaptDP-FedAvg consistently achieved lower training loss than De-FedAvg. Lower training loss generally corresponds to smaller gradient norms, which numerically confirm in the following paragraph for part of tasks. Since the noise size in AdaptDP- FedAvg is determined by the gradient norm, it typically injects less noise size compared to De-FedAvg. This reduced noise injection may be key factor contributing to the superior performance of AdaptDP-FeddAvg Thus, AdaptDP-FedAArg was empirically effective under the conditions of lower training loss (i.c., smaller gradient norms). In some cases, the test. accuracy of AdaptDP-FeddAr is lower than that of De-FedAvg, which is likely due overfitting. Note that our theory primarily focuses on effectively reducing training loss, and low test accuracy in some experimental scttings does not contradict that the variance of DP-Fed OP-FedAvg is very large (e.g., 5.5 and 7.25), whereas the variance of AdaptDP -ooP--FeAvg remains small. While DP-FedAve Avd may appear to achieve better average performance, this is likely due to a few seeds that happened to yield exceptionally good results. Therefore, it does not indicate that DP-FedAvg consistently outperforms AdaptDP-FedAyy
theory. It should also be noted
To further analyze the experimental results, Fig. presents the convergence curves using training loss. Fig. shows (al) transitions of the adaptive clipping radius C, in AdaptDP-FedAArg and (a2) transitions of the quantile-mean ratio Ty for T1) artificial dataset, while the corresponding results for T3) MNIST are shown in Figs. (b1) and (b2), respectively. As shown in Figs. (a2) and (b2), Ta is approximately 1.0 when 0.7, indicating that 70% of data samples have gradient norms smaller than the average gradient norm. Furthermore, when 2 0.8, does not diverge but instead converges to a moderate value. This observation supports Corollary 1, which theoretically asserts that the training loss can be lower compared to DP-SGD, Empirically, as shown in Fig. 1, the training loss of AdaptDP-Fedd improvement can be attributed to the adaptive changes the clipping radius C, as the communication round progresses, rated in Figs. 2(a1) and (b1). In contrast, DP-Fed employs fixed clipping radius
lower than that of DP-FedAAg This
11

---

Published in Transactions on Machine Learning Research (06/2025)
throughout the training process. When large proportion of data samples undergoes gradient clipping, performance will be degraded. However, this result. suggests that as the training progresses, the number of data samples that remain unclipped increases, leading to improved performance of Adapt Moreover, through our experiments, we obtained empirical insights indicating that r = 1 and approximately G 0.05 lend to yield good performance for the models and datasets we used. However, the optimal values of r and & vary depending on the models and datasets, and there currently no principled method determine them universally. Similar to learning rate selection, a greedy search remains necessary.
6 Conclusion
We proposed AdedlDD-Fed,Avg, method designed to enhance the utility of standard differential private optimization algorithms by introducing adaptive clipping radius in federated learning. Our adaptive clipping radius was determined based the root-mean-square of the gradient norms, motivated by the interpolation property and smoothness of the objectives. We theoretically demonstrated the privacy analysis of the pro- posed algorithm in general settings. Furthermore, for smooth and non-strongly convex objectives, we showed that our method achieves an improved utility bound compared to DP-SGD in specific cases, Through sev- eral numerical experiments, we confirmed that our AdaptDP-Fed.e outperforms standard DP optimization (DP-FedAvg) with a fixed clipping radius.
References
Martin Abadi, Andy Chu, Tan Goodfellow, H Brendan McMahan, Nyya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Securily, pp. 308 318, 2016.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International conference on machine learning, pp. 242-252 PMLR, 2019.
Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswany. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems, 34:17455 17466, 2021.
Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. In International Conference on Machine Learning, pp. 383-392. PMLR, 2021.
Hilal Asi, Karan Chadha, Gary Cheng, and John Duchi. Private optimization in the interpolation regime: faster rates and hardness results. In International Conference on Machine Learning, pp. 1025-1045, PMI.R, 2022.
Raef Bassily, Adam Smith, and Abhradcep Thakurta. Private empirical risk minimization: Efficient algo- rithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pp. 464-473. IEEE, 2014.
Louis Béthune, Thomas Massena, Thibaut Boissin, Aurélien Bellet, Franck Mamalet, Yannick Prudent, Corentin Friedrich, Mathieu Serrurier, and David Vigouroux. DP-SGD Without Clipping: The Lipschitz Neural Network Way. In ICLR, 2024.
Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J. Su. Deep learning with gaussian differential privacy, 2020. http//1.///11/.....///
URL
Zhiqi Bu, Yu-Xiang Wang. Sheng Zha, and George Karypis. Automatic clipping: Diffcrentially private deep learning made easier and stronger. arXiv preprinl arXiv:2206.07138 2022.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Cynthia Dwork. A firm foundation for private data analysis. Communications of the ACM, 54:86 95, 2011.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Cynthia Dwork and Moni Naor. On the difficulties of disclosure prevention in statistical databases or the case for differential privacy. Journal of Privacy and Confidentialityy, 2:93-101 2010.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pp. 265-284, 2006.
Cyuthia Dwork, Guy Rothblum, and Salil Vadhan. Boosting and differential privacy. In IEEE Annual Symposium on Foundations of Computer Science, pp. 51-60, 2010.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9:211-401) 2014.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic counterneasures. In ACM SIGSAC Conference on Computer and Commuunicationss Security, pp. 1322 1322-1333, 2013.
Jie Fu, Zhili Chen, and Xiao Han. Adap dp-fl: Differentially private federated learning with adaptive noise. Tn 2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TruslCom), pp. 656 663. IEEE, 2022.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? In Advances in Neural Information Processing Systems, volume 33, pp. 16937 16947, 2020.
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXie preprint 1i12:111..070 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael Jordan. A short note concentration incqualities for random vectors with subgaussian norm. arXie preprint wxXi:1902.033966 2019.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint 1XX0:1160000492, 2016.
Andrew Lowy, Ali Ghafelebashi, and Meisam Razaviyayn. Private non-convex federated learning without a trusted server. In International Conference on Artificial Intellgennce and Statistics, pp. 5749 5786. PMLR, 2023.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efiiieend learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, volume 6, 2018.
Πly Mironov. Rényi differential privacy. In IEEE Computer Security Foundations Symposium, volume 30, pp. 263 275, 2017.
Tomoyta Murata and Taiji Suzuki. Diff2: Differential private optimization via gradient differences for nonconvex distributed learning. In International Conference on Machine Learning, volume 202, pp. 25523 25548, 2023.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprchensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In IEFE Symposium on Securily and Privacy, pp. 739 753, 2019.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Venkatadhecraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXiv preprinl or Xiv: 1ec:1908..00 008.07643, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEFF Symposium on Security and Prinacy, pp- 3-18, 2017.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In IEEE Global Conference on Signal and Information Processing, pp. 245 248, 2013.
Aleksei Triasteyn and Boi Faltings. Federated learning with bayesian differential privacy. In 2019 IEEE International Conference on Big Data, pp. 2587 2596, 2019.
Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of sgd for overpparameterized models and an accelerated perceptron. In The 22nd international conference artificial intelligence and slatistics, pp. 1195-120 1204. PMLR, 2019.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant, In International Conference on Artificial Intelligence and Statistics, volume 89, pp. 1226-1235, 2019a.
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Beyond inferring class representatives: User-level privacy leakage from federated learning. In IEEE INFOCOM Conference on Computer Communications, pp. 2512-22222 25200, 2019b.
Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33:6281-622 6292, 2020.
Tianyu Xia, Shuheng Shen, Su Yao, Xinyi Fu, Ke Xu, Xiaolong Xu, and Xing Fu. Differentially private learn- ing with per-sample adaptive clipping. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 10444 10452, 2023.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie Yan Liu. Normalized//lippel sgd with perturbation for differentially private non-convex optimization. arXiv preprint arXic:2206. 18033, 2022.
Ziqi Yang, Jiyi Zhang, Ee-Chien Chang, and Zhenkai Liang. Neural network inversion in adversarial setting via background knowledge alignment. In ACM SIGSAC Conference on Computer and Communications Security, pp. 225 240, 2019.
Samuel Yeom, Trene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: An- alyzing the connection to overfitting. In IEEE Computer Security Foundations Symposium, pp. 268-282, 2018.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion allacks against deep neural networks. Tn IEEE/CVF Conference Compuler Vision and Pattern Recognition, pp. 253 261, 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, volume 32, 2019.
14

---

Published in Transactions on Machine Learning Research (06/2025)
A Preliminary
A.1 Review of Rényi Differential Privacy
Our DP analysis relies on Rényi diffcrential privacy (RDP) technique. This sub-section is excerpted from (Murata & Suzuki, 2023), which summarizes several known results about RDP used in our theoretical anal- ysis.
Definition 2 ((a, ()-,P) (Definition 4 in Mironov (2017))). A randomized mechanism M D - R. salisfies (DD (a E (1, and = > 0) if for any datasets D, D' D with du(D, D") 1, it holds that
M M(D')(o)
log BOD
where (D) denotes the density of M(D) at Lemma (Post-processing Property of RDP (Mironov (2017))). Let M D + R be (a,E)-RDP and 9 R + R' be any function. Then, go M: D + R' is also (R.E)-RDP.
Lemma 2 (Composition RDP Mechanisms (Proposition in Mironov (2017))). Let Mr R1 X x R,-1 D + R, be (a,Er))RDP for r [R]. Then, M + R1 x RR defined by M(D) = D)) is (a, Er= -
(D)
Lemma 3 (From RDP to DP (Proposition in Mironov (2017))). If a randomized mechanism M is (a,E)-
RDP, then M is (e + log(1/6)/(o 6)P for every 5 10,
Definition 3 slyssiiiti (h) == (D,')(D [l/(D) - h(DD'llq is called lq-sensitivity for function h, where the marimum taken over any adjacent datasets D. D' E D.
Lemma (Gaussian Mechanism (Corollary in Mironov (2017))). Given a function h, Gaussion Mechanism M(D) h(D) +O, ofT) satisfies (a, (2)))))))o))RRPPP for every E (1,
Lemma (Subsampling Amplification (Theorem 9 in (Wang et al., 2019a))). Let M be a randomized mechanism that takes a dataset of b S n points as an input and 7y == b/n. M Musbsample, be defined as: (1) subsamples, subsample "n points without veplacement from the inpul datasel with size n, and (2) apply M taking the subsampled points as the input. For every integer 2 2, if M is (a, (a))) M subsample, is (a, (a,s((a))-RDP, where
8' e'(a) < log 1 (1 min 4(c2(2) - -1), -4(E20))11,,e
(()) min (6 -1)3}
cl 1)6() min (c=(20) - -1)3
j=3
We can derive simple upper bound using Lemma 5.
Lemma 6 (Subsampling Amplification (Simple Upper Bound)). On the same settings as in Lemma 5, if s(a) is monotonically increasing with respect to a, it holds that
2 4+ 2) (2)
é'(a) < 3
a -
under e(a) s 1/3 log(1/(27a) and 7 < (22)((ca) for any > 0.
Proof. See the proof of Lemma B.8 of (Murata & Suzuki, 2023).
A.2 Concentration Inequalities
In this subsection, we provide known concentration inequalities for norm-subGanssian random vectors (Jin et al., 2019) used in our analysis. Note that any random vector X satisfying || s a 20-norm- subGaussian (and any g-bounded centered random vector c-norm-ubbGaussian) Thus, the following concentration inequalities is also applicable to bounded random vectors.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Lemma 7 (Martingle version of Azume-Hoeftding" incquality for norm-subGaussinn (Corollary 7 in Jin et al. (2019))). Lel XI...., Xa be random vectors in Rd. Suppose that {X;) and corresponding filtra- tions (3.)==1 satisfy the following conditions:
E [[ 81-1] and P( 2 s 81-1) S 2e 207
E [n]
for fixed {a:)-=1- Then, for any u E (0, 1), with probability at least holds that
2d
X
S
Σ
0 log
U
where c == V2.
Lemma 8 (Martingle version of Azume- Hoenee--e inequality for norm-subGaussin (Jin et al., 2019)). Let
XI...., Xn be random vectors in IRd. Suppose that {X;}=_ and corresponding filtrations (3:)21 satisfy the following conditions:
E[X 31-1] and P([|Xe] 2 (1-111 S 2e
20
V A.ER [n]
for random variables {o; {o with as E 8i-1(i E [n]). Then, for any (0, and A 0, with
probability at least 1 - q it holds that
Σ of 2
Σ Xi
Sc max Σ of.aa )
Eoj.a
log 2d
loglog A )
where C c:= V2.
Remark 6. In Lemma 8, in practice, we can choose sufficiently small a O and a deterministic value A > such that Σn= < A and then we have essentially the same bound as Lemma because 1 l/a and A will always have only polynomial dependencies the other parameters such as K, R, P, n, G, L, Cpron, d, 1/s and 1/6 in analysis. This approach sacrifices some rigor but yields correct results and simplifies the analysis.
B DP Guarantee Analysis
B.1 Proof of Proposition 6
Here, we focus on DP analysis with respect to client E [P].
RDP Analysis of C,
Let minibatch (P)) be given. Then, we can see that the L2 sensitivity with respect to client E [P] of G(P) is GG where A == L. because |_!P) (4) c.P).C (I') s 202 /bc for adjacent minibatches I and 1', where GfP). (I) denotes the averaged squared per-sample gradient and 1, we conclude that C, is
the samples I. From Lemmas 4 (a) (a)) where (a) == a A 2) (2)/((((()))
Then, we apply the subsampling amplification technique to the proven results for the fixed minibatch. From Lemma 5, it holds that for every integer a 2 2, C, (a, ((,,(( (a))--DDP with oo := /n/pp-
RDP Analysis of (p) (=l/-1
In the following, we omit round index We consider RDP property of 2 conditioned on C, and 2.(2) To realize this, first, focus the L2 sensitivity of gk conditioned the previous randomness. Since 0k is
16

---

Published in Transactions on Machine Learning Research (06/2025)
constituted of b samples, we have that 119k (1) oh (T'Il < 2C//b for adjacent minibatches and I', where (P) (I) denotes the averaged gradient on the samples I. Thus, the mechanism that takes minibatch with size as an input and returns the sum of the average of the clipped per-sample gradients and Gaussian noise e(p)
9k
SR ~ (0, of leads to (a, Ep(a))-RDP for every (1, oo) from Lemma 4, where Sp(a) 20/ (6 Then, from Lemma 5 and Lemma 1, taking into account the subsampling amplification effect, we immediately obtain that = is (a,S( Yp) (p) (R))-RDP for every integer a 2 2 with "pp == b/np-
Using the composition theorem of RDP (Lemina 2) finishes the proof of Proposition 6.
B.2 Proof of Theorem 4
Let a 2 2 be some integer. Combining Proposition 6 with Lemma 2, holds that the outputs of client. p [P] is (a, RRS(s (), Tp (a) + RKS((e()))))))) (a))- - From Lemma 3, we can see that the outputs of client p E [P] is (s/2 + log (1/0)/(a 1), DP under RS(E)()) (f)))))))))) (a), RKS(Ep()) (a) & E/4 4 = Étarget Thus, using a, := 1+ [2 log /e] implies (D..).DP. Thus, we only need to determine the minimum noise size of and Op satisfying RS(ep ())1))) (a))(a) S Etarget and RKS(Ep()), (a)) S Etarget as (1).
Finally, we estimate the order of the noise size. Then, we only need to determine the minimum noise size op satisfying K RR of the noise size op. Suppose that
(a.) S Etarget in (1). We first estimate the order (a..) Etarget and of satisfying Re
K |K (mpp)2 (1mpe)))
log b
of =
is sufficiently large and
b I Rpp
4np
A
20ax.
Since 27,a, I 1/e, it holds that log(1/(27,a.)) 2 1. Thus, E(a.) = 2b./()o0)) S 1/3 A log(L/(27,a.) is satisfied under E = (log(1/6)). Also, note that 7/p S 4/(a,b"ap) = E(2) a. holds. Thus, the conditions of Lemma 6 are satisfied. Finally, from Lemma 6, we have
Enp (a.)
2 (a.) 50
(
1 ppa)5((2)
=0
a+ -
under 8 = (llog(1/)).
Thus, choosing appropriate on gives K ()), S Starget. Similarly, setting
Rlog (mpe)2
log (bc)2E)
(00)² =
is sufficiently large and
8c < np
4np
2ea
aa (o())
gives (S(E 3p (a.) Etarget
This finishes the proof of Theorem 4.
C Utility Analysis
In this section, we provided utility analysis of Algorithm All the minibatch sampling is assumed to be uniform sampling with replacement. Also, to simplify the arguments, we assume that the gradients in the
17

---

Published in Transactions on Machine Learning Research (06/2025)
computation of C, are deterministic, i.c., we focus the full batch cases. Moreover, assume that np ny for every p.p' E (P) and define Moc Up-
First, we give some notation used in our analysis.
Notation
Let ey (x) min min(1,CC//lV/((()))))))) (x) = (1/np)E ExED, (x) and (0 == (1/P) pelly fo (x).
Also, V (z) denotes min C/IV Similarly, V (x) denotes (1/np) Exen, (T)()) and
(x) denotes (1/P) EpE(P) VI (z). Note hat (e)(2) Ve (I) in general.
Also, we define E. = (1/P) Ep- dl.cc E = (f - (00)2 = (1/P) Ep- and o² = (1/P)
As long as no confusion, we omit the round index For example, we write (p) Algorithm 1.
in r-th round of -ky-1 I (p)
C.1 Useful Concentration Results
Lemma 9. Suppose that Assumption holds. Let E 1). With probability least u, for every [K], r E [R] and p E [P], it holds that
Σ
Σ
Σ Cli(Ve,(a2)))...
np =ED,
=EI!
VKGC,
Ōō
vb
Proof. Observe that (V (x (,) np EtED, CLip(VV(() C.M S 2GC, Applying
Lemma 8 with A = 2KPG and sufficiently small a > and taking union bounds over [K], p E [P] and [R] give the desired result.
Lemma 10. Let u E (0, With probability at least w, for every & E [K], E [R], and E (P), il holds that
- Σ Clip(V(,(=()))..) Sō
C
Σ
b
Tpp ZED,
vb
=ET(")
Similarly, with probability least 1 u, for every E [K], r (R), it holds that
Σ CIp((Ve((a)))...)) 1 Σ Chp((e s sō (VPb)
C. {0(p))
np zED,
PE(PI =E
Proof. Applying Lemma 7 and taking union bounds over E [P], E [K] and r E [R] give the desired results.
Lemma 11. Let u E 104). With probability at least 1 - 4u, for every p E [P], t E [K] and r E [R], it holds that
(oo)2
(c()).0 < Ō 5 =5
18

---

Published in Transactions on Machine Learning Research (06/2025)
1 (p) 1 s Solo,C.Va), 1ER.--11150 <ō (
and
Clip(gh + Sh.r-1 Cprot) = -h.r-1 + c(p)
(p)
Skir-1
under Gproj 2 ov,aj).
Proof. Since g/m.C N(,, of we have P(|sft). > s) S 2exp((s>/(((o for any 0. This implies 15 I &2 /log/o/(2/u) with probability at least u. Taking union bounds over p E [P] and r E [R] gives the desired result. The proof the case of EC is perfectly same. Thus, the first result is proven. Similarly, we have (s(e) 11 < 20 log(2/u) with probability at least - Taking union bounds with respect to j E [4], E [P], E [K] and r E [R] gives the desired result. The case Exr completely same. This gives the second result. The last result is trivial since C, < G.
In the following, we always assume that the events defined in Lemmas 9, 10 and 11, which simultaneously hold with probability at least 1 - 7u for any u E (0,
Remark
Due to a technical reason, we modify Algorithm of line 16 as follows: -R,--1 11-1 Cllip(o + <(p)
-k.--1: 1..r-1:prroj for Cproj > 0. First. note that this modification does not affect the DP guarantee anal- ysis thanks to the post processing property of RDP. We particularly use sufficiently large Cproj := @((1 + maxpe(P) Vd)d)). Under this choice, we can see that (i) the sequence of model parameters { is bounded almost surely, which is necessary to utlize Lemma 8, and (ii) the clipping does not happen for every and r with high probability from Lemma 11. Practically, we do not need to implement this clipping process thanks to (ii). Additionally, the choice of Cproj is quite robust. because Cproj only depends on the convergence rate with log-log order. Thus, we can omit this clipping process practice as in Algorithm 1.
C.2 Analysis of Clipping Radius
Lemma 12. Suppose that Assumption holds. For any 21, Rdd and C > O, we have
min(L,C//Vee(=>)) < min(L,C/\VVL + L| 22ll
Proof. We will prove the lemma by considering different cases below
(i) (222222 S
In this case, Cin < min(L,C//VV,(<>)l| holds.
(ii) [e and [+:]l <
Observe that
minn2, (x2) -
[V22(x2)
|| (x2) (xi))
L||m] - x2ll
C
and thus min(1,CC/IV(((m))) [min[1C//|V(((>>]]]]]
-
19

---

Published in Transactions on Machine Learning Research (06/2025)
(iii) Ve (x2)) C. and [21((=))] C
In this case, we have
-
min(I,C//VV<(()))) - min(1,//1VV((===)]
[[ (x2]
[[)]) (x2)))
The remained proof is similar to the arguments of case (ii).
Lemma 13. Let E (0, 1/11). Suppose that the requirements in Proposition are satisfied. Then, with probability 1 1lu, for every [K], TE [R] and RR,, it holds that
1/ KC
1/ K2?
K
| Boll 2 0
K0
fp (x
Pb
P
pE|Pl
In porticular, we have
10s - zoll2 =ō (22K3t
n2 T2Koodd?
P
(p) Here, 3x fx== 7 EpEpp o Similarly, have
| z 11p1 00 C
for every p E (P].
Proof. Let r be fixed and we omit index r. We define gk := + Epe 9h and EN == + EmE(p) 5 Observe that
Toll²= = ||ā
Toll2 + 77(āx-1 - To, T6 x2 1) + 1²² [2 - Tz-
1 + K || -Bol? - (7x - Zu, E(gg])
1
| 1 Toll2 27(Øx-1 Zo. 9h E[ge]) 27 Z0. Ex)
K
+ 0172)
1Eig + (0(1)))19 E[gal12
+
|| -
æoll 2| + O0 K [|
K
| 20 - 217( To. 9h Eigx]} 2n( Io. Ex)
2K
+ +0(1))
Eigx]11
1 +
|| Toll 12
||
zol12 - 2
To. 9h EEgh]} 2n( Io. Ex)
2K
+0 17202
K)
-
V (xtt))
Ph
PE[P)
Here, for the last inequality, we used Lemma 10.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Now, we use this inqquality recursively. Since (1+ 1/K** S = O(1), we have
1
Toll
& Σ
k
k
$ 2K
Tkk-1 - zoll3 21
((k)( Io. 9k Elgal) 2n Σ (xxx - 20. Ex)
x=1
K)
(x
+0
lp
Ph
[P[P]
where o(K) = (1+ 1/K)
Then, applying Lemma to the third and fourth terms of the right hand side, with probability 1 -2, we get
k
CP
21
2 (Tx-1
Io. gx - B[g+])
Ō 2m
æoll2
pb
x=1
1)² KCK
xol1
Pb
4K
and
&
O2Cd -7.
2n
Σ
o (k) - To.Ek)
5Ō
211
Σ
((K)))|Txx111 - Foll2
P
k=1
J - xoll?
P
4K
where ,² == (1/P) og. Here, note that it holds that | Toll S K proj.
Hence, we obtain
PPKC3 Pb
Koo
1H =0
+((PK)
Σ
PE(P
Noting / (of2) < C, finishes the proof of the first result. The proof of the second result is completely same as that of the first result.
Proposition 4. Let (0,1/11), q [0.1] and 0. Suppose that Assumptions 3 and hold. Under G and O(1((L))))))OOOOO WP/AAPP////// Ko2dL)), with probabilily at least 1 11u, for every k E [K], r E [R], it holds that
Q. S / IV (a( Cr,
when we define
C₂
2r
Σ CUpp(Vic((a.-)))P.3)) + +7+E nnc.
21

---

Published in Transactions on Machine Learning Research (06/2025)
where
(o0)2 0²
2PR
E ;== c
log
P
and == = (1/P) 5h) Here, c= 22.
Proof. Let p. and be fixed. T denotes 2 -kr-- From Assumption 4, we have [=xx(== =
V[ (To))l + (1- 4) [+ - (To))) - (1+ (z)) + L3 (1+ || -oll? for
any = E D, p E [P] and 7 > O. Also, note that (i) vlg(o from Assumption and G G. Thus, have
)EED)
(1 + (To)
(Z) + (1+ ! L xx-1 - -2ol2
Σ
P
Rp =ED,
PE[P]
(1 + (T0)
Σ (62((r-1)11 1]1
(+ 7
_ δ 1²K2112 +
'2KL'O'
C²
P
Up zED,
P
pE[P]
1 + 1 o
(1 (1+
(2r-))))11
+
P
Pup
16
PE[P]
ZED,
for some p E [P]. Here, the third inequality holds from Lemma 13. The last inequality holds by choosing appropriate 7 S O(I((KL) vP/(/KodL)) Thus, setting = 1/3 gives
(1++) o
I (g
O2 +
2
16
{1 Tol To( C
s max
Here, we used the fact that 72 & from 11.
This finishes the proof of Proposition
C.3 Convergence Analysis
Proposition (One-round Analysis). Suppose that Assumptions 1, 2, and 4 hold. Let a E (0, 1/11) and g 1). With probability at least 11u, Algorithm 1 satisfies
22

---

Published in Transactions on Machine Learning Research (06/2025)
1k,r-1- - -x.11
K
< |0o - x.112-5(11 -1(1)
Σ
Elsc. (fx- B]
k=1
1f Kood
r²K
+0
T2 L +
C2
P
Pb
Σ
1e (1(() (p)
Ω
pE[P] np zEDp
min (1, C, C,
K
2n
Th 1, Ih [(1(e) (160) 1C(F) (
K
2n
1Tr- - :o. 11)
k=1
for every r E
[R] under T S Tmax and S O(1/(VTTmx /TKL) A 1/(V /T Ko3dL), where s ==
max Ty (T)-1).
Proof. First note that from Lemma 11, it holds that Clip(ok (p) probability.
+ c(p) 1 Cpros) =
(() - + c(p)
with high
We define 3s = 1 Eperyy 2 T = Eperty 9% (p)
Since E (r(p)) [Tx] (1/P) PElP 7 we
have
1h 1=11
| - -val? 185-2111 -21(Tx=11 Tk) -21(5x-1 -r, + 11x + 21
Σ V.S (o ) 41²
< | 1 - -xaa12 -221 2n1
Jx- Io,; p pE(M
(
Σ
V v.p (
pE
2n i - Fa. 9k - E B )e((p) 19kl) - 2n7 xK- - Ex)
+ +112 0N
E (((00) 1 [
+ +21221 1Eal1
Note that
1
Σ
VI (a(()
Tk-1
Ina P
PE(PY
1
(p)
Fk 1, Ve
V
x. -
(p) 3e (x
P
np
PE[
zED,
Since
K-1 - 12-1, (lr(o(2))) S (e((a - (0xx111) + L
|102/01 - 12-
x
2
from Assumption 4 and
(p) 11 v(s(((2)))) (2. - -(((f
1
23

---

Published in Transactions on Machine Learning Research (06/2025)
from Assumptions 1, 2 and 4, get
(B6-1- ve,(o - (# 2 110f2 Ik 111
4
Ve, (x 11.
2L
Multiplying the both side by Cim (x
1 we have
Tk-1 - Fo, - (2)
L (p) < - (I) +
11
VC (ofi)1.
=2
2 min (o
Now we bound the first term. Suppose that From Proposition 2, for any Dr and p [P], have [[x S (IV(((6-))//)CC, and thus holds that
( 2 1/
Choosing appropriate
/T) KoždL), from Lemma 12, we
have
min(1,C./IV(L(/---)))) < mi((1,Cc//V4(/221111 D| (p)
- Ix-1ll
C,
O K
Thus, we get - m1, (1,
-mm.1..c.//V444442)))44420. (3/4)) (Ix-1).
In summary, we obtain
ăt-1 -o, -05 (-111
< Cr (Tx-1) + L 22- 2 m1nn C
|
VC (p) 112
2
Averaging this inequality with respect to = E Dp and p E [P] results in
1
Ik-1 Vo:
V
(1
[E
< 3
3
C. (āx-1) +
| |a
(p)
12-1
2
P
[P[
Σ
|V 11.
2L PE[P np sED, min C,
Next, we bound
E (1(0)) pelal [on
[ax ||
C
[0x E B(gge)
pb
24

---

Published in Transactions on Machine Learning Research (06/2025)
from Lemma 10.
Since
A Σ
(- )
"p sED,
PE[M]
pe[r]
combining all the results yields
1 - x.112
31 jc. (āx-1) + 1 /
1| -1.12-
nT
Σ
|o-l1 112
PE[P]
"1 E 1 Σ
1
Σ
VI (p) 11?
2L P
np
min [1,
pE[PY
DED
21
TR-1 - Iz. Ik
E )
[ul)
2n (T=1= Fz En)
r102Pd
+0
+
C
Ph
under n 1/(4KL).
From Lemma 13, we have
Σ 1
|a
(p)
Σ
-
(p)
221'=00(1p'''' x +n²Koad) C²
p2
[P[
P.P'E[P]
and then, we get
[ - x=1l"
| xx.11= - -c.11=-S((1)) Cr (Tx-1)
12 L oOdd
1/2
+0
C?
P
Ph
- Ω 7 ト
(p) 11
pE[Pi Ppp ZED, min 1, C./l (1 I|B
2m T - Fa, Ok E. (1 DeEm [xx) - 2n - Ex.
Recursively using this inequality and rearranging the result gives
25

---

Published in Transactions on Machine Learning Research (06/2025)
|ix - v.,1?
K
< 100 x.112 11 - -((n)
|B]
k=1
KKKodd 1p
+
3 K x Lo2d
C
11
Pb
Ω
1
V (a (p)
(x1=))1 112
Rp =ED, min C+
(x (p)
pE[P)
21
Tk-
No. 9k
R
Jccee) [x)) )
2n
(T
Fo. En)
This is the desired result.
Theorem 3 (Convergence Rate). Suppose that Assumptions 1, 2, 8 and 4 hold. Let u E (0,1) be sufficiently small. Then, with probability at least - u, Algorithm 1 with defined in Proposition 2 and (a ,a) defined in Theorem & satisfies
Cr (ok-1x-1)
KR
k=1
<ō
Fini
12 B²
TVRG2
I1kKL+
n Rd
7 Ph
+ [nL + 1 nd
) 12g
nRKR
MoeePe
n Pg
P
under G 2 G with G=O(G), T T Tiaax and
n 3 δ (O(T)K A
1
Mooe
mic Pe
pb (TT
KGB ((T) v RdL
(((T
where Φ(T) = /Tppax/ /T + (T1/ (bT) = /Tpuax and mmax is defined in Proposition 3.
Proof. From Proposition 3, we have
||,, - xx
K < | - xx.12- ((2n) Σ
(Ix (fx--(())))r-1)) E
h=1
KKod
r2K
+0
+
C²
P
Ph
Ω
(2 (p)
- 112
P
up zen, min (1, /...c/1/Ve
K- x-1r-1 1
(p)
k=1
PP(P]
K
2m
Th 1, Fo. -h.r- (( D)mm [e.--
K
2n
( 2z. Ek.r-))
Here, we used the fact that [[Clip(æ, B) xzll S |1 2.Il for any Rd and ||>.|| S from the elementary projection property.
26

---

Published in Transactions on Machine Learning Research (06/2025)
Recursively using this inequality from 1 to R yields
IR x.ll"
R
K
S $1000 - x.112 -Q(1)
E
B Cr (Tx-
E)
==
n Kood
+0
1| K +
Σ
c
P
Pb
r=1
R K
Σ sED,
1
C
(a
(==-1--)) 1l
(p)
Rp
min
(1, (x (p)
11
pE[P]
1 pejpy (
2n
2k-1,r-1 2x g
E
2nj
1Tk-1,r-1 Io, Ex.
Observe that | x2112 11100-1111000-1111111 20
PK (0 =,
+ +B²) s
δ 2 d= + B²) from Lemma 13 with high probability under nKG S B.
Then, applying Lemma 8 to the third and fourth terms of the right hand side, with probability 1 - 2u, get
K
K2
C ph
2n
1k-11
Ix, -
Eig 11
5 17 ΣΣ
+ B²
1 V ad
K
5ō
Σ C.+0(B3)
PVb
Ph
r=1
11 Kodd
K
R
Σ
C. C2+0(B3)
P
Ph
r=1
and
K ΣΣ
K2
dOdd
1.1-1 Fa, Ekr--1)
5
211
B
1/2 Ko
R
Thus, have
27

---

Published in Transactions on Machine Learning Research (06/2025)
12 x.11
R K
$ 1100 x.ll2 ==1l=+0(BB) - ((1))
E Cr (Tk- E]
17 Kodd
r2
+0
L
Σ C2
P
Pb
R
- 2
(12-1,--111?
1|??
PE[P]
n
=dD,
k-1,
From the definition C, and Proposition 2, for each round r it holds that
Oº =
Σ
+0(TV)
P
PE[P]
2T
Σ
Σ
(
=
P
up sED,
pE[M)
27 / (1(e-1))
Σ np
mi O(T))
PE[P]
=ED,
Here, for the incquality, we used Proposition 2.
Since Tmmax choosing appropriate S
A 1/( T/ /T Ko3dL), from Lemma 12,
we have
min(1,C./IV4((6--I]] < min(1,C./1VV//22_))11 1 Ix-1ll
C,
[ 1,
Observe that
<
hanp::////.....ll111......1----- 122 12
< O(min (1 +
(0) 1133
0 (1
(a ( K2 + PKDO30C))
Here, for the second inequality, we used Assumption 4. Furthermore, if appropriately choose S T
and 17 < (100 1/ Kozdl.), we gel
V Tixxx
C.
50
50
(2()) (11 1 +(TT)
K
pE[P! np
zeD, min(1, (
28

---

Published in Transactions on Machine Learning Research (06/2025)
Let
KKod
rf
A 1=>KKL=k-'LOM
+
P
pb
Then, if we appropriately choose satisfying
max
A
5
K
to cancel out the term Ö(AC)), from Proposition 3, we obtain
1 - Ix 12 11:0- x.11² + - -((n)
Σ
Σ
(3x-
r=1k=
which implies
jor (5x-1r1-1
KR
1==
Fini 12112 + B²
TUA
n.R.
Ō |iini 1.112 + BB
TV TV
bood
P + ps. )
+
nKR
VP
Ō
Fini
I. 112 + B
TV RVG
12
n
+ (PKLd++T))
TjK R.
Moe
Pg
10c P&
ph
Here, we used
R
10 ===0
(Moo=))
and
f ==
KR
1
(MincE)2
1128
Finally, we summarize a sufficient condition on IF
KGB ((T) Rdl. n 5 [[
1
n PE (TT) RRIL.
nloe
A Pb
((()L.)
where ((T) = / max /T (7 and ((7) = /Tma
This finishes the proof.
Corollary 2. It is assumed that || xx B, G and L are @(1). Then, for any q E [0.5, there exists D, C D such that Dyl 2 4) and Algorithm 1 with (s, ODP guarantees satisfies
JDz (out)
VIV
maxx
T
uti
DP..GDD
10
(D(T)TVP) dd
DP-SGDDS (cuth
VT
(Cril
(DD-SGD)
7 DD-SSDD
(cutl
T
db
di
d
(E
(Eutil
DP
DP-
(())
DDDS (cuti
r
(Eril
DP.SGD
+ (b( (PP
dt
dt
(o(T)d)
under the conditions in Theorem 2, where fDr(x) = (1/1 E (x(x) and @(Va/(moeVPel).
DP-SGD
29

---

Published in Transactions on Machine Learning Research (06/2025)
Proof. Let q [0.5, 1]. First, note that
[K] (T ( vI
To( Cr
D 2 FED.Vr SED:WEEll E
satisfics [D9] 2 D from Proposition 2..
Hence, when we define
fD= = | DE
1
Σ (x)),
from Theorem 2 we have
(2(uu))
fDv
R
K
R
K
R
Cr (TR-LN-1)
|iinl
12 12 B²
TVTCC
1222IL+
nK
pb + [nKL+ 1)
md
nRR
MooC PE
1 Pe?
bZE
1V
max
Zini 11
+ B²
Tnk TnKRAGd
TVRG
(12K2L+
"
+ InLL+
+ 1
nd b2e
nKR
(Moc p
Tilc VP
pb
p
T max
1
TnK Rãd
TVR
T2 +
1 Pb
(Λı nK
1 md
RKR
(Mic v /P))
Moo Moxe VPs
Here, for the first inequality, we used convexity of fDs. The second inequality holds because
fDo (2 2
le (Ix-1,1)11)
zED* Do
Ia (#r-1)
min LmnLL.C//VV44/.-----1111...
ZE D2
2 1 V Ta (#y-1)
min
n
RED
so VI (1( for (FA-1011)
To derive the best achievable utility, we need to determine the optimal choices of n. K, and R.
We first determine R.
R Ō
(M) =: Ri is sufficient condition for 1st term > 2nd term (Td) (1))))
R < δ 3 (,)
VPes
R2 is a sufficient condition for 1st term
3rd term,
30

---

Published in Transactions on Machine Learning Research (06/2025)
Also, from the condition in Theorem 2, need
R [(T)((n)'d
(Minee))
R3,
R R
(Mic)PP)) o(r)nka
R4.
Thus, if RI, R2, R3, R4 2 1, setting := R1 A R2 A Rs A R4 gives
(out)
DD
74
max
0
RKR.
T
<ō
Tg
(Td)
rinK
T
1
1 Pb
+
1 d (
+
(nK) 3 (Mo VP))
(Moc v PP))
K (moc v PP)
P BZE)
(())d
((k)
(Mloc 2P))2
(M))0))
Note that 3rd term can be arbitrary small by reducing y and increasing K for any fixed optimal choice of (n))), when we focus the asymptotic utility bound. Thus, we ignore this term. Now, we need to determine the optimal choice of nK.
.nK nnKs
d3 (nK) is sufficient condition for 1st term 2nd term,
(miwe PP)
nk 5
(nK)2 2 is a sufficient condition for 1st term 5th term,
o(r)+ VPH
nk Ô
(Nin PP)) (nK)s is sufficient condition for R 2 with sufficiently small r and Vrd
large K,
nk 5
(Moc vP)} PP))) =: (n))) is a sufficient condition for R₂ 2 1,
nk $ Ō
Nlaee o(r)v/))
=: (n))ss a sufficient condition for R3 2 1,
nk < (Mioe VPe)) (nK)) is a sufficient condition for R4 2 1.
((r)d
Also, from the condition on in Theorem 2, need
nk δ ( ((()) b(T))
KPh
(nkK).
Thus, setting (nK). (nK), A (nK)2 (nK)s ^ (nK)a ^ (nH)) ((nk))nnnnn))eeoo A (nK), yields an asymptotic utility bound of
31

---

Published in Transactions on Machine Learning Research (06/2025)
x (out)
mmxx
(Td) 8
(T))
1v
+
(nK (Moe P
(Nie VPE)
max
(Td)
(T)d
IV lvve
(nK) (moov
(MocVPe))
mmax
rd (Nioe p
7 d (Mine VPe) 18
(Moe PP)
(Mone) (nioe PP))
0( Vd
&
(rd) g
((()d
(Moce) (Moc /PE)
(Moc /PE)
(Moc VPe)2
max (o (o(T)7²
T
SGD to + (p(T)TVP) DP.SGD $
V
1b (DPSGD
util
(Ctill
DP.SGD
$
(Cuil
di
dt
di
d
(T))
d3
DP.SGD z +
d$
(ttil
D
(d(r)d)
$
(c
util
DP-SGD g + (c) DP
This finishes the proof.
De-FedAvg
Here, we describe the implementation details of DP-FedAvg, which is natural baseline in our experiments.
Algorithm 2 DD-FFedAvg
for to R do
for [P] in parallel do
Compute op based on (3).
(p) = -1
4:
5: for k: == to do
Sample minibatch (p))
Dp with size
7:
Ikr-1 E.ero,) k-
(p)
Clip( Ve, (x) (x._1,r-))
(p)
I -kr-1 = I -k-l---1 n + (p) <())_)) whore c/op- ~ N00
8:
end for
10: end for
11:
T, + EpEp K
(p)
12: end for
13:
Return:
The concrete procedures of DP-FedAvg given in Algorithm 2 for record level centralized DP.
As in Section 4, we assume that the minibatch sampling is random permutation. The following proposi- tion and theorems are natural counter parts of Proposition 6 and Theorem 4 respectively, The proofs are completely same as the ones of Proposition and Theorem 4, and we omit them.
Proposition 6 (One-round RDP analysis). Suppose that b S np for p E [P]. Let A == and Tp := b/np. We define (a(a) aA²/(20f) Given I,-I, ( BReeK] is (a, ( for any integer a 2 2.
32

---

Published in Transactions on Machine Learning Research (06/2025)
Theorem 4 (DP noise size). Let a, == + [2log(1/6)/s] Then, the output information of client p [P] in Algoritlom 1 is (s,8)-DP, if
op :p:=mink 1 20 (KS(ep())))))))) (p)() S Starget) }
(3)
with Etarget = E/2. In particular, the DP noise size op in (1) satisfies
|KRlog
log 1
=0
bi/e
under b S Fp 200.
A
4 0.0g)
and ε (l1g(1)))).
33

---

Published in Transactions on Machine Learning Research (06/2025)
E
Additional experiments
E.1
Hyperparameter Tuning
We briefly describe the hyperparameter tuning process for both the conventional DP-FedAvg and the pro- posed AdaptDD-Fed.Avg. Both methods require setting the clipping radii C, G, as well as the learning rate T, prior to model training. These optimal values generally depend on the target privacy level and task. Therefore, we performed an empirical search to identify the hyperparameters that minimize the training loss for each dataset, using the three privacy levels listed Table 1. The results of this hyperparameter search are presented in Table 3. For cach privacy level and task in Table 1, the configuration that achieved the lowest training loss is used to report the final results in Table 2.
Table Training loss for each hyperparameter setting tested,
Datasct
Hyperparamcter
DP-FedAvg
AdaptDP-Podd/M
7
C' or G
Lv.1
Lv.2
Lv.3
Lv.1
Lv.2
Lv.3
T1) Artificial datasct
0.1
0.5
3.88c
3.89c
3.91c
3.88c
3.92-
3.93e
(R. (1500)
0.1
8.1le
3.24e
9.01e
3.3de
4.72e
2.34e
6.77e
2.71e-1
7.53e
5.02r
3.81e
1.82e
0.1
1.88e
7.52e
2.09e
2.48e
1.82e
5.30e
0.3
0.5
6.73e
2.69e 5
7.62e
4.29e
9.13e
1.14e
0.3
2.80e
1.12e-4
3.15e
3.96e
1.69e
2.56e
0.3
2.52c
1.01c
2.83c
3.69c
1,66c
2.49c
0.3
2.80c
2,80c
7.86c
3.94c
1,61e
2.40c
0.5
0.5
1.36e
5.44e
1.61e-4
1.37e
1.77e
2.27e
0.5
5.08c
2.04e-1
6.19e
1.36c
6.20e
8.91e
0.5
4.57e
1.84e
5.59e
1.18c
6.80e
1.09e
0.5
1.27e
5.10e
1.55e
7.85e
5.99e
8.85e
T2) Bank Marketing
0.01
3.15c
3.16c
3.14c
3.15c
3.17e
3.14c
(R, (1000 226)
0.01
3.16e
3.14e
3.16e
3.14e
0.03
3.13e
3.14e
3.14e
3.12e
3.13e
3.14e
0.03
3.21e 1
3.13e
3.13e
3.15e
3.12e 1
3.13e
0.03
3.15e
3.12e -
3.14e
3.15e
3.14e
3.14e
0.1
0.05
3.15e 1
3.14e
3.14e
3.14e
3.13e
3.15e
0.1
0.1
3.15c 1
3.14c
3.13e
3.15c
3.12c
3.16c
0.1
0.5
3.16c 1
3.14c
3.15c
3.15c
3.12c
3.16c
T3) MNIST
0.1
0.05
6.91c
5.96c
5.62c
4.49c
3.49c
5.70c
(R, (1000))=--100.)11 15)
0.1
0.1
1.54c
2.86e
4.06e
3.27c
2.22
2,51c
0.1
0.5
7.80e
7.98e
1.28e
2.63e
2.03e
9.16e
0.3
0.05
4.84e
5.57e
7.79c
4.25e
3.38e
4.02e
0.3
0.1
4.57c 2
3.10c
9.31e
4.30c
3.97c
4.96c
0.3
2.97e
4.40e
6.32e
2.0de
4.04e 5
8.36e
T4) PNSSI
0.03
1.30c
1.45c
1,25c
-1
1.37c
(R, x(==(00, 15)
0.03
1.38e
1.05e
1.28e
9.65e
1.09e
1.45e
0.03
0.5
2.06G 22
3.39e 2
8.64e -1
1.03e
1.97e
7.84e
0.1
0.05
2.39e
1.23
1.52c
1.62e
1.04e
2.04c
0.1
0.1
1.97e
4.82e
1.60e
1.54e
8.07e
1.78e
0.1
0.5
3.26e
9.57e
9.9de
4.1le
2.30e
9.48e
0.3
0.05
2.02e
4.23c -
5.94c
2.55c
7.45c
1.27e
0.3
0.1
1.52c
3.94c
1.80c
5.96.e
2.94.e
8.10c
0.3
0.5
1.37e
3.14e-1
8.15e
4.05e
3.53e
2.97e
E.2 Evaluation under Varying Number of Clients
Here, we describe how the performance of DP-FedAvg and AdapiDP--eddAvg changes with varying numbers of clients. We empirically evaluated the training loss for each dataset using 2, 4, 6 clients. For each datasot, we set the number training samples per client to Merain = 3000, 6000, 9000 for P 2, 4, clients, respectively. The results are presented Table As shown in Table although the training loss varies depending on the number of clients, AdaptDP-FedAAy generally outperforms Fed
34

---

Published in Transactions on Machine Learning Research (06/2025)
Table Training loss for each hyperparameter setting tested.
Dataset
Num of client
DP-PedAvg
AdaptDP-FedAwg
Lv.1
Lv.2
Lv.3
Lv.1
Lv.2
Lv.3
T Artificial dataset
6.73e
2.69e
7.62e
3.34e
3.81e
1.82e
(R, ((..))=(11) (150, 20)
4.44c
1.78e
4.93e
2.00e
2.49e
1.23e
3.05e
1.22e
3.38e
1.39e
1.72e
7.76e
T3) MNIST (100, 15)
4.84e
5.57e
4.06e
4.25e
3.38e
4.02e
3.79e 5
4.28e-5.
7.53e
3.09e
3.38e
4.1le
1.99e
4.29e
1.19c
1.60e 3
4.5le
T4) MNSHI (R, (1000 15)
2.02e
3.39e
1.28e
5.96e
2.94e "4 1.27e
4.96e
5.75e
5.98e
4.92e
5.75e -2 5.86e 2
4.04e
5.55e
1.09e
3.88e
5.52e
1.08e
E.3 Evaluation using non-convex model
Although AdeptDD-FedAvg is theoretically guaranteed only for convex models, experimentally evaluated its performance compared to DP-FedAvg on the image classification task (T3: MNIST) with P = 2 using non-convex model. Specifically, we employed MobileNet as the model architecture, with 0.3 and C, 0.05. As shown in Table 5, even when using non-convex model, AdaptDP-FedAvg achieves lower training loss compared to DP-FedAvg. While this observation lacks theoretical guarantees, it aligns with our expectation that the proposed method remains effective for overparameterized deep neural networks, even in the non-convex setting.
Table 5:
Training loss using non-convex model.
Datasel
De-FedAvg
A FedAvg
Lv.1
Lv.2
Lv.3
Lv.1
Lv.2
Lv.3
T3) MNIST
(R, (1000 15) 8.12e
1.246 -4
4.40e
1.35e
9.13e
2.69e
E.4 Evaluation additional computational costs
As noted in Remark 1, AdaptDP-FedAva requires additional computational costs compared De-FeddAvg We conducted additional experiments on the image classification task (T3: MNIST) with P = 2 clients to measure the runtime per communication round well as the peak memory usage. As shown in Table 6, we experimentally found that both the runtime and peak memory usage of AdaptDP--FedAne and DP-FedAvg are nearly identical, indicating that the additional cost introduced by AdeptDD-Fed/Avg is practically negligible.
Table 6: the runtime per commuication round and the peak memory usage.
Dataset
De-FedAvg
Fed.Av
runtime(Sec]
CPU/GPU memory usage|GB
runtime[Sec]
CPU/GPU memory usage GB)
T3) MNIST
19.60
2.10/1.62
19.63
2.10/1.62
(R, (1000 15)
F
Impact statement
We propose a new adaptive clipping algorithm for local record-level DP federated learning. While our method provides theoretical guarantees of statistical data privacy under differential privacy, it does not ensure complete immunity to data leakage. Consequently, there remains potential risk of privacy breaches.
35


Reviews:
[{"review_id": "ruo1g8BkhF", "reviewer": "Reviewer_1V8F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nProblem Relevance: Addresses a critical challenge in FL—balancing privacy and utility—by reducing clipping bias and wasted privacy budget.\n\nTheoretical Rigor: Combines RDP-based privacy analysis with utility guarantees, rigorously linking adaptive clipping to improved convergence rates in interpolation regimes.\n\nEmpirical Support: Demonstrates consistent improvements in training loss across tasks, validating the theoretical claims.\n\nPracticality: Highlights negligible additional communication/computation costs (scalar-valued clipping radius updates).\n\nWeaknesses:\n\nGeneralization Concerns: While training loss improves, test accuracy occasionally underperforms DP-FedAvg (e.g., MNIST), suggesting potential overfitting. The paper attributes this to training loss focus but lacks deeper analysis.\n\nHyperparameter Sensitivity: Introduces new hyperparameters (τ, Ĝ) with limited guidance on tuning. The theoretical benefits depend on dataset-dependent quantities like τ_q, raising questions about robustness.\n\nLimited Baseline Comparison: Focuses on DP-FedAvg; comparisons with recent adaptive clipping methods (e.g., PAGAN, Clipless DP-SGD) are missing.\n\nComputational Overhead: While claimed as negligible, no empirical profiling of runtime/memory costs is provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Adaptive Clipping Algorithm: The authors propose AdaptDP-FedAvg, an adaptive gradient clipping method for differentially private (DP) federated learning (FL). The clipping radius is dynamically adjusted based on the root-mean-square of gradient norms, exploiting the interpolation property (where per-sample gradients vanish near optimality).\n\nTheoretical Analysis: They provide (ε, δ)-DP guarantees via Rényi DP (RDP) and utility analysis for smooth, non-strongly convex objectives, showing improved convergence rates over DP-SGD under interpolation.\n\nEmpirical Validation: Experiments on synthetic and real-world datasets (MNIST, FashionMNIST, Bank Marketing) demonstrate that AdaptDP-FedAvg achieves lower training loss compared to DP-FedAvg with fixed clipping, especially in high-privacy regimes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Essential Revisions:\n\nGeneralization Analysis: Discuss why lower training loss sometimes correlates with reduced test accuracy (e.g., MNIST). Explore regularization techniques or early stopping to mitigate overfitting.\n\nHyperparameter Guidance: Provide practical guidelines for selecting τ and Ĝ, possibly via sensitivity analysis or dataset-specific heuristics.\n\nComparison with State-of-the-Art: Include experiments/baselines against other adaptive clipping methods (e.g., Asi et al.’s PAGAN, Clipless DP-SGD) to contextualize improvements.\n\nComputational Overhead: Quantify the runtime/memory impact of adaptive clipping (e.g., per-round time vs. DP-FedAvg).\n\nRecommended Improvements:\n\nTheoretical Limitations: Clarify assumptions (e.g., interpolation) and their real-world applicability. Discuss how non-interpolation settings might affect performance.\n\nBroader Impact Statement: Address ethical considerations (e.g., fairness implications of adaptive clipping in heterogeneous FL).\n\nVisualization: Include plots showing the relationship between τ_q and utility improvement to reinforce theoretical claims.\n\nOptional Enhancements:\n\nCode Release: Publish implementation to ensure reproducibility.\n\nLarge-Scale Experiments: Test on larger models/datasets (e.g., CIFAR-10, ResNet) to assess scalability.", "broader_impact_concerns": "I do not find any concerns of broader impact", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0OSofBfM68", "reviewer": "Reviewer_dzJ3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Smart Noise Reduction: By adapting the clipping radius to actual gradients, the method reduces noise without sacrificing privacy, a key improvement over one-size-fits-all approaches.\n2. Solid Theory: The paper provides a theoretical analysis linking the adaptive strategy to better utility.\n3. Practical Use: Addresses a real need in federated learning, where balancing privacy and model accuracy is critical (e.g., in healthcare or finance data).\n\nWeaknesses:\n1. Assumptions: The theoretical benefits rely on convexity, which may not hold for all real-world models (e.g., deep neural networks with non-convex losses).\n2. Hyperparameter Tune: Introduces a new parameter (τ) that needs manual tuning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces AdaptDP-FedAvg, a method to improve privacy-preserving federated learning by dynamically adjusting the \"clipping radius\" (a parameter that limits gradient sizes to protect privacy). The main contributions include:\n1. Adaptive Clipping: Instead of using a fixed clipping radius (common in prior methods like DP-SGD), the new algorithm adjusts it based on the average size of gradients. This reduces unnecessary noise, improving model accuracy while maintaining privacy.\n2. Theoretical Guarantees: The author proves the method meets differential privacy standards (using Rényi Differential Privacy) and shows it achieves better \"utility\" (model performance) than fixed-radius methods for smooth and non-strongly convex problems. The improvement is rooted in the \"interpolation property,\" where gradients shrink as the model nears the optimal solution.\n3. Experiments show the effectiveness over the baseline method (DP-FedAvg).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Relax Assumptions: \nConsider cases where objectives are non-convex.\nSimplify Tuning:\nProvide guidance or an automated way to set the new hyperparameter (τ).", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ygj6Vr1k2Q", "reviewer": "Reviewer_Ed7n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The investigated differentially private federated learning problem is of paramount importance in practice.\n\n2. The proposed algorithm is novel and achieves a better theoretical bound.\n\nWeaknesses:\n\n1. Assumption 2 (the interpolation regime assumption) restricts the application scenario of the proposed algorithm to overparameterized models and fails to generalize to more practical settings.\n\n2. In the numerical experiments, the authors consider only a small-scale federated regime with two clients. I suggest that the authors evaluate the proposed algorithm with a larger number of clients to demonstrate its general performance, and further investigate the impact of the number of clients on the algorithm's behavior by varying the client size in the experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new adaptive clipping algorithm that adaptively sets the clipping threshold based on the gradient norms of the current model in a federated learning regime. The proposed algorithm provides differential privacy guarantees and achieves a better utility bound than the classical DP-SGD in theory. The experimental results support the theoretical findings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My detailed requested changes are listed in the Weakness part; please refer to it.", "broader_impact_concerns": "There are no concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)