Adaptive Sigmoid Clipping for Balancing the Direction–Magnitude Mismatch Trade-off in Differentially Private Learning
Authors: Faeze Moradi Kalarde
Ali Bereyhi
Ben Liang
Min Dong
Venue: NeurIPS 2025 poster ()
Keywords: Differential Privacy
Stochastic Gradient Descent
Deep Learning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper proposes a novel clipping strategy that enhances the learning–privacy trade-off by adaptively balancing the direction and magnitude mismatches between the aggregation of clipped sample gradients and the true batch gradient.

Abstract: Differential privacy (DP) limits the impact of individual training data samples by bounding their gradient norms through clipping.
Conventional clipping operations assign unequal scaling factors to sample gradients with different norms, leading to a direction mismatch between the true batch gradient and the aggregation of the clipped gradients. Applying a smaller but identical scaling factor to all sample gradients alleviates this direction mismatch; however, it intensifies the magnitude mismatch by excessively reducing the aggregation norm.
This work proposes a novel clipping method, termed adaptive sigmoid (AdaSig), which uses a sigmoid function with an adjustable saturation slope to clip the sample gradients. 
The slope is adaptively adjusted during the training process to balance the trade-off between direction mismatch and magnitude mismatch, as the statistics of sample gradients evolve over the training iterations.
Despite AdaSig’s adaptive nature, our convergence analysis demonstrates that differentially private stochastic gradient descent (DP-SGD) with AdaSig clipping retains the best-known convergence rate under non-convex loss functions.
Evaluating AdaSig on sentence and image classification tasks across different datasets shows that it consistently improves learning performance compared with established clipping methods.

Adaptive Sigmoid Clipping for Balancing the Direction-Magnitude Mismatch Trade-off in Differentially Private Learning
Faeze Moradi Kalarde
Ali Bereyhi ECE Department
ECE Department
University of Toronto
University of Toronto
faeze. mooradi@mailo utwronto..o..o ca
www.beeeuerorrootto.ooooo ca
Ben Liang
Min Dong
ECE Department
ECSE Department
University Toronto lianggece wtw.:ecte.tooooooooooomo ca
Ontario Tech University
Min.Dong@ontarrottechuu.u
Abstract
Differential privacy (DP) limits the impact of individual training data samples by bounding their gradient norms through clipping. Conventional clipping operations assign unequal scaling factors sample gradients with different norms, leading to a direction mismatch between the true batch gradient and the aggregation of the clipped gradients. Applying a smaller but identical scaling factor all sample gradients alleviates this direction mismatch; however, it intensifies the magnitude mismatch by excessively reducing the aggregation norm. This work proposes a novel clipping method, termed adaptive sigmoid (AdaSig), which uses sigmoid function with an adjustable saturation slope to clip the sample gradients. The slope is adaptively adjusted during the training process to balance the trade-off between direction mismatch and magnitude mismatch, as the statistics sample gradients evolve over the training iterations. Despite AdaSig's adaptive nature, our convergence analysis demonstrates that differentially private stochastic gradient descent (DP-SGD) with AdaSig clipping retains the best-known convergence rate under non-convex loss functions. Evaluating AdaSig on sentence and image classification tasks across different datasets shows that it consistently improves learning performance compared with established clipping methods.
1
Introduction
Deep learning models can expose individual data samples to privacy risks and are vulnerable to multiple types practical attacks 5 26 29 To mitigate these risks, differential privacy (DP) 10 has emerged as a widely adopted standard in machine learning. Differentially private training constrains the impact of individual data samples on the training process, ensuring that the resulting model statistically resembles the one trained by the exclusion of an arbitrary data sample. As result, inferring the presence of an individual data sample in the training set becomes challenging when observing the model during training. Differentially private stochastic gradient descent (DP-SGD) is prominent
algorithm that incorporates DP into the SGD algorithm through two steps: (i) bounding sensitivity through clipping, and (ii) introducing uncertainty by adding Gaussian noise [ The first step limits the fe-seniitvviyy of the batch gradient against the addition or removal of data samples by upper bounding the magnitude of the sample gradients using clipping operation.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A widely used clipping operation, which we refer to as vanilla clipping 1 8] scales each sample gradient gi,t with norm greater than C' to constrain its norm C, as
Sit = Si, min (1, lgg.lll
(I)
where the constant C' is termed the clipping threshold. In the second step, a zero-mean Gaussian noise added to the aggregation of the clipped gradients with standard deviation that is proportional to the clipping threshold.
Applying the clipping operation sample gradients introduces a deviation between the aggregation of clipped gradients. i.c., EiEB, gi.e. and the true batch gradient, i.e., Eies, St.t, which is referred to as bias B3 13 19]. The bias caused by clipping manifests two forms: (i) direction deviation, and (ii) magnitude deviation from the true batch gradient. Direction deviation can be particularly severe in many classical clipping methods that are based on vanilla clipping operation in (T). This follows from the fact that the vanilla clipping maps all gradient norms greater than C' to C, and hence neglects the information contained in the diversity of the sample gradient norms within the batch. This can be interpreted as assigning unequal weights to sample gradients, which causes significant direction deviation. This misalignment can steer training steps towards undesired regions of the loss landscape and hinder training convergence I
In response, the per-sample adaptive clipping (PSAC) method [36] was proposed to mitigate the direction deviation. Figure shows the vari- ation of the clipped gradient norm ||@...l| versus the gradient norm |gg for the vanilla clip- ping and compares it with PSAC for the same threshold C. 0.1. We also include the curve for the automatic clipping method (Auto-S) (4) which is proposed to eliminate the need for tun- ing the clipping threshold. This figure shows the effectiveness of each method in reducing direction deviation: the more closely curve can be approximated by a line passing through the origin, the better the clipping method ap- proximates equal scaling of gradient samples with different magnitudes. We observe that the vanilla clipping method hard-clips the magni- tude of sample gradients that are larger than C, resulting in limited linear region") on the curve. Using Auto-S, the thrd-hhhessolding be- havior of the vanilla method is slightly reduced:
Clipped Gradient Norm ((g..))
0.10
0.08
0.06
0.04
Vanilla
AdaSig
0.02
AdaSig (a= 0.8)
Auto-S
PSAC
0.00
0
2 Gradient Norm (lgg)ell)
3
Figure 1: Clipped gradient norm vs, gradient norm. Comparing linear regions for different clipping methods. PSAC has wider linear region com- pared with the vanilla clipping. The linear region of AdaSig can be adjusted by altering a.
nonetheless, its restricted linear region leads a similar scaling as the vanilla method. In contrast, the PSAC clipping method has a wider linear region than vanilla clipping and Auto-S, as it preserves the same scaling factor across broader range of sample gradient norms, thereby providing a more accurate approximation of equal scaling.
While expanding the linear region can reduce direction deviation, it also decreases the norm of the aggregation of the clipped gradients compared with that of the true batch gradient (i.e., larger magnitude deviation). This occurs because the wide linear region of the clipping curve results in small scaling factor, causing sample gradients with smaller magnitudes to have excessively small norms after scaling. This suggests that the span of the linear region on the clipping curve needs to be carefully adjusted to better balance the trade-off between direction deviation and magnitude deviation. Moreover, since the sample gradient statistics change across different training iterations, it is necessary to adaptively adjust the linear region to balance the trade-off throughout the training process.
Contributions Motivated by the above observation, we propose a novel clipping method called Adaptive Sigmoid (AdaSig), where the magnitude of the clipped gradient determined from the original gradient magnitude using a scaled and shifted sigmoid function. AdaSig features saturation slope parameter that adaptively adjusts the span of the linear region throughout training balance the trade-off between direction and magnitude mismatches.
"Linear region" is used in contrast the "lazy region" to refer the part of the curve that can be well approximated by straight line passing through the origin.

---

Figure I also shows the clipping curves for two choices of the saturation slope (denoted by a) in AdaSig. As a shrinks, the linear region of the clipping operation expands resulting better approximation of equal gradient scaling and thus smaller direction deviation. Conversely, larger preserves the magnitude of smaller sample gradients thus reducing the magnituee deviation. To adaptively adjust the clipping operation based on the gradient statistics throughout training, we treat as a learnable parameter and update it in each iteration to minimize the empirical loss. To the best of our knowledge, this is the first study that designs clipping method to balance the trade-off between the magnitude deviation and direction deviation. This work makes the following contributions:
We introduce AdaSig, novel clipping strategy that allows balancing the direction deviation and magnitude deviation by adjusting a parameter a. We treat a as a learnable parameter and derive its SGD-based update for empirical loss minimization.
We develop new algorithm for DP-SGD with AdaSig clipping (DP-SGD-AdaSig) by establishing an update rule for a that effectively preserves privacy.
AdaSig's unique clipping structure and varying slope complicate its convergence analysis such that existing analyses are inapplicable. However, we derive a convergence bound for BP-SGD-AdaSip in non-convex loss settings, showing that its privacy- utility trade-off matches the best-known bound in the literature.
We conduct experiments on image and sentence classification tasks. Performance compari- with existing clipping methods shows the efficacy of AdaSig in learning enhancement through a proper balance between direction deviation and magnitude deviation.
2 Related Work
Extensions of Vanilla Clipping In vanilla clipping, the clipping threshold substantially impacts training performance [19]. Large C. results in large privacy noise variance, while small C' leads to aggressive gradient clipping, causing significant bias. Several studies adaptively adjust to improve the bias varinnn trade-off. In [3]. the optimal threshold is estimated using gradient quantiles during training. In [13,, the clipping threshold is treated as a learnable parameter, allowing it to be optimized dynamically over the training iterations. However, these works focus on the bias-variance trade-off for vanilla clipping and cannot balance the magnitude and direction deviations introduced by the bias for fixed variance level. This limitation arises from the inherent nature of vanilla clipping, where reducing the direction deviation (increasing C) inevitably increases the variance.
Some studies combine vanilla clipping with error feedback to mitigate the bias introduced by clipping 41 16 This technique is applicable to any clipping method, including ours, and is orthogonal to the focus of this study.
Other Clipping Methods Automatic clipping (Auto-S) and normalized SGD (NSGD) (39 are similar approaches that aim to eliminate the need for tuning the clipping threshold. Their key idea is to normalize gradients as gi Tg.. for some small positive constant Although these methods effective for their main goal, they suffer considerably from direction deviation, they assign larger weights to gradients with smaller magnitudes. These small gradients are often opposite to the true batch gradient direction, as empirically shown in [36]. To address the direction deviation issue, Cg.. for a positive r. This adjustment PSAC 36 modifies the clipping operation to gi, |g:. -=.
Cgg.e_
reduces the weights assigned to small gradients and thus mitigates the direction deviation. While PSAC effectively reduces the direction deviation, cannot adaptively balance the trade-off between magnitude and direction deviations during training because its linear region span remains fixed throughout the process (Figure D. To address this limitation, we propose AdaSig, which dynamically adjusts its saturation slope to better balance this trade-off.
3 Background and Preliminaries
Learning Model and SGD We consider supervised setting, where the training dataset consists of N samples, denoted as D ((r,, w with x; E X' representing the vector of input features and y E ) denoting the corresponding target output. The goal is to learn a model fo X' - V, with parameters 0 Rdd that maps input features to the target outputs. The discrepancy between
3

---

the predicted outputs 0; = (z.)) and the actual targets y; is measured by the sample loss function, denoted & yxy + R. Let h' (0) denote the loss of sample as function of the model parameters, i.c., h (0) - {z)( v.). The population loss is defined as L(O) E(z,))><l [e(fo(z), v)], where P denotes the unknown data distribution. In iteration f, stochastic gradient descent (SGD) updates the model parameters using the average gradient of the loss over the samples within a batch of data as
A
Oc+1 0.
+EV Voh'
B
0=0.
REB
where Veh' (0) denotes the gradient of the loss function with respect to (w.r..)) the model parameters. 9, denotes the model at iteration t, B, S 'D is the batch of data at iteration t, B denotes the expected batch size, and À is the learning rate.
Differential Privacy The randomized mechanism M : S - R. with domain S and range R satisfies
(e, 6) -differential privacy (DP) if, for any two neighboring datasets S, E S, i.c. S is formed by adding or removing single sample from S, and for any output set R C R,
(S) E RJ S e Pr(M(S') E R]
(3)
where Pr[A] denotes the probability of the event A. Inequality (3) implies that for small and 6, the output distributions of the mechanism under neighboring datasets approach each other, making harder to detect the contribution of an individual sample from the mechanism's 's output.
Differentially Private SGD (DP-SGD) Let gi, Rd denote the gradient of loss for sample iteration t, i.e. g - (eh'(0)le= e DP-SGD achieves privacy protection via two steps: (i) The gradient samples within the batch are first passed through a clipping method that returns a clipped gradient g., whose f--norm is bounded by a constant C. Basic DP-SGD uses the vanilla clipping method in (I). The aim of clipping is to bound the fz-sensitivity of the aggregation of the gradients by C. (ii) After aggregating the clipped gradients over the batch, zero-men pero-mean Gaussian privacy noise is added to introduce uncertainty. To ensure the privacy of training, the variance of the noise process scaled with C22 Consequently, in iteration t of DP-SGD, we update the model parameters as
X
0 = 0
Σ
E +m).
B
where n, ~ (0, C²oo1aa) is the Gaussian privacy noise with a being the noise multiplier determined by the privacy budget (c, 6).
4 AdaSig Clipping
Vanilla clipping in outputs a vector with the smallest error norm relative to the input vector. however introduces a notable direction deviation between the true batch gradient and the sum of clipped gradients due to unequal scaling of different sample gradients. is to scale all sample gradients by the same coefficient such that all resulting norms fall below the clipping threshold. This approach requires the scaling coefficient to be small enough so that the largest gradient norm is reduced to the clipping threshold. As a result, gradients with smaller norms are scaled down to very small norms, leading to an aggregation with excessively small magnitude (magnitude deviation). Since the privacy noise is proportional to the clipping threshold, this maggiiite aggregation results in a very low signal-to- noise ratio, which hinders training performance.
naive solution to this issue
In light of the above discussion, proper clipping operation should not only have a bounded range to constrain the lg sensitivity, but it should also ensure that the output norm monotonically increasing w.r.t. the input norm, with an adjustable linear region span that balances the rrade-off between direction and magnitude deviations. Since the sigmoid function, with an adjustable saturation slope, satisfies both of these properties, we propose clipping operation based on the sigmoid function, termed adaptive sigmoid (AdaSig). Further details on the choice of the sigmoid function for clipping are provided in Appendix A This operation is illustrated in the sequel.
Clipping with Sigmoid Function For the clipping threshold C, we define AdaSig clipping as
vo (xx = (i -1).
2
(5)

---

91.
6: 9:
9. 9.
9.. 0
9: 9
8..0
0.2
0.1
0.1
0.0
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
(a)
(b)
(c)
(d)
Figure 2: Illustrative example: sample gradients shown by solid arrows and their aggregation is indicated by a dashed arrow. The horizontal and vertical axes correspond to the first and second entries of the two-dimensional vectors, respectively. (a) shows true gradients and their summation. (b) shows the gradients clipped by the vanilla method and their aggregation gradients clipped by AdaSig for 15 and in green and red, respectively. The adjustability of a enables AdaSig to balance the trade-off between direction and magnitude deviations.
orange. (c) and (d) show the
where >> 0 is the saturation slope. Using the AdaSig function, the sample gradient gi, is clipped as gi. = v/ (Il
gi,
One can see that this clipping addresses the two mentioned desired properties: (i) since th (x) E [0, C*) for x the of the clipped gradients by AdaSig has bounded fz-sensitivity equal to C. (ii) The scaling is a monotonic function of the input norm and can adjust its linear region by varying a. Figure Saarreesshwwww shows the clipping curves of AdaSig for two choices of a. As the figure shows, the parameter a enables us to sketch an adjustable trade-off between the two extreme cases of clipping: by increasing a, the curve approaches the vanilla clipping method, which results in larger magnitude after aggregating the clipped gradients at the expense of higher direction deviation. Conversely, decreasing a leads to a better approximation of equal scaling, and thus less direction deviation, while reducing the magnitude of the sum of the clipped gradients (i.e., larger magnitude deviation).
Illustrative Example Before developing our differentially private training algorithm via AdaSig, provide a numerical example to illustrate AdaSig's capability to balance direction and magnitude deviations. Let's consider sample gradients
0.3
-0.08
0.3
0.05
that are shown by solid blue arrows in Figure 2 The sum vector, i.e., true batch gradient, is shown with dashed blue arrow. The clipped gradients using the vanilla method with C 0.1, and their aggregation are shown in orange by solid and dashed arrows, respectively. Comparing the sum the clipped gradients with the true batch gradient, we observe notable direction deviation.
We next consider AdaSig clipping with the same threshold C' 0.1 and different choices of the saturation slope, namely, a = 15 and a = 8. The result for a 15 is shown by green arrows with the dashed one indicating the sum the clipped gradients. Evidently, the direction deviation is slightly reduced in this case compared with the vanilla method. As we reduce to = 8, shown by red arrows, the direction deviation is further reduced. It however observed that better alignment in this case is achieved at the cost of a smaller magnitude of the aggregated vector. This explains the natural trade-off between direction deviation and magnitude deviation. Appendix H.1 provides additional insights into the effect of a on this trade-off.
5 DP-SGD with AdaSig Clipping
We develop a DP-SGD algorithm with AdaSig clipping, which we refer to as DP-SGD-AdaSig. Nonetheless, AdaSig can be readily incorporated into other differentially private optimizers. DP-
5

---

SGD-AdaSig consists of two steps: (i) Differentially private update of the model parameters, and (ii) Differentially private update of the saturation slope a.
5.1
Differentially Private Update of Model Parameters
Rewriting AdaSig clipping in terms of Ox in 66, we have
gi vo. (IlB..l))
gi,
(8)
[B..e
where we add the subscript to the slope, i.e., 0g,, indicate that it is adjustable parameter and changes over iterations. We further define s(O,, oy; BB) as the aggregation of clipped gradients in iteration i, and s(0r, B,) represent the private estimation of batch gradient, i.e.
(0,, B.)
Vai (IBB..
gi,t
(9a)
||gr,
s az: B.) _ s s(O, oy; BB) +
(9b)
where n ~ NO,, Ia) is privacy noise with per-coordinate variance C2 It is worth mention- ing that s(0,, 04; B,) includes 0, as an argument, since it depends on 0, through sample gradient g computed at O,.
Using the above notation, the differentially private SGD update of model parameters is
0 = =0 30--5((0.)) -0, az By).
(10)
Note that in (T)), the parameter C can be absorbed into the learning rate X, since s(0,, ou; BB) linearly scales with fact, the noise term n and all clipped gradients s(O,, oy; B:) are scaled by C (see (5) and note that the variance of nž is C²2o?). We hence simplify the parameter space in the remainder of this work by setting =
5.2 Differentially Private Update of Saturation Slope
At iteration f, we aim to update to minimize the empirical loss. To this end, we update a in the descent direction of the loss gradient. Developing differentially private update rule for or therefore involves two main steps: (i) computing the derivative of the loss w.r.t. a, and (ii) obtaining differentially private estimate of this derivative. Based on this estimate, a differentially private update rule can then be constructed using one-step gradient descent. We detail these steps in the subsequent sections.
9h (0x) denote the derivative of the loss sample w.r.t.
Loss Derivative with Respect to Let at iteration 6. We approximate it as
(ht(
A
R3
(1I)
B
where
2e a ||gi. gi.t c as || 11)?
r(0,, aq; B.)
(12)
1EB,
The derivation of the approximation in is provided in Appendix C
Let
(h.).)
denote the batch-averaged loss derivative w.r.t. at iteration t. We approximate it as
dar
ah(O.)
(0.(
das
B
day
iEB,
gi,
r(0,-1, (101-1104-1:BB-11))
(13)
Ba
kc8,
where (a) applies the approximation in

---

Differentially Private Estimate of Loss Derivative We next compute a differentially private estimate of the loss derivative approximation in ([3). The right-hand side (RHS) of (13) comprises two factors. For the first factor, i.e., LiEB, Bi,t, a private estimate is given by 8(0,, 04; Br), which is dk(0.) computed for update of 0, in TOP. We can hence obtain differentially private estimate for da, by finding private estimate of the second factor, i.e., r(0,-1, -4-1; B,-1). The standard approach for differentially private estimation is to (i) restrict the fz-sensitivity of r(0,, cri B:), and (ii) add a privacy noise term, whose standard deviation is proportional to the bounded ez-sensitivity, However, the former step is not required in this case, as the fz-seniitiviyy of r(0,, ac; B.) is bounded. This shown in the following lemma.
Lemma 5.1. The lz-sensitivity of on; Bi) is bounded from above by Da. = 0.448/at-
Proof. See Appendix D
Using Lemma 5.1 we compute the private estimate of r(0,, 0t; Br) as
f(0,, oz; (3) = r(0,, ay; 53) + n
(14)
where n. (0, A L2) is the Gaussian privacy noise with the noise multiplier o,. Replacing the first and second factors in (13) with their private estimates s(O,, ari B,) and (1-0-1,.-11;B--- respectively, we obtain the differentially private update for ar as
dad
a++1 =4 + B² $(0,, or; (1; (4-111))
(15)
for some learning rate Aa- Since both 8(O,, oz; B,) and f(0,-1, 04-1; B1-1) are noisy, their product can deviate notably from the derivative approximation in (13). To make the update rule robust, we consider only the sign of their product, which is more resilient to noise. Also, ensure that the value of o remains positive, we adopt a standard exponential update, by rewriting (15) in the exponential form
0++1 a..c sosign
(16)
The DdsSGD-AdaSig algorithm is outlined in Algorithm DP-SGD-AdaSig incurs negligible extra computational and memory costs compared with the vanilla clipping method. A complexity analysis provided in Appendix
Algorithm DD-SGID-AdaSig
1: Input: D, T, (kp > 0, Xa, A, B, 0., Po
2: Output: {0,)
3: (f--,001;;B11
// Initiialization
4: for E {O....,
do
5: Form B, via Poisson sampling with rate B/N.
|/ Poisson sampling
6: for (I,, B, in parallel do
7:
Σ1,1 Vac (Ilg,, Bit |l.
20 2e gi
8: Pit (1+e- (1tt-cc1
9:
end for
Ao. 0.448
|/ Sensitivity bound
as 8(0,, 01; B,) = EiEs, Bitt + +N(0 La)
11:
12:
f(0,, ar; (B)) = =EEE,, iit + NO, A2 Ia)
01+1 0, 500,, B.)
Model update |/ update
14: ar+1 a, (
15:
end for
6 Privacy Analysis of DP-SGD-AdaSig
The privacy guarantee of the proposed algorithm is provided directly by extending the results for SGD To this end, we first recall that each iteration of the DP-SGD algorithm is Gaussian
7

---

mechanism. The entire DP-SGD algorithm is then viewed as composition of these Gaussian mechanisms, where each iteration's output serves as the input for the next iteration. The privacy accountant method is then used to obtain the noise multiplier in ( which ensures the algorithm satisfies (c, 5)-D after T iterations D. We denote this guarantee as a Accountant (g. T, T,,,)) where q = B/ B/N is the batch sampling probability.
Unlike the DP-SGD algorithm,, PP-SGD-AdaSii in each iteration consists of two parallel Gaussian mechanisms: the first one returns the Gaussian approximation of the batch gradient, 8(O,, O2e; Br), that used for updating both the model (1O) and the saturation slope 0g in (16). The second mechanism returns the Gaussian approximation of r(0:-1. Qt- 11 denoted by f(0)-1. - -1) which used update a in (16). From a privacy perspective, these two parallel Gaussian mechanisms together behave as a single Gaussian mechanism [3] Theorem I]. This is formalized in Proposition 6.1
Proposition 6.1. In each iteration of DP-SGGD-AdaSa the two parallel Gaussian mechanisms s(-) and f(-). with noise multipliers dg and or, are equivalent to a single Gaussian mechanism with noise multiplier = [o,²² 2 + o. 2)-1/2, Hence, each iteration of DP DP-SGD-AdaSig incurs the same privacy cost as an iteration of DP-SGD with noise multiplier o.
Compared with Theorem R] Proposition 6. uses o, instead of 2a, for privatizing the second mechanism. This is because the privacy noise of the second mechanism in [3] privatizes positive scalar, but in our algorithm, the noise privatizes r(0,, 04; B,) which may have non non-positive entries.
7 Convergence Analysis of DP-SGD-AdaSig
Since the clipping structure of AdaSig is substantially different from established methods in the literature, existing convergence analyses are not directly applicable. Furthermore, the adaptive adjustment of a adds complexity to the analysis of training convergence. In this section, we provide theoretical guarantees on the convergence of DP-SGID-AdaSig. We consider the following assumptions on the population loss L(O) and its gradient, denoted by gr LL...)
Assumption 7.1. (Lower bounded loss) There exists a scalar L* such that L(0) Z L Rd. Assumption 7.2. (Smoothness) L( is 8-smooth. i.e., 3 such that
B |10 02l1 10,, 02 CRd
L(O,) s L(02) + (VL( 01 02)
(17)
2
Assumption 7.3. (Bounded gradient). The é 2n-norm of the gradient of L(0) is bounded, i.c., there exists G > 0, such that Ig < G for any 0g E cdd.
Assumption 7.4. (Unbiased per-sample gradient). The per-sample gradients, i.e., gi, for i E Br are i.i.d. and are unbiased estimators of gr- This means gi, Ve, Vi, where v. g: A,, Vt, with
E [ 0. Additionally, gi, is distributed centrally symmetric around g:, i.c., A 2 A..
Assumptions 7.1 and 7.2 are standard assumptions used for amalyzing the convergence of first-order
optimization algorithms
Assumption 7.3 widely used for analysis of clipping in DP
algorithms 32
35
41 Assumption 7.4 (unbiasedness and symmetric distribution) is commonly
used in both literature not concerned with DP
6
30
33 38 and DP literature 44. Specifically, the
symmetric per-sample gradient distribution assumption is also empirically verified in 77] Figure 3].
We next present the main convergence result in Theorem 7.5 which shows that AddSGD
yields a bounded weighted average of the expected squared gradient norms. The bound is expressed explicitly in terms of the privacy budget and other parameters.
Theorem 7.5. Let Assumptions
7.1 7.4
hold. For a constant G. let on & 1/r. Moreover, let
2N [((0))-L) L*)
Xo 1/T and V
Then, under the privacy budget (c, there exist
ST (2222 "))
constant v3 > 0 such that DD-SSD-AAdaat satisfies the following inequality:
Σ Pr (1 0-0
(18)
T
8

---

reduces to for some constant G > 0, where
L(0o) - 1i* (1+
dv log(3)
J
(19)
T
22
Proof. See Appendix E.2
We note regarding the bound in Theorem 7.5 (i) The weight of expected gradient norm squared in the expression on the left-hand side (LHS) of 18 corresponds to the probability that the magnitude of the deviation of sample gradients from g: does not exceed r. (ii) Tighter privacy guarantees, i.e., smaller values of (e. 8), lead to higher values on the RHS. This reflects the trade-off between privacy and learning performance.
Corollary [7.6 below, we observe that DP-SGD-AdaSig achieves the
asymptotic convergence rate of Ol Ne the literature for vanilla clipping that does not allow adaptation
d ), which matches the known privacy-utility trade-off bound in
Corollary 7.6. As T approaches infinity, the RHS of 18 converges O (l/og(1/))
Ne
When the sample gradients have a bounded variance, the weights in the LHS of the expression in Theorem 7.5 can be further simplified. This yields upper bound for the average of the expected gradient norm squared, which is given in the following corollary.
Corollary 7.7. Let E A.ll S C² for any iteration t. Then, the bound in Theorem 7.5
T-1
Σ
E g 12 s
2RG 2ROV33. /BJ,
(20)
t=
r2 and r, max max G}.
where R r.-3.
Proof. See Appendix
E.3
8 Experiments
We evaluate AdaSig on image and sentence classification tasks Under equal privacy budget (c. 5), compare the performance of AdaSig against four baselines (i) Vanilla method (1. (ii) method proposed in 3 (iii) Auto-S [4. and (iv) PSAC [36]. We further present an ablation study for AdaSig
Appendix where we compare the performance of AdaSig with sigmoid clipping using fixed a.
8.1 Image Classification Task
We conduct experiments on five image classification datasets including MNIST (0). FashionMNIST 37], CIFAR-10 10 [18, ImageNette [15] (a 10-class subset of ImageNet 9 and CelebA 23
Setting For MNIST, FashionMNIST and CIFAR-10. 10, we train the CNN architectures used 27
34 4] i.e., a 4-yyeer CNN for MNIST and FaShionMNIST, and an 8-layer CNN for CIFAR-10. The simulation setup and hyperparameters are adopted from [44) For ImageNette, we use ResNet-9 with group normalization (instead of batch normalization) and without scale normalization, following the setup considered in [17 with the minor exception that the learning rate does not decay. For CelebA, we use the same ResNet-9 architecture for ImageNette, with the same simulation setup as in [4]. Since each image in CelebA dataset has 40 labels, we run two sets of experiments on this dataset: (i) We consider binary classification problem that aims to predict the 'Smiling' label only. (ii) We consider the problem of multi-label classification with all the available 40 labels in the dataset. The detailed settings, including hyperparameters for each method, provided in Appendix F22
Performance Comparison Table T (first six rows) shows the average test accuracy across five differ- ent runs of each dataset. AdaSig outperforms all baselines by achieving higher average accuracies
com/faazeeezeorriii/// "The GitHub repository for implementation is available at httt:sss//gttttiioo/////eereooar AdaptiveSigmo1 wwtpeiimmmiiippppppmeepp
git
"Methods such as [41] are not directly comparable, as they do not propose new clipping strategy but instead incorporate error feedback into existing ones, which be applied to any clipping method, including Adašig.

---

Table 1: Average test accuracy (in percentages) with 95% confidence intervals five runs. Bold numbers indicate p-value < 0.05 in the t-test performed to identify statistical significance.
DATASET
MODEL
[c.6)
VANILLA
METHOP IN (
AUTO-S
PSAC
ABASIG
MNIST
4-LAYER CNN
(3,10-*)
98.11
0.04
98.13 2: 0.03
98.04 1: 0.04
98. 1: 0.04
:: 0.05
FASHIONMISST
4-LAYBR UNN
(3,10-*)
86.27 0.24
86.28 0.19
86. +: 0.26
86. +: 0.19
86.68
CIFAR-ID
8-L CNN
(3,10-*)
0.13
0.11
61.21
61.93 +
62.54
IMAGENETTE
RESNET-9
(8,10-1)
62.07
62.30
64.65
CELEBA (MULTI-LABEL)
RESNET-9
x
0.03
0.02
88.48
88.46
88.78
(SMILING)
RESNET-9
(8,5
90.90
90.95 0.11
88T-2
ROBERTA- BASS
(3, mke)
87.45 0.49
87 /6 0.47
87.18 + 0.56
87.48 + 0.32
88.44 +: 0.17
QNLI
ROBERTA-RASH
(3:
84.32 0.16
84.36 0.17
84.51 + 0.13
84.58 + 0.14
84.92 0.12
Ads
AdaSiy
AdaSig Neihod In
Probabling
0.10
Auto-S
0.10
PSAC
0.10
Vin
0.10
0.05
0.05
0.05
0.05
0.00
0.6
0.00
Cosine Similaniy
Cosine Similarity
Similarity
Cosine Similantt
Figure 3: Comparison of cosine similarity during training ResNet-9 on ImageNette.
across all datasets. Conducting statistical t-test further confirms the statistical significance of this improvement (p-value < 0.05) on out datasets [14].
Cosine Similarity Analysis To assess AdaSig's 's ability to balance the direction deviation and magni- tude deviation, we measure the cosine similarity between the true batch gradient and the aggregation of clipped gradients. The results are shown in Figure where we compare the histogram of the cosine similarity while training ResNet-9 on ImageNette. Notably, AdaSig's histogram concentrated at larger cosine similarities as compared with the baselines, which indicates the effectiveness of AdaSig clipping in reducing the direction deviation. Additional results are presented Appendix G to elaborate on the trade-off between the direction and magnitude deviations.
8.2 Sentence Classification Task
We conduct sentence classification experiments SST-2 (31. for sentiment classification, and QNLI 28 for question-answeriigg inference both taken from the GLUE benchmark.
Setting We use the pre-trained RoBERTa-base model and conduct full parameter fine-tuning with DP consideration following the same setting as in [21 The detailed settings, including hyperparame-
ters for each method, are provided in Appendix F.3
Performance Comparison Table I (last two rows) shows the average test accuracy five runs for each dataset. We observe that AdaSig achieves statistically significantly (p-value < 0.05) higher average test accuracies as compared with all baselines both datasets.
9 Conclusions
We proposed AdaSig clipping, novel clipping operation for differentially private training that balances the trade-off between direction and magnitude deviations incurred by clipping. It achieves this balance by adaptively adjusting its saturation slope throughout training based on information obtained from sample gradients. Our convergence analysis demonstrates that DP-SGD with AdaSig clipping retains the best-known convergence rate in the non-convex loss setting. Experiments on image and sentence classification tasks demonstrate that AdaSig clipping consistently improves training performance compared to existing clipping methods.
10

---

Acknowledgments
This work was supported part by the Natural Sciences and Engineering Research Council of Canada.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the ACM SIGSAC Conference Computer and Communications Security, 2016.
[2] Jason M Altschuler and Kunal Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
[3] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. In Proceedings of the International Conference on Neural Information Processing Systems, 2021.
[4] Zhiqi Bu, Yu- YuaXiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. Proceedings of the International Conference on Neural Information Processing Systems, 2023.
[5] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium, 2021.
[6] Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In Information Theory and Applications Workshop (ITA), 2018.
[7] Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In Proceedings of the International Conference Neural Information Processing Systems, 2020.
[8] Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horváth, and Eduard Gorbunov. Conver- gence of clipped-SGD for convex (Lo, -mmoohh optimization with heavy-tailed noise. arXiv preprint 2rXiv:2505.20817. 2025.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, 2009.
[10] Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy, Now Publishers, Hanover, MA, USA, 2014.
[11] Maria S Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, and Jesse C Cresswell. Disparate impact in differential privacy from gradient misalignment. In International Conference on Learning Representations, 2023.
[12] Huang Fang, Xiaoyun Li, Chenglin Fan, and Ping Li. Improved convergence of differential PrD with gradient clipping. In International Conference on Learning Representations, 2023.
[13] Filippo Galli, Catuscia Palamidessi, and Tommaso Cucinotta. Online sensitivity optimization in differentially private learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[14] Sander Greenland, Stephen J Senn, Kenneth J Rothman, John B Carlin, Charles Poole, Steven N Goodman, and Douglas Altman. Statistical tests, p values, confidence intervals, and power: guide to misinterpretations. European journal of epidemiology, 31(4):337-350 2016.
11

---

[15] Jeremy Howard. ImageNette: A smaller subset of 10 easily classified classes from ImageNet, March 2019. /ttttts:::// /tttps://github.... http://thh....mo//aaaaaaaaaaaaaa
Dataset; fast.ai.
[16] Rustem Islamov, Samuel Horvath, Aurelien Lucchi, Peter Richtarik, and Eduard Gorbunov. Double momentum and error feedback for clipping with fast rates and differential privacy. arXiv preprint 26X., 2025.
[17] Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis. Differentially private training of residual networks with scale normalisation. arXiv preprint 2tti::20030003244 2022.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[19] Alexey Kurakin, Shuang Song, Steve Chien, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv: 12328, 2022.
[20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[21] Xuechen Li. Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language mod- els can be strong differentially private learners. In International Conference on Learning Representations, 2021.
[22] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X.v:190..11692, 2019.
[23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[25] Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint (9KKv:/908..000
2019.
[26] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman. SoK: Security and privacy in machine learning. In IEEE European Symposium on Security and Privacy, 2018.
[27] Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[28] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint 1rXiv:1606.05200 2016.
[29] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017.
[30] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
[31] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference Empirical Methods in Natural Language Processing, 2013.
[32] Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private GLMs. In International Conference on Artificial Intelligence and Statistics, 2021.
[33] Mandt Stephan, Matthew Hoffman, David M Blei, et al. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1-35, 2017.
12

---

[34] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). International Conference on Learning Representations, 2021.
[35] Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. In Proceedings of the International Conference Neural Information Processing Systems, 2017.
[36] Tianyu Xia, Shuheng Shen, Su Yao, Xinyi Fu, Ke Xu, Xiaolong Xu, and Xing Fu. Differentially private learning with per-sample adaptive clipping. In Proceedings of the AAAI Conference Artificial Intelligence, 2023.
[37] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 0777477777 2017.
[38] Zeke Xie, Issei Sato, and Masashi Sugiyama. diffusion theory for deep learning dynamies: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2021.
[39] Xiaodong Yang. Huishuai Zhang. Wei Chen, and Tie-Yan Liu. Normalized/clipped SGD with perturbation for differentially private non-convex optimization. arXiv preprint 130033, 2022.
[40] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2 109, 12298, 2021.
[41] Xinwei Zhang, Zhiqi Bu, Steven Wu, and Mingyi Hong. Differentially private SGD with- out clipping bias: An feror-feedback approach. In International Conference on Learning Representations, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main contribution of the work is proposing AdaSig, a clipping method for balancing direction-magniiude mismatch trade-off in differentially private learning. We explain this trade-off, limitations of existing methods, and how our AdaSig addresses them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clarify the existence of trade-off between direction and magnitude deviation and discuss even though our method can balance this trade-off, these deviations cannot be eliminated simultaneously. We also clarify the scope of the work, which is petrrenttiall-prrvatte learning, making non differentialll-privaal training beyond its scope. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We state all assumptions for our theoretical proof's and convergence analysis, which are all established assumptions that were used in previous works too. All theorems in the work are also supported by a proof.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide complete explanations on our experimental setup such as the datasets used, model architectures, etc. in Section g We also explain our hardware and software information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If contribution primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We release all our implementation code enable reproducibility. All datasets that were used in this work are established benchmarks that are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wtipps.cc/public////
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed explanations on our experimental settings in Section g
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: our experiments were repeated five times and the confidence intervals for the five runs are also provided. Furthermore, the statistical significance of the improved performance offered by our method was validated for each dataset by conducting a statistical t-test.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information on both software and hardware used to conduct experiments in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics httph:t////// http://eeurrip..ccpebeeeee htt.c///eulttttiiiiiiiiieeeeeeee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: This work complete alignment with the NeurIPS Code of Ethics. does not involve any human studies, is in complete agreement with the data-related criteria, and there's no anticipated societal impact associated with
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
17

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our study proposes method for improving models, which we consider positive societal impact. We do not anticipate any negative socictal impact for our work.
training of ML
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data or model that has high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our work is licensed under CC BY-SA 4.0, and we are aware of its terms of
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[Yes [es]]
Justification: Our proposed methodology well as the released software are well- documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The methodology presented in this paper does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httpps///eeerrppcccc///ooerrrerp for what should or should not be described.
20

---

APPENDIX
The appendix is structured as follows: Appendix A presents details on the choice of the sigmoid function for clipping. The complexity analysis for DP-SGD-AdaSig is given in Appendix B An approximation of the sample loss derivative is provided in Appendix [ The proof of Lemma 5. is presented in Appendix [ The proof of Theorem 7.3 and the preliminary lemmas and theorems related to the convergence analysis are included in Appendix E The detailed experimental setup is presented in Appendix [F] A numerical analysis of the direction and magnitude deviations is provided in Appendix [G] The ablation study of AdaSig is presented in Appendix [H] Finally, Appendix andd Appendix repport the variation of a across training iterations and the numerical convergence results, respectively.
A Details on the Choice of Sigmoid Function for Clipping
The sigmoid function is simple yet well-suited to our goal of adaptively balancing the trade-off between direction and magnitude deviations. particular, it enables control over the range of the linear span in Figure [ through the parameter a. Although PSAC (36] and Auto-$ 4 could be extended to variants that adjust their linear region span by varying the constant parameter r throughout training. the sigmoid function provides greater flexibility and control.
Specifically, in the PSAC clipping function, varying r alters the span of its linear region; however, the change in this span is confined to a limited range. This behavior can be illustrated by analyzing the PSAC clipping function, gi,: lig lia,
Cga, in extreme cases. Due to the term [| in
the denominator, varying r from 0 to oo causes the clipped gradient norm curve || (shown in Figure to transition only between two limiting curves: C' and C|E +++++|++- Consequently, the span of the linear region in PSAC is restricted to lie between these two curves, and adaptively updating r cannot effectively balance the trade-off between direction and magnitude deviations. In contrast, in the AdaSig clipping function of (6), the clipped gradient norm curve llg.. (illustrated in Figure ! can vary from C' (as greater flexibility in adjusting the span of the linear region.
oo) down to values arbitrarily close to 0 (as a -> 0), thereby providing
While the Auto-S clipping function, gi. |iiil++>>* does not have the restricted variation of the linear region span seen in PSAC, it is still less flexible than the sigmoid function. Its linear region span is less responsive to changes in than the sigmoid function is to a, larger adjustments of are required to produce a noticeable effect. In contrast, the sigmoid function, with its exponential dependence on a, offers more flexible control over the linear region and thus the dirention-magnitude mismatch trade-off.
CR..
B Complexity Analysis
Compared with DP-SGD (with vanilla clipping), one can observe that DP-SGD-AdaSig (Algorithm I introduces a negligible increase in computational and memory costs. Specifically, with respect to computation, the algorithm performs lines 8, 12, and 14 in addition to the standard operations in vanilla clipping. Line 8 is carried out efficiently using the per-sample gradient norms computed in line 7, the computational cost of step 12 is identical to that of step II, and line 14 incurs only a constant additional cost per iteration. Consequently, the overall computational cost introduced by these additional steps is dominated by other operations and the cost of per-sample gradient computation, making the additional overall computational cost minimal.
With respect to memory, OP-SGD-AdaSip requires extra memory to store f(0,, or; B,) (line 12), which is required for the slope update in the next iteration (line 14). This additional memory cost is negligible as compared with the memory required for storing the model parameters and per-sample gradients.
21

---

C Approximation of the Sample Loss Derivative with Respect to a
Since the loss on sample i iteration t, denoted by h²(0)). does directly depend on at, derive an update rule for we unroll the expression for h0,) and write it in terms of 0 and 0t In particular, using (10). we obtain
X
h' h',) 11' 10-10-1
(21)
B
For sufficiently small learning rate for (i.c., da in (6)). at-1 and a; differ only slightly, and hence we adopt the approximation 0t-1 R 04- Consequently, the loss derivative at ot, da, approximated by the loss derivative at at-1, 80,
(0.) can be
(h+(0.)
ie..
δh (0.) 0h (O.)
(22)
da,
da--1
8h"(0,) daa-1 as
We now apply the chain rule to (1 compute
Øh (0.)
00:
(Vo. dai-1
(23)
da.-1
where the first factor in the RHS of 23 is the sample loss gradient w.r.t. model parameters, i.c., gi, To compute the second factor, we write
00:
0
A
B)-
0o111
8a.-1 11
Ova.. (1gi
(u)
gi
day-1
gite-1
VEB.-.
(b)
2e ap-1 1i,r-1 gi
(24)
B
(1 + te
01-1
8i.t-1
iEB.-1
where (a) follows from the definition of 6(0, 11, 01-10-111 1, (9)), and (b) is obtained by computing
the derivative of the AdaSig function Va. (.). Using the definition of r(0,, az; B,) in 12 rewrite (24)a
we
00,
B 1(0011100-1::11111
(25)
0a,-1
Substituting
255 in 23
we finally get
oh (h,(
= B (001000,,0111::1
(26)
aa--1
Using together with 26) we have
(h.)
(0,1(00,1, B.
(27)
dog
D Proof of Lemma 5.1: Sensitivity Bound
Proof. The Ez-sensitivity of r(0r, 04; Be) w.r.t. the training samples in iteration is defined
Ar max 8,8
r(0,, og B.) r(0,, o. B3)
(28)
where B, and B, are two adjacent batches of data that differ by exactly one data point. Without loss of generality, we assume that B' contains all elements of B, together with an additional data point (x', y²), where x' the input feature and y is corresponding target output. That is, Bi = B, U ((a ./') Moreover, let g' denote the gradient of sample loss at (a', 3y), i... Vol(fe(z').))
22

---

Considering the definition of 2r ar: B:) in [22 we can conclude that the terms under summation in r(0,, ay; (B.) and r(O,, a;; B,) are identical, except single term that arises due the extra element in By. The fz-sensitivity in (28) can hence be upper bounded by the norm of this extra term, i.e.,
2e 2aaal a:lll''1 lg
Ar S max 8'
(29a)
(1 + e a xx
2e 2 1s'll gg'
max
(1 + a 1g'1)2
(29b)
g'
1
20
max
(29c)
=20
=
where (a) follows from the variable exchange - arllg'||. We next find the solution to the maximiza-
tion in (2c)) by setting the derivative of the objective to zero, i.e., 8 20= the following fixed-point equation
05 (14e-277 0, which results in
2+1 2
(30)
2
Let's denote the solution for (30) by 2.. Substituting the solution, i.e., 24, into objective (29c). we get the upper bound for Ar
Ar 1 2. 2 -
(31)
an (1 + 2.
22,04
Solving 30) numerically for and substituting in 31 results in Ar 0.448/aa, which completes the proof.
E Theoretical Convergence Analysis
first introduce the preliminary lemmas and theorems required for proving the main theorem.
E.1
Preliminaries
| population loss difference in two consecutive iterations is upper-hounded by
Lemma E.1. Suppose that Assumptions 7.2 and 7.44h Under Algorithmm]
the expectation of the
E ((0++1) L(0:) 0,, az) s <->qT ( - 1) |vall
2
V:
0., az +B8 322211
do?
(32)
2B²
where g: = VI)))), TE. as) denote the expectation over the rondomness in iteration for given O, and at, and v, the random process from which gi, is sampled, i.e., gi Ve E B.
Proof. Based on Assumption 7.2 the population loss is B-smooth. Thus, we have
L(0,+1) - L(O,) < gT (0.-1 - 0.) + 2 0 -2112.
(33)
23

---

Taking expectation from both sides of (33 for given Og and dg we have
E ((042)) I((0)) 2c, 04)
8
g: (0 - -0.) 2c, 0tt
R
0 - 0:ll²
0,,
(34)
2
A
B1 E
Σ
caa gs,eel))
gi,e
+ n; 0r. an
B
||g.ell
KEBL
Bx
Way (llga,el)
gi,t
+ nf
0z, 0r
(35)
2B
g
1EB,
(h)
B
A
E
Σ
Vor (Ilgi:)l)
|g
gi,t
0, Ctz
BEB
A12
AA
E
(Ilgg,)l))
gi,
0r, 0q
+
E
| |1
pc. aq
(36)
2B²
2B²
B 81
X
Σ
Vaa (|ll).)))
gi.t
0,, 0z
[gg.e||
iEB.
112
E
Mar (llgr,)))
gi,t
10000 04
2
82
2B do
(37)
2B
||gr
YEB.
X
8i
Σ
Vo. (llg,,))))
git
Or,
+ B
B
E
do²
(38)
B
[g;t
2B2
2B
BEB.
(c)
X
81
E
aa (ll gr.ll))
gi,t
0,, 0t
+ B32
do
(39)
B
|g.ee||
2B
VeB.
where (a) follows from 10). (b) comes from the fact that gi, for i E B, are independent of the process n.. and the fact that na is zero-mean, (c) obtained by applying the Triangle inequality to the second term in (36) and substituting the variance of n². (d) results from the upper bound
Σ Ve. (llg..)))
gif
S [Bol.
(40)
Igght ||
IEB
since Vo, (x) < 1 for any E I,, and (c) follows from the fact that, under Poisson sampling with rate B/N, we have
B 27
B
B+14
1
< 1 2
(41)
B2
B
N
where the last inequality holds since the expected batch size B 2
Based on Assumption 7.4 hence write
the sample gradients are identically distributed, i.e., gi, ~ Vt, Vi. We can
E B
Σ
Vor
(llga
gi Igg
Oc, at E
-B[wor
(llvel))
| Or, ar)
(42)
sEB.
Substituting into and replacing 1/ (1l ([lvel) with its definition, 32 is concluded.
39
Lemma F.2. Under Assumption
7.4 i.e. V. = ge + A. the following equality holds:
g E
1)
Ve
Oz, or
ggll E.cc (scc(s....))
s
(43)
1 e
- Vvlll
|ve||
2
where 71 A a |gl..
||
|g. c
| ||
x A;
and
1 +
2
f(s,)m))
VI + 2sc
sC
- -++e-reI++==000 -1)
2
+
(44)
vI +
2sc
24

---

Proof. We adopt an approach similar to that proposed in 4 Let 7 and H. denote the following two halfspaces:
* _ fo E F T o 2 0).
(45)
H for
R R $ 0).
(46)
Using V; gt
A., we can write
2
Vt
g:
E
1)
0,,
(47)
|v
Ea: [(
2
+
(48)
140
81 git
g:
Arl
Ea.
81 g:
+
g
A.
A E [EH
2
+ e
11g.
2+ A. 1... A.
g 1 +
A
2g.
1
2
g1 g: g
A.
+
EA
A A
(49)
2
eee ||B || A
/1g:11 + A Al|
112 +g2 A
(b) 1
2
gr g: + B: A.
EA.
A. A. AE
2
1te avv
|AA
P
2g A
gr
+
A 8 1|
2g A
1
2
A
+ EA.
A Ht) (50)
2
1 [g:
2g
gg +
A 1
2g.
where (a) follows from conditioning the expectation w.r.t. A, on two halfspaces, and using the fact
that under the symmetric distribution assumption on A, (i.e., A, 2 A, in Assumption 7.4). we have
1
Pr(A, E h* = Pr(A, E H
(51)
2' and (b) follows again from symmetric distribution assumption on A, and along with the variable exchange A, = -A, in the second term of (49).
We now rewrite (50) in terms of the random variables and [-1,1] introduced in the lemma. Noting that the event A, E Ht corresponds to 2 O, we can write
2 EA, [(,
2
g
g
A
A E Ht)
1te avv
1r 1|
+1| 2 A.
2
+
A 1|
+ 2g A
+ 2
| I(
2
g gt g AA
A. E H+]
1+0 aa |18 2+ A 2
A A.
Vlgg11 1| +
A 1
2g A
(52)
||ge|l Es.c
1 1 sc
2
-1]ssccsi]
2
v1 + 82 + 25c
1+-e""vv++*" +28'+2e0
||g:
1 1- 1-sc
1)
+
Ex,c
(53)
2
VI+ 52 2sc
+
where the equality follows directly from the definitions of and Expressing 53 in terms of f(s, c, 76) yields the result.
Lemma E.3. The function Vala) with (x) defined in 5 is a non-increasing function in for a 2 0, x 0.
Proof. Computing the derivative of the function w.r.t. and simplifying the terms yields the following:
(1)(())/
(4a(() 1
dr x x² Va(x) 2c az (ar sinh ax)
dr
(54)
x2 (1 + caz)²
25

---

where the RHS is non-pen con-positive term, since oz I sinh a.r for a 2 0, 2 0. Thus, the derivative is non-positive, which shows that the function non-inereasing.
Lemma E.4. The function 0, 0,0,
(t)(#) with (x) defined in is non-increasing function in for 8x
0.
Proof. We compute the derivative of the function as
a(0raa (z)/Ox)
2ae
(1 + e e-x 2 /0x
Ox
- -2 200 (1 0000)
(55)
(1 + e ax))
which is non-positive for and 0, which leads the desired conclusion.
Lemma E.5. For the function d (z) defined in S). the following holds:
(v)(2)
Va
0 O.
(56)
dx
Proof. We begin with the inequality az S sinh az for x 2 and 2 O. We have
az S sinh or 3 2aze
< - c-a -aa
(
2ae
5 1
+ -00 1)
(1 (1+)-a))
ax)
( Or (x) (a(x)
(57)
Ox
where (a) follows from multiplying both sides of the previous inequality by 2e-ax, (b) follows from dividing both sides by xx1 + e e-ax)² and (c) results from writing both sides in terms of da (x) and
dx
Theorem E.6. Let fs, c, 71) be the function defined in Lemma E.2
Then, the following properties
hold:
[(s, 0, 71) non-increasing in 8 for 0 c and 7t 2 O.
2. f(s, (, m1) for 0 c and it 2 0.
3. f(s, C, 71) is non-increasing in e on the interval 0 S f. s 2 1 and 7t 2
Proof. We first prove the first property, showing that f(s, c, c/)) is non-increasing in Based on Lemma E.2 we rewrite f(s. c, 7t.) as
44
f(s,c,7)) ff f1 (x1) f2 c) (22),
(58)
where we define
+ 2sc 2 VI+ 32 2sc, and
1 + sc
(i(
(59)
VI + 8² + 2sc
[2(s,)) v1 s2 2sc
SC
(60)
Taking the derivative of f(8, (1), w.r.t. 8, we have
@f(s,em)
(f1(s,
Ow, (21) Ox
Uy (x)) fi fi(s,)
8s
Os
Ox 8x
(f)
Dij (I2) Or2
wr (Iz) + fe(s, c)
(61)
Oxz
0s
26

---

Substituting the derivatives into the above expression results in
05 C. 7.))))
-8(1- - (²)
Vay, (a1)
8(1 (τ)
4 (x1) +
c+
Os
(1+ 82 +
8z1
1 52 -2c
-s(1- )²)
Vy (x2) + O (122)
(-ct c+
s(1 c2)
(62)
(1+ - 2sc)
0.2
1 82 28c
Rearranging the terms, we have
0s(s,C)7))
-0 Tb (112)
(x1)
0s
O02
s(1 - -2))
1/ (x1)
Drine (x1)
1 52 + 2sc
21
Ox1
s( [1 - )²)
Ui (12)
Oehr (x2)
(63)
1 1 82 2sc
22
Oxz
The first term in the RHS (63 is non-positive since, by Lemma E.4
the derivative of th (x) is a
non-increasiin function of x, and have x2 24 for c 2 0, Ó. The second and third terms
also non-positive because e < and, by Lemma E.5 it holds that
y (a)
Ori (x) Ōr
Vr
V4 20.
(64)
I
Thus, we conclude that 0((s.cm))) 8s
s implying that f(s, c, %) non-increasing s.
To prove the second property, we first evaluate the limit of the function as approaches infinity:
lim f(s, c, n) 0, 1 2 c 2 0, % 2 0.
(65)
8-000
By the first property, f's c, 7)) non-increasing 8, which implies that its minimum value is zero.
To establish the third property, we compute the derivative of f(s, c, (, w.r.t. Based on (58). we have
0f(s,c, dc
(si(s,c)
Orp (xI) ar
Of2(s,
Oc Vo (21) + (s(
+
Un (x22
Oan
dc
de
Orly (x2) 8z2
+ (2(,
(66)
002 Oc
Plugging the derivatives into the above expression and rearranging the terms, we obtain af(s, Tt)
by (xI)
(
Vy (x,
52 c)
dc
21
1+ 2sc
82
1+ 52 2sc
Ouhy (rI)
s(1 + sc)
Svi (x2)
s(1 - (sc)
(67)
dm
1 x2 2sc
Orz
1 52 2sc
Next, we find an upper bound for 0f(s,e))) in x. Thus, we have
Based on Lemma
E.3
Vre (x)
is a non-increasing function
de
My (x1)
Vy, (x2)
(68)
20
which follows from the fact that 22 S Iı for 2 0 and 2 2 0. Furthermore, by Lemma E.4 the
function 0v-, (z) is non-increasing in æ. Thus, since D2 S 21, it follows that Oyy (m1) Deh, (x2)
On
(69)
dx
Or
Substituting the upper bounds from 68 and 69 into 67
obtain
c8,
dc
Wyy (122)
82
(8 c) c)
Ow (22)
8(1 + sc)
8(1 - sc)
P2
1 11+ 52 2sc
1 $2 2sc
Or2
1 52 + 2sc
1 1 s2 2sc
Dhy (x2)
2cs² (1 - s2)
2cs² (1 s2)
(70)
T2
(1 + 52 2sc) (1 + 82 2sc)
Orz
(1 + 32
2sc)
82 2sc)
27

---

The RHS of the upper bound in (70 is non-positive for 2 1 and >>
0, since
(12)
Ou2
Miy (xaa
by Lemma E.5
Lemma E.7. For any r 2 1gll the the following inequality holds:
E.c f(s. c, ((s.c.m) sessssss 2 f |
C=
1,70
Pr srss5
(71)
g:
Proof. For any E R. we have
(0) E.c (lac)
Pr
g
|g:
+E. ((.em
Pr
(72)
gg
gg
{
1.55
Pr
(73)
|g1
lg
No E. [ c, c Pr
(74)
g
where (a) follows from conditioning the expectation on the events
r and Tx (b) is
concluded by dropping the second term in (72) and noting that f(s, c, 7e) 2 0 based on Theorem E.6
and (c) follows from the fact that [88 c, 7c) is non-increasing to Theorem E.6
for any c 2 and Y 2 0 due
We now restrict 2 |ggell and use the fact from Theorem E.6 that f(s,c, c, 71) non-increasing in for any 2 1 to conclude that for any 2 llgel.
Ec f c )o which completes the proof.
r
1.70 Pr p
(75)
g::
gg
Lemma E.8. For s 2 L, c( 1, (i) is lower bounded as
f(s,c= 1, cosh(s7):)
(76)
Proof. By substituting = 1 into the definition of f(s, c, 78) given in Lemma E.2 and simplifying the expression, we obtain
f(s, c = 1,7))
VI + + 1 1 1+c-ivin 7 -1)
1+s
2
1 1-8
2
1) (77)
VI+ s² - 2s 11c
(78)
+ 77(#11
1te 1+--7((--1)
2sinh(?e)
(79)
cosh(>x) + cosh()8):)
We next note that 2 1. Using the fact that cosh(æ) is an increasing function on æ 2 0, we can
conclude that cosh(s7)) cosh(>x) We then use the lower bound sinh(r) 2 for 0 write
2sinh(7))
(80)
cosh(Tc) + cosh(ss))) cosh)ss)))
which completes the proof.
Lemma E.9. Let Xa k1/T and ag = k2/r for some k1 > O and K2 > O. Under Algorithm saturation slope is bounded on both sides as
1 the
Ki
< K2
< a4
0
(81)
where e -h and R2 A kt
28

---

Proof. First, observe that for t = O we have &z which satisfies
61
K2
since 1, > 0. We next show that the same bound also holds for S1<T-1 Substituting ), 4/2 and ag 12 into the update rule (16). we obtain for any 2 1,
22 k
at
(82)
r
where qi is defined as
sign (:
(83)
r=0
with r(0-1,a-1::...
0. Since -1 -1 q S and 1 1.1 - 1, we have
h2
- (T-
h2 ki
04 2
(84)
r
h2
bg
(T-
1:2
gh
K2
04
I
(85)
which completes the proof.
Theorem E.10. Let Assumptions 7.1 7.4 hold, and suppose that N x 1/T and ag 1/r for some
7.4
constant r 2 G. Then, under the DP-SGI SGD DDD-GDD-ddSig algorithm described in Algorithm the following inequality holds:
T-1 Σ 1=
[((0) + B2(1 +
da?
Sr
(86)
T
XT
2B
where & is a constant.
Proof. Using the results of Lemmas E.1
E.2
E.7
E.8
and
E.9
we have
do
x E ((0.) L(0
1(0m.))0...
Or,
+ 3>
2B
Lemma
A0
2
g:
E
1) |v
02 02.
(87)
+
lvell
|
Ex,
S scsi
(88)
2
IV3 IV
|gel
c= 1,70)
Pr | sr}
(89)
Ore 0r./g
Pr (1A sr]
(90)
cosh (raz)
Lemma
Ri killg
Pr | sr)
(91)
2r cosh (R:))
where (a) is obtained by applying Lemma E.7 follows directly from Assumption 7.3
which requires the bound > Ig: Ill Vt. This bound G, Vt) together with the assumption r 2 G.
(i.c.,
g
Inequality (b) is obtained by applying Lemma E.8
using the bound 2 Ig Vt. The last inequality
follows from Lemma E.9 since under the assumptions Xy x 1/T and ag x 1/r. we can equivalently
write Aa T and ag k2/r for constants k > 0 and ky 0. Substituting these
expressions into the lemma, the inequality holds with K1 kze- Ri A2g *1
29

---

Let us now define _ 2 cosh (k2)/K1. After taking expectation from both sides of the last inequality. summing over iterations from to T 1, and dividing by T. have
Σ
Pr
A
S
GGT
t=
T-1
Σ
L(O.) 1(0
+ T
T-1
E
BA
(1+
do
(92)
XT
2
(
L(Og) L*
S1(1+
do? 2
+
(93)
XT
where (a) follows from Assumption 7.1
Rearranging the terms yields the inequality in (86
E.2 Proof of Theorem 7.5
Proof. We use the result in Theorem E.10 and set the noise multiplier o, such that privacy is guaranteed. To ensure the privacy guarantee, we use the result in Theorem I in [. For clarity, we first restate Theorem [I].
Theorem E.11 (Theorem 1 of [I]. There exist constants and such that, given the sampling probability q = B/N, for any E S TT the composition T Gaussian mechanisms, each with noise multiplier a, satisfies (c, (.8))PPj if
22 log((1))
a²
(94)
2
According to Theorem E.11 the composition of T Gaussian mechanisms ensures (e, 8) provided that the noise multiplier a is set to
v2 T log(1/0)
(95)
N2
Note that proposed algorithm (DP-SGD-AdaSie described in Algorithm 1 requires two noise multipliers, a, and or,, since each iteration involves two Gaussian mechanisms, s(.) and f(.). Accord- ing to Proposition 6.1 these two parallel Gaussian mechanisms equivalent single Gaussian mechanism with noise multiplier a, where
+
(96)
σ²
03
o²
To ensure (e, d)-DP for the overall algorithm after T iterations, we set ag and o, such that the equivalent noise multiplier in (6 satisfies follows:
(5). This can be achieved by setting 0g and o, as
(97)
N222
V² B²T log (1/0)
o²
(98)
N
with constants 24 and v, satisfying
1
1
(99)
The choices of a, and or in 197) and 98 together with constants v, and v, that satisfy the equality in 199, ensure that (96) holds and thereby guarantee (e, O)DP for the DP-SGD-AdaSig algorithm.
Substituting of from (97) into (86) in Theorem E. 10 rate X, which yields
we then optimize its LHS w.r.t. the learning
2N2 (L2)) L*) 3T (2N2,2 do² log (og())) (
(100)
Substituting this learning rate into the LHS concludes the result.
30

---

E.3
Proof of Corollary 7.7
Proof. The probability Pr A Sr can be lower bounded as
(a)
E
A
Pr | sr)
2
(101)
(b)
(102)
99
2
(103)
where (a) follows from Markov's inequality, (b) follows from Jensen's inequality, and (c) is due to the bounded variance assumption.
Using the derived probability lower bound in (03) on the LHS of 18 in Theorem
7.5 and consider-
ing r 2 max (in((SGG)) G}, we divide both sides by to obtain:
2
B ((((Oo) Tn) (1+ du? log())
T 1=
EI
T
2N
(104)
We next set r such that the RHS of the above inequality is minimized. Note that when r 2 S. the - function 7-7 achieves its minimum value at 26, and for 2 2Ç, the function is increasing.
Noting that Theorem 7.5 2 G, two cases may occur: namely, G s 2 G 2 2Ç. By considering these two cases, separately, we can conclude that
2
2C,
if S 26, G 2 2
T argmin
max (2ax)2(,7 G}.
(105)
G.
Substituting r, into 104) completes the proof.
F
Experiments Settings
This section provides further information on the experimental settings considered in the paper. We present more details the datasets and architectures used for the numerical experiments presented in the paper.
F. General Settings
Privacy Settings In all experiments, we fix the DP budget 6), and compute the noise multiplier, i.c. σ, numerically using the Opcaus library [40]. such that the DP budget spent after T iterations (or equivalently Tqq epochs) equals to the fixed budget (c, 6). Considering Proposition 6.1 for given different pairs of a, and a, can obtained, where decreasing one increases the other. To simplify the parameter space, we set choice prioritizes a more accurate update of the model over the saturation slope, as roughly updating in the descent direction is sufficient.
1.01 in all AdaSig experiments, making smaller than a,- This
Hyperparameters To ensure fair comparison, we the same batch size and number of epochs across all methods. For AdaSig, we tune the learning rates X, Xo and the initial saturation slope 0g. For vanilla clipping, we tune C and X. For Auto-S and PSAC, we the tuned value of C for vanilla clipping following their original papers 36), and tune hyperparameters and learning rate A. For the method in (3], tune the initial clipping threshold C*, clipping threshold learning rate learning rate X, db. and ?.
F.2 Settings for Image Classification
MNIST and NNSSI We use the 4-layer CNN with tanh activation proposed in [27 and described in Table of [34), with cross-entropy loss and the DP-SGD optimizer. The DP budget is set to () (3. 1)- For batch size, number of epochs, and momentum, which are common across different methods, we use the values reported (34. These values are summarized in Table Table 3 presents the best values of other hyperparameters for each method.
31

---

Table 2: Common hyperparameters' across different methods used for training CNN MNIST,
FashionMNIST, and CIFAR-10.
Parameter
MNIST
FasHionMNIST
CIFAR-10
Batch size (B)
512
2048
4096
Number of epochs (Tq)
40
40
60
Momentum
0.9
0.9
0.9
Table 3: Hyperparameters selected for each method for training CNN MNIST, MNSST and CIFAR-10.
Method
Parameter
MNIST
MNshionMNIST
CIFAR-10
C
0.1
0.1
0.1
Vanilla
0.5
4.0
3.0
0.01
0.01
0.01
Auto-S
C
0.1
0.1
0.1
0.5
4.0
3.0
r
0.1
0.1
0.1
PSAC
C
0.1
0.1
0.1
0.5
4.0
3.0
C
0.1
0.1
0.1
0.5
4.0
3.0
Method in
1CC
0.05
0.01
0.01
σb
40.0
30.0
25.0
Y
0.5
0.5
1.0
00
5.0
1.0
1.0
AdaSig
0.05
0.4
0.25
Aa
0.01
0.01
0.01
CIFAR-10 We use the 8-layer CNN with tanh activation from 27), detailed in Table 7 of [4] with cross-entropy loss and DP-SGD optimizer. The DP budget use the same batch size, number of epochs, and momentum across different methods as reported in The best values of other hyperparameters for different methods are reported in Table
(c, (3. 10 10-5). We
Table
ImageNette We use ImageNette, 10-class subset of ImageNet [9]
with an image size of 160 x 160. 25 and cross- loss. For training, the Westerov-accelenn
We consider ResNet-9 architecture (about 2.5 million parameters), with the Mish activation function
Adam (DP-NAdam) optimizer utilized [ We consider DP budget of (e, 6) = (8, 10-4). We follow [17] by using group normalization instead of batch normalization without scale normalization. The only difference is that we not apply the learning rate decay schedule. The ResNet-9 architecture can be found at [T7] All methods the same batch size and number of epochs, which are set according to the values reported in [6). as shown in Table Table lists the best values of the remaining hyperparameters for each method.
CelebA We use the same ResNet-9 architecture as for the ImageNette dataset, with the DP-Adam optimizer. The CelebA dataset contains 40 labels per image and we use it for both single-label and multi-label classification tasks.
*Note that the batch size and the number of epochs differ from those reported in 3 and used in 36 Our experiments showed that these hyperparameter changes yield better performance across baselines compared with those reported in the references.
advanced optimizers than SGD,GD, namely, DA-N and DP- Adam, respectively, to achieve improved performance these more challenging datasets. These experiments further demonstrate the wide applicability of AdaSig approach with different optimizers.
For ImageNette and CelebA,
Check also https: //gis///it/ttt/e github. com/ gkaissis/ 6ab667777199045525667776666
32

---

Table 4: Common hyperparameters across different methods used for training Resnet-9 ImageNette and CelebA.
Parameter
ImageNette
CelebA (Multi-labe//mmllnng)
Batch size (B)
1024
512
Number of epochs (Tq)
50
10
Table 5: Hyperparameters selected for each method for training ResNet-9 on ImageNette, and CelebA.
Method
Parameter
ImageNette
Celeb((Multi-label)
CelebA (Smiling)
C
1.5
0.1
0.1
Vanilla
X
5 x 10
10-3
10-3
0.01
0.01
0.01
Auto-S
C
1.5
0.1
0.1
X
10-4
10-3
10-3
T
0.1
0.1
0.1
PSAC
C
1.5
0.1
0.1
X
5 x 10
10
10
CO
1.5
0.1
0.1
X
5 x 10
5 x 10
2 x 10-1
Method in 3
0.01
0.01
0.05
30.0
20.0
25.0
7
3.0
1.0
1.0
00
5.0
1.0
1.0
AdaSig
X
10-3
x 10-4
5 x 10-4
0.02
0.001
0.02
For the single-label task, we perform binary classification considering the label 'Smiling', where the binary cross-entropy loss for training. In this scenario, the output layer of ResNer-9 consists of single neuron.
For the multi-label classification task, all available 40 labels are considered for prediction. Here, the output layer contains 40 neurons, and we use scalar loss function that averages the 40 binary cross-entropy losses from each label.
In both cases, we set the DP budget to (c, = (8,5 x 10 9).
The batch size and the number of epochs are set the same across all methods, following the settings 36) and are reported Table 4 The best values of other hyperparameters for each method are presented in Table
Setting for Sentence Classification
SST-2 and QNLI We pre-trained RoBERTa-biie model (about 125 million parameters) [22] and perform full parameter fine-tuning. We use the cross-entropy loss function. The AdamW optimizer 24 without weight decay is applied. A learning rate scheduler is used to linearly reduce the learning rate from its initial value to zero throughout the training process. The DP budget is set to (c, 6) = (3, 2N ). where N is the number of training samples for each dataset. It is worth mentioning that SST-2 and QNLI contain 67,349 and 104,743 training samples, respectively.
The batch size, number of epochs, and maximum sequence length for all methods are set according to 4, and are given in Table 6 Table 7 lists the best value of other hyperparameters for each method.
33

---

Table 6: Common hyperparameters across different methods used for RoBERTa-base full parameter fine-tuning on SST-2 and QNLI.
Parameter
SST-2
QNLI
Batch size (B)
1000
2000
Number of epochs (Tq)
Max sequence length
256
256
Table 7: Hyperparameters selected for each method for fine-tuning RoBERTAa-base SST-2 and
QNLI
Method
Parameter
SST-2
QNLI
C
0.1
0.1
Vanilla
X
x 10-4
x 10-4
r
0.01
0.01
Auto-S
C
0.1
0.1
À
5 x 10-4
5 x 10
0.1
0.1
PSAC
C
0.1
0.1
X
5 x 10
5 x 10
C
0.1
0.1
X
x 10
5 10
Method in 3
JJC
0.01
0.01
σb
35.00
25.0
1.0
1.0
00
1.0
1.0
AdaSig
X
5 x 10-4
x 10-4
Xa
0.005
0.01
F.4 Hardware and Software Information
All experiments are performed on server equipped with Intel Xeon E5-2683 v4 CPUs, NVIDIA V100 GPUs, and 251 GiB of memory. The operating system used AlmaLinux 9.3, and the CUDA Toolkit version is 12.2. The implementation of all training procedures is based on Py PoTch 2.3.0 and Opacus 1.4.1.
G Numerical Analysis of Direction Deviation and Magnitude Deviation
In this section, we provide additional figures to further assess the direction deviation and magnitude deviation achieved by the proposed AdaSig clipping method and the basclines.
G.1 Defining Metrics
Cosine Similarity To evaluate the direction deviation, we compute the cosine similarity between the aggregation of the clipped gradients, i.c., gi e EiEB, gi.,, and the true batch gradient, i.c.,
(gr.ge) ge Eies, gi,t: as cos ci. = TR. g: which measures the cosine the angle between g. and gi-
SNR To characterize the magnitude deviation, we define the signal-to-noose-ratio (SNR) as normalized magnitude of the aggregation of the clipped gradients. Let us denote by |l.. the magnitude of the aggregation of the clipped gradients after clipping, i.c., || || Σies, Si. We define the SNR to be the ratio of this magnitude to the standard deviation of the added privacy noise denoted by on. i.c., SNR [& For AdaSig, on a.. For vanilla clipping, Auto-S, and PSAC,
34

---

On Co. For the method in [3, on C3 This metric can describe the magnitude deviation against various approaches with potentially different privacy noise variances: as the SNR decreases, the performance degradation caused by the privacy noise becomes more severe.
G.2 Performance Comparison
ImageNette Figure shows the histogram of the SNR during training on the ImageNette dataset. The results depict that using AdaSig, the SNR is concentrated around larger values. We further recall the earlier observations reported Figure 3 in the main paper. There, we observe that using AdaSig, direction deviation is reduced compared with the baselines. Considering these two results, i.e., Figure 3 and Figure 4 we conclude that using AdaSig, the deteriorating impact of clipping is reduced with respect to both direction deviation and magnitude deviation. This demonstrates that, for ImageNette, AdaSig provides highly effective clipping that preserves closeness the true batch gradient.
CIFAR-10 We next present the results of similar experiments on CIFAR- 10. Figures 5 and show the histogram of cosine similarity and SNR while training the 8-layer CNN on CIFAR- 10, respectively. As observed in these figures, the cosine similarity distribution of AdaSig is skewed toward higher values relative to the baselines. This means that AdaSig incurs less direction deviation. Nevertheless, the histogram of the SNR achieved by AdaSig is close to those of the baselines. Thus, for CIFAR- 10, AdaSig clipping is still able to improve the trade-off between the direction deviation and the magnitude deviation.
Auto
Ad55g
0.4
PSAC
Ard
0.4
Vanilia
Ad15
0.4
Method
AdaSls
0.3
0.3
0.2
0.2
0.1
0.1
0.00
0.10
0.00
0.00
0.05
0.30
0.00
0.10
SMR
SNR
SNR
SNR
Figure 4: Comparison of normalized magnitude for different approaches during training ResNet-9 on ImageNette.
Ads
AdeSig
0.100
0.08
0.08
0.08
0.075
0.06
0.06
0.050
0.04
0.02
0.07
0.02
0.000
Cosine Similarity
Similarity
Similarity
Similarity
Figure 5: Comparison of cosine similarity for different approaches during training 8-layer CNN on
CIFAR-10.
0.4
Ad15g
0.4
AddSg)
0.4
A015g
0.4
AdaSig
Probebiity
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.2
0.1
0.0
0.00
0.05
0.10
0.15
0.0
0.00
0.05
0.10
0.15
0.0
0.00
0.05
0.10
0.15
0.0
0.00
0.05
0.10
0:15
Figure 6: Comparison of normalized magnitude for different approaches during training 8-layer CNN on CIFAR- 10.
35

---

H
Ablation Study: Sigmoid Clipping with Constant Cr
An ablation method for the AdaSig approach is to fixed across all iterations, i.e., 0z a, Vi. We examine sigmoid clipping with various fixed a values for training an 8-layer CNN on the CIFAR-10 dataset. The experimental setup is based on Appendix H2
H.1 Impact of a on Direction Deviation and Magnitude Deviation
In this section, we study the impact of on the trade-off between direction deviation and magnitude deviation. Figure shows the histogram of cosine similarity and SNR during the training for three fixed values of ax. As seen, decreasing the a value results in an increase in cosine similarity, while reducing the SNR. This occurs because decreasing expands the linear region of the sigmoid function, providing equal scaling for different gradient samples during clipping (see Figure [ in the main paper), which reduces the direction deviation. This observation also aligns with the numerical example presented Section of the main paper.
0.1
0.20
1.0
0.15
a= 1.0
Probability
50.0.
Probability
a= 50.0
0.15
0.10
0.10
0.05
0.05
0.00
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.05
0.10
0.15
Cosine Similarity
SNR
Figure 7: Trade-off between direction deviation and magnitude deviation for several fixed values during the training of an 8-layer CNN on CIFAR- 10. left: Cosine similarity histogram, right: SNR histogram.
H.2 Impact of Test Accuracy
Next, we present the average final test accuracies for sigmoid clipping with several fixed a values in Table 8 As observed, very low values of a (less than 0.1) result in low accuracy due to low SNR, while very large a values (greater than 5) also degrade accuracy due to direction deviation. However, for in the middle range, achieve the highest accuracy, resulting from balance between direction deviation and magnitude deviation. It is worth noting that the highest accuracy achieved by a = is still lower than the accuracy attained by the AdaSig method, shown in Table iin the main paper, which highlights that adaptively adjusting a over iterations further improves accuracy.
Table Average test accuracies with 95% confidence intervals over five runs for sigmoid clipping with various fixed values.
0.01
0.1
1.0
5.0
50.0
100.0
Average test accuracy (%) 45.27 + 0.67
61.74 0.21
62.13
0.12
61.64
0.27
60.62 + 0.38
60.39 + 0.31
I Variation of a; During Training
In this section, we illustrate how a evolves during training on CIFAR-10 for three different random seeds. As shown in Figure g the general trend across seeds is that a initially increases from its starting value 1.0 up to a peak in the middle training, and then gradually decreases toward the end. The exact trajectory varies across seeds since each seed corresponds to different batch sampling process, leading to distinct gradient statistics in each run.
When examining a; over training iterations across different datasets and models, we do not observe consistent pattern. This behavior arises from the dependence of or on individual sample gradients, which vary substantially across datasets and model architectures.
36

---

1.4
1.3
1.2
8
1.1
1.0
0.9
0
100
200
300
400
500
600
700
Iteration (t)
Figure Evolution of a across training iterations on CIFAR-I0 for three random seeds.
J Convergence Plots
In this section, we present convergence plots for several experiments. Figure SesurePillsser Gillustrates the average test accuracy and test loss over five runs versus epochs during training on the CIFAR-10 dataset. The shaded regions around the curves represent the 95% confidence intervals. As shown, the increasing test accuracy in the left plot and the decreasing test loss in the right plot demonstrate convergence for all methods, including AdaSig. Additionally, the figure highlights that AdaSig achieves lower final test loss and higher final test accuracy compared with the baselines.
60
2.25
Vanilla
AdaSig Method in [3)
Test Accuracy (%)
Auto-S
Test Loss
2.00
PSAC
40
60
1.75
Vanilla Auto-S
AdaSig Method
20
1.50
1.25
20
40
60
20
40
60
Epoch
Epoch
Figure 9: Convergence plot for CIFAR-10 dataset, left: Test accuracy vs. epoch, right: Test loss vs. epoch.
Additionally, Figure I0 shows the average test loss and accuracy throughout the training the SST-2 dataset. As observed, all methods progressively reduce the test loss while increasing accuracy, with AdaSig outperforming the other methods.
90
Test Accuracy (%)
80
88
70
Vanilla
87
Test
Auto-S
60
Vanilla
Ada5ig
1.0
PSAC
Auto-S
Method [3]
AdaSig
PSAC
Method
[3]
50
1
1
3
Epoch
Epoch
Figure 10: Convergence plot for SST-2 dataset, left: Test accuracy epoch, right: Test loss vs.
epoch.
37


Reviews:
[{"review_id": "wYLVZLoN3S", "reviewer": "Reviewer_n68q", "summary": "The paper addresses an important question in differentially private SGD. In standard DPSGD, each gradient is clipped to a fixed length C, to restrict the sensitivity of the corresponding data point. \n\nHowever, this clipping means large gradients are clipped more than small ones, thereby fundamentally affecting the direction of the SGD path. The alternative is to scale down all gradients such that the largest gradients are at most C. But in this process, the smaller gradients become tiny, affecting the magnitude of the SGD steps. \n\nThe paper proposes a non-uniform scaling given by the sigmoid eq (5) and (6) that is meant to address the tradeoff of keeping sensitivity bounded, retaining some of the relative importance of large vs small gradients, and ensuring the small gradients are not too small. \n\nThe parameter $\\alpha$ in the equation is updated dynamically during the training process in a privacy preserving manner. \n\nThe paper includes theorems about privacy guarantees and convergence of the training.", "questions": "1. Can you explain why a dynamic adaptation to $\\alpha$ is needed and why a fixed $\\alpha$ does not suffice? \n2. Can you explain what your algorithm is to update $\\alpha$? \n3. Can you explain the novelty of the idea with respect to Auto-S, NSGD and PASC? These also seem to address the same issues with somewhat similar ideas. I read the discussion in related works, but did not quite catch the difference. Examples may help.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe paper is on an important and timely topic. It contributed interesting ideas to the area. It has theoretical guarantees for privacy and convergence. Experimental results on a range of models and data show good performance. \n\nWeaknesses: \n\nThe need for adaptive changes to $\\alpha$ are not explained. The process for deriving the right $\\alpha$ in 5.2 is also unclear. \n\nMinor comments: \n\nThe explanations in lines 68-75 and 152-166 could be made more clear. They are important for understanding the paper, but not easy to follow.", "was_revised": true, "final_justification": "After the discussion, I feel my intitial understanding of the paper was accurate enough. I am thus retaining the rating and support acceptance of the paper. \n\nI am have not checked the proofs in detail.", "rebuttal": "We thank the reviewer for their comments and address their concerns below.\n\n> 1- Can you explain why a dynamic adaptation to $\\alpha$  is needed and why a fixed $\\alpha$ does not suffice?\n\nThe value of $\\alpha$ affects the span of the linear region, as illustrated in Fig. 1: a smaller $\\alpha$ leads to a larger linear region (e.g., see the curves for $\\alpha = 0.8$ and $\\alpha = 10.0$). Thus, varying $\\alpha$ allows us to adjust the span of the linear region and control the trade-off between direction and magnitude deviations throughout training. In contrast, a fixed value of $\\alpha$ results in a static linear region span. Since the distribution of sample gradient norms $||\\mathbf{g}_{i,t}||$ can vary significantly during training, using a fixed $\\alpha$ across all iterations cannot effectively balance this trade-off at all times.\n\nTo make this more concrete, consider Fig.1 and assume we have selected a fixed $\\alpha = 0.8$ across all iterations. After several iterations, the maximum norm of the sample gradients within a batch may decrease significantly. In this case, it becomes possible to shrink the span of the linear region to reduce the magnitude deviation without causing excessive direction deviation. Recall from Fig. 1 that smaller linear regions, which correspond to larger $\\alpha$ values such as $\\alpha = 10$, introduce less magnitude deviation and more direction deviation. Thus, when the gradient norms are smaller, a larger $\\alpha$ (that is, a narrower linear region) becomes preferable, as the risk of large direction deviation is mitigated due to the smaller gradient magnitudes. It is worth mentioning that this is just an illustrative case; in practice, gradient statistics may vary arbitrarily over time. Furthermore, the optimal adjustment of $\\alpha$ depends not only on the maximum norm but on the overall distribution of the sample gradients. Our algorithm leverages this richer information to dynamically update $\\alpha$ during training.\n\nWe have experimentally validated the benefit of using an adaptive $\\alpha$ in Appendix H.2 in the supplementary file. In Table 8, we show that the performance achieved with a fixed $\\alpha$ remains consistently below that of our adaptive approach, highlighting the benefit of dynamically adjusting $\\alpha$ during training.\n\n\n\n> 2- Can you explain what your algorithm is to update $\\alpha$?\n\n\nOur algorithm updates $\\alpha$ to reduce the learning loss while preserving privacy. In brief, the procedure for updating $\\alpha$ in the $t$-th iteration is as follows:\n\n1. Computing $\\hat{s}_t$ which is defined in equation (9b). This value is already computed in the model updating step in equation (10).\n2. Computing $\\alpha_{t+1}$ from equation (15), using the value of $\\hat{s_t}$ and $\\hat{r}_{t-1}$, which was computed in the previous step. \n3. Computing $\\hat{r}_t$ using equation (13) and the definition of $r$ provided in Lemma 5.1. $\\hat{r}_t$ will be used in the next iteration for $\\alpha$ update in step 2.\n\nFor further clarity, we kindly refer the reviewer to Algorithm 1 in Appendix A. We will also include additional clarifications in the algorithm description in the final version of the paper to make this procedure more transparent.\n\n\n\n> 3- Can you explain the novelty of the idea w.r.t. Auto-S, NSGD and PASC? These also seem to address the same issues with somewhat similar ideas. I read the discussion in related works, but did not quite catch the difference. Examples may help.\n\n\nAuto-S clipping (also known as NSGD) was proposed to eliminate the need for tuning the clipping threshold $C$ in vanilla clipping, which affects the bias-variance trade-off. This clipping operation is **not** designed to mitigate the *deviation effects* introduced by clipping. As shown in Fig. 1, Auto-S has a small linear region, which leads to significant direction deviation. This observation has also been validated numerically in [1].\n\nLater, PSAC clipping was proposed to address the direction mismatch of Auto-S by assigning smaller weights to sample gradients with smaller norms. As shown in Fig. 1, the PSAC curve has a wider linear region to serve this purpose. However, the PSAC clipping has a **fixed** linear region; that is, it cannot adaptively expand or shrink its linear region to balance the trade-off between direction and magnitude deviations (see also our response to the first question).\n\nIn contrast, our proposed clipping method is based on an **adaptive** sigmoid function that can dynamically balance the aforementioned trade-off by adjusting its linear span through varying $\\alpha$.\n\n\n> Minor comments: The explanations in lines 68-75 and 152-166 could be made more clear. They are important for understanding the paper, but not easy to follow.\n\n\nIn lines 68–75, we discuss the issue of magnitude deviation and its trade-off with the direction deviation. The span of the linear region in Fig. 1 governs this trade-off. A larger linear region—corresponding to smaller values of $\\alpha$—reduces direction deviation but increases magnitude deviation. This occurs because, for gradients with small norms (points close to 0 on the horizontal axis), the red curve (with a larger linear region) scales down the norm of the clipped gradients more severely than the green curve (with a smaller linear region), as shown in Fig. 1. This effect is further illustrated in Fig. 2(c) and 2(d). The dashed green arrow in Fig. 2(c) corresponds to the aggregation of clipped gradients for a large $\\alpha$ value (i.e., smaller linear region), the dashed red arrow in Fig. 2(d) represents the aggregation of clipped gradients with a small $\\alpha$ value (i.e., larger linear region), and the dashed blue arrow shows the summation of unclipped gradients. As shown, the dashed green arrow has a closer norm to the dashed blue arrow compared with the dashed red arrow. In contrast, the dashed red arrow exhibits reduced direction mismatch relative to the dashed blue arrow, but at the cost of increased magnitude deviation.  In summary, adjusting the value of $\\alpha$ controls the size of the linear region and thus allows for balancing the trade-off between direction deviation and magnitude deviation. \n\nLines 152-166 also refer to the same aforementioned trade-off, providing further elaboration. We will revise the indicated sections of the paper to enhance clarity.\n\n\n[1] Tianyu Xia, Shuheng Shen, Su Yao, Xinyi Fu, Ke Xu, Xiaolong Xu, and Xing Fu. Differentially 426 private learning with per-sample adaptive clipping. In Proc. of the AAAI Conference on Artificial 427 Intelligence, 2023."}, {"review_id": "soMDrw62x7", "reviewer": "Reviewer_R2gd", "summary": "The authors propose a learnable sigmoid‐based clipping scheme for DP-SGD that smoothly trades off gradient direction and magnitude errors. Upon privately learning the sigmoid’s slope $\\alpha$ in addition to the model weights, they provide a sharp non-convex convergence rates while improving on the accuracy-privacy trade-off.", "questions": "1.) How are the seemingly similar looking performance numbers in Table 1 justified as an improvement? Please provide some additional context to help understand the suggested gains. For example PSAC 86.35 ± 0.19 Vs. proposed method at  86.68 ± 0.04 in the 2nd row of Table 1 with Fashion MNIST.\n\n2.) How does the following clipping result for DP compare with your proposed result? https://arxiv.org/pdf/2502.11682 . Please cite and compare. Similarly, do so with https://arxiv.org/pdf/2505.20817? which is for a narrower class of functions.", "limitations": "They haven't compared against this paper: https://arxiv.org/pdf/2502.11682 where authors claim that for non-convex smooth distributed problems with clients having arbitrarily heterogeneous data, they achieve an optimal convergence rate and also near optimal (local-)DP neighborhood.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\nThey provide the following equivalence in the extreme case showing an $O(\\sqrt{d} /(N \\varepsilon))$ that matched the vanilla DP-SGD in the asymptotic case when the iterations $T$ tend to infinity. The convergence analysis is done in good detail.\n\nRoom for improvement:\n1.) The authors show that privacy accounting leads to the same effect in proposed method and DP-SGD when $\\sigma_r=\\left(\\sigma^{-2}-\\sigma_s^{-2}\\right)^{-1 / 2}$. But they don't comment on the other settings, on how these two compare. For instance, the work in https://arxiv.org/pdf/2205.13710 shows that privacy-loss or degradation converges and is quite an ideal property to have as opposed to an unending loss. \n\nCurrent privacy analyses of Noisy-SGD give an $(\\alpha, \\varepsilon)$-RDP upper bound of $$\n\\varepsilon \\lesssim \\frac{\\alpha L^2}{n^2 \\sigma^2} T$$ which increases ad infinitum as the number of iterations $T \\rightarrow \\infty$. The Altschuler-Talwar work above contains that increase as it shows a cut-off where the privacy loss begins to converge.\n\nHow does this above issue and corresponding sota result benefit from your adaptive clipping based version of DP-SGD?\n\n2.) How does the following clipping result for DP compare with your proposed result? https://arxiv.org/pdf/2502.11682 . Please cite and compare. Similarly, do so with https://arxiv.org/pdf/2505.20817? which is for a narrower class of functions.", "was_revised": true, "final_justification": "Due to the effect of alpha being unclear in the experiments, I uphold my score. The authors have done a good rebuttal on rest of the points.", "rebuttal": "We thank the reviewer for their valuable feedback.\n> Room for improvement 1.)  RDP bound in Noisy-SGD in [1]:\n\nWe appreciate the reviewer’s reference to [1], which provides an important contribution by showing that the RDP privacy loss of Noisy-SGD can converge to a bounded value as the number of iterations $T \\to \\infty$, in contrast to standard analyses where the privacy loss grows unbounded.\n\nHowever, it is important to note that the analysis in [1] assumes **unclipped** gradients, as also mentioned in the discussion section of their paper. In contrast, our work focuses on the **clipped** setting; specifically, our analysis focuses on how adaptive clipping influences both training convergence and privacy guarantees.\n\nWe believe extending the theoretical results for RDP analysis in [1] to the clipped regime remains an open and nontrivial direction. Once addressed, it could potentially open the door to analyzing more advanced clipping strategies, including our proposed AdaSig method.\n\n\n> Room for improvement 2.) How does the following clipping result for DP compare with your proposed result? [2]. Please cite and compare. Similarly, do so with [3]?  which is for a narrower class of functions.\n\nWe thank the reviewer for pointing out these two recent works [2] and [3]. We will cite these two works to suggest potential directions for future research. Inspired by [2], one possible direction is to extend our AdaSig approach to the federated learning (FL) framework. Moreover, drawing on the methodology of [3], developing a high-probability convergence analysis would be a valuable theoretical extension of our work. Below, we highlight the key distinctions between our work and these papers:\n\n\n**Reference [2]:**\n\nThe work in [2] considers a Federated Learning (FL) setting and proposes an algorithm that combines vanilla clipping, momentum, and error feedback. The authors show that this combination achieves an optimal convergence rate under client heterogeneity.\n\nWhile this is an important contribution for the FL setting, we note that the paper does *not* introduce a new clipping strategy, i.e., it relies on standard (vanilla) clipping. In contrast, our work focuses on the design of a new adaptive clipping mechanism, AdaSig, which dynamically balances direction and magnitude deviations in the clipped gradients, while maintaining optimal convergence guarantees. We note that a comparison with vanilla clipping used in [2] is already included in our experiments. \n\nMoreover, our work focuses on centralized (non-distributed) learning, whereas [2] addresses the FL setting, where client heterogeneity is a key challenge. This challenge does not arise in our scenario. That said, our proposed AdaSig clipping strategy is general and can also be applied in the FL setting. Like any clipping method, it can be combined with techniques such as error feedback and momentum to enhance performance under heterogeneous clients. We view this as a promising direction for future studies.\n\n\n\n**Reference [3]:**\n\nThe work in [3] provides a high-probability convergence bound for vanilla clipped SGD under $(L_0, L_1)$-smoothness assumptions and heavy-tailed noise. In contrast, our work focuses on proposing a new adaptive clipping strategy, AdaSig, which aims to balance the distortion introduced by clipping in both gradient direction and magnitude, and establishes a convergence bound for this adaptive method.\n\nFrom a theoretical perspective, we note that the high-probability convergence analysis in [3] for vanilla clipping could potentially be extended to our adaptive clipping method. However, this extension is non-trivial and represents a promising direction for future work.\n\nFrom an experimental standpoint, we emphasize that our proposed method has already been compared against vanilla clipping, which is the clipping method in [3].\n\n>- How are the seemingly similar-looking performance numbers in Table 1 justified as an improvement? Please provide some additional context to help understand the suggested gains. For example, PSAC 86.35 ± 0.19 Vs. proposed method at 86.68 ± 0.04 in the 2nd row of Table 1 with Fashion MNIST.\n\nWe would like to clarify that we performed a statistical $t$-test to assess the significance of the results. Specifically, the bold entries in Table 1 indicate cases where the $p$-value is less than 0.05, demonstrating that the observed improvements are statistically significant [4].\n\nIn the specific example mentioned (second row), although the mean values are close (86.35 vs. 86.68), the $p$-value is below 0.05, confirming the statistical significance of the improvement. Furthermore, our method achieves a smaller confidence interval compared with PSAC in this case, indicating greater stability. We also note that in several other cases (e.g., rows 3, 4, and 7), the improvements over PSAC are more pronounced.\n\n\n> – Figure 3 caption: ImageNette should be ImageNet.\n\nWe clarify that ImageNette [5] is a publicly available subset of ImageNet. It consists of 10 easily distinguishable classes from the original ImageNet dataset and is commonly used when full ImageNet training is computationally prohibitive. We will include a citation to [5] in the final version of the paper for clarity.\n\n\n[1] J. Altschuler and K. Talwar, “Privacy of noisy stochastic gradient descent: More iterations without more privacy loss,” Advances in Neural Information Processing Systems, vol. 35, pp. 3788–3800, 2022.\n\n[2] R. Islamov, S. Horvath, A. Lucchi, P. Richtarik, and E. Gorbunov, “Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy,” arXiv preprint arXiv:2502.11682, 2025.\n\n[3] S. Chezhegov, A. Beznosikov, S. Horváth, and E. Gorbunov, “Convergence of Clipped-SGD for Convex $(L_0, L_1)$-Smooth Optimization with Heavy-Tailed Noise,” arXiv preprint arXiv:2505.20817, 2025.\n\n[4] Greenland, S., et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. In European journal of epidemiology, 2016.\n\n[5] J. Howard, ImageNette: A subset of 10 easily classified classes from ImageNet, fast.ai, 2019."}, {"review_id": "xkfgOhDQxj", "reviewer": "Reviewer_8LLB", "summary": "This work proposes AdaSig, an adaptive clipping strategy for Differentially Private Stochastic Gradient Descent (DP-SGD). Specifically, AdaSig clips the sampled gradients based on a sigmoid function for a proper balance between direction deviation and magnitude deviation. Extensive experiments on both sentence and image classification tasks demonstrate that AdaSig consistently outperforms existing clipping methods.", "questions": "Overall, I have the following questions, labeled as Q1-Q4. If all these concerns are addressed convincingly, I would increase my rating of this paper.\n\nQ1. Is it necessary to adopt the sigmoid function in this context? What are the unique benefits of using it compared to other functions, such as $\\tilde{\\mathbf{g}}_{i,t} = \\frac{C \\mathbf{g}_{i,t}}{\\Vert \\mathbf{g}_{i,t} \\Vert + r}$\n        in [3] and                \n$\\tilde{\\mathbf{g}}_{i,t} = \\frac{C \\mathbf{g}_{i,t}}{\\Vert \\mathbf{g}_{i,t} \\Vert + \\frac{r}{\\Vert \\mathbf{g}_{i,t} \\Vert + r}}$ in [32]. A theoretical comparison and experimental results would be helpful.\n\nQ2. Would introducing a learnable parameter $r$ in the equation $\\tilde{\\mathbf{g}}_{i,t} = \\frac{C \\mathbf{g}_{i,t}}{\\Vert \\mathbf{g}_{i,t} \\Vert + \\frac{r}{\\Vert \\mathbf{g}_{i,t} \\Vert + r}}$\n         in [32] also achieve comparable performance to AdaSig? It would be valuable to include experimental results evaluating a learnable $r$ in [32] for a direct comparison.\n\nQ3. What is the main technical difference between AdaSig and the method in [11]? AdaSig appears to be a simple application of the method in [11]. A more detailed explanation would be beneficial for better understanding the novelty of AdaSig.\n\nQ4. How does $\\alpha$ evolve throughout the training process? Providing a traceable trajectory of $\\alpha$ could offer valuable insights into the adaptive behavior of the model.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths (labeled as S1, S2, S3, etc.):\n\nS1. This paper addresses a practical and significant problem, i.e., how to clip gradients during training to achieve a better trade-off between magnitude and direction deviations.\n\nS2. The workflow of this paper is clearly presented, and the related works discussed in this paper are comprehensive and representative.\n\nS3. The experiments are thorough, and the results provide compelling evidence for the effectiveness of the proposed method.\n\nWeaknesses (labeled as W1, W2, W3, etc.):\n\nW1. The authors propose a clipping operation based on the sigmoid function, and treat $\\alpha$ as a learnable parameter for adjusting the deviations adaptively. However, it appears that the existing solution [32] can also reduce the direction deviation effectively. The key difference is that its linear region span remains fixed, as the corresponding parameter $r$ is not learnable. So is it necessary to leverage the sigmoid function? It seems that a learnable parameter $r$ in the equation \n$$\\tilde{\\mathbf{g}}_{i,t} = \\frac{C \\mathbf{g}_{i,t}}{\\Vert \\mathbf{g}_{i,t} \\Vert + \\frac{r}{\\Vert \\mathbf{g}_{i,t} \\Vert + r}}$$\n     in [32] could also achieve comparable performance. A discussion on the necessity of the sigmoid function and the experiments on the learnable $r$ in [32] would be helpful.\n\nW2. The authors focus on the trade-off between magnitude and direction deviations, but the proposed method appears to be an application or extension of that in [11]. What is the main technical difference between AdaSig and the method in [11]? A more detailed explanation would be beneficial for clarifying the novelty and contribution of the proposed method.\n\n\nW3. The authors treat $\\alpha$ as a learnable parameter for adjusting the deviations adaptively. Although Appendix H analyzes the impact of different $\\alpha$ values, the evolution of $\\alpha$ during the training process still remains unclear. Providing a traceable trajectory of $\\alpha$ during training would bring valuable insights.\n\nW4. There are a few writing issues in the paper.\n\n* The title of Section 3 contains a typo: \"Backgound\" should be corrected to \"Background\".\n\n\n* In line 127 in Section 3, a period is missing between \"output\" and \"The\".\n\n* In Table 1, the values in AdaSig column should be consistently highlighted in bold.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s feedback.\n\n> Q1. Is it necessary to adopt the sigmoid function in this context? What are the unique benefits of using it compared to other functions, such as $\\tilde{\\mathbf{g}}{i,t} = \\frac{C \\mathbf{g}{i,t}}{\\Vert \\mathbf{g}{i,t} \\Vert + r}$ \nin [3] and $\\tilde{\\mathbf{g}}{i,t} = \\frac{C \\mathbf{g}_{i,t}}{\\Vert \\mathbf{g}{i,t} \\Vert + \\frac{r}{\\Vert \\mathbf{g}{i,t} \\Vert + r}}$ in [32]. A theoretical comparison and experimental results would be helpful.\n\n\nThe sigmoid function is simple yet well-suited to our goal of adaptively balancing the trade-off between direction and magnitude deviations. In particular, it enables control over the range of the linear span in Fig. 1 through the parameter $\\alpha$. The sigmoid function provides greater flexibility and control compared with the clipping functions proposed in [32] and [3], even if those functions are extended with learnable parameters. \n\nThe structure of the clipping function in [32] is such that the span of its linear region changes within a small range as $r$ varies. We are not allowed to include plots at this stage, but this effect is visible when the function is plotted for various values of $r$. We will incorporate these plots in the final version of the paper. This behavior can also be understood by analyzing the function in extreme cases. Due to the presence of the term $\\frac{r}{x + r}$ in the denominator of PSAC, varying $r$ from $0$ to $\\infty$ causes the function to transition between two limiting values: $1$ and $\\frac{x}{x + 1}$. Hence, the span of its linear region is restricted between these two curves, and adaptively updating $r$ does not effectively balance the trade-off between direction and magnitude deviations. In contrast, the sigmoid function used in AdaSig can effectively span from values close to 1 down to curves near 0, thereby flexibly balancing the trade-off. \n\nWhile the clipping function in [3] does not suffer from this limited range of the linear span, it is less flexible than the sigmoid function in balancing the trade-off, as its values are less responsive to changes in $r$. Therefore, learning the optimal parameter requires greater variation across different values, whereas the sigmoid function, with its exponential change in $\\alpha$, provides more flexible control over the linear region and thus, the direction-magnitude mismatch trade-off.\n\n\n> Q2. Would introducing a learnable parameter $r$ in the equation $\\tilde{\\mathbf{g}}{i,t} = \\frac{C \\mathbf{g}{i,t}}{\\Vert \\mathbf{g}{i,t} \\Vert + \\frac{r}{\\Vert \\mathbf{g}{i,t} \\Vert + r}}$ in [32] also achieve comparable performance to AdaSig? It would be valuable to include experimental results evaluating a learnable $r$ in [32]  for a direct comparison.\n\nWe first would like to clarify that deriving a differentially private update for the parameter of the clipping function is not straightforward and involves several steps that depend on the specific form of the clipping function. Specifically, the loss function must first be expressed in terms of the learnable parameter to compute its derivative w.r.t. that parameter. Then, to construct the private update, it is necessary to identify a constant upper bound on this derivative to guarantee bounded $\\ell_2$-sensitivity, which allows the update to be privatized by adding a suitable amount of Gaussian noise.\n\nFollowing the reviewer’s suggestion, we have carried out the derivations and conducted experiments for the method in [32]; however, since the calculations exceed the allowed space, we are unable to include them here. Furthermore, due to the extensive tuning required to find the optimal initial value for $r$, we were only able to perform experiments on the CIFAR-10 dataset. The best accuracy obtained by this method was $61.96 \\pm 0.15\\%$, which is not significantly different from that of the original method. Considering the limited span of the linear region in [32], as discussed in our response to Q1, we believe these results are intuitive, since even with a learnable $r$, it has limited capacity to balance the direction-magnitude mismatch trade-off.\n\nWe will conduct experiments on other datasets and include the results in the final version of the paper if the reviewer considers them necessary.\n\n\n> Q3. What is the main technical difference between AdaSig and the method in [11]? AdaSig appears to be a simple application of the method in [11]. A more detailed explanation would be beneficial for better understanding the novelty of AdaSig. \n\n\nFirst, we emphasize that AdaSig does **not** use the method in [11]. In [11], vanilla clipping is extended by optimizing the clipping threshold over iterations to balance the trade-off between bias and variance. Therefore, [11] does not introduce a new clipping methodology, but rather a new approach to applying vanilla clipping. Furthermore, it is unclear whether the method in [11] provides a theoretical convergence guarantee, as none is given in that paper.\n\nDespite both [11] and our work optimizing a learnable parameter, there are three key differences between them:  (i) AdaSig introduces a novel clipping methodology, whereas [11] builds on existing vanilla clipping as mentioned above.  (ii) The optimized parameters differ, with AdaSig tuning the saturation slope and [11] tuning the clipping threshold.  (iii) The purpose of the optimization is fundamentally different. AdaSig aims to balance the trade-off between the two components of bias, namely the direction and magnitude deviations, while keeping the variance fixed. In contrast, [11] seeks to balance the trade-off between bias and variance. More specifically, since AdaSig uses a sigmoid-based clipping function, optimizing the saturation slope $\\alpha$ does not affect the output range. As a result, the $\\ell_2$ sensitivity remains constant when changing $\\alpha$ in AdaSig, leading to a fixed variance for the Gaussian noise used in the privacy mechanism. On the other hand, in [11], updating the clipping threshold changes the $\\ell_2$ sensitivity, which in turn affects the variance of the added privacy noise.\n\n\n\n\n\n> Q4. How does $\\alpha$ evolve throughout the training process? Providing a traceable trajectory of $\\alpha$ could offer valuable insights into the adaptive behavior of the model.\n\nSince we are not allowed to include figures in our response, we provide a verbal description of the $\\alpha_t$ behavior here and will include the plots in the final version.\n\nFor example, on CIFAR-10, we observe that the average $\\alpha_t$ increases from the initial value of 1.0 to just below 1.5 within the first 20 iterations. It then steadily decreases until around iteration 200, after which it begins to rise again, ultimately surpassing 1.1 by the end.\n\nObserving the changes in $\\alpha_t$ over iterations across various datasets and models, we find no consistent trend in its variation. We conjecture that this is due to $\\alpha_t$'s dependence on the gradients of individual samples, which can vary significantly across different datasets and model architectures.\n\n\nWe thank the reviewer for pointing out the typos. We will carefully revise the manuscript to correct them."}, {"review_id": "0Tgqm896WZ", "reviewer": "Reviewer_e8WR", "summary": "The paper proposes a new adaptive clipping method for DP-SGD.\n\nIn the standard DP-SGD algorithm, a fixed clipping threshold is applied uniformly to each gradient in a batch. \nThis can lead to a significant mismatch between the direction of the sum of the clipped gradients and that of the original gradients, potentially resulting in degraded model performance.\n\nLinear scaling offers a potential solution: if all gradients are scaled by the same factor, the direction of their sum is preserved. \nThis motivates the design of clipping mechanisms that approximate linear scaling for most gradients, while still enforcing an upper bound on gradients with large $\\ell_2$ norms to ensure differential privacy after adding noise. Several prior works have explored this direction.\n\nHowever, compared to fixed-threshold clipping, these methods often reduce the total $\\ell_2$ norm of the clipped gradients more aggressively, which may in turn lower the signal-to-noise ratio once Gaussian noise is added. \nStriking a balance between preserving the direction and maintaining the norm of the summed gradients remains a key challenge.\n\nThis paper addresses this challenge by introducing a new clipping mechanism based on the sigmoid function. \nThe method includes a parameter that explicitly controls the trade-off between preserving gradient direction and magnitude. \nMoreover, this parameter is adaptively optimized for each batch in a differentially private manner.\n\nExperimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches across various benchmarks.", "questions": "Does the deep learning model studied in the experimental section (Section 8) satisfy the assumptions made in the theoretical analysis (Section 7)? \n\nIf not, it is unclear what the role of the theoretical section is, given that the motivation and intuition behind the proposed clipping method are already well-articulated earlier in the paper. \nClarifying the connection between the theory and the experiments would strengthen the overall contribution.", "limitations": "See questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper studies an interesting problem and provides a concise and insightful description of existing solutions, clearly highlighting their advantages and limitations. \nIt is carefully written and easy to follow.\n\nThe proposed method for automatically updating the clipping threshold is elegant, and the experiments demonstrate improvements over the existing state-of-the-art.\n\nThe only shortcoming is that the theoretical section contributes little to the experimental results, and the connection between the two could be made stronger.", "was_revised": true, "final_justification": "While I understand that the theoretical analysis aims to provide insight under standard assumptions, the paper would be strengthened by more clearly explaining how these assumptions hold or partially hold in practice. \n\nI will keep my score unchanged.", "rebuttal": "We are grateful for the reviewer’s insights.\n> – Does the deep learning model studied in the experimental section (Section 8) satisfy the assumptions made in the theoretical analysis (Section 7)? If not, it is unclear what the role of the theoretical section is, given that the motivation and intuition behind the proposed clipping method are already well-articulated earlier in the paper. Clarifying the connection between the theory and the experiments would strengthen the overall contribution.\n\nThe purpose of our theoretical analysis in Section 7 is to show that the proposed adaptive clipping retains the best-known convergence rate in the non-convex loss setting. This analysis serves as a basis for understanding the convergence behavior of the algorithm. It is natural to impose some assumptions on the loss function in order to theoretically characterize convergence. While these assumptions may not hold exactly in practice, they still offer valuable insight into how different parameters can affect the convergence bound.\n\nThe assumptions we use—such as smoothness and bounded gradients—are standard in the literature [1-4] and broad enough to cover a wide range of models and loss functions. Although some of these may only partially hold for the deep learning models used in our experiments, the theoretical results remain informative and provide useful guidance for the design and behavior of our method in practical settings.\n\nWe thank the reviewer for their suggestion on clarifying the role of the theoretical analysis, and we will reflect this discussion in the final version of the paper.\n\n\n[1] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. In NeurIPS, 2023.\n\n[2] Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In NeurIPS, 2020.\n\n[3] Di Wang, MinweiYe, andJinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. In NeurIPS, 2017.\n\n[4] Xinwei Zhang, Zhiqi Bu, Steven Wu, and Mingyi Hong. Differentially private SGD without 440 441 clipping bias: An error-feedback approach. In ICLR, 2024."}]

Author rebuttal:

Meta-review: This work proposes AdaSig, an adaptive clipping strategy for Differentially Private Stochastic Gradient Descent (DP-SGD). Specifically, AdaSig clips the sampled gradients based on a sigmoid function for a proper balance between direction deviation and magnitude deviation. Extensive experiments on both sentence and image classification tasks demonstrate that AdaSig consistently outperforms existing clipping methods.

Strength: (1) The paper is on an important and timely topic. It contributed interesting ideas to the area. It has theoretical guarantees for privacy and convergence. Experimental results on a range of models and data show good performance.

(2)  The workflow of this paper is clearly presented, and the related works discussed in this paper are comprehensive and representative.

Weakness: (1)The theoretical section contributes little to the experimental results. 

As the clipping operator is very important in DP-SGD, the paper makes a fundamental contribution to the DP community. Although there are some improvement of the paper such as a finer theoretical analysis or the scalability issues. I would like to accept the paper.
Decision: Accept (poster)