Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective
Authors: Enea Monzio Compagnoni
Alessandro Stanghellini
Rustem Islamov
Aurelien Lucchi
Anastasia Koloskova
Venue: ICLR 2026 Poster (2026)
Keywords: Stochastic Differential Equations
Differential Privacy
Primary area: optimization
TL;DR: With SDEs, we show that while DP-SignSGD is better under tight privacy or noisy batches, DP-SGD is better otherwise, and adaptivity needs far less hyperparameter tuning across privacy levels.

Abstract: Differential Privacy (DP) is becoming central to large-scale training as privacy regulations tighten. We revisit how DP noise interacts with _adaptivity_ in optimization through the lens of _stochastic differential equations_, providing the first SDE-based analysis of private optimizers. Focusing on *DP-SGD* and *DP-SignSGD* under per-example clipping, we show a sharp contrast under fixed hyperparameters: *DP-SGD* converges at a Privacy-Utility Trade-Off of $\mathcal{O}(1/\varepsilon^2)$ with speed independent of $\varepsilon$, while *DP-SignSGD* converges at a speed *linear* in $\varepsilon$ with a $\mathcal{O}(1/\varepsilon)$ trade-off, dominating in high-privacy or large batch noise regimes. By contrast, under optimal learning rates, both methods achieve comparable theoretical asymptotic performance; however, the optimal learning rate of *DP-SGD* scales linearly with $\varepsilon$, while that of *DP-SignSGD* is essentially $\varepsilon$-independent. This makes adaptive methods far more practical, as their hyperparameters transfer across privacy levels with little or no re-tuning. Empirical results confirm our theory across training and test metrics, and empirically extend from *DP-SignSGD* to *DP-Adam*.

Published as a conference paper at ICLR 2026
ADAPTIVE METHODS ARE PREFERABLE IN HIGH PRIVACY SETTINGS: AN SDE PERSPECTIVE
Enea Monzio Compagnoni, Alessandro Stanghellini
Anastasiia Koloskova
Rustem Islamov, Aurelien Lucchi
Department of Mathematical Modeling and Machine Learning
Department of Mathematics and Computer Science University of Basel
University of Zurich
Basel, Switzerland {enea wwo. wooopiiccomaagonii, a.stanghellini, rustem. is islamov, aurelien. j
Zurich, Switzerland anastasi ia koloskova@uzh.ch
ABSTRACT
Differential Privacy (DP) is becoming central to large-scale training as privacy regulations tighten. We revisit how DP noise interacts with adaptivity optimiza- tion through the lens of stochastic differential equations, providing the first SDE- based analysis of private optimizers. Focusing on DF-SGD and OP-SignSGD under per-example clipping, we show sharp contrast under fixed hyperparam- eters: DP-SGD converges at a Privacy-Utility Trade-Off of O(1/=²) with speed independent of E, while DP-S1 gnSGD converges at a speed linear in with a ((1/e) trade-off, dominating in high-privacy or large batch noise regimes. By contrast, under optimal learning rates, both methods achieve comparable theo- retical asymptotic performance; however, the optimal learning rate of DP- SGD scales linearly with while that of DP-SignSGI is essentially e-independent. This makes adaptive methods far more practical, as their hyperparameters transfer across privacy levels with little re-tuning. Empirical results confirm our the- ory across training and test metrics, and empirically extend from DP-Si DP-SignSGD gnSGD DP DP-ada..
INTRODUCTION
The rapid deployment of large-scale machine learning systems has intensified the demand for rig- orous privacy guarantees. In sensitive domains such as healthcare or conversational agents, even the disclosure of a single training example can have serious consequences. Legislation and policy initiatives show that AI regulation is tightening rapidly. In the United States, the Executive Order of October 30, 2023 mandates developers of advanced AI systems to share safety test results and techniques such as differential privacy (Biden] 2023). Additionally, the NIST. a U.S. federal agency, released draft guidance (SP 800-226) on privacy guarantees in AI a key dimension in its Al Risk Management Framework (RMF 1.0) (NIST) 20236). : In Europe, the EU A/ Act sets binding obligations for high- risk systems (European Parliament 2023), while ENISA recommends integrating data protection into AI development (European Union Agency for Cybersecurity (ENISA) 2025). In this context, DP (Dwork et al. 2006) emerging as the de l'cto standard for ensuring example-level confidential- ity in stochastic optimization. By injecting carefully calibrated noise into the training process. DP utility.
promotes
NIST 2023a) and included "privacy-enhanced"
optimizers protect individual data points while inevitably trading off some population-
A central open question is how differential privacy noise influences optimization dynamics, and in particular, how it interacts with adaptivity and batch noise. In this work, we revisit this problem through the lens of SDEs, which, over the last decades, have proven to be powerful tool for analyzing optimization algorithms (Li et al. 2017 Mandt et al. 2017 Monzio Compagnoni et al.) 2023). While SDEs have not yet been applied to DP methods, we utilize them here to uncover key and previously overlooked phenomenon: DP noise affects adaptive and non-adaptive methods in structurally distinct ways. We focus on two fundamental DP optimizers: DP-SGD (Abadi et al.) 2016) and DP -e-SiinnGD. The former serves as the baseline for DP optimization; although the latter not widely used in practice, it is substantially simpler to analyze than the popular DP optimizer

---

Published as a conference paper at ICLR 2026
DP-Adam (Gylberth et al. 2017 Zhou al. 2020 Li et al. 2022b McKenna et al. (025). Relying on SignSDD as a proxy for Adam is standard in prior work (Monzio Compagnoni et al. 2025c
Balles & Hennig 2018 Zou et al. 2023 Peng al. 2025 Li al. (025), and this motivates our focus on DP-S 1 g SGGG for the theoretical development. We leave the study of more advanced DP optimizers future work, each would require separate technical treatment. Under standard assumptions and with per-example clipping, our analysis isolates how the privacy budget E, which governs the overall level of privacy, influences the dynamics.
In practice, private training is usually performed across a range of privacy budgets e, and for each value one searches for the best-performing hyperparameters. A change can therefore arise either from this exploratory sweep or from stricter regulatory requirements as discussed above. To capture these two possibilities, we study two complementary protocols. Protocol A (fiee hyperparame- ters): To examine the situation when re-tuning is not feasible, e-g., due to low computational budget, we first fix privacy budget e and find the optimal configuration (1), C, .....) ) via grid search. Then, we analyze how performance changes if training is repeated under different e, without adjusting hy- perparameters, thereby isolating the impact of & on performance. Protocol (best-tuned per s): When re-tuning is allowed, we search the optimal hyperparameters for each E, thereby isolating the intrinsic scaling of the optimal learning rates with respect to E.
Contributions. We make the following contributions:
1. We provide the first SDE-based analysis of differentially private optimizers, using this framework to expose how DP noise interacts with adaptivity and batch noise;
Protocol A: We show that DP- DSD converges at a speed independent e, with a privacy-utility
trade-off that scales as O (1/=), consistent with prior work;
3. Protocol A: We prove a novel result for DP-Si gnSGD: DPSS:: its convergence speed scales linearly in
E, while its privacy-utliity trade-off scales as O (1/a);
4. Protocol A: When batch noise is sufficiently large, uP-SiggnGn always dominates. When
batch noise is sufficiently small, the outcome depends on the privacy budget: for strict privacy (s < {*), DP- Sp--ingDDDD is preferable, while for looser privacy (s s*), DP-SGD has better performance;
5. Protocol B: We theoretically derive that the optimal learning rate of DP-SGD scales as 1y" x s, while that of DP-SignSGD s-independent. This tuning allows the two methods to reach theoretically comparable asymptotic performance, including at very small E;
6. We empirically validate all our theoretical insights real-world tasks, and show that the quali- tative insights extend from training to test loss and from DP -SignSGD to DP addm.
In summary, our results refine the privacy-utility landscape, which, to our knowledge, has not yet yielded definitive answer on which of DP- SGD or DD-DDP-daaaa - SignSGD performs best, and in what regimes. Under Protocol A, adaptivity is preferable in stricter privacy regimes: DP DS-SignSGD converges more slowly but achieves better utility when is small batch noise is large, whereas DP-SGD converges faster but suffers sharper degradation. Under Protocol B, both methods achieve comparable asymptotic performance; however, adaptive methods are far more practical, as their op- timal learning rate is essentially g-independent, allowing it to transfer across privacy levels with litrle or no re-tuning. This matters not only for computational cost but also for privacy, since each hy- perparameter search consumes additional budget (Papernot & Steinke, 2022). In contrast, DP-SGD requires an e-dependent learning rate tuned ad hoc, making it brittle the sweep grid misses the "right" value. Intuitively. adaptive methods inherently adjust to the scale of DP noise, whereas non- adaptive methods require explicit tuning the learning rate to counter the effect of privacy noise.
2 RELATED WORK
SDE approximations. SDEs have long been used to analyze discrete-time optimization algo- rithms (Heimke & Moore 1994 Kushner & Yin 2003). The most popular framework to derive SDEs was rigorously formalized Li Li et al. (2017). Beyond their foundational role, these approx- imations have been applied to practical tasks such learning-rate tuning (Li et al. 2017: 2019) and batch-size selection (Zhao et al. 2022). Other works have focused on deriving convergence
bounds (Monzio Compagnoni et al. 2023 2024 2025c 2026), uncovering scaling laws that govern
2

---

Published as a conference paper at ICLR 2026
optimization dynamics (Jastrzebski et al. 2018 Monzio Compagnoni al., 2025b c) and reveal- ing implicit effects such as Legularization (Smith et al. 2021; Monzio Compagnoni et al. 2023) and preconditionig (Xiao et al. 2025 Marshall et al. (022).. In particular, SDE-based techniques have been used to study a broad class of modern adaptive optimizers, including RMSProp, Adam, AdamW, and SignSGD, as well minimax and distributed variants (Monzio Compagnoni et al.) 2024 2025 c Xiao et al. 2025). Despite this progress, prior work has exclusively focused on non- private optimization. To our knowledge, ours the first to extend the SDE lens to DP optimizers, including explicit convergence rates and stationary distributions as functions of the privacy budget.
Differential privacy in optimization. Differentially private training is most commonly imple- mented via DP-SGD (Abadi et al., (2166., which clips per-example gradients to a fixed norm bound to control sensitivity and injects calibrated Gaussian noise into the averaged update. Advanced accounting methods such as the moments accountant (Abadi et al., 2016) and Rényi differential privacy (Mironov) 2017: Wang al., 2019), combined with privacy amplification by subsam- pling (Balle et al. 2018 2020). allow practitioners to track the cumulative privacy cost tightly over many updates and have made large-scale private training feasible. A central challenge is that clip- ping, while essential for privacy, also alters the optimization dynamics: overly aggressive thresholds bias gradients and can stall convergence (Chen al. 2020), prompting extensive work on how to set or adapt the clipping norm. Approaches include rule-based or data-driven thresholds, such as AdaClis iP (Pichapati et al.] 2019) and quantile-based adaptive clipping (Andrew et al. (02T), as well as recent analyses that characterize precisely how the clipping constant influences conver- gence (Koloskova al. 2023). Together, these contributions have positioned DP-SGD and its vari- ants as the standard backbone for differentially private optimization.
Adaptive DP optimizers. Adaptive methods such as AdaGrad (Duchi et al., 2011 McMahan & Streeter 2010), RMSProp (Tieleman & Hinton 2012). and Adam (Kingma & Ba 2015) generally outperform non-adaptive SGD in non-private training. However, under DP constraints, this perfor- mance gap i) narrows considerably (Zhou et al. 2020 Li et al. 2022a) and ii) essentially vanishes when both optimizers
carefully tuned, as observed for large-scale LLM fine-tuning in Li et al. (20226) App. S). Consistent with non-DP training, non-adaptive methods are sometimes still pre- terred in vision tasks (De et al. 2022). Therefore, which of DP-SGD and DP-Adam is preferable remains an open question. Under assumptions that include bounded/convex domain, bounded gra- dient norm, bounded gradient noise, convexity of the loss, and possibly without performing clipping of the per-sample gradients, several strategies have been theoretically and empirically explored to mitigate the drop in performance of adaptive methods DP. These include bias-corrected DP-Adam variants (Tang& & Lécuyer 2023 Tang et al. 2024). the use of non-sensitive auxiliary data (Asi et al., 2021) 1, and scale-then privatize techniques that exploit adaptivity before noise injection (Li et al, thini-atte-ii
2023 Ganesh et al.) 2025). A recent related work by (Jin & Dai 2025) studies Noisy SignSGD; Conceptually, they investigate how the sign compressor amplifies privacy, and argue that the sign operator itself provides privacy amplification beyond the Gaussian mechanism. Their analysis estab- lishes convergence guarantees in the distributed learning setting while relying on bounded gradient norms and bounded variance assumptions, thereby avoiding the need for clipping and explicitly leaving its study to future work.
We view these contributions as providing valuable theoretical and empirical advances in the design of adaptive private optimizers, clarifying many important aspects of their behavior as well as try- ing restore the aforementioned performance gap. Yet, the fundamental question of which privacy regimes are most favorable to adaptivity remains largely unanswered, and addressing it could explain least one aspect of the nature of this gap. Our work addresses this open question by analyzing why and when adaptivity matters under DP noise, identifying the regimes where adaptive methods dom- inate and where they match non-adaptive ones. Crucially, we incorporate per-example clipping, a central element of DP-SGD, and a heavy-tailed batch noise model that captures unbounded variance.
PRELIMINARIES
General Setup and Noise Assumptions. We model the loss function with a differentiable func- tion f: Rd + R with global minimum f"=00 0: This is not restrictive, one can always consider the suboptimality f fz)) ff and rename it as f. Regarding noise assumptions, recent literature com- monly assumes that the stochastic gradient of the loss function on a minibatch y of size B > 1 can be decomposed as ff, (x) = fz(z) + Z,, where batch noise Z,, is modeled with a Gaussian (Ahn al. Chen
Chen et al. 2014 Mandt al. 2016 2017
Zhu et al. 2019 Jastrzebski al. 2018 Wu
3

---

Published as a conference paper at ICLR 2026
et al. 20200 Xie et al. 2021). often with constant covariance matrix (Li al. 2017 Mertikopoulos & Staudigl 2018 Raginsky& & Bouvrie 2012 Zhu et al. 2019; Mandt et al., 2016: Ahn et al. 2012 Jastrzebski et al. 2018). In this work, we refine the standard noise assumption to distinguish the two regimes induced by per-example clipping in DP training. In this setting, each mini-batch contains mix of clipped and unclipped example-level gradients. For unclipped examples, we follow the usual literature and model the batch-averaged noise as Gaussian, e.g., Zy (o./vB)N Ia). To capture the heavy-tailed gradient noise observed at the per-example level (B = 1), we model (x) J- ty. (0, : Student-t with v degrees of freedom (recovering the Gaussian case as - oo). See Assumption B.2 and Remark [.2 for more details. Finally, we use the following Ff(a)
a,va" The approximation is valid under two assumptions: i) The parameter dimension d is sufficiently large (d = 2(10")) consistent with modern deep learning models that often reach billions of trainable parameters; ii)
approximation, formally derived in Lemma A.2: E
(f(z))
The signal-to-noise ratio satisfies [V(x)) << d: This condition has been thoroughly empirically 20";
studied by Malladi et al. (2022) (Appendix G), who observed that across multiple tasks and archi- tectures the ratio |
never exceeds O112²), well below typical values of d. See Remark A.1
for more details, including experimental validations. We highlight that our experiments confirm that the insights derived from our theoretical results carry over to real-world tasks. Importantly, while our theory is developed for DP-S insSigg ggnSGD, we further validate that the same insights hold empiri- cally for DP-Adam, showing that our insights extend directly to this widely used private optimizer, as well as also transfer from training to test loss. This is testament to both the mildness of the assumptions and the robustness of the analysis.
SDE approximation. The following definition formalizes in which sense a continuous-time time model, such solution to SDE, can accurately describe the dynamics of discrete-time pro- cess, such as an optimizer. Drawn from the field numerical analysis of SDEs (see shtein (1986)) it quantifies the disparity between the discrete and the continuous processes. Simply put, the approximation is meant in a weak sense, meaning in distribution rather than path-wise: We re- quire their expectations to be close over a class of test functions with polynomial growth, meaning that all the moments of the processes become closer at a rate of 17 and thus their distributions.
Definition 3.1 Let O 1 < 1 be the learning rate, T > O and T 1,,1-- We say that continuous
time process X, over [0, an order-o weak approximation of discrete process Tk. if for any polynomial growth function g. 3M > 0, independent of the learning rate n, such that for all ks 0, I.. [g((x,)) - Eg(xx)) S M1f".
While we refer the reader to Appendix B for technical details, we illustrate with basic example. The SGD iterates follow 2 I& 1, fru (Ix)), and, as shown in Li et al. (2017). can be approximated in continuous time by the first-order SDE
dx, = - (f()) + VTV VTt.VV//////... E(X))WW,
(1)
(V)=)) - T is the gradient noise covariance. where Σ(x) 1 E (v)(z) Intuitively, the iterates drift along the gradient while the stochasticity seales with this covariance.
Differential Privacy. Here, we outline the relevant background of foundational prior work in DP optimization. We adopt the (e,8)-DP framework (Dwork et al. 2006)..
Definition 3.2
random mechanism M : R is said to be (e, diifferrrrtl private if for any two adjacent datasets d, d' E 'D (i.e., they differ in I sample) and for any subset of outputs holds that P PMM(d) E S ( S]
R SCR.
In this work, we consider example-level differential privacy applied by central trusted aggregator. We implement this using the sub-sampled Gaussian mechanism (Dwork & Roth 2014; Mironov et al., 2019) to perturb the SGD updates: At each iteration, random mini-bated is drawn, per- example gradients are clipped to a fixed bound to limit sensitivity, and Gaussian noise is added to the averaged clipped gradients. The following definition formalizes these mechanisms and provides the update rules for DP-SCD and DP -S---igp -i---iggnne gnSGD.
4

---

Published as a conference paper at ICLR 2026
Quadratic
IMDB
StackOverflow
0.25
DP-SGD
DP.SGD Di-SigDSGD
20
DP-SGD
DP-SignSGD
DP-SignSCO
DP-Adam
DP-Adam
15
DP-Adam
0.15
Bound DP-SD
oie "
2100x11
dle !
Bound DPSSigSSiD
1.0
ole
30
dee -1
0.8
0.05
0.6
0.4
0.6
0.8
1.0
1.2
0.5
1.0
3.5
2.0
2.5
0.00
0.25
0.50
0.75
1.00
1.25
Figure 1: Empirical validation of the privacy-utility trade-off predicted by Thm. 4.1 and Thm. 4.3
comparing DP-SGD, DP-SignSCD, and DP-Adam: Our focus on verifying the functional dependence of the asymptotic loss levels in terms of Left: On quadratic convex function f(x) x Hr, the observed empirical loss values perfectly match the theoretical predictions 6 Eq. 9). Center and Right: Logistic regressions verflow dataset (right), confirm the same pattern: the utility of DP DP-SGD scales as 2 while the utility of DP-Si gnSGD DP-SignSGD scales linearly as s. Across all settings, we observe that the insights obtained DP-S8 OP-SignSS SignSGD extend to DP-Adam as well as to the test loss (see Figure C.7) For experimental details see Appendix C.2
(Eq.
the IMDB dataset (center) and the
Definition 3.3 For k 2 O, learning rate n) variance hhp. and batches " of size B modeled i.i.d. Let gb be the private gradient, defined as uniform random variables taking values in , ....
1 Σ B
c[Vfi(ax)] + B N10, C(0,C220ppl) cbblla))
(2)
BET
and C[-] be the clipping function C[x] min
c 1
|>||2
The iterates of DP-SGD are defined as
Øk+1 2k - 199k.
(3)
while those of DP-SignSCD are defined as
Ik+1 2g n sign [gx].
(4)
where sign(:] is applied component-wise. Finally, those of DP-Adam are defined in Eg. 196 (Ap-
pendix Ct.
The following theorem from (Abadi et al. 2016) gives the conditions under which DP SGD, and thus also DP-S ap--iinnSGD, is differentially--riveee algorithm.
Theorem 3.1 For g where B is the batch size, n is the number of training points, and number
of iterations T, 3c,, 02 s.1. Ve cI,2T, the noise multiplier opp satisfies aDp C2 logg//))/ OP-SGD is (e, d)-iffferentially private for any ō > 0.
In the following, will often onp multiplier.
TT where Tb 9//log(1/5) to indicate the DP noise
THEORETICAL RESULTS
this section, we investigate how the privacy budget influences convergence speed and shapes the privacy-utility trade-offs in both the loss and the gradient norm. To do so, we leverage SDE models for DP-SGD and DP-SignSGD, which can be found Theorem B.3 and Theorem B.10 respec- tively, and are experimentally validated in Figure C1 In addition, we provide the first stationary distributions for these optimizers, presented in Theorem B.9 and Theorem B.15 in the Appendix. This section is organized follows:
5

---

Published as a conference paper at ICLR 2026
IMDB
DP-SGD
DP-SignSGD
DP-Adam
0.9
3.3.39
0.8
c=2.71
r=2.26
t=1.70
c-0,68
6-0.45
50
100
Epochs
Epachs
Epochs
StackOverflow
DP-SGD
DP-Sign SGD
DP-Adam
4.0
.1111
t-0.85
c=0.6
3.6
g-0.36
3.4
g-0.29
3.2
c-0.24
3.0
20
40
20 40
20
40
Epochs
Epochs
Epochs
Figure 2: Empirical validation of the convergence speeds predicted by Thm. 4.1 and Thm. 4.3 compare SSD. DP- -SigGGDD and DP -Adaa as we train a logistic regression on the IMDB dataset (Top Row) and on the StackOverflow dataset (Bottom Row). In both tasks, we verify that when DP converges, its speed is unaffected by e. As expected, it diverges when E is too small. Regarding DP-SignSCI and DP-Adam, they are faster when is large and never diverge even when this is small. Crucially, Figure C.8 shows that these insights are also verified on the test loss. For experimental details see Appendix
We
1. Protocol A (Section 4.1 Section 411.1
analyzes DP-SGD, yielding bounds for the loss (Thm. 4.1) and the gradient norm (Thm. 4.2 in the μ-PL and L-smooth cases, respectively: We observe that the convergence speed is independent of e, while the privacy-utility trade-off scales as (1)=)). Section 4.1.2 analyzes Do--iignSG and Thm. 4.3 and Thm. 4.4 show a qual- itatively different behavior: Convergence speed scales linearly with E, while the privacy-utility terms scale (1))), making adaptivity preferable if the privacy budget is small enough. Finally, Theorem(4.5 Section|4.1.33 1.53sss that when batch noise is large enough, DP-S gnSGGD always dominates. When batch noise is small, the outcome depends the privacy budget: There exists e* such that for strict privacy (s < E*), DP- OP-SignSDD is preferable, while for looser privacy (s >> e*), DP-SGD better.
2. Protocol B (Section 4.2). In this section, we derive the optimal learning rates of DP-SGD and DP DP-S SignnGG: That of DP- SGD scales linearly in e, while that of SigSGDD is independent of it. Under these parameter choices, they achieve the same asymptotic neighbourhoods.
We empirically validate our theoretical insights on real dataset: Crucially, the same insights de- rived from DP-SignSGD empirically extend to DP-Adam as well as to test metrics: This under- scores the mildness of our assumptions and the depth of our analysis.
Notation. In the following, we use the symbol S to suppress absolute numerical constants (c.g.. R, etc.), and never problem-ddpendent quantities such as d, μ., L, or e: This convention lightens the presentation. We will often use E to highlight the privacy budget in relevant formulas. We use "high privacy" to mean a small privacy budget, i.e., O. Whenever we state an E-scaling such as O(1/e), we refer to the leading e-dependence as 4. O with all other quantities fixed under the protocol being discussed.
4.1 PROTOCOL A: FIXED HYPERPARAMETERS
Following the tuning routine of [I. et al. (202)), we conduct an extensive grid search to select op- timal hyperparameters (1), C, B,...) for a chosen e. To isolate the effect of E, we then keep the optimal hyperparameters fixed as we vary e. In particular, 17 does not depend on e or on other hyper- parameters. This absolute comparison exposes structural differences in how DP noise interacts with adaptive vs non-adaptive updates.
For all our experiments, we use the official GitHub repository http://gthhi..cmm//eiiyylluu// released with the Google paper[ Li 22033

---

Published as a conference paper at ICLR 2026
4.1.1 DP-SGD: THE Privacy-Unlitty Trade- Off Is (1/2)
By construction, mini-batch can contain both clipped and unclipped per-example gradients. For exposition, we first study two idealized regimes: Phase 1, where all examples are clipped, and Phase 2, where none are clipped. This separation is purely pedagogical and allows us to isolate how the privacy budget enters the dynamics. The fully realistic mixed setting, where clipped and unclipped examples coexist within the same batch, handled by Theorem B.8
Theorem 4.1 Let I be p-PL and L-smooth, then during
Phase 1: when each gradient is clipped, the loss satisfies:
"yya" ( (1-c myvi"
Tnd LCo.
E² BdT
+
E f(Xo)
(5)
p
B
Decay
Privacy-tility Trade-off
Phase 2: when gradient is clipped, the loss satisfies:
TndL ( BT
D B
1
[[f(X,] S f(Xo)e +
+(1++1111000 e1-e mt)
C
The decay rates are independent of E: in Phase they depend only on μe. while in Phase I nor- malization spreads the signal over the sphere of radius C. (Vershynin 2018 Ch. 3), giving rate proportional C/(or.vd). In both phases, the privacy-utility trade- off term scales 1/2² in the high-privacy regime.
We now turn analyzing SDE dynamics assuming only L-smoothness of The following theorem presents bound on the expected gradient norm across both phases together: We observe that the expected gradient norm admits the same (1/2) scaling in the high-privacy regime.
Theorem 4.2 Let f be L-smooth, Ki max(1, eyva, and K2 max B
G
c² Bd
Then,
E [IVS(X)))) S Ki
"I ndl (K2 + (C3)2)
where is a random time with uniform distribution on [0, r].
Takeaway. Thm separates two effects: the decay terms, which encode the convergence speed, and the privicy-uiilty terms, which determine the asymptotic neighbourhood under DP. shows that the convergence speed of DP-SGD unaffected by the privacy budget a: Fig. 1gg.2een empirically shows that, whenever DP - SGD does not diverge. its convergence speed is independent of E. Additionally, when E the privacy-utility trade-off scales as (1/e), consistent with Thm 4.2 This insight is validated in Fig. T on quadratic function (left panel), the empirical loss matches the theoretical values from Thm 4.1 and the same scaling reproduced when training classifiers on IMDB and StackOverflow (center and right panels). The behavior persists the test loss (Fig. C.7
4.1.2 DP-SI THE Privacy-Utility Trade-Off IS (2/=)
As for DP-SGD, we isolate the effect of & the dynamics of DP-Si gnSGD and study the loss in each phase separately. Theorem B. coverrs the general case.
Theorem 4.3 Let f be p-PI. and T-smooth. Then, during
Phase 1: when the gradient is clipped, the loss satisfies:
)
e-e St
VTnLdao +b
E[f(X.) & f(x))
BB
Decay
Privacy-Uoliry Trade-off
Phase 2, i.e., when no gradient is clipped, the loss satisfies:
-met
E[f(X.)] & f(X))
C282
n2
T +
1 -
VTLLd
c2 B
(9)
Decay
Privacy-Utiltty Trade-off

---

such that Published as a conference paper at ICLR 2026
DP-SCD Batch
DP-SignSGD
DP Adam
.-ange <" 0.835
0.88
< 1,20
= 0.965
é" 0.940
<' 0.765
2188/11
0.72
0.64
0.75
Figure 3: Logistic regression on IMDB Dataset: From left to right, we decrease the batch noise, i.e., increase the batch size, taking values B {48, 56, 64, 72, 80}: As per Theorem 4.5 the privacy threshold e* that determines when DP-SignSGt is more advantageous than DF-SGD shifts to the left. This confirms that if there more noise due the batch size, less privacy noise needed for DP-SignSGD to be preferable over DP-SGD. For experimental details see Appendix C.4
In the high-privacy regime, the decay rate is proportional to both phases (Eq. g and Eq. 9
unlike DP-SGD, where it is independent of E (Eq. [5]ndd Eq. [ At the same time, the privacy-utility 554.5land
trade-off term both phases scales as (2/s). This might be more favorable than the O (4/=²) scaling of DP-DDPP- SGD in high-privacy regimes.
For L-smooth functions, the following result presents a bound on the expected gradient norm across both phases together. As the bound scales as O (1/s), it suggests that adaptivity might mitigate the effect of large privacy noise on performance. Intuitively, the sign[-] effectively clips the privatized gradient signal, capping the update magnitude and reducing sensitivity noise corruption.
Theorem 4.4 Let f be L-smooth and K3 == max
{Vor
C2 o. va.. Then, B
E [[VT S K3
(((xa) TVT
+ +ndd
(10)
where Ü is a random time with uniform distribution on [0,
Takeaway: Theorem suggests that the privacy noise directly enters the convergence dynamics of DP- DP- gnSGG, making its behavior qualitatively different from DP-SGD: The center column of gp-Sig gnSGD converges faster for larger e. Additionally, it also shows that it never diverges as drastically as DG-SGD for small This is better shown in Figure ! where validate that the asymptotic loss scales with 1. while that DP SGD seales with + Therefore, adaptive methods are preferable in high-privacy settings, and all these insights are verified also for DP-Adam and generalize to the test loss (Figure C.7).
Figure confirmm that DP-Si
4.1.3 WHEN ADAPTIVITY REALLY MATTERS UNDER FIXED HYPERPARAMETERS.
In this subsection, quantify when adaptive method such as DP-SignSGH achieves better utility than DP-SGD. To this end, we compare Privacy-Utilin terms Phase 2 for both methods and derive conditions on the two sources of noise that govern the dynamics: the batch noise size o and the privacy budget
Theorem 4.5 If o 2 B, then DP-SignSeR always achieves a better privacy-utility trade-off
than DP-SGD. If o² < B, there exists critical privacy level e = nz B (B-o2)
C2TB log
OP-SignSGR outperforms DP-SCD in utility whenever E <
This result makes the comparison explicit: i) Under large batch noise (a 2 B). DP-SignSCD achieves better utility than DP-SGD; ii) Under small batch noise (o? B), the best optimizer depends on the privacy budget. For strict privacy (s < E*), De-SignnSGi has better utility, while for looser privacy (e > =*), DP-SGD achieves better overall performance. Thus, e* marks the threshold at which the advantage shifts from adaptive to non-adaptive methods when batch noise is small. By contrast, when batch noise is large, adaptive methods are already known to be more effective (Monzio Compagnoni et al. 2025blc), and the effect of DP noise is only marginal relative to the intrinsic stochasticity of the gradients. We verify this result empirically in Figure As increase the batch size B, decreases, in accordance with our theoretical prediction.
Takeaway:
8

---

Published as a conference paper at ICLR 2026
DP-SCD
Train Loss
Optimal g
DP-SignSGD
DP-Adam
DPD Tuned
0.5
StackOverflow
DP-SGD
Optimal
DP-SignSGD
Troin
4.0
DP-SGD Tuned
Figure 4: Empirical verification of Thm. 4.6 and Thm. 4.7 under Protocol B the IMDB dataset (Top Row) and on the StackOverflow dataset (Bottom Row). We tune (17, C) of each optimizer for each and confirm that: i) all methods achieve comparable performance across privacy budgets; ii) the optimal v of DP- SGD scales linearly with s, while that of adaptive methods is essentially e- independent; dii) failing sweep over the "best" range of learning rates causes DP-SO- SGD severely underperform, whereas adaptive methods are resilient. On the left, DP-SGD degrades sharply for small e. Indeed. the right panels shows that the selected optimal rj flattens out, while the theoretical one would have linearly decayed more: The "best" re-running the sweep with larger grid (DP-SGD Tuned) recovers the scaling law and matches the performance of adaptive methods. For experimental details see Appendix
simply missing from the grid. A posteriori,
Practical Implication. If hyperparameter re-tuning is infeasible and the target regime involves stronger privacy constraints, e.g., lower privacy budget or high stochasticity from small batches, adaptive methods are preferable. Otherwise, DP-SGD the method of choice.
4.2 PROTOCOL B: BEST-TUNED HYPERPARAMETERS
We now mirror standard practice by allowing (1) C) to be tuned over an extensive grid search for each target privacy budget e. In contrast to Protocol A, this leads to derive the theoretical optimal learning rates, which, just as in empirical tuning, are allowed to depend on explicitly.
To select the optimal learning rate 77' for DP-SGD, we minimize the bound in Thm. 4.2 and conse- quently derive the implied optimal privacy-utility trade-off for DP-SGD in the L-smooth case.
GT } then the expected gradient norm
Theorem 4.6 (DP-SGD) Lei nj min
((xo) dLT
((Xo) at.
CV/L/(Xo) en
bound of DP-SGD is ①
as we ignore logarithmic terms and those decaying T.
This result aligns with the best-known privacy-utility trade-off obtained in prior works in these set- tings KKoloskova al. 2023 Bassily al. 2014). Importantly, notice that the optimal learning rate of DP-SGD scales linearly in E and the resulting asymptotic performance scales like 1/E.
To derive the optimal learning rate 17 of DP-Si DD-SignSGD, minimize the bound in Theorem and derive privacy-utility trade-off in the I-smooth case.
Theorem 4.7 (DP-SignSGD) Let
The expected asymptotic gradient norm bound of
4LT
DP-SignSGD is CVVLL((Xo) as we ignore logarithmic terms and those decaying in T.
Importantly, we observe that the asymptotic neighborhood DP DP-SignSGD matches that of DP-SGD, while the optimal learning rate is independent of This suggests that adaptivity automat- ically handles the privacy noise injection: This facilitates the transferability of optimal parameters setups that require higher privacy. In contrast, DP-SGD needs retuning of the hyperparameters.

---

Published as a conference paper at ICLR 2026
Takeaway: Our theory shows that while optimal learning rate scalings differ, the induced neigh- borhoods match. As shown in Fig our experiments verify that: 2) DP-SCD, g and DP-Adam exhibit similar asymptotic performance across a broad range of even small values; vi) the optimal learning rate of DP- is linear in e, while that of adaptive methods is almost independent of it.
Practical implication. Hyperparameter searches are not free under DP: each evaluation consumes a portion of the privacy budget (Papernot & Steinke 2022). making fine learning- rate grids costly. This asymmetrically impacts the two optimizers. For DP-SGD, the optimal step size scales linearly with & (Thm. [466), so the "right" 'y" moves as privacy tightens. If a fixed sweep grid misses a value close to n," the performance of DP-SGD can degrade sharply. This is illustrated in our experiments (Fig. - in the left panel, the performance of DP- SGD drops because the selected "optimal" plateaus instead of decaying linearly as predicted (right panel) - the true 17" was simply absent from the grid. By contrast, the optimal step size of DP DP-SignSGD (and empirically DP DP-Adam) is almost e-invariant (Thm. 4.7). so a single well-chosen v) transfers across privacy levels with little or no re-tuning. This helps explain prior empirical reports that con-adaptive methods deteriorate more severely under stricter privacy (Zhou et al., 2020 Fig. 1). (Li et al. 2023] Fig. 5), (Asi et al.) 202T Fig. 2): plausible cause is that their fixed grids did not track the e-dependent 17" for DP- SGD. However, when both optimizers are carefully tuned, DP-SGD SGD and DP -Adla match in performance in large-scale LLM fine-tuning (Li et al. 2022b App. S).
5 CONCLUSION
We studied how differential privacy noise interacts with adaptive compared to non-adaptive opti- mization through the lens of SDEs: To our knowledge, this is the first SDE-based analysis of DP optimizers. Our results include explicit upper bounds on the expected loss and gradient norm, opti- mal learning rates, well as the first characterization of stationary distributions for DP optimizers.
Under a fixed-tsperparammttr scenario (Protocol A), the analysis reveals a sharp contrast: i) SGD converges at a speed independent of the privacy budget e while incurring O (1/2)) privacy-utility trade-off; ii) DP-SignSCH converges at a speed proportional to & while exhibit- ing O (1/=) privacy-utility trade-off. Additionally, when batch noise is large, adaptive methods dominate in terms of utility, as the effect of DP noise is marginal compared to the intrinsic stochas- ticity of the gradients, confirming known insights from non-private optimization. When batch noise is small, the preferable method depends on the privacy budget: for strict privacy, DP-Si SigGGDD yields better utility, while for looser privacy, DP-SGD achieves better overall performance.
Under a best-tuned scenario (Protocol B), the picture changes: the optimal learning rate of DF-SGD scales linearly with E, whereas the optimal learning rate of DP-SignSCD (and empirically DP-Adam) is approximately e-independent. With this tuning, the induced privacy-utility trade-offs match order and the methods achieve comparable asymptotic performance, including at very small A practical implication is that adaptive methods require less re-tuning if regulations mandate tighter privacy budgets.
theory and experiments agree that
We validated these theoretical insights on both synthetic and real datasets. Importantly, we also demonstrated that the qualitative behavior observed for DP-SignSCB extends empirically to OP-Adam and to test metrics, underscoring the strength and generality of our framework.
Practitioner guidance. Under higher privacy requirements, e.g., regulations mandate a smaller E. if per-s re-tuning of the hyperparameters is impractical because retraining/tuning expected to be costly (Protocol A), prefer an adaptive private optimizer such as DP-S1 gnSGD (or DP-Adam): their performance scales more favorably as e decreases compared to DP-SGD.
When re-tuning is feasible (Protocol B): Both DP-SGD and adaptive methods can reach comparable asymptotic performance. However, hyperparameter searches are not free under DP: each sweep consumes additional privacy budget (Papernot & Steinke, 2022), making fine grids expensive. This creates an asymmetric risk: DS- requires an e-dependent learning rate (17* o E), so if the sweep grid does not track this scaling, its performance can degrade sharply. In contrast, adaptive methods retain a portable, E-independent learning rate, making them more robust and less costly to tune across privacy levels.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Enea Monzio Compagnoni, Rustem Islamov, and Aurelien Lucchi acknowledge the financial support of the Swiss National Foundation, SNF grant No 207392. We acknowledge the use of OpenAl's ChatGPT as a writing assistant help us rephrase and refine parts of the manuscript. All technical content, derivations, and scientific contributions remain the sole responsibility of the authors.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, lyy Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS' 16. ACM, October 2016. doi: 10.114552976749.997333
Sungjin Ahn, Anoop Korattikara, and Max Welling. Bayesian posterior sampling via stochastic gradient fisher scoring. In Proceedings of the 29th International Conference Machine Learning (ICML). pp. 1771-1778. Omnipress, 2012.
Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially pri- vate learning with adaptive clipping. In Advances in Neural Information Processing Systems, volume 34, pp. 17455-17466, 2021.
Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. pp. 383-392. PMLR, 2021.
Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In Advances in Neural Information Processing Systems, volume 31, 2018
Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy profiles and amplification by subsampling. Journal of Privacy and Confidentiality 10(1), 2020.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 404-413. PMLR, 2018.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proceedings of the 55th Annual IEEE Symposium on Foun- dations of Computer Science (FOCS), pp. 464 473. IEEE, 2014. doi: N..1109FFOS.20111
Joseph R. Jr. Biden. Executive order 141 10 safe, secure, and trustworthy development and use of artificial intelligence. Executive Order, October 2023. URL
edu/do suunents/exccutiiv--rleee 141 -w--aaa--seuuue-and-----e---ee-a http:///iSS.tttttttttttta/ttttt.
Accessed: 2025-09-24.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In Pro- ceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1683-1691. PMLR, 2014.
Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In Advances in Neurul Information Processing Systems, volume 33, pp. 13773-13782, 2020.
Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlocking high- accuracy differentially private image classification through scale, 2022. Presented at TPDP (ICML 2022 workshop).
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 07 2011.
11

---

Published as a conference paper at ICLR 2026
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3 40:34141-141177 August 2014. ISSN 1551-305X. doi: 10.1561/0400000042.
URL https://dii.rr// 15/00000001000000000000
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. Serge Vaudenay (ed.), Advances in Cryp- tology- EUROCRYPT 2006, pp. 486-503, Berlin, Heidelberg, 2006. Springer Berlin Heidelherg.
ISBN 978-3-540-34547-33.
European Parliament. Eu ai act: first regulation artificial intelligence. https:/wwww.curopa..ccu aet-act--iine wewnsttreglationoon--ooo on-aarificiaalmmmlliennn 2023. Published: 2023-06-08; Last updated: 2025-02-19; 19; Accessed: 2025-09-24.
European Union Agency for Cybersecurity (ENISA). Engineering data protection in the wake of ai,
April 2025. URL httpp://wwww.miis.........eeeeee -ata -ata--rottcction- in- the take- of-ai. Accessed: 2025-09-24.
Arun Ganesh, Brendan McMahan, and Abhradeep Thakurta. On design principles for private adap- tive optimizers, 2025. PPML 2025.
Roan Gylberth, Risman Adnan, Setiadi Yazid, and T. Basaruddin. Differentially private optimization algorithms for deep neural networks. 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pp. 387-394, 2017. doi: 10.1109/CAASISS2017.. 355063.
F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5(4), December 2015. ISSN 2160-6455. doi: 10.1145/222787. URL https://coi.orr//100 105110/82777/2
Uwe Helmke and John B Moore. Optimization and Dynamical Systems. Springer London, 1st edition, 1994.
Stanistaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Ben- gio, and Amos Storkey, Three factors influencing minima sgd. Artificial Neural Networks and Machine Learning ICANN 2018, volume 11141 of Lecture Notes in Computer Science, pp- 392-402 Springer, 2018
Richeng Jin and Huaiyu Dai. Noisy SIGNSGD more differentially private than you (might) think. In second International Conference on Machine Learning. 2025.
Kaggle. Stack overflow data on kaggle. http://www..aggcccommmmeeeececee low 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Anastasia Koloskova, Hadrien Hendrikx, and Sehastian U. Stich. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In Proceedings the ference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp- 17343-17363, PMLR, 2023.
the 40th International Con-
Harold Kushner and G George Yin. Stochastic approximation and recursive algorithums and appli- cations, volume 35. Springer Science & Business Media, 2003.
Bingrui Li, Wei Huang, Andi Han, Zhanpeng Zhou, Taiji Suzuki, Jun Zhu, and Jianfei Chen. On the optimization and generalization of two-layer transformers with sign gradient descent. In Interna- tional Conference on Learning Representations, 2025. ICLR 2025 Spotlight.
Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2101-2110. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2026
Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. The Journal of Machine Learning Research, 20 (1):1474-1520, 2019.
Tian Li, Manzil Zaheer, Sashank Reddi, and Virginia Smith. Private adaptive optimization with side information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10p.13066 13086-13105. PMLR, 2022a.
Tian Li, Manzil Zaheer, Ken Ziyu Liu, Sashank J. Reddi, H. Brendan McMahan, and Virginia Smith. Differentially private adaptive optimization with delayed preconditioners. In International Conference on Learning Representations, 2023.
Xuechen Li, Florian Tramer, Perey Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference Learning Representations, 2022b.
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling SGD with stochastic differential equations (SDEs). In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguisties. URL http::/aaclanttooo..ooooog/
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDEs and scaling rules for adaptive gradient algorithms. In Advances in Neural Information Processing Systems, 2022.
Stephan Mandt, Matthew Hoffman, and David Blei. A variational analysis of stochastic gradient algorithms. In Proceedings the 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 354-363. PMLR, 2016.
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approxi- mate bayesian inference. Journal of Machine Learning Research, 18(1 (8(134):1-33 2017.
Noah Marshall, Ke Liang Xiao, Atish Agarwala, and Elliot Paquette. To clip or not to clip: the dynamics of SGD with gradient clipping in high- digh-dimessoons. In The Thirteenth International Conference on Learning Representations, 2025.
Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette- Choo, Badih Ghazi, George Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, and Chiyuan Zhang. Scaling laws for differentially private language models. In Proceedings of the 42nd In- ternational Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 43375-43399 PMLR, 2025.
H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex opti- mization. In Proceedings of the 23rd Annual Conference on Learning Theory (COLT 2010), pp. 244-256, 2010.
Panayotis Mertikopoulos and Mathias Staudigl. On the convergence of gradient-like flows with noisy gradient input. SIAM Journal on Optimization, 28(1):163-197 2018.
GN Mil'sh Weak approximation of solutions of systems of stochastic differential equations. Theory of Probability & Its Applications, 30(4):750-766, 1986.
Ilya Mironov. Rényi differential privacy. In 20/7 IEEE 30th Computer Security Foundations Sym- posium (CSF), pp. 263-275. IEFF, 2017.
Ilya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism, 2019. URI htttpss/aaaaiiooggaaa////... 10530
13

---

Published as a conference paper at ICLR 2026
Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, and Aurelien Lucchi. An SDE for modeling SAM: Theory and insights. In Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 25209-25253. PMLR, 2023.
Enea Monzio Compagnoni, Antonio Orvieto, Hans Kersting, Frank Norbert Proske, and Aurelien Lucchi. SDEs for minimax optimization. Proceedings oh 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. 4834 4842. PMLR, 2024.
Enea Monzio Compagnoni, Rustem Islamov, Antonio Orvieto, and Eduard Gorbunov. On the in- teraction of noise, compression, and adaptivity under (o.441)-mmothness An SDE approach. In High-dimensional Learning Dynamics 2025, 2025a.
Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, and Aurelien Luc- chi. Unbiased and sign compression distributed learning: Comparing noise resilience via SDEs. In The 28th International Conference Artificial Intelligence and Statistics, 2025b.
Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of SDEs: Theoretical insights on the role of noise. In The Thirteenth International Conference on Learning Representations, 2025c.
Enea Monzio Compagnoni, Rustem Islamov, Frank Norbert Proske, Aurelien Lucchi, Antonio Orvi- eto, and Eduard Gorbunov. On the interaction of batch noise, adaptivity, and compression, under (1..1) -m..o)t)mmssssssss An sde approach. Submitted to ICLR 2026, 2026. OpenReview submission 4H8xZA4zuj.
NIST. Nist offers draft guidance evaluating a privacy protection technique for the ai era. htips: draft- saaaft--uidaccee-- gwwdance-evallneeeeeee evaluating privacy-
//www.nist.g www.wnist.oo/nnww...oo www.ggwwww/eeennneeee wwwpeeee//eew//////2222///
2023a. Accessed: 2025-09-24.
NIST. Artificial Intelligence risk management framework (AI RMF 1.0). Technical Report 1000,, National Institute of Standards and Technology, January 2023b. URL https:
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with Rényi differential privacy. In International Conference on Learning Representations, 2022.
Courtney Paquette, Kiwon Lee, Fabian Pedregosa, and Elliot Paquette. SGD the large: Average- analysis, asymptotics, and stepsize criticality. In Conference Learning Theory, pp. 3548- 3626. PMLR, 2021.
Hanyang Peng, Shuang Qin, Yue Yu, Fangqing Jiang, Hui Wang, and Zhouchen Lin. Simple con- vergence proof of adam from a sign-like descent perspective, 2025.
Venkatadhecraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank Reddi, and Sanjiv Kumar. AdaCliP: Adaptive clipping for private SGD, 2019.
Maxim Raginsky and Jake Bouvrie. Continuous-tinn time stochastic mirror descent on a network: Vari- ance reduction, consensus, convergence. In 2012 IEEE 51 Annual Conference on Decision and Control (CDC), pp. 6793-6800. IEEE, 2012.
Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In International Conference Learning Represen- tations, 2021.
Qiaoyue Tang and Mathias Lécuyer. DP-Adam: Correcting DP bias in adam's 's second moment estimation. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
Qiaoyue Tang, Frederick Shpilevskiy, and Mathias Lécuyer. DP-AdamBC: Your DP-Adam is tually DP-SGD (unless you apply bias correction). Proceedings of the AAAI Conference on Artificial Intelligence, volume 38-14, pp. 15577-15283,, 2024.
14

---

Published as a conference paper at ICLR 2026
TensorFlow Federated. Tensorflow federated stack overflow dataset. httpss/wwww.eensorrooo.oo//// http://ww..ooootooooo//s////s//e
2022.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running
average of its recent magnitude., 2012. URL https//wwww...ooooontccddd/////d 1/slides/
decturessiides peeiuee.p
Roman Vershynin. High-dimensional probability: an introduction with applications in data science. Cambridge series statistical and probabilistic mathematics 47. Cambridge University Press, Cambridge, United Kingdom ;. 2018 8. ISBN 978 108415194.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential pri- vacy and analytical moments accountant. Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Second International Conference Artificial Intelligence and Statis- tics, volume 89 of Proceedings of Machine Learning Research, pp. 1226-1235. PMLR, 16-18 Apr 2019.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes SGD. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10367-10376. PMLR, 2020.
Ke Liang Xiao, Noah Marshall, Atish Agarwala, and Elliot Paquette. Exact risk curves of signSGD in high-dimensions: quantifying preconditioning and coiis--ompressionn effects. In Forty-second International Conference Machine Learning, 2025.
Zeke Xie, Li Yuan, Zhanxing Zhu, and Masashi Sugiyama. Positive-negative momentum: Ma- nipulating stochastic gradient noise to improve generalization. In Proceedings of the 38th In- ternational Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11448-11458 PMLR, 18-24 Jul 2021.
Jim Zhao, Aurelien Lucchi, Frank Norbert Proske, Antonio Orvieto, and Hans Kersting, Batch size selection by stochastic optimal control. Has Trained Yer? NeurlPS 2022 Workshop, 2022.
Yingxue Zhou, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Arindam Banerjee. Pri- vate stochastic non-convex optimization: Adaptive algorithms and tighter generalization bounds, 2020.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochas- tic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. In International Conference on Learning Representations, 2023.
15

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
Technical Results
16
Theoretical Framework
18
B DP DP-SGD
20
B.2
SignSGD
27
C Experimental Details and Additional Results
35
DP-SGD and DP SP-Sig SDE Validation (Figure C. ................................
37
Asymptotic Loss Bound (Figures C.2 and C.7
38
Convergence Speed Analysis (Figure) and
................................
39
When Adaptivity Really Matters (Figure) and Figure
40
C.5
Best-Tuned Hyperparameters (Figures
and C.5
40
C.6
Stationary Distributions
42
Additional Results Test Loss
42
D I
Limitations
43
A TECHNICAL RESULTS
In this section, we introduce some technical results used in the derivation of the SDEs.
Lemma A.1 Let ~ NN,, Iala) and fix a tolerance e 0. If that I
< e, for + oc, we have
20"
de + e0 (2tT)-
|
Proof:
Let us remember that X NN Ial,,
X
r +1 -1 5
k:
+
11m113 202
E
F1
(11)
|
(202) 6/2 ($ +1)
2
2
where Fa 2) is Kummer's confluent hypergeometric function. We know that
( k=1
V2
2
lim d
+
(12)
r ( +1)
d3/2
Let
1el 20
If d > by expanding the series, we have
F
1 d
+1;-z)
=1 (-2)
+0
2
51-6
(13)
2'
b(4)m
d+2 (
(1xT1a 1x
X =V1 + c0 (11)).
Combining everything together, we obtain E
Lemma A.2 Let K(v) = - T(:)
and X ~ ll(u,, 11), forv 2 1. Fix a tolerance > 0: If
20 [d=22 < <s for - 0o, we have that x
||
= K(v) 10t + +c0 237
T
16

---

Published as a conference paper at ICLR 2026
Proof: One write X 4 <2 where Z ~ ~N(0 Ia) and S ~ x² independent. Define
/5/2"
then, conditioning on S and applying Lemma A.1 we have
8/v
X
V d + c0 (d-312)
(14)
Remembering that E[(S]] v2 [ (111
we have
r(3)
[Var
TH dr + co (d-112))
E
= Es
(15)
X 12
= -VV2o VV Eslv S]
P
t0 (d=3/2
(16)
-K(N 1u + co
1
(17)
d3/2
Remark A.1 As discussed in the main paper, our analysis is based on the following two assump-
tions:
The number of trainable parameters large, specifically d 2((1*))
[V 20
ii) The signal-to-noise ratio satisfies
d.
First, they ensure that the approximation of the confluent hypergeometric function in Equation 13 is highly accurate. Second, neither condition is restrictive for modern deep learning models. The dimensionality assumption trivially satisfied by contemporary architectures, which routinely have millions of parameters. Regarding the second assumption, Malladi et al.) (2022) empirically measured the signal-to-noie ratio [(())1 across a wide range of large-scale architectures and 2n
datasets, and consistently found values of at most O(10²). Thus, the regime in which our approx- imation is valid closely matches the regime observed in practice. This is further supported by experimental results, which confirm our theoretical predictions across all models and tasks consid- ered in this paper. In Figure A.T numerically evaluate the confluent hypergeometric function for varying values of d, and show that, for sufficiently large parameter counts, the approximation remains tight throughout the realistic signal-to-nold ratio range reported (Malladi et al.) 2022).
Confluent Hypergeometric Function
1.0
0.0
d=
d-10P
d=107
W.0.4
d=
z= 01103)
0.
0.0
10:
10a
10a
10'
Figure A.I: Numerical validation of the approximation used in Equation For several values of we plot the confluent hypergeometric function as a function of the signal-to-noise ratio In the realistic range observed in (Malladi et al. 2022), approximating this function by 1 is extremely accurate.
13
17

---

Published as a conference paper at ICLR 2026
B THEORETICAL FRAMEWORK
In this section, we introduce the theoretical framework, assumptions, and notations used to formally derive the SDE models used in this paper. We briefty recall the definition of L-smooth and p-PL. functions, Then we introduce the set of functions of polynomial growth G.
Definition B.1 A function f Rd Lipschitz continuous, namely
+ IR is L-smooth if it is differentiable and its gradient is L-
[V)(()
v(will2 S 1/ - Vx, y-
(18)
Definition B.2 A function f Rd I admitting a global minimum x² satisfies the Polyak- Lojasiewica inequality if. for some > and for all x E R it holds
1 2p
f(x) f -
[V(())
(19)
In this case, say that the function f is μ--11
Definition B.3 Let G denote the set of continuous functions R2 + R of most polynomial
growth, namely such that there exist positive integers kı, 22 > 0 such that [g(x)] < ki (1 + + 1x1 for all x E R.
To simplify the notation, we will write
b(æ + 77) = b0(x) + nb1(2)) + 0(7))
whenever there exists g G, independent of n, such that
|b(r + n) - bo(x) - nbb())) S g(x))}²
We now introduce the definition of weak approximation, which formalizes in which sense the solu- tion to an SDE, a continuous-time random process, models a discrete-time optimizer.
Definition B.4 Let < <n < 1, 0 and T -,,1-- We say that a continuous time process X₃ over
[0, r). is an order a weak approximation of a discrete process Tho for k = O,..., N, if for every g E G, there exists M, independent of 11, such that for all k 0, ,.....,T
[Eg(XE>)) Eg(xx)] S Mn
This framework focuses approximation in weak sense, meaning in distribution rather than path- wise. Since G contains all polynomials, all the moments of both processes become closer at a rate of 11th and thus their distributions. Thus, while the processes exhibit similar average behavior, their sample paths may differ significantly, justifying the term weak approximation.
Remark B.1 (Validity of the SDE approximation) To guarantee that the SDE model is a first- order weak approximation of the optimizer dynamics in the sense of Definition 3.1 one shows that the first two moments of the one-step increments of the optimizer and of the SDE match up to O while all higher-order terms in the Taylor expansion are collected in an O remainder (see Ap- pendix B This implies that the discrepancy between the two processes scales as O(π) for any test function of polynomial growth and any finite time horizon. The neglected O(n²) terms could, in principle, be retained by deriving higher-order SDEs. However, to the best of our knowledge, such second-order models have been derived ([.i el al. 2017], but have not yet led to additional practical insight in the analysis of optimisation algorithms until recently Monzio Compagnoni al. (2025a). Finally, Figure [.I empirically compares the discrete algorithmss with their SDE counterparts on quadratic and quartic objectives, confirming that for the step sizes used in experiments, the first-order SDEs closely track the discrete dynamics.
The key ingredient for deriving the SDE is given by the following result (see Theorem 1, (Li et al. 2.17h, which provides sufficient conditions to get weak approximation in terms of the single step increments of both X, and Fg. Before stating the theorem, we list the regularity assumption under which we working.
18

---

Published as a conference paper at ICLR 2026
Assumption B.1 Assume that the following conditions are satisfied:
Ji E (R R);
f, ft and its partial derivatives up to order 7 belong to G;
VJ. V. satisfy the following Lipschitz condition: there exists I. > 0 such that
- ( + Σ V - Ll|u ul12i
VJ, Ja satisfy the following growth condition: there exists M >> such that
[Vf(2)]2 + Σ DIVA((=))2 S M(1 (|
2m1
Assumption B.2 Assume the per-example (n has size 1) stochastic gradient admits the decomposi- tion (z)) V f(x) (z),, with centered, approximately isotropic noise (z). In Phase (clipping regime), we model the per-example noise as heavy-tailed, e.g., Zz((x) ~ Gy ty (0, Ia), multivariate Student-t with degrees of freedom (for S 2 the variance is unbounded). In Phase 2 (non-clipping regime), we consider the mini-batch average (x)
recover the Gaussian case; for
Eeen (x) and approximate its noise by Z,(x) = Vf) Vf(x) VB" (00 La), reffecting the averaging effect of i..dd per-sample gradients.
Remark B.2 The distinction between the two phases stems from how per-example clipping alters the effective noise seen by the optimizer. In Phase 2, clipping is inactive and the mini-batch average of i..dd. per-sample gradients admits the usual Gaussian approximation with variance / B. In Phase I. clipping is applied before the average and the update is driven by normalized/runcated per-example gradients. While averaging still reduces the variance of the average of clipped direc- tions by facter 1/ B, the distributional shape is governed by the per-example gradient law and can deviate substantially from Gaussian. therefore model the per-example noise with a multi- variate Student-t, which captures heavy tails and yields tractable expressions, while recovering the Gaussian case as V + CO.
Lemma B.3 Let < 1) < 1. Consider a stochastic process Xe, t 2 0 satisfying the SDE
dx. = b(X:)dt + (X:)dWW
Xo=
(20)
where b, logether with their derivatives belong to G. Define the one-step difference A X,, and indicate the i-th component of A with A.. Then we have
1. EA,- b.n + [ b,0,by 17 + O(1")
2. EA,A, = [beb, + [o! 12 + 0(T²)
= Li.j=I..... d:
3. EE A1,
Vs Ns23 3, is = l....,d.
All functions above are evaluated at x.
Theorem B.A Let 0 < 0 and set T = [T/n]- Let Assumption B.I hold and let X, be stochastic process as Lemma B.3 Define A E1 I to be the increment of the discrete-time algorithm, and indicate the i-th component of A with Dj. If in addition there exist K1, K2, K3, K₁
so that
1. EA. E S K2
-i= 1..,...,dd;
2. EA,A, - E < K2(x))
Vi,j = 1,...,d;
3. EII; A., - EII Ay/
<Ka()))
Vs 2 3, Vi, = 1,...,d; d;
19

---

Published as a conference paper at ICLR 2026
4. EIT Ku(z)n² = wy.=1...,,d
Then, there exists constant C so that for all 0, ..... N we have
(x(X - Eg(xx)) S Cn.
(21)
We say Eq. 20 is an order 1 weak approximation of the update step of Tk-
B. DP-SGD
This subsection provides the formal derivation of the SDE model for DP-SGD (Theorem B.5 to- gether with the formal versions of the loss and gradient-norm bounds used in the main paper (The- orems and B.7 and the quadratic stationary distribution (Theorem B.9
Idealized clipping regimes vs. the mixed regime. Following the exposition in Section 4.1.]
first analyze two idcalized per-iteration regimes to make the dependence on the privacy budget explicit: (i) fully-clipped regime ("Phase 1*), where all sampled per-example gradients exceed the clipping threshold and the update is dominated by normalized directions with magnitude on the order C: and (ii) unclipped regime ("Phase 2"), where clipping inactive and the update reduces to standard noisy gradient step. These two regimes used only to isolate how clipping changes the drift and diffusion terms. The fully realistic mixed case, where clipped and unclipped samples coexist within each mini-batee mini-batch, is handled separately in Section B.1.1 (Therem
B.8
Theorem B.5 Let 0 < 1, and set T [T/n] and K(v) = /2 (1
Let x TRd and Assumption
r(
denote sequence of SGD iterations defined in Eq. 3 Assume Assumption
B.
B.2 Let X, be the solution of the following SDEs with initial condition X 20.
Phase 1:
CK(v)
dX,
dt VT E(X))aW
(22)
a,Vd
where E(x)= 6²
Bg
Vc(())V(<(>) 11V/(())18
K()) 73d
Vf(=)V(()TT) + B2
C Id.
Phase 2:
dX, vf(,))tt + +VVV E(X.)dW.
(23)
where E(x) 10
c pp
la-
B²
Then, Eq. 22 and Eq. 23 are an order approximation of the discrete update of Phase and Phase of DP SGD, respectively.
B Proof: Phase I: Let Zpp N N0, C' Ia) be the differentially--rivaal noise injected via
the Gaussian mechanism and denote with á x1 r the one-step increment for Phase 1. In the fully-clipped regime, each sampled per-example gradient is clipped to norm C, hence
C B
(x(
Ā=
Zpp
(24)
fe(x))]
SET
Applying Lemma A.2 with tolerance
=1 and by linearity of expectation have
ff(())
CK(v)
E A = CE
= -17 vv Vf(z) +0))))(()))
(25)
MA and (x) = Eger Ye(x). Since the samples in 7 are i.i.d. conditional v, have Cov(u,) 1/ CYy(YY), and u, is independent of
To compute the covariance, define Yx(x)
20

---

Published as a conference paper at ICLR 2026
ZDP. Therefore
Cov(A) 11² (I)) + Cov(ZDp)) + +(11²))
(26)
n²² C Cov(Ye(>)) Co la
+ 0(n')
(27)
B²
C Eg 12²
V[fe()
(²K(V)²
Coope B²
V((=)
la
+ O11)).
[
Bogd
(28)
Define now
CK(v) v
b(z)
(29)
vd
C²
Ve
K(v)
v(())((a))
Coope
[xx) = B
E
+
la.
(30)
V (e()
o2d
1
Then, from Lem B.3 and Thm. B.4 the claim follows.
Phase 2: Following the same steps as above, one obtains:
E [ = - [V)(z) Zpp] = -
(31)
and
Cov(A) = 2 (v) + Zpr V[(=)
ZDP -DP V\(x))
(32)
Vf v/(=)) + 172 2 C'OOOp
= B (z)
la
(33)
1
C²
17
la.
(34)
B
B²
Define
b(x)
Vf(();
(35)
02
Coo
E(x)
Lai
(36)
B²
Finally, from Lem B.3 and Thm. B.4 the claim follows.
Theorem B.6 Let f be L-smooth and p-PL Then, for t [0, we have that
Phase 1, i.e., when the gradient is clipped, the loss satisfies:
T TndiLCos BIS(X.) & f(Xo)e "oaa' + +(1- -ivatt
E2 BdT
22
(37)
1
B2
62
Phase i.e., when the gradient not clipped, the loss satisfies:
2 f(Xo)e at + (1- TrdL (=2o=²
C $2
1
(38)
BT
B
z
Proof:
Phase
By construction we have
C² Coop
Tr (E(x) < B
+ B2
(39)
Since f is p-PL and L-smooth it follows that 2p < [z)]] and f(x N Llg. Hence, by applying the Itō formula we have
df(x,) CK(v)
V, |V
7
Tr (Xx)((XXXXXx)))) di + O(
(40)
2
CK(v)
(Xt + ndL
2
C²
< 24
+
dt + (Noise).
(41)
V,
2
Bd
B
21

---

Published as a conference paper at ICLR 2026
Therefore,
2p Kioic, "yva"
2p aive"
nd Lo, 4uCK(p)
(C²
CoOBP
[[f(x)) & f(Xo)e
1--6
+
(42)
Bd
B
Let us now remind that
9V T log(1/8)
DDP
(43)
8
then
-2p ayvil
t
1 (1-e-
ayvi"
gd LCog Apk(v)
1
Tq² log(1/8)
B[f(X)) & f(Xo)e
(44)
Bd
B
Phase 2: Similarly to Phase I, we have
Coopp
Tr (E(x)) d
(45)
B²
Again using the fact that f 4-PL and L-smooth and by applying the Itō formula, one obtains
ndL
o²
C
if(),) - - [V
+
O(Noise),
(46)
B
B
from which we have
ndL
o²
Caope
B[S(Xi) S ((Xo)e 2pt + -1- =ue)
+
(47)
Ap
B
B2
Hence, by expanding ODP
ndL
2
log(1/))
B[f(X)) S 5(xoee 2pt (1
2pt
(48)
4p
B
Finally, let Φ = log((1)) and suppress all problem-indeependen constants, such 2, n, K(v), to obtain the claim.
Theorem B.7 Let f be L-smooth and define
K = max 1. Vd
C² of
and
K₂ = max
(49)
CK(v)
Bd B
then
[(Xo)
IdI
C2 q (2(4))
log 1/0)
E [X))( $ Ki
K₂ +
B
(50)
n/T
22
where i & T
Proof:
Since f is L-smooth and by applying the Itô formula to Phase we have:
df(X,) < CK(u)
VV
+
" 2
Tr (LE((x,) di + O(Noise)
(51)
V.Vd
CK(v)
ndL
C² Coope dt + (Noise)
|VF(X)))
(52)
vd
2
Bd
Similarly, in Phase 2 we obtain
df(x,) < (1
! Tr (LE(X)) dt ONNois)
(53)
ndL
Q2
C.obp
+
dt + O(Nois)).
(54)
2
B
B²
22

---

Published as a conference paper at ICLR 2026
Let K1 and K2 in Eq. 49
Then, by integrating and taking the expectation, have
E !
(((X6) f(x+)+ + Tddl.
Coor
dt = Ki
K2
(55)
2
B²
([(x))=((Xx) + ndL
C
E
c2 S Ki
K2 +
(56)
2
B
=E [IV Ki (2x - [(x+)) + +fddTTK2 ndLTCC 1og(1/6)
1
2T/T
B²
E2
where the last step follows from the Law of the Unconscious Statistician and ~ Unif(O, r). Finally, by suppressing problem-independenn constants, 2, ग., we obtain the claim.
B.1.1 MIXED-PHASE GRADIENT BOUND
In this section, we extend the two-phase SDE derivation single mixed setting. This is important because, at any point during training, some per-example gradients may exceed the clipping threshold while others remain below it. We show that in this scenario the same bound holds in Theorem B.7 where it was previously derived under a worst-case approach.
Theorem B.8 Let f Fd R. be L-smooth. Then, we can write the SDE of DP-SGDO as
dX, bmix dt + vn (X,)
(57)
where the drift and covariance satisfy, for all x,
(V f(x)), (mix),mmix((x)) S - - [VF(2))
1
(58)
Tr (xxx(x) S K2 + C²OOP"
(59)
B
where K1, K2 are defined in Equation 49 Therefore, for i ~ Unif(0,
f((X))
ndL
CaoBr
E [(x)(1 < K1
K2
+
(60)
T/T
2
B
i.e., the same L-smooth convergence bound as in Theorem B.7 holds for any mixture of clipped and unclipped samples in each mini-batch.
Proof: We proceed in three steps: i) drift under mixed clipping, ii) covariance under mixed clip- ping using an explicit decomposition of Gş into clipped and unclipped parts, and iii) ItO's formula and the final bound.
Step 1: Drift of the mixed batch. The DP-SGD update be written
Øk+1 = 2x
G+ + B
1
ZDP
where G = Σ c(V
(61)
B SE
Let
S1,A = (i mk [V)((xx)) 2 c),
S2,6 = (i (m [V[((=)] <C,,
(62)
Bg
:= |5
Pk =
B [0, 1).
(63)
Intuitively, py represents the probability of sample being in Phase 1. Define the per-sample contri- butions
:= C(f((()))))), ie SL.k
X; Vf((x)),
i S2,6
(64)
and the corresponding batch averages
9k
(1)
Σ Y₁
9k
2
Σ X,,
(65)
B
B
VEST.
VES2k
23

---

Published as a conference paper at ICLR 2026
so that
G3 =
+
(66)
In the same way in the proof of Theorem
B.5 we have that
CK(2)
B|y] (a(s))
01 o,Vd
(67)
and
E (x(x).
(68)
Conditioned on the sets S1,k: S2.k. the Y, are i..d. over S1,x and the X, are iid. over S2 so
Bluk
(1)
SI,A]
Σ
B[Y]
BB EMY] PEat (s(w)),
(69)
B
B
VEST.
1
B
B
Bigk
S2,11 =
Σ
E|X
E = (1 Pk) Vf)())
(70)
B
B
iE.2.
Thus
EEGK& S1,k, S2,1) = paa V[((+) + (1 (((A))
(71)
Recall the definition of K₁ from Equation 49
ovvd CK(v) f
K1 max
(72)
Then, holds ai = [C(())/O,VV)) 2 1/K1 and 02 = 2 1/Ki. Since Umix (pR) is convex combination of 01 and d2
1
(pix Pkαl + (1 pk)02 min 2
Vpk [0.
(73)
Ki
Therefore, the drift in the SDE limit satisfies
Dixix (x) = amix (mmxx(((()))) (f(x²) Vf()),
(mi(
S
Ki |V
(74)
Covariance of the mixed batch. We now compute the gradient noise covariance and show it is convex combination of the pure-phase covariances, which already derived in Theorem B.5 Define the centered contributions
Step 2:
U = -
iE S1,k.
-
E[Xy],
j E S2
(75)
Then
9k
Elght (1)
|1111
Σ Us
9k
(2)
Elg
12,4
1
Σ
Vj.
(76)
B
B
SES
JES2,
Hence
G₁ - G S1.2. S2,11
Σ
U
Σ
VjJ
(77)
B
B
VESLK
JES2.x
Let
X ssingle (xx) = Cov = Cov(U.),
I (Λx) = Cov(X,) Cov(V))
(78)
be the covariances of single data-point. Conditioned on the sets S1,k, S2.4s the random vectors (U: : ) and (V; j } } are independent and zero-mean. Thus
Cov(G& S1,4. S2.4) = Cov
U.
Σ
Vs
S1,6, S2.k
(79)
B
B
iESL
JES2.
Cov
Σ U:)
+
Cov
Σ v1).
(80)
B
B
iES1.
JESo,u
24

---

Published as a conference paper at ICLR 2026
where cross terms vanish by independence. Using i.i.d. within each group, we have
Cov Σ
U.
- BB 21 Egingo (rx),
Cov Σ
v3) = (B Ba) Σg Esingle (Ix);
(81)
1ES
JES2,
therefore
BR singto Ba 41
B Bk yoinnle (xk). (82)
Egrad (Ik; S1,ks S2,k) = Cov(Gr Ik. S1,k, k2,k) =
(IK) +
222
B
Since Ps = B2/ B and - Pk = (B B&)/B, B, we obtain
Egrad (Irs Si,k, S2,a)
E1
Epingle (axx) +
1-Ph Egingle (FR).
Dn2
(83)
B
B
In the pure-phase SDEs of Theorem B.5
the batch-level (gradient and DP) covariances are given by
singge
(a)
C
Id
Σz(x)
1 siinggl (x)
(x)
C C
la-
(84)
B
B²
B
Ba
From equation 83
the gradient part of the mixed -hase covariance is
CaoBr
Ca
Egraa (1R;; A1 S2,)
Ph:
(xE))
B²
Ia
+ pp)) [2(2k)
B²
Ta
(85)
c
Adding the DP noise term mixed batch is
Ia back in, the full covariance of the DP-SGD increment in the B²
Cº
Emix S1,6: S2.1) = Egrad (1 S1,k S2,x) +
Ia Ps (xx) + (1 - pk) (x(x)
(86)
(87)
Thus, at the SDE level, the mixed-phase covariance is exactly convex combination of the pure- phase covariances Σj and Eg. From Theorem B.6 we have the trace bounds
C
CoOB
TTE
+
(88)
B²
0
T2
(89)
B
B²
Let K2 as in Equation 49 we can write, for E [1,2],
C
T+ < d K2
(90)
B²
Using equation 87 for any pa E [0, 1].
CooPP
Tr (xx) = Pk Tr (xx) + (1-PH) TI 12(x)))) S
K₂
(91)
B
Hence, the mixed- phase covariance satisfies exactly the same worst-case trace bound as the pure- phase covariances.
Step Itô bound and convergence. Finally, we can rewrite the SDE of DP-SGD as follows:
dX, bmix (X,) dt + vn Emiix EX, 1/2
(92)
where, for all x,
(2)( (2)xx()))) S [+)(2)1
(93)
K
Tr Emix(() S K2 C²aBp
(94)
B
Since ʃ is L-smooth, (²)(x) 3 Lla. By Itô's formula.
df(X,) = (Vf(())) D (xixx(()) (X,)) dt + " Tr (X, (Xx)) dt + (Noise).
(95)
2
25

---

Published as a conference paper at ICLR 2026
Using the drift and covariance bounds,
[Vf(x)) + ndLL
(
C'o
df())) - K
K2 +
dt + O(oosee)
(96)
2
B²
Integrating from 0 to r : r/l',
1
[ |XV)()))
ndL
C
- K1
K2 +
-+ ((Noie)).
(97)
2
B²
Rearranging,
K1 / IV)(())) S f(Xo) - f(x+) + ndL (
Cºo
K2 +
T (NNise).
(98)
B²
Taking expectations,
K l [V = Bif( - s(x+)| ndL
C
K2 +
T.
(99)
2
B
Let i Unif r). Then, by the Law of the Unconscious Statistician,
3[ - h |VS(X,) < K1 [(Xx))
ndL
Cºopp B2
K₂
(100)
Since nT,, this is exactly the gradient-norm bound as in Theorem B.7 with the same constants K1, K2, now rigorously shown to hold under arbitrary mixtures of clipped and unclipped samples at each iteration.
We now derive the stationary distribution of DP-SGD at convergence: We empirically validate this result in Figure C.6
Theorem B.9 Let f(x) 2=2 convergence of DP-SGD is
Hæ where H diag(A1..... da). The stationary distribution at TH
Tn e log(1/)))) (1 (1-8-281)H-1)
(ovX,), = X Hr
+
(101)
22
BT
B²
Proof:
Since H is diagonal, we isolate each component. Furthermore, since f(.)) is quadratic we can rewrite the SDE as:
02
Co
dx Xi, vTV
dW...
(102)
Ba
We have immediately that
E[X,,,1 = Xo,se del
(103)
Applying the Itô isometry, we obtain:
3[X
02
Cho
02
C² dW,
n/E
dW,
e -i((tt)))
B
B2
B
B²
a2
Coo
-22,(1-0) ds
1
B²
o²
C²
(logT//((1/))
-2dit)
+
2
B
B²g²
Tn
6²002
0²
log 1/8)
(1 (1-e -2ht)).
BT
B2
26

---

Published as a conference paper at ICLR 2026
B.2 DP-SIGNSCD
This subsection provides the formal derivation of the SDE model for DP-SignSGD (Theo- rem B. together with the formal loss and gradient-norm bounds used in the main paper (Theo- rems B.12 and B.13) and the quadratic stationary distribution (Theorem B.15).
Idealized clipping regimes vs. the mixed regime. As the main paper (Section 1.2.2 we first analyze two idealized regimes: (i) fully-clipped regime ("Phase I"), where sampled per-example gradients are clipped and the sign update acts on privatized, normalized direction; and (ii) un- clipped regime ("Phase 2"), where clipping is inactive and the privatized gradient is approximately Gaussian after averaging. These regimes are used only to isolate how & enters the drift and diffusion. The realistic mixed regime (a mixture of clipped and unclipped samples within each minibatch) is treated in the mixed-phase analysis below (Theorem B.14).
V V T($)
Theorem B.10 Let 0 < 1, and set T [T/n] and K(v)
Let
E E Rd denote sequence of DP-Si gnSGD iterations defined in Eq. 4 Assume Assumption B.1
and Assumption B.2 Let X, be the solution of the following SDEs with initial condition Xo zn:
Phase 1:
(x.))
dx. BBy Erf "x
dt VTV E(X,)dW,
(104)
where, a natihbbtch 7 of size B, we define the averaged normalized direction
1 -11 Σ
(fe(z)
(x(x)
(105)
B
[e(e2))
and we keep the diffusion covariance implicit as
Znp ~ N(
C la
Σ(x) Cov, (opign
(Su, (Cu,(w) + Znp)),
(106)
B
Phase 2:
vf(,,)
dx, Erf
dt + VTV E(X,)aWe
(107)
C3oBr B² +
2
where E1x) = I8 Erf
ff(x)
and Erf(.) is applied component-wise.
o2
Then, Eq. T04 and Eq. 107 are order approximation of the discrete update of Phase I and Phase 2 of DP- SignSGD, respectively.
Proof: The proof is virtually identical to that of Theorem B.5 Hence, highlight only the necessary details for each phase. Let A = I1 be the one-step increment.
Phase In the fully-clipped regime, the privatized direction used by DP -iggnngg is
(x)
g(x)
+ ZDP = Cu, + Zpp.
(108)
B
[e((2)]
EEN
with u.y in Eq. 105 and 7pr (0, 0 Ga ggeetT) Ta). Therefore
E E
D. DP isign (Cu, (x) + ZpP)]
(109)
Remember that, for scalar random variable Y, we have Elsign(YY) 1 - 2P(Y < 0), and for Y A/(0, 1) it holds ((v) 1 + Erf (1) Fix i E {I,..., Conditioned on 7,
27

---

Published as a conference paper at ICLR 2026
the random variable (x) + Zpp; is Gaussian with mean (() and variance C² /r/ B², hence
(-
B
Epplsign (C), + ZDP;)) | =1111 2Φh
un,((t)
DDP
B
Erf
ODPV (y,((7))
(110)
Collecting the coordinates and taking expectation over y yields
E A
E,y
Erf
B
uy 21
(111)
@DDV2
For the second moment, define S(x) = sign (z) + ZDP) and m(x) = [[S(x)] Then
Cov(A) = Cov (5 =/²"(a))
(112)
We emphasize that do assume the coordinates of S(x) to be independent after averaging loff-diagonal entries of E(x) may be non-zero). Nonetheless, since Sz(x) E {+1}, we have for each coordinate
Var(S:(x)) = - -m;(2)²
(113)
and therefore the trace satisfies
d
Tr(E(t)) Σ Var(S.(x)) = =d (m)))( < d.
(114)
This trace bound is the only property of the diffusion used later in Lyapunov arguments. Define now
B
b(x) -B, Erf @DPV2 (10))
(115)
Σ(x) Cov- (Cu,(1) ZDP)).
(116)
Then, from Lem B.3 and Thm. B.4 the claim for Phase follows.
Phase 2: Remember that, from Assumption B.2 V Vf + Zy, where Zr N (0, ) We
calculate the expected increment
E| = B sign V fz()) + ZDP)]
(117)
Elsign V((x) + Z, + ZDP)
(118)
Vs(x)
-rn Erf
(119)
/
C2a be a B
B
Instead, the covariance becomes
Vf(x)
Cov(A)is B DP
sign(V. Znp) Erf
(120)
C3o
o7
+
13
B3.
vf(x)
sign Vf + (DP) EErf
(121)
Coo B
o²
B
=1 B,, DP [sign V ff, + ZpP);
sign (V,, + ZDP)s
(122)
0,f(z)
0,f(x)
-7² Enf
Erf
(123)
/2
C2 13a
C2
+
o2
B
28

---

Published as a conference paper at ICLR 2026
If i j, we have
d.f(z)
Cov(A); 12
1 Erf
(124)
Coopp
B + 03
B
while if
0xf(x)
0,f(x)
Cov(A)is =1p Erf
Erf
(125)
CRoBp 132
an
1/2
Coabr o
0xf(x)
0zf(x)
1 Erf
Erf
0,
(126)
1/2
C2o
1²
0² B
1/2 1²
CRohe
+
Define now
Vf(z)
b(x) = Erf
(127)
C2 o2
+
Vf(x)
Σ(x) = Ia Erf
(128)
Ca 87 +
Then, from Lem B.3 and Thm. B.4 the claim follows.
Corollary B.11 Under our assumptions, using the linear approximation Erf(z) z in neigh-
V
borhood of 0 together with Lemma A.2 the drift terms in Eq. [04 and Eq. T07 simplify as follows. We keep the diffusion covariance implicit (since our Lyapunov bounds depend on only through its trace).
Phase 1:
2 (K) ff(X,) (X,) dW.,
dX,
(129)
dm DDPO-
where Ei(x)= = (D (Cu, (x) ZDP)) with u, as in Eq. 105 and ZDP as Eq. 106
Phase 2:
dXX, -
ff(X)) + V Ex), dW,
(130)
C2 0²
B
2
where Σz(x)= La Erf
V((w)
2207
Moreover, the diffusion satisfies
Tr(E,(())) sd.
(131)
which is the only property of the diffusion used in our trace-based Lyapunov bounds.
29

---

Published as a conference paper at ICLR 2026
Proof: Let us recall that
Vfe(z)
(x)x) B EEY
(132)
Then, [x),(()] < 1 for all i = ,,...,d d by definition. Additionally, let us consider the term which appears in Eq. 107 in the argument of the error function Erf. That can
[0,1(1))
y coeBp +o2/B)
be interpreted as Signal-to-Noise Ratio (SNR), and it is clear that if this is large, the descent is stronger, while if it is small, the descent weaker. To derive our results, we put ourselves in the < 1. We add that this is
worst-case-scenario where the SNR is small, e.g.
(8,f(x)
C B²
+0//B)
particularly true when + 0, which is the high-privacy regime we study in this paper.
Phase 1: Since (x)) 1, we Erf(2) /=² and write
By Erl (BB
@DPV2 (n(())
B
By
V
B
(x)
TDDP
2 B
V
V
Eg
TDDP
2 (K(v) f()),
V dx appoy
(133)
where in the last step we used Lemma A.2 in Eq. 129
Therefore, the drift in Eq. 104 becomes the one stated
For the diffusion term we keep Σ (x) DD (z)) implicit, where S (x) = sign(Cin (x) + ZDP) and m2 (z) = E (x)]. Since (1,) {=1}, we have (ar() = (z) hence Tr(2,
(m)(z)] < d.
Phase 2: Since
(=)(
< 1 for 1,.... d, we can use the same linear approxi-
c2o g
+o?/B
mation of the error function. In detail, one has
fz(z)
2
Vf(x)
2
1
Erf
vf(x). (134)
Cza B²
a²
V
C2 B
o2
π
Coopp
az
B
B
Therefore, Eq.
107 becomes Eq.
130
The diffusion covariance E2(x) Ia - S2 with S2(x) =
Erf
v((w))
also satisfies Tr(E2(x)) = d (152( s by the same argument.
2 DP n2
o2
Remark B.3 In the loss and gradient-none bounds that follow, the diffusion enters only through the trace term (TXV)((X))(()))) < TT()).))). Since E(x) Cov (S(x)) for some S(x) E (+1]d, we always have Tr(S)())) S d; hence we keep E, implicit and do not require any diagonal approximation mordinate-independeen
assumption.
Theorem B.12 Let f be L-smooth and p-PL. Then, for E [0, T), we have that
Phase 1, i.e. when the gradient is clipped, the loss satisfies:
E[((X,)) & e e $1 +
e $1 )
VTnL
(135)
BB
30

---

Published as a conference paper at ICLR 2026
Phase 2, i.e, when the gradient is not clipped, the loss satisfies:
((((x,)) f(xoe
c2
C²
B T
VTnId
600 BT
C²
B E (136)
First of all, observe that in both phases E(r) = Cov(S(z)) for some S(x) E {=1]]. In particular, T+{S(x)) S d (and thus Tr(V2 ((X))(())) S Ld when V 3 Lla).
Proof:
Since f is μ-PL and L-smooth it follows that 2xf(x) & [V and v2 1 Lis- Then, By applying the Itō formula, we have
Phase 1:
2 K(v) df(x)) S - duT oy q glog(1/6)
Be VV(X)) + " Tr (V2 di +
(137)
2 (
Be
ndL
$ <-2 dul on q glog(1/8) f(X))dt dt + O(Noiee).
(138)
Therefore,
E[f(X,)) sf(X))e -2H
Kiui
(139)
+ 11-
-2p
Riol
o )))
TT ndă Lo, (log(1/0)
(140)
4uK(v) Be
Phase 2: As for Phase 1, by applying the Itô formula one has
df(X,) S - 2
1 B 20 log(//0)T (
(141)
+ " Tr (V f(x))_(X)) di O(Nois))
(142)
2
2
((X))l ndL di + O(Nois)). (143)
< 201
π
2 o + B 2² (1/61
2
Therefore
Ve 1 20 (05 61
E[f(Xi)
(144)
IT ndT. 4μ
jeo? log(1
+ 1-
+
BT
B²
Finally, by suppressing problem-independeed constants, such as 2, ग,, K (v), the thesis follows.
Theorem B.13 Let f be an L-smooth function. Define
dm 0. log(1/0)
2o
C² log(1/0)
K3 max (V2
(145)
BK(v)
BT
B²
Then
[|SX $ K₂ ((Xo)
TVT + +ndINT)
1
(146)
:.
where T ~ Un).
31

---

Published as a conference paper at ICLR 2026
Proof: Since in both phases E(z) Cov(S(x)) for some S(x) {=1}², we have Tr(E(x)) I d, hence Tr(V2 ((x)) Σ Ld when f 3 Lla. Therefore, for worst-case analysis the drift
is the only term worth comparing. Let then K3 as in Eq. 145 Applying the Itō formula to the worst-case SDE we have
df (X, $ - - =(VTKs))
dt +
? Tr (V))(X)))())) dt - (NNise)
(147)
2
<-<(VTKa)~ lEdl + ndL
dt O(Noise).
(148)
Then, by integrating and taking the expectation
11 Latt S K3VT f(Xo)) + ndLT
E
|
[Vx
E-
(149)
2
K3
(s(Xa) + adder
=
E
| (xx,) (x
--
(150)
nVT
2
f(Xo) nVT
ndLVT
=E | K3
+
(151)
2
where in the last step used the Law of the Unconscious Statistician and E~ Uni Finally, by suppressing problem-independent constants, we get the thesis.
B.2.1 MIXED-REGING GRADIENT BOUND
Analogously Section B.I.I. we extend the idealized tully-clipped/unclippedd analysis the re- alistic mixed regime in which each mini-batch contains both clipped and unclipped samples. The following result shows that the same ((1/e) gradient-nore norm scaling in Theorem B.13 continues to hold in this mixed regime (up to constants).
Theorem B.14 Let f Fd + R be L-smooth. In the mixed regime, we can write the SDE of DP DP-SignSGu as
dx, Umix (Xt) di + vn Emix (omix(X,)) (x,) 1/2
(152)
where
B
((o))
I (x) TF.
Erf
(153)
Canpy2
[((z) T kDDP)) ))
Emix (x) = DD
sign
(154)
with
G(x) = Σ C(V)
Zpr N(O, C² TP
(155)
B
EET
In particular, letting mmix(()) (x)
=
E
Erf
CODPYE Gox))((())))) we have Tr Tr(Emix(:)) = d
B
[m)ix(())B3 d.
Define
Rd a.a (/g(1//)
Cq Cqy (oVlog(1/))
K4 max
(V>
BK(D)
2
B
(156)
Then
E [|V((X)))
< K4
f(Xo) nVT
ndLTT
1
(157)
2
where E E~ Unif(0,77) T).
Remark B.4 By construction have K₃ < K3. Theorem B.14 provides formally tighter upper bound than Theorem B. 13 However, note that the first term in the definitions of Ka and KA (Equations| 145 and 156 respectively) scales as Vd. Since d is assumed to be large, this term typically dominates the maximum in both constants. As consequence, in the dime-siima regime of interest we effectively have Kg = Ks, and the improvement from the mixed -hasee analysis negligible in practice.
32

---

Published as a conference paper at ICLR 2026
Proof:
We divide the proof into two steps: i) SDE derivation, ii) gradient bound.
Step 1: SDE derivation.
Write one update of DP- SignSDD in the mixed regime as
1 Sk sign Gk ZDP
Tk+1 Ik
Sk:
(158)
B
where
Gk 1111 E C(V
DPP N(O. C² o
(159)
EETN
Conditioned on 7k, each coordinate of G2 + 1/4 ZDP is Gaussian with mean (G&); and variance CP BB, hence
(
B
(G&).
E DP (ms Erl
CODPV2
(160)
Collecting the coordinates and taking expectation over "/k gives
B
B| - mx] -1
Erf
Coopr2
Gk
n mixx (x)x
(161)
For the second moment, define
Emix(x) CoV,,DP(St) with x Ik.
(162)
Then
Cov(#x+1 Izk] = 2 E (Ix)) Since Sp E {=1]d, we have Tr(Emix (2)) S d. Therefore, by Lem B.3 and Thm. B.4 we obtain the SDE
(163)
dX,- bmix (X,) dt Emix (X, 12 dW.
(164)
Step 2: Gradient bound. As argued in the proof of Corollary B.11 nentwise) without loss of generality. Using Erf(2) linearized drift
B
G(x) << 1 (compo-
CODPV2
2 in a neighborhood of 0, obtain the
2 B B[G(=)).
b (z) - Copp
(165)
In the mixed regime, we can write (for some Pk E [o, 1] encoding the expected clipping rate at mr))
[e(x)) [V
B[G())) = = C
(1 - pk: Reg [c(())]
(166)
Using Lemma A.2 for the clipped component and unbiasedness for the unclipped component yields
CK(v)
+ (1 pb)) (1-pk) ||T((1)]
vf(x) = Pk:
(167)
vd
Therefore, applying the Itô formula and using Tr(v (X,) (Xi) & L Tr(EmixxX))) S Ld, have
2 (LXX).) di + Tr (x²)(X))E di O(Nose))
df(X)) -v Capp vs(X,) B(G(X.)
< (ppaa + pk )2)) [ ndL
dt O(Nois)).
(168)
where 01 = = B K(v)
TOPF aDF v.vd and d2 = Copp
Expanding ODP and defining K₁ in Eq. 156 have
cVT-IK K < ppad (1 pk p2)a2,
Vpk E [0,
(169)
Thus
ndL
df(x,) < - [f
+
+ (Noise).
(170)
2
33

---

Published as a conference paper at ICLR 2026
Taking expectations and integrating over [0, 7] gives
E ! |V |1 di se- K4 (f(X)) + ndlr
(171)
2
! + [I
K1
(f(X0) ndLT
E
(172)
nTT
((Xo) qdI.vT 1
E [IVS K4
(173)
n/T
2
where in the last step we used the Law of the Unconscious Statistician and & ~ Unif(0,
Finally, we derive the stationary distribution of DP-SignSGD: We empirically validate in C.6
Fig.
Theorem B.15 Let f(x) -z0 HE where diag(A)..... Aa). The stationary distribution of Phase 2 is
E [Xy] =Xye KHT;
(174)
Cov(X,) =Xpe- -2KH (c (e-Nk(HT-1)
(175)
+ (2KH +112+)) -1 (1-
(1-E-2KH-RR1P
(176)
where = / V22n-lo5+ 1 20 (og 1og(1/
Proof:
Since H is diagonal, we can work component-wise. Let us remember the SDE:
2
1
2
d'x,i
dt vT. 1
a² x dW
(177)
π
C +
o B
π
Co B
B
To case the notation, we write K = Ven ln? 202 Ing(1/8)7 in closed form
Hence, we can write Xe,
X To KAdt + vi f feee-KKiee ---iittts) V1 K2A3X,
(178)
Due to the properties of the stochastic integral, we immediately have
VeBB-lo 202, dde
E x = Xo,ze
(179)
Using the Itô formula on g(x) = x², we have
d (x?,) 222, X2,dt +
11 2dt 2dt
"
2K
Xxx2,,tt dt + O(Noise)
(180)
2
2
-EXX? = x2 42
2KΛ, nAtK2 (1 11-e2kkkttk33)11
(181)
therefore
Cov(Xr.) E(X 2[
(182)
-X -(2Kh-(2k -22
2K>; + nXtK2 (1 (1-c - (1-e-2kAttwk333)) X²
1
2KA
e 2KXrt ( -pk3 11 1) 2K + ! (1-e (1-e
7
(183)
Finally, we present a result that allows us to determine which of DP DB-SignSGD and DP-SGD is more advantageous depending the training setting.
34

---

Published as a conference paper at ICLR 2026
Corollary B.16 If o- O 2 L, then DP DP-SignSGD always achieves better privacy-utility trade-off than DP-SGD, though its convergence is slower. If
B
there exists critical privacy level
C²TB
log (1)
(184)
V
n² (B
o22)
such that DP-Si gnSGD gnSGD oulperforms DP-SGD in ufility whenever < Et, but still converges more slowly than DP-SGD.
Proof: The Phase 2 asymptotic terms at T are
ASGD
TndL J
o C²
Asign
VTndL 2 TB
C 42
(185)
TB
1
We compare Asign
< ASGD.
Cancelling the common factor ydL
gives
VT
+
T
Pe'd TB
C
4² B
(186)
TB
Multiplying by x2 and dividing by the positive square root yields
0 C SVT
(187)
All quantities are non non-negative, so squaring preserves the inequality:
2T
< T
TB
+
2
42
B
+
(1-
< C 2 T.
(188)
Using + = /lo(((1)) gives
(1-%)
C²
Tlog !
(189)
n²
If 2 1, the left coefficient is non-positive and the inequality holds for all > 0. If B < 1, solving for E yields
C
log (1) Ex
(190)
n² B o??)
which proves the claim. Interestingly, by keeping 1) and C depend on the optimizer, we get
C2
$2 B
o² C $2
VT Msign
+
Tyigdd
+ BT
(191)
BT
B2
We observe that a.y + oo, DP-SignSGD always better than DP-SGD, while if o, - 0, there always threshold =*. Since the algebraic expressions complex, we believe this is enough to show that our insight is much more general than the case derived here and presented the main paper.
C EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
Our empirical analysis is based the official GitHub repository https://gittub.. com/kenziyuliu/ DP2 released with the Google paper (Li et al. 2023). In particular we consider the two following classification problems:
IMDB Maas et al. 2011) is a sentiment analysis dataset for movie reviews, posed as a binary classification task. It contains 25,000 training samples and 25,000 test samples, with each review represented using a vocabulary of 10,000 words. We train a logistic regression model with 10,001
parameters.
35

---

Published as a conference paper at ICLR 2026
StackOverflow (Kaggle, 2022),
TensorFlow Federated 2022) is large-scale text dataset derived from Stack Overflow questions and answers. Following the setup in (TensorFlow Federated (022). consider the task of predicting the tag(s) associated with given sentence, but restrict our experiments to the standard centralized training setting rather than the federated one. We randomly select 246,092 sentences for training and 61,719 19 for testing, each represented with 10,000 features. The task is cast as a 500-class classification problem, yielding a model with approximately 5 million parameters.
MovieLens-1000 (Harper & Konstan 2015 widely used movie-rating dataset for recommen- dation systems. consists of 100,000 ratings provided by 943 users across 1,682 movies (=:6% non-zero entries). We consider a non-convex matrix factorization problem with an embedding di- mension of 100, resulting in 262,500 parameters. Each observed rating is treated as an individual "record" for differential privacy, and the non-zero entries are randomly split into training and evalu- ation sets.
Optimizers. We train both classification problems using DP-SGD, DP-Si gnSGD and DP -ddam. For 0, learning rate 7,, variance op. and batches ? of size B modeled i.i.d uniform random variables taking values in {l....,my nj. Let It be the private gradient, defined as
1 gk
Σ BET
c[VS(a==]]
1 + - N(0, oppla)
(192)
B
B
and C[:] be the clipping function
Cix] = min { C .1
|| 1
(193)
The iterates of DP-SGD are defined
Tk+1 2k 79
(194)
while those of DP-Si are defined as
Tk+1 23 17 sign [9x],
(195)
where sign[-] is applied component-wise The update rule of DP-Adam is defined as follows:
mk++1
m Bim; + (1 B1) gk.
m
1 - 3 k+I
Uk+1 B24k + (1 32) (gk @ gk),
Wk+1
1 3 k+1 11
Uk++1
(196)
m
Ik+1 Ik
/x+1 +
where gk is the privatized stochastic gradient and defined in Equation 192
Hyper-parameters. Unless stated otherwise, we fix the following hyperparameters in our experi- ments: for IMDB, StackOverflow and MovieLens respectively, we train for 100. 50, 50 epochs with batch size B = 64. The choice of batch size follows the setting in (Li al. 2023). We also aimed to avoid introducing unnecessary variability, keeping the focus the direction suggested by our theo- retical results. Finally, we set 5 = 10 10-5 110- 10-6 corresponding to the rule á = 10-k where the smallest integer such that 10- 1/n for the training dataset size n.
Protocol A.
We perform a grid search on learning rate 7) {0.001, 0.01, 0.1, 1, and clipping threshold C (0.1,0.25 0.5, 3,2,3,5) 5} for SGD, DP ne--ignnGD and DP-Adam both datasets, using ODP = 1: this gives E = 2.712 and E = 0.424 for IMDB and StackOverflow respectively. For MovieLens
use the same hyperparameters grid, but for the noise multiplier use instead o pp = 0.5, which yields s = 1.329. We summarize the best set of hyperparameters for each method on both datasets in Table C.1
36

---

Published as a conference paper at ICLR 2026
Dataset
DP-SGD
DP-SignSGD
DP-Adam
IMDB
(5, 0.5)
(0.1, 0.5)
(0.1, 0.5)
StackOverflow
(3, 0.25)
(0.01, 0.5)
(0.01, 0.5)
MovielLens
(0.1,
(0.001, 3)
(0.001, 1)
Table C.I: Tuned hyperparameters for different methods across the two datasets. The values refer to
(learning rate, clipping parameter); For DP -Adam we also used 31 = 0.9, 32 0.999 and adaptivity c 10-8 in both cases.
Protocol B. For each noise multiplier, we tune a new pair of learning rate and clipping pa-
rameter by performing a grid search. IMDB: For DP-SignSGD and DP-Adam, we consider the following learning rates = {0.01, 0.05, 0.10, 0.15, 0.22, 0.27, 0.33, 0.38, 0.44, 0.50] and
clipping thresholds C {0.05, 0.1, 0.25, 0.5}, while for DP-SGD we consider a different range of learning rates 7 = {0.5, 0.7, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0} and
C c= {0.1, 0.25, 0.5). This tuning is designed to identify the best hyperparameters across a broad range of privacy budgets
{0.01, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0), which correspond to the following noise multipliers: {271.23, 13.56, 6.78, 4.52, 3.39, 2.71, 2.26, 1.94,
1.70, 1.51, 1.36]. StackOverflow: For DP Adam we consider the following learning rates {0.001, 0.003, 0.005, 0.01, 0.03, 0.05, 0.1, 0.5). for DP- SignsSD we add {0.008, 0.015, 0.02, 0.03, 0.04] the list while for DP-SGD we consider different range of learning rates 7 = {0.1, 0.5, 1.0, 2.0, 2.5, 3.0, 3.5, 4.0, 5.0}. For the clipping thresholds we consider C (0.05, 0.1, 0.25, 0.35, 0.5, 1.0) for every method. This tuning is designed to identify the best hyperparameters across a broad range of privacy budgets & = {0.01, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, L.4), which correspond to the following noise multipliers: (42.384, 2.119, 1.060, 0.706, 0.530, 0.424, 0.353, 0.303). Movielens: For DP - SiignngnDD and OP-Adam, we consider the fol- lowing learning rates 77 {0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3} and clipping thresholds C' = {0.1, 0.25, 0.5. 1.0. 2.0, 3.0}. while for DP-SCD we consider different range of learning rates 7) = {0.01, 0.03, 0.05, 0.1, 0.3, 0.5, 1} and C {0.1, 0.25, 0.5, 1.0, 2.0, 3.0). This tuning is designed to identify the best hyperparameters across a broad range of privacy budgets& & = {0.01, 0.05, 0.1, 0.2, 0.5, 1}. which correspond to the following noise multipliers: {66.48, 13.30, 6.65, 3.32, 1.33, 0.66).
C.I DP-SGD AND DP-DD-SSIEEEEE GIGSGD: SDE VALIDATION (FIGURE C.I
In this section, we describe how we validated the SDE models derived in Theorem B.5 and Theo- In line with works in the literature Monzio Compagnoni al. (2025c b). we optimize quadratic and a quartic function. We run both DP-SCD and DP-Si gnSSCD, calculating the full gradient and injecting noise as described in Assumption B.2 Similarly, we integrate our SDEs using the Buler-Maruyany algorithm (See, e.g., (Monzio Compagnoni et al.) (025c). Algo- rithm 1) with Al- = y/ Results are averaged over 200 repetitions. For each of the two functions, the details are presented in the following paragraphs.
Quadratic function: We consider the quadratic function f(x) f Hx. with 0.1 diag I.... 1), in dimension d = 1024. The clipping parameter is set to C = 5, and each algorithm is run for T 10000 iterations. The gradient noise scale is a. /d. The learning rate 0.01 for DP-Si gnSGD. The differential privacy parameters are (c, (5, 10-4 10-1), corresponding a noise multiplier of opr 0.03. The initial point is sampled as xg = vd - (0, Ia), using an independent seed for each method. In this experiment and in all the following where we use a Student's 4, we use 1.
is = 0.1 for DP-SGD and
Quartic function: We also test on the quartic function fxx - Hag? + A
E>E 22 where / = diag(-2, 1,...,1) 1)., x = 0.5, and & = 0.1. The problem dimension, clipping, and number of iterations are the same; d 1024, = 5, T 10000. with gradient noise 0y 1/Vd. Both methods use learning rate of 17 0.01. The differential privacy parameters are (,4)) (5, 10-10 10-1) for DP-SGD and (5, 10-1 2 x 10-1) 1) for DP-S1 igSGGD corresponding
37

---

Published as a conference paper at ICLR 2026
to noise multipliers ODP 0.03 and ODP 0.06, respectively. Initialization is zo for DP-SGD and yo -2 for DP-SignSGD.
30
(0,
va
DP-SGD
--
SDE DP-SGD
DP-Sign5GD .... SDE DP-SignSD
Gradient Norm Quadratic
Trajectories Quadratic
Gradient Norm Quartic
Trajectories Quartic
10a
=(199001")
0.5
10t
-05
-1.0
10
107
10
0
10:
107
Iterations
X1
Iterations
X5
Figure C.I: Consistent with Theorem B.5 and Theorem B.10 we empirically validate that the SDEs of DP-SGD and DP SiGGD model their respective optimizers. For convex quadratic function (left two panels) and a nonconvex quartic function (right two panels), the SDEs accurately track both the trajectories and the gradient norm of the corresponding algorithms, averaged over 200 runs.
C.2 ASYMPTOTIC LOss BOUND (FIGURES)
C.2 AND C.7
This section refers to Figure Figure and Figure C.7 We consider three different scenarios: A quadratic function, IMDB, StackOverflow and MovieLens. Each setup is optimized using DGDS DP-SignSCD, and DD-Adam, and we plot the final averaged training loss across range of privacy levels. In the left panel, we include the exact bounds from Theorem 4. and Theorem 4.3 to show agreement with theory; in the central and right panels, we compare the final losses with the trends in predicted by the same theorems. Experimental details are as follows.
Quadratic: f(x)= 1 Hz, H 104,; d 1024, C 5, T' 50000, o,, 0.01; learning rate 7 0.01 77t with 7/t = (1+1)l) 111) (a decaying schedule for stability: our theory assumes constant so we use this toy experiment for qualitative validation): Adam parameters: 81 = 0.9, Bg = 0.999, 10-8, We used noise multipliers, linearly spaced from to 2, which with q 10 4,8 10-4 correspond to E foo, 6.78, 2.38, 1.19, 0.79, 0.59, 0.48, 0.40, 0.34).
IMDB: Hyperparameters are given in Table C.I We performed 10 runs for each noise multiplier (0.5, 1.0, 2.0, 4.0, 6.0. 8.0, 10.0. 12.0), yielding the following values for (5.425, 2.712, 1.356, 0.678, 0.452, 0.339, 0.271, 0.226). respectively. We report the average training and test loss of the final epoch with confidence bounds (Figure Figure C.7).
Hyperparameters are given in Table each noise multipliers {0.1, 0.3, 0.5, 1.0. 2.0, 4.0, 6.0, 8.0 yielding the following values for e (4.238, 1.413, 0.848, 0.424, 0.212, 0.106, 0.071, 0.053), respectively. We report the average training and test loss of the final epoch with confidence bounds (Figure[aand Figure C.7).
StackOverflow:
We performed 3 runs using for
We performed 10 runs for each noise multiplier (0.06, 0.08, 0.1. 0.2, 0.3, 0.5. 1.0, 1.5 yielding the following values for (11.079, 8.310, 6.648, 3.324, 2.216, 1.329, 0.665, 1.44). respectively. We report the average
MovieLens: Hyperparameters are given in Table C.I
training and test loss of the final epoch with confidence bounds (Figure C.2
38

---

Published as a conference paper at ICLR 2026
MovieLens
DP.5GD
150
DP-SignSGD
DP.mdam
((RX-))
100
10
Figure C.2: Empirical validation of the privacy-utility trade-off predicted by Thm. 4.1 and Thm. 4.3 comparing DP-SGD, gngGG, and DP A:aAdmm: Our focus is on verifying the functional de- pendence of the asymptotic loss levels in terms of 8. Train MSE for matrix factorization MovieLens- 100k, confirm the same pattern: the utility of DP scales as 2. while the utility of DP-SiggnCE scales linearly as J. We observe that the insights obtained for DP DP-Si SignSGD extend to -ddm as well to the test loss.
C.3 CONVERGENCE SPEED ANALYSIS (FIGURE 2 AND C.3
This section refers to Figure and Figure 3% We consider three different scenarios: IMDB, Stack- Overflow and MovieLens. Each setup optimized using DP-SGD, DP-SignSGD, and DP-Adam and six different privacy levels: We plot the average trajectories of the training losses and observe that, when it converges, the convergence speed of DP-SGD does not depend the level of privacy, while the two adaptive methods are more resilient to the demands of high levels of privacy, but their convergence speed changes for every E, predicted Theorem 4.3
IMDB: Hyperparameters are given in Table C.I each noise multiplier {0.8. 1.0, 1.2, 1.6, 4.0, 6.0} and (3.390, 2.712, 2.260, 1.695, 0.678, 0.452]. We report the average trajectories of the training loss with confidence bounds (Figure 2
We performed 10 runs for
corresponding epsilons
StackOverflow: Hyperparameters are given in Table C.I for each noise multiplier {0.37, 0.5, 0.64, 1.19, 1.46, 1.73] and corresponding epsilons {1.146, 0.848, 0.662, 0.356, 0.290, 0.245]. We report the average trajectories of the training loss with confidence bounds (Figure 2
We performed 3 runs
MovieLens: Hyperparameters are given in Table C.1
We performed 10 runs
for each noise multiplier (0.06, 0.1, 0.2, 0.3, 0.5, 1.0F and corresponding epsilons (11.080, 6.648, 3.324, 2.216, 1.330, 0.665) We report the average trajectories of the train-
ing loss with confidence bounds (Figure C.3
MovieLens
DP-SignSGD
DP-Adam
60
r=11.08
e-6.65
s-2.22
c=1.33
r=0.66
20
40
20
40
20
40
Epochs
Epochs
Epochs
Figure C.3: Empirical validation of the convergence speeds predicted by Thm 4.1 compare DP DP-SGD, DP- Di-SiggSGD, and DP-Adam problem on the Moviel.nns dataset. We verify that when DP-SGD converges, its speed is unaffected by As expected, it diverges when too small. Regarding DP-Si gnSGD and DP-Adam, they are faster when is large and never diverge even when this is small.
4.1 and Thm. 4.3 We
optimize non-convex matrix factorization
39

---

Published as a conference paper at ICLR 2026
C.4 WHEN ADAPTIVITY REALLY MATTERS (FIGURE AND FIGURE C.4
This section refers to Figure 3 and Figure C.4
Each setup is optimized using DP-SGD. DP DP-SignSGD, and DP-Adam. We consider different batch sizes and for each plot the final loss values for different privacy levels, similarly to Section C.2 We highlight the possible range of e* and dash-dotted line to mark its approximate value, suggested by each graph. As predicted by Theorem |.5, the empirical value of s* shifts left as we increase the batch size. Experimental details are as follows.
IMDB: Hyperparameters are given in Table C.1 We select wide range of noise multipliers: {0.5, 1.0, 1.2, 1.5, 1.8, 2.0, 2.2, 2.5, 2.8, 3.0, 3.2, 3.5, 3.8, 4.0, 4.5, 5.0, 6.0, 8.0, 10.0, 12.0) and increasing batch sizes B (48, 56, 64, 72, 80). The corresponding epsilons are
B 48: {4.698, 2.349, 1.879, 1.566, 1.342, 1.174, 1.044, 0.940, 0.854, 0.783, 0.723, 0.671, 0.626, 0.587, 0.522, 0.470, 0.391, 0.294, 0.235, 0.196];
B 56: (5.070, 2.535, 2.028, 1.690, 1.449, 1.268. 1.127, 1.014, 0.922, 0.845, 0.780, 0.724, 0.676, 0.634, 0.563, 0.507, 0.423, 0.317, 0.254, 0.211};
B 64: (5.425, 2.712, 2.170, 1.808, 1.550, 1.356, 1.205, 1.085, 0.986, 0.904, 0.835, 0.775, 0.723, 0.678, 0.603, 0.542, 0.452, 0.339, 0.271, 0.226];
B 72: {5.740, 2.870, 2.296, 1.913, 1.640, 1.435. 1.276, 1.148, 1.044, 0.957, 0.883, 0.820, 0.765, 0.717, 0.638, 0.574, 0.478, 0.359, 0.287, 0.239];
B 80: {6.070, 3.035, 2.428, 2.023, 1.734, 1.517, 1.349, 1.214, 1.104, 1.012, 0.934, 0.867, 0.809, 0.759, 0.674, 0.607, 0.506, 0.379, 0.303, 0.253).
For each batch size, we performed runs and plotted the average final value of the Train Loss and the empirical e*: these observed values follow the direction indicated in Thm. 4.5 For visualization purposes, we show only a smaller window of e values satisfying 0.75 < & S 1.25.
OverkOv: Due to the higher computational cost required, with our limited resources we man- aged to select only a restricted range of noise multipliers: {0.1, 0.3, 0.5, 1.0, 2.0, 4.0, 6.0, 8.0) and batch sizes: {48, 56, 64). The corresponding epsilons are
B = 48: (1.223, 0.734, 0.367, 0.184, 0.092, 0.061, 0.046);
B 56: {1.322, 0.793, 0.396, 0.198, 0.099, 0.066, 0.050);
B = 64: (1.413, 0.848, 0.424, 0.212, 0.106, 0.071, 0.053).
For each batch size, we performed 3 runs and plotted the average final value of the Train Loss and the empirical e*: these observed values follow the direction indicated in Thm. 4.5 For visualization purposes, show only smaller window of E values satisfying 0.15 < < 0.75.
MovieLens: Hyperparameters are given in Table C.I We select a wide range of noise multipli- ers: f0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95) and batch sizes (56, 64, 72). The cor- responding epsilons are
B 56: (4.146, 2.487, 1.777, 1.382, 1.131, 0.957, 0.829, 0.732, 0.655):
B = 64: (4.432, 2.659, 1.899, 1.477, 1.209, 1.023, 0.886, 0.782, 0. 0.700):
B = 72: (4.700, 2.820, 2.014, 1.567, 1.282, 1.085, 0.940, 0.829, 0. 0.742).
For each batch size, we performed 10 runs and plotted the average final value of the Train Loss and the empirical e*: these observed values follow the direction indicated in Thm. 4.5 For visualization
purposes, we show only a smaller window of E values satisfying 0.75 < E 1.5.
C.5 BEST-TUNER HYPERPARAMETER (FIGURES AND C.5
This section refers to Figure and Figure C.5 On top of the hyperparameter sweep performed described in Section C we additionally tune DP-SCB for the smaller values of e. As predicted by Theorem 4.6] the optimal learning rate for DP-SGD scales with while those of the adaptive methods are almost constant. Furthermore, we observe that once we reach the limits of the hyper- parameter grid, DP-SGD loses performance drastically.
40

---

Published as a conference paper at ICLR 2026
DPSGDD
DP SignSGD
DP.Adam
Batch size
Batch size
Batch size
4.00
0.458
0.418
3.75
F(RX1)]
3.00
0.11
0.30
0.45
0.60
0.75
0.15
0.30
0.45
0.60
0.75
0.15
0.30
0.45
0.60
0.75
DP-SGD
SignSGD
Dd-d
Batch size 56
Batch size 64
Batch size 72
=1131
1.001
60
F1RXh1]
20
0.5
1.0
8 0.5 1.0
1.2
1.4
1.2
1.4
0.8
1.0
1.2 1.4
Figure C.4: (Top Row) StackOverflow and (Bottom Row) MovicLens: From left to right, we decrease the batch noise, i.c., increase the batch size, taking values B = 48, 56, 64: As per The- orem 4.5 the privacy threshold s* that determines when DP-Si DP-SignSGD is more advantageous than DP-SG- shifts to the left. This confirms that if there is more noise due to the batch size, less privacy noise is needed for DP-S pe-Sig to preferable over DP-SGD.
IMDB: We additionally tune DP-SCD using = {0.001, 0.005, 0.01, 0.05, 0.1, 0.5] and C = {0.1, 0.25, 0.5) and add the corresponding values using the cyan line. On the left, we plot the average of the final 5 train loss values and confidence bound for each method against the privacy budget e; On the right, we focus on the scaling of the optimal learning rate with respect to e.
We additionally tune DP-SGD using 1j {0.001, 0.01, 0.05) and add the corre- sponding values using the cyan line. As above, on the left, we plot the average of the final training loss values and confidence bounds for each method against the privacy budget e; on the right, we focus on the scaling of the optimal learning rate with respect to E.
StackOverflow:
MovieLens: We additionally tune DP-SGD using 1) (0.0001, 0.0003, 0.001, 0.003) } and add the corresponding values using the cyan line. On the left, we plot the average of the final train loss values and confidence bound for each method against the privacy budget e; On the right, focus on the scaling the optimal learning rate with respect to
MovieLens
DP-SignSGD
DP-SGD
Train Loss
40
Optimal
0.3
0.4
DP-ddam
35
0.2
DP-SGD Tuned
0.1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.5
1.0
Figure C.5: Empirical verification of Thm. 4.6 and Thm. 4.7 under Protocol dataset. We tune (1), C) of each optimizer for each & and confirm that: i) all methods achieve comparable performance across privacy budgets; zi) the optimal n of DP-SGD SGD scales linearly with s, while that of adaptive methods is essentially e-independent; iii) failing to sweep over the "best" range of learning rates causes DP-SGD to severely underperform, whereas adaptive methods are resilient. On the left, DP-SGD degrades sharply for small e. Indeed, the right panels shows that the selected optimal 7) flattens out, while the theoretical one would have linearly decayed more: The "best" was simply missing from the grid. A posteriori, re-running the sweep with a larger grid (DP-SGD Tuned) recovers the scaling law and matches the performance of adaptive methods.
the MovieLens
41

---

Published as a conference paper at ICLR 2026
C.6 STATIONARY DISTRIBUTIONS
In this paragraph. describe how validated the convergence behavior predicted in Theorem B.9
and Theorem B.15 To produce Figure C.6 we run both DF and DP- DP-SiqnSGD on f(a)
x Hx, where H diag(2, 1). To (0.01, 0.005), 1) 0.001, oy ODP = 0.1, C = 5. We average over 20000 runs and plot the evolution of the moments compared to the theoretical prediction provided in Theorem and Theorem B.15
x10
X10
2.5
DP-SGD
1.0
2,5
SignSGD
Theor. Pred.
Theor. Pred.
2.0
0.8
1.5
Var(X;]
0.6
1.5
Var[X;]
1.0
1.0
0.5
0.5
DP-5GD
DP-SignSGD
0.0
0.0
Thear. Pred.
0.0
0
Theor. Pred.
2
[[X,]]
-10-
VarlX,paor
E[X]]
x10-1
Var[X,]
Figure C.6: The empirical dynamics of the first and second moments the iterates X, DP-SGD (left two panels) and of DP-Sign.SD (right two panels) match that prescribed in Theorem B.9and and respectively.
Theorem B.15
C.7 ADDITIONAL. RESULTS TEST LOSS
Interestingly, the insights provided Theorem 4.1 and Theorem 4.3 regarding both asymptotic bound and the convergence speed extend, in practice, also to the test loss. In the same set-up of Section C.2 we plot the asymptotic values of the Test Loss and interpolate with (11Ez) and (1)=²) show that they match the predicted scaling.
IMDB
StackOverflow
DP-SGD
DP-SignSGD
DP-Sign5GD
DP-Adami
DP.Adam
Test Locs
ele-i)
10'
oir
oie-1)
ole
10
10
6x 10-1
x
0.5
1.0
1.5
2.0 2.5
0.00
0.25
0.50
0.75
1.00
1.25
Figure C.7: Privacy-utility trade-off on the test loss, comparing DP-SGD, DP-SignSGD, and DP-Adam. Left: Logistic regression on the IMDB dataset. Right: Logistic regression on the StackOverflow dataset. In both cases, the empirical scalings predicted by Thm. 4. and Thm. 4.3 carry over from training to test: DP-SGD follows the trend, while adaptive methods follow the trend. This demonstrates that not only do theoretical insights generalize the widely used DP DP-Adam, but also extend from training to test loss.
Similarly, in the same set-up Section + observe that once again the convergence speed of DP-SGD is not affected by the choice of E, while adaptive methods clearly present different e-dependent rates.
plot the trajectories of the Test Loss (Fig. C.8
42

---

Published as a conference paper at ICLR 2026
IMDB
DP-SGD
DP-Sign
DP-Adam
0.9
3,3339
0.8
c=2.71
t=2.26
c=1.70
E-D,68
6-0.45
40
20
Epochs
Epochs
E Epochs
StackOverflow
DP-SGD
DP-Sign SGD
DP-Adam
4.0
E-1.15
3.8
6-0.85
3.6
11
t=0.36
3.4
c=0.29
3.2
E-0.24
20
40
0
20
40
20
40
Epechs
Epochs
Epochs
Figure C.8: We compare the Test Loss DP-SGD, DP-SignSCD, and DP-Adam as we train
logistic regression the IMDB dataset (Top Row) and the StackOverflow dataset (Bottom Row).
D LIMITATIONS
As highlighted by Li et al. (022)) the approximation capability of SDEs can break down when the learning rate 77 large when certain regularity assumptions V and the noise covariance matrix are not fulfilled. Although such limitations can, in principle, be alleviated by employing higher-ord ooder weak approximations, our position that the essential function of SDEs is to provide simplified yet faithful description of the discrete dynamics that offers practical insight. We do not anticipate that raising the approximation order beyond what is required capture curvature-dependen effects would deliver substantial additional benefits.
We stress that our SDE formulations have been thoroughly validated empirically: the derived SDEs closely track their corresponding optimizers across a wide range of architectures, including MLPs, CNNs, ResNets, and ViTs (Paquette et al. 2021 Malladi et al. 2022 Monzio Compagnoni et al. 2024
2025b
Xiao et al. 2025 Marshall et al. 2025).
43


Reviews:
[{"review_id": "YRXiRczRwT", "reviewer": "Reviewer_4NeB", "summary": "Thru discussing how DP noise interacts with adaptivity in optimization, DP-SGD and DP-SignSGD are proposed in this work, where DP-SGD is shown to be converged at a speed independent of ε, DP-SignSGD is with convergence speed scales linearly in ε. Under optimal learning rates, both methods reach comparable theoretical asymptotic performance, while this leaves potential issues in practice.", "questions": "- The theoretical analysis is derived for DP-SignSGD, while the conclusions are empirically with DP-Adam. Please provide more discussions and valiations.\n- The experimental validation for Protocol B on the StackOverflow dataset needs to be provided.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- SDE-based analysis of differentially private optimizers, using this framework to expose how DP noise interacts with adaptivity and batch noise.\n- DP-SGD is shown ito be converged at a speed independent of ε.\n- DP-SignSGD: its convergence speed scales linearly in ε, while its privacy-utility trade-off scales as O (1/ε)", "weaknesses": "- The assumptions on SNR (signal-to-noise ratio) are built on linear approximations that are only valid in a high-noise, low-signal regime.\n- A general Student-t distribution for batch noise is used to capture heavy tails, while it is not used consistently in assumption B.2..\n- The experimental validation for Protocol B on the StackOverflow dataset is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQKYr5Gvjw", "reviewer": "Reviewer_a1Sd", "summary": "-  This paper investigates differentially private learning through the lens of a stochastic differential equation. Based on theoretical understanding, the paper investigates how the DP-SGD, DP-SignSGD, and its variant DP-Adam perform in various privacy budgets. The authors argue that DP-SignSGD is epsilon-dependent, which reduces the burden of parameter tuning in DPDL.", "questions": "-\tThe paper investigates the difference between DP-SGD and DP-SignSGD in terms of differentially private deep learning. Is there any related work on non-private optimization, or does this comparison solely rely on a DP sense?\n-\tFor protocol A, how did the authors choose the parameters? For private learning, the clipping value is also as important as the learning rate. Can the authors provide more results while varying hyperparameters in both protocols A and B?\n-\tThe paper’s analysis is based on that the optimal performances of DP-SGD and DP-SignSGD are almost similar (without considering parameter search). However, as far as the reviewer knows, the current methods prefer DP-SGD compared to DP-SignSGD. Does DP-SignSGD still provide comparable results with private fine-tuning or bigger architectures? Refer to [1] or recent tuning methods in larger vision or language-based DP papers.\n\n    [1] Unlocking High-Accuracy Differentially Private Image Classification through Scale, 2022.\n\n-\tWhat about the case of DP-SGD-based Adam, instead of DP-SignSGD-based Adam?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper investigates the optimization process of differentially private learning in terms of SDE, which has not been actively investigated.\n-\tThe authors provide a theoretical analysis of why DP-SGD and DP-SignSGD differ in training dynamics, especially with hyperparameter setups.\n-\tBased on their observations, the authors argue two protocols that cover both fixed and tuning parameters.", "weaknesses": "Please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6oOv3t1KV9", "reviewer": "Reviewer_B6gH", "summary": "The paper studies DP stochastic gradient descent and its variants via modelling the DP-SGD dynamics via stochastic differential equations. It develops continuous-time approximations for adaptive algorithms and distinguishes between two training “phases”: an initial exploration phase and a later convergence phase. The analysis aims to explain how noise injection from differential privacy interacts with the optimization dynamics, and how the privacy parameters and optimizers' hyperparameters (clipping bound, batch size and learning rate) influence convergence. The authors attempt to unify previous heuristics (DP-SGD, DP-SignSGD etc.) under a common SDE framework, claiming to provide an interpretable connection between privacy, optimization noise, and generalization. The paper also reports empirical results to illustrate the phase-transition behavior. There is a very interesting phenomenon that is supported both by theory and experiments: DP-SGD has $\\varepsilon^{-2}$ behaviour for small $\\varepsilon$-values whereas DP-Adam and DP-SignSGD have $\\varepsilon^{-1}$ behaviour.", "questions": "DP-Adam refers to the algorithm considered by (Balles and Hennig, 2018), i.e., to the plain Adam with DP-SGD gradients, right? Why not to analyze other adaptive optimizers like the versions of DP-Adam that are tailored for DP (e.g., Tang and Lécuyer, 2023, or Li et al., 2023, the references you also list)? Were DP-Adam and DP-SignSGD chosen because the analysis is amenable for them?\n\nRecently, certain filtering methods have turned out to give good privacy-utility trade-offs, see e.g.\n\nZhang, X., Bu, Z., Balle, B., Hong, M., Razaviyayn, M., & Mirrokni, V. DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction. In The Thirteenth International Conference on Learning Representations 2025.\n\nCould this SDE view allow analyzing those methods as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is well written, seems to be of very high quality.\n\n- The fact that the SDE view is able to capture the experimental behaviour that DP-SGD has $\\varepsilon^{-2}$ behaviour for small $\\varepsilon$-values whereas DP-Adam and DP-SignSGD have $\\varepsilon^{-1}$ behaviour (see Figure 1) is very impressive. \n\n- The SDE view is well motivated and also commonly considered in the literature (e.g., Blei et al. 2018).", "weaknesses": "- The paper focuses on only on few adaptive optimizers, and I am a bit surprised about their choices: DP-Adam (Adam with DP gradients) and DP-SignSGD (which is not that well-known). The reason might be that the analysis is amenable for them (questions below), and I think the contribution is very valuable neertheless.\n\n- Due to the fact that very few adaptive optimizers seem to actually fit into this SDE framework (or can be seen as discretizations of SDEs, meaning that the weakly converge to them in the vanishing step size limit), I have a feeling that this framework does not actually help in designing new hyperparameter adaptive DP optimizers. The paper seems thus to give an analytical explanation of certain differences in the optimizers' behaviors ( $\\varepsilon^{-1}$ vs.  $\\varepsilon^{-2}$ error behaviour).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWeIe2vi63", "reviewer": "Reviewer_g3be", "summary": "The main methodological contribution is to approximate the discrete optimization algorithm with a stochastic differential equation (SDE). It focuses on DP-SGD and DP-SignSGD under per-example clipping and provides the first theoretical framework that connects SDE with convergence dynamics.", "questions": "1.The paper could provide more heuristic insight into why DP-SignSGD, which can be seen as a post-processing of DP-SGD and thus should not improve efficiency, nevertheless achieves better empirical and theoretical utility. In particular, what property of the sign operation makes it more robust to DP noise?\n\n2.Additionally, the role of the clipping bound C deserves clarification: while a larger C should intuitively reduce gradient bias, the theory suggests that a larger C moves the iterates further from the optimum.\n\n3.The two-phase assumption (clipped vs. unclipped regime) is central to the analysis but not well justified; a short discussion of why it is valid to analyze the optimization dynamics separately in the clipped and unclipped phases would improve the theoretical clarity of the paper -- it should noted that in a single batch of samples, some gradient would be clipped and some would not but we will only have one update at the end. The analysis in the main theorem is confusing.\n\n4.The paper also lacks comparison with SOTA DP-SGD benchmark, e.g., \"unlocking high-accuracy differentially private image classification through scale\", \"a theory to instruct differentially-private learning via clipping bias reduction\", and \"differentially private image classification\nby learning priors from random processes\".", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents the first SDE-based theoretical framework for analyzing differentially private optimizers, bridging discrete training dynamics with continuous-time analysis. It provides clear scaling laws for DP-SGD $O(1/\\epsilon^2)$ and DP-SignSGD $O(1/\\epsilon)$, offering strong insight into how adaptivity interacts with privacy noise. Theoretical predictions are thoroughly validated by experiments across multiple datasets. The work also gives practical guidance for tuning and transferring DP optimizers under different privacy budgets.", "weaknesses": "While the SDE framework is well motivated, the paper does not clearly justify why the continuous approximation is valid or which small terms are neglected. A short discussion explaining that the discretization error between the algorithm and its SDE counterpart can be ignored would make this assumption more convincing. Additionally, although the authors argue that higher-order SDE approximations are unnecessary, a simple analysis or experiment illustrating how the first-order approximation deviates from the discrete dynamics would strengthen the theoretical rigor. Clarifying these points would improve both the transparency and completeness of the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)