DPQuant: Efficient and Private Model Training via Dynamic Quantization Scheduling
Authors: Yubo Gao
Renbo Tu
Gennady Pekhimenko
Nandita Vijaykumar
Venue: ICLR 2026 Poster (2026)
Keywords: differential privacy
quantization
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Differentially-Private SGD (DP-SGD) and its adaptive variant DP-Adam are powerful techniques to protect user privacy when using sensitive data to train neural networks. During training, converting model weights and activations into low-precision formats, i.e., quantization, can drastically reduce training times, energy consumption, and cost, and is thus a widely used technique. In this work, we demonstrate for the first time that quantization causes significantly higher accuracy degradation in DP training compared to regular SGD. We observe that this is caused by noise injection, which amplifies quantization variance, leading to disproportionately large accuracy degradation.
To address this challenge, we present DPQuant, a dynamic quantization framework that adaptively selects a changing subset of layers to quantize at each epoch. Our method combines two key ideas that effectively reduce quantization variance: (i) probabilistic sampling that rotates which layers are quantized every epoch, and (ii) loss-aware layer prioritization, which uses a differentially private loss sensitivity estimator to identify layers that can be quantized with minimal impact on model quality. This estimator consumes a negligible fraction of the overall privacy budget, preserving DP guarantees. Empirical evaluations on ResNet18, ResNet50, and DenseNet121 across a range of datasets demonstrate that DPQuant consistently outperforms static quantization baselines, achieving near Pareto-optimal accuracy-compute trade-offs and up to $2.21\times$ theoretical throughput improvements on low‑precision hardware, with less than 2% drop in validation accuracy. We further show that our framework extends to DP-Adam with similar gains.

Published as a conference paper at ICLR 2026
DPQUANT: EFFICIENT AND DIFFERENTIALLY-PRIVATT MODEL TRAINING VIA DYNAMIC QUANTIZATION SCHEDULING
Yubo Gao University of Toronto Toronto, Ontario
Renbo
University of Toronto Toronto, Ontario renbo. www.:ouummaluooooooo.o
Gennady Pekhimenko
Nandita Vijaykumar
University of Toronto
University of Toronto
Toronto, Ontario pekhimenko@cs. wtw.eoeocc.ttoooo..edoeoo
Toronto, Ontario
ABSTRACT
Differentially-Priiaal SGD (DP-SGD) and its adaptive variant DP-Adam are pow- erful techniques to protect user privacy when using sensitive data to train neural networks. During training, converting model weights and activations into low- precision formats, i.., quantization, can drastically reduce training times, energy consumption, and cost, and is thus widely used technique. In this work, we demonstrate for the first time that quantization causes significantly higher accuracy degradation in DP training compared regular SGD. We observe that this is caused by noise injection, which amplifies quantization variance, leading to dis- proportionately large accuracy degradation. To address this challenge, we present DPQUANT, a dynamic quantization framework that adaptively selects changing subset of layers quantize at each epoch. Our method combines two key ideas that effectively reduce quantization variance: (i) probabilistic sampling that rotates which layers are quantized every epoch, and (ii) loss-aware layer prioritization, which uses differentially private loss sensitivity estimator to identify layers that can be quantized with minimal impact on model quality. This estimator consumes negligible fraction of the overall privacy budget, preserving DP guarantees. Em- pirical evaluations on ResNet18, ResNet50, and DenseNet121 across a range datasets demonstrate that DPQUANT consistently outperforms static quantization baselines, achieving near Pareto- coretal accuracy compuee trade-offs and up to 2.21 x theoretical throughput improvements low-precision hardware, with less than 2% drop in validation accuracy. We further show that our framework extends to DP-Adam with similar gains.
INTRODUCTION
Differentially Private Stochastic Gradient Descent (DP-SGD) (Abadi al. 2016) enables training neural networks on sensitive data while providing formal privacy guarantees. To improve the efficiency of such training on modern hardware, the use of low-precision arithmetic and data formats, i.e.. quantization, has gained widespread interest (Gholami et al., 2021 Jacob et al. 2017). Quantization can significantly reduce the amount of computation and memory required, thus reducing the latencies, cost, and energy consumption during training and inference, often with little to no loss in model accuracy (Micikevicius et al. 2022). These benefits are especially important in resource-constraine settings such federated learning with edge devices, where support for full full-precision arithmetic is limited and compute budgets are constrained.
Modern accelerators, ranging from datacenter GPUs to mobile NPUs, are rapidly adopting ultra-low precision formats such as FP8, INT4, or FP4. NVIDIA Blackwell (NVIDIA Corporation 2024) provides 4x throughput for FP4 matmuls compared to FP16; AMD Instinct GPUs, Google TPUs

---

Published as a conference paper at ICLR 2026
and AWS Trainium support FP8 (Advanced Micro Devices, Inc. 2023
2025 Vahdat 2025
Amazon
Web Services 2025), and Qualcomm Hexagon supports INT4/INTI '8 (Qualcomm Technologies. Inc. 2024). Leveraging these compute capabilities in model training would enable significant performance and scalability improvements.
In this work, observe that applying low-precision quantization directly to DP training often leads to significant accuracy degradation, as severe as a 40% drop. While non-DP training is typically robust quantization, the gradient clipping and noise addition steps in DP-SGD/DP-Adam interact poorly with low-precision arithmetic leading to poor convergence as explained in Section I
Our goal is develop automatic mechanism effectively quantize DP training, while minimizing the impact on model accuracy and the privacy budget. We observe that quantizing only subset of layers and selectively varying this subset every epoch can preserve most of the efficiency gains from quantization while maintaining model accuracy. DPQUANT uses two core techniques that are implemented in a differentially private framework for dynamic quantization scheduling:
1. Probabilistic layer sampling, which rotates which layers quantized every epoch to dis- tribute quantization variance across the network, decreasing overall quantization variance;
Loss-aware prioritization, which uses a loss sensitivity estimator to selectively quantize layers that have minimal impact on model accuracy.
Specifically, in this work we make the following contributions:
1. We are the first to demonstrate and explain the significant accuracy degradation that arises when employing satra-low-precsiion (FP4/INT4) quantization with DP training (DP-SGD and Dd-Ad,,), compared to the mild impact observed in non non-private SGD and 16-bit DP training.
2. We introduce DPQUANT, a differentially private lightweight mechanism that minimizes quantization-induced loss by (a) probabilistically sampling which layers to quantize every epoch and (b) prioritizing layers with lower sensitivity- incurring only negligible cost to the overall privacy budget.
3. We demonstrate that DPQUANT achieves near Pareto-optimal accuracy tradeoff's across range of compute and privacy budgets, outperforming static (fixed-layer) quantization baselines.
2 RELATED WORKS
Post-training quantization (PTQ) (Banner et al., 2019: Jacob et al.) 2017: Nagel et al. 2021 aims to accelerate inference through low-precision computations. A neural network is first trained in full-precision and then its weights are quantized. The conversion to quantized formats typically involves a small calibration dataset (Hubara et al. [022] Nagel al., 2019) allocate quantization bit-widths in different parts of the model and to perform bias correction. PTQ methods are orthogonal to this work since they do optimize training.
Quantization-awan training (QAT) (Krishnamoorthi 2018) ameliorates the aforementioned accu- racy loss by training in lower precision. However, QAT requires additional overhead such hardware simulation Wang et al., 2019), sensitivity estimation (Dong et al. 2019)), and quantizer tuning (Esser
al. 2020 (akr et al. (2022,, which typically cancels out raw bit-width speedups, making QAT
suited for accelerating inference but not training (Chen et al. 2024).
Gradient compression (Lin al. 2020; Alistarh al.) 2017: Alimohammadi et al. 2023 Wen
et al. 2017 Shi et al., 2020) reduces communication costs by compressing gradients in distributed settings, either through sparsification (Stich al.] 2018 Yu et al. (017)) or low-rank approximation (Vogels et al. 2019; Idelbayev and Carreira- Carreira-Perpinán 2020). Notably, Youn et al. (2023) combine quantization and the noising mechanism to achieve differential privacy while reducing communication. However, compression does not lower the arithmetic cost of training. In addition, these methods often rely on assumptions such as full gradient availability or error feedback accumulation (Karimireddy et al. 2019), which are difficult to satisfy under DP constraints.
Mixed Mixed-precision training (Choi et al., 2018 Zhou et al. 2018 Sun et al. 2020 Chmiel et al. 2024
Micikevicius et al. 2022) aims to reduce training cost by operating nower-precision data types (e.g..
2

---

Published as a conference paper at ICLR 2026
FP16, BF16, FP8, and FP4). Prior work has shown that 16-bit miixed-precision training (FP16/BFI1) compatible with DP-SGD with little accuracy loss (Li et al. 2022 Bu et al. (023). However, pushing ultra-low precision (FP4/INT4) under DP remains challenging: show in Section that that DP noise amplifies quantization variance at these bit-widths, causing significant accuracy degradation that does not arise at 16-bit precision.
Quantization as a privacy mechanism. Several works use quantization to provide or strengthen
DP guarantees in federated decentralized settings (Kang et 2024 Kim al. 2022, Wang and Basar 2022 Xiong et al. 2016). In these approaches, quantization is applied after gradients are computed in full precision, so there no effect on subsequent forward/backward passes. DPQUANT instead quantizes activations and weights during training to reduce arithmetic cost.
Improved DP mechanisms. complementary line of work improves the DP algorithm itself for better utility at a given whether through modified optimizers (Wang et al. 2024] Wei et al. 2022 Park al. 2023; Denisov et al., 2023) or by reducing the trainable parameter space via pruning or layer selection [Adamczewski and Park (203). These methods are orthogonal to DPQUANT, which keeps the full dense model and instead optimizes how much computation can run in low precision.
Gradient clipping optimizations for DP-SGD. (Li et al. 2022 Bu et al. 2022 Lee and Kifer 2020; SSbbamani et al.) 202T) optimize the per-sample gradient clipping for DP-SGD by eliminating redundant computation and increasing vectorized computation, leading to better hardware utilization. These works are orthogonal to DPQUANT.
PRELIMINARIES
3.1 DIFFERENTIALLY PRIVVTTE DNN TRAINING
We first recall the standard definition of differential privacy:
Definition 1 (Differential Privacy, (Dwork and Roth 2014)). randomized algorithm A satisfies (E, 8) diffferential privacy for all adjacent datasets D, D' differing on at most one example, and for all measurable sets S in the output space,
Pr(A(D) E e Pr(A(D') S)+
Definition 2 (DP-SGD, (Abadi al., 2016)). Difserentially Private Stochastic Gradient Descent (DP-SGD) is a variant of SGD that satisfies (e, d-fffferrentaa privacy by clipping and perturbing per-example gradients. At each iteration the update rule is:
Oc+1 O.
Σ
clipc( +.)00,,0,))+++000000))) IEB
B
where B is a minibatch of training examples, clipe(-) scales the gradient to have &g norm at most C, and (O, c2C2 is Gaussian noise added ensure privacy.
3.2 QUANTIZATION AND MIXED PRECISION TRAINING
Modern hardware accelerators such as NVIDIA GPUs, Google TPUs, and Qualcomm Hexagon NPUs provide dedicated support for low-precision arithmetic, including fp16, bfloat16, and increasingly lower bitwidth formats like fp8, fp6, and fp4. These formats enable faster computation by reducing arithmetic complexity, memory footprint, and data transfer costs.
While prior work et al. 2024) has demonstrated that full training in low-bit formats (e.g.. fp4) can often retain accuracy under standard SGD, extending these techniques to differentially private training remains challenging. The clipping and noise injection steps in DP-SGD amplify quantization errors and increase gradient variance, making DP training more sensitive to precision loss. Fully quantized DP-SGD thus often results in severe degradation unless carefully tuned.
4 DEGRADATION OF DP TRAINING FROM QUANTIZATION
Figure I presents a case study of ResNet18 trained on GTSRB, where the forward and backward convolution operators quantized to FP4 evaluate the effects training. Figure Ta shows the
3

---

Published as a conference paper at ICLR 2026
spoch
aecumnay diffemous
0.15
epoch
spoch
Unmiied Goalio Norm
0.5
*poch
50
0.05
60
0.00
0.0
10
SG0
SGD+ Note
DP-SOD
mamber of quantized lagaer
Togs (ng/lee)d/nnoiee
(a) Accuracy loss due to quantiza-
(b) Distribution of gradient/noise ra- tios in convl of ResNet18 Gradi-
(c) Distribution of gradient norms between SGD. SGD and only noise injection, and full DP-SGD
tion between DP and non-DP SGD
ent mostly dominated by noise
Figure I: Comparing quantized SGD vs DP-SGD ResNet18 training on the German Traffic Sign
Recognition Benchmark (GTSRB) (Stallkamp et al. 201T) dataset
accuracy loss compared to the non-quantized baseline for different degrees of quantization (in terms of the number layers quantized), and the error bars represent the results when different subsets of layers are chosen for quantization, both for DP and non-DP training. For the non-DP SGD baseline, fully-quan qully-quantized training results in only a modest accuracy drop around 1 1%. In contrast, DP-SGD experiences a much greater degradation, up to 5%. Furthermore, the variance in performance due to different layers being quantized is substantially higher under DP-SGD. We observe similar trends in other neural networks and datasets (included in Appendix A.6).
We hypothesize that the increased sensitivity to quantization can be attributed to noise injection in DP-SGD as follows. In iteration in the DP-SGD training, the gradients g: are first clipped to obtain E. where ||B:|2 12 S C' (section 3.1 Next, noise n, ~ ~N(00, c²C**1 "N(0,."***
is sampled, and finally
the weights are updated using the sum of the noise and clipped gradient:
W++1 W. (B: +
We assume the noise scale is (0.5, 10) (Abadi et al. 2016 De al. 2022). in line with common configurations reported
DP-SGD literature. Since the standard deviation the injected noise n, is equal to the 2-norm of the clipped gradients, the co-norm of the noise n, (i.e. its largest component) is roughly on the same order as |gg|e-- Since in higher dimensions || l2 > || due to the 2-norm growing much faster than the oc-norm, combining we have:
||m:|>> R ||
||
This relation is also demonstrated empirically in Figure lb. where on average the magnitude of the clipped gradient elements of g 2 times smaller than that of the injected noise
The weight update (Equation D with noisy gradient updates amplify the norms of raw gradients in subsequent iterations. To show this, we first write the weight update as:
Aw; W w, (g n:)
The weight update Aw, 2 1j n. due to being much larger in norm than g., and thus:
Awo x |n/loc = ( 11))
(4)
where the asymptotic equality holds due to Equation loss with respect to the gradients:
Assuming L-Lipschitz-smoothness of the
||g+1 -glloo S L L| wello L Lwwoooloo
(5)
We now show that the co-norm of the raw gradients (i.e. before clipping and noising) of the next iteration is bounded by || using the inverse triangle inequality with Equation 4 and
1/
||g+1 lo
/ O(1gg(12)
(6)
Equation 6 shows that elements of the raw gradients in the next iteration are bounded by the much larger ( II2) rather than the usual O(|gg in normal SGD. As a result, we expect elements of the raw gradients in DP-SGD be larger in magnitude than in non-DP training.
4

---

Published as a conference paper at ICLR 2026
Notably, the batch size has negligible effect norms, though larger batch size leads to smaller variance of the stochastic gradients. Therefore, we omit it in this analysis; we include further discussions and empirical evaluations of this in appendix A.1
To show this empirically, we plot the norms of intermediate gradients under both SGD and DP-SGD using the same hyperparameters in Figure Tc where the DP-SGD intermediate raw gradients are 2x larger in the average and worst case. This phenomenon has also been observed in prior work (Du al. 2022), showing even larger gap than what we observe in gradient norms later in training.
We now demonstrate that the much larger raw gradients under DP-SGD result in much higher quantization variance.
Proposition 1. Let q R" - R" be unbiased (i.e. Elg(x)) = x) and scale invariant (i.e. ((xx) = xx())) quantizer. Assume q(x) quantizes values onto some finite grid. Let be sampled from an absolutely continuous distribution. Then the quantizer variance Var(g(x)) ([|[x
Proof: See Appendix
A.10
Using Prop.
we can more precisely express the variance of the quantization as follows:
(under DP-SGD)
Var(a(g(+1) gc-1) 0
(00g(:g1)))) giggg++llll: (12x)) O (1g.
(under SGD)
Var(q(g(+1) gc-1) (1lg: (1
The quantization variance above is in addition the existing variance of the stochastic gradients, as well as noise injected by DP-SGD. In higher dimensions, l|g/ll2 >> ||g/||s. quantization contributes much more variance to the gradients, hence leads to slower and less reliable convergence (Johnson and Zhang] (013) and accuracy degradation. To address this challenge, we aim to reduce the quantization-induced variance.
Extension to OP-Adam/DP-Adamyss
The above argument for DP-SGD relies on DP noise domi- nating the raw gradients, which leads to disproportionate quantization variance. In DP-Adam, noise is added to the clipped gradients first, and Adam's 's preconditioner then rescales the noisy gradients coordinate-wise Since the preconditioner applies the same scaling to both signal and noise, preserves the relative scale between them, quantized DP-Adam incurs the same quantization overhead as DP-SGD.
We reason about this more directly by considering the per-coordinate iigallto-noin ratio. In DP-Adam, each noisy gradient coordinate is 9: = gi + n4s where gi is the true gradient and n, ~> (0, o²C The preconditioner divides by running second-moment estimate, stationarity the preconditioned coordinate is μą (gi + ms/) v of c²C² with Var(u: S 1.
In the strong DP regime (where oC >> ||
l >
(gi)). the mean (i.e. signal) is:
lgsl
I9d 1 oC σ
E[u,]]
gf C
where the last inequality is due 1gs| < ||Bl22 S C. We define the per-coordinate SNR in the preconditioned space, which is directly comparable to non-preconditiooee DP-SGD:
(Eju,))² SNR, = Var(u:)
~ (02C2
C2
Thus the SNR in DP-Adam matches that of DP-SGD. Therefore, DP turns each preconditioned coordinate into unit-variance noise with very small mean. A fixed-precision quantizer u, then adds zero-mean noise whose variance does not depend on this small mean, so the signal-to-quaniizztionn noise ratio behaves like of (o2c²) and deteriorates privacy noise increases.
DPQUANT: OUR PROPOSED SOLUTION
5.1 PART I: PROBABILISTIC LAYER SAMPLING
Each time we perform randomized and unbiased quantization on a layer, we introduce additional variance to its gradient updates. While this added variance might be acceptable in standard training,
5

---

Published as a conference paper at ICLR 2026
it results in significant performance degradation under DP-SGD, where gradient stability is already challenged by injected noise.
Suppose layer is quantized with probability p. let Sip denote its full precision gradients and Squam to be its gradients computed under quantization. By Section quantization incurs additional variance, hence Var(gip) Var Var(guum)- The expected gradient variance is:
4
E (Var(g)) = (1 - p) Var(gip) + p Var (gguan) S
Var(gguam)
From this it follows that whenever < 1- that is, when only a subset of layers quantized each epoch - the average quantization-indueed variance is strictly lower than in full quantization. Furthermore, by rotating which layers are quantized every epoch, single layer repeat- edly incurs the full quantization variance, and hence their expected variance remains smaller than Var Var(gguam))
Analysis
Quantization Policy Statistics
Baseline
B2
pn
Measure
Loss
Compute Difference
5.2 PART II:
LAYER PRIORITIZATION
Not all layers contribute equally to model performance. Intuitively, prefer retain higher precision in layers that have a greater impact on the loss or accuracy. Given constrained quantization budget, our goal is prioritize quantization in lower-impas layers, thereby minimizing the overall loss in model quality.
Randomization Privatization
Update statistics
Training
Query optimal policy
We define a quantization policy p to be the set of layers to compute under quantization. We define R(p) as the expected loss increuse incurred by applying quantization policy p instead of full precision:
Quantized sGo
R(p) = Ep (c(,((D)
(M)ps)(D))]
Figure DPQUANT system overview
Our goal is to find policies that minimally increase loss (or equivalently, a policy p with small R(p)). We note that key challenge in evaluating R(p) for given policy p is that the expectation taken over the full private dataset D. This makes direct computation both expensive and incompatible with tight privacy guarantees. To address this, we instead estimate R(p) by subsampling D and running limited number of DP-SGD iterations under policy to obtain a proxy loss, then the same training iterations are done obtain the baseline full-precision loss, and the difference is used an empirical estimate of the loss impact.
Since this quantity is computed on the private training dataset D, any estimation of R(-) must be performed in differentially private manner, and therefore consumes part of the overall privacy budget. Any computation using the private data incurs privacy cost that must be accounted for to ensure that the privacy budget is not exceeded. We outline how DPQUANT privatizes and accounts for loss measurement in Section 5.4
5.3 DYNAMIC LAYER SELECTION FOR QUANTIZED DP TRAINING
Building on the insights from the previous sections, we design dynamic layer selection strategy for quantized DP training that combines: (i) probabilistic sampling of quantized layers to reduce variance (Section 5.1) and (ii) loss-aware prioritization to preserve performance by avoiding quantization of high-impact layers (Section 3.22)
Let R(4,) denote the estimated quantization loss impact of layer i. We define the probability of selecting layer for quantization as:
exp( (R()))
for i I,...,7..
== cxp(-
where B > 0 is scaling parameter that controls how strongly we prioritize low-impact layers.
As outlined in Figure 2 to quantize out of n layers at each epoch, we sample a subset without replacement according to the distribution {m:). This allows to adaptively choose the least sensitive

---

Published as a conference paper at ICLR 2026
layers for quantization, while still randomly rotating layers with similar loss impact to minimize variance over time. This selection procedure detailed Appendix(A.15) DPQUANT provides a set of tunable parameters that govern the frequency of the analysis, well as other privacy parameters.
5.4 PRIVACY ACCOUNTING
Our method begins by measuring loss differences each user's private dataset. Specifically, we compute L( C(M(D) which requires inspecting raw data and inherently risks exposing sensitive information if released directly. Without these loss-difference measurements compromises the privacy guarantee DP-SGD provides.
measures, simply publishing the
Definition (Sampled Gaussian Mechanism (SGM), (Mironov et al. 2019). Let I be a function that maps subsets of a dataset S to R The Sampled Gaussian Mechanism, denoted SG is defined with sampling rate 0 < q < 1 and noise parameter a > 0 as:
(S) = 5 (1x E S x is independently sampled with probability q}) + (0,00000))
where each element in S is independently included with probability a, and N/ of11)) denotes d-dimensional isotropic Gaussian noise with variance o² per coordinate.
To protect privacy. we frame this loss computation as a Sampled Gaussian Mechanism (SGM): draw a random subsample of D, clip the resulting loss value to bound sensitivity, and then add Gaussian noise of scale o. These operations correspond step 3 of Algorithm
Algorithm 1 COMPUTELOSSIMPACA
1: Input: P (policies), B (batches), R (iterations), (decay), C (norm), (noise)
2: Let po be the baseline policy (no quantization)
3: Initialize map for average losses, E
4: for each p E PU [Po} do
Compute avg. loss for baseline and all policies
5: total loss 0
6: for = 1 to A do
7:
RESTOREMODELO
8: for each (x, y) E B do
9:
With policy p. run DPSGD- PPSGD-UPDATE(MN, loss (x). v))
10: end for
total_loss +- tosal_loss + [B Elayen (oss(M(())
12: end for
13:
Elp] <- total Loaa__oss/RRR
14: end for
R(p)
Elp] Eppo] for all p P
Compute loss differences from baseline
16: R + [ R(px]]
17: R + R min 1,, | + +N(0,02c²1)
b Privatize differences
18: UPDATEPRIVACY rate [B/ steps=11 1, noise_scale o)
19: for each p E P do
20: L[p] f (1 1--a] a) Lip] + R(p
Update Policy EMA
21: end for
22: return L
Proposition 2. Algorithm 1 is a Sampled Gaussian Mechanism (SGM) with sample rate q = |B/ D and noise scale ameanure Proof: See Appendix A.77
To account for the privacy cost we incur by performing the analysis in Algorithm we rely on Opacus's privacy accountants (Yousefpour et al.) 2022). This is for two reasons: First, these accountants measure the cumulative privacy loss of SGMs (Makni al. 2025), where by Prop. 2 we can reuse its implementation. Second, by leveraging the advanced composition theorem (Abadi et al. 2016), obtain a much tighter upper bound the total privacy expenditure incurred by both the SPDSGD training process and any subsequent analyses performed under the same privacy budget. We explain this in more detail in Appendix A.14
7

---

Published as a conference paper at ICLR 2026
portion of privary budgees
<pelvecy consuned
8
0.15
0.0
no analysis
0.75
6-
0.5
0.6
4
0.75
1.0
2
0.00
epoch
beseline privacy spent
(a) privacy consumption of analysis training
(b) Fraction privacy spent on analysis
Figure 3: Privacy cost of analysis for ResNet18/GTSRB; performing analysis every 2 epochs
In Figure 3 we report the cumulative privacy loss from both the analysis and training components across various configurations. Our results empirically demonstrate that the privacy cost of analysis is negligible compared to training, and does not meaningfully affect the quality of the resulting model.
6 EVALUATION
Models and Datasets. We evaluate our approach commonly used neural networks for differentially private training (Jagielski et al. 2020 De et al. (022):: ResNet18 8 (He et al. 2015)), 5e and DenseNet121 (Huang et al. 2018) from the torchvision (maintainers and contributors 2016 library. We also test BERT (Devlin et al. 2019), trained with AP-Adam W. We additionally evaluate with DP-Adam in Appendix A.5] These models are trained on the Extended MNIST (Cohen et al.) 2017), German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp et al. 2011), CIFAR- 10 (Krizhevsky) 2009) and SNLI (Bowman et al., 2015) datasets.
Implementation. DPQUANT is implemented on top of Opacus (Yousetpour et al. 2022). DP training framework which provides Poisson sampling, gradient clipping, and noising. The DPQUANT parameters can be found in Appendix A.2
Low Precision Format. For low precision computations, we used LUQ-FP4 (Chmiel et al. 2024) format, the highest-performing 4-bit quantization format. LUQ-FP4 uses 4-bit representation of floating point numbers, consisting of sign and 3 exponent bits. In Appendix A.9 we evaluate DPQUANT other low-precision formats including FP8 and 4-bit uniform quantization.
6.1 QUANTIZATION-QUALL TRADE-OFF
Quantizing more layers proportionally increases the speed of training. However, also increases the accuracy degradation in DP-SGD training. Thus, there is a speed-accuracy trade-off depending the number of layers quantized. For a given number of quantized layers, the resulting model accuracy can significantly vary depending on which layers are quantized any given epoch. DPQUANT aims to automatically identify the subset of layers for each epoch that provides the best accuracy, assuming certain number of layers quantized. We refer the desired number of quantized layers as "computational budget" because it determines the speed and compute resources needed.
In Figure we sample 50 random subsets of layers to execute in fp4. We plot the empirical Pareto front using these sampled measurements, in addition to the resulting validation accuracy when using DPQUANT's scheduling technique for given computational budget.
We make two observations. First, we note that randomly selecting the quantized layers can lead to significant loss in accuracy, as much as 40%. Second, DPQUANT generates scheduling configurations that provide validation accuracy close to the Pareto-front for all evaluated networks and datasets.
6.2 SENSITIVITY TO PRIVACY BUDGET
We compared method to the baseline for two privacy budgets 4 and 8. In Table I report the validation accuracy for different privacy budgets. For 8/55 we obtain these values
Batch size decreased improve convergence under
8

---

Published as a conference paper at ICLR 2026
ResNer18 EMNIST Serotch
ResNet18 CIFARIO Ima Weights
ResNot18 GTSRR ImagoiNot Weights
masiom
Validation Areurecy
85
ouns
70
80
76
15
20
ResNet50 GTSRD ImageNNtt Weights
DeaseNeti21 CIFARIO ImageNNt Weights
DeaseNeri21 GTSRD ImageNet Weights
Valilation Arvirayy
8
60
50
Number of Quantized Leyers
Number of Quantized Layers
Number of Quantized Layers
Figure 4: Comparing policies generated by DPQUANT the speed-accuracy Pareto front
Model
Dataset
Percent Quantized
==4
Buseline
E
Ours
Buseline
Ours
0.5
81.27
= 1.29
3.14
82.16
3.04
EMNIST
0.75
80,51
0.37
80.09
3.04
0.9
78.82
0.30
3.01
79.03
3.04
0.5
42.34
5.53
4.01
49.09
3.99
69.06
5.63
8.01
76.75
7.99
R8sNet18
GTSRB
0.75
39.98
= 3.99
4.01
42.62
3.96
63.62
+ 5.59
8.01
70.07
7.99
0.9
37.94
2.23
39.48
3.99
57.49
+ 4.46
67.67
7.99
0.5
64.37
1.42
4.06
65.39
3.94
69.26
1.46
7.12
70.51
7.17
CIFAR-10
0.75
62.17
0.61
4.06
63.57
3.94
67.80
0.81
7.12
69.84
7.17
0.9
61,09
1.66
4.06
61.22
3.94
67.21
= 1.24
7.12
68.68
7.17
0.5
38.76
= 8.16
4.01
42.11
3.99
75.99
+ 7.33
8.01
80.23
7.99
ResNet50
GTSRB
0.75
29.48
= 4.72
4.01
33.67
3,99
58.13
+ 8.50
8.01
69.03
7,99
24.78
2.67
29.00
47,40
7.23
59.87
55.38
3.97
5.42
71.05
GTSRR
0.75
44.60
5.06
4.06
47.36
3.97
56.14
7.57
8.01
63.30
7.93
0.9
40.52
+ 2.83
4.06
44.15
3.97
$1.06
+ 5.41
8.01
52.60
7.93
DenseNet121
0.5
59.22
+ 1.15
4.03
61.08
3.97
67.96
+ 0.93
7.12
68.96
7.28
CIFAR- 10!
0.75
56.43
1.72
4.03
60.31
3.97
64.81
+ 1.71
7.12
66.48
7.28
0.9
55.18
1.38
58.89
3.97
63.03
1.69
7.12
65.13
7.28
0.5
62.54
4.54
7.48
67.80
7.48
BERT
SNLI
0.75
52.04 + 3.95
7.48
63.61
7.48
Table 1: Model quality across datasets and privacy levels.
by truncating the training at the respective privacy budgets (i.c. without additional hyperparameter tuning), and select baseline data points with larger than ours wherever possible.
In most cases, DPQUANT outperforms the baseline performance by at least standard deviation whilst not exceeding the privacy budget. In particular, despite the privacy cost of analysis being more dominant during the = case, DPQUANT produces near-optimal quantization schedules, demonstrating its robustness with respect
We have also evaluated DPQUANT on extremely small privacy budgets (e.g. & == 1). Appendix A.3 we show that DPQUANT still demonstrates the same benefits under this setting.
6.3 ABLATION STUDY
To better understand the contributions of the two approaches, compared approach (probabilistic layer sampling loss-aware layer prioritization) with probabilistic layer sampling (PLS) alone. In Figure 5| we observe that PIS consistently performs better than the baseline where the quantized

---

Published as a conference paper at ICLR 2026
layers selected statically. However, there is still large gap between PLS and the best best-performing layer selections, suggesting that some crucial layers are consistently being subjected to quantization which significantly degrades the quality of trained models.
When PLS is combined with loss-aware layer prioritization, the layers crucial to model training are left in full precision, even when most of the layers quantized. The benefits of prioritization begins to surface as the proportion of quantized layers increase, as the critical layers have a larger probability of being quantized in the randomized baselines. Furthermore, observe that the best training outcomes are achieved by combining both approaches. We include more details in Appendix A.7
analysia
train fy16
oveetseou
train
Vadidatiion Accurasy
80
1.87x
30000
Noraalzed Ieeraioos
2:21xx
75
25000
70
15000
1.88x
L-P8&
65
PLS
baseline
10000
10
12
14
16
18
20
5000
Number of Quantized Layens
18 EMNIST
ResNor18
ResNer18 GTSRB
ResNet50 GTSRB
122eeeee2ee GTSRD
CIFAR10
CIFAR1O
Figure 5: Ablation study, PLS: prob- abilistic layer selection, LLP: loss- aware layer prioritization
Figure 6: Theoretical speedups for DPQUANT assuming 90% of the layers are quantized.
6.4 THEORETICAL SPEEDUP
As hardware with support for FP4 MatMuls and Conv2D (e.g.. NVIDIA Blackwell) are not yet widely available, we are unable evaluate the speed benefits of quantization with DPQUANT. Instead, we use estimates from prior work, along with performance statistics published by NVIDIA (NVIDIA)
estimate that Corporation 2024) to estimate speedups. Based profiling existing hardware,
FP4 can provide a 4x speedup over the FP16 baseline. Separately, prior works (Sun et al. 2020 7.3x speedup when using FP4 supported hardware. To remain conservative, we use the lower bound (4x) in our estimates. We assume matrix multiplications, convolutions, and element-wise operations can be accelerated 4x, and characterize the total runtime as a linear compute cost model:
Choi al. 2018 Abdolrashidi al. 2021) report
Tours aaaalyss (1 p + p/ p/4) (Tiriin irain baseline Toverhead) + Toverhead
where T amalysis the time taken by algorithm 1 and Tuverbead captures the time taken by operations that do not have performance benefits from low precision (details in appendix [A.13).
We show our speedups in Figure Quantized training with DPQUANT 1.75x to 2.21x faster than the fp16 6 baseline. In particular, the loss-aware prioritization mechanism in DPQUANT incurs minimal runtime overhead, which is crucial preserve the performance gains of fp4 computation.
CONCLUSION
In this paper, we introduce DPQUANT, mechanism for efficient quantized DP training. We observe that existing quantized training techniques significantly degrade the accuracy of models trained with DP-SGD and Ad-Adam, and provide a detailed analysis demonstrating the amplified quantization eπror. To address this challenge, DPQUANT employs techniques to dynamically select layers to quantize such that the impact of quantization on model accuracy is minimized. DPQUANT itself is a differentially private mechanism that incurs only small privacy cost. We empirically demonstrate that DPQUANT achieves near-optimal compute-to-accuracy tradeoffs during quantized training, generalizes to different models, datasets and privacy budgets, and can provide up 2.21 speedup while minimally impacting accuracy. DPQUANT enables efficient and practical differentially-prii privvtee training for both centralized and distributed training deployments.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, lyya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS' ACM. October 2016. doi: 29114///27674..29788111 (ttt:/t://dxx.o
orq/10.g///oo 0T..0..14//297749999 2978318
AmirAli Abdolrashidi, Lisa Wang, Shivani Agrawal, Jonathan Malmaud, Oleg Rybakov, Chas Leichner, and Lukasz Lew. Pareto-optimal quantized resnet is mostly 4-bit. In 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), page 3085-3093. IEEE, June 2021. doi: 10.1 10.11099cyprw.5309888.8 2022.0022.......
URL http: //tp:://xx/ doi org./1/ org/10
1109/CVPRN5 30 9 8. 2 0200211.10 ...21..0.. 00345
Kamil Adamczewski and Mijung Park. Differential privacy meets neural network pruning, 2023.
URL htttts://aaxiiiiis org/ abs/a/b33/333.3 04612
Advanced Micro Devices, Inc. AMD Instinct MI300X Accelerator Data Sheet: Leading-Edge Accelerator Module for Generative AI, Training, and High-Performance
Computing. Technical report, Advanced Micro Devices, Inc., 2023. URL https
//www , amd com wwww//oooeettttoeoooooe http://ow..eonoaaaaocooo/////oaa
data det
data sheetassshet....... Accessed: 2025-05-13.
Advanced Micro Devices, Inc. Data types and precision support. https::/roen:: docs amd
serrecssioo--ii sppeeciionnsupppttttt html March 2025. ROCm Docu- mentation; Accessed: 2025-05-13.
Mohammadreza Alimohammadi, Ilia Markov, Elias Frantar, and Dan Alistarh. L-greco: Layerwise- adaptive gradient compression for efficient and accurate deep learning, 2023. URL https
//arxiv.oor wtt.v/..rr///bbb/211.777
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication- efficient sgd via gradient quantization and encoding, 2017. URL
1610.02133 02132]
Amazon Web Services. AWS Trainium purpose-built for high-performance, cost-efficient Al at scale. Amazon Web Services, 2025. URL https: aws. amazon. com/ai/ Accessed: 2026-03-01.
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment, 2019. /RL/httss///axxx..rr///////
05:23
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and D.ristopher D. M.nning large annotated corpus for learning natural language inference, 2015. URLpttts://arxxv wwapxiii.rrr/aaee Bargg//bs//11 05326
Zhiqi Bu, Jialin Mao, and Shiyun Xu. Scalable and efficient training of large convolutional neural networks with differential privacy, 2022. URL /ttp:t/a///a.....ooor////////2/2
Zhiqi Bu, Justin Chiu, Ruixuan Liu, Sheng Zha, and George Karypis. Zero redundancy distributed learning with differential privacy, 2023. URL htt.aaaaa//arrvv../////////11111
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.1106. 2024.
Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Accurate neural training with 4-bit matrix multiplications at standard formats, 2024. URL ltttps://arxiv
org/abs/ 21 12. 10769
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, 2018. URL htttps://axxii www..orgg...... 1000. 8005.06085

---

Published as a conference paper at ICLR 2026
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: extension of mnist to handwritten letters, 2017.
abs 05373
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlocking high- accuracy differentially private image classification through scale, 2022. URL httttt//a//ax.
org/aba/2204 13650
Sergey Denisov, Brendan McMahan, Keith Rush, Adam Smith, and Abhradeep Guha Thakurta. Improved differrnntaal privacy for sgd via optimal private linear operators on adaptive streams, 2023, wwtp:/ii.oorooooooooooe abw//2/2/22222 08312
Jacob Devlin, Ming- Wein
Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxlvvorg
abs/T810 04805
Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed sixed-precision, 2019. URL httpp::/
abs/1905. 03696
Jian Du, Song Li, Xiangyi Chen, Siheng Chen, and Mingyi Hong. Dynamic differential-privacy preserving sgd, 2022.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations
and Trends in Theoretical Computer 9
9(3 4):111411--447 2014. doi: 10. 1004600/4400000044.
URL https: www.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, CaahinakkmmarrAppussssaaaa and Dharmen- dra S. Modha. Learned step size quantization, 2020. BtLLitttp///axxi.orrgg
1902
08153
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference, 2021. URL https
//arxiv. wwwprr/aab///aa 13630
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https:://arxiii org/abs /1512.0338
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, 2018. URL/attss://arxxy
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference Machine Learning, pages 4466-4475. PMLR, 2021.
Yerlan Idelbayev and Miguel A Carreira-Perpinán. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 1149-8059, 2020.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithneticonlyy inference, 2017. UtLhttpps: htt.ttppp///a.x.....re http///aaww..oo/aas/////aaa/ss// /wt.b/////1...... 05877
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd?, 2020. TRLLittps://axxiv.orr/// wttp:////////...////////// 07709
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive vari- ance reduction. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. U htt.:/tts///prrceeeiiiiiie
neuri ps wwwpsccc/ppaee cw/paper fi files/paper/ Pww.bbbbeepPppp.ppp....
Tianqu Kang, Lumin Liu, Hengtao He, Jun Zhang, S. H. Song. and Khaled B. Letaief. The effect of quantization in federated learning: A rényi differential privacy perspective. In 2024 IEEE International Mediterranean Conference on Communications and Nenworking (MeditCom). IEEE, 2024.
12

---

Published as a conference paper at ICLR 2026
Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages
PMLR, 2019.
Muah Kim, Onur Günlü, and Rafael F. Schaefer. Effects quantization on federated learning with local differential privacy. In GLOBECOM 2022 2022 IEEE Global Communications Conference. IEEE, 2022.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint (rXiv:1806008444 2018.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical re- port, University of Toronto, April 2009. URL httph:tt/s//wet cs, wtt.sttoooooooootottto
Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping, 2020. UR/.accpes//aamxxv.or//////////0 03106
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be
hbtp://abb/211100777.. strong differentially private learners, 2022. URL httlttss///axxii..ommm /ttpp///////.....rr////////10011
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. 2020. URL lnttps://axxx: org htte/abb/17122. 01887
Torch Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. https //giihab.cco//yyttrcc////t/
2016.
Mchdi Makni, Kayhan Behdin, Gabriel Afriat, Zheng Xu, Sergei Vassilvitskii, Natalia Ponomareva, Hussein Hazimeh, and Rahul Mazumder. An optimization framework for differentially private sparse fine-tuning, 2025. URL /ttt:s://axxi..org//bb/bs
12822
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenth- waite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu. Fp8 formats for deep learning, 2022.
URL httth:/s::/: org. abc://bs/220 05433
Ilya Mironov. On significance of the least significant bits for differential privacy. In Proceedings of the 2012 ACM Conference on Computer and Communications Security, CCS 12, page 550-661, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 450316444, doi: 011144/2282296222222622 URL : //doi. http://:/ee.ooooooo/...111////// 23822264
Ilya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism, 2019. URL http://ppps//aaa...c///a////////
Felix Morsbach, Jan Reubold, and Thorsten Strufe. R+raunderstanding hyperparameter effects in dp-sgd, 2024. URL
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1325-13344 2019.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint w:Xh:2206.008295 2021.
NVIDIA Corporation. Nvidia blackwell architecture overrii.eeeeeeeiee.... 2024. URL https:/ resources www.urrees.iiiia................ nvidia.
en http:///eennns..aaaaaaaaaaaaaera
Accessed: 2025-05-
05.
Jinseong Park, Hoki Kim, Yujin Choi, and Jaewook Lee. Differentially private sharpness-awan aware training, 2023. URL htlpttts:: /ttppiiv/////vi////aee 22650
05651
13

---

Published as a conference paper at ICLR 2026
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How to dp-fy ml: A practical guide to machine learning with differential privacy. Journal of Artificial Intelligence Research, 17:1113-1200 July 2023. ISSN 1076-9757. doi: 10.ar13/jair...1 146499. URI http
7/dx.do: www.:/o0..rgg//.1. 16300.66333610 jair 1.4499
Qualcomm Technologies, Inc. Ai hardware cores/accelerators, 2024.
URL
https
wtopiiiirrrreoiiiaaaooooooccccce (ropicss80-63300 (80-63195-1
AI Ll-Taaaware-corre--ccceeeeeeeeee htmll Accessed: 2025-05-05.
Charbel Sakr, Steve Dai, Rangha Venkatesan, Brian Zimmer, William Dally, and Brucek Khailany. Optimal clipping and magnitude-aware differentiation for improved quantization-aware training. In International Conference on Machine Learning, pages 19123-19138 PMLR, 2022.
Shaohuai Shi, Xianhao Zhou, Shutao Song, Xingyao Wang, Zilin Zhu, Xue Huang, Xinan Jiang, Feihu Zhou, Zhenyu Guo, Liqiang Xie, Rui Lan, Xianbin Ouyang, Yan Zhang, Jieqian Wei, Jing Gong, Weiliang Lin, Ping Gao, Peng Meng. Xiaomin Xu, Chenyang Guo, Bo Yang, Zhibo Chen, Yongjian Wu, and Xiaowen Chu. Towards scalable distributed training of deep learning on public cloud clusters, 2020. httpss/:: /tteps://arxxee wtw.v.ivvv. org/ abs/ 201 0. 10458
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In The 2011 International Joint Conference on Neural Networks, pages 1453-1460 1460, 2011. doi: 10.1 10.1109/1ICNN.200 1011.0.33.1110.03399 66033395.
Sebastian U. Stich, Jean- Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory, 2018. 8.
URL http:htts///xxi..cott hbtp://bb///////////// 07599
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath. Enabling fast differentially private sgd via just-in-time compilation and vectorization, 2021.
2010.09063
Xiao Sun, Naigang Wang. Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swa- gath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural In- formation Processing Systems, volume 33, pages 1796-1807. Curran Associates, Inc., nwur 2020. URL /ttttp::/prroceeeeeeet http://nwwn.uuuur.ccceeeeeeeceee cc/papr files//paprr2222/ pdf
File/ / 2tta://011333311cccaaaaa66777771
Amin
Vahdat.
Ironwood:
The first Google TPU for the age of infer- Blog. April 2025.
ence. attp:aaaiooonnddaaaaiaaaaaaaaaaa
Google
URL https://blog google/
tronwood-tpu-age. of coffinnfeennee//
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automatedd quantization with mixed precision, 2019. httttstt/t/xiiiiae www.aaawww..raaaaaaaaaaaaaaaaape wttp://g/////...oo/er/oo//aaa..t
/ltpatttps////ar....orr///////.. 08886
Yonggiang Wang and Tamer Basar. Quantization enabled privacy protection in decentralized stochas- tic optimization, 2022. URL https: wwwpsxiiv...or/oeoooooooo abs 2200 04845
Zihao Wang, Rui Zhu, Dongruo Zhou, Zhikun Zhang, John Mitchell, Haixu Tang, and XiaoFeng Wang. Dpadapter: Improving differentially private deep learning through noise tolerance pre- training, 2024.
org/ abs//433. 02571
Jianxin Wei, Ergute Bao, Xiaokui Xiao, and Yin Yang. Dpis:
An enhanced mechanism for differen- tially private sgd with importance sampling. In Proceedings of the 2022 ACM SIGSAC Confer- ence on Computer and Communicattinn Security, CCS 22, page 2885 2899. ACM, November 2022. doi: 10.1145//548000.5565553 URL http: : dx doi www.wdoo.o.oooo....oooo 1144///4541133358113
3560562
14

---

Published as a conference paper at ICLR 2026
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Temgrad: Ternary gradients to reduce communication in distributed deep learning, 2017. URL https
07878
Sijie Xiong, Anand D. Sarwate, and Narayan B. Mandayam. Randomized requantization with local differential privacy. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016.
Yeojoon Youn, Zihao Hu, Juba Ziani, and Jacob Abernethy. Randomized quantiiaaionn is you need for differential privacy in federated learning, 2023. URL https:://arx...oo/////a///
11913
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in pytorch, 2022. URL https;
//2xivv.rr//ab///00...22 12298
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7370-7379, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients, 2018. URL https:
7/arxiv. srrxv..orc/bbs//065.00000
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX / SUPPLEMENTAL MATERIAL
A.1 EFFECT OF BATCH SIZE
Our analysis of quantization error in DP-SGD is independent of batch size. While larger batches reduce stochastic gradient variance, our argument hinges on the magnitude of the final noised gradient, which remains large regardless of batch size.
1. In DP-SGD, the added noise's scale is proportional to the per-example clipping constant, C, not the batch size.
2. This noise dominates the averaged gradient signal, causing the raw gradients in subsequent steps have significantly larger norms than non-DP training.
3. As shown in Proposition I, quantization variance is proportional to the square of the gradient's s norm (Var(g(z)) @(1== )). Therefore, the larger gradients DP-SGD lead to much higher quantization variance, which destabilizes training and degrades accuracy.
To demonstrate this empirically, we ran the same training job with batch sizes ranging from 1024 to 8192 and measured the numerical range of the weight gradients, similar to that in figure Ic. Across the batch sizes, there is negligible difference the gradient ranges, which confirms our hypothesis.
Table 2: Weight gradient norm range across various batch sizes, showing the negligible impact of batch size on the final gradient magnitudes.
Batch Size
Norm Range Mean
Norm Range Std
1024
0.159
0.137
2048
0.161
0.127
4096
0.158
0.116
8192
0.156
0.119
A.2 EVALUATION SETUP AND PARAMETERS
Parameter
Default
Description
n
60
number of epochs train layers to execute in low precision. test samples for loss measurement. epochs to train before the next measurement.
k
sasmmle
1
Tenterval
2
R
2
repetitions during measurement.
mensure
0.5
Noise scale used during loss-difference privatization. Clipping norm used during loss-difference privatization.
Ceessure
0.01
Table Configurable Hyperparameters of DPQUANT
Selecting DPQUANT parameters in practice. our experiments, have found repetitions = 2 and sampling frequency 1 to be the most optimal. Adopting these recommended defaults, the user needs pick:
Remark:
1. one of k (number of layers to quantize) and the analysis frequency
2. clipping norm used in loss sensitivity analysis
The process of determining clipping norm for analysis is similar to that of finding the clipping threshold C for normal DP-SGD training. We want to pick a value such that the differences between policies are expressed.
16

---

Published as a conference paper at ICLR 2026
A.3 EVALUATION UNDER EXTREME PRIVACY BUDGETS
As shown in Figure 3 the privacy consumption of DPQUANT's analysis accounts for higher fraction of the total privacy near the beginning of training. We wish to evaluate DPQUANT under more strict privacy budgets.
In these cases, both the parameters DP-SGD and DPQUANT need be updated, namely the noise scale a and measurement noise scale measure both need to be increased. Table below shows that DPQUANT achieves optimal accuracy even when
1.
Table 4: Training accuracy of ResNet18 with GTSRB under the strict privacy budget (E = 1)
Baseline
DPQUANT
Count
Accuracy (%)
Accuracy (%)
50%
44.14 +: 4.61
1.05
48.26
0.99
75%
40.13 + 3.65
1.05
43.14
0.99
90%
35.20 + 1.12
1.05
38.66
0.99
A.4 TRAINING HYPERPARAMETERS
A.4.1 IMAGE MODELS
While the learning rate might seem too high for regular SGD training, previous results Morsbach et al. (2024): Ponomareva et al. (2023) have shown that large learning rates are more beneficial for DP-SGD training.
Table 5: Experimental configurations (6 runs)
2
Model Dataset
ResNet18 EMNIST
ResNet18 CIFARIO
ResNet18 GTSRB
ResNet50 GTSRB
DenseNet121 CIFARIO 1
DenseNet121
GTSRB
a
1
1
8
10-5
10-5
10-5
10-5
10-5
10-5
Clipping norm
1
1
Batch size
1024
1024
1024
1024
512
512
Physical batch size
128
128
128
128
128
128
Weights
None
ImageNet
ImageNet
ImageNet
ImageNet SGD
ImageNet
Optimizer
SGD
SGD
SGD
SGD
SGD
Learning rate (Ir)
0.5
0.5
0.5
0.5
0.5
0.5
Epochs
30
60
60
60
60
60
A.4.2 LANGUAGE MODELS
We conducted new NLP experiment using BERT for sequence classification on the Stanford Natural Language Inference (SNLI) corpus. In this task, the model classifies pair of statements (e.g., "Children smiling and waving at camera" and "There are children present") as "entailment," " "contradiction." "neutral."
Due to the high number of parameters BERT, we have followed the tutorial from Opacus and frozen 12 out of 13 BERT layers, and trained the last BERT layer and subsequent classification layers.
We have compared our method (DPQUANT) with a random static baseline (similar to section 6. I). We the same training parameters, trained for single epoch, and used budget.
the total privacy
In these experiments, DPQUANT outperforms the baseline in accuracy. DPQUANT consistently avoids quantizing the last few layers (including the trainable ones) without prior information about the importance and trainability of the layers.
17

---

Published as a conference paper at ICLR 2026
A.5 EVALUATION WITH DP-ADAM
We evaluate DPQUANT using DP-Adam as the optimizer. All hyperparameters match the DP-SGD configuration in Table with the optimizer changed Adam (learning rate 0.01, default [1 0.9,
B2 = 0.999).
Table shows that DPQUANT consistently outperforms the static baseline under DP-Adam while consuming a strictly lower privacy budget, with the largest gains at 75% and 90% quantization.
Table 6: DP-Adam: DPQUANT vs. static random baseline (ResNet18)
Baseline
DPQUANT
Percent Quantized
Dataset
Accuracy (%)
Accuracy (%)
E
0.50
54.18 + 1.48
5.05
55.03
4.97
CIFAR-10
0.75
49.86 + 1.57
5.05
52.40
4.97
0.90
47.36 + 1.36
5.05
51.18
4.97
0.50
7
6.05
75.87
5.91
GTSRB
0.75
58.14 7.64
6.05
72.32
5.91
0.90
50.88 + 5.11
6.05
66.45
5.91
Additionally, the BERT experiment on SNLI (Table
1 uses DP-AdamW as the underlying optimizer.
A.6
ACCURACY DEGRADATION FOR DP-SGD UNDER NAIVE QUANTIZATION
Prior works Sun et al. (020); Chmiel et al. (2024) Micikevicius et al. (2022) have demonstrated
minimal degradation during quantized fp4/8 training compared to the full precision counterpart. We
tabulate their results below:
Table 7: Ultra-Low and LUQ vs. baseline accuracy
Model
Baseline
Ultra-Low Sun et al. (2020)
LUQ Chmiel et al. (2024)
ResNet-18
69.7%
68.27% (-1.43%)
69.09% (-0.61%)
ResNet-50
76.5%
74.01 % (-2.49%)
75.42% (-1.08%)
MobileNet-Vi V2
71.9%
68.85% (-3.05%)
69.55% (-2.35%.
ResNext50
77.6%
N/A
76.02% (-1.58%)
Transformer-baut
27.5 (BLEU)
25.4 (-2.10)
27.17 (-0.33)
As demonstrated in the Figure the performance degradation DP-SGD under quantization much larger.
Table 8: Validation accuracy for DP-SGD training: baseline vs. LUQ-FP4 (all layers quantized)
Model
Dataset
Pretraining
Baseline
LUQ-FP4
A
ResNet-18
EMNIST
None
83.4%
77.8%
-5.6%
ResNet-18
CIFAR-10
ImageNet
71.0%
65.8%
-5.2%
ResNet-18
GTSRB
ImageNet
85.6%
64.0%
-21.6%
ResNet-50
GTSRB
ImageNet
89.8%
49.0%
-40.8%
DenseNet-12
CIFAR-10
ImageNet
67.0%
62.9%
-4.1%
DenseNet-122
GTSRB
ImageNet
82.0%
53.0%
-29.0%
A.7 SENSITIVITY OF TEMPERATURE 3
In our method, the temperature parameter 3 provides a crucial mechanism to balance two comple- mentary strategies:
18

---

Published as a conference paper at ICLR 2026
Deterministic Selection: This approach prioritizes elements based on their loss sensitivity, selecting those that are most impactful.
Randomized Sampling: This approach introduces stochasticity, ensuring diversity and exploration in the selection process.
A low 3 value favors randomized sampling, while high value makes the selection process more deterministic and reliant on loss sensitivity. Below, we tabulate the training accuracy for different value of B, and observe that better training outcomes be obtained by favoring loss-based layer selection while retaining some stochasticity. Namely, it performs strictly better than selection purely based random layer sampling.
Table 9: Model performance across various counts and temperature (3) values
Temperature (3)
Count
0.1
0.22
0.47
1.03
2.24
4.86
10.57
22.99
50.0
10
66.49
67.58
67.53
67.01
70.25
70.37
71.59
70.96
71.67
15
58.47
58.47
60.03
59.07
60.86
65.00
60.54
65.04
63.75
18
51.60
54.08
55.73
53.73
53.86
53.49
60.90
55.45
56.05
A.8 EMA ABLATION
The exponential moving average (EMA) in step Algorithm smooths the per-layer importance scores across measurement iterations so that single noisy batch does not drastically change the layer ranking. Table I0 shows the effect removing EMA on ResNet18 trained on CIFAR-10 10 with DPDSGD. Across all quantization levels, EMA consistently improves accuracy, confirming that smoothing the noisy sensitivity estimates is beneficial.
Table 10: Effect of EMA DPQUANT (ResNet18, CIFAR-10, DP-SGD).
Percent Quantized
With EMA
Without EMA
0.50
56.06
54.01
0.75
51.59
50.07
0.90
49.97
47.22
A.9 EVALUATION ON OTHER QUANTIZERS
To assess the versatility of our method, DPQuant, we evaluated its performance with different numerical precisions and quantization schemes. We conducted two primary experiments: one using 8-bit floating-poit point (fp8e5m2) for training to test different bitwidth, and another using uniform 4-bit quantizer test different quantization strategy.
A.9.1 FP8 QUANTIZATION
Our experiments with FP8 training show that quantized DP-SGD does not suffer significant performance degradation. This minimal performance gap suggests that in nigher-precision settings like FP8, the benefits of more complex techniques like layer subset selection may be less critical. We show the results in table
Table 11: Performance comparison with FP8 training.
Count
Base Ace( %
Base
Our Acc( %
Our
50%
67.56 + 0.47
4.05
67.12
3,93
75%
67.76 + 0.67
4.05
67.65
3,93
90%
67.38 + 0.59
4.05
68.01
3.93
19

---

Published as a conference paper at ICLR 2026
A.9.2 UNIFORM 4- B-BIT QUANTIZATION
Next, evaluated a more aggressive quantization scheme using a uniform FP4 quantizer. In this setup, the value range is discretized into 2 = 16 levels via stochastic rounding. The results reveal more substantial drop in accuracy for our method compared to the baseline. This outcome is consistent with our observations of the LUQ-FP4 quantizer discussed in Section 6.2, highlighting the inherent challenges of applying DP-SGD with very low-bitwidth uniform quantization. We show the results in table 12
Table 12: Performance comparison with uniform FP4 quantization.
Count
Base Acc( %)
Base
Our Acc(%)
Our
50%
63.56 + 0.89
4.53
62.15
4.44
75%
57.85 + 0.90
4.53
59.09
4.44
90%
55.82 + 0.80
4.53
56.27
4.44
A.10 PROOF OF PROPOSITION
Proposition 1. Let q Rn - Rd be an unbiased (E[g(x)] = x) and scale-invarian (q(Ax) =
Xx(x)) quantizer whose outputs lie a fixed finite grid. If x is drawn from absolutely continuous distribution, then
Var(g(x)) (11x)
Proof. We begin by first showing the upper-bound: Var (g(x)) = O (1xx11)) We define M =
||x|| oo and v = x/M, so l|vl = 1. By scale-invariance of q.
Var (a())( Var(g(Avv) Var q(v)) = M² Var Var(g(v)).
Since q(v) [-1, there exists finite C' such that Var(g(v)) - C, giving
Var(a(x)) = M Var (a(v)) S CCM² = C |
Next, we show the lower-bound: Var (g(x)) = Ω (2)x On the compact set {v: || 1}. the continuous function v + Var(g(v)) attains a minimum m 2 O. Because the finite quantizer grid has measure-zero, absolute continuity of ensures the probability of landing on the grid is 0, so Var(g(v)) > 0 and hence m > 0. Therefore
m 2 I Var(g(x)) S SCM²,
From this we conclude Var(g(x)) ([|x1118). ) as desired.
A.11 JUSTIFICATION OF PRIVACY GUARANTEES (THEOREM 2)
Proposition 2. Algorithm 7 is a Sampled Gaussian Mechanism (SGM) with sample rate q |B| D and noise scale o meaaure-
Proof. We first characterize Algorithm T Tas an analysis on the user's private dataset. The function accepts subsampled batch of size B from a dataset with DJ samples.
Using this batch of user data, as well some non-private sources of data such as the model weights, we compute the loss differences which is vectorized in R E R?, where is the number of available quantization policies.
we clip the vector R. to norm C, to which independent Gaussian noise proportional to C²C² is added to obtain R.. This is equivalent to adding noise proportional to o² when the sensitivity of R is 1 through scaling argument.
In step 3 of Algorithm 1
Algorithm ceases access private data in B after the computation of R. which results in step (updating the policy EMA) being post prosessnnsssingg Dwork and Roth (2014) which does not impact the privacy consumption.
20

---

Published as a conference paper at ICLR 2026
Furthermore, the privacy accounting step makes use of Opacus' Yousefpour et al. (2022) privacy accountant, which assumet the noise scale o is proportional to the clipping constant (i.c. equivalent to adding noise proportional to c²C?.
1.12 LOw PRECISION SIMULATION SETUP
As FP4 hardware support is forthcoming, we employ the following simulation setup to emulate the effect of training under FP4. Notably, we quantize both inputs the conv2d forward, wgrad, and dgrad operators as well as its output.
image
quantizer
fp32 conv2d
forward
quantizer
out
weight
quantizer
Figure 7: Quantization simulation setup
A.13 THEORETICAL SPEEDUP CALCULATION
Due to the unavailability of accelerators and reliable software support for FP4, we instead rely performance model to estimate the theoretical throughputs of FP4 computation.
We first decompose the DP-SGD training computation into the following operations, listed in table 13
We performed profiling on the models (on their respective datasets) stated in the paper, and we plot the runtime decomposition in Figure g Using this data, we can compute the amount of "overhead" (i.e. the time spent on operators which will not benefit from lower precision) for each model/dataset. This is tabulated in Table 14
Table 13: Decomposition of DP-SGD training
Benefits from FP4
Computation Stage
Description
Total Forward
Time spent on the forward pass through the model, where input data is processed layer by layer to produce the output.
Total Backward
Time for backpropagation, where gradients are calculated for model parameter updates.
Optimizer Clip
Time for clipping gradients to a predefined threshold to en- sure stability and prevent large updates during training.
Optimizer Noise
Time for adding random noise to the gradients to ensure differential privacy by masking individual data point contri- butions.
Optimizer Scale
Time for scaling the gradients after clipping to adjust the magnitude of the updates.
Other Optimizer
Time spent on other optimizer-related operations, such as learning rate management.
Other Time
Time for all other operations during the training iteration, including data loading, synchronization, and auxiliary tasks.
This assumption stated in httph:
21

---

Published as a conference paper at ICLR 2026
Opacus DP-SGD Training Breakdown
resnet50 gtszb
Total Forward
Total Barkward
resnet50 emnist
Optimizer Clip
resnet50 eifar10
Optimizer Noise
Config
1esne118 glscb
Optimizer Seale
Other Optimizer
renel18 ermmist
Other Time
18nell18 cifar10
dereened11) giscb
densemet121 cifar10
0.0
0.2
0.4
0.66
0.8
1.0
1.2
Time
Figure 8: Runtime decomposition of DP-SGD training
Table 14: Breakdown of total time, good ops, bad ops, and overhead percentage for different model configurations.
Config
Total Time
Ops with Speedup
Overhead Ops
Overhead
DenseNet121 CIFARIO
1.15 x 109
1.10 x 10"
5.23 x 102
4.55
DenseNet121 GTSRB
1.08
109
1.01 10
6.74 x 102
6.23
ResNet18 CIFARIO
1.82 10a
1.66 10
1.68 x 10?
9.20
ResNet18 EMNIST
1.86 10a
1.49 x 10
3.68 x 10?
19.81
ResNet18 GTSRB
1.74 10a
1.63 10
1.04 x 102
5,99
ResNet50 CIFARIO
4.31 10a
4.05 108
2.55 107
5.92
ResNet50 EMNIST
3.88
10a
3.36 10
5.13 X 10?
13.22
ResNet50 GTSRB
4.05 10a
3.76 X 108
2.87 X 10
7.10
A.14 OPACUS PRIVACY ACCOUNTING
Opacus maintains tuple of the form (sample rate, noise scale, number steps), which is incre- mentally updated during training. At any point, we can query the current privacy cost in terms of (c, 6) by specifying a target å and using either the rdp or prv accountant. This mechanism enables flexible and precise tracking of privacy usage, allowing us to assess how much additional privacy is consumed by our analysis relative to standard training.
A.15 SAMPLING OF QUANTIZED LAYERS
We outline the algorithm DPQUANT select layers to compute under quantization in Algo-
rithm 2
Algorithm 2 SELECTTARGETS
1: Input: L (EMA scores), P (set of policies), s (temperature), (number to sample), layers
(set of layers to quantize under policy p)
2: < [L[p] for p E P
3: v + (u (u-nu (min(:))/(max))
- min(v)
> Normalize
4: softmax(-s
5: Q < Multinomial(z, m, without replacement)
Sample m policies
6: S 0)
7: for each 2 do
8: S S 1ayersls
9: end for
return
22

---

Published as a conference paper at ICLR 2026
A.16 REMARK: APPLICABILITY NON-DP TRAINING
The mechanisms in Sections 5.1 and 5.2 are general and could in principle apply to non-DP settings. However, prior work such LUQ-FP4 CCmiiel al. 2024 shows that FP4 non-DP training suffers negligible degradation, so DPQUANT's improvements would be marginal there. Our main novelty and motivation are DP-specific:
identify and analyze setting where non-DP FP4 training shows negligible degradation, but DP + FP4 suffers substantial accuracy loss (Figure Ta attributing this to the interaction between fz-calibrated DP noise and -diive quantization. Furthermore, our loss voss-sensttiviyy measures and selection rules are designed to be compatible with DP accounting and composition, relying only on quantities that are already DP-released or incorporable into the privacy accountant, which is not concern in standard non-DP quantization work.
A.17 REMARK: VULNERABILITY TO FLOATING POINT ATTACKS
Differential privacy implementations must carefully consider the vulnerabilities highlighted by Mironov (2012). Mironov identified that the floating-point implementation of noise sampling for mechanisms such as Laplacian or Gaussian introduces a "porous" distribution that lacks translation invariance. This issue is prevalent in both fp64 and fp32 arithmetic.
To ensure robustness against this vulnerability, our method has been meticulously designed. The critical step of noise addition in our framework occurs under standard conditions, prior to the application of our novel quantization technique. The process is as follows:
1. Gradients are maintained in full fp32 precision.
2. Noise is sampled and added to these fp32 gradients, also in fp32 precision. Only after the noisy gradient computed quantized for use in the forward/backward pass of select layers.
3. Thus, the noise injection process maintains a vulnerability profile identical to that of standard DP-SGD implemented in fp32. The of lower-precision representations for computation does not alter or exacerbate the known properties of the initial noise addition.
Additionally, our method is fully compatible with established defenses against this vulnerability. The 'snapping mechanism' proposed by Mironov, a post-processing step applied directly to the noisy output, would be applied to the full-precision fp32 gradients immediately after noise addition and before quantization our pipeline.
23


Reviews:
[{"review_id": "PGghHWPQPs", "reviewer": "Reviewer_oPEM", "summary": "This paper observes that quantization variance is detrimental in DP-SGD. The authors propose two methods to reduce the variance, which achieve good Pareto-optimal performance when used in combination. Specifically, the loss-aware method takes care of DP guarantee. Some experiments on small datasets and models show a good utility-compute tradeoffs while leveraging the quantization speedup.", "questions": "Can the authors add DP-AdamW at least for NLP experiments?\n\nDoes non-DP quantization works better in finetuning? like comparing fp4 with quantization to fp32 without quantization, will the gap in pretraining be larger than in finetuning?\n\nHow necessary is DP quantization in finetuning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the identification of the effects of quantization variance is novel, and the solutions to reduce the variance is novel. Hence this work has fair significance in improving DP efficiency. The motivation of the solutions is reasonable, as variance in DP noise is already known as the key blocker in DP convergence like \"Pre-training Differentially Private Models with Limited Public Data\". Hence it is not surprising another type of variance from quantization also harms the convergence.", "weaknesses": "In practice, especially in NLP, models are optimized by adaptive optimizers like AdamW or Muon, whereas this work has a narrow focus on DP-SGD (maybe not even with momentum?) I would encourage the authors to put DP-AdamW in empirical experiments which should not require extra efforts.\n\nIn \"Pre-training Differentially Private Models with Limited Public Data\", it is claimed that variance only harms the early stage of training, e.g. in pretraining regime, to which this paper's experiments are limited. In fact, many existing works including \"Large Language Models Can Be Strong Differentially Private Learners\" have shown that DP finetuning is comparable to non-DP finetuning, suggesting variance is less detrimental in the finetuning regime. I wonder would DPQuant be less effective in this regime?\n\nAlso please use a different notation for policy. Right now both policy and probability is p.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AEPYaaT4w2", "reviewer": "Reviewer_Bdkn", "summary": "This paper studies methods to reduce quantization variance in DP-SGD, which slows down the convergence a lot. Technical efforts are dedicated to achieve good performance, maintain DP guarantee, and avoid extra computation. A number of experiments on computer vision and one language modeling are presented on relatively old architectures, showing a near Pareto-optimal utility-compute tradeoffs and about 2X theoretical speed improvement.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper is clearly written and shows the originality in combining quantization and DP. The problem of DP training efficiency is an important one, highlighting the significance of this work. The experiments are careful and convincing, especially the ablation study. I enjoy reading the details in this paper, such as Pareto optimality and error bars.", "weaknesses": "1. Wrong positioning of mixed-precision training\n\nThe authors state \"While effective for standard SGD, mixed-precision training degrades significantly under DP-SGD. To our knowledge, no prior work has explored mixed-precision training when differential privacy mechanisms are employed.\" Both sentences are wrong. Mixed-precision training with DP is comparable to DP-SGD with full-precision. See Appendix T of \"https://arxiv.org/pdf/2110.05679\" (no code though) and Section 3.4/Appendix C of \"https://arxiv.org/pdf/2311.11822\" (code is open-sourced). The caveat is the loss scaling when using fp16, and DP training with bf16 can just run without loss scaling.\n\n2. Limitation to DP-SGD\n\nI recommend the authors to analyze DP-Adam or DP-SignSGD, where the gradient norm is always \\sqrt{d} and line 230 won't hold. How would you extend your analysis in this case? Also, DP-Adam and other adaptive optimizers are the main-stream for large models. Can your method work directly? What will be the performance of quantization empirically?\n\n3. Limitation to training from scratch\n\nDP pre-training is hard. I think this is why the authors stick to small and toy datasets. However, the methodology should be applicable to small-scale finetuning as well. Did the authors try that?\n\n4. What about non-DP?\n\nThe methods in Section 5.1 and 5.2 should also improve non-DP quantization, even though additional care was taken to make it DP. Are these methods novel? Are there evidence these methods also work in non-DP scenario?\n\n5. Not practical for throughput\n\nAnother minor weakness (and my review score does not depend on this point) is that the improvement of throughput is theoretical, not applicable in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oZo7IEwGRg", "reviewer": "Reviewer_ixjt", "summary": "This paper focuses on differential privacy (DP) and the impact of quantization variance. The authors hypothesize that quantization variance leads to disproportionately large accuracy degradation in DP settings. To address this issue, they propose DPQUANT, a dynamic quantization framework that adaptively selects a changing subset of layers to quantize at each epoch. The main procedure consists of (i) probabilistic sampling of layers and (ii) loss-aware layer prioritization.", "questions": "It is unclear whether the Exponential Moving Average (EMA) update plays an essential role in the proposed framework. While the paper mentions EMA as part of the post-processing step for maintaining differential privacy, there is no ablation study isolating its effect. It would strengthen the work to report performance both with and without EMA, or to justify why EMA is necessary for stability or privacy preservation in DPQUANT.\nFurthermore, it would be helpful to discuss whether other optimization or averaging techniques (e.g., momentum-based methods, adaptive optimizers such as Adam or Adagrad, or other moving-average variants) could serve a similar role. Clarifying whether the proposed approach is tightly coupled with EMA or can generalize to other optimizers would provide deeper insight into the method’s robustness and applicability.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The mathematical analysis is clearly presented, and the theoretical treatment of differential privacy in the proposed algorithm, such as the post-processing of the update exponential moving average (EMA), appears sound. The empirical settings are well constructed, and the figures and tables are clear and easy to interpret. Compared with vanilla DP-SGD, the proposed method demonstrates meaningful performance improvements.", "weaknesses": "However, the related work and comparison baselines raise significant concerns. Quantization is a widely used technique across various applications, and it is natural to consider its potential benefits within differential privacy. Indeed, several prior works have explored the interplay between quantization and DP, including:\n- Kang, Tianqu, et al. \"The effect of quantization in federated learning: a rényi differential privacy perspective.\" 2024 IEEE International Mediterranean Conference on Communications and Networking (MeditCom). IEEE, 2024.\n- Youn, Yeojoon, et al. \"Randomized quantization is all you need for differential privacy in federated learning.\" arXiv preprint arXiv:2306.11913 (2023).\n- Kim, Muah, Onur Günlü, and Rafael F. Schaefer. \"Effects of quantization on federated learning with local differential privacy.\" GLOBECOM 2022-2022 IEEE Global Communications Conference. IEEE, 2022.\n- Wang, Yongqiang, and Tamer Başar. \"Quantization enabled privacy protection in decentralized stochastic optimization.\" IEEE Transactions on Automatic Control 68.7 (2022): 4038-4052.\n-Xiong, Sijie, Anand D. Sarwate, and Narayan B. Mandayam. \"Randomized requantization with local differential privacy.\" 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016.\nWhile some of these works focus on federated learning, the authors should still acknowledge them and discuss how their method relates to or differs from these approaches.\n\nBeyond these, it is also important to consider recent advances in DP optimization methods beyond DP-SGD. Since DP-SGD is known to suffer from excessive noise and slow convergence, relying solely on it as a baseline is insufficient to demonstrate the full benefit of the proposed approach. Several improved variants could be relevant comparisons:\n- Wang, Zihao, et al. \"{DPAdapter}: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training.\" 33rd USENIX Security Symposium (USENIX Security 24). 2024.\n- Wei, Jianxin, et al. \"Dpis: An enhanced mechanism for differentially private sgd with importance sampling.\" Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 2022.\n- Park, Jinseong, et al. \"Differentially private sharpness-aware training.\" International Conference on Machine Learning. PMLR, 2023.\n- Denisov, Sergey, et al. \"Improved differential privacy for sgd via optimal private linear operators on adaptive streams.\" Advances in Neural Information Processing Systems 35 (2022): 5910-5924.\n\nIn summary, while the proposed method is technically sound and the presentation is clear, the related work section and choice of baselines need substantial improvement to strengthen the novelty and empirical validity of this paper.\n\nIf these concerns are properly addressed, I would be pleased to raise my evaluation score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Byht92WIi7", "reviewer": "Reviewer_ieCQ", "summary": "This paper investigates dynamic quantization for differentially private SGD. The authors propose a specific method that converts specific layers into a low-precision format and design a selection criterion based on the difference between the quantized and non-quantized layers. They empirically validated the strength of their dynamic quantization approach across various settings, including different configurations and datasets.", "questions": "•\tIn the related work section, the authors mention that all existing methods are orthogonal to this work. However, for readers who are not familiar with quantization, it is hard to understand the relationship between these methods and the authors' work. The authors need to improve their related work section. Maybe including more on DP network pruning or layer selection, such as [1]. \n\n[1] Differential Privacy Meets Neural Network Pruning, arXiv:2303.04612\n\n•\tThe reviewer is not convinced why the authors shift their perspective from the L2 norm (for noise addition) to the L-inf norm (for analysis). Can the authors clarify this reason or explain why the L2 norm is not sufficient for their analysis? As far as I understand, Figure 1 does not support the reason for the L-inf analysis, and the empirical part uses the L2 norm.\n\n•\tDoes the quantization method meet the properties in Proposition 1, such as the unbiased and scale-invariant properties?\n\n•\tIf a low-precision format induces the DP accuracy drop, can using higher precision mitigate the accuracy drop in DP training?\n\n•\tWhen quantizing, is there any difference between floating-point and fixed-point quantization in terms of differential privacy?\n\n•\tThe authors should carefully check the paper for readability. For example, the GTSRB dataset is not explained or cited when it is first used. And the indicator function in Eq. 1 should be clarified.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "•  The paper investigates the quantization of differentially private deep learning.\n\n•  The authors provide some basic analysis of why DP training interacts poorly with quantization.\n\n•  Based on their observations, the authors’ new training method enables faster training compared to FP32 training.", "weaknesses": "Please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)