On Optimal Hyperparameters for Differentially Private Deep Transfer Learning
Authors: Aki Rehn
Linzh Zhao
Mikko A. Heikkilä
Antti Honkela
Venue: ICLR 2026 Poster (2026)
Keywords: differential privacy
hyperparameters
deep learning
transfer learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We study how the difficulty of the transfer learning task affects the optimal hyperparameters in differentially private deep learning.

Abstract: Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the-art approach for training large models under privacy constraints. 
We focus on two key hyperparameters in this setting: the clipping bound $C$ and batch size $B$.
We show a clear mismatch between the current theoretical understanding of how to choose an optimal $C$ (stronger privacy requires smaller $C$) and empirical outcomes (larger $C$ performs better under strong privacy), caused by changes in the gradient distributions. 
Assuming a limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning $B$ do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. 
We also highlight how the common practice of using a single $(C,B)$ setting across tasks can lead to suboptimal performance. 
We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as a form of gradient re-weighting and examining cumulative DP noise.

Published as a conference paper at ICLR 2026
ON OPTIMAL HYPERPARAMETERS FOR DIFFEREN- TIALLY PRIVATE DEEP TRANSFER LEARNING
Aki Rehn 1 Linzh Zhao",, Mikko A. Heikkilä', Antti Honkela' Department of Computer Science, University of Helsinki, Finland Equal contribution (aki. rehn, linzh. zhao, mikko.a.heike
@helsinki. .fi
ABSTRACT
Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the approach for training large models under privacy constraints. We focus two key hyperparameters this setting: the clipping bound C' and batch size B. We show a clear mismatch between the current theoretical understanding of how choose optimal C' (stronger privacy requires smaller C) and empirical outcomes (larger C' performs better under strong privacy), caused by changes in the gradient distributions. Assuming limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. We also highlight how the common practice of using a single (C, B) setting across tasks can lead to suboptimal performance. We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as form of gradient re-weighting and examining cumulative DP noise.
INTRODUCTION
The current state-of-the-art approach for training large models sensitive data is differentially private (DP) transfer learning: after pretraining backbone on non-private data, the model fine-tuned on the private task using DP optimizer, such DP-SGD or DP-Adam (De al. 2022 Due to the high computational requirements of DP optimization, commonly only the learning rate is tuned for each separate problem, while other hyperparameters, most importantly batch size and/or clipping bound, are assumed to be stable and fixed to a single value across different privacy levels, backbones,
and computational budgets (see, e.g., De al. 2022 Panda al. 2024] Sander al. 2023)
Fasy (ViT-Rase,
Medium (ViT-Ting
Hard (ViT-Tiny. 0.25)
0.30
Batch Size
0.74
213
0.72
0.62
0.70
0.60
0.26
260
Clipping Bound
Figure 1: Macro accuracy heatmaps under increasing learning-problem difficulty (left to right): strong model with loose privacy, weaker model with same privacy, and weaker model with tight privacy. Experiments use DP-Adam with 8 10-3 on SUN397 "Full'=87002) averaged over 5 seeds with epochs each. Learning rates are tuned separately for each configuration using a fixed grid. No single clipping bound batch size performs best across all settings.
In this paper, we demonstrate that fixing hyperparameters often leads to suboptimal performance and explain why this happens. We find across multiple datasets that the optimal clipping bound and batch size are affected by the overall learning learning-problem problem difficulty- that is mainly governed by the privacy budget, available data and compute, dataset difficulty, transfer complexity (Tobaben al. 2023

---

Published as a conference paper at ICLR 2026
and the pretrained backbone capability, which reflects both model complexity (Hu al. 2021b) and pretraining quality.
Figure Tillustrates how fixing the clipping bound and batch size across problems fails to account for this difficulty: the settings that perform best in easy cases degrade clearly under harder ones, and vice versa. Ignoring these shifts and fixing clipping bound and batch size across tasks systematically suppresses DP transfer learning performance, reducing overall accuracy and especially harming difficult examples and the classes they dominate. We summarize implications for hyperparameter tuning in Table
Table 1: Practical implications for hyperparameter tuning.
Change in condition
Tuning implication
Lower (tighter privacy)
Increase C, decrease B. Use smaller C.. Try larger C.
Stronger backbone easier dataset
Weaker backhone / harder dataset
Fewer epochs (limited compute)
Avoid large B (= more steps).
More epochs
Larger B becomes viable.
General guidance
Tune (C, B, 1) jointly for each task.
Contributions
We present systematic study of how the optimal clipping bound C' and batch size vary- primarily with the privacy budget (s), but also with other factors that affect the difficulty the private fine-tuning task, such as the capability of the pretrained backbone and the amount of compute in DP deep transfer learning.
2. We demonstrate through extensive experiments that contrary to what previous theory sug- gests, increasing C can improve results with smaller e (Section 5.T), 1 and with less capable pretrained models (Section [.2]p. To explain these findings, we derive a novel optimal clipping result (Theorem 5.2) and show how it affects optimization progress (Theorem 5.4 Corollary 5.5). which results might be of independent interest.
3. We propose a rule for selecting the optimal batch size under bounded compute (fixed epochs) through combination of a lower bound on the number of optimisation steps and maximising the number of steps without increasing the total cumulative DP noise (Section 6
BACKGROUND
DP (Dwork et al. 2006b) provides formal guarantee that an algorithm's 's output distribution does not depend too strongly on any single individual's 's record. This guarantee is defined with respect to neighboring datasets, which differ by one record. Formally:
Definition 2.1. (Approximate differential privacy, Dwork et al. 2006b
An algorithm M D - R is (e, -iiffferrntl private if for all neighboring datasets D' e D and for all subsets S C R, it holds that
Pr[M(D) E S) Pr(M(D') E S]+ 6
We use the add-remove neighborhood relation with sample-level privacy, mcaning the neighboring datasets differ by the presence or absence of a single training example. and ò E [0, 1] control the strength of the privacy guarantee with smaller values implying stronger privacy. We set 8 = 10-5 in all the experiments.
In machine learning, DP is typically enforced by adjusting the optimization process by suitably randomizing each optimizer step. Algorithms such as DP-SGD (Song al., 2013) Abadi al.. 2016, Rajkumar & Agarwal 2012) and DP-Adam (Abadi et al.) 2016 Kingma & Ba 2015) clip per-example gradients to fixed norm in order to bound the sensitivity the maximum change any single record can have on the model update. The optimizer then adds noise, calibrated to the sensitivity, to the aggregated gradients to guarantee DP. Each training iteration then contributes to the cumulative privacy loss. We use the Privacy Random Variable (PRV) accountant (Gopi al. (021). numerical method that tightly tracks the privacy guarantees under subsampling, to calibrate the noise level to match the target privacy budget.
2

---

Published as a conference paper at ICLR 2026
Gradient clipping and noise addition generalize to any gradient-based optimizer. We focus DP-Adam due to its widespread use, but the same principles apply to any first-order optimization method (McMahan et al., 2019). In experiments, we a variant that decouples the learning rate and the clipping bound (Al (Algorithm[]) [T. For analysis we adopt the standard (unnormalized) variant in which the DP noise term scales explicitly with C. See Appendix B for details of these two variants.
3 RELATED WORK
In this work, we mainly focus on analyzing two hyperparameters in DP optimization: clipping bound C' and batch size B. As these are critical parameters that need to be somehow addressed by every DP deep learning work, limit our discussion the most significant works.
Algorithm 1 Generic DP optimization, normalized, from (De et al. 2022)
Input: iterations T, dataset D, sampling rate g. clip- ping bound C, noise multiplier o, learning rate initial weights 0g. initial optimizer state On. for t = T do ~ PoissonSamplle(
for (.,, Vi E B do
Clipping bound There are some theo- retical reasons expect that the clipping bound C' is an important hyperparameter for model training: Chen et al. (020)) and
Jr < VL(so, (x:). clip 9i
+ min
end for 1
Koloskova et al. (2023) show that clipping DP-SGD will in most cases cause bias.
g
Eien
93 clip + NN o.))
B
The bias due to clipping tends to amplify fairness disparities (see, e.g., [Bagdasaryan et al. 2019
(0.+1, O++1) + OptimizerStep(f, (r, n. Oi)
end for
Esipova et al. 2023).
For properly tuning fixed C' in PPnoomarea et al. (2023) propose using a non-private model
to identify the smallest C' that slightly degrades utility, then applying this during private training, while Tobaben et al. (0023) perform Bayesian optimization over C' [0.2, 10) to directly optimize
utility.
However, many recent papers simply fix C to a constant for all experiments, c.g., to C' = 0.1 (Trmmer
& Bonch 2 C De et al. [022 Berrada et al. 20233 Sander et al.) 2023 Panda et al. [024]]
Koskela & Kulkarni 2023). In the same vein, Bu et al. 2023) eliminate the need to set any specific
C by introducing automatic clipping, which normalizes each per-sample gradient individually, which
is equivalent to setting an extremely small clipping bound. Liu & Bu (024) further build on this
idea and propose a hiperparameter-ffre framework that combines automatic clipping with adaptive learning rate schedules.
Adaptive clipping (Andrew et al., 2021) translates the problem of choosing fixed C estimating it dynamically from quantile of the gradient norm distribution. The adaptive approach can still lead to suboptimal results for example with highly bimodal gradient norm distributions that commonly arise under strong privacy. As a final alternative, Zhhang et al. (2024) introduced an error-feedback method to trade the elimination of clipping bias to adding more noise.
In summary, there is plenty of contrasting and partially contradicting claims and recommendations on the importance and tuning of clipping bound the existing literature. In this work we seek to clarify these issues.
Batch size bound on training epochs. This is in contrast to prior work mostly assuming fixed number of training steps. The fixed-epochs setting introduces trade-off between batch size and the number of steps.
Our work focuses on the bounded compute setting, which we implement through a
Based on theoretical analysis a single-step setting, Räisä et al. (2024) show that effective DP noise variance, o² //², where a is DP noise and is the subsampling probability, monotonically decreases as batch size increases. As the number of fixed steps increases, the effective DP noise variance becomes invariant of B, leaving only the mini-bated induced variance which decreases B increases. Based on similar reasoning and the fixed-steps setting, PPonomreva et al. (2023) recommend large batches, but note point of "diminishing returns"" beyond which performance plateaus.
3

---

Published as a conference paper at ICLR 2026
Empiricallyy, Abadi al. (2016) note that batch size has relatively large impact accuracy and suggest approximately vN as the optimal batch size for dataset with N observations. Conversely, Panda et al. (2024) find that under their linear learning rate scaling, batch size has only very small
impact.
Recent empirical work in DP deep learning generally recommend using relatively large batch sizes. In image classification, De al. (2022) show that without any other constraints, increasing the batch size monotonically increases accuracy over a broad range of values, but larger batches require more epochs and thus more compute. |Mehta al.) (2023) show that for certain models and tasks, performance may improve up to batch sizes of IM.
In large language models, McKenna et al. (2025) study the optimal batch size by simulating the effect of varied batch sizes from experiments performed using fixed physical batch size of 1024. For largest compute budgets, the optimal simulated batch size can exceed 1M. They also make the surprising discovery that even smaller physical batches can perform better in high noise (low E) settings.
Some works treat batch size as a fixed, lata-independent hyperparameter. [Bu al. (2023): Liu & Bu 2024) argue that batch size does not need tuning, while Morsbach et al. (2024) consider it Naatings,TTbbben full-batch) in their hyperparameter search, though they do not analyze its effect.
unimportant. In few-shot settings, Tobaben al. (2023) include a wide range of batch sizes (including
While much of the recent work favors large batches, we find that under fixed-epoch imposed by limited compute--moderate-siiii moderate-sized batches (see Section 6) can outperform large ones, especially under high learning-problee difficulty, where more iterations are needed for convergence.
METHODOLOGY
Models For image tasks, we employ Vision Transformers (ViTs) (Dosovitskiy et al. 2021) from the PyTorch Image Models library (Wightman 2019), using ViT-Base and ViT-Tiny to represent high- and low-cipabllity pretrained backbones, respectively. All image models for fine-tuning tine-tuning tasks are pretrained on ImageNet-21k (Ridnik al. [021] with the "AugReg" setup that includes multiple data augmentation methods as well as regularization methods (Steiner et al.) (022). Depending on the classifier head size, ViT-Base has approximately 85M parameters, and ViT-Tiny has around 5M. For text classification, we use the DistilBERT Sanh et al. (2019) model which has roughly 66M parameters. Morcover, the experiments for training from scratch employs WideResNet- ZZaoruyko & Komodakis 2016), which contains about 2.8M parameters.
Fine-tuning & from-scratch training For image tasks, we follow the experimental setup of Tobaben et al. (2023) for differentially private fine-tuning. Specifically, we use the feature-wise linear modulation (FiLM) parameterization (Perez et al. 2018) by freezing all the other parameters and train only the scale and bias of the normalization layers in addition to the classification head. This results in approximately 0.5-1.5% of trainable parameters, depending on the size of the backbone and the classification head. Following the DP-FiLM methodology. we initialize the classification head weights to zero. Furthermore, we also experiment training with low-rank adaptation (LoRA) (Hu) ct al., 202 aa under DP with image models (see Appendix G. and lastly with with full fine-tune (see Appendix I.3). For natural language tasks, we follow the standard practice of fine-tuning them using LoRA. For training from scratch, we use the WideResNet-
(Zagoruyko & Komodakis 2016 also used in from scratch experiments by De et al. (2022) a CNN with approximately 2.7M parameters, on the CIFAR-10 10 dataset.
Hyperparameter grids We perform a structured grid search over all three hyperparameters: learn- ing rate, batch size, and clipping bound (see AlI. Batch sizes range exponentially from 256 up to full-batch (i.c., the full training); the range of clipping bounds depend on the dataset; and similarly learning rates chosen geometrically from dataset-specific ranges. We provide details in Appendix [
We found this exhaustive tuning is necessary expose the subtle interactions between DP-specific hyperparameters and training dynamics. For instance, the learning rate often dominates optimization, masking the influence of the other hyperparameters in less thorough hyperparameter tuning methods.
4

---

Published as a conference paper at ICLR 2026
Effect of e
Effect of backbone
A Accuracy
0.01
8.0
VIT- -TT-Bacc
0.25
Wir-Tny
00.02
0.1
10 26 42
0.1
10
26 42
Clipping Bound
Figure 2: Accuracy difference the best, mean over 5 repeats with min-max bands; SUN397 dataset, 8 epochs, 5=10- Learning rate and batch size are tuned jointly with clipping bound for each case. Left: optimal clipping constant larger with tighter privacy (small E). Right: low-capability pretrained backbone (ViT-Tiny) performs better with larger clipping bounds compared to a high- capability model (ViT-Base); 0-225 In both cases, harder tasks (orange) shift the optimal clipping bound upward, while easier tasks (blue) tolerate or even benefit from smaller bounds.
Datasets We evaluate models on four datasets: SUN397 (Xiao et al. 2016 2010), Cassava (Wwe baze et al. 2019), CIFAR-100 (Krizhevsky & Hinton 2009). and 20 Newsgroups (Mitchell 1997) (see Appendix C.2 for details). Additionally, we include 10% subset of CIFAR- TO0 to (i) study how scarcity of training data increases learning-problem prrolemoooemmmoe difficulty, and (ii) analyze how models capable of overfitting the training dataset affect the clipping bound. For each dataset, we merge the validation split (if available) into the training set to maximize training data for each configuration. We evaluate the final accuracy on the test dataset. SUN397 is a highly imbalanced real-world scene recognition dataset with 397 classes and ~ 87k training examples. Cassava is imbalanced plant disease classification dataset with classes and ~ 6k training examples, and is significantly out-of- distribution relative to the ImageNet-211 k pretraining dataset. CIFAR-100 is widely used, balanced natural image classification dataset with 100 classes and 50k training examples. 20 Newsgroups is a text classification dataset consisting of roughly 18k posts taken from 20 online newsgroups. For the image datasets, all input images are resized to 224 x 224 resolution to match the pretraining dimensionality.
5 OPTIMAL CLIPPING DEPENDS ON PROBLEM DIFFICULTY
In this section, we argue that the optimal clipping bound depends fundamentally on the learning problem difficulty, and consider clipping form of gradient re-ieighting. Fig. 2 shows results for ViT-Tiny on SUN397 under two different privacy levels (left) and for two pre-trained backbones (right) for varying C: with easy problem (shown in blue; left larger E, right: more capable backbone ViT-Base) optimal C is clearly smaller than with a hard problem (shown in orange; left: small e, right: less capable backbone ViT-Tiny). [ We provide additional image results (including ResNet- 50) in Appendix F text classification results in Appendix G.2 results with LoRA fine-tuning in Appendix G.T and lastly from scratch and DP-SGD training results in Appendix [..]] all of which appear to show connection between clipping bound and learning-problem difficulty. Lastly, the odd-one out is fine-tuning all the model parameters where we find that-especially with large models-the the increased parameter count, resulting in larger average gradient norms prevents using smaller clipping bounds and rendering both easy and hard task prefer large C' (see detailed experiments and discussion in Appendix [33
In the rest of this section, we first analyze how these two factors affecting problem difficulty. DP noise level and pretrained model capability, affect the optimal clipping bound, in Section [51] and Section 5.2, respectively. To understand better how clipping affects the gradient distributions, we look at clipping as form of gradient re- re-weight in Section 5.3
Considering hyperparameter tuning, our results imply that using any fixed C should give close to optimal performance only for settings of similar difficulty, and especially that using small C' only works with easy-enough settings.
Note that we do not claim that for decreasing smaller C' is always better, rather, we want to understand when and why this is the case. See Appendix for more results.
5

---

Published as a conference paper at ICLR 2026
5.1 How DP NOISE AFFECTS OPTIMAL CLIPPING
De et al. 2022 note that gradient clipping introduces a bias-variance tradeoff in DP optimization, but do not provide formal analysis. The closest theoretical work connecting DP-SGD optimization performance to the DP noise level a and the clipping bound C comes from Koloskova et al. (2023). who provide convergence guarantees for DP-SGD with per-example gradient clipping. However, any optimal clipping bound derived from their work would depend on quantities that are not known in advance (such as the loss evaluated at the unknown optimum; see Appendix DD. Furthermore, these bounds do not explain why-as as seen in our experiments-tighttt privacy can favor larger clipping bounds.
To understand when and why this happens, we analyze clipping in DP-SGD through mean squared error (MSE) decomposition into clipping bias and DP noise variance. This allows us to characterize how the tradeoff depends on both the noise level and the gradient norm distribution, and to derive a clipping bound that minimizes the per-step gradient MSE. We further connect this MSE to op- timization progress, showing that reducing this MSE tightens the bound on per-step optimization progress.
To understand the empirical results, we therefore derive the optimal clipping threshold under standard unnormalized DP-SGD formulation. In this setting, per-example gradients are clipped individually and isotropic Gaussian noise with per-coordinate variance o²C²2 is added. In the following, we use the standard (unnormalized) DP optimizer variant, where gradients are clipped individually and isotropic Gaussian noise with per-coordinate variance c²C² added. This variant adds the same amount of DP noise as the normalized one used in our experiments (see Appendix (B)
Assumption 5.1. Assume there is mini-batch sampling noise, we use standard per-sample constant clipping with C E [min, || Il. max; [[9.1]]. and the Gaussian mechanism with noise level a to privatize the sum of clipped gradients.
Theorem 5.2. Under Assumption 5.1 an optimal clipping constant C* that minimizes the mean squared error between the per-sample clipped DP gradient 9 and the true gradient 9 for fixed mini-batch satisfies
[Ig:|l for some i, or Ge No.
C C==
where the dimensionality of the gradient, Gc := Eiele Ii and Ne Eiele [g: Ic = :(::1lg.] > +C denotes the indices of the clipped gradients.
and
Proof. See Appendix for a proof.
Note that we do not Theorem 5.2 to find C* for doing DP optimization (this would require additional DP mechanisms as both Assumption [.1 Theorem 5.2 depend on the actual gradients), but only describe how the clipping works. Next, connect the per-step gradient MSE to the optimization performance. We again first state the assumptions and then give the theorem.
Assumption 5.3. Assume an L-smooth loss function E (Definition E.7y. and that we want to minimize L via gradient descent, using step size 1) S t. and the per-sample clipped and noisy DP gradients
g. E, Di
+ elf) at step I....,T TT instead of the true gradients gt E. (t)
(().))
Theorem 5.4. Under Assumption 5.3 the expected improvement the loss from step t + is bounded as
E[C(0(+1)00) (().) !7 [V (MSE((C)
(2)
Proof. See Appendix for proof.
The following Corollary 5.5 follows immediately from Theorem 5.4 as MSE(C) is non-negative.
Corollary 5.5. Under Assumption 5.3 minimizing MSE(C) minimizes the upper bound given in
Theorem 5.4 for the expected per-step improvement the loss function at any given step t.

---

Published as a conference paper at ICLR 2026
In summary, from Theorem 5.2 we see that C* depends not only directly o, but also the actual gradients: as a simple example, assuming that the optimal clipping is not exactly one of the well constant gradient direction and fixed number of clipped gradients, then increasing a implies smaller C* if the gradient distribution is not much affected by the noise, while larger gradient norms would always push towards larger C*. We can also connect C' directly to the actual optimization performance: as we show in Corollary 5.5 under Assumption 5.3 using optimal clipping that minimizes MSE directly improves the bound on the per-step loss improvement.
gradient magnitudes
As shown in Fig. B] the true gradient norm distributions do shift toward larger gradient norms under tighter privacy; ; in Appendix [ we empirically verify that the gradients shrink toward the end of training. Appendix J shows the similar plots for C' = 0.1 and C = 42. We argue that this more complex picture explains the empirical results we see with real data (Fig. 2 left), where smaller can counterintuitivel benefit from a larger clipping bound: increasing shifts the gradient distributions, effectively pushing the terms in Eq. ① towards larger values as the examples that were easy to learn with larger become harder.
E 8 (Easy)
E = (Medium)
$ = 0.25 (Hard)
Epoch
100
100
Gradient norm
Figure 3: Gradient norm distributions over training epochs for ViT-Tiny on SUN397 with FILM; 8 epochs, 8=10-5, C=1). Learning-problem difficulty increases from left to right as & decreases. Thicker regions imply higher probability mass, As difficulty increases, the distributions shift toward larger gradient norms.
5.2 HOW PRETRAINED MODEL CAPABILITY AFFECTS OPTIMAL CLIPPING
As mentioned in Section Fig. shows how the choice of pretrained model can have similar effect on the optimal clipping bound changing (switching to more capable backbone can roughly correspond to using larger e for fixed backbone). Furthermore, Theorem (.2 explicitly shows how the optimal clipping constant C* depends the DP noise a, the subsampling rate q (batch size), and the gradient distribution.
Considering how C* in Theorem 5.2 is affected by switching the pretrained backbone from less capable one to more capable one under fixed a, as previously hard examples are now easier to classify, this generally reduces the gradient norms. For example, in the simple case where the gradient directions or the number of clipped gradients does not change, this directly leads to smaller C'* in Theorem 5.2 While the true effect of changing the backbone is obviously more complex, as the gradient directions and number of clipped gradients can also be affected, we argue that this general effect the backbone has on the gradient distributions explains our empirical results.
Others have explored the features and their effect on learning: Tramer & Boneh (2021) discussed this prior the DP fine-tuning era. More recently. Wang et al.) (2024) find that high-quality features make learning more robust; aligning with our findings. Lastly, Zhaa et al. (2025) note that poorly tuned hyperparameters (especially the learning rate) degrade the feature quality of the pretrained model.
5.3 CLIPPING AS GRADIENT RE- RE-GIIGGG
To understand how clipping interacts with shifts in the gradient distribution besides just possibly causing bias (see, e.g.. et al. 2022; Koloskova et al. 2023), we can interpret clipping as form of gradient re-weighting: decreasing C gives more weight easy examples/caasses while dein-weighting the harder ones, whereas larger C' weights all examples/classes more equally. We formalize this by defining the retained weight for class y at clipping C' as
granular level,
wy(C)
1
Σ min (1,
C
(3)
My $19:=9
1g:1122
7

---

Published as a conference paper at ICLR 2026
Relative Retained weight
Easy (ViT-Base, 5
Medium (ViT-Tiny, 8)
Hard (ViT-Tiny, 0.25)
1.0
1.0
1.0
0.5
0.0
0.0
0.0
200
400
200
400
200
400
0.1
0.1
necuricc
AMA
0.0
0.1
00.1
-0.1
200
400
200
400
200
400
Class rank (easy to hard)
0.1
1.0
42.0
Figure 4: Per-class effects of gradient clipping under increasing task difficulty (left right), SUN397, 8 epochs; 5=10-5 Class rank is based on sorting classes according to their per-class accuracies. Top: relative retained weight after clipping, computed per class and normalized the class with the largest retained weight in the baseline (C==1). Flatter and higher curves indicate that gradients are preserved more uniformly across classes, while steep drops reveal that hard classes suffer disproportionately from clipping. Small clipping bounds risk over-clipping gradients from hard classes, while large bounds better preserve gradient signal across the classes under high learning-problem difficulty.
where ny -E, y) (number of samples with label y). Retained weight values near mean gradients are mostly preserved; lower values indicate stronger clipping. This is shown in Fig. 4
where the top panel shows per-class retained weight (see Appendix for details; see Appendix K
for the experiment settings, Appendix G.2 for results on text classification, Appendix [I]] for the
training from scratch image classification, and Appendix [.2 fine-tuning with DP-SGD) and the bottom panel accuracy relative to the C=1 baseline, with learning-problem difficulty increases from left right. As difficulty increases, the gap between clipping bounds widens; with small C hard classes are severely down-weighted compared to easier ones. A larger C' better preserves hard-class gradients, recovering their performance at the cost of slightly reduced accuracy on easier classes. In short, clipping acts as a tunable knob that re-weights example (and class) contributions-itt impact grows more asymmetric
ceariing-rrobeem difficulty increases.
This re-weighting view to clipping explains why methods for tuning C' based on optimizing the performance without DP noise to save compute Pon.maovoa et (023) only work well in specific settings: optimizing C' without DP noise works in the regime where adding noise does not make the problem much harder (in the sense of not affecting the gradient distributions too much cf. Fig. 3. Similarly, would expect automatic clipping (Bu al. [023 essentially, using very small C: see Appendix H} to work well in cases where focusing overwhelmingly on the easiest examples suffices for good performance or when class distributions are balanced so that no single group dominates the training. As shown in Fig. 5 this matches our empirical results: automatic clipping roughly matches tuned C' in easy-enough settings (larger E, simpler dataset), while it clearly under-performs under harder settings (smaller e, more complex dataset).
OPTIMAL BATCH SIZE DEPENDS ON PRIVACY AND COMPUTE BUDGETS
We study how batch size affects model performance under limited compute, focusing on the fixed- epochs setting. The number of epochs explains the compute quite well because the computational cost is dominated by feedforward and feedback computations that scale with the total number of samples processed. Under this setting, doubling the batch size halves the number of optimizer steps.
Prior works (Ponomareva et al., 2023 Räisä et al. 2024h take a fixed-steps view and suggest tuning the batch size by finding the sweet spot when plotting the standard deviation of the DP noise in the averaged gradient, thereby finding the smallest batch size with (nearly) optimal per-step signal-to- noise ratio.
8

---

Published as a conference paper at ICLR 2026
CIFAR-100
Cassava
SUN397
A M.croo Accurncy
0.00
AUTO-S
-0.03
Tuned
-0.05
C 0.5
2
c= 0.5
8 -
Figure 5: Comparison of AUTO-S S (Bu et 2023 and properly tuned flat clipping, mean accuracy with min-max bands over 3 repeats, ViT-Tiny model, 8 epochs training for SUN397 and CIFAR- 100, 32 epochs for Cassava; were tuned over fixed grid for both methods. AUTO-S performs notably worse on harder datasets (SUN397, Cassava), especially under tight privacy, as predicted by our analysis. See Appendix H for absolute accuracies.
10-5 Lines show difference best accuracy. Batch size and learning rate
10
0.05
Effecttive Noise
Effective Noise
102
Cumulative
Accuracy
103
0.03
10a
10a
0.02
10a
0.01
0.00
2
212
244
Full
2
240
12
24
Full
Batch Size
Figure 6: Neither per-step averaged gradient noise standard deviation suggested by Ponomareva et al. (2023) (left) or its cumulative version (right) saturates in the fixed-epochs setting, thereby failing to provide useful guidance for selecting the optimal batch size. The lines show the noise levels needed for fine-tuning ViT-Tiny for CIFAR-100 for epochs with 0.5, 1, 2, 8 (from top bottom), 5 10 The color indicates the difference to best accuracy each privacy budget; brighter is better.
The algorithm for finding the optimal batch size does not work in the fixed-epochs setting. As shown Fig. 6, the standard deviation plot does not have a sweet spot but would always suggest full batch which is suboptimal.
To develop an alternative that is better-suited to the fixed-epochs setting, we analyze how DP noise accumulates when the number of steps compute the noise multiplier, using the PRV accountant (Gopi et al. (222)) and from that derive the cumulative noise standard deviation: /T. This cumulative noise captures the total DP noise accumulated over training steps. We find that the cumulative noise, under the constraint of minimum number of steps, can explain which batch sizes perform the best.
E N/B increases as the batch size decreases. We
As Fig. depicts, using the cumulative DP noise, we recover weaker form of the plateau that reported for average gradient noise standard deviation: in low-s regimes, / remains nearly constant across wide range of batch sizes.
In addition the cumulative noise, the number of steps plays an important role in determining the optimal batch size. This is illustrated in the right panel of Fig. which shows that minimum number of steps (indicated by the vertical dashed line at 20 steps this case) is needed good performance. As illustrated for SUN397 in Fig. A35) the actual minimum number can vary between datasets. Increasing the number of epochs pushes training further into the asymptotic regime, flattening the T curves and reducing sensitivity to batch size, similarly as in the asymptotic theory of Räisä al. 2024).
As a rule, tighter privacy (lower E) broadens the flat region in the cumulative noise and favors smaller batches compared to weaker privacy. The effect of increasing epochs is more subtle: it simultancously allows using larger batches while still meeting the minimum steps requirement as well as broadens the flat region in cumulative noise.

---

Published as a conference paper at ICLR 2026
7 DISCUSSION & CONCLUSIONS
2pochs
We have presented systematic study of fac- tors influencing the optimal clipping bound C and batch size in differentially private deep transfer learning.
4
10.
For clipping bound, various factors such as tighter privacy budgets, less capable pretrained backbones and limited compute- compute-whiel which can be informally characterized difficulty-caure a shift in the distribution of gradient norms: gradients grow larger and more variable, especially for harder examples. In these cases, larger clipping bounds, despite in- jecting more noise step, better preserve the optimization signal and result in higher overall accuracy. Our experiments across privacy levels, model sizes, datasets, and training budgets con- firm this: hard examples become increasingly dominant under high difficulty, and training ben- efits from higher clipping threshold to avoid disproportionately discarding their gradients.
Curulaaive Nuise (ovT)
learning problem
Epochs
10'
10a
16 Epochs
10a
Voll
2015
Batch Size
Number of Steps
Accuracy
0.00 0.02
0.02
0.03
0.04
Figure 7: Cumulative noise, batch size and the number of steps under fixed-epoch DP-Adam on CIFAR-100 (ViT-Tiny, FiLM; N=50000, 8=10-5). Results are averaged over 3 seeds. The learning rate and clipping bound are tuned sepa- rately for each point. Left: cumulative noise o vs. batch size. Right: the same data shown against the number of steps. The color indicates the dif- ference to best accuracy at each privacy budget (e) and epoch count; brighter is better. When pri- vacy is tight, cumulative noise exhibits noise plateau where moderate-sized batches can outper- form large ones. The vertical lines illustrate that certain number of steps is required obtain the optimal accuracy, regardless the number of epochs.
In batch size, have focused fixed-epoch training where compute constraints limit the total number of samples processed. We have demonstrated that existing guidelines for tuning the batch size that focus on fixed-steps setting where the required compute grows with batch size fail to provide useful guidance in this set- ting. To fill this gap, we have proposed com- bination of minimum number of steps and min- imizing the batch size under (nearly) optimal cumulative noise as an effective predictor op- timal batch size across a wide range of settings.
Together, these findings challenge the common practice of fixing C' and B across tasks. Fixed hyperparameters ignore how privacy, data, and model interact shape gradient distributions and class separability. By tuning C' and B to match learning-pre -proilnm difficulty, models can learn more effectively and equitably across examples- especially in the high-difficulty regimes where DP training typically struggles. We also found that tuning learning rate, C, and B jointly, rather than in isolation, is crucial: for instance, the best learning rate under DP-Adam often scaled with v B, echoing non- non-private scaling rules Youu et al. 2020 Malladi al. 2022).
While our main experiments focus on parameter-efffiient (FiLM) fine-tuning for image classification, the underlying mechanisms are general, which we also demonstrated for many settings, such DP- LoRA for images and text classification. We also found that when fine-tuning all model parameters on larger models, the average gradient norm grows due to the increased parameter count, preventing the use of smaller clipping bounds. In these settings, both the easy and hard tasks prefer abnormally large clipping bounds.
What our results make clear is that good DP training requires matching the optimizer to the problem: C' and B are not just noise parameters, but levers for steering learning. Defaults often suppress the very gradients that matter most, waste compute on high-variance updates when more steps could be taken at no extra privacy cost. Tuning these hyperparameters can turn good models into great ones, especially in challenging tasks. As DP moves from research into practical deployments, understanding and exploiting these dynamics will be essential for building useful, reliable, and fair private models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence, FCAI, Grant 356499 and Grant 359111), the Strategic Research Council the Research Council of Finland (Grant 358247) as well as the European Union (Project 101070617 17). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them. The authors wish to thank the CSC IT Center for Science, Finland for supporting this project with computational and data storage resources. We acknowledge CSC (Finland) for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium. The authors acknowledge the research environment provided by ELLIS Institute Finland.
REFERENCES
Martín Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016. doi: 10.1145/2976749.229788
Kareem Amin, Alex Kulesza, Andres Munoz, and Sergei Vassilvtiskii. Bounding User Contributions:
A Bias- Variance Trade-off in Differential Privacy. In Proceedings of the 36th International Conference on Machine Learning, ICML. PMLR, 2019.
Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy, Differentially Private Learning with Adaptive Clipping. In The Thirty-Fifth Annual Conference on Neural Information Processing Systems, NeurlPS, 2021.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential Privacy Has Disparate Impact on Model Accuracy. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ahh--Buc, Emily B. Fox, and Roman Garnett (eds.), The Thirty-Second Annual Conference on Neural Information Processing Systems, NewrlPS, 2019.
Leonard Berrada, Soham De, Judy Hanwen Shen, Jamie Hayes, Robert Stanforth, David Stutz, Push- meet Kohli, Samuel L. Smith, and Borja Balle. Unlocking Accuracy and Fairness in Differentially Private Image Classification, 2023. URL /ntt:://aaxxii org/ abs/ 2308 10888
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger. In The Thirty-seventh Conference on Neural Information Processing Systems, NeurIPS, 2023.
Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding Gradient Clipping in Private SGD: A Geometric Perspective. In The Thirty-Third Annual Conference on Neural Information Processing Systems, NeurlPS, 2020.
Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlocking High- Accuracy Differentially Private Image Classification through Scale, 2022. http:///aaax
org/abs/2204 13650
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In The 9th International Conference Learning Representations, ICLR, 2021.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our Data, Ourselves: Privacy Via Distributed Noise Generation. In Serge Vaudenay (ed.), Advances in Cryptology 25th International Conference the Theory and Applications of Cryptographic Techniques, EUROCRYPT 2006. Springer, 2006a. doi: 10. 10.1007/11761699 29.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Advances in Cryptology- 25th International Conference the Theory and Applications of Cryptographic Techniques. EUROCRYPT 2006, volume 3876. Springer, 2006b. doi: 10.1007/1 1681878_14,
11

---

Published as a conference paper at ICLR 2026
Maria S. Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, and Jesse C. Cresswell. Disparate Impact in Differential Privacy from Gradient Misalignment. In The 1 11th International Conference on Learning Representations, ICLR, 2023.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical Composition of Differential Privacy. In The Thirty-Fourth Annual Conference on Neural Information Processing Systems, NeurIPS, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In The 9th Interna- tional Conference on Learning Representations, ICLR, 2021a.
Xia Hu, Lingyang Chu, Jian Pei, Weiqing Liu, and Jiang Bian. Model complexity of deep learning: A survey. Knowledge and Information Systems, 63(10), 2021b. doi:
Shuqi Ke, Charlie Hou, Sewoong Oh, and Giulia Fanti. Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion. NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability. FITML 2024, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations, ICLR, 2015.
Anastasia Koloskova, Hadrien Hendrikx, and Sebastian U. Stich. Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees. In Proceedings of the 40th International Conference Machine Learning, ICML. PMLR, 2023.
Antti Koskela and Tejas D. Kulkarni. Practical Differentially Private Hyperparameter Tuning with Subsampling. In The Thirty-Seventh Annual Conference on Neural Information Processing Systems, NeurlPS, 2023.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Toronto, ON, Canada, 2009. URL http:://ww/ cs wtt.toorottt.caaake.eto
feattueg-fetturess-
pd.
Ruixuan Liu and Zhiqi Bu. Towards hyperparameter--free optimization with differential privacy. In The 13th International Conference on Learning Representations, ICLR, 2024.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDEs and Scaling Rules for Adaptive Gradient Algorithms. In The Thirty-Sixth Annual Conference Neural Information Processing Systems, NeurIPS, 2022.
Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette- Choo, Badih Ghazi, Georgios Kaissis, Ravi Kumnr, Ruibo Liu, Da Yu, and Chiyuan Zhang. Scaling Laws for Differentially Private Language Models. In The Forty-second Interna- tional Conference on Machine Learning, ICML, 2025.
H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, and Peter Kairouz. A General Approach to Adding Differential Privacy to Iterative Training Procedures, 2019. htt.:ttpp/aaxxv..oot
abs 181211111111 .6210
Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Towards Large Scale Transfer Learning for Differentially Pri- vate Image Classification. Transactions on Machine Learning Research, 2023, 2023.
Tom Mitchell. Twenty newsgroups, 1997. doi: 10.24432/C5C323.
Felix Morsbach, Jan Reubold, and Thorsten Strufe. R+R: Understanding Hyperparameter Effects in DP-SGD. In Anmual Computer Security Applications Conference. ACSAC, 2024. doi: 10.1 ACSAC63791 110.3799...220009999
Emest Mwebaze, Timnit Gebru, Andrea Frome, Solomon Nsumba, and Jeremy Tusubira. iCassava 2019 Fine-Gre Grained Visual Categorization Challenge, 2019. URL http:://// http://xxiv...g///aa// 1 1908. 02900
12

---

Published as a conference paper at ICLR 2026
Ashwinee Panda, Xinyu Tang, Saeed Mahloujifar, Vikash Sehwag, and Prateck Mittal. A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization. In The Forty-fire International Conference Machine Learning, ICML, 2024.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual Reasoning with a General Conditioning Layer. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), 2018, 8. doi: 10.1609/AAAA.V321.11110
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Thakurta. How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy. Journal of Artificial Intelligence Research, 77, 2023, doi: 10. 10.1613/jair. 1.649
Ossi Räisä, Joonas Jälkö, and Antti Honkela. Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation. In The Forty-first International Conference on Machine Learning, ICML, 2024.
Arun Rajkumar and Shivani Agarwal. A Differentially Private Stochastic Gradient Descent Algorithm for Multiparty Classification. Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS, 2012.
Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. ImageNet-21K Pretraining for the Masses. Joaquin Vanschoren and Kit Yeung (eds.), The Thirty-fifth Conference on Neural Information Processing Systems, NeurlPS, Track on Datasets and Benchmarks, 2021.
Tom Sander, Pierre Stock, and Alexandre Sablayrolles. TAN Without a Burn: Scaling Laws of DP-SGD. The Fortieth International Conference on Machine Learning, ICMI., 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distllee version of BERT: Smaller, faster, cheaper and lighter, 2019.
Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In IEEE Global Conference on Signal and Information Processing, GlobalSIP. IEEE, 2013. doi: 10. 1099//abalSIPP22011666666
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your VIT? Data, Augmentation, and Regularization in Vision Transformers. Transactions on Machine Learning Research, 2022.
Marlon Tobaben, Aliaksandra Shysheya, John F. Bronskill, Andrew Paverd, Shruti Tople, Santiago Ganella-Beguelle Richard E. Turner, and Antti Honkela. On the Efficacy of Differentially Private Few-shot Image Classification. Transactions on Machine Learning Research, 2023.
Florian Tramèr and Dan Boneh. Differentially Private Learning Needs Better Features (or Much More Data). In The 9th International Conference Learning Representations, ICLR. OpenReview.net, 2021.
Chendi Wang, Yuqing Zhu, Weijie Su, and Yu-Xiang Wang. Neural Collapse meets Differential Pri- vacy: Curious behaviors of NoisyGD with Near-Perfed Representation Learning. In Proceedings of the 41st International Conference on Machine Learning. PMLR, 2024.
Ross Wightman. Pytorch image models.
com/ http//wightaaaa
pytorch-image-monn
2019. doi: 10.5281/2emooo.444881.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN Database: Large-scale scene recognition from Abbey to Zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, 2010.
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN Database: Exploring a Large Collection of Scene Categories. International Journal of Computer Vision, 119 (1), 2016. doi: 1007/sl
13

---

Published as a conference paper at ICLR 2026
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In The 8th International Conference Learning Representations, ICLR. OpenReview.net, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Procedings of the British Machine Vision Conference 2016, York, UK, 2016. British Machine Vision Association. ISBN 108-1-901725-59-99 doi: 10.5244/C.30.87.
Xinwei Zhang, Zhiqi Bu, Zhiwei Steven Wu, and Mingyi Hong. Differentially private SGD without clipping bias: An error-feedback approach. In Proceedings of the International Conference on Learning Representations,, ICLR, 2024.
Yue Zhao, Xia Yutong, and Chendi Wang. Why Does Private Fine-Tuning Resist Differential Privacy Noise? A Representation Learning Perspective. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025.
14

---

Published as a conference paper at ICLR 2026
A
LLM USAGE
We used large language models (LLMs), including OpenAl's ChatGPT and GitHub Copilot, at various points during the development of this paper.
These tools assisted with grammar and phrasing, clarification of technical concepts, code generation and refactoring code for data processing, filtering, and visualization, well interpretation of intermediate results, such as analyzing plots and drafting captions.
LLMs were especially helpful for rapidly prototyping visualizations by providing data and iterating on plotting ideas interactively at the drafting phase. In several cases, they were used to generate the start version of the code, and then reviewed and modified to ensure correctness. Moreover, all such scripts were reviewed and verified by us. We treated generated code as drafts and retained only what interpretable, correct, and replicable.
We also used LLMs to assist in writing portions of the experiment code, such as data collection logic in our apperiment-runiingg system. The core logic was written by the authors and all code was reviewed, tested, and finalized by us.
We also experimented with LLM-assisted initial drafting of the related work section, but due to factual inaccuracies and limited coverage, we abandoned the LLM-generated content and wrote the section manually. We did not use any LLM design experiments, develop hypotheses, or formulate scientific claims.
We reviewed and finalized all the contents, and take full responsibility for the submission.
B EQUIVALENCE OF NORMALIZED VS. STANDARD DP UPDATES
We analyze with the standard (unnormalized) variant when clarity of the DP noise's dependence is needed, and we train with the normalized variant for simpler hyperparameter tuning. The two mathematically equivalent reparameterizations.
Standard (unnormalized) update In standard DP-SGD (Abadi et al. 2016), the learning rate and the clipping bound C interact
g. 9: min (1. (111s
C
0.41 0g
Ust
Σ D. + oCE i
&~ N(O,
[B]
where & E Id with d the number of trainable parameters, Ia E Rdxd is the identity, and |B] is the expected batch size.
Normalized update In normalized DP-SGD De et al. 2022). normalization is embedded into clipping
9: 9: min (c 1 (laulle) θ.41 0g
Tnarm
Σ vEB
To
of
|B]
Here, C sets sensitivity, while Thorm sets the step size.
Equivalence and accounting The two forms equivalent under the reparameterizztioo Thorm CTja.- This mapping yields identical updates for the same (g, T, C), and the privacy accounting is unchanged. We therefore use the standard form in derivations where explicit C- dependence of the DP noise is useful (e.g., Section and the normalized form in experiments for simpler tuning.
C EXPERIMENT DETAILS
C.I HYPERPARAMETER GRID DETAILS
We fine-tune all image models using FiLM (Perez et al. 2018). following the DP-FiLM setup of Tobaben al. 2023), where train only the scale and bias of normalization layers and the
15

---

Published as a conference paper at ICLR 2026
Table AI: Hyperparameter grids used for each dataset and model. Method denotes the training method, e.g. FiLM, or from scratch training. Batch sizes are powers of two, with full-batch included. Learning rates are drawn from geometric ranges (number of values shown in parentheses). Clipping bounds are explicitly enumerated.
Model
Method
Subset
Batch Sires
Learning Rates (grom.)
Clipping Bounds
CIFAR-100
ViI-Tiay
DP-Adam,
29_214
[0,001, D.02] (S)
1e-5, 1o-4, 16-5, 1e-2, 1,
CTFAR-100
ViT-Bese
DP-Adim, RIM
1.0
24-214 + full
[0.001, 0.02] (8)
Ie-5. 1e-3, 1e-2, 1,
CIFAR-1O0
ResNe: 50
DP-Adam, FILM
2 2--224 + fall
[0.0005, 0,01] (10)
le:S. le-4, Ie-3, Je-2, 0.1, 1.
CIFAR-100 100 ( 100)
ViT-Tiny
DP-Adam, RLM
0.1
25-219 Fall
[0.001, 0.03] (8)
1e-4, 1e-2,
1,
100 (00%
Base
DP-Adam,
2 29-221 full
le-4 2.-
CIFAR-100 (10%)
Re-Nor500
DP-Adam, RIM
0.1
2"-21 Full
[0.0005, 0.01] (10)
le-3, 1e-2, 0.1, I, 10, 26,42
SUN397
Vir-Tay
DP-Adam, FIM
1.0
28-244 4 full
[0.001, D.02] (8)
0.1, 1, 10, 12, 14, 18, 26,
SUNW?
ViT-Rese
DP-Adam, RIM
2"-21" full
[0.001, 0.02] (8)
0.1, 10,
SUN397
Vir-Tay
DP-Adam, LORA
1.0
28-220 + full 2h_210 fill
(0.005. 0.025] (8)
0.1. 10, 12, 14, 18, 26,
SUN397
ViT-Rise
DP-Adam, LoRA
[0.005, 0,025]
10, 42.
SUN397
ViI-Tay
DP-Adam, Full
2²_242
[0,0001, 0/002] (30)
1o.3, 1e-2, 10, 29.5, 34.22,
40.03, 46. 84, 54.79, 64.11,75. 75.0
SUN397
ViT-Bise
DR-Adam, Full fine-tune
28_212 a full
[0.0001. 0.002) (10)
1e-2,
29.5, 34.22,
40,03, 46. 46.84, 54.79, 66.11,
Coeaava
ViT-Tey
DP-Adam, FILM
1.0
25-214 full 2 2 full
[0.001, 0.02] (8)
0.1, 10, 26, 42
ViI-Base
DP-Adam, HLM
[0,01, 0.022
0.1, 10, 26,
Newagroups
DisiBERT
DP-Adam, LORA
2"_243
[0,001, 0.02) (8)
1o-3, 16-2, 0.1, 10.
CIAAR-10 TO
WKN-16-4
DP-Adam, From scratch
1.0
_ + full
10.0001, 0.011 (10)
1e-5, 10, 50
classification head. Following standard practice, we fine-tune language models with LoRA (Hu cct al. (021a), using rank 16 adapters on the query and key matrices of all attention layers. We use maximum sequence length of 512, which is the DistilBERT (Sanh al. 2019) default.
We perform a structured grid search over three hyperparameters: learning rate, batch size, and clipping bound. For each model and dataset, define geometric learning rate range (with number of values in parentheses). a discrete set of clipping bounds, and a power-of-two batch size sweep up to full-batch training. Subset size refers to the fraction of the training set used.
For datasets that include a validation split, we merge the training and validation sets prior to any subsampling, and train on the combined set. Final performance is reported on the test set.
We do not report all evaluated clipping bounds in the main paper; values unimportant for our results omitted for clarity.
C.2 DATASETS
We use four standard image classification datasets: 1000 CIFAR- (10%), SUN397, and Cassava. We also evaluate 20 Newsgroups Mitchell (1977 text classification dataset. All are publicly available through either the Hugging Face datasets library tensor flow_ _lww_datasee__eeeeeeeeeeeeeeess datasets. We use canonical train/validation/tes splits, and when a separate validation set exists, we merge it into the training set.
16

---

Published as a conference paper at ICLR 2026
Table A2: Public datasets used in our experiments. All are available via Hugging Face TensorFlow Datasets.
Dataset
Source
Description
CIFAR-100
Krizhevsky & Hinton 2009)
Balanced dataset natural images with 100 classes and 50k training examples.
CIFAR-100 (10%)
Krizhessky & Hinton
20099
Stratified 10% random subset of CIFAR-100 used to study limited data regimes.
SUN397
Xiao el al. (2010)
Large-scale scene recognition dataset with 397 classes and roughly 87k training images.
Cassava
Mwebaze et al. 2019
Imbalanced plant discase classification dataset with 5 classes and ~66 examples. Considered out-of- distribution relative ImageNet pretraining.
20 Newsgroups
Milchel 1997)
Balanced text classification dataset roughly 18k Usenet posts across 20 topics, commonly used for hench- marking document categorization.
C.3 EXPERIMENT ENVIRONMENT
Our experiments are performed on the clusters, which equipped with AMD EPYC Trento CPU and AMD MI250x GPU.
C.4 RETAINED WEIGHT
For Fig. (top), we train the DP model with the best hyperparameters under three different clipping bounds, and use C as a baseline. Models are evaluated eval mode.
To analyze gradient clipping, capture the classifier activations and compute per-sample gradients via a backward pass, and calculate retained weight according to Eq. (3). In the plots, classes are ordered by per-class accuracy. All results seed 1.
D KOLOSKOVA ET AL. (2023) BOUND
Koloskova et al. 2023 bound
The full bound the expected gradient norm from Koloskova et al.
(003)) is.
Ln opp + Ln + minfor, C
of nL 1 B
o2 +
Fo nf''
FJ C2
O
where Fo = [(Og) - L()') is the difference in loss evaluated at initial weights and at the optimal weights. Denoting dDP Co makes the dependence on clipping bound (and noise level) explicit.
As the bound depends on the unknown optimal weights, any derived bound with respect to C' cannot be determined advance. Moreover, in contrast our work, this bound does not model how the gradient norm distributions change the privacy level changes.
E OMITTED PROOFS
Optimal clipping from MSE For convenience, we first restate the theorem and then give the proof.
As far as we know, optimal clipping in the sense of minimizing the MSE has not been derived in any existing work. The closest existing work we are aware of is Amin et al. (2019), who derive an upper bound for MSE for pure DP = 0) using Laplace noise. We instead focus on the Gaussian mechanism, and derive exact MSE, not an upper bound.
Assumption 5.1. Assume there is mini-batch sampling noise, constant clipping with C' E [min; || max, [[9.]]]. and the Gaussian mechanism with noise level o privatize the sum of clipped gradients.
standard per-sample
17

---

Published as a conference paper at ICLR 2026
Theorem 5.2. Under Assumption 5.7 optimal clipping constant C* that minimizes the mean squared error between the per-sample clipped DP gradient 9 and the true gradient 9 for a fixed mini-batch satisfies
c=1[1e C² -
[1g:l] for some i, or
No. Ge No. Ner+o2d"
where d is the dimensionality of the gradient, Gc := EiEle 9i and Nc == Eiele |asi and
Ic {i: 11:11 ) denotes the indices of the clipped gradients.
Proof. Let 9:, = 1,...,
, B denote the true per-example gradients in a given mini-batch, g. the corresponding clipped gradients, and the DP noise. The mean squared error of the total clipped noisy mini-batch gradient as function of C
MSF(C) =
Σ Di + E
Σ
9t
Σ 9:
Σ
gi
(5)
I C
|g
C
-2d
Σ
C²o
Σ ieTc
9:
Ji
|9
1=11
where Ic {i: |1 C} denotes the indices of the clipped gradients. Noting that MSE(C) is continuous function defined on bounded interval [min, I19:l1, max; 1| II]. there exists minimizer
C If C * lgg|l for all then C* (Ils)ll I]) for some i, j. On this open interval, MSE is differentiable with the derivative
T
dMSE(() 2Co dC
- iele
Fi
Σ Te
||ail|
C
||
119ill
gi
(6)
T
=Cood
Σ 9i Aele
2C
Σ
9i
Σ
9t
||gi||
|
|
HEle
iele
iele
By denoting Go := EiElo 9i and No = Eiele
St |ss.ll
this simplifies to
dMSE(C) dc 20 2d +222 -2
The zero of the derivative C* satisfies
Ge. C= No. No- o2d
(8)
which concludes the proof.
Connecting MSE to optimization progress restate the theorem and finally continue with the proof. Definition E.1 (Smoothness). A differentiable function f is L-smooth, if
We start by stating some helpful standard results, then
[V/(x) - V V((ii]l & L||x vll. Va, TRd.
Assuming smoothness, we have the well-known quadratic upper bound given next in Lemma E.1 Lemma E.1 (Quadratic upper bound). Assume f f + IR is L-smooth on convex domain Q. Then
f(y) < f(x) (Vf(x), y - 2) + 2 llv x012.
(10)
Proof. See any standard lecture notes smooth optimization.
18

---

Published as a conference paper at ICLR 2026
Assumption 5.3. Assume L-smooth loss function L (Definition E.1 and that want to minimize
via gradient descent, using step size 1j S L. and the
clipped and noisy DP gradients
9r = Σ; 9: (1) {()) at step t l...... T instead of the true gradients 9t E.9. 9h VL((e).
Theorem 5.4. Under Assumption 5.3 the expected improvement in the loss from step t + 1 is
bounded as
E[[(0++1) & ((),)
" (IV())) 2
" (C)
(2)
Proof. Let ex(C) Ør It. i.e., the error when using C' for the per-sample clipping at step 6, so (C) MSE, (C)) as defined in Eq. (5) at step In the following, we write e ex)C) for
brevity without any risk of confusion.
We have
L
[(0,+1) S ((0+) 0,t1 01)
11 0e11
(11)
2
Lyp²
$ ((a)) vo + +.)) +
119e + es 112
(12)
2
Lap
s ((.)) 7119:112 m/g,,
+
(llge 1l + 2(gr, ex) + (lex11))
(13)
2
[(O.) ( (Lnp² -1) 11a.11 (In/² 1) (gi, et)
Lnp llee1
(14)
[nn? -1) |1ael12 + (Ln) 1)) | e1)
Lnp
S c(0)) +
|ee
12
(15)
where we first use Lemma 1 with the assumption that 9: VL().), then note that 0 -n(gt ex), and expand the terms to get Eq. (13).
0
To bound the cross-terms, we have Kyr, ex)l S |19.112 le 22 S 1 + f 12, due to Cauchy-
Schwarz and the elementary inequality 2ab S n² 12. Since we assume 7) S T/T. + nT. < we Ln²² Continuing from Eq. 15
have Laj² 71 (LT) 1) s 0, and therefore LT)² n|
Lap²
|19a 12 + |
Lij²
[(1r+1) s <L(.))
2 -7)
11ae112 112 + (1 Ln²))
1
(16)
2
2
L(())
" 99 112
2 | le 1
(17)
2
Taking expectations both sides, writing the current parameter value 6 explicitly as conditioning the expectation, and using the assumption that g. VL(0,) we finally have
E[C(0++))00] = ((0.) " [[V(())) + " MSEE(()
(18)
2
which concludes the proof.
19

---

Published as a conference paper at ICLR 2026
F ACCURACY BY MODEL AND BY PRIVACY BUDGET F.1 BY MODEL (FIXED PRIVACY BUDGET)
Dataset: CIFARI00 (10%)
chedee oot
0.500
brest
-0.04
accuracy to
-0.500
_0.008
-0.0125
0.530
-0.0150
-0.3175
Sate
-0.14
VT
-0.15
(a) Clipping bound vs. accuracy comparing ViT-Base ViT-Timy on CIFAR-I00 (10% subset, 32 epochs, 10 >) at 1.0.
(b) Clipping bound vs. accuracy comparing ViT-Base and ViT-Tiny on CIFAR-100 (10% subset, 32 epochs. 5 = 10 ") at c 8.0.
Figure A1: Clipping bound accuracy comparison CIFAR- 100 (10% subset)
Dataset: CASSAVA
(105) caaros D03 pot pet opacte-32
--3 curme ECCE poh patt-tir apeche-32
0.300
0.000
-0.355
0.065
6.300
-0.13
accuracy ba
-6.335
0.635
6.330
-4:225
-02220
Basc
0.50
-0.235
MTFliry
(a) Clipping bound vs. accuracy comparing ViT-Base and ViT-Tiny on Cassava (full dataset, 32 epochs, 8 = 10 5) c 0.5.
(b) Clipping bound vs. accuracy comparing VIT ViT-Base and ViT-Tiny on Cassava (full dataset, 32 epochs, å 10 at 8.0.
Figure A2: Clipping bound vs. accuracy comparison on Cassava
20

---

Published as a conference paper at ICLR 2026
Dataset: CIEAR100 (100%)
-0.05
-0.00%
-0.230
-0.003
-0.33
-0.05
VTBaas
WTBass
VITMlny
WiTleyy
-0025
(a) Clipping bound vs. accuracy comparing VIT-Base and ViT-Tiny 110 (full dataset, epochs, 8 at 0.5.
(b) Clipping bound vs. accuracy comparing VTT-Base Base and ViT-Tiny CIFAR- 100 (full dataset, epochs, 8 10 5) 8.0.
Figure A3: Clipping bound vs. accuracy comparison on CIFAR 100 (100%)
Dataset: SUN397
e030 sydtt coccett
0.500
-0.0023
-0.001
brest
-0.3050
-0.02
accuracy to
-0.2325
-0.05
6.3158
Vare
4.006
MT
-0.007
(a) Clipping bound vs. accuracy comparing ViT-Base and ViT-Tiny on SUN397 (full datasct, 8 epochs, = 10 ) at
(b) Clipping bound vs. accuracy comparing ViT-Base and ViT-Tiny on SUN397 (full dataset, 8 epochs, 8 10 at 8.0.
0.25.
Figure A4: Clipping bound accuracy comparison SUN397
21

---

Published as a conference paper at ICLR 2026
F.2
BY PRIVACY BUDGET (FIXED MODEL)
Dataset: CIFARIO0 (10%)
INOMECE CENSRO 3PAN pat--ii 180077--3
ViTaaa chisks (2551 pet--iin spocke-12
1,00
6.300
-1,35
-4.7225
-8.1550
1,10
-0.322
3,15
-1029
(a) Clipping bound vs. accuracy ResNet-50 on CIFAR-100 (10% subset, 32 epochs, 6 10 Privacy settings: E (2, 8}.
(b) Clipping hound vs. accuracy ViT-Base on CIFAR-100 (10% subset, 32 epochs, 5 10 B).
Privacy settings: E (1,
10001
3.611-
-10023
-1.033
1-10
(a) Clipping bound vs. accuracy for ViT-Tiny CIFAR-100 (10% subset, 32 epochs, 5 10 10-
Privacy settings: (1,
Figure A6: Clipping bound vs. accuracy comparison CIFAR100 (10%).
22

---

Published as a conference paper at ICLR 2026
Dataset: CASSAVA
Whibed: canere (1MPS)) petoriie postar
1,30
6.350
-0.35
-1025
-0.22
4.000
-0.23
-6.23
-1055
0.375
-0332
-0.335
(a) Clipping bound vs. accuracy for ViT-Base on Cassava (full dataset, 32 epochs, 10 9), Privacy settings: E {0.25, 8/.
(b) Clipping bound vs. accuracy for ViT-Timy on Cassava (full dataset, epochs, 10 "), Privacy settings: E {0.25, 8}.
Figure A7: Clipping bound vs. accuracy comparison on CASSAVA.
Dataset: CIFAR100
Widee (1t000/0 rppeevd
witthe ofer109 SCRN
-0.354
+4010
-1015
-8.333
-3.00
1-05
-0.322
m5
(a) Clipping bound vs. accuracy for ViT-Base CIFAR-100 (full dataset, epochs, 8 10-"), Pri- vacy settings: e E (0.5,8).
(b) Clipping bound accuracy for ViT-Tiny CIFAR-100 (full dataset, 8 epochs, 6 10 10- Pri-
vacy settings: E {0.5,8 8}.
Figure A8: Clipping bound vs. accuracy comparison CIFAR100 (100%).
Dataset: SUN397
-20335
-0.3025
20952
-8.1550
-8.3075
2A102
-50023
63350
mm0
40175
1-50
-8:3275
23
(a) Clipping bound vs. accuracy for ViT-Base on SUN397 (full dataset, 8 epochs, 8 = 10 Privacy
(b) Clipping bound vs. accuracy for VIT ViT-Tiny on SUN397 (full dataset, 8 epochs, = 10 3). Privacy settings: {0.25, 8).
settings: e {0.25,8 8).
Figure A9; Clipping bound vs. accuracy comparison SUN397.
23

---

Published as a conference paper at ICLR 2026
G FINE-TUNING WITH LORA
In this appendix, we will study fine-tuning with LoRA and its effects on the clipping bound.
G.1 IMAGE CLASSIFICATION WITH LORA
We evaluate fine-tuning image models with LoRA. We fine-tune ViT-Tiny (6=1, hard) and VīT-Base (c=8, easy) on SUN397 using rank-1 LoRA adapters on all attention layers, training only the LoRA parameters and the classifier head. After performing an initial sweep over LoRA ranks 1, 4, 8, 16 while running HPO the other parameters, we opter for rank-1 LoRA adapters they gave the best performance (see Table A3
We DP-Adam with the normalized update, Poisson subsampling, PRV accounting, and d 10 running for epochs. We jointly tune learning rate, batch size, and clipping bound for each case (see Appendix C.I for details).
Fig. A10 shows the effect of the clipping bound on test accuracy. the easy setting (ViT-Base, 8), performance is remarkably stable across wide range of clipping values, with small C performing best. In contrast, in the hard setting (ViT-Tiny, & = 1) benefits from larger clipping bounds, with accuracy improving C' increases. Under higher bearning-problent difficulty, the optimal clipping bound shifts upward.
To further analyze if the findings match analysis, in Fig. All plot the per-class relative retained weight and the resulting per-class accuracy change. In the easy ViT-Base case, all clipping bounds result in remarkably stable accuracy. However, in the hard (ViT-Tiny, ==1 case, small clipping bounds reduce retained weight for the hardest classes, while C = 10 preserves more gradient signal and improves their accuracy.
Finally, Fig. A12 provides the full accuracy grid over (B,C) C) for both models.
sun397 (100%)
peft=lora
epochs= ViT-Tmy (e 1.0) ViT-Base (s 8.0)
0.000
Accuracy (est(s)
-0.005
-0.010
Privacy level Vin 1.0
-0.020
VIT-Base, 8.0
-0.025
0.01
Clipping bound
Figure A10: Effect of clipping bound on image classification with LoRA. Accuracy difference to the best clipping bound each for SUN397 fine-tuning with rank Lora adapters, epochs, 10-3 The hard configuration (ViT-Tiny, e 1) benefits from larger C, while the easy configuration (ViT-Base, 8) clearly prefers smaller clipping bound.
24

---

Published as a conference paper at ICLR 2026
SUN397, 8, LoRA
Asceurcy Relative Retained weight
ViT-Base (Easy)
ViT-Tmny (Hard)
1.0
1.0
0.5
0.3
0.0
0.0
100
200
300
100
200
300
400
0.1
0.0
0.0
-0.1
-0.1
100
200
300
400
100
200 300
400
Class rank (easy to hard)
10.0
Figure A11: Per-class relative retained weight and per-class accuracy change for SUN397 with LoRA rank I adapters, 8 epochs, 5 = 10 and E 8. Left: ViT-Base, ==8(easy); right: ViT-Tiny, s=1 (hard). Classes are ordered from easy to hard by per-class accuracy. Top: relative retained weight. Bottom: per-class accuracy difference relative C 1. For ViT-Tiny, small C reduces retained weight for hard classes and degrades their accuracy, while C = 10 preserves their gradients and improves performance.
sun397 (100%) - -eet-lora--pochs-s -
ViT-Tiny vs ViT-Base
ViT-Tiny (E = 1.0)
ViT-Base (e 8.0)
Batch size
240
0.797
0.507
0311
0318
0.75
0.19
0.59
0.747
216
0.294
11306
0312
6316
0744
0746
6.44
0.743
212
213
0.297
11500
0.5
0316
03007
0312
0.52
0.743
0744
0.241
0.25
0,735
210
0.409
0.464
28
0.56
0.261
0.395
0.375
0.001
0.1
10
0.001
0.1
10
Clipping bound
Figure A12: Accuracy for each configuration of batch size and clipping bound when fine-tuning
SUN397 with DP LoRA (8 epochs, å = 10- Left: ViT-Tiny with 1; right: ViT-Base with & = 8. In the hard regime, the best configurations consistently use larger clipping bounds (C ~ 10),
whereas the easy regime remarkable stable across C's (see Fig, A10).
G.2 TEXT CLASSIFICATION WITH LORA
We further validate our observations on text classification task using parameter-efficient fine-tuning. Following standard practices, we fine-tune DistilBERT [annh et al. (2019) on the 20 Newsgroups dataset [Mtchelll (1997) using LoRA with rank-16 adapters (see Appendix C.I for details) for epochs. We use maximum sequence length of 512 (model default). We use DP-Adam with the normalized update, Poisson subsampling, and 10-5 while varying the clipping bound and privacy budget e.
Fig.AT3 AT3 shows that, although the absolute accuracy differences between clipping bounds are small on this task, different clipping choices still affect performance
theory suggests. Fig. A14 explains these differences using relative retained weights, revealing the same underlying mechanism: small clipping bound suppresses signal from hard examples, especially under higher learning-problem difficulty. We speculate that the accuracy differences would become larger with a more difficult task or if switching a lower capability pretrained model.
problem
Finally, Fig. A15 reports the full accuracy grid over all configurations. We always tune the learning rate separately for each configuration.
25

---

Published as a conference paper at ICLR 2026
Test accuracy of LoRA with sweep over ranks. ViT-Tiny on SUN397, epochs, 6=10- Each row shows the accuracy results from the best configuration found by HPO over learning rate, batch size, with clipping bound fixed to reduce HPO costs. Higher ranks can degrade performance, especially under tighter privacy.
Table A3:
LoRA rank
Accuracy at c-0.5
Accuracy at ==8
1
0.4107
0.6512
2
0,4020
0.6514
4
0,4055
0.6535
0.3924
0.6482
16
0.3800
0.6481
Accuracy - best(E)
distilhert-hhxe-uucasedd 20 newsgroups (100%) peft=leoarraaao. epochs=8
0.000
-0.005
Privacy level
1.0
0.010
8.0
Clipping bound
Figure A13: Effect of clipping bound on text classification with LoRA adapters. Accuracy difference to the best clipping bound for each privacy level, DistilBRRT Sanh et al. (2019) on 20 Newsgroups dataset Mitchell (1997) with rank 16 LoRA adapters, 8 epochs, and à = 10 Lines indicate median three repeats with max bands. Tighter privacy (e 1) shifts the optimal clipping bound towards larger C, while looser privacy (E 8) prefers a smaller bound, consistent with our findings.
H AUTOMATIC CLIPPING COMPARISON
We compare against the automatic clipping method proposed by Bu et al. (2023). Specifically, use the AUTO-S variant, which the authors report to outperform AUTO-V V. and set 0.01 as recommended by the authors. The results in Table A4 correspond to those shown in Fig. providing exact accuracy values across datasets and privacy levels.
26

---

Published as a conference paper at ICLR 2026
20 20-newsgroups, distllher--ass-uucaaee LoRA
Aecuracy Relatiee Retainod weight
(Easy)
(Hard)
1.0
1.0
0.5
0.0
0.0
0.00
0.0
5 10
16
20
10 15 20
Class rank (easy hard)
Figure A14: Per-class retained weight and accuracy for text classification with LoRA adapters on 20 Newsgroups dataset Mitchhell (1997) using the DistilBERT (aanh et al. (2019), rank 16 LoRA adapters, 8 epochs, = 10 Classes are ordered from easy to hard by per-class accuracy. The batch size and the learning rate are tuned separately for each case. Top: change per-class accuracy relative to 1 Bottom: relative retained weight for different clipping bounds. Under tight privacy (s = 1). small over-clips gradients from the hardest classes, while larger C' preserves more of their signal and improves their accuracy.
DistilBERT - 20 newsgrrups (100%) peft=lora epochs=8
e=1.0
8=8.0
Batch size
Full
0,482
0.482
0.46
0.513
0.513
0.542
2
0,517
0.515
0.596
0.6
2
0.515
0.523
0.532
0.615
0.616
0.621
210
0.552
0.56
0.562
0.633
0.636
0.632
2
0.549
0.557
0.562
0.61
0.613
0.615
0.1
0.1
10
Clipping bound
Figure A15: Absolute accuracies achieved with each configuration for DP LoRA fine-tuning 20 Newsgroups dataset Mitchell] (1997) DistilBERT Sanh et al. (2019), rank 16 LoRA adapters, epochs, 8 10 Learning rate is tuned separately for each case.
I OTHER TRAINING METHODS
In this appendix, extend the analysis in Figure 4 by examining the per-class effects of gradient clipping under three additional training regimes: (i) training from scratch, (ii) DP-SGD, and (iii) full-parameter fine-tuning.
For comparability across models and optimizers, we plot the relative retained weight after clipping for each class, normalized by the class with the highest retained weight under the baseline setting (C=1). This metric quantifies the fraction of gradient signal preserved after clipping. Flatter and higher curves reflect more uniform gradient preservation across classes, while steep drops indicate that difficult classes lose gradient mass disproportionately, Excessive clipping therefore suppresses learning for these classes, whereas sufficiently large clipping bounds maintain gradient signal even as task difficulty increases.
I.1 TRAINING FROM SCRATCH WITH DP-ADAM
We examine training from scratch to assess whether the same clipping effects arise a setting without pre-trained features. We train WideResNet-166 16-4 (Zagoruyko & Komodakis 2016) CIFAR-10
27

---

Published as a conference paper at ICLR 2026
Table A4: Comparison of AUTO-S (with 7 0.01) against our best-tuned clipping bound across datasets and privacy levels. Each row shows mean macro accuracy with [min, max] across three random seeds. These are the absolute accuracies for the lines plotted in Fig.
Dataset
AUTO-S
Tuned (Ours)
SUN397
0.5
0.403 [0.399, 0.405]
0.430 [0.428, 0.434]
SUN397
0.510 [0.508, 0.512]
0.536 [0.534, 0.537]
SUN397
2
0.584 [0.581, 0.586]
0.601 [0.598, 0.603]
SUN397
4
0.628 [0.624, 0.631]
0.639 [0.636, 0.642]
SUN397
8
0.650 [0.645, 0.655]
0.660 [0.655, 0.663]
Cassava
0.5
0.604 [0.584, 0.620]
0.637 [0.629, 0.648]
Cassava
0.699 [0.691, 0.709]
0.709 70.701, 0.718]
Cassava
0.754 [0.752, 0.756]
0.759 [0.757, 0.762]
Cassava
4
0.795 (0.792, 0.801]
0.797 (0.792, 0.802)
Cassava
8
0.812 [0.799, 0.825]
0.814 [0.805, 0.827]
CIFAR 100
0.5
0.781 [0.781, 0.782]
0.786 [0.784, 0.788]
CIFAR 100
1
0.812 10.811, 0.814]
0.815 [0.815, 0.817]
CIFAR 100
2
0.830 [0.828, 0.832]
0.832 [0.831, 0.833]
CIFAR-100
4
0.841
[0.840, 0.841]
0.842 [0.841, 0.843]
CIFAR-100
0.846 [0.845, 0.847]
0.847 [0.847, 0.848)
CIFAR-100
SUN397
10
Optimal Batch Size
Optimal Batch Size
10"
10
10a
epocbe4
upocte8
epoch-16 16
10a
0.25
0.5
2
0.25
0.5
Epsilon
Figure A16: Optimal batch size under fixed-epoch DP-Adam on CIFAR-100 and SUN397 with ViT-Tiny and FiLM (6 = 10-5; NEIFR 100 50,000, NsuN397 87,002). Curves show means across epochs and e, with shaded areas indicating variation over 3 seeds. The optimal batch size increases with training epochs, most steeply at low epochs. Smaller favor smaller batches, while gains from very large batches taper off as E or epochs grow.
for 8 epochs using DP-Adam without any PEFT method (namely, parameter-efficient layers; see Appendix [C.1 for details), varying the clipping bound C' and privacy budget s. Figure A18 summarizes the accuracies achieved under each hyperparameter configuration.
Despite the simplicity of the task, different clipping bounds still lead to measurable accuracy differ- ences. Figure AA7een explains these differences through per-class retained weights. As in our other results, small clipping bounds disproportionately suppress gradients from the hardest classes, while larger bounds preserve more signal and improve their accuracy, especially under higher learning- problem difficulty.
28

---

Published as a conference paper at ICLR 2026
CIFAR-IO, wm-16-4
Relative Retained weight
1.0
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
0.0
10
0.1
0.1
0.1
accuracy
0.0
0.0
0.0
0.0
-0.1
-0.1
0.1
0.1
10
10
5
10
5
10
Class rank (easy to hard)
Figure A17: Relative retained gradient weight per class on CIFAR-10, 10, under increasing task difficulty (left right). The model is WideResNeet1666 trained for 8 epochs from scratch using DP-Adam and FiLM (Perez et al. 2018); 8-10-
wmn-
16-4
cifar 10 (100%) peft=none epochs=40
e=1.0
e=2.0
2
0.497
0.517
0.519
0.537
0.567
0.542
Batch size
0.491
0.513
0.516
0.455
0.533
0.557
0.561
0.536
2
0.498
0.518
0.519
0.45
U.536
0.555
0.561
0.539
2
0.499
0.517
0.518
0.438
0.527
0.552
0.554
0.52
le-05
1e-05
10
e=
8=8.0
21
0.59
0.614
0.623
0.599
0.646
0.67
0.667
0.645
Batch size
2.0
0.592
0.61
0.607
0.58
0.633
0.653
0.651
0.625
29
0.576
0.596
0.604
0.586
0.602
0.626
0.631
0.613
28
0.56
0.581
0.588
0.562
0.587
0.607
0.613
0.592
le-05
43
Je-05
10
Clipping bound
Figure A18: Accuracy achieved with each configuration when training WideResNet-16-4 from scratch on CIFAR-10 with DP-Adam; epochs, 10 The batch size and learning rate are tuned separately for each E and clipping bound C.
I.2 FINE-TUNING WITH DP-SGD
We evaluate the behavior of clipping under DP-SGD, focusing on ViT-Tiny fine-tuning SUN397. We train for 8 epochs using DP-SGD with FiLM, Poisson subsampling, and = 10-5, while varying the clipping bound C and the privacy budget e. Figure A20 summarizes the accuracies obtained across the hyperparameter configurations.
Consistent with our earlier findings, Figure A19 shows that small clipping bounds reduce the retained gradient weight for the hardest classes, especially under high probbleg difficulty. Larger clipping bounds preserve more class-specific gradient signal and thus lead improved accuracy for these harder classes, demonstrating that the same mechanism appears also with DP-SGD.
29

---

Published as a conference paper at ICLR 2026
SUN397, DP-SGD, ViT-Tiny
Jeceuacy Relative Retained weight
(Easy)
0.25 (Hard)
1.0
1.0
0.5
0.3
0.0
100
200
300
100
200
300
400
0.1
0.0
0.0
-0.1
100
200
300
400
100
200
300
400
Class rank (easy to hard)
Figure A19: Per-class relative retained gradient weight on SUN397 with ViT-Tiny trained for epochs using DP-SGD with FiLM; 8 10-5, Classes are ordered from easy hard by per-class accuracy. Small clipping bounds disproportionately suppress gradients for the hardest classes, while larger bounds maintain more signal and improve their performance under higher task difficulty.
ViT-Tiny sun397 (100%)
peft=film epochs=8
c=0.25
==1.0
8=8.0
16834
0.281
0.284
0.299
0.307
0.305
0.432
0.444
0.486
0,462
0.474
0.458
0.472
0.527
0.485
0.501
Batch size
22
0.269
0.271
0.284
0.299
0.526
0.528
0.582
0.619
0.632
0.629
200
0.272
0.274
0.291
0.304
0.305
0.453
0.465
0.499
0.515
0.513
0.56
0.573
0.592
0.59
0.594
24
0.274
0.277
0.291
0.298
0.295
0.425
0.434
0.458
0.455
0.447
0.501
0.51
0.537
0.556
0.555
1e-05
0001
76
42
16-05
0.001
1e405
0.001
26
Clipping bound
Figure A20; Accuracy across hyperparameter configurations for DP-SGD training. We tune the learning rate separately for each configuration.
1.3 FULL FINE-TUNE
Here we evaluate if our findings hold for full fine-tune; meaning that instead of employing parameter- efficient fine-tuning (PEFT) method, we fine-tune all the model parameters. We find that full fine-tuning is different from PEFT fine-tuning: the clipping bound does not clearly depend on task difficulty-inseaad very large clipping bound seems be the most effective, both for easy and hard tasks, which is illustrated in Fig. A22 We speculate this is due to the increase in the number of trainable parameters: with PEFT methods we train just few percent of all the parameters depending on the method and classifier head size, whereas with full fine-tune we train all the parameters (50 x to 200x more parameters), leading an increase in average gradient size. Fig. A2T depicts to situation in terms of relative retained weights. Even in the hardest case (ViT-Tiny, c=0.5) the gradients of the most difficult classes are not over-clipped with the smallest (useful) clipping bound.
Fig. A22 shows the full grid results, demonstrating that both the easy and hard classes benefit from a large clipping bound. As the amount of noise increases (smaller E) the model starts to prefer a slightly smaller (though still large) clipping bound. We explain this in Figs. to A25 which show the gradient norm distributions for three clipping bounds picked from the grid from a reasonable range around the optimum. In all cases most of the gradient norms way above the typical clipping
30

---

Published as a conference paper at ICLR 2026
SUN397, Train All Parameters
accuracy Relative Retained weight
VIT-Base, c
VIT-Tiny, =
VIT-Tiny, = 0.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
200
400
0.05
0.02
0.02
0.00
0.00
0.00
0.02
200
400
200
400
0
200
400
Class rank (easy to hard)
40.03332
54.
Figure A21: Per-class effects of gradient clipping under increasing task difficulty (left to right), SUN397, 8 epochs; 8=10-5 Class rank is based on sorting classes according their per-class accuracies. Top: relative retained weight after clipping. computed per class and normalized to the class with the largest retained weight in the baseline (C=54.795), The effective clipping bounds are large enough- duggh due to the increase in the average gradient size to keep even the hardest classes gradient signal alive.
bounds used (e.g. 0.1, 1.0) and even above the clipping bounds that have used in this paper for the hard cases (e.g. 10, 26, 42). We speculate that with smaller models (fewer trainable parameters), the easy-vs-hard effect will reappear, as hinted in our from-scratch experiment with WRN-16-4 (see Appendix I.I.
Lastly, we note that training large models with full fine-tune is costly and typically leads it bets to on-par accuracy with PEFT methods Tobaben et al. (2023) or even degraded accuracy compared, especially under tight privacy Ke et al. 2024 Tobaben et (2023).
sun397 (100%) - peet=none epochs=8 ViT-Tiny ViT-Base
ViT-Tiny (E 0.5)
ViT-Base (E 8.0)
Batch size
Full
0.145
0.145
0.146
0.146
0.14
0.147
0.142
0
0.454
0.454
0.454
0444
0.451
0:454
0.455
0,458
245
0,213
0.216
0.217
0218
0.217
214
0.229
0.23
0.232
0.252
0.234
0.236
0.256
0.27
0.684
0.685
0.688
0.689
0.69
0.692
0.664
0.696
212
0.246
0.511
0.667
0.673
2.0
0.23y
0.24
0.241
0.243
0.444
0.346
0.246
0.2
01001
0.004
0.006
0.009
0.614
0015
0.609
0.622
2a
0.217
0217
0218
0.219
0.2
0222
0.222
0217
0
0.313
0,512
0,523
0,525
25
29.25
34.22
40.00
46.84
54.79
64.11
29.25
34.22
40.03
46.84
54.79
64.11
75
Clipping bound
Figure A22: Comparison of the accuracies achieved over (B, C) grid with hard (ViT-Tiny, 0=0.5) and easy (ViT-Hard, ==8) tasks while tieettning all the parameters, SUN397, 8 epochs; 8=10 Learning rate is tuned separately for each configuration. Due to the increase in average gradient size, the optimal clipping bounds are much larger than with PEFT fine-tuning. With full fine-tune hard tasks would benefit from even larger clipping bounds, but these start deteriorate learning due the increase in injected noise.
31

---

Published as a conference paper at ICLR 2026
Figure A23: Histogram of gradient norms in easy (ViT-Base, 5=8), medium (ViT-Tiny, e 8), and hard (ViT-Tiny, ==0.5) tasks, C=40.03; SUN397, 8 epochs; 6=10-5, The average gradient norm distribution has mass way above the usual clipping bounds.
VIT Rac
vitTir
Vittie
Figure A24: Histogram of gradient norms in easy (ViT-Base, ==8), medium (ViT-Tiny, e = 8), and hard (ViT-Tiny, e=0.5) tasks, C=54.8; SUN397, 8 epochs; 8=10-5 The average gradient norm distribution has mass way above the usual clipping bounds.
Bar (rpal)
VITTle (ypal)
VilTley (mpar
Figure A25: Histogram of gradient norms in easy (ViT-Base, 8-8), medium (ViT-Tiny, 8), and hard (ViT-Tiny, 0.5.) tasks, C5 SUN397, 8 epochs; 8-10-5 The average gradient norm distribution has mass way above the usual clipping bounds.
32

---

Published as a conference paper at ICLR 2026
J GRADIENT NORM DISTRIBUTIONS DURING TRAINING FOR SMALL AND LARGE C
8 (Easy)
(Medium)
0.25 (llrdd)
Epoch
50
100
50
100
50
100
Gradient norm
Figure A26: Gradient norm distributions over training epochs for ViT-Tiny on SUN397 epochs; 5-10-5, C=0.1). Learning-rroblen problem difficulty increases from left to right as e decreases. Thicker regions imply higher probability mass. As difficulty increases, the distributions shift toward higher gradient norms.
(Easy)
(Medium)
0.25 (Hard)
Epoch
100
Gradient
Figure A27: Gradient norm distributions over training epochs for ViT-Tiny on SUN397 (8 epochs; 5=10-5 C=42). Searnnng-robbles difficulty increases from left right as e decreases. Thicker regions imply higher probability mass. As difficulty increases, the distributions shift toward higher gradient norms.
33

---

Published as a conference paper at ICLR 2026
K
PER-CLASS EFFECTS OF GRADIENT CLIPPING
0.5
Relative Per- Per-Class Class Accurucy Relative Retained Weight
1.0
1.0
0.8
C-0.1
C-26
3
0.02
0.01
0.00
0.00
-0.01
-0.05
0.02
3
2
Class rank (easy to hard)
Figure A28: Per-class effects of gradient clipping under increasing task difficulty (left to right).
Trained on Cassava with epochs and tuned hyperparameters; 5 10-5
Easy (s 8)
Hard (s 0.5)
Relative Retained Weight
1.0
0.8
0.6
0.4
C-1e:05 05
C-10
40
60
100
40
60
80
100
0.05
A accuracy
0.00
-0.05
-0.15
20
40
60
80
100
20
40
60
80
100
Class rank (easy to hard)
Figure A29: Per-class effects of gradient clipping under increasing task difficulty (left to right). Trained on CIFAR-100 with epochs and tuned hyperparameters; 8 10-5.
34

---

Published as a conference paper at ICLR 2026
L
MEAN GRADIENT NORM OVER ITERATIONS
Mua Oraden Nom.
Epoctse2, Epillo--05
Rpsioo-0
Epilloee1
Epsinoz200
Gralnnt
Mor Crasker
Figure A30: The average norm of gradient trace on SUN397 with ViT-Tiny, epochs
Mea Oraiier Nom.
Epochsel,
Epecheed, Hpstoon00
Epoctssod, Upoiloee1
Epectrood, Epslion22.0
Gralien
Mas
Doraion
Denation
Figure A31: The average norm of gradient trace on SUN397 with ViT-Tiny, 4 epochs
Mea Oratier Nen
Fioc Epulenat) 25
Man Oralian Non
Epocheack Ppuloratt
Max Non
Ppodhwak Upilena1.1
Mesa Gralion Nor
Epocherk Uprioontt.
Mest Grallinn Nea
Upocher Ppeikenko
g
9
Figure A32: The average norm of gradient trace on SUN397 with ViT-Tiny, 8 epochs
Moan Oracian Neem
1pochest6, 25
Upecheet6,, Epoloontt
Ppochus16 Upeloon1100
Epochsaté, Ppplime77
Mesa
Mon
Deration
Deration
Figure A33: The average of gradient on SUN397 with ViT-Tiny, 16 epochs
35

---

Published as a conference paper at ICLR 2026
M
CUMULATIVE NOISE VS. EFFECTIVE NOISE
10a
10
2 Epochs
10
10
100
10
10a
102
4 Epochs
10
10a
Effective Noise
10
10
10
102
8 Epochs
10
10a
16 Epochs
10a
10a
10
10a
28
210
21
21
216
Full
16
.22
64
128
256
512
1024
2048
3136
Batch Size
Number of Steps
Figure A34: Effective noise, batch size and the number of steps under fixed-epoch DP-Adam on
CIFAR-100 (ViT-Tiny. FiLM; -50000, 1=. Results are averaged over 3 seeds. The learning rate and clipping bound are tuned separately for each point.
36

---

Published as a conference paper at ICLR 2026
2 Epochs
10t
4 Epochs
10a
Cumulative Noise
10a
8 Epochs
10a
0.
16 Epochs
102
240
21
212
21
244
255
Full
128
256
512
1024
2048
Batch Size
Number of Steps
A Accuracy
0.00
0.01
0.02
0.03
0.04
0.05
Figure A35: Cumulative noise, batch size and the number of steps under fixed-epoch Ad-Adam on SUN397 (ViT-Tiny, FiLM; N887002, 8=10-5) Results are averaged over 3 seeds. The learning rate and clipping bound are tuned separately for each point.
37

---

Published as a conference paper at ICLR 2026
10a
2 Epochs
104
102
4 Epochs
10
Effective Noise (a/9
8 Epochs
201
10
16 Epochs
1
0.25
0.5
10a
1.0
2.0
240
211
213
24
218
Fall
128
256
512
1024
2048
Batch Size
Number of Steps
A Accuracy
0.00
0.01
0.02
0.03
0.04
0.05
Figure A36: Effective noise, batch size and the number of steps under fixed-epoch AP-Adam on SUN397 (ViT-Tiny. FiLM; -87000, 8=10-5), Results are averaged over 3 seeds. The learning rate and clipping bound are tuned separately for each point.
38


Reviews:
[{"review_id": "vjRRSbf5Ey", "reviewer": "Reviewer_J42k", "summary": "This paper systematically studies how two critical hyperparameters, gradient clipping bound (C) and batch size (BS), affect performance in fine-tuning large pretrained models under DP constraints. Empirical and theoretical analysis showing that the optimal C depends on the privacy noise scale $\\sigma$ and gradient distribution, which itself shifts with privacy level, model capacity, and dataset difficulty.\nThe paper finds that \n1. Larger clipping bounds C can improve performance under stronger privacy (smaller $\\varepsilon$) and for weaker pretrained backbones.\n2. The optimal BS depends jointly on privacy and compute budget: under tight privacy, a smaller BS may perform better, while a moderate BS is optimal under limited compute.\n3. Fixing (C, BS) across tasks degrades performance, especially for harder datasets and tighter privacy settings.", "questions": "1. Line 30-33 claim that usually only LR is tuned for each separate problem, while other hyperparameters (BS and C) are fixed, with reference made to De et al. 2022. Could the authors clarify how De et al. 2022 is an instance? According to my understanding, De et al. 2022 conducted careful hyperparameter tuning.\n2. For the plots that present the change of optimal C/BS under different settings, how are the other hyperparameters (LR, BS/C, epoch) selected? For example, each experiment in Fig. 2 (left) might require its own LR and BS, and it would be helpful to describe how they are selected. Do all the runs in Fig. 2(left) use the same LR and BS? \n\nOverall, I suggest augmenting the experiments, improving Section 5.3, and improving the writing to make the insights for hyperparameter tuning more executable. After the clarification questions and concerns are addressed, I would be happy to raise my score.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Clear motivation: Hyperparameter selection is an important and open question in DP training.\n2. Novel theoretical contribution: Theorem 5.1 provides a principled explanation for observed discrepancies between prior theory and practice regarding the clipping bound.\n3. Insightful conceptual framing: Interpreting clipping as gradient re-weighting is intuitive and helps explain asymmetric behavior between easy and hard examples.", "weaknesses": "1. Since the scope of this paper is DP fine-tuning, it would be complete if language classification tasks were considered. Additionally, this paper only considered fine-tuning the classification head plus the scale and bias of normalization layers. It would be complete and more convincing if more baselines were compared, such as LoRA, linear probing, and full fine-tuning, given that the latter are widely used fine-tuning methods.\n2. The section \"Clipping as Gradient Re-Weighting\" is insightful, and it would be better if this section were written more mathematically rigorously rather than in storytelling.\n3. Writing: It would be clearer if the authors could summarize and highlight all the implications for practical hyperparameter tuning in bullet points in the introduction/Section 5 & 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "svIHssYmBs", "reviewer": "Reviewer_frYa", "summary": "This paper investigates a mismatch between theoretical results and empirical findings regarding the optimal clipping threshold C in relation to the privacy budget $\\varepsilon$. Thus, for DP transfer learning, the paper proposes using a gradient re-weighting mechanism to justify using a larger C with a smaller $\\varepsilon$, and analyzing the total accumulated DP noise to inform batch size selection.", "questions": "- The reviewer believes the gradient norm analysis needs to be further developed to fully support the paper's hypothesis. In Figure 3, the authors show the gradient norms after training with various $\\varepsilon$ and C, but the gradient norm also varies during training. A discussion of this dynamic behavior seems to be missing.\n-  Furthermore, the claim that the gradient norm gets bigger when not converged needs more support. In standard (and possibly DP) training, the gradient norm is often observed to increase towards the end of training (see [1], [2]), which seems to contradict the paper's explanation (i.e., that large norms are simply due to non-convergence). Please provide more backing for this claim.\n\n    [1] Why Gradients Rapidly Increase Near the End of Training, 2025.\n\n    [2] Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning, ICML 2022.\n\n-  Is there any significant difference between analyzing 'from scratch' and 'transfer learning' in terms of your analysis or empirical findings? Would your conclusions hold for 'from scratch' training?\n-  Can you test your proposed framework on other SOTA DP training methods, such as those mentioned in the related works?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper investigates an interesting idea: that previous approaches using a small, fixed C (e.g., 0.1 or 1) might be suboptimal.\n- The authors provide a basic analysis of how DP noise affects the optimal clipping bound.\n- Based on their observations, the authors’ proposed tuning approach performs better than using static hyperparameters.", "weaknesses": "Please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SwBEgf9snM", "reviewer": "Reviewer_yubP", "summary": "This paper investigates the optimization strategies for hyper-parameters in DP finetuning, including the clipping bound $C$ and the batch size $B$. For the clipping bound $C$, the author presents a finding that under strict privacy budget, a larger clipping bound leads to improved model performance. For the batch size $B$, the paper proposes a principle for selecting the optimal $B$ on the fixed-epochs setting. This rule aims to overcome the limitations of traditional fixed-step theories by maximizing the number of optimization steps $T$ (via minimizing $B$) while ensuring that the cumulative DP noise ($\\sigma \\sqrt{T}$) remains within its near-optimal plateau region.", "questions": "See the weaknesses part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper demonstrates that the clipping bound $C$ and batch size $B$ should be dynamically adjusted based on the difficulty of the learning task, which is different from the traditional approach with fixed hyper-parameters.\n2. The paper introduces the perspective of \"gradient re-weighting,\" which provides a detailed analysis of how the clipping bound $C$ affects the learning process.\n3. The paper highlights the necessity of jointly tuning $C$, $B$, and the learning rate, noting their complex interactions.\n4. Comprehensive experiments are conducted.", "weaknesses": "1. The study relies only on the Vision Transformer; it would be beneficial to explore various models to verify the effect of pre-trained model capability on optimal clipping. At a minimum, some discussion of future work is necessary.\n\n2. It would be great if the authors could discuss other related works concerning optimal hyperparameters in DP fine-tuning, especially the learning rate. For instance, [A] pointed out that if the features of a ViT are extracted near-perfectly, then DP fine-tuning is less sensitive to hyperparameters. This implies that in some extreme cases with near-perfect features, optimal hyperparameter tuning may not be necessary—a conclusion that differs from this paper, which considers cases where pre-training extracts near-perfect features for the fine-tuning datasets. However, [B] shows that if the features are not extracted perfectly, then optimal hyperparameters, especially an optimal learning rate, are necessary; otherwise, the features may be degraded by the noise, a conclusion that aligns with this work.\n\n\n[A] Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learning. Wang et al., ICML, 2024.\n\n[B] Why Does Private Fine-Tuning Resist Differential Privacy Noise? A Representation Learning Perspective. Zhao et al., ICLR 2025 Data-FM Workshop, 2025.\n\n3.  The term \"learning problem difficulty\" is central to the paper but seems to be used informally (please correct me if I have missed a definition). It would be beneficial to define or characterize it more formally, perhaps as a function of the privacy budget (or noise level), model capability, the size of datasets. A brief discussion in Section 5 would suffice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tj5jkJEmAv", "reviewer": "Reviewer_GYoL", "summary": "This paper focuses on clipping threshold and batch size. The paper studies how these two usually-fixed hyperparameters in differentially private (DP) transfer learning should depend on privacy level, compute budget, and backbone strength.\n\nThe authors (i) document that empirical optima contradict a common theoretical takeaway (\"tighter privacy $\\Longrightarrow$ smaller C\"), (ii) provide an analysis that explains when larger C helps under tight privacy via a new MSE-based optimal clipping result (Theorem 5.1), and (iii) argue that in fixed‑epoch (compute‑bounded) training, cumulative DP noise ​$\\sigma\\sqrt{T}$ together with a minimum-steps constraint better explains optimal batch size than per‑step noise heuristics developed for fixed‑steps regimes. The empirical study uses DP‑Adam with PRV accounting ($\\delta$= 1e−5) and extensive grid search over LR, B, C on SUN397, Cassava, CIFAR‑100 (and a 10% subset), comparing ViT‑Base vs ViT‑Tiny (and some ResNet‑50 in the appendix).\n\nFor the batch size part, the authors show that “effective noise” plots used in fixed‑steps heuristics (e.g., $\\sigma/q$ ) monotonically favor full batch under fixed‑epochs, hence are uninformative (Figure 6). Instead, plotting cumulative noise ​$\\sigma\\sqrt{T}$ versus B reveals plateaus under tight privacy where moderate B can match or beat large B, provided a minimum number of steps is met (Figure 7). The appendices give dataset‑specific grids (Table A1) and many robustness plots across $\\epsilon$'s and epochs.", "questions": "- In Figure 4, could you explain more on how you determine which classes are harder than others?\n\n- Could you turn Theorem 5.1 into a procedure? In other words, can you propose a privacy‑respecting estimator for the statistics that define the optimal $C$? How sensitive would such a method be to noise?\n\n- How would you incorporate DP hyperparameter tuning (e.g., subsampled validation with DP bandit/BO) into your workflow to preserve the reported gains when tuning itself must be private?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Many DP transfer‑learning pipelines fix C and B across tasks; this paper convincingly shows that practice leaves accuracy on the table and can be especially harmful in hard regimes (e.g., small privacy budget, limited compute, weaker backbones)\n\n2. Theorem 5.1 explains when the optimum moves the other way because the gradient distribution itself shifts under DP. Figure 3 and the Figure 4 make this intuitive and actionable.\n\n3. The fixed‑epoch framing matches many real‑world budgets. The \"cumulative noise plateau\" and \"minimum steps\" interpretation is intuitive and well‑visualized.\n\n4. Implementation details like hyper-parameter grids, (un)normalized-DP version, and class imbalance conditions are well documented and explained.", "weaknesses": "**Major concerns**\n1. Previous works have shown that the choice of fine-tuning methods and initialization of heads could influence the DP performance, e.g. [1] [2]. However, all main experiments in this paper fine‑tune with FiLM (scale/bias of normalization layers + head), leaving about 0.5–1.5% of parameters trainable (Section Fine‑tuning). While the authors vary backbones (ViT‑Tiny/ViT‑Base and some ResNet‑50), the parameterization is fixed; Table A1 confirms all grids are under FiLM. In the discussion they state the mechanisms are “general” and expect transfer to full‑model tuning, but this is not demonstrated. This design choice narrows the external validity of the paper’s conclusions about optimal clipping C and batch size B.\n\nReferences:\n\n[1] Want et al. 2024. Neural Collapse meets Differential Privacy: Curious behaviors of NoisyGD with Near-Perfect Representation Learning. ICML.\n\n[2] Ke et al. 2024. Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion. NeurIPS Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability.\n\n**General weakness**\n\n1. Most results are with DP‑Adam on image classification and parameter‑efficient fine‑tuning. While some ResNet‑50 and dataset diversity exist in the appendix, it remains unclear how fully the conclusions transfer to DP‑SGD, full‑model fine‑tuning, or other modalities (NLP, speech). The paper argues mechanisms are general, but evidence is image‑centric.\n\n2. The \"minimum steps + cumulative noise plateau\" guidance is compelling visually, but a practitioner‑ready recipe (how to pick/estimate the steps threshold per dataset/task without extra privacy spend) is not spelled out. For example, in Figure 7 the dashed 20‑step threshold is illustrative, and Appendix A notes dataset‑dependence, but there is no principled estimator.\n\n3. The study performs extensive grid searches, tuning LR jointly with C,B per configuration, which is appropriate for understanding behavior but raises the usual question of privacy leakage from tuning in practical deployments. The paper cites work on DP hyperparameter tuning, but it does not quantify or discuss the privacy cost of its own search.\n\n**Theorem 5.1 is confusing to me for several reasons.**\n\n- Weak theory–practice bridge. The current MSE criterion is explanatory rather than prescriptive. Without a descent‑type link (the small lemma suggested above) or a correlation study tying MSE proxies to accuracy across your sweeps (Fig. 2 and Fig. 4), the reader is asked to accept that “lower MSE $\\approx$ better” on faith. Clarifying this would substantially strengthen the contribution.\n\n- State prominently that $N_C$ and $G_C$ are used only for analysis in this paper, and provide one of the DP‑compliant procedures above for any future adaptive use. As written, a well‑meaning practitioner could implement a non‑private adaptation.\n\n- Recasting Theorem 5.1 as a piecewise‑explicit optimizer (with convexity and boundary cases spelled out) would eliminate the fixed‑point ambiguity and address the \"hand‑wavy\" perception. Please centralize all assumptions next to the theorem.\n\n**Some other minor issues.**\n- In Figure 2, the plots only show relative accuracy differences. It would always be helpful to show the absolute accuracy value of each experiment result here. Readers would always prefer complete information rather than a visually straightforward but incomplete figure. And it would be helpful to provide a zero-shot baseline (i.e. no fine-tuning at all). The zero-shot baseline would provide a basic sanity check when the authors argue that the number of parameters is positively correlated to the \"model capacity\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)