Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion
Authors: Shuqi Ke
Charlie Hou
Sewoong Oh
Giulia Fanti
Venue: Accepted by TMLR ()

Abstract: We show that **d**ifferentially **p**rivate **f**ull **f**ine-**t**uning (DP-FFT) can distort pre-trained backbone features based on both theoretical and empirical results. We identify the cause of the distortion as the misalignment between the pre-trained backbone and the randomly initialized linear head. We prove that a sequential fine-tuning strategy can mitigate the feature distortion: first-linear-probing-then-fine-tuning (DP-LP-FFT). A new approximation scheme allows us to derive approximate upper and lower bounds on the training loss of DP-LP and DP-FFT, in a simple but canonical setting of 2-layer neural networks with ReLU activation. Experiments on real-world datasets and architectures are consistent with our theoretical insights.   We also derive new upper bounds for 2-layer linear networks without the approximation. Moreover, our theory suggests a trade-off of privacy budget allocation in multi-phase fine-tuning methods like DP-LP-FFT.

Published in Transactions on Machine Learning Research (11/2025)
Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion
Shuqi Ke Carnegie Mellon University
shugkkBondrew.cmu.omm
Charlic Hou Carnegic Mellon University
Shandres@andrew.coo wwt.trewwwww...omommmmm eduu
Sewoong Oh University of Washington
sewonng@ls..uwshhigggooo...
Giulia Fanti
Klanti@adrew.ccmet. edu
Carnegie Mellon University
Reviewed on OpenReview: https: // operreveew..eee... net/ forum? id- LuT8aDv502
Abstract
We show that differentially private full fine-tuning (DP-FFT) can distort pre-trained back- bone features based on both theoretical and empirical results. We identify the cause of the distortion as the misalignment between the pre-trained backbone and the randomly initialized lincar head. We prove that sequential fine-tuning strategy can mitigate the fea- ture distortion: first-linear- prt---o-tooinen--iiieee featt-iinettuininn tuning (DP-LP-FFTT). A new approximation scheme allows us to derive approximate upper and lower bounds on the training loss of DP- LP and DP-FFT, in a simple but canonical setting of 2-layer neural networks with ReLU activation. Experiments on real-world datasets and architectures are consistent with our theoretical insights. We also derive new upper bounds for 2-layer linear networks without the approximation. Moreover, our theory suggests trade-off of privacy budget allocation in multi-phase fine-tuning methods like DP-LP-FFT.
1 Introduction
Today, many siterenntillly-priatte (DP) machine learning pipelines proceed in two phases: (1) A model is pre-trained (non-privately) public dataset. (2) The model is then fine-tuned private data, using DP optimization techniques such DP stochastic gradient descent (DP-SGD) and its variants (Hoory et al.
2021
2023 Zhang et al. 2024b). Pre-training backbone model on public data enable bles differentially private fine-tuning to achieve improved performance across various downstream tasks
De et al. 2022 Tang et al.
Yu et al. 2022 and is proven to be necessary in some cases
Gannss et al. 2023a
Despite these advances, the effect of DP on fine-tuning train- ing dynamics remains poorly understood. Several key questions are yet to be answered: (1) how does randomness (both of ini- tialization and DP optimization) impact the pre-trained rep- resentations? (2) What are the convergence rates of common fine-tuning methods, such as DP full fine-tuning (DP-FFT) and DP linear probing (DP-LP, where feature representations are frozen, and only the linear head fine-tuned]? (3) Prior work suggests that combining an early stage of DP-LP with later stage of DP-FFT yields better privacy-utility tradeofis
Linear probing
Pull fine-tuning
Linear head
Trainable leyers
Festuru extractor
Figure 1: Linear probing (LP) freezes the lower layers and optimizes the last lincar layer while full fine-tuning (FFT) optimizes the whole network.

---

Published in Transactions on Machine Learning Research (11/2025)
92
75.0
Feature Quality (%6)
72.5
88
86
67.5
DP-FFT
DP-LP
84
DP-FFT
Test
DR-LP-FT(E-33 DP-LP-FT(=-2
62
DP-LP-FFT
features
62.5-
DP-LP-FT(r=1)
044
1.0
FFT Epochs
Ratio Privacy Budget Alotted to LP
Figure 2: Left: Backbone feature quality evaluated by top-1 kNN accuracy on the downstream task, for ResNet-50, through public pre-training ImageNet-IK and differentially private fine-tuning STL-10. Right: Privacy budget trade-off in DP-LP-FFT, predicted in our theory, for WideResNet-16-4 on CIFAR- (Tang el al.
10
2023). For a detailed explanation, refer to
Tang al. 2023), yet there is no theoretical understanding of this phenomenon, nor is it clear how to optimally combine these fine-tuning methods.
Answering these questions theoretically requires an analysis that can capture the fine-grained optimization dynamics of DP fine-tuning. We sock a model of DP finctuning that satisfics 2 propertics.
1. Architecture--ensiitiittyy The convergence dynamics must differentiate between representation learn- ing in the backbone and learning in the linear head. The analyses of Bassily et al. (2014) Wang et al.) (2022 Fang et al. (2023) Ganesh al. (2033b) focus only on the network's s dimension, failing to capture this distinction.
2. Ability to model nonlinearities: The model should account for the nonlinearities introduced by multi neural layers, unlike existing methods that simplify analysis by linearizing neural networks Ye et al. 2023a Wang et al. 2024).
We propose novel approximation of DP-SGD training dynamics based on linearizing Langevin diffusion around the noise term. This approach offers new insights into DP fine-tuning and significantly simplifies analysis by converting stochastic differential equations into ordinary differential equations (ODEs). We validate our thcoretical predictions with real experiments.
Main contributions. In summary, key contributions
1. New approximation technique: In Section 2 we derive first-order ODE via an asymptotic ex- pansion of the stochastic noise in Langevin diffusion. Unlike previous methods, which linearize neural network parameters, our technique preserves the multi-layer structure of deep learning models while sim- plifying the analysis. This approach, commonly used in physics and control theory (Skorokhod et al. 2002), novel in the context of private machine learning and bridges the gap between non-private neural network theory and the private regime.
2. Understanding of feature distortion: In Section 3 we provide theoretical understanding of how DP fine-tuning affects feature representations. Using our approximation, we prove that, in 2-layer ReLU networks, randomly initialized linear heads distort pre-trained backbone features in the early stages of DP-FFT. Empirically Figure 2 demonstrates that feature quality evaluated on private data initially degrades during DP-FFT but later improves and surpasses pre-fine-tuning quality. Our theory also predicts that running a single epoch of DP-LP before transitioning to DP-FFT can mitigate this initial feature distortion, as shown empirically in the DP-LP-FFT curve of Figure 2 (left). This insight extends the findings of Kumar et al. (2022), who showed that LP-FFT roduces feature distortion in oon-private, OOD scenarios, in-distribution settings for both DP and non-DP cases.
3. Theoretical convergence bounds: In Section we present new upper and lower bounds on the training loss of DP-LP and DP-FFT for 2-layer ReLUU networks using approximation technique.
2

---

Published in Transactions on Machine Learning Research (11/2025)
We also prove upper bounds for 2-layer linear networks without the approximation. To the best of our knowledge, this is the first convergence analysis of DP-SGD on non-linear neural network architectures.
4. Mitigating feature distortion by combining fine-tuning methods: Prior work by TTng et al. (2023) empirically showed that combining DP-LP and DP-FFT (DP-LP-FFT) can achieve better test accuracy than either method alone. Figure 2b] we demonstrate that allocating approximately 20% of the privacy budget to DP-LP yields optimal test accuracy. In Section D explanation for this phenomenon. Specifically, our bounds suggest that DP-FFT may underperform relative to DP-LP at lower privacy budgets, while DP-LP-FFT can outperform both methods under moderate privacy budgets. These predictions are empirically verified across various architectures and benchmarks in Section 5.3
we provide a partial theoretical
1.1 Related Work
Similar empirical phenomena have been explored in non-private, out-of-distribution (OOD) contexts by Aghajanyan et al. (2021). Kumar et al. (2022), Trivedi ct al. (2023), and Chen et al. 2024 Kumar et al. (2022) demonstrated that non-DP line-tuning distorts pre-trained features, leading to degraded OOD
performance. But their theory relies on the assumption that OOD test data exists in an orthogonal subspace to the fine-tuning training data, leaving their results unable to explain why, in many transfer learning tasks, lincar-probe fine-tuning (LP-FFT) still outperforms both LP and full fine-tuning (FFT) in in-distribution (ID) settings. Our work seeks to fill this research gap.
Wang et al. (2024) examined how pre-trained representations enhance DP fine-tuning within the neural collapse framework, though their analysis was restricted to the final layer. Mcanwhile, [Tang et al. (2028) empirically observed the privacy budget trade-off for WedeResNNt models pre-trained on synthetic data, but without accompanying thcoretical insights.
Analyses by Wang et al. 2019), Chen et al. (0200a], Ganesh et al. (003bb), and Fang et al. (022 rely standard convexity/ ceiiityyooor /non-convexity and smoothness assumptions, which abstract away the simultaneous dynamics between the backbone and linear head. Other works (Ye et al. 2023b Wang et al. 2024 focus on
linearized models, limiting their ability to capture the nuanced interactions between these components. Our explanation of representation alignment builds on the theoretical foundation of Min et al. (2024). which we extend to a DP context using novel approximation tools.
2 Continuous modeling of differentially private fine-tuning
Notation. We use Ô to denote both the deterministic and stochastic differential operators. The dot product between vectors y is x ITy, the Euclidean norm of vector æ is || and the infinity norm is || The trace of a matrix is denoted by tr, and the ReLU activation is d. For any twice differentiable function f(x), its gradient denoted Vzf and its Hessian as Hzf- Li denotes the disjoint union. [i] == (1.....4). The
""p
We denote the privacy cost cosine similarity between two vectors u, is defined as cos(u, Tulla estimated by Rényi divergence as r.
DP-SGD Dynamics. Different privacy (DP) is a widely used framework for evaluating privacy leakage in datasct accessed through queries (Dwork & Roth 2014). In machine learning, DP ensures that an ad- versary cannot confidently determine whether specific training samples are part of the dataset. Differentially Private Stochastic Gradient Descent (DP-SGD), introduced by Abadi et al. (2016), is the standard algorithm for training deep neural networks while maintaining privacy.
Our fine-tuning theory built on an analysis of DP-SGD dynamics. Although real-world algorithms are dis- crete, contimuou approximations- such as stochastic differential equations (SDE) like Langevin diffusion are often used to study these dynamics (Chourasia et al. 2021 Ye et al. 2023b). In a similar vein, Kumar ct al. (2022) use gradient flow, a continuous approximation of SGD, to study fine-tuning in a non-private context.
Definition 2.1 (Langevin diffusion GGanesh et al. 2023b dynamics of system influenced by both deterministic and random forces [Lemons & Gythiel 1997). For
Langevin diffusion an SDE that models the
3

---

Published in Transactions on Machine Learning Research (11/2025)
DP-SGD, we define p-dimensional Langevin diffusion follows:
00 - -V + v2020Q..
(1)
where 0 E I' represents the neural network parameters, f is the network architecture, L(1 : R
+ R
the Iraining loss, and a > 0 is the noise multiplier (Abadi et al. 2016). (Qt)220 is the standard Brownian motion in R modeling the Gaussian noise mechanism.
By Ito's lemma
Ito 1951
the Langevin diffusion of the training loss is given by
0c = 1 oo (vr((()) ot + v20 (VoC(()))
(2)
Ye et al. 2023b study how random initialization affects DP-SGD performance in linearized neural networks via Langevin diffusion. To facilitate theoretical analysis, they lincarize the entire neural network using 1st order Taylor expansions at the initial parameter to-
0f(x) +
f(x) = fun(z) f(x)
(0 -0-00 00).
(3)
00
0=
8=#to
Recently, this lincarization technique has gained popularity for explaining key deep learning phenomena (Ortiz-Jimenez et al. 202Ip. However, fully linearizing the model removes critical multi-layer interactions, making this approach unsuitable for our analysis.
To address this, view the optimization trajectory of neural networks as dynamical system, with noisc in gradient updates treated as random perturbations. We first rewrite a Langevin diffusion like Equation [ in the following form
00 = =F(() + oG())0Q
(4)
where F is the drift coefficient and G is the diffusion coefficient. We then introduce small noise (regular) perturbation expansion of the Langevin dynamics in the spirit of Freidlin -reezeln-Wentzell (Freidlin et al.) 2012
In particular, we decompose Langevin diffusion (e.g. Equation (1) to power series of the perturbation scale
0= 0 + (2 + 020 +...,
(5)
where we define each g(+)
(60
Σ
V'F(l(i))
( +
VGG(( ...... lat =1,
Σ
(6)
Like Taylor's expansion, we can approximate 0 with the partial sum Σ ((()) and the remainder 0 Σi 00000 is infinitely small compared with σ,, uniformly on any finite interval [0, T]. The approximation order N gives various accuracies for the deviations caused by the random perturbations.
Applying the zeroth-order asymptotic expansion (N = 0) for the paramcter dynamics 0 (Equation and the loss dynamics L (Equation (2), we approximate:
00 00 -V (0 at.
(7)
In the zeroth-order expansion, we ignore the noise term aQ; and only keep the noise effect term C) in the loss dynamics. This zeroth-order expansion helps circumvent the complex analysis of stochastic, non- linear equations. By substituting the approximate parameter i into Equation (2), our modeling partially preserves the noisy behavior characteristic of DP-SGD. We further explore this property the next scction.
2.1 Zeroth order approximation
The noise multiplier o remains explicitly in our convergence bounds. We retain the key noise effects for the loss dynamics by keeping the second-order term from Ito's lemma in Equation 2 and preserving the second-order terms associated with Brownian motion.
4

---

Published in Transactions on Machine Learning Research (11/2025)
This approach allows us to capture the essential stochastic characteristics of DP-SGD without modeling the full noise term directly on the parameters. In essence, this approximation enables us to analyze the expected behavior of parameter updates while preserving the noise-sensitive behavior of the loss itself. By isolating these core clements, we provide insights into the overall training dynamics under diffcrential privacy without losing the major noise effects that influence convergence properties and feature alignment.
To support our claim that this approximation does not introduce too much error, we have proved an error approximation guarantee, which shows that our approximated model does not differ too much from the original Langevin diffusion model. We present the theorem based Langevin diffusion with gradient clipping. We use the subscript t in 0, to denote the parameter A at training step t.
Clipped Langevin diffusion: 00;- - Σ (V + v2020Q.
iE/m
Zeroth order approximation: 00,-
Σ iE
clipe (Vl: (0:l.)) 01,
(8)
C
where clipe == min 1,
Tulla
Thcorem 2.2 (Zeroth order approximation error). Denote the model parameter vector in original Langevin diffusion as 9e and its zeroth-order approrimated version as A. For any training time t >> 0 and clipping threshold C > 0,
01 10--0,11 0.119] < (o(2))tth+ +2nCt)
E
Note that this approximation error significantly improves upon the ((exp(T) error found under standard regularity assumptions (Freidlin al. 2012 Theorem 1.2, Chapter 2.1). The approximation does not remove the effect of noise, nor is the resulting model equivalent to gradient flow. We defer the proof to Appendix [F]
The the best of our knowledge, this is the first analysis of clipped Langevin diffusion as continuous model of DP-SGD. We present more technical details in Appendix F
3 Representation Alignment
In this section, we introduce the concept of representation alignment, present our thcoretical findings, and validate them with experiments. Representation alignment. refers to the process by which the classification head aligns itself with the pre-trained backbone features. During the DP-FFT process, this alignment creates characteristic trend in feature quality: initially, the randomly initialized linear head distorts the pre-trained features, but as it better aligns with the backbone, the distortion diminishes, and the overall quality of the backbone features improves over time.
3.1 Theory
Our goal is to understand (1) how does DP fine-tuning distort the pre- trained features in the backbone, and (2) under what conditions this dis-
S+
S_
tortion can be mitigated. We consider the simple binary classification setup from Min et al. (024)) which provides a clear and intuitive un- derstanding of representation alignment. The results generalize to our experiments in Section 3.2 Specifically, we 2-layer fully-connected neural network with h hidden nodes and ReLU activation o,
h
f(x) (x))=o](()) = uT (TToWTT 1) Σ 250(uJa) x).
(10)
-
fine-tuning on a dataset = (a,, , with n inputs z; Rd, and Figure 3: Visualization of As- binary labels ze (-1,1)- The objective is to minimize the training sumption 3.1
5

---

Published in Transactions on Machine Learning Research (11/2025)
loss [(flf) = Ei f(:)), using the exponential loss Ely. 9) =
exp(- Similar results hold for logistic loss (Min et al. 2024
Our use of a two-layer surrogate and a zeroth-order ODE is a local approximation around the pre-trained weights. In the short horizon that governs the distortion phase, it has been previously shown that deep networks behave approximately like their linearization (Jacot et al., 20188) Lee al. 2019; Kumar et al. 2022); the dominant term is the interaction between the head's random initialization and the backbone ' Jacobian under DP-SGD updates. This is precisely what surrogate captures.
For simplicity, we make the two assumptions.
Assumption 3.1 (Data correlation (Min et al. 2024]). For any pair of data (24,8)), (xy, vy), the inputs are positively negaaavveyy correlated if the labols are the
inf (way)) AgE(m]
2 2 [,y12
(11)
| lla
|
We define two cones in &d that separate subspaces spanned by data points in the positive and negative classes, respectively: 5+ = {= E Rd= Vi E [n].Ta] 2>0 = -,,=),,, S_ = {= E TR Vi E [n], Th, =>0 = =1
Min el al. (2024) prove that S+ n S_ 0, and x E S+/- if yą 1/ (see Figure 3 We define the mean data directions of class c [-1,1} by T. = Σie(n) Zi y.=e-
We assume that a "clustering" bohavior emerges in the pro-trained features, which allows the features to work well in transfer learning Gallant et al. 2022). This phenomenon is well-documented in the neural collapse literature (Kothapalli 2023). suggests that pre-trained features Wy tend to converge around the mean direction for data in class c(7).
Assumption 3.2 (Collapsed neural features). For cach wj in Equation 10 where j E [h] (with h denoting the dimension of the linear head), it holds that wj E S+ or wy S_. We define c(j) = if wy S,, and c(j) = -1 1 if wy E S_. Thus, there is a partition [h] = F+ UF_ over the index set [h], such that for each wy.
j F+ uf S+. jEF Ey ES..
(12)
Feature quality. Assumption 3.2 says that data with positive label (resp. negative) only activates the j-th neuron if j E F+ (resp. j E F-). As a result, any positive data pair, (x,y) and (z,y') with y = activate the same set of neurons. From contrastive learning viewpoint, this assumption makes the representations of them semantically similar (Saunshi et al., 2019). Namely, when the features wy and data inputs x, are normalized unit vectors, the difference between representations of a positive data pair is bounded by:
llg(x) - =(x))l <
max cos(w,,z)) x;),
(13)
24 ===)))=y
which represents the maximum cosine similarity between the fcatures wy and the data points.
Note that our assumptions are We further discuss the relaxation of the assumptions in Appendix B.1
and serve to make the distortion mechanism transparent.
However, FFT or DP-FFT with random initialization may reduce the feature quality.
Theorem 3.3 (Random initialization causes feature distortion). If Assumption 3.1 and Assumption 3.2 hold, and the linear head is randomly initialized by Up ~ (O, 3Inxu); then with probability 1- 2 2-h vB 0, 3j E [M], At > such that during the time interval (0, At), DP-FFT distorts wi reducing ils alignment with the data cluster. The cosine similarity between wj and the data cluster mean Fe(s) decreases monotonically:
i ôt
cos (w., (wg. <0, Vt (0,
(14)
For a pre-trained wy that aligns with labeled data, DP-FFT (as modeled by Equation makes it deviate from Fof): the mean direction of those data. wy optimal when cos( (Wy, Fe()) 1. This result holds

---

Published in Transactions on Machine Learning Research (11/2025)
for both DP and non-DP settings and explains the potential feature distortion observed in in-distribution and non-private settings, such those studied by Kumar et al. (2022) ). The stochastic analysis of non-smooth loss, activation, cosine similarity functions is challenging without our approximation.
Next, we show that running (DP-)LP before (DP-)FFT could mitigate feature distortion.
Theorem 3.4 (DP-LP first mitigates feature distortion). Suppose Assumplion 3.1 and Assumplion 3.2 hold,
and the linear head is randomly initialized by vo ~ NN(0, (lxxh) for any 3 > 0. There exists At such that after running DP-LP for time Ât, switching full fine-tuning ensures that DP-PFT does not distort the pre-trained features. Specifically, cos(w,, is non-decreasing for all j E [h]:
θ Ot
cos (wj. Re(s)) 2 0, A (At,
(15)
See complete proofs of Theorem 3.3 and Theorem 3.4 in Appendix C.1
Corollary 3.5 (Non-DP feature distortion). The results in Theorem 3.3 and Theorem 3-4 still hold in non-
DP case (o 0). In particular, if Assumption 3.1 and Assumption 3.2 hold and the linear head is randomly initialized by Uo ~ (hwh)::))):
1. Then with probability 1 1 2- v3 > 0,3j E [k] At > O such that during the time interval (0, At), FFT distorts wj:
cos (wz, Re(s)) at
< 0,
Vt At).
(16)
c
2. There crists At such that after running LP for time At, FFT does not distort the pre-trained features. Specifically, cos(wy, Baiy) is non-decreaingg for all j [h]:
Ōt cos (wy, {daI 20, Vt E (At, +oo).
(17)
3.2 Experiments on Representation Alignment
In this section, we show empirical evidence supporting Theorems
3.3 and 3.4
Pro-training and Model. We pro-train Vision Transformers (ViT) and ResNet-50 backbones on ImageNet- IK using Self-Supervised Learning methods, including BYOL Grill et al. 2020) and MoCo v2 (Chen et al., 2020b), as well as distillatinn methods Touvron et al. 2021). Then we fine-tune the backbone with a linear
classification head on CIFAR-10 and STL-10 using DP-SGD.
Experiment protocols. We conduct public pre-training for 100 epochs with a batch size of 256. Following this, we implement DP-SGD using the pre-trained weights and a randomly initialized linear head for 30 epochs. Each DP fine-tuning process repeated with random seeds and a batch size of 1000. We evaluate the backbone features both the pre-training and fine-tuning datasets, measuring feature quality through top-1 kNN accuracy (Chen et al. 2023).
Private fine-tuning initially distorts features. Figure 4 qualitatively visualizes the effect of DP-FFT on feature quality with respect to the private test data. We pre-train (BYOL) a ResNet-50 backbone on ImageNet-1K and DP fine-tune (DP-SGD, e == 1) it on STI-100 We qualitatively assess the features of the private test data within the ResNet-50 backbone by visualizing the backbone mappings (outputs from the penultimate layer) of data points using UMAP (Mclnnes et al. 2020). For simplicity, we only plot 3 classes in CIFAR-10.
Figure indicates that during the initial phases of DP-FFT, the randomly initialized lincar head interferes with the pre-trained features in the backbone network, leading to degradation in feature quality on both the pre-training and fine-tuning datasets. This observation validates Theorem 3.3 Concurrently, the linear head begins adapting to these pro-trained fcatures, a process we refer to as "representation alignment." As this alignment progresses, the backbone starts to regain portion of its original feature quality, which had been degraded by DP noise and shifts in data distribution.
7

---

Published in Transactions on Machine Learning Research (11/2025)
End pre-training
Early stage DP-FFT
End of DP-FFT
End of pre-training
Early DP-FFT
End DP-FFT
Rinetturing step-0
Fine-tuning step=100
Fine-tuning step-1500
Fine-tuning step-0
Fine-tuning step-100
Fine-tuning step-1500
(a) STI-10 (in-distribution)
(b) CIFAR-I0 (out-of-disste) distrriiibbt)n)
Figure 4: We pre-train (BYOL) a ResNet-50 backbone on ImageNet-IK and DP fine-tune (DP-SGD, it STL-10. We qualitatively evaluate the features in the ResNet-50 backbone by visualizing the backbone mappings (penultimate layer outputs) of data points via UMAP (McInnes et al. 2020 These results suggest
that DP-FFT distorts feature quality before improving it, as predicted by Theorem 3.3
End of pre-training
DP.LP-FFT step=5
DP-LP-FFT step-15
DP-LP-FFT step=25
Figure 5: UMAP of pemultimatc-layer features subset of MNIST (labels {0,3,7)). We visualize the features at the end of non-private pretraining and the end of DP fine-tuning. As we increase the number a of DP-LP steps (q E {0, 20))) before DP-FFT (5 steps), we observe that the the severity of the feature distortion reduces.
Linear probing mitigates fcature distortion. To illustrate the benefits of linear probing, we first run DP-LP for 1 epoch before transitioning to DP-FFT for the remaiting epochs. In the initial steps of DP-FFT, the feature distortion significantly weaker (Figure 2a if we first run DP-LP. This supports the claim of Theorem 3.4 Similarly, we evaluate features on the pre-training domain (see Figure 7
We also visualize with UMAP the penultimate-layer features MNIST (labels 0,3,7) in taken at two checkpoints of the training pipeline: non-private pretrain and final DP-FFT (after some early DP-LP steps). In Figure 5] the pretrain panel (left-most) shows three compact, well-separated clusters. We switch to DP-LP after the pre-training stage. We consider three settings with different DP-LP steps. In the second, third, and fourth plots -legttrright. Figure 5 we run DP-LP for 0, 10, 20 epochs respectively, and then run DP-FFT for epochs after the DP-LP phase. We fix the noise multiplier to
for DP-LP and a for DP-FFT.
As our theory predicted, private updates in DP-FFT induce the expected feature distortion: class prototypes drift from their pretrain locations, clusters elongate and partially mix along shared manifold, and the inter- class margin narrows relative to the increase in intra-class spread. This behavior is consistent with our theory that, al the onset. of DP-FFT, gradients are misaligned due to (i) the random or poorly aligned classification head and (ii) DP noise injected into per-sample gradients; as result, the backbone momentarily adapts in directions that do not presorve the pretrain geometry. When we increase the number of DP-LP, we effectively mitigate the feature distortion: the clusters are better aligned and separated (though not identical to the pretrain configuration).
8

---

Published in Transactions on Machine Learning Research (11/2025)
4 DP Fine-tuning Convergence Rates
Scction 3 showed that DP-LP-FFT can mitigate feature distortion. A natural question is, for fixed privacy budget, how do DP-LP and DP-FFT allect the convergence of fine-tuning loss function? We study this question under two models: (1) our zeroth-order approximation of Langevin diffusion (Section 4.1 and (2) a two-layer neural network without our zeroth-order approximation (Section 4.1.1p. The second result will be used to study the budget allocation of DP-LP-FFT in Section 5 To our knowledge, these are the first convergence guarantees (approximate or not) for DP fine-tuning on explicit nonlinear neural network architectures.
Privacy guarantees We begin by establishing the privacy guarantees of Langevin diffusion by bounding the Rényi divergence of its trajectory distributions on neighboring datasets (Mironov) 2017). Both Ganesh et al. (023b) and Ye al. (2023b) show that the Rényi divergence increases linearly over time. We this guarantee for all tine-tuning variants.
Thcorem 4.1 (Rényi privacy guarantee (Ganesh al. 2023bl). Suppose we initialize a pair of neural nelwork paramelers 0,0' by some i.i.d. distributions On 0- We fine-lune 0,0' respectively on neighboring datasets D, D' via Langevin diffusion. Denote the distribution of the trajectory of by Øto.1] over T) Similarly, denote the trajectory distribution of 0' by 100.11 Then for any 1, the Rényi divergence Ra is bounded linearly in time,
(0527
(AATT
r: R.. (0000000))) =0
(18)
where is the noise multiplier, and Ag IVVL((; VL(O;D') is the upper bound of gradient difference between meighboring datasets. Thus, for any ó E (0, the Langevin diffusion satisfies
(ALT 4
5) differential privecy. log(1/6)
(19)
a
4.1 Convergence Rates under the Zeroth-ordes Approximation
We follow the approximation scheme outlined Equation 7 to derive convergence results for two-layer ReLU neural networks. These results are derived from our zeroth-order approximation; recall that we bound the error of this approximation relative to the Langevin dynamics model in Theorem 2.2 To support these findings, also include separale convergence proof without the zerolh--rder approcimation for two-layer
linear neural network in Section 4.1.1
Theorem 4.2 (Approximate DP-LP loss convergence). If Assumption 3.1 and Assumption 3.2 hold at t 0, we can bound the loss after running DP-LP for t=
Z.(0 BIT + 11 (1-E-BITT S (T(T) S
(20)
[. e B2T + Ag (1 e B2T)
B2
where Lt(t) denotes the training loss of data points labeled c (-1,1), -= L1 + L2, and
A Ewyes, m wJ x=]=
Bi f0²²² (Enmeo |[relu Ix 1=)
(21)
A2 = Emyes. [ming: wz xz]
B, 102 Eyyze |rolu(WTT 1/2
constants for DP-LP.
When we set n h 2, 1/1 achieve tight bound on the DP-LP loss.
122 W1
21 WW2 -x2,
the upper and lower bounds are equal and we
9

---

and Published in Transactions on Machine Learning Research (11/2025)
Theorem 4.3 (Approximate DP-FFT loss convergence). For simplicity, we assume that ||2:||2 R for all E [n]. If Assumplio 3.1 and Assumplion 3.2 hold, and we consider balanced inilialization | 1) ||vol
MMin et al. 2023a) at t = 0, then
(i) we lower bound the loss after running DP-FFT for > 0:
Lc(T)
(22)
E.[0] Ea exp0= + B: 11e eA---
where we define A || 2R C R o" and de 2RC.
(ii) we upper bound the loss after rumning DP-FFT for T 0:
(c(T) S Bi E (. e A.C.T
(23)
Cu
where we define A = Ew,ys, [2),-0 + |00 || Bu = 1222² and C. = 122
4.1.1 Theory without the zeroth-order approximation (2-layer linear network)
We complement the results in Section 4.1 by removing the zeroth-order approximation in simpler sctup:
2-layer linear networks for a regression lask. We define a linear network by replacing the ReLU activation e with an identity function in Equation 110 We collect the data inputs in a matrix X E Ruxd, and put the labels in a vector Y IR, For simplicity, we assume that n 2 and x = ld,xdd We consider the MSE training loss [Vv, W) = Eiemm(( W WTx;- y1)² 2 X YI2..
Note that the loss function is nonconvex in the parameters being fine-tuned, the gradient descent training becomes nonlinear dynamical system. This significantly complicates theoretical analysis. Prior works
have dealt with the challenging analysis by using heavy approximations (Bu et al. 2023 Ye et al. 2023b). We overcome these theoretical difficulties by using conservation laws and geometric properties of Langevin dynamics (sce Appendix for more detail).
Pretrained features. We evaluate a backbone W by the least square error:
>() == inf C(u, W) Y - XW(WW)))
(24)
wERt
where (-)t denotes the pseudo inverse of a matrix. This metric measures the optimal loss for LPP when fixing the current features. 7 >(W) denotes the initial least square error. We suppose Wo has orthonormal
columns, following prior works (Tripuraneni et al. 2020 Kunar et al. 2022).
Theorem 4.4 (DP-LP loss convergence). If we randomly inilialize the lineur head vo run liner probing for time T, then
1 (h. + (| + (7 (1 TT B[C(T)) < 2
(25)
In this thcorem, the first term describes that the loss tends exponentially decrease, while the second term describes the limiting behavior induced by linear probing and the added noise.
Theorem 4.5 (DP-FFT loss convergence). If up N 3lxxk) and Assumption holds, and we run fine-tuning (Equation 127 for time T, then the loss converges:
B[C(T)] < 1 (h3 + 1 AT + +L AAT
(26)
A h.3 20 (1 + d,))
where
- o2
[1+ Y +2
10

---

Published in Transactions on Machine Learning Research (11/2025)
This upper bound has similar form to Equation 25
while the factor A of the exponential terms depends
the initialization and the noise. When we take limit - 0 in Theorem 4.4 and 4.5 the Langevin diffusion
degenerates to a gradient flow and the loss converges exponentially to zero as T 0c. This recovers known results from the non-private optimization literature (Min et al. 2023a
The bounds in Section 4.1 and Section 4.1.1 exhibit different dependencies on the hidden dimension h and the data dimension dı due the differing curvature properties of the loss functions in each setup. The underlying reason is that the noise term introduced by Itô's formula (Equation (2)) is inftuenced by the curvature of the loss function. While the square function has constant curvature, the exponential function does not, leading to varying noise impacts.
5 Budget Allocation between DP-LP and DP-FFT
Finally, we consider the DP-LP-FFT fine-tuning strategy, which first applies DP-LP for some portion r of the privacy budget (i.e for some number of training iterations), then uses the remaining privacy budget for DP-FFT. In this section, we ask: given fixed privacy budget, how should we allocate it across DP-LP and DP-FFT? Our results, both theoretical and empirical, suggest that at low total privacy budget, should allocate more of the total privacy budget to DP-LP.
5.1 Results under Zeroth-order Approximation
We first show how to allocate privacy budget to avoid the feature distortion analyzed in Section using the zeroth-order approximation.
Theorem 5.1 (Estimated privacy budget allocated to DP-LP). If Assumption 3.1 and Assumption 3.2 hold = 0, then for any p E (0, with probobility (1 -p),, we can avoid feature distortion by spending
r r o pl((2pp)
(27)
amount r of privacy budget on DP-LP, where a is the noise multiplier. That is, we ensure that Vj E [h], and any after DP-LP,
cos (wy, Ra())
(28)
Ōt
According to Theorem 5.1 greater proportion of the privacy budget should be allocated to DP-LP when the total privacy budget is smaller.
5.2 Results without approximation (2-layer linear network)
Complementing the result of Section 5.1 we use the 2-layer linear model of Section 4.1.1 to show that DP- LP-FFT may work better in some set selling Lings than linear probing or full fine-tuning alone. Linear probing first can accelerate fine-tuning by aligning the linear head. The following result provides a convergence bound for DP-LP-FFT when we lincar-probe for time tip, and then fully finc-tune for time
Proposition 5.2 (Convergence of DP-LP-FFTT, Suppose randomly initialize the linear head to N(0, xTxx) and Assumption E.7 hold. We run linear probing for time tip and then fine-tuning (Equa- tion equation 127 for time t then the loss is upper bounded by:
E(((t)] < E[Lipe AAt (1-
(29)
where Lip the expected loss after linear probing, A = - da), and L (1+d2) Y The coefficient A E. Ammax (D)) increases as lp increases when we run linear probing in fovile time interval tip
hlo²-9)
[,, XTYIE
Corollary 5.3. Suppose randomly initialize the linear head vo NO 3lxxh) and Assumption E. trobinn-then-fiieturee (LP-FFT), could achieve tighter
hold. Then the two-phase method, loss upper bound Ihan lineur probing or fene-luning in expectation is we forst run linear probing for lip <
In 3 + [W. X
1(02-8)
11

---

Published in Transactions on Machine Learning Research (11/2025)
Corollary 5.3 suggests that when we fix other hyperparameters (e.g. the total training time T), the perfor- k(a -8f [IBa**YB
mance of LP-FFT depends on the noise scale a. If is large cnough such that T < In 3 +
then LP may be the best; if a is small enough such that In 3 + the best; LP-FT could achieve the best performance when the noise scale is in proper interval o² e (3-2 V 3 + (e" 3) Y11
k(a2-8) x
I 0, then FT may be
In our theory without approximation, these predictions are based only on upper bounds, so we cannot conclusively say that any fine-tuning approach outperforms another. Nonetheless, thcoretical results in two approaches suggest that the smaller the total budget, the more privacy budget should be allotted to DP-LP.
5.3 Experiments
MoCo-v2
MoCo-v3
MoCo-v2
MoCo-v3
Tedt accurncy (%)
Text accurecy
Teat accurey
50
203cp 800pp
500cp
200cp
303cp
1000gp
1000ep
10
16
LP upochs
LP spochs
epucts
upochs
(a) Private utility curves (ar 0.3)
(b) Non-private utility curves
Figure 6: Utility curves for pretraining on ImageNet- 1K and fine-tuingg on CIFAR-10 over ResNet-50, with pretrained features from MoCo-v2 and MoCo-v3 (Chen et al. 2020b Chen et al. 2021]. We compare the performance from pre-trained weights of different pre-training epochs (200/800 800 epochs for MoCo-v2, 300/ Ik epochs for MoCo-v3). The x-axis sweeps the number of LP epochs from 0 10; the remaining epochs (out of 10) use FFT.
To illustrate the privacy budget trade-off, we empirically evaluate the benefits of DP-LP-FFT on real data and architectures. For experiments in Table [ and Table 2 we use clipping thresholds C=0.1 and C=1, use batch size 1000 and sweep over learning rates {9, 5, 1, 0.5, 0.2, 0.15, 0.1, 0.05, 0.025J. These values are based on cstablished empirical studics that explore optimal clipping thresholds for DP-SGD, In particular, Appendix B.1 De el al. (2022) provides an in-depth analysis of clipping norms, concluding with the choice of C=1 for their primary experiments. Our experimental settings also draw from the methodologies outlined in Tang et al. (2023).
DP-LP-FFT outperforms other fine-tuning methods: Pre-training on synthetic data. We fol- low the setup in TTagg et al. (2023) and generate utility curves for e = 1,2, (Figure 2b We pre-train WideResNet with synthetic images generated from StyleGGANorriente (Baradad et al. 2021 and fine-tune it with DP-SGD on CIFAR-10. 10. The x-axis sweeps the fraction of privacy budget allocated to DP-LP, and the remaining budget is used for DP-FFT. We find that at various privacy levels, DP-LP-FFT gives clear advantage over either DP-FFT or DP-LP alone.
Figure 2b presents different trend from our theoretical prediction, where expect the optimal budget ratio for DP-LP to increase as the privacy noise grows. A possible intuitive explanation is that, in the Figure 2b experiments, the pre-training data is synthetic, making it 'distant' from the CIFAR-10 fine-tuning data distribution. This divergence may violate our assumption that the pre-trained weights wy are well-aligned with the fine-tuning data Z,.
DP-LP-FFT outperforms other fine-tuning methods: Pre-training on ImageNet-1K. Figure 0 illustrates the utility curves ResNet-50 for 0, 0.3. Here fix and vary eLP to trace the full utility
12

---

Published in Transactions on Machine Learning Research (11/2025)
curve predicted by Corollary 5.3 Table 1 instead varies (hence c) at fixed eLP 5. | To demonstrate utility curves for DP-I.P-FFT, we vary the number of epochs of linear probing from eLP = 0 to eLP 10; all remaining epochs (out of 10 total) are allocated to full fine-tuning, i.e, EFFT = 10 ELP. Note that full fino-tuning corresponds to ELP = 0 (the leftmost point of our subplots), and lincar probing corresponds to PLP = 10. We observe that for non-private optimization (Figure Gbb, full fine-tuning achieves the highest test accuracy. However, for DP-SGD (Figure (a), linear probing outperforms full fine-tuning, and DP-LP-FFT
outperforms both DP-LP and DP-FFT.
Model
ResNet1s
MobileNet,s
Transformelhorr
E
00
1.29
0.57
0
1.29
0.57
00
1.29
0.26
LP
68.54..0z 02
67. 12
66.60p.04
71.120.31
69.54o.c .os
67.32..om .03
95. 95.740.04
93.61o.08
94.21o. 0.
LP-FFT
72.660.12 12
68.65g.os
59.791.03
71.30o.11
71. 11.180.00
66.94..08
96.82p.o8
93.66g.
93.62..05
FFT
73.69o.03
59.79
53.82p.37
77.02p.31
63.06g.o0
45. 12
96.170.08
90.3
84.19a.so
Table 1: Test accuracies of DP-LP, DP-LP-FFT, and DP-FFT various architectures.
Comparing DP fine-tuning methods. As suggested by Theorem 5.1 and Corollary 5.3 as the noise scale a increases, the best. fine-tuning strategy changes from DP-FFT (small σ, low privacy regime) to DP- LP-FFT, to DP-LP (large 0, high privacy regime). To qualitatively test this prediction, we sweep over different noise scales a and fix other hyperparameters in each benchmark and model architocture. We sort the rows by the number of parameters of each model and the noise scale in an ascending order. For each experiment setting, we report average test accuracies with standard errors. As expected, among the three fine-tuning methods (Table [ DP-FFT almost always does the best under small noise scales (including the non-private setting where o = 0), DP-LP-FFT does the best under moderate noise scales, and DP-LP does the best under large noise scales. The close non-DP (<) performance of FFT and I.P-FFT on transformer
architectures is consistent with previous observations in Kumar et al. 2022
Table 1).
Transformerpair
c
12.28
1.29
0.57
0.26
LP
95.81p.05
95.55g.05
94.80p.06
94.21g.os
92.48p.27
LP-LoRA
96.20.o0
95.90p.03
94.81o.08
94.18p.05
91.99o.19
LoRA
96.26p.u 05
95.50o.06
94.766.0s
93.05p 0.00
91.28p.43
Table 2: Test accuracies of LP, LP-LORA, LoRA on Transformeroai..
More experiments on paramctor-efficien fine-tuning (PEFT) methods. We conduct experiments with another fine-tuning trick: differentially private LoRA (Hu el al. 2022a).. We run experiments on the Mini-DeiT-Ti architecture, where we use LoRA instead of full fine-tuning In these experiments (Table 2 our batch size 1000, and our LoRA rank is full fine-tuning; namely, as we increase the noise scale (i.e., as we reduce epsilon, giving a stronger privacy guarantee), it becomes more beneficial to use I.P.LoRA even just L.P.
We obscrve the same trend as what we saw for
6 Conclusion and Discussion
We characterize the training dynamics of DP fine-tuning under simplified theoretic setup (2-layer neu- ral networks, separable datasets with -1/1 labels) using Langevin diffusion-based approximation of DP- SGD, with an asymptotic expansion of random perturbations in dynamical systems as an approximation for Langevin diffusion. Our theory identifies and explains the phenomenon of representation distortion and alignment during DP fine-tum fine-tning, which we confirm empirically. Our work takes a step towards understand- ing how different private fine-tuning strategics can be mixed to improve performance, which could be useful
The model performance is compromised because we replace the BatchNorm loffe & Szegody 2015 in the pro-trained weights with GroupNorm (W & He 2018 BatchNorm relles batch statistics, which conflicts with the principles of differential privacy.
13

---

Published in Transactions on Machine Learning Research (11/2025)
for designing or mixing other strategies, such as memory--fficient zeroth-order optimization with differential privacy (Zhang et al. 2024a).
Limitations and open questions There are several open questions we cannot cover in this work, such as generalizing our results to multi-layer neural networks with our approximation technique, the effect other loss functions on the fine-tuning dynamics, and loss lower bounds for DP-LP /FFT without the zeroth-order DPFFP
approximation. Moreover, et al. 2022b), as well as generative models for which neural collapse does not happen. Understanding whether the zeroth-order approxination can facilitate analysis in these settings is an interesting and important question for future work.
unclear how to apply our theory to other fine-tuning methods like LoRA (Hul
Reproducibility Statement. We have included full proofs for all theoretical results and sufficient exper- imental details appendices to reproduce our results. We will also release our code under a permissive open-source license upon acceptance.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Πya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Confer- ence on Computer and Communications Security, CCS '16, pp. 108-318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394. doi: 10.114//297674929988888 URL
https://doi. 10. 11w.//97717679...0 2978311 2978318
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representationall collapse. In International Conference on Learning Repre- sentations, 2021. URL /atpp::/openrevveer htt.://www......oooooooooooooooo
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit accelcration by overparameterrvation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 244 253. PMLR, 10-15 Jul 2018. URL htttps://proceeddngg/ mll. press/ hatss/v80/aaaaa 2000/rroa18a... html
Manel Baradad, Jonas Wullf, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. In Advances in Newral Information Processing Systems, 2021.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algo- rithms and tight error bounds. In Proceedings of the 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, FOCS '14, pp. 464 473, USA, 2014. IEEE Computer Society. ISBN 9781479965175. doi: 10. 10.1109/FOCS. 201 4.56. URL https: /rgp::////ooooo////////// org/10r.//1 10. 1109/FOCS. /01.///............... 2014.56.
Louis Béthune, Thomas Massena, Thibaut Boissin, Aurélien Bellet, Franck Mamalet, Yannick Prudent, Corentin Friedrich, Mathieu Serrurier, and David Vigouroux. DP-SGD without clipping: The lipschitz neural network way. In The Twelfth International Conference Learning Representations, 2024. URL
https://openreview wwwpeenreviee.eeeforruueeee http:/oorrmmiddBByyzzZZZez
Zhiqi Bu, Hua Wang, Zongyu Dai, and Qi Long. On the convergence and calibration of deep learning with differential privacy. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
https://opeereviiee net:/forumid==KOCGG
Sandra Cerrai. Second Order PDE's in Finite and Infinite Dimension: A Probabilistic Approach. Springer, Berlin, 2002.
Annie S Chen, Yoonho Lee, Amrith Setlur, Sergey Levine, and Chelsea Finn. Project and probe: Sample- efficient adaptation by interpolating orthogonal features. In The Twelfth International Conference on Learning Represenlations, 2024. URL http:://pppaaeeeeeeeeee//rrrraaa wwtt::/foormmmii1-ff/002....
Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding gradient clipping private sgd: geo- metric perspective. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020a. Curran Associates Inc. ISBN 9781713829546.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved basclines with momentum contrastive learning. aixiv preprint arXi::203.04220 04277, 2020b.
Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision trans- formers. arXiv preprint arXiv:2104. 02057, 2021.
Yubci Chen, Zeyu Yun, Yi Ma, Bruno Olshausen, and Yann LcCun. Minimalistic unsupervised represcntation learning with the sparse manifold transform. In The Eleventh International Conference on Learning Representations, 2023. URL https::/openrevieen wttpiiiwwweweeeeeeeee foedffonm? nBVKAhhD
Rishav Chourasia, Jiayuan Ye, and Reza Shokri. Differential privacy dynamics of langevin diffusion and noisy gradient descent. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Newral Information Processing Systems, volume 34, pp. 14771-14781. Cur- ran Associales, Inc., 2021. URL https://proceeeeee inw.rrrreennnnn..e.rri.i.sp..p.s newrips wwtprris.ccpppppppp httllppppprppappllallllappaaapee /wwes//appr///222/ 7c6cla770441175566666666222 htt.aaac7bbbbbbb133322222ccaaae pdf
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215 223, Fort Lauderdale, FL, USA, 11 13 Apr 2011. PMLR. URL https: 7/proceedings mml.iiinnnmmm......mmmmmm.m..e.m
Soham Do, Lconard Berrada, Jamic Hayes, Samuel L Smith, and Borja Balle. Unlocking High-Accuracy Diflerentially Private Image Classification through Scale. arXiv preprinl 2fie:2244.13500 2022.
Simon Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- Associates, Inc., 2018. URL https::/proocedd wwwrwps.cc/ppap....pe httpppppe..leeeeepppr/////////// pdf
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Compat. Sei., 9(3 4):211 407, aug 2014. ISSN 1551-305X. doi: 00.1561/0400000042. URL https:
//doi org/0. 1561/0400000042
Huang Fang, Xiaoyun Li, Chenglin Fan, and Ping Li. Improved convergence of differential private SGD with gradient clipping. In The Eleventh International Conference on Learning Regresentations, 2023. URL
httpps:/
M.I. Freidlin, J. Szües, and A.D. Wentzell. Random Perturbations of Dynamical Systems. Grundlehren der mathematischen Wissenschaften. Springer, 2012. ISBN 9783642258473. URL htttp:/bbooks http://bookk.ogggge.e..o google. books??ddrP@LPMIIAANEC
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembligg for visual domain adaptation. Tn International Conference on Learning Representations, 2018 URL http::///pparevieeeeeeeettrorres
rkpoTaxA-
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Representations, 2022. URL httpps://ppearrriie..eeee//////s ittperrrrreeeeeteffouuumiiit
SwI [41410
Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Guha Thakurta, and Lun Wang. Why is public pretraining necessary for private model training? In International Conference on Machine Learning, pp. 10611-10622 PMI.R, 2023a.
Arun Ganesh, Abhradeep Thakurta, and Jalaj Upadhyay. Universality of langevin diffusion for private optimization, with applications to sampling from rashomon sets. In Gergely Neu and Lorenzo Rosasco (ods.), Proceedings of Thirty Sirth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 1730-1773. PMLR, 12-15 Jul 2023b. URL https: https://prooeediings. mlr. .ares// #195/ganesh233 html
15

---

Published in Transactions on Machine Learning Research (11/2025)
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila. Pires, Zhaohan Daniel Guo, Mohammad Cheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning, 2020.
Shlomo Hoory, Amir Feder, Avichai Tendler, Alon Cohen, Sofia Erell, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, and Yossi Matias. Learning and evaluating a differentially private pre-trained language model. In Oluwaseyi Feyisetan, Sepideh Ghanavati, Shervin Malmasi, and Patricia Thaine (eds.), Proceedings of the Third Workshop Privacy in Natural Language Processing, pp. 21-29, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.1 20118665112202 22p1p////// 3. URL https://aclantholl. ottpoooooogoooo/22111111 wrt.gy/ooor//.......aaaaeell/l/. privatenlp-1.3
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022a. URL. /ttps:://ppenreviett http://wwweeeeoooon/nn/////eeeee
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Represen- lations, 2022b.
Sergey Tolle and Christian Szegedy. Batch normalization: accelerating deep net network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- - Volume 37, (,, pp. 448-456. JMLR.org, 2015.
Kiyosi Ito. On stochastic differential equations. Mem. Amer. Math. Soc., 1951(4):51, 1951. ISSN 0065-9266.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https://proceeddngg: meurips.cc/paate httppppeiilleeepppee//////////// becbetts3466bbaaaeettttttta22243 Paper. pdf.
Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions on Machine Leurning Research, 2023. ISSN 2835-8856.
URL http:://openrevvieeeneeferrrrrre net//oruniff
QTXocpAP9p
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Canadian Institute for Advanced Research, 2009.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang, Fine-tuning distort pretrained features and underperform out-of-distribation. In International Conference on Learning Representations, 2022. URL
Jaehoon Tee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models un- der gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alchó-Buc, E. Fox, and R. Garneut (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: https:///proeee http:///poeeeeenn..eeee neurdnng...n...n..oeeeeeeermn www.npps.c.ppppprrr.eeeeeeeee cc/paper ftlees/paper/2199/LL.. pdf.
Don S. Lemons and Anthony Gythiel. Paul Langevin's 1908 paper "On the Theory of Brownian Motion"
["Sur la théorie du mouvement brownien," " C. R. Acad. Sei. (Paris) 146, 530-533 (1908)]. American
Journal of Physics, 65(1)):1079 1081, 11 1997. ISSN 0002-9505. doi: 10.1119/1.18725, URL https:
//doi.org/10 10. 1119/1. 18725
Jan R. Magnus and Heinz Neudecker. Matriz Differential Calculus with Applications in Statistics and Econo-
metrics. John Wiley, second edition, 1999. ISBN 0471 196321 9780471986324 0471986333 9780471986331.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Xuerong. Mao. Stochastic differential equations and their applications / Xuerong Mao. Horwood series in mathematios & applications. Horwood Pub., Chichester, 1997. ISBN 1898563268.
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: conservation laws for gradient flows. In Teirby-sevenlh Conference on Neural Information Processing Systems, 2023.
URL https:// wttp://oooooomm/////aaaaaaaaaaa
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. URL httts://aarzi: http://aarivv.orrra/////////// 03426
Hancheng Min, Salma Tarmoun, Rene Vidal, and Enrique Mallada. On the explicit role of initialization the convergence and implicit bias of overparametrized linear networks. In Marina Meila and Tong Zhang (eds.); Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings mlr.
of Machine Learning Rescarch, pp. 7760-7768. PMLR, 18- Jul 2021. URL httptttt////tceeeeetees press/v139/min2c..hr
Hancheng Min, Rene Vidal, and Enrique Mallada. On the convergence of gradient flow multi-layer linear models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 18550-2487.. PMLR, 23 23-29 29 Jul 2023a. URL https::/proceeddngg .ml....sss.......m.............. wttp:/ggs......rrr///eeee// ppress/v02/mmm22222 html
Hancheng Min, Rene Vidal, and Enrique Mallada. On the convergence of gradient flow multi-layer linear models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (ods.): Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 24850 24887. PMLR, 23 29 Jul 2023b. URL
mlr, S.lr.press/vee aress/v202/min225 html
Hancheng Min, Enrique Mallada, and Rene Vidal. Early neuron alignment in two-layer reLU networks with small initialization. In The Twelfth International Conference on Learning Representalions, 2024. URL
https: https://openrevier: htt.rr/rw.....ooooouooooooooaaaa
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-212111 2017. doi: 10. 2017.1017.11.
Bernt Øksendal. Stochastic Differential Equations: An Introduction with Applications (Universilent)
Springer, 6th edition, Jamuary 2014. ISBN 3540047581. URL http://www. amazon, com/ /ete//exccccc./////////////////e
Guillermo Ortiz-Jimenez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neu- ral networks actually say about generalization? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 8998 9010. Curran Associates, Inc., 2021. URL httpss:/prroeeeiing..eem httpg//nw...iiiieeeeperreeeeeeee wwwpppapppaaiiaeeeeee files/
Natalia Ponomareva, Sergei Vassilvitskii, Zheng Xu, Brendan McMahan, Alexcy Kurakin, and Chiyaun Zhang. How to dp-ly ml: practical butorial to machine learning with differential privacy. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Dato Mining, KDD '23, pp. 5823-56 5824, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/ 3580305.3599561 URL https: org/ 1145//5803355.... 35995611
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-25) 2015. doi: 10.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A the- oretical analysis of contrastive unsupervised representation learning. Tn Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, vol- ume 97 of Proceedings of Machine Learning Research, pp. 5628-5637. PMLR, 09-15 Jun 2019. URL
httpss:/prroeeednng mlr. http:///////uushhiitta
Anatoli V. Skorokhod, Frank C. Hoppensteadt, and Habib Salehi. Random Perturbation Methods with Applications in Science and Engineering. Applied mathematical sciences (Springer-Verlee New York Inc.) V. 150. Springer, 2002. ISBN 0387954279. doi: 1..1115/..159455
Xinyu Tang, Ashwinee Panda, Vikash Sehwag, and Prateek Mittal. Differentially private image classification by learning priors from random processes. CoRR, abs/2330.06077/ 2023. URL https:://ool orrooi..rrr/1/0 10.
48550/arXiv. 2306.06076 06076
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolle,, and Herve Je- gou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, vol- ume 139 of Proceedings of Machine Learning Research, pp. 10357. PMLR, 18 24 Jul 2021. URL mmr.inngg.........m...m.....m..m www.prees//viee (ieess/v33/touvron222 htmm
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning; The im- portance of task diversity. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systeme, volume 33, pp- 7852-7862 Curran Associates, Inc., 2020. URL 55627FFFee1177443344422012116661
hetgiss.eurip... www.ips.cc/ppppree.ee ftless/paper/222///i/ pdf
Puja Trivedi, Danai Koutra, and Jayaraman J. Thiagarajan. A closer look at model adaptation using feature distortion and simplicity bias. In The Eleventh International Conference on Learning Representations, 2023. URL http:::t//////tt /tttps://openrevieer www.eeww.eeeeeeeeeeeeeeeeeeeeeee fode://frummdddwk._...m 11..-44-i100 IwTZ
A J Veretennikov. On strong solutions and explicit formulas for solutions of stochastic integral equations. Mathematics of the USSR-Sbornik, 39(3)3387, apr 1981. doi: (..1070/SM10811133900....
Chendi Wang, Yuqing Zhu, Weijie J Su, and Yu-Xiang Wang. Neural collapse meets differential privacy: Curious behaviors of NoisyGD with near-perfect representation learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Leurning Research, pp. 52334 -23660. PMLR, 21 21-27 Jul 2024. URL press/v235/wast / 1ttml
mlr.
Di Wang, Changyou Chen, and Jinhui Xu. Differentially private empirical risk minimization with non- convex loss functions. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6526 6535. PMLR, 09 15 Jun 2019. URL pattps://proceesee
mlr. presss/997/vaagg1..c.t hhtml
Puyu Wang, Yunwen Lei, Yiming Ying, and Hai Zhang. Differentially private sgd with non-smooth losses. Applied and Computational Harmonic Analysis, 56:306-333 2022. ISSN 1063-5203. doi:
.9.0011. URL https::/www.scienneee
com/sscience/articc....
pi1//81063520220008811
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Compuler Vision (ECCV), September 2018.
Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, and Volkan Cevher. Initialization matters: Privacy- utility analysis of overparameterized neural networks. In Sherth- Conference on Neural Information Processing Systems, 2023a. URL http::///ppaaaeeeeee.eee////ttta
18

---

Published in Transactions on Machine Learning Research (11/2025)
Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, and Volkan Cevher. Initialization matters: Privacy- utility analysis of overparameterized neural networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 5419 5446. Curran Associates, Inc., 2023b. URL https://prroceedinggs www.e/iiiiinnnnnnnnnneeeeeeeeeee www..mee...ppppppeeeee files/paper/
pdff.
Tian Ye and Simon Shaolei Du. Global convergence of gradient descent for asymmetric low-rank malrix factorization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advences in Newral Information Processing Systems, 2021. URL https://openrevieet wwt. http:/pparrreieeeeeeeeerrrrrrraa
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarmi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differ- entially private fine-tuning of language models. In International Conference on Learning Represenlations. 2022. http:://pppprrrrreeee..//ooooera
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In British Machine Vision Conference 2016, York, France, Jamuary 2016. British Machine Vision Association. doi: 10.5244/C.30.86 URL https://enpc.. hall hci.laacccccce//a0011255555
Jinnian Zhang, Houwen Peng, Kan Wu, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In 2022 IEEE/CW ECVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12135-12144, 2022. doi: 10. 20000//VVR2266822222222
Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. DPZero: private fine- tuning of language models without backpropagation. In Forty-first International Conference on Machine Learning, 2024a.
Xinwei Zhang, Zhiqi Bu, Steven Wu, and Mingyi Hong. Differentially private SGD without clipping bias: An frror-feedback approach. In The Twelfth International Conference on Learning Representations, 2024b. URL httpps://operrevier net/ffrumm SittimidduuuFuuFuilia
A Additional experiment results
In this section, we provide more experiment results and detailed configurations.
Evaluations back in the pre-training distribution (Figure 7). We also evaluate the feature quality on ImageNetl-K, the pre-training dataset. The representation alignment for the pre-training domain is different: once a proper alignment is achieved, the backbone gradually recovers portion of its original feature quality, which had been compromised due to DP noise and distribution-shff.
Pretraining on Public Data
fine-tuning on Private Data
Feature Quality
0.3
Feature Distortion
0.2
Representation
0.1
Alignment
0.0
0
16
61
Epochs
Figure 7: Backbone feature quality evaluated by average top-1 kNN accuracy the pro-training dataset, for ResNet-50, through public pre-training ImageNet-1K and differentially private fine-tuning on STL-10.
Experiment setup in Table 1. We use batch size 1000, clipping thresholds C=0.1 and C=1, and sweep range of learning rates (9,5, 5, 1, 0.5.5, 0.2, 0.15, 0.1, 0.05, 0.025).
19

---

Published in Transactions on Machine Learning Research (11/2025)
Summary of experiment configurations. We run experiments five deep learning models and four transfer learning benchmarks to verify if our theoretical prediction, the existence of concave utility curves, generalizes to deep neural networks and real datasets. Each experimental setting comprises: (1) model architecture, (2) a (larger) dataset for public pretraining, and (3) a (smaller) datasct as the private data for fine-tuning. The benchmarks we use are:
LingeNee-IKK-CIFARR-- ImageNet-IK large-scale dataset. We initialize pretrained features of ResNct-50 from MoCo-v2 Chen et al. (2020b) and MoCo-v3 Chen ct al. (2021). trained on ImageNet-113 Russakovsky et al. 2015 without privacy. We then privately fine-tune the ResNet-50 on CIFAR-10.
ImageNet-IK-STTI-1 We pretrain a DefT model ImageNet then pretrain Mini-DeiT-T model with weight distillation from the DeiT model Touvron el al. (2021); ); Zhang et al. (2022) After that, we privately fine-tune the Mini-DeiT--D model on STL-10 Coates et al. (2011) for 20 epochs.
CIFAR-10-STI-10. We pretrain the feature extractor CIFAR-10 Krizhevsky (2009) using stochastic gradient descent without privacy mechanisms. Then finetune the pretrained features and randomly initialized linear head on STL-10. This benchmark has been studied in the context of domain adaptation French et al. (2018); Kumar et al. (2022). The training subset STL-10 only contains 500 images. To align with the small scale fine-tuning data, we run the experiments with smaller and data-efficient models: MobileNet-v3 and ResNet-18.
CIFAR-. To reproduce the results of Tang et al. (2023 and verify the general existence of concave utility curves, we also consider a slightly non-standard pretraining protocol. We pretrain a wide residual network (WRN) Zagoruyko & Komodakis (2016) on synthetic images generated by random diffusion processes. We follow the settings in Tang et al. (023
We employ early stopping, and select the optimal learning rate based on the accuracy of the in-distribution validalion.
A.1 Privacy-utility curves
We further plot the privacy-utility curves to aid the information in Table
As expected, accuracy increases with epsilon for every method and backbone, and the results generally (but not always) qualitatively match our theoretical predictions.
For Mini-DeiT-Ti ,th ViT-style backbone is comparatively robust. DP-LP-FFT retains the lead in high epsilon regimes while DP-LP wins for small epsilons, predicted by our theory.
For MobileNet-v3 and ResNet-18. 18, the cross-over pattern is different from Mini-DeiT: even at moderate epsilon, DP-LP-FFT outperforms DP-LP, and under strong privacy DP-LP is best. And DP-FFT retains the lead over the high epsilon regime. This suggests that small conv-nets are more prone to head-induced distortion, so the front-loading budget into LP pays off sooner.
With deeper conv-net, the trends predicted by our theory persist: DP-FFT wins at large epsilon, DP-LP-FFT at moderate epsilon, DP-LP at small epsilon. The DP-LP-FFT curve sits close DP-FFT in the high-epsilon regime (no downside when noise is small) yet clearly exceeds it as epsilon shrinks, which is exactly the "mitigale-then--nne-tuee" behavior predicted by Theorem 3.3 and Theorem 3.4
A.2 Explanation on side examples
Figure 2b follows Tang et al. 2023 protocol, which introduces EMA smoothing and gradient-averaging across augmentations before clipping. These two ingredients are absent from our theoretical sctup, and these modifications dampen the representatinn-dittttioon predicted by Theorem 3.3 Our interpretation of Figure 2b is currently heuristic and is early-stage conjecture rather than formally proved result.
20

---

Published in Transactions on Machine Learning Research (11/2025)
ResNet-18 18 (CIFAR-1O-SSL-10)
ReaNot-50 LingeNes-KK-CIAARRI
D0P1P D7 FFT
01-2
DP-FFT
Test socuracy (%)
Test sccurazy
90
50-
40-
73902
25.1
3.1
73902
928
20.1 Fysilon
Ppsilon
(a) ResNet architectures
MobileNet.v3 10-1AR-1---TL-100
Mimi-DeiT-Tii (ImageNet -S-LL-00
DP-LP
75
DP DP-LP.FFT
90-
10
Test accuncy (%)
65
Test sccurnry
60
20
50
DP-1P-11 -17
65
DP-IFT
Epeilon
Epoilon
(b) Other architectures
Figure
21

---

Published in Transactions on Machine Learning Research (11/2025)
1. EMA: Tang et al. (2023) maintain EMA copy of the network parameters and report accuracy with that averaged model. E.M acts as low-pass filter on the parameter trajectory, effectively smoothing out the rapid weight adjustments induced by the large imitial head-gradient. This could delay the transient distortion our theory attributes to the first few DP-FFT steps.
2. Gradient averaging gradients of multiple augmentations of the same image. Averaging reduces variance and shrinks the expected norm of each per-example gradient, lowering the probability that the clipping threshold is hit. Consequently, the random-initialisation error injocted by the head could have a smaller effective magnitude. This potentially mitigates the early distortion phase.
augmentations: Before per-example clipping, Tang et al. (2023 average the
Technical results
Lemma B.1 (Holder's inequality for sums). For a sequence x = [m:]]=1 of posilâve real numbers and p > 0, define |æ]|p (E== x;) Then for any pair of positive real numbers 0, 0 with 1 any puir of sequence of posilive real numbers and y.
1, and
| I |a>||p
Lemma B.2 (Reverse Holder's 's inequality for sums). For sequence x = [m.]]=1 of positive real numbers and p O, define ||æ||p- == Eii= x²") 1/. Then for any pair of posilive real numbeer p > O, q > with 1 1 and any pair of sequence of positive real rumbers x and y,
|zy|| 2 ||01bel1al1
Lemma B.3 (Reverse QM-AM incquality for sums). For sequence [x;] of positive real numbers,
x
Lemma B.4 (p-coherent data conic hull (Min et al.
2024
Lemma 5)). Define a conic hull == CH CH(lya
E [n]])= (Σ 0:y4: Va, 2 0, [n]]- Tf Assumption
3.1 holds, i.e. the dataset separable, then K is
p--coherent:
Azn 22 KV (O), cos(21, 22) 2 μ
Corollary B.5 (Orthogonally separable = linearly separable MMin et al. 2024 ). If Assumption 3.1 holds,
then By > 0 and z E SD-1 such that
Vi E [n]. (e(2, xx) 27
Proof of Corollary B.5
We prove the existence statement by picking a valid pair of 2,7. Take = 21
a la
Then Vi E [n].
(r(2, x.) = -llxalle cos(y1z1,y)z2
//by Lemma B.4
|111112/
2/ /4 min ||ar|la
*E[m]
Therefore 7 μ mineej) ||r:||2.
B.1 Relaxed assumptions
We relax Assumption 3.1 by allowing non-zero cross-class correlation, controlled by parameter p|0, 1), and we relax Assumption 3.2 by allowing bounded activation leakage of a feature wy onto the opposite class, also controlled by p (setting p recovers the original assumptions).
22

---

Published in Transactions on Machine Learning Research (11/2025)
Assumption B.6 (Relaxed data correlation). Let Te be the class means defined in the paper. There exists Min and E [0, 1) such that for all i A
(rr, 2.g}
(within class) yi = V/
2 Mins
(30)
|| =1:ا
{I zz)
(across class) y: + yj
1 ||اا S PMin
(31)
Equivalently, the (label-signed) pairwise cosine similarity has positive gap
inf (o5( sup cos(s1, my) 2 (1 p)liin > 0. y
(32)
This weakens Assumption 3.1 which enforced sign separation, to gap separation that permits some positive cross-class correlation. The original Assumption 3.d and its cone construction.
Assumption B.7 (Relaxed neural feature collapsing). Let c(j) {+1,-1} be the class index associated with feature wy (same convention as Assumption 3.2) Define the "activated mass" at t=0 for wy under the exponential loss weights 4 = xxp(-y:5(x:))
A3
Σ ((0))(u;;())=z 0},
(33)
bw=e((
AJ Σ {(0] > 0}
(34)
Assume Icakage is bounded by the same above and below:
Vj. A S PA
(35)
And the pre-trained features are not well aligned yet with the downstream data, i... we need to fine-tune the features. We describe this by an upper bound upon the alignment.
cos(To(): w)) Min p²))
(36)
This is quantitative relaxation of Assumption 3.2 (the old statement implied Aj =).
Then we show that, based on the relaxed assumptions, we can similarly prove similar result to Theorem
Theorem B.8 (Random initialization causes feature distortion). If Assumption B.6 and Assumption B. hold at f = 0, then for each j,
di cos(wj, Fe(r)
(,))),,()),
(37)
1=
with the positive lower bound
(2(0) 2(w,, Te(s)) A3 (Mm -p2) cos(Teet)) wz))-
(38)
In particular, if v0 < 0 then मे cos(w,, Te(ii)
< 0. By continuity of the Langevin dynamics, there
1=
exists At such that
cos(ws, Fo(i))
> 0. VE (0,
(39)
dt
1=
Since vo (o,,BTn), with probabilily al least 1 - 2-h lhere erists some j with vy(0) < 0.
Hence early-stage feature distortion occurs with the same high probability as Theorem 3.3 now with strength scaled by the factor (1 - p). Setting = recovers exactly the sign identity used in the proof of
Theorem 3.3 The bound monotone in the leakage: larger p weakens but does not flip the sign as long p << 1.
23

---

Published in Transactions on Machine Learning Research (11/2025)
Proof of Theorem B.8
1. Zeroth order DP-FFT dynamics. For j-th head/backbone pair, at = 0 the zeroth order ODE gives
W3 = Σ (0)) x; > 0)
(40)
EE|m]
my (0)
Σ
(,, > 0)
(41)
kE(m]
= (0)
(42)
where Zj is defined as the activated, label-signed data aggregate.
2. Derivative of the cosine. Using the exact identity for the time derivative of cos(w,, Te(t))
d cos( cos(w,, Fc(ti)
Fe()) (S w1)) 2(w0,, Fa(n) Uy (0)))S)) Zz),
(43)
dt
1w5112
1w,11
1==0
S = ||w 12 [²)r())
(De(i): wy) Wy,
(44)
that (Sy. and (S). = || || (Zn()), 22 2 0. From geometric perspective, Sj define the component of Fe(t) orthogonal to w/y-
3. Lower bound (S), Zz).
(S = Σ
( (1),, Σ (10
03(x. B(Fc(1)
(45)
y==d())
=||W,|13
{(0)1(w5 2, (2, Tc()
Σ =1
0) 03
(46)
(==
T.
- (Too), (y))
Σ [(0)1[wJx; x2 > 0) - Σ ((0)1[w]=; > 03(a, w))
(47)
Term Ta. By Assumption
B.6 and Assumption
B.7
Ta pin (in pAj) pin -²²)
(48)
Term Tb.
TB S |1w
(49)
Combine the two bounds to get
(S,, Zz> 2||w5| 11 (mn - 1²) - (āee), myy) 1
(50)
> ||lw, AA (mm (1 - p²) cos( F (()) w;))
(51)
/ /b Assumption B.7
(52)
>0.
(53)
Consequently, ae cos(w,, 2e(i)) has the same sign as (0)
24

---

Published in Transactions on Machine Learning Research (11/2025)
C
Appendix:
Representation alignment
C.1 Theory
The Langevin diffusion of w/ on a n-sized data cluster (j E [hl) is
w/ yi ( (wj x;)x; + oQQu:
(54)
where Qt is a vector containing independent 1-dimensional Brownian motion.
The Langevin diffusion of a n-sized data cluster
Σ cxxp-vuffee W, w))rclu(W x;) + odQt,
where Q: is a vector containing h independent I-dimensional Brownian motion.
We rewrite the Langevin diffusion by asymptotic expansion (Freidlin et al. 2012 Equation 2.1, Chapter 2.2),
(1) R w00) + σuj +... W Rs w + ow +...
(55)
(1)
1.c. we expand the Langevin diffusion as linear combination of the original gradient flow and a lincar stochastic diffusion.
Ci=1 yı exp (oo0v-- (x;; W ( relu(()) ) (0)
(56)
(0)
n 1-1 21 exp(-x: W )), relu''lll (ww) (T)))) x;)x;.
Tx:))
Lemma C.1 (Zeroth order invariance of locally linearized LD). If we rewrite the Longevin diffusion by asymptotic exponsion (Freidlin et al. 2012 Equation 2.1, Chapter 2.2),
Dj Djzy ;j (0) + ovj (1) uy wj + tow
then the layer invariance still holds for zeroth order approzimation
d
1 |
(57)
dt
This result is similar to the imbalance matria in gradient flow (Arora et al. 2018 Du et al. 2018 Min al. 2023a)..
We are ready to prove Theorem 3.3
Proof of Theorem
3.3 The explicit expression of the cosine value is
w Te(t)
Fe(()) Iur, 11a Fot
(58)
Without loss of generality, let |[ 1. To show that the cosine value decreases with high probability, we only need to prove that the derivative of (w:] s is negative at t = 0 with high probability. The explicit derivative expression is
wyll
i cos(w, Fo(i))
2(wJ Zo(D))
Ow; awi x " at
|| || 11
(59)
Öt
||: l1
Øt
25

---

Published in Transactions on Machine Learning Research (11/2025)
2(w} Te(i)
Jr
[|uy1BBa0)) 20)))) ot
(60)
oly 11
1/by Assumption
3.2
(61)
sign ( Öt
=sign ([1we) [1101 (z) /hoi 0t
cos( ( Fe(()))Fo())))
(62)
sign (1,(11u,,1B (z)
(63)
swg)(w;)
(64)
Since we initialize v ~ ((0,8Taxx)) with probability - 2- there exists j such that vy < 0 at t = 0 = dt cos(wy, Ba(r)) < 0 at f = 0. By the continuity of the approximated Langevin diffusion, there exists At > 0 such that for any & E At),
a cos cos(w,, Fc() 0.
(65)
Proof of Theorem 3.4 In the proof of Theorem 3.3 we show that for wy E S {-1,1},
sign cos (ws, w/, mo()))) sign(u,) sign(e)
(66)
To mitigate the feature distortion after some time index At, we only need c. by > 0. For DP-LP, every oc 11 increases/ serresss//decreaeess /eereases if c 1/-1. Therefore, for any initialization, there exists Al such that sign(y,) = sign(c) after time index At. If we switch to DP-FFT after At, & cos( W,. Re(s)) > for any j E [h]. Thus cos(wr, Tc(i) d6
is non-decreasing in DP-FFT.
Approximate convergence of DP-LP and DP-FFT
D.1 Approximate DP-LP convergence
We add some extra notations for the following proofs:
Posilive data subset Z, = {i [n] / 0)
Negative data subset I_ := {i E [n] 1 < 0}
Positive head cluster (+) {,r [h] sign(u, 0}
Negative head cluster V- (t) = { E [h] sign( (t)) < <0}
Index function I F:R. > {I+, I_} maps feature vector to its cluster
I+
w E S+
(ww)
I__ w WES_ otherwise
We first derive the upper bound for approximate DP-LP.
Upper bound proof of Theorem proximation
We construct lower bound of the drift terms in the zeroth order ap-
|
12
Σ
#i exp( exp(-yif(c) (2i ((),, v (e TT x44)
(67)
26

---

Published in Transactions on Machine Learning Research (11/2025)
y/s expl( W(O w )
(68)
j=1
(E))
2
2
min
relu((wg))) xi)
Ys expp( (:r;; W00) (((1)))
(69)
())))
2
min relu((u,))) (T) zi) (w))))
Σ
exp(- yif W (((oo))
(70)
(w))
2 )
Txs) 2 min relu((w)) (1) xz)
Σ
min iEz
(c(0))2
(c
(71)
JEV,
JEV
2
2 xi²)
2 min
min relu relu((w, (1)) ( x;} SEZ
min relu((u,))) ) xxi) iET
[1c40] +
(72)
JEV+
JEV-
22 min
min relu((n,))) x:4) FEI+
Σ
min relu((w,))) xi) -EI-
[100 + c!o
(73)
JEV+
JEV-
=2 min
Σ
min relu((u (0)) xi))
Σ
min relu((w)")) mi) JEI_
(C101)2
(74)
(EZ+
JEV
JEV_
We construct an upper bound of the diffusion terms in the zeroth order approximation
EVVi fxx; ( ( ( xx)
2
Myy, f(z;; (it) f(Coon (
=
0
2
1=1
1/by Lemma
B.1
1/2
1/2
2220
e (y: f(x; W( ((00))
(elll(( Tx;)
1/by Lemma
B.3
1/2
1
Eun f(; w(O), (000)))))))
TT Tx;))12
'50¹
1/2
0² c()) (())
rrllu( (rllu(( (w"")) xx))1
2
Then we have an upper bound
2 (T) 5
BT 1 (1 c- BT)
e
B
where constants A, are defined as
A 1 min Ejer, [mminez, relu((u,))) (T)aa) xxa)
Ejev [minez )elu((uo)0)Txz)) [=
B f02 (E ( 22)111) 1/2
27

---

Published in Transactions on Machine Learning Research (11/2025)
We give the lower bound approxiante DP-LP below. We first give a loose lower bound as warm-uip. Then we improve the techniques and provide a tight lower bound.
Loose lower bound proof of Theorem 4.2 We rewrite the Langevin diffusion by asymptotic expansion (Frei- dlin et al. 2012 Equation 2.1, Chapter 2.2)
n y (y, f(z;; w( (re
( = V c0) 12
o
V
L
Eyvs (at; W (0)))) (eel((W((0)) xx)))l
| (())
min iEV!
[[ [x:)1
1 0022 Σ
Kyr, f(z; w b(0)))
iEV!
min iEV"
|eellu ( T2,)13 0² Σ
(Vi (a;;; W(0) ((;;; ()))
SE
| c
min |eluu (()(0) xx,1)1 (E(M)
My, fag w/ ,
2
kE(m)
V (())
11
+
min (E(n)
22))11
-
1
c(
2
h
Yt exp(-uf(#; (()), ,0(0)) )rllu(()) (0)) xxt)
min ( [E(n
2 - c(0)
+
j=1
//by trapping
Σ
exp(-f(xx; (( )rluu)))) (
=-
jEvio
IEI+
2
exp(f(z;; ((O), relu (ww)))))a Tx;))
IET
+ min
iE(n [[relu((W:()) Tx:))1 0
exp(-f(zz; 10 (,110 (((o))) 2
max (relu((w)) (relu(((())T=)))) Tx:))²]
Σ
Σ
jEV(
iEz
(T xx)))²
Σ
(xp(f(x;; W(O), ((0)))
max
jEv(
FET
+ min (Irelu((W(0)) xx:>)1²
000 (())
iE(m)
1/02 + 6² < (a + 6)2 when
0,b 0
2- max (relu( (relu(ww))) Tx:))²) (EMM)
Σ
Σ
exp( exp(-f(on f(x (0)) (2(01))
JE(N]
[E[m)
28

---

Published in Transactions on Machine Learning Research (11/2025)
+ min [z) iE(n]
2 ((0)
Σ exp(-f(1x) W w (10))) )
2
1 0² c( min [rehu(((())) >>)))²² iEfn
max (relu((w)))) [2:))²
jE[R],EE|x]
(c(m)
max 2-h
(relu((w) 2x1)3 Toz)) (C:O))++ JE(A)
iE(m) min ||rell
22)11 Tx
o²
In linear probing, the coefficients h (relll [Txx)²² and 1²²²² (mingelm) [|relu((W())) (
are constants. We replace them with dummy notation A and B. We solve the first-order nonlinear ODE by turning it into a first-order linear ODE.
[ 2 - )( + BC
1
(()) 2 A+ BB 1
(C(0))2
[())
d
1
1
2 A+ B
dt
((0)
(()
1
(T)
BT + (1 e-BB
(0)
Remark D.1 (On the qualitative properties loose DP-LP lower bound). If we take the limit initial point, then the lower bound degenerate the initial loss value.
lim 2
BT + 41 e BBT) = _ 0) [(
(75)
If we take the limit infinite time,
oo² (minge(w) [eellu(( (w) Ta))) x=)))l))
B
lim
(76)
2 e BT + A (1 - 61-e-BB B-BT)
A (rax ( )3
the following interpretation holds:
1. For larger noise a 1, the lower bound higher, i.e. worse performance.
2. For bad alignment between pretrained features (0 and data points, both the denominator and the numerator could shrink. Tt is not obvious how the lower bound changes.
In the following result, we modify the proof, replace the min(-), and provide a tighter bound.
Tight lower bound proof of Theorem 4 This is alternative construction of lower bound for drift terms
in the zeroth order approximation
(1)
1
Σ yi
W(o), (((0) )cluu((() (0) (T)))
Σ
Σ
exp( - exp(-f(xx; (0))) v (0 (ell Tx:)
jEV!O)
1EI
29

---

Published in Transactions on Machine Learning Research (11/2025)
exp(f(xx W )reuu))) T xz)
JEV
iET
1/by Lemma B33
Σ
Σ cxp(-f(x;;
W 2000) )chu(()) Ta:)
2
Σ jEV!O) IET
Σ Σ exp(-f(æ; W (elu())))(
JE[N iEim]
Σ Σ
exp(-f(x;; W(0) (relu((( Txxe)
iE|ni JE[N]
r 2.)) exp(-f(x1; w ((10))) )²
JE(N]
se|n
by Lemma B.1
max relu rclu((w))) 241) /)
Σ exp(=f(x; W(0) ((01))
jE(M
se|n
iE|ml
by Lemma
B.3
12)
Σ
max relu ( x25)
Σ
exp(-5(xx; W01), ,,(0))))))))
[E(M
sEin
KE|n]
max relu((u (1)) 2i) (c(0))2 [e[h]
ie|m
We replace the A constant by Eneto maxxej relu((u;)) = TT.)) lower bound for diffusion-resulter terms the zeroth order approximation
This is an alternative construction of
1
(yn (ay W w (reu((())))
2
1
Ely,, f(rs (() 2210)) (Ieel((( x;))13]
2
1=1
1/by Lemma
B.2
1 2
ell U (0) (00)) [rell((W(0))T: xx:)l=
//by Lomma B3
1 0²
Σ Ely, f(x; W()) (loon
(relu((((0)) az))lzz
30

---

Published in Transactions on Machine Learning Research (11/2025)
2 00 c()) [
1
We replace the B constant by 1>in [ T<:)) ) in the previous proof of loose lower bound of Theorem 4.2 Similarly,
1
(T)
[10(0)
E
BT
+
4 (1 e BB) B
where Eie(a)
max ( Tva))
B 3
|Σ ||relu |[ell(((W1))] Tx.p r;)1122}
The limit of this
lower bound is
lim 1-000
B
2 0²
| (2)) 22
max relu (1o) (0) T xza) jEl
BT
A(1 BT)
A
2 (0)
EE
Example D.2 (On the downstream alignment of pretrained features (Theorem 4.2) ). Here we provide an example on how the pretrained feature space affects the linear probing lower bound in Theorem 4.2 the overparametrizee regime. Consider one data point x4. and two pretrained features 14,1, W4,2 with || | = ||w+,212 I, cos(4+, w+.2) ST.
1. If we get lucky such that w4,1 Bt, then the limit is
B A
24 02
2. If the W4,1 is not so good for the downstream task such that cos(#4, w+,1) C then the limit becomes
16 06.2
we can tell that when the pretrained features do not align well with the downstream task, the lower bound gets higher, i.c. worse performance.
Since 24
16 >
15
24
D.2 Approximate DP-FT convergence
Analysis of DP-FFT loss diffusion. In the following -oddr approximation of loss Langevin diffusion, denote the drift term by W-gradient as Ti, the drift term by v-gradient as T2, the diffusion term by Whesssan as T3, the diffusion term by u-hessian as T4.
((0) = VW C()) C(1)
V (0)
(77)
Ti
[Irelu((W10)) Tx:) xx) + Σ
0
y ((vi, fzz (0)) ((0))
) (rell'((w,))) x. 2
(78)
2
1=1
h
2
1/2 (- ((0), (00) rellu(w:))))
(79)
2
m
Σ
yi exp yi (x;; W(O), oooii)e (1o T2
(80)
1
oe (vi, (2x ((0) (,00),,()))) [[ee)(([W(0)) r;)111 + Σ
(0)
2
m: 12::11
(81)
2
(m)
(mn
j=1
h
2
i/s exp( y: -a:ppp-flllee (ax; (()), v0 )elu((()))
(82)
31

---

Published in Transactions on Machine Learning Research (11/2025)
Us W(,, (o ( 22
(83)
1
vi Eyy, f(e;; W( ) Irel((((111)) xxx)
(84)
2
1
vt lyvi f(r W(o) )())
> (0000))21 ) 12:11 11
(85)
2
23
Upper bound proof of Theorem 4.3 1. Upper bounds for Tı, T3. For Ti, the key idea || 2 2) for any unit vector
2
T
Ys (0000(0(( W( (0 )u, 1011 )T,
1=1
//since Vx E R.D 2 ESD || 2 2x)
2
Yi exp( 1 f
w v ) (0) ) TT,0
j=l
h
2
Σ
y/ exp( - yi (x;; W( )0; (0) 1 (1000) T2,00 (0 2)
j=1
(0)
Ut exp( (0:00: W(O) (.()) (()))) (w: a;20 (1 2)
T,,20
2/
//pick
by Corollary B.5
|01112"
h
--
(u)
exp( - Vi (a,; (()), ".(0))) (wn.0)yT T2
J=1
Y=1
2 h
00)3
exp(-y; w ((00)))
j=1
(
2
h
2
(1
10)
Σ
Elyi, f(r; W(O) ,,0))
(1
For Ts, we align its form with Ti.
T3 Ey f (Iz W(0), w(00)
>
(1 22
(m (1))))) 110 11
j=1
/ //since Vi [n]] 1u: =1
1
Mm, f W(1) (00) Σ (1 (0) (w)0)) |o
2
1
(0) 2 Σ
| 221 (ي
( (yr, f(a;; (1) (0))
2
32

---

Published in Transactions on Machine Learning Research (11/2025)
1
0
max
< 1
h Σ j=1
(0)
)2) ya
n
1 (wg ( To (Vn. f(z;; w( v((0)) v(())
2
/E|n|
1=1
h Σ
1
max
|
11
(0)
)2
Σ
f ((0))
2
iE|n]
j=1
(E
2. Upper bounds of T2, T4. For T2, use linear separability.
h T3
2 Ui exp(-1//f W(o), ( (elu(())) (0)) xi))
J=1
//by Corollary
B.5
2
Σ
exp( wepp- yef (ax; W(O), 2(()))) (1o)0))) 1 ?lll 10/ l12
J=1
iE|m]
h 7²
(0) 112
Σ
(xp(-yi((a;) W ((ooon
2
|
10
Σ
(yr, W( ((
()))
For T4, we align its form with T3.
y Elye, f(x;;; (()) (00) (W)) Tx:)
0
2
//since Vi E [n]] 1ual =1
Uy, fla;; w v ( (1 2x,)1
2
n
Σ
elyi [a, W (())))
Σ
1 (w ( (
(10 (0) 2
JE(N
2
Eyis (x1; (()) ))))
Σ
1 (w (o) 2 Mu (0) 12112:112
i=1
JE(N]
000
max /E|ml
12
00)
12
( )
(Wr. f(r:; W()), ((0)))
<E(M)
1 _
max ||a:112 :EEm)
||
|
1011
112
Σ
EVv W(0), (0))))))))()))
j=1
(w))))
3. Combine upper bounds of Tı, T2, T3, T4.
( =T, + T2 + T3 T4
h
S -->2 Σ (1100) + 100 (0) [2
Σ
e(yr, farz W(0) b(00)))
(w)))
)
33

---

Published in Transactions on Machine Learning Research (11/2025)
h (0) 02 + 1
+ 2
0²
max | selm|
Σ
Elyi, f(r;; W( ))
//abbr. == flyi: (1;;; W ))
72
(1)0) + 1100f |
Σ ls
(E (0))
1 00 +
| | Σ (wgg و -² + 11 10
Σ li (1eg)))
max iE(m]
( )}
=
(u) 1 + 110e (0) 12
-72 Σ
ls
+ 2 20²
max ||1:13 vE/n
Σ 4
Σ
(6)
(
(E
(0)00) + |/03
10)
12 N 0² + |1 1
:. Whee the drift term (negative) still dominates the dynamics, we lake L = 0 for (u)) (1 + 11100 || 113.
( < (1))=00) )2 + 11 1002
Σ
max iE|m)
| 12
Σ le
2
(w)))
(wa
4. Decompose loss by trapping. If the trapping condition holds, can decompose the loss (()) c(0) + _00), where _. is only controlled by w, if w/y" ES, (+
2
200) S Σ
(w 12 + w الممي
-22
Σ
es
+ t0²²
1 mmx iE(n!
Σ li
jE(N],uef0) 10) Es.
(1)))))))))
(
Σ
(0) )2 + w اامي
{ -,² ? (c.(0)
+ 2
0²
max sEln]
|2zil2 c.
jE(N),u, (0) Es.
Let u == 1 = Σ EEEM, ( ES. [105000] + 00001=001 B = -2,200 -=
= -0² (maxie(m) | 122) Then
du
dt < AB + ACu
d (cr)
ABexp(ACt)
S dl
B
C (oxp(ACL) - Suc ACt ug
B
C (exp(ACt) - 1) + 40 Sue ACE
C (1 - cxp-xx(AACt)) + uoee-eec Act <u
c
<
B e AC1)
+ 1 c- ACE
C
2,2)0
The time limit of the upper bound is
100
lim (0)
C
B
0²
max iE|n
|
12
2 min
max
||; 2 1
|0:
2 1
34

---

Published in Transactions on Machine Learning Research (11/2025)
5. Combine clustered losses.
( _ + 4
1
1
(1 - (1-E- A-) + A400 - E11-E-A-C1)
A_
E
i-
---0,-
Lower bound (type 1) proof of Theorem 4.3 1. Upper bounds for Ti, T3. For Ti, the key idea is | (z, z)² for any unit vector
T
Yi (xxp(>ii W ))0 "
/ //since Vo E ERD, = E -1x113 2 (ar, 2)²
2
S Σ
Vi exp( (z:; w v (0))) 10) ) =
j=1
h
2
Σ
y/ exp( (x,; w v )o (w0 (1000) (1 2)
f=1
h
Σ
(0)
Σ Yi expp--9/(( (v;;; W(O), ( (1n00))) (
(2res 2)
j-1
//pick x
11
by Corollary B.5
|011122
h -- Σ
(n 10
Σ expl - (xx W() (()),,(0) (10)
1=1
h
=- 72 Σ
10)
exp(-u:((x;) Wl0) ((0o))
3=1
2 Elyi, f(a; (()) 2((0))
10)2
(w))))
For T3, we align its form with Tí.
T3
1
o fly f(r;; (00)
> 10)0)22 (m)) (m S T 1
2
//since Vie [n]. lyi =1
h ( f(a (()) (0)))
1
(1 (0) 00) (n. (T 11
==0²
2
1
(0)
|
2 (mn
( (1t, W(0), o(0)))
2
1=1
000
max [E|n]
10
Σ
10
10) 2
1 (w
( (yr f(a;; w(0) (0)))
i=1
35

---

Published in Transactions on Machine Learning Research (11/2025)
h
1 =²
max |E|n|
1
Σ (0) )²
Σ
My, fax; ((;; .o(0))
2
2. Upper bounds of T2, T4. For T2, we use linear separability.
A T3 =
2 Σ y exp(( (x):)000) rrel((()))) zi))
Σ
W1
5=1
//by Corollary
B.5
h
exp( - fa W(O), ,000) (0)))) ( 0 llll l12
[E(m)
2
2
(0)
112
Σ
exp(-yef(as)) W ((ioon
(w))
2
100
(0)
11
Σ
fly; W(O) ((00na
(w))))
For T4, we align its form with T3.
T4 2 v (ve, f(x;; W(O), "/" x1
//since Vi E [m]: wil=1
2
Ey,, fx: ( O) xx:)112
2
L(y, [ay , ()
Σ
1 (w ( ( (w) ( xx,
JE(N)
Clss, (111; ((0))
Σ
1 (w (0)) -
lu (0) 1111::112
jE(N)
2200
1 2
max iE(m)
||2:1
Σ
||
(0) 11
Σ
1 )2 f(z;; w(0) )(0)))
j=1
<E(m)
00
1
max ||~:112 iE|u
(0) 12
Σ
f W(0), ((0)))
j=1
(w))(
3. Combine upper bounds of Tı, T2, T3, T4.
(00)- =T, + T2 + T4
2 ely,, flr; w/ ((00))
< 2 Σ j=1
+ 1100 00) [2]
(w))( (0))
h |
+ 2
02
max iE|n]
Σ (2 2 + || 10) 1
Σ
E(Vi, f(;; W(0) )))
36

---

Published in Transactions on Machine Learning Research (11/2025)
//abbr. l, := llys: fa;; W )(0)))
h = =11 7²
(1) + 1lare (0) [
Σ &s
j=1
(E )
+ 02
max 12:
Σ (0)
)2 + |11| 111u0
Σ l
sc|m|
(w
}
h
= Σ
|2 + |1 1 1 12
Σ li
+ 00²0
max l1::1:] iE(n)
Σ ls
j=1
::
((og0) + ||
11) 12 N (0 (3 + || (0)
:. Whee the drift term (negative) still dominates the dynamics, we lake L = 0 for (1) + 10
2
}
( S [1//0/000 + 11 ooll
2
Σ
li
+
2
max iE(m)
Σ le (E
2
(1
4. Decompose loss by trapping. If the trapping condition holds, we can decomposc the loss L()) (0) _.)) where _ is only controlled by wy if w/) E ES. (+ (+,-)))
2
((0) Σ
(0)0000) 2 + 1/ 1, 12
(0)
Σ
ls
+ 20²
1
mmx [|1:113] iE/a
Σ Co (E
ue 100 Es.
(w
Σ
(1 || w (0)
2 (!0))
+ 2
0²
max (E(n)
|2::18] c. }
jE(N), J es.
Let u = 1/c_ A A= Ese(, ES. [009000] + |003]=
(0)
-C= =0²² (maxietm) 12:118). Then ,B -2,
du
dt S - AB + ACru
ABexp(ACt) < d dt
(or)
C 8 (oxp(ACI)) - -1) SucACe usg
B
B
C (exp(AAC)) 11) + u0 Aue
E exp(- + uge ACt Su
B
( < B (1 e-ct + 1 ACt
The time limit of the upper bound is
lim
(0)
<
C
0²
max [E(n]
|2
1 =
max
|0: 2 0
||::13 1
B
2.
2 min
37

---

Published in Transactions on Machine Learning Research (11/2025)
5. Combine clustered losses.
( _ + (
+ (1 ((1-e-A-Ch) e-A-CC)= -- A.C.
(1 c-A- + 2
-
2,20-
Lower bound (type III) proof of Theorem 4.3 (maxke(m) || (2)
Lower bounds for T₁, T3.
For Ti we use
T1 =
Vi exp( (x;; W(O), v 1)) (0) (w) (
//abbr. l == exp( - yf fx;; w 2(0)))
h
Σ
Σ
y
(0) Ti
j=1
|E (m )
h Σ
Σ
(0)
iE (E (0))
2
=-
Σ En
Σ
E
JEIM
(E))(
12
Σ
(0
Σ
es
|0ill2
jE(H]
(E
max
| 12
Σ
Σ Pi
kE[n]
jElA]
(ES
For T3, we align its form with Ti.
T3
y Cy,, (x;;; W()) (0))))
00
20
1
2
(0) (0)
2
1=1
j=1
1
01)3
Σ
21110:113
2
jE(m]
(w))))
2
2002
0²
1
min | 11
Σ
(1)
Σ le
kE(m)
jE(A]
2. Lower bounds for Ty, TT. For T2, we use (a. & ||21||||ll
h
2
Tb
Σ
Σ yi
(0)
(ell((()))))
38

---

Published in Transactions on Machine Learning Research (11/2025)
2
Σ
Ye expl eeep((-if(aa Unf w (w0) )
(w))))
2
Σ
Σ
Ci(wn] (0) ai)
JE[
(
2
Σ
Eillees 10 12210al12
JE[A]
(w
max ||,28112 kE(N)
Σ
(1) 12
Σ li
SE[N]
(w
For Ta, we align its form with T2.
Ta = 0²
1
e(ye, fa; W(O), v (. (WV() xx;)11
2
//since Vi E [n], lyi =1
1
Eii,, faa; W()), rf ( xx))
2
Elyi, f(x; W(o), (1))))
Σ
1 ((mf0) o0)
(0) xx))²
0"
jE(h]
1
4/w,0,,,) xx1)
2
JE[N]
(w))) 1
1/by Lemma B.4
Σ
Σ
1 (0) 1131111013
0²
2
JE(N] (wo)))
1
2
Σ
|/m 1012
Σ
JE
(()))
1 2 n2 2 200º
min || 1
11 10
Σ le
kE(a)
jE(A)
((
3. Combine lower bounds of Ti, T2, T3, Tr.
(0)) =Ti+ + To + 73
2
2 - max
||ax|122
Σ
(1o00)02 + |/ 11 1/
Σ
li
(E(())
JE(A]
11 10))
+ +2
1 0²
min [k(m)
Σ
(0g 00) 22 + 1 100
(0)
12
Σ
es
jE(A)
(E
//by balancedness, ||.,,00 (u,5')13 1/ 0
39

---

Published in Transactions on Machine Learning Research (11/2025)
σ (1 + n²)
2 max
2
Σ jE[N]
(0)
Σ (
min kE|n
| 112
Σ jEIN
11
11
Σ e
kE/n
iE
4. Decompose loss by trapping. If the trapping condition holds, can decompose the loss L()) ( _ where _.0) is only controlled by w w// ES, (+,
Iee 10) 12 + 0² + x²)
(( 2
max | kE
Σ
min KE(n
|
Σ
Iuwy
1 12
2
uel es.
_ ES.
(0)
11
{
| (c.0)²= + (1 + n²) kE(n]
max
min |1ax|122 c! kE(n)
2
JE es.
The time limit of the loss lower bound is
min |
1 + 1²²
lim c.
max
ke[n] | 12
By the previous lower bound proof,
I 1
1| (0)
1 1 (maxxe)))) (2:1)))))
Let u _. A " 12 [( Then consider integrating factor exp(AC/Mo exp( (22)))
||= [0
(0) B 2 maxxe(1] | C a²(1+a²)
[ |
d
Cu) d ABe AAC dl
dt u
d
ABe exx(AC/22 2AC aawp:///AAAaaaaaaaaooaaaaaa////a
d http:///AAaaapo/la22ccaccaa/aaa/
d
dt (exp(AC/))
(u exp exp()2t)))
dt
B
? lexp(ACC/aaxp((An - exp(AC//))) -xp(A)/))) 2u exp(AC/ (xpp)l(/eeexxxxaaxppaaa exp(xx))))))x(x exp()2()) - uo exp(AC/>2)
B
j ell-ss/Ac// Zu uo exp(AC) exp(A2t)))
cC/As
B E [| eAC/sa(1-esp((el)
5. Combine clustered losses.
C (0)
1
+ (C/AA((-oxp((x))) + B - (11cC(((1-oopp()t))) exp()at))]
AC/Aa exp((A)) + B -1 (C-cA//s((-ooool
2
2 -_1=0
D.3 Privacy budget allocation
Proof of Theorem 5.1 For any j [h], with probability 1 1 p, its initial absolute value is bounded by
l0yl 22 (n
(86)
40

---

Published in Transactions on Machine Learning Research (11/2025)
Then with probability (1 - p)", the maximum worse initial value is bounded by
max (c: Ug)) S v ln(pp)
(87)
jE(h]
where we define oj by wy E Sey- The approximate DP-LP dynamics
i = Σ Nelluul xz)
(88)
Say wy E Se for some E {-1,1}, then during DP-LP, when sign(e;(T)) = (O))
(T)(T) - (o))) L/
Σ [rclu(w] ax)dt
(89)
min (rel(([] x:)1 / Lo(l)dl wec
(90)
/by Theorem 4.2
(91)
1 { Σ = (rell(WT 22) mz)l13 -2
minrelu(ug [1] xz)
(92)
W=
E Emjese [maxy, w. Iz]
y= relu(w) 244)
Σ Mi=c
||relu( (Tx)lll
(93)
E Ewyes. [maxy w] xi)²
2 Q²Q
(94)
where we define a constant Q to describe the pre-training quality. If the pre-trained features are better, Q becomes larger. To mitigate the feature distortion, we need u- > O, then the necessary DP-LP run-time is
B (2/p)
(n(2/p))
(95)
where we ignore B as it is typically pre-determined in real implementations (e.g. the Linear layers in PyTorch).
E Appendix: Theory without approximation
For convenience, we use different notations for the data input dimension d = dg and the backbone weight matrix = W in the following proofs.
E.1 Itô's formula and its consequences
We denote Mm, (R) as the space of m-by-n real matricos.
Theorem E.1 (Itô's formula). Let X, be -aluud Itō process satisfying the stochastic differential equation 0x, = X1) A2(1, Xz)aW, with Al,1,,,) being IR -oluued A2(1, X,) being (R)-(RR)auuedd,, and W, being a standard re-dimensional brownian motion. Let f [0, oc) x 1²" + RR be a function with continuous partial derivatives. Then Y, = Xt, is also an Itō process, and its stochastic differential equation is
(,((, 0t + (VS(1, X,), Ai X.)au + A2(1, X.)OW,) + (A2(1, x.)aw,, A2(1, X.)OW,) (96) 0Y, = Øt
8 and (Xt); denotes the i-th where Hy is the Hessian matrir of f over X, defined as (H o(x)):((.)); entry of random vector X.-
41

---

Published in Transactions on Machine Learning Research (11/2025)
Corollary E.2 (Loss dynamies during linear probing). During linear probing (Equation equation 121 the
stochastic differential equation describing the loss dynamies is
1 - -BB v -- B3 Bo - + (202(Bl
XTy) ow, + 01.
(97)
Proof of Covollary
E.2 By Itō's formula (Equation equation E.1 the loss dynamics is
1 =0 2
X [B - Y1?
(98)
=(XB/% V- xB de (0u)" Boxx* x (0v)
(99)
=(XB)) Y) x) 0v 2
(0v) (0o)
(100)
//by Definition
E.5
(101)
=(B) v- -o -
- V2220W + ho
(102)
+ hoo
(103)
X --(B' v- x² - x))+
(B v -B BJ oW, + ho²01
(104)
Corollary E.3 (Loss dynamics during fine-tuning). During fine-tuning (Equation equation 122 the stochas- tic differential equation describing the loss dynamics is
8hn = - (B''v (3
(" vōt +B"T - (V"=X""""
(105)
+ + + 0² d1
where we use Ô as the differential sign and use d as the data inpul dimension.
Proof of Corollary E. fine- tue-tuning is
Similar to Corollary E.2 we use Itô's formula (Equation E.1
the loss dynamics of
thn = 2 JBB e- -112
(106)
Voll(BB' 8- -
V vooBB)
vcc(aB))
(107)
(dc) + (x) =(XB² -
1 (B0) (0 " (0x
(108)
du O - (
(109)
U1
+ (00) x + 1
(vec(
U1
vec(aB)
(110)
dxh
Uh
-- -BB
o XT - BT 22
(111)
- -BBT v x8"'
XTY v (
(112)
t (BB* 0t + 0² u att
(113)
= =-BB"v - B x - X"
(114)
(115)
+ 1
(116)
42

---

Published in Transactions on Machine Learning Research (11/2025)
Remark E.4 (Noise effects on linear networks). In the loss dynamics of fine-tuning (Corollary E.3 the noise induced deterministic terms
( + d1
does not explicitly depend on the linear head size We do a sanity check for this result a discretizod setting (so that skip Ito's lemma and stochastic calculus). Say inject noise AB to B, where AB is h x d-matrix, and its entries are independent and follow Gaussian distribution Then the expectation of the perturbed loss is:
[[C] BII AB)" U- 111
(117)
1 -
XBB 0 211 + ( v)
(118)
IXB Y112 + Blu AB(AB)
(119)
2
1 0² d ||112 1 111 + 2
(120)
As a result, we find that, in the discrete updates, the noise induced deterministic terms does not explicilly depend on the linear head size either. So our findings the continuous case matches the discrete case.
E.2 Modified Langevin diffusion
Definition E.5 (Langevin diffusion for linear probing). Let Qt be the standard h-dimensional Brownian motion. Then the Langevin diffusion for linear probing defined by the following stochastic differential equation:
dv V.L(v, Bo)at V2020Qi
= B,X" 55xx"(XBB -- Y)ðt + v2020Q.
(121)
Here we use "0" as the differential notation.
Definition E.6 (Langevin diffusion for fine-tuning). Let Q: be the standard f--dimensional brownian motion and Q be a matrix whose entrics are standard and independent brownian motions. Then we define the Langevin diffusion for fine-tuning a two-layer linear network as
dv V.L(v, B)at V20"0Q.
-BX (XB" -BX"(BBT 0- YJδt v202@Q,
(122)
OB NCC( B)8l+ + v202aQ'
= Xit 220
Here we introduce an assumption based on random initialization. It describes common phenomenon in differential privacy deployment: the loss might not converge if the privacy mechanism perturbs the gradients too much (Ponomareva et al. 2023). To ensure that DP-SGD works for full fine-tuning, we assume that the noise scale (or variance) in the privacy mechanism is upper bounded by a constant.
=|I* y [2 Then we assume that the noise scale a > O we add for privacy in the fine-tuning process is upper-bounded by
Assumption E.7 (Upper bounded noise scale). Let 8
(13 + | |
h3
hh3h3 + || (1+ + - #}}
min
(123)
2h
V2 + d) 1+ 22
Equation upper monotonically decreases in time if Assumption E.7 also holds.
To understand the properties of a dynamics analysis problem, it can be useful to identify invariants, functions whose output is conserved during optimization. Such conservation laws be seen "weaker"
43

---

Published in Transactions on Machine Learning Research (11/2025)
form of implicit blas, helping to clucidate which properties (e.g., sparsity, low-rank) preferred by the optimization dynamics among a potentially infinite set of minimizers (Marcotte et al. 2023). To prove the convergence of our optimization, we study the imbalance matriz, an invariant for multi-laye linear networks that has previously been studied in the context of gradient flows (but not Langevin dynamics, to the best of our knowledge).
Definition E.8 (Imbalance matrix). For a two-layer linear network, we define the imbalance matrix as
D := === BB".
(124)
Prior work on gradient flows has found that the imbalance matrix remains invariant over the evolution of gradient flows modeling gradient descent (Arora et al. 2018 Du et al., property can used to derive tight convergence bounds (Min et al. analysis has not materialized for Langevin diffusion models of DP-GD.
2018 Marcotte et al. 2023). This
2021 2023a).
However, a similar
We observe that prior work Langevin diffusion to analyze private optimization has implicitly assumed that the sensitivity of each layer in a neural network is the same (Ganesh et al. 2023b Ye et al.) 2023b). Hence, they fix a uniform noise scale for every parameter of the nelwork. Under these conditions, we show that, when we ignore the sensitivity of each layer and use uniform noise scale 0, the imbalance matrix is not invariant in expectation, unlike in (noisc-free) gradient flow (Arora et al. 2018 Du et al. 2018 Marcotte et al. 2023); that is, its derivative over time is nonzero. This complicates the use the imbalance matrix for theoretical analysis (Ye & Du 2021
Lemma E.9 (Imbalance matrix in tuningg.. During fine-tuning (Equation imbalance matrix Definition E.8
122 ) the derivative of the
E[D] (1 Inxho
(125)
01
where d is the dimension of data inputs (B R.
Our main observation is that by modeling differences in sensitivity different layers, we can recover the invariance property of the imbalance matrix. The following proposition characterizes the sensitivity of the lincar head and the feature extractor, and illustrates why they have differing scnsitivitics at initialization.
Proposition E.10. We assume that the training dataset D = (X,Y) is normalized such that X Laxd, |||||2 1. We initialize the linear head by U/ (0,0, (Inxk) and 3 h/vd. At the initialization of full fine-tuning, the linear head has greater layer sensitivity (Béthune al. 2024) than the feature extractor B:
A(V.L(vo. Bo)) θ (Vd (VVef(voBo)))
(126)
Based on this observation, we propose a modified version of Langevin diffusion for full fine-tuning, which accounts for layer-wise sensitivity. With this modified definition, the imbalance matrix is again invariant expectation.
Definition E. (Modified Langevin diffusion for fine-tuning). Let Q, the standard h-dimensional brow- nian motion. Let Q/ be a h d matrix whose entries are standard and independent brownian motions. Then define the modified Langevin diffusion for fine-tuning two-layer linear network as
av = V.L(v, B)at doo2d0Q.
XXTX(BU--XTX 2d0
(127)
aB VnL(v, V2000''
= - v(XB" wー Y)T x0l 222
The only difference between this diffusion and Equation 122 the additional factor of Vd, shown in red, reflecting the fact that the linear head has greater function sensitivity than the feature extractor.
44

---

Published in Transactions on Machine Learning Research (11/2025)
60
0.10
Relative eгror
0.05
Sensitivity
40
Layer Layer 2
0.00
20
-0.05
0
-0.10-
20
40
60
80
100
0
25
50
75
100
Epoch
Epoch
(a) Sensitivity
(b) Drift of sensitivity ratio
Figure 9: Exaluation of layer-wise sensitivity when rumning DP-GD on 2-layer linear networks and synthetic
data (Béthune et al., (024)). We initialize the network parameter according to Proposition E. 10 average on 10 random seeds with standard error smaller than 10
We take
E.3 Linear probing loss upper bound
The main idea of the proofs for convergence is to replace gradient terms with loss terms. By doing so, we obtain inequalities containing only loss terms and some other constants.
For the linear probing setting, we first show the strong convexity of the loss function. Then we can use the Lojasiewicz inequality to replace gradient terms with the loss terms.
Lemma E.12 ((Strong) convexity of linear probing phase). The empirical risk = ( v.) is 1-strongly convex.
Lemma E.13 (Initial loss before linear probing). we initialize the linear head by Ut-0 NO, 3lxxh) then the expected empirical risk before linear probing is
B|Co|
(18 + 2|
(128)
Proof of Lemma E. 15 pected initial loss is:
We initialize the linear head with Gaussian distribution N BIxx). So the ex-
Co
E
B| to - -11
(129)
2
x Elug Bo X vo + YTy x vol
(130)
Elug By + YT]
(131)
2
1/we assumed in section 3.1 that Bo has orthogonal rows
(132)
Elug to + YTy]
(133)
2
//by Vt=0 No Inxh)
(134)
2 (h. + ||||13)
(135)
45

---

Published in Transactions on Machine Learning Research (11/2025)
Theorem E.14 (Expected loss upper bound of linear probing). The expected empirical risk in linear probing is upper bounded by
[[Cip(()] - [B[Co + (1 ">> ha²)
(136)
Proof of Theorem 4-4 By Lemma E.12 C is 1-strongly convex, we have the Lojasiewicz inequality, Here we abuse the notation and consider it as a function of the linear head v because we fix Bo in the linear probing process.
((v) - {min L) S (..(v))11
(137)
2
For simplicity, we denote E[C]- == 2. Consider the Langevin diffusion in Equation equation ((v) - (min, L) ho? > 0, by Corollary E.2
121 when
ac(v) -V.c((), - VV + + ho²0t
(138)
ac(v) IV. + (V.[(v), 2222 + ho
(139)
/ // Lojasiewicz inequality
(140)
@C(v) ( ((v) + (min L)) + (V.((u), 2o, + ho
(141)
a(E[C(v) [min C)- - h - (B([(v)) {min 2)) + ho
(142)
a(c {min C} ho S (c - {min [}} ho²)@at
(143)
//When 2 - {min {} ho > 0
(144)
0ln 12 - {min L}- - ho < 11l
(145)
In í2 - {min C} - ho | < In (C(v)) - {min L} - h1o2
(146)
L - (min C] - hoz se ((((m) (min L} h
(147)
L Se - ((((v) {min L}} ho ) + {min L} + ho?
(148)
L 5e- (((o) + (1 <1 () L} + ho²)
(149)
5 *C(oo + ") + ho²)
(150)
When we substitute the initial loss L(vo) with the hyper-parameters use in the random initialization, we obtain the following corollary.
Corollary E.15 (Expected loss upper bound of linear probing from random initialization). If we initialize the linear head by 0 ~ NO, Inxa); then the expected loss is upper bounded by
(t) < (h3 (| + +(1 -1- e + ho?)
(151)
Proof of Corollary E.15 The result is immediate when combine Lemma E.13 and Theorem 4.4
j
E.4 Imbalance matrix from linear probing
In the convergence analysis of fine-tuning, we climinate variables and simplify the Langevin dynamics by the imbalance matrix. this part, we characterize how the imbalance matrix changes the linear probing phase. The following results will later help analyze I.P.FT.
46

---

Published in Transactions on Machine Learning Research (11/2025)
Lemma E.16 (Eigenvalues of imbalance matrix at the beginning of fine-tuning). During the linear probing
phase (Equation equation 121), for the imbalance matrir defined in Definition E.8
the minimum eigenvalue of the imbolance matria is always 1;
2. other eigenwalues evolue in this way:
E E E |11112
-1
(152)
Proof of Lemma E. Consider any eigenpair (A, u) of matrix D, we have
Du =uu
(153)
(voJ BB )u =Au
(154)
(vv" Inxhuu )u =\u
(155)
(w u)u =(A + 1)us
(156)
(157)
We can take any I. v and (u, -1) an eigenpair of D. So is always an eigenvalue of D. We need discuss two different cases here:
1. X = -1, we only know that u 1 V.
2. If 11, then and u are parallel. Say u av, then
(158)
ar ||10112
02
(159)
X +1 = w113 2 -1
(160)
Proposition E.17 (Expected eigenvalue of imbalance matrix the beginning of fine-tuning). Say we run
linear probing for time t.. If we initialize the linear head by Ut=o (O, Taxt), then for the imbalance matrix defined in Definition E.8 we have
B|[|u||] h3e- 21 X1 + (1 ho²)(1 21
(161)
throughout the linear probing process. Then by Lemma E.16 for those eigenvalues not equal -1, we have
= - hBe 2t 21 2) (I ho -21)
(162)
at the beginning of fine-tuning after linear probing.
Proof of Proposition E.17 By Equation equation 121 the Langevin diffusion of lincar probing is:
dv - B,xx B-xx(xXBB Y)αt + 20 aw, vvt + B,x v20
(163)
We consider the evolution of uTu: by ltt's formula (Equation equation E.1
I du' v =2 de + (I (Qv)
(164)
do 2v (v B, at + 20 V202aW, 2ho20t
(165)
du v =( = T B Y) + 20 20W + 2tt
(166)
47

---

Published in Transactions on Machine Learning Research (11/2025)
To solve the above equation, we need to solve the dynamics of o² BaXTy:
0Y X Y XB (c B + V2020W,
(167)
0 X Byg v| E|V v wldt + | YY
(168)
0
E|Y XB.
1b r1 E| x 1B
(169)
0t
0
In X
| y|]]]
(170)
EY X 01 Bo " Y 1 =|E| XB 80 | exp(-t)
(171)
When we initialize the linear head by -==0 (0, /xx)), we have B X BB vol 0. Then
X |B
= X | Y |||| exp(-t)
(172)
E|| B, Y - _
= B y X uol exp(-t)
(173)
So we can rewrite Equation equation
166
as:
u| [|
2t [|
+
B Y|) /
(174)
0E|| u ( 10-21|11111 + =
y X -xp--- + |1, + y 2ho
(175)
0
[ |||0||] = = x
yT
0o) exp(-t) + BB X yl12 ho²)
(176)
2 0t
Let 01
E|] Bo
- u0] = |1:00220BBBBXX10. 12 ho? f(t) |||| and rewrite the above equation:
1
2 f(1) f(t) f(t) =aje + 02
(177)
f()) + 2f(() -201 +t + 202
(178)
2h f'((()) + f(() f(()
=2a1 c' + 20zc
(179)
" f(t) =(2a1e' + azze2))
(180)
" f(t) =f( 2a 2e' 1) + az(c
(181)
f(t) (()e 2t 2a c 21 + az²1 6
(182)
Since we initialize the linear head by 0==0 ~ NO, Inxh); we have ʃ(0) hB and 01 = =1
Lemma E.18 (Imbalance matrix in fine-tuning). During fine-tuning (Equation equation 122 the imbalance malric D in Desinition E.8 evolves as
i Øt
E|D] =(1-do=* dd
Inxh
(183)
where d is the dimension of data inputs (B Raxd).
Proof of Lemma E.9 We prove this lemma by analyzing the infinitesimal generator A of imbalance matrix
at any time:
()(((())) - (D),
A(D); = lim
(184)
440
=0 + o Σ Σ
-i==j'ii-
(185)
KE P'
2
Σ
Σ
11'= 1" =i= and j'= = j"=
(186)
"EM
48

---

Published in Transactions on Machine Learning Research (11/2025)
the generator is zero for + j. So we just consider the case where i = j.
A(D); =o? E
EE'=
(187)
YEI &'
E|
0²
Σ
Σ
1f' 1" = andd j =
(188)
N' ''Eld ("
= =(1 d)o²
(189)
Lemma E.19 (Monotonic eigenvalue of imbalance matrix in fine-tuning). Denote Dip as the imbalance matrix right after linear probing phase. All eigenvalues of the imbalance matrix are decreasing in expectation during fininnnning. Specifically,
B[{(D)) = [[((((,)] + d)o²t
(190)
where the time-span of fine-tuning process.
Proof of Lemma
E. 19
Pick any eigenpair (A, u) of imbalance matrix D (Definition E.8 such that ||u|/2 1.
By Itô's Jemma
Equation equation
E.1
ax =aD)u + (OD)(M - (DDY'
(191)
=(1 1u + aM, + (1- dd " (AI-D) =(1 =|1-d)o²00 0M, + +(1 (1-d)'o'u (NI -D
(192)
(193)
where M, the martingale induced by the Brownian noise and (.) denotes the pscudo inverse of certain matrix. Say the the singular value decomposition (SVD) of D is
A
0
D = UEUT =
X2
UT
(194)
0
where we have À E diag& and u being a column vector in U. So we can write the SVD (AT D) as:
A1
AI - M-D= M-D==VEV'V
h A2
VT
(195)
where obtain V by removing in the columns of U and we obtain > by removing in E. Then the pseudo inverse of (AI AV-10 D) is
A-
(AT - D) -VE"
Vr
(196)
Since is orthogonal, we shall have yT 0. Then we can rewrite the stochastic dynamics of D as:
E[A] = (1 - d
(197)
Ot
49

---

Published in Transactions on Machine Learning Research (11/2025)
E.5
Fine-tuning loss
Lemma E.20 (Bounding the norm of linear head (1012)) During sine-luning (Equalion equation 122 can bound the norm of || with the imbolance matriz D in Definition E.8 as
1² + w 1
X
12
w|
1011
(198)
2
where we denote A = Amia (D), A Amax
Proof of Lemma E.20 Given the information of imbalance matrix, we can bound the linear head norm.
Denote A = Amin (D), X = (D). Denote w = BB'' and multiply D with on both sides:
o" De =(u" u)² ()"")(B"')
(199)
T Do =111111 ||012
(200)
We have range for the Rayleigh quotient:
x DD E [A, A]. So we obtain two inequalities:
xxx
||0112
||
(201)
||e||
||0
||a|| A/la11
||||| 2 0
(202)
||a|| x||a||
|||||
To get a lower bound of v, we can solve two quadratic inequalities. For the first quadratic equation, since the smaller root non-positive, A
1² + 4||xr||2 < O, we just bound ||o| with the larger root:
A² 4||w|||
||0||
(203)
2
similarly, for the second quadratic equation, we obtain an upper bound for ||aa|2 with the right-side zero point:
Ā X2 w||
||0|
(204)
Lemma E.21 (Bounding eigenvalues of Bd B (re-stated from Min et al. (2023b))) )). During fine-luning
(Equation equation 122), we can bound any nonzero eigenvalue X of B" B as
X2 4( wF) -X+
A(2Tww)
A
(205)
2
where we use the imbalance matrix D in Definition
E.8 and demote
Xmax (D) Amin
(206)
Proof of Lemma E.21 The proof of this lemma follows the proof of Lemma in Min et al. 2023b). Bd B is symmetric and positive semidefinite (xT BT Br = |B 2 2 0). So every eigenvalue of D" is non-negative.
D has most one positive eigenvalue: if D has more than one eigenvalues, then the subspace R spanned by the all positive eigenvectors has dimension at least. 2, which must have non-trivial intersection with
50

---

Published in Transactions on Machine Learning Research (11/2025)
ker(v") dim(ker(v') hh11 Then there exists nonzero vector ker(v") such that 2D which would imply BB= =² > 0, contradiction.
For any eigenvalle--iggevector pair (A,, zi) of B B where X; & O and 2, E gd-1,
X = (B" (B*B))² B)
(207)
/ //replace something with imbalance matrix
(208)
X = (2) w)² - 2f B" DBz,
(209)
17 (zf w) 2t B² DB
(210)
X - wx) E(2, (B" B)z;) ( [-Amax- miin
(211)
X (=Tw)² Edi 1-Amax- Aminn]
(212)
again, we can rewrite this two quadratic inequalities
x + Amax Ai - (2/w)) 2
(213)
x2 + Amin - (2f w)² 00
from them know that there are two possible intervals:
A Amax
mmax + 4a w)
Amux + v A +4(2 w)
400
(214)
Amin
min +4(27 w)
Amio
main 4(= 2(="=]=
A E
Note that we must have A 2 since BT B is positive semidefinite. So we can rewrite the bounds:
A Amax + 22 4(2/w)) w)2 -Amin + D 4(2 w)2
(215)
since the function f(x) = -2 V is monotonically decrcasing, we have f(mmax) S f(Amiu), i.c. the lower bound is no greater than the upper bound, i.e. the above interval is always non-empty.
E.6 Numerical conjecture the eigenvalues
Conjecture E.22 (Small relative error induced by Jensen gap (Equation 247 ) We denote the minimum
eigenvalue of the imbalance matrix D as A. The relative error Elmax(0,- 1 F(A
increases slowly in time
and is smaller than 1% under reasonable number of training epochs. Here we provide an empirical example with huge noise scale (much greater than the common noise scale in real-world applications). We observe that the relative approximation error is insignificant even with huge noise scale.
E.7 Fine-tuning loss upper bound
Lemma E.23 (Imbalance matrix in fine-tuning under layerwise noise). During fine-tuning (Equation [12 the imbalance matria D in Desinition E.8
evolues as
dD
E
0
(216)
dl
Proof of Lemma E.23 We prove this lemma. by analyzing the infinitesimal generator A of imbalance matrix D:
DD Duj (Do)
A(Do(u. B)); := lim
(217)
51

---

Published in Transactions on Machine Learning Research (11/2025)
viriee ever
3062
30035
8008
36494
(a) 50
100
Figure 10: Growth of the relative error 3mxx(0,-)) in the experiment setting: (1) two-layer linear
X(max(0,-2)" /3/
network with linear head size lo and feature extractor of size d x 10; (2) we train the linear network with DP-SGD; (3) we repeat the experiment with large noise multipliers a = 50 and 100.
0²
Σ
Σ
=1==-==-- 5l
(218)
PEJM F'LM
0²
Σ
Σ
11'=-= and j' = =1
(219)
the generator is zero for x j. So we can just consider the case where i =
A(Do(v, B))a =02
Σ
Σ
1F==
(220)
d²
Σ
Σ
111"= = 1" = and -== = j="
(221)
VE[A],'
[
(d d)o²
(222)
=0
(223)
Theorem E.24 (Loss upper bound of fine-tuning). In Sine-buning under layerwise noise (Equalion equa-
tion 127 we have
E & A + 1 c(-AtVBo(2+d))1)
(224)
where L 2² h- (1++()
X
Proof of Theorem
4.5
We first simplify the loss dynamies:
OC =
XB v 2
(225)
1 - voc(BB)
V.IIXBT -Y +,
(226)
+ (00)" 11 (1x 1
H
(227)
=(XB² ; Y) XB² d0 (XB - y)
(228)
+ 2 (00)" DD" H
(229)
= -XX="=="""="'''""X
- BX" (XT + +XB YY 20
(230)
-(XB" 1 -
XX vot u XY
(231)
"oPtrace(BB" )0: + o2
(232)
=- (B"'-x²Y)² B(B B xT (B
(233)
52

---

Published in Transactions on Machine Learning Research (11/2025)
-BB u -B-
(B'v-''')") (
(234)
+ tracee B)ot d
(235)
By Lemma E.20 and Lemma
E.21 we have
OEC - El(w x² (B" B J vlaxa)( + B1 + ddlw
(236)
A +
4 1|2
Øt 110 - X2 - V X2 + 4 4(2 w)²
<
- 1/00 - x 12
at
(237)
2
2
-X + 1²
4 T
w)²
X v X2 + 4|| 2|1
+ E
O'd
Ot + o²d
o²d
Öt
(238)
2
1 o w X 1 (Amin + Amax +
El|
o'E(dl
[T min + r max 1t
(239)
2
2
where we define
Amin = A + V
1 41| | 2 max (0, 22)
Amax = - +
12 + 4(Ziin w)2 2 max (0, -2X)
(240)
T Tiia
= -
122 + 4(2 w)² & max (2|| w|| |||w|| - 22) = 2||w|| + 2 m -
T maa A
X2 + 4|| w| v max(2) 1|| 2| wm + 2X) = 2||w|| + 2 max(0, X)
Denote the probability measure of the state time as ve. Then by using Jensen's inequality, reverse Hölder's inequality, etc., we can bound the first term:
B|[[ w. (l..2(Amin + Amax)) )] = ||w w. wall(Amin + Amaxx
(241)
> (f ||0 w. w.l1a'dd dus -2 (f(Mmino + Ammax) =Amaa)
2
(242)
= = w.
11 w.11=
2 (Amin + 2
(243)
B|[[ w. E + Ammx) 1/222 )
(244)
according our empirical observation (Conjecture E.22
(245)
we ignore the Jensen gap for the second multiplier
(246)
2
Billw - w. [(X
(247)
By Lemma E.19
(248)
= B|ll w. |1 (- E[((Do)] + (d- - 1)021)
(249)
=2 -2(-E(((Do) + (d 1)o²t)- -
(250)
Then we rewrite the upper bound:
0E[4] < | Fl|| X Y + Amxx)) 3 2 [ min + r max]] al
(251)
0E[4] & - XEE[CjJ + (V2( 4 + (1+ + d)] YI dd)a
(252)
[E[4] (
20 (1+
0² (1
dxjat
(253)
E (
+1
(254)
where /_ = 22 x- (1+d)
(1+d)
53

---

Published in Transactions on Machine Learning Research (11/2025)
F Theory with Clipping
In this section, we present the first thcoretical investigation on Langevin diffusion with clipping. We believe that our contribution is significant for the Langevin diffusion and private optimization research community. We summarize our findings and contributions in the following list:
A new definition for Langevin diffusion with clipping (Definition F.1
Zeroth order approximation error for the clipped Langevin diffusion (Theorem F.3)
Privacy guarantee for the clipped Langevin diffusion (Theorem F.4
The exact "discrete vs. continuous" algebraic correspondence between the clipped Langevin diffusson and vanilla DP-SGD (Remark F.2
Feature distortion analysis for the clipped Langevin diffusion (Theorem F.5
The existence proof of unique strong solution for the clipped Langevin diffusion (Corollary
F.7
Definition F.1 (Clipped Langevin diffusion). Say we work on paramcter 0 E R minimize a group of loss functions {Civeen) The parameter evolve according to the following stochastic differential equation.
00
Σ (V 000. aE/n]
(255)
This equation is the clipped Langevin diffusion. & is vector containing independent 1-dimensional Brownian motion. The clipping function is defined by constant C' 0 and
clipc((((()) == min (1 VE,(0).
This definition allows us establish the first exact 'discrete vs. continuous" algebraic correspondence between clipped Langevin diffusion and vanilla DP-SGD, creating continuous analytical framework that closely mirrors real DP-SGD implementations.
Remark F.2 (Algebraic correspondence between the clipped Langevin diffusion and DP-SGD). The update rule of the vanilla DP-SGD with step-size 17 > can be written as (Abadi et al. 2016
0k+1 02
1 1
Σ iEB.
(clipc(V&,()) + tN C²1)
(256)
where is the batch size and B& is the batch of data points sampled at step k. We can rewrite the update rule by assuming full sampling, فH and ő = oC:
0kt1 = 02 ñj Σ (VL(() + +ON(0.
(257)
=E|n]
One can compare this update rule with the clipped Langevin diffusion (Equation 255
00
Σ
ciip((V(())))tt++++ obG.
(258)
EE|m]
It is easy to see the algebraic correspondence between the above two equations. We provide rigorous derivation of DP-SGD update by discritizing the clipped Langevin diffuusio with the method.
Suppose that we want solve the clipped Langevin diffusion on some interval of time |0, T). Then the Fuler- Mally-Maruyama approximation to the true solution 0 is the Markov chain 0 defined as follows:
54

---

Published in Transactions on Machine Learning Research (11/2025)
Partition the interval [0.T] into K equal subintervals of width î > 0:
T
To <
TK -Taan and ñj K
(259)
Let 0 0o at the initialization.
Iteratively compute 0% for 1 1 k K by
0 = 0
Σ *Eim]
(VL((O-))) + +
(260)
In this way, we rediscover the update rules for DP-SGD by discretizing the clipped Langevin diffusion.
We give an approximation error bound following Freidlin et al., 2012 Theorem 1.2, Chapter 2.1). Theorem F.3 (Zeroth order approximation error). For all > 0,8 0, we have
E 00. 0000 19] (0(2)) 3 +2nc2)²
(261)
Proof of Theorem
F.3
E[o| - 13 (0, 0000 ,0,0 - -00 + 220020 0i
(262)
0E - 1 <E[4nC||0 1 t + 2po 0t]
(263)
B1100 1 & (4nC- E[ 0 2po 00t
(264)
B1100 - 0. 0 ||2 < /'/ (4mC v VE110, 0.00 |2/| + 2po2
(265)
E B|10, - 0 |131 <2po²T + 4nC 1-" VEllOo, - 0000 12:0t
(266)
By Lemma F.10
have
B[[[0, -0.00 119 < (0(p)) +2nci²² z
(267)
Note that this approximation error significantly improves upon the O(expTT)) error found under standard regularity assumptions (Freidlin et al. 2012 Theorem 1.2, Chapter 2.1).
We present privacy guarantee for the clipped Langevin diffusion by deriving an upper bound on the KL divergence.
Theorem F.4 (KL Divergence Bound for Clipped Langevin Diffusion). Let 9, 6 have the same distribution Oo. 6. Or be the solution to Equation (255) given initial condition On and database D, O be the solution to Equation (255) given initial condition 0 and database D', such that D D'. Let O be the distribution of the trajectory Ore(o,T)- Then for any T 0:
KL p.z (..7))) < 2n²C² T
(268)
Proof of Theorem F.4 By Theorem B.1 & 3.1 of Ye et al. (2023a).
1 202
/ E Σ
clipo (VL((; -
clipe (V; D'))
dt
55

---

Published in Transactions on Machine Learning Research (11/2025)
S
1
J
{
An An²C2dl
20
2n C
6² T
We demonstrate that our main result feature distortion holds for clipped Langevin diffusion, reinforcing paper's key insight. Here, approximation technique is essential, the stochastic analysis of Langevin diffusion with nonlincar & nonconvex coefficients would be extremely challenging without it.
Theorem F.5 (Random initialization causes feature distortion). If Assumption 3.1 and Assumption 3.2 hold, and the linear head is randomly initialized by ~V(0, 3laxh), then with probability 1 - 2 2-h vB
0, 3j E [4], At > 0 such that during the time interval (0, At), DP-FFT distorts wy reducing its alignment with the data cluster. The cosine similarity between Wy and the data cluster mean Belj) decreases monotonically:
1 cos (w), Ro(oil <0, E (0,
(269)
Proof of Theorem
F.5 The per-sample gradient for the i-th data point (before clipping)
Je4relu(WTx) yelvirell (w) (wTxy) 11)x4
relu(W xx:)
Virelu' (w 22, )x;
V (o,w))+
Voli veol V (w
1el,ogrelu (wg 23)or
m
egrelu' (wg 24 22)0
(270)
yl (w) 24)x,
uyrell' (w xx))
where the vec(-) operator is defined as an operation that converts tensor to a vector (Magnus Neudecker 1999. Chapter 2.4). We use vec(-) to collect the gradients of and W into vector. Then can write the clipped per-sample gradient for the i-th data point as:
relu(WT Tara) Uyrelu' (w a;) U2relu' (w) on ax)
(V(,wv))) min ("
C
yili
(271)
vgrelu' (W.. x,)2,
Therefore, the dynamics the parameter my for any j E [b] under gradient clipping is,
Ow, = min (1 ysl, - v,relu' (w) x;)x;
C
(272)
àt
Note that the clipping operation only multiplies the gradient with a normalization term min (1, IV we As result, does not change the signs of the gradient entries. Then are ready to analyze the cosine similarity between wy and the mean data direction:
2(w} FeGi)
Owl T wyyl 1 12 ot 2
Ow: Ot
cos(w,, Te(i))
(273)
Ot
|| kuy
2 Fe(r)
T Ow: Ot.
[|,,, 113 (5 (z)))
(274)
|/yy ||
1/by Assumption
3.2
(275)
sign (
-sign (IIm,, 1| - (,
T Ow,
cos(w,, Fo()))
(276)
0t
Ot
56

---

Published in Transactions on Machine Learning Research (11/2025)
/ //h clipping operation perserves the sign =sign (1,(11 (z))))))))))
(277)
(278)
(279)
Since we initialize e ~ NO,,, (Tx,,,)) with probability - 2- there exists j such that by < 0 at t = 0
कं cos(we, F(()) < 0 at / = 0. By the continuity of the approximated Langevin diffusion, there exists A/ > 0 such that for any E (0, At),
a cos cos wj Fe(r)) 0.
(280)
We establish that a unique and strong solution exists for the clipped Langevin diffusion. This result is par- ticularly noteworthy because bypasses the standard regularity assumptions typically required in existence proofs for stochastic differential equations (Mao) 1997 Øksendal 2014). Standard conditions demand that both the drift and diffusion coefficients exhibit lincar growth in their parameters and are Lipschitz contin- nous. However, such assumptions are often impractical for the loss functions prevalent in modern machine learning. Additionally, deep learning architectures frequently introduce non-differentiabliiy (as seen in the discontinuities of ReLU activation functions, for instance). In response, we propose relaxed regularity criteria to address these challenges.
Theorem F.6 (Criteria of unique strong solution for SDE with irregular drift (Veretennikov 1981 Theorem 1)). Consider the following stochastic differential equation:
dæ, = a)zr,t)dt + b(x,, t)dX,
(281)
where
Xt denotes the standard Wiener process.
bounded, d-dimensional veclor-valued, measurable function.
b is bounded, matrin-onlued, continnous measurable function of size d x d. b salisfies the following properties:
-Uniformm elliptic condition): For any x E Rd, v Rd, t 2 0, there emists constant X >> 0 such that
o' bI, 1)6² (x, DJu 2 Xo
(282)
(Pixed time uniform continuity):: For every T > 0 and any E [0,T], b(,tt) is wniformly continuous on any compact metric subspace U C Rdd
Then a unique strong solution X, exists for the stochastic differential equation.
Corollary F.7. If the per-sample loss function é has discontinuity set with Lebesgue measure 0, then the clipped Langevin diffusion (Equation (255) has a unique strong solution.
Remark F.8 (Toy-case example of Corollary F.7 Consider 2-layer ReLU network f paramctrized by U Rd W Bddk
f(x) = v relu (W" "))
(283)
a singleton training dataset D = {(00, yn)}
æ0
(284)
57

---

Published in Transactions on Machine Learning Research (11/2025)
and exponential loss (yy. exp(-vÿ) Then the drift coefficient (e.g. a(zs,t) 1) in Theorem F.6 of the loss Langevin diffusion is
-ciipe (Vlo(vo. f(x)))) clipe (Vlo(z0, f(zo)))
(285)
relu(W xx)) virelu (w) 2;)xx; usrelu' (wy 24)xZ
C
min (VIVoe.,)oole)))
yil,
(286)
relu' (m, m: a.)
The set of all discontinuitics of this drift cocfficient has Lebesgue measure zero in the parameter space RA Rdxh, This drift coelficient is a measurable function. So we can apply Theorem F.6 in this example.
Theorem F.9 (Exitence of stationary distribution (Cerrai 2002 Theorem 2.2.1)). Consider the following stochastic differential equation:
dx, = a(re)dt + b(x
(287)
where Xt denotes the standard Wiener process, is d-dimenssionll vector-valued continuous function, and b is malria- malris-valued, continuous function of size d x d. If the following conditions hold:
There exists 2 such that
sup [6(x)|]
+00
(288)
ZER4 1+ |01
The function is locally Lipschitz continuous and there erists m 2 k such that
||(x)|]
sup FERd 1 2x 2m++
+00
(289)
For any 2 1 there erists cp such that for each 2, Rd
(a(x) a(y), x 2) + p1b(()) - b(y) S c, x v1
(290)
There exist v,y > 0, e E R such that. for any x. Rdd
(a(x + h) h) < -k|h|2m+2 + c((==1 + 1)
(291)
Then there erists at least one stationary distribution for the stochastic differential equation.
F.1 Technical results
Lemma F.10 (Gronwall type inequality IV). Let x: [a, b] + R4 be a continuous function that satisfies the incquality:
x(t) S M + L' ((s)w(x(ss)ds, te [a, b]
where M 2 0, 1. R4, is continuous and wi R__ + RR is continuous and monolone-increasing Then.
the estimation
x(1) S -- (D(M) - (((oddd). 1 E [a, b
holds, where R is give by
1
((u) = No (())
ds, u E R
58

---

Published in Transactions on Machine Learning Research (11/2025)
Proof of Lemma F.10 This proof is done by Sever Silvestru Dragomir.
We just copy the proof here for completeness.
Denote y(t) as
v(t) v(t)== = S. ((s(s)) (a,
we have y(a) = 0, and by the recursive integral condition of a, we obtain:
(1) (+)))((), tE (a,
v1( S =())(())
+
1
(((()) (((d)
(M + v(t))
By integration on (a, t) we have
(1000
(00)
wMM =) ds (MM) - (ss)ds
w(M + 8) ds
(s) + ((M)
that is,
( + M) < J( ((s)dss ((M)
By laking the inverse mapping of + on both sides, we finish the proof.
59


Reviews:
[{"review_id": "ZsEjOdeaCb", "reviewer": "Reviewer_UYLt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper studies the training dynamics of DP-SGD for pretrained models. The authors model clipped DP-SGD as a Langevin diffusion and derive a zeroth-order approximation for analysis. Under this lens, they show that naïve full-parameter DP fine-tuning initially misaligns a randomly initialized head with pretrained features, causing a temporary drop in representation quality (feature distortion). They propose a hybrid strategy, DP linear probing followed by full fine-tuning (DP-LP-FFT), and provide guidance on splitting the privacy budget between LP and FFT. Experiments on CIFAR-10/STL-10 with ResNet/ViT backbones (plus LoRA variants) support the theory: small noise favors DP-FFT, moderate noise favors DP-LP-FFT, and large noise favors DP-LP.\n\n**Strengths And Weaknesses**\n\nStrengths:\n1. The paper addresses an interesting problem and presents clear takeaways.\n2. Splitting the privacy budget between LP and FFT is a promising idea.\n3. The hybrid method is theoretically grounded and empirically well supported.\n\nWeaknesses:\n1. Assumption 3.2 (and Assumption 3.1 partially) seems quite strong for a binary classification setting. In practice, this assumption is likely violated, which could undermine the validity of the theory.\n2. The theoretical analysis is conducted on simplified two-layer ReLU/linear networks, which differ substantially from the architectures used in practice. This gap limits the external validity and practical utility of the results.\n3. More details regarding the hyperparameters of DP-SGD in the experiments should be presented, such as the clipping norm C, learning rate, etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are accurate and well supported by the theory or experiments.", "audience_yn": "Yes", "audience_explanation": "Yes, I think some audience will be interested in the results of this paper.", "requested_changes": "1. Authors should better discuss the experimental designs in more details.\n\n2. It is better to plot the privacy-utility curve (accuracy, epsilon) for DP-LP, DP-LP-FFT, and DP-FFT (Table 1) if possible.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tpv0wjB5iW", "reviewer": "Reviewer_kFvb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is an interesting paper! \n\nThe authors study the dynamics of fine-tuning a model under differential privacy constraints. As I understand it, such models are first trained on public data and then they are appropriately fine-tuned on private data. To reduce recall of specific instances from the private data, techniques like DP-SGD are used, which includes clipping and noise injection. Such properties can make formal analysis challenging. \n\nTo study such dynamics, typically researchers resort to various approximations, e.g., Taylor expansions of models, perturbations, etc., which make analysis tractable, albeit at the cost of losing some of the information about the true behavior of the model. \n\nThe authors focus on the alignment dynamics between a randomly-initialized head and a pre-trained backbone under various fine-tuning schedules: full fine-tuning of both parts (DP-FFT), fine-tuning of only the head (DP-LP) or short fine-tuning of the head followed by longer fine-tuning of the backbone (DP-LP-FFT), specifically for the DP case. \n\n# Contributions \n\n## Theory \n\nThe authors construct a zeroth-order theory that captures the feature distortion when jointly fine-tuning a randomly-initialised linear probe with a pre-trained backbone during DP-SGD. \n- To construct the zeroth-order theory, the authors introduce an ODE derived by setting the noise term of a Langevin diffusion model of DP-SGD to zero, getting $\\partial \\tilde{\\theta}=-\\nabla L(\\tilde{\\theta}|f)\\partial t$. This ODE is used to analyze the behaviors of the various schemes. \n- The authors provide an upper bound for the $L_2$ error of $\\tilde{\\theta}_t$ from the exact $\\theta_t$. \n- Theorems that demonstrate the initial distortion due to DP-FFT as well as the stability of features when using DP-LP-FFT, as well as more theoretical analysis between the zeroth-order approximation and the initial Langevin diffusion. \n- A study of the privacy budget allocation for users of DP-LP-FFT, with an explicit formula for the zeroth-order approximation. \n\n## Experiments \n\nOn ImageNet→STL-10/CIFAR-10 and LoRA-style parameter-efficient tuning, the results track the theory (except for the case of Figure 2b, also highlighted by the authors as a case of synthetic pre-training data). \n\n\n# Strengths \n- As you can see from the above notes, there is a lot of content in those 12 pages, but at no place did I feel like the writing was dense. \n- The zeroth-order analysis is presented clearly, with a lot more content in the appendix. \n- Experiments even look into parameter-efficient fine-tuning techniques like LoRA, and broadly agree with the theoretical results. \n- There is clear experimental guidance provided, e.g., the privacy budget allocation for the DP-LP-FFT method. \n\n# Weaknesses \n- It would be interesting to see experiments in other modalities, e.g., text.  \n\n\n[1]: As in the work of Min, H., Mallada, E. and Vidal, R., 2023. Early neuron alignment in two-layer relu networks with small initialization. arXiv preprint arXiv:2307.12851.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are:\n\n1. the explanation of the feature distortion effect under the zeroth-order theory, including the practical suggestions re: budget allocation and\n2. the experimental results on vision examples. \n\nPoint 1. is backed by the theory, as developed in the main paper and expanded in the appendix. The assumptions are clear and the proofs seem to me to be solid. \n\nFor the experiments, sufficient detail is given to reproduce the experiments, which use standard datasets and models.", "audience_yn": "Yes", "audience_explanation": "TMLR readers who care about differential privacy, transfer learning, and the theory-practice gap in DP-SGD will find the paper worthwhile. The zeroth-order surrogate offers a novel analytic tool for understanding how DP noise perturbs feature representations during fine-tuning, and the DP-LP-FFT schedule provides actionable guidance on privacy-budget allocation.", "requested_changes": "# Necessary changes \n\n1. In \"Assumption 3.1\", there is a typo, $i,j$ are not used in the expression (Equation (8)). \n2. \"We follow the approximation scheme outlines in Equation  (4)to derive\" -- equation label needs some space from \"to\". \n3. Proposition 5.2 has some typos, e.g., \"Assumption #.7 hold**s**\".\n\n# Changes that are not necessary, but may strengthen the work. \n\n1. The sentence \"From a contrastive learning viewpoint, it makes the representation of them semantically similar.\" could probably be tightened up a bit to be more specific, e.g., \"it=DP-SGD\" (?), etc. \n2. Perhaps highlighting a bit more the example in Figure 2b -- it's always interesting to look into examples that don't follow the theory closely.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sDbJlXDEL1", "reviewer": "Reviewer_P6M5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "One way to obtain a private neural model is to pre-train a representational or \"backbone\" model $w$ on public data then fine-tune with a linear head $v$ on private data. There are two variants of fine-tuning under consideration: full fine-tuning (FFT), where all weights are updated, and Linear-probing-first (LP-FFT), where the head is fine-tuned before a full fine-tuning pass. The authors extend the prior work by Kumar et al., who found that LP-FFT mitigates OOD errors caused by random initialization of the linear head, to the case where test data is in-distribution and where Gaussian noise to SGD is added for differential privacy (DPSGD).\n\nTheir theoretical results begin with a bound on the error of a zeroth order approximation of Langevin diffusion, which prior work by Ganesh et al. has used to model DPSGD. This is then used to explain why random initialization of $v$ is initially bad for $w$ when performing DP-FFT (Theorem 3.3), as well as why LP alleviates the distortion in representation(s) (Theorem 3.4). The zeroth order approximation is also used to derive bounds on convergence after running DP-LP (Theorem 4.2) and running DP-FFT (Theorem 4.3). Finally, the authors use the approximation to derive how much \"privacy budget\" to spend on the LP pass to avoid distortions (Theorem 5.1).\n\nIn Figure 4, the authors validate Theorems 3.3 & 3.4 by pre-training on Imagenet-1K, fine-tuning on CIFAR-10 & STL-10. In Table 1, they validate Theorem 5.1---\"a greater proportion of the privacy budget should be allocated to DP-LP whenthe total privacy budget is smaller\". In Table 2, they present evidence that such a pattern still holds when FFT is replaced with LoRA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The introduction gives a good exposition of the objectives of the paper. The theoretical results are generally presented in an easy-to-parse manner. And, in my opinion, the experimental results cover all the bases.", "audience_yn": "Yes", "audience_explanation": "DP training and fine-tuning is a very active area of research. Additionally, the zeroth order approximation definitely would be appealing to those already aware of the Langevin diffusion analysis framework.", "requested_changes": "Clarify the following:\n- Theorems 3.3 and 3.4 concern DP processes, but the text surrounding it makes claims about non-DP processes as well (see also 2. in \"Main contributions\"). Are the authors claiming to have contributed a result for both DP and non-DP fine tuning, or just DP?\n- I am unsure what story is told by the paragraph beginning with the bold text \"DP-LP-FFT outperforms other fine-tuning methods: Pre-training on ImageNet.\" Given that $\\sigma$ is fixed to 0.3, doesn't this amount to a (extra) column of Table 1, where privacy budget is fixed to some quantity?\n\nMinor (typographical) errors:\n- Inconsistency between N in (5) and n in Theorem 2.2\n- Assumption 3.1 should use i,j instead of 1,2", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)