DP-SGD-LF: Improving Utility under Differentially Private Learning via Layer Freezing
Authors: Qiaoyue Tang
Mathias Lécuyer
Venue: Submitted to ICLR 2023 (2023)

Abstract: Differentially Private SGD (DP-SGD) is a widely known substitute for SGD to train deep learning models with privacy guarantees. However, privacy guarantees come at cost in model utility. The key DP-SGD steps responsible for this utility cost are per-sample gradient clipping, which introduces bias, and adding noise to the aggregated (clipped) gradients, which increases the variance of model updates. Inspired by the observation that different layers in a neural network often converge at different rates following a bottom-up pattern, we incorporate layer freezing into DP-SGD to increase model utility at fixed privacy budget. Through theoretical analysis and empirical evidence we show that layer freezing improves model utility, by reducing both the bias and variance introduced by gradient clipping and noising. These improvements in turn lead to better model accuracy, and empirically generalize over multiple datasets, models, and privacy budgets.

Under review as a conference paper at ICLR 2023
DP-SGD-LF: IMPROVING UTILITY UNDER DIFFEREN- TIALLY PRIVATE LEARNING VIA LAYER FREEZING
Anonymous authors Paper under double-blind review
ABSTRACT
Differentially Private SGD (DP-SGD) is a widely known substitute for SGD train deep learning models with privacy guarantees. However, privacy guarantees come at cost in model utility. The key DP-SGD steps responsible for this utility cost are per-sample gradient clipping, which introduces bias, and adding noise the aggregated (clipped) gradients, which increases the variance of model up- dates. Inspired by the observation that different layers in a neural network often converge at different rates following a bottom-up pattern, we incorporate layer freezing into DP-SGD to increase model utility at fixed privacy budget. Through theoretical analysis and empirical evidence we show that layer freezzig improves model utility, by reducing both the bias and variance introduced by gradient clip- ping and noising. These improvements turn lead to better model accuracy, and empirically generalize over multiple datasets, models, and privacy budgets.
INTRODUCTION
Deep Neural Networks (DNNs) have seen a growing success at many tasks under various domains in recent years. As result DNNs are now deployed in numerous applications, including some involving sensitive data, such as users' medical history, purchasing records, or chat histories. In these sensitive applications, data privacy is a concern. However, there strong evidence that deep learning models memorize, and thus leak, information about their training data (Shokri et al.) 2016 Carlini al. 2020 [222] |Feldman& & Zhang 2020). To prevent data leakage, common DNN training algorithms such as Stochastic Gradient Descent (SGD) and its variants have been adapted to enforce Differential Privacy (DP) (DP) et al, (2013); Dwork et al. (2006), rigorous privacy guarantee which provably mitigates data leaks. As convenient drop-in replacement for SGD, the DPDSGD algorithm is commonly used for
machine learning, and numerous efforts haave improved its theoretical privacy analysis (Abadi et al. 2016 Mironov 2017 Mironov et al. 2019).
However, the privacy guarantees offered by DP-SGD still come at a substantial cost in model utility (accuracy), despite substantial practical improvements over time De et al.] (2022); Papernot et al. (202T) There are two key changes to SGD that DP-SGD introduces in each model update step. Each change is required prove privacy guarantees, and contributes to utility costs. The first change is to clip per-sample gradient to a fixed Lg norm bound, which introduces bias in the estimation of the gradient descent direction. The second change is add noise from a standard Gaussian to the aggregated (clipped) gradients, which increases the variance of model updates.
We show using theoretical analysis that increasing the gradient clipping norm of a given DNN layer in DP-SGD reduces the variance introduced by DP noise and, under some assumptions, the clipping bias well. Both lead to better convergence upper-bound the observation that different layers in a DNN trained with SGD converge at different rates following a bottom-up pattern-which which we empirically verify also holds for DP-SGD-and and introduce the DP- SGD Layer Freeze (DP-SGD-LF) algorithm. This algorithm freezes the lower layers (closer to the input) of a DNN towards the end of training, which increases the norm of clipped gradients for the remaining layers, thereby decreasing the bias and variance introduced by DP-SGD when updating these parameters. Since the remaining layers benefit more from updates at this point of traininig, the finial accuracy increases.
for DP-SGD. We combine this result with
We apply DP- DP-SGD-LF to state of the art DP-SGD implementations on three datasets ee al. (002);; Papernot et al. 2021), and show that improves the final model's accuracy by up 1.3 percentage

---

Under review as a conference paper at ICLR 2023
points, and is particularly effective in the high privacy (low DP e) regime. We also show that DP- SGD-LF is not sensitive to typer-parameters, and propose and use easy to set, reasonable defaults.
The rest of the paper describes our contributions: after introducing the necessary background in $2
3 introduces our algorithm, and supports its design through empirical and theoretical analysis. $ then empirically confirms the expected behavior, and shows that DP-SGD-LF improves the accuracy of different models over multiple image classification datasets.
2 BACKGROUND
Mini-batch SGD is one of the most commonly used optimization algorithm in non-private deep learning. For each iteration I, and calling ग the step size, SGD updates the parameters of the model / by stepping into the direction of steepest descent, estimated with the averaged gradients over B samples mini-batch.
B
0+11 - 01
1/t
1
Σ (1x)))
B
The convergence analysis of the SGD algorithm often rely on the following fundamental result.
Lemma 2.1 (Decent Lemma (Bottou et al. 2018)). Assuming the objective function R - R to be continuously differentiable and the gradient of f. Vf R - Rd to be Lipschitz continuous with the Lipschitz, constant L 0, [[V[(v) V [ww)l Lllu - w|l, Vv, w, then f(v) < f(w) +
V (u w)
$ ||u w 1 Vu, w.
Under privacy constraints, the DP-SGD algorithm provides a convenient substitution to SGD for training DNNs with differential privacy guarantees (Abadi al.) 2016). The DP-SGD algorithm protects privacy by clipping the per-sample gradient vector, 9r (æ;) < fe, 2;), and adding noise drawn from a Normal distribution to the aggregated clipped gradients. Let C be the L2-norm clipping threshold, be the noise multiplier, and d be the dimension of the model's parameters. the update rule for DP-SGD in each iteration is:
Σ clip(ar(a)).)) + +z
B
x = NN,N NN(N00,0
0+11 + 0g - -tt
B
(2)
(x:() O) + gr()))/ max |19 C
where controls the maximum influence that an individual sample can have on the gradient (the sensitivity), and controls the noise level scaled with respect the sensitivity. We use the analysis based on Rényi Differential Privacy (RDP) (Mironov) (017)) for privacy accounting. The compo- sition over t steps of training and the conversion of the RDP guarantee to the ( guarantee follow from the results in Mironov et al. (2019). we use the publicly available implementation of the RDP privacy accountant Opacus (Yousefpour et 2021).
DIFFERENTIALLY PRIVATE LEARNING WITH LAYER FREEZING
We propose incorporate layer freezing with DP-SGD, and demonstrate its effectiveness in in- creasing the trained model's predictive accuracy at fixed privacy budget. The intuition behind the performance gain as follows. The two key steps in DP-SGD, clipping and noising, provide DP guarantee at the cost of degrading model utility: clipping potentially introduces bias into the estimated descent direction, since it truncates individual gradients before aggregation to control sen- sitivity (Chen et al. [2021] Pichapati et al.) 2019; Zhang et al.) 2019); noising introduces variance on top of the biased estimate by adding random noise to the aggregated clipped gradients. Freezing parameters limits the model capacity in learning representations, but could bring benefits by reduc- ing the bias and variance caused by clipping and noising on the remaining trainable parameters. Given the observation that lower layers (closer to the input side) converge faster than higher layers (closer to the prediction). we can freeze the parameters in lower layers during training, to minimally sacrifice model capacity in exchange for the benefits of better updates for the upper layer parameters.
2

---

Under review as a conference paper at ICLR 2023
We detail approach in the rest of this section. $3.1 shows the algorithm propose. 3.2 presents empirical evidence that lower layers in a DP SGD trained neural network converge faster than the upper layers. $3.3showw theoretically that clipping and noising can be expected to degrade the convergence of model training, and presents our key metric to quantify this negative impact: the distortion angle in estimating the descent direction. In $3.4 we show that under some assumptions, layer freezing can improve both bias and variance caused by clipping and noising with respect to the trainable parameters. $ then empirically evaluates our claims and demonstrates the effectiveness of layer freezing in improving model utility under multiple settings.
3.1 THE DP-SGD-LF ALGORITIM
Algorithm D shows the pseudo code of our method. When the current iteration index & exceeds preset threshold L², we apply parameter freezing on the first m layers (closest to the input) of the model architecture. t and m are two hyperparameters of the algorithm. In $4 we empirically show that the performance is not sensitive to these hyperparameters a wide range of values, and provide generic defaults that
in experiments. Before freezing, the algorithm behaves exactly as DP-SGD. After freezing (t >> t/), the frozen layers' gradients are set to 0 before clipping, and not noised. The frozen parameters
not updated. The rest of the parameters usual, following Equation 2 Since layer freezing is decided by hyper-parameters without adapting the data, the privacy analysis of DP-SGD-LF is identical that of DP-SGD, and existing privacy accounting can be re-used.
updated as
Algorithm 1: DP-SGD-LFF
Output: Model parameters 8
Input: Dataset D = (ai v)) loss function f. learning rate m,s batch size B, noise multiplier a, 1.2norm clipping threshold C, privacy parameters e, 6, freezing parameters t/, m!
Initialize Oo randomly
Calculate the total number of iterations T(e, B,N,
for t 1......... .........
Take a random batch with sampling probability B/N
tft= > tf then
Partition 6g into Ofoozen and 6 grrainable according to the first m layers
Update Ofrainable as in Equation 2
Merge Ofrozen with Opginable as 0,+1
else
Update Oc+1 as in Equation]
end
end
3.2 LAYER CONVERGENCE FOLLOWS A BOTTOM-UP PATTERN IN PRIVATE TRAINING
There is strong empirical evidence to suggest that for DNNs trained in the non-private setting, layers in the DNN architecture converge at different rates, and exhibit bottom-up pattern (Wagg et al.) 2022 Raghu et al.) 2017 Morcos et al. 2018 Yosinski et al.) 2014; Rogers et al.) 2020). In this section, we verify that similar observation holds under private training. The dataset, model, and algorithm used for demonstration are CIFAR- 10, a 5-layer CNN (with the last layer being softmax- activated classification layer) and SG- with C 3 and 1. The accuracy of the full model is around 0.63 at the last training step, when is around 7.
We examine the privacy-utility trade-off of the DP-SGD trained model by training only a single layer after = 3 where the model achieves moderately high accuracy but is not fully trained yet. We note that the final layer the softmax activated classification layer and not frozen these experiments. As shown in Figure [ (Left), only Layer trained further in steps while a larger increase is observed when only upper layers are trained. The final accuracy obtained by training Layer I only is also considerably lower than when only training the other layers. To support this observation, we also measure the convergence quality using post-hoc analysis tool, PWCCA (Morcos et al. 2018), which compares different layers'
observe that there is a minor gain in accuracy when
3

---

Under review as a conference paper at ICLR 2023
0.60
Layer
Layer x Trained Only
0.25
Layer
0.55
Layer
Test Accuracy
0.50
0.20
Layer
0.45
Layer 1
PWCCA
0.15
0.40
Layer 2 Layer
0.10
0.35
Layer
Monn
0.05
0.30
2
6
20
40
60
80
100
Epsilon
Training Step
The privacy-utility trade-off when only single layer is trained with every other layer frozen (except for the final classification layer) after 3. Layer has the least amount of gain in accuracy when other layers are frozen while the upper layers perform similarly. Right: The PWCCA score over training steps for different layers. lower PWCCA score indicates model converges better. Layer 1 shows weak sign of convergence while the upper layers has strong evidence of converging.
Figure 1: Left:
intermediate activation vectors throughout training to the converged activation vector from fully- trained model (i.e., the final iteration the model converges to, in which the accuracy is reasonably high). Figure (Right) shows the PWCCA score for the first layers over training steps. A lower PWCCA score means that the layer converges better. We observe that only Layer shows a weak sign of convergence, while the upper layers are likely to be dominated by the accumulated noise and have clear sign of convergence. These experiments confirm that: lower layers converge better and earlier during training with DP-SGD, and focusing training on higher layers leads to more utility.
3.3 DISTORTIONS IN DESCENT DIRECTION DEGRADES DP-SGD PERFORMANCE
((m,))
Before freezing
After freezing
8
8
9 (mma)
Tt
B1
Tt
1t
't
BP
17B
YE
Tt
Figure Left: example true (V (V.), unbiased 4 biased (s("), private (s signals and the distortion angles (yPP) 7 UB BBP) as in Definition 3.1 and 3.2 Middle: Assuming the unclipped and clipped gradients remain unclipped and clipped after freezing, when s more aligned with the clipped gradients, then 7 is reduced after freezing since the clipped gradient is clipped less (magnitude increase prior freezing) due to an decrease [1g)( Right: For an arbitrary noise vector, since of increases in magnitude 7f BP is reduced after freezing such that the noising makes sf" less variable in direction.
We first define the true, unbiased, biased and private signal vectors (which refer to the true descent direction, and different estimates of it from minibatch: without changes, with clipping, and with clipping and noise, respectively) and the corresponding distortion angle in each step t of the opti- mization algorithm. Figure 2 (Left) shows demonstration using 2-dimensional vectors. We adapt the Decent Lemma (Lemma 1. to DP-SGD, and show that bias and variance in the distortion angle will increase the upper-bound convergence. This suggests that larger bias (mis-oriented descent direction) and higher variance in the distortion angle is likely to lead to slower convergence.
Definition 3.1 (The True, Unbiased. Biased and Private Signal Vectors). For each step t, the true signal vector V is the gradient of the loss function evaluated at 02 all the training data.
4

---

Under review as a conference paper at ICLR 2023
is the true direction of steepest descent for the empirical loss. The unbiased signal vector s is the mean gradient over samples in a batch drawn following a uniform sampling scheme. It is an unbiased estimate of ft (Bottou et al. 2018). sy == B Iz(x.). The biased signal vector se is the mean of sample- clipped gradients on batch drawn following uniform sampling scheme. is a biased estimate of 1 since per-sample gradients are re-scaled differently before aggregation, se = = clip(ge(x,), ()) Let 21 ~ (0, (1/B3)02C21) be the random noise vector, the private signal vector is the sum of B and 2, and is the actual descent direction in DP- SGD updates, sf. = = 2t.
With Definition 3.1 the parameter update rule of DP-SGD can be rewritten as:
0++1 < 0r 17
2t.
We note that gradient clipping in DP-SGD has been shown to be biased and a bias vector by can be decomposed following the analysis of Chen et al.) (202T). We define the distortion in estimating the descent direction by the angle between V and s.
Definition 3.2 (Distortion Angle). For each step the distortion angle " is the angle between fe and st.
(V of V sell
== arccos
where (.) represents the dot product and ||-|| represents the L2-norm. "t can be decomposed into B and BP which is the angle between V ft and sy, and between of and sf, respectively; % It. gbb ht 7t
R s.. Yt. 7/2 and %t" BP are taken with In what follows, the expectations and variances of s St
respect the data sampling and the DP noise distribution when applicable.
We show next show that the bias and variance introduced by clipping and noising leads to a worse convergence bound rate bound for DP-SGD. The proof of Lemma 3.1 is in Appendix A
Lemma 3.1 (DP-SGD convergence bound). Following the proof of Bottou al. (2018) for DP- SGD, we show the following result. Assuming the objective function f : - R. to be continuously differentiable and the private gradient of J, V IRd + Rd to be Lipschitz continuous with the Lipschitz constant L O, [VV[(v) [(w)] < Lll wil, Vu, the convergence bound of DP-SGD is.
min E V < f(0o) [((()r)
BE cos (
2 o DP
m
The
Finally, we explain why % is an effective metric for utility. From the convergence bound above, we the negative effect of % due clipping and noising: a larger bias (T and variance (oDP) make the upper-bound larger, and could potentially lead to worse model performance:
(1) A negative value of E[ cos (7t) (O))l could be caused by large bias angle 7B. This makes the convergence bound worse by adding a positive term. A smaller 7 (1 E [0, g) means that the DP-SGD descent direction is better aligned with the true direction of steepest descent (7 smaller), and EE| cos (1 [0.)]l]))
positive. However, h > still
makes co)(1 < 1, increasing the bound.
(2) Since 2; is drawn from zero timation. However, from the convergence bound we see that a larger variance of the noise (L/2B")obpCC also makes the bound worse. higher variance means a larger noise is more likely added to sf therefore would expect 7, to be large.
Normal distribution, adding noise does not bias the
5

---

Under review as a conference paper at ICLR 2023
3.4 LAYER FREEZING MITIGATES DISTORTIONS IN OPTIMIZATION DIRECTION
In this section, we introduce (strong) assumptions, but supported by empirical measurements in Appendix H under which we prove that freezing some parameters benefits the remaining trainable parameters, by reducing the bias and variance in their distortion angle 7/-- Intuitively, since sub- set of the parameters are frozen, the gradient's s L2-norm is reduced (the frozen parameters have a gradient of zero). This in turn leads to an increase in the magnitude of each sample-wist clipped gradients on the remaining parameters. Under our assumptions, clipping gradients less aggressively reduces bias in the distortion angle y, since each clipped gradient is closer its original value. A larger biased signal is also more robust to noise, and 7: is smaller for any fixed noise draw, lead- ing reduced variance. Figure 2 (Middle and Right) demonstrates this intuition with 2-dimensional vectors, and the formal results are stated below, following from results which we prove in Appendix aadd
Lemma 3.2 (The magnitude of the biased signal of trainable parameters increases after freezing). Let s (0)) and s (0()) be the biased signal of the trainable parameters before and after freezing, then lg (x:4 I| S (r;)] Va, and || [.sB.a (6))) 2 ||s (B)))l
Assumption 3.1. The mean of per-sample gradients g(æi) with a larger magnitude such that [[g(x:)] > C' are more aligned (smaller angle) with the gradient direction V ft. whereas those with a smaller magnitude such that [[g(x;)]| S C are less aligned (larger angle).
Assumption 3.2. Rescaling the gradient norm of [[g(z;)]] by freezing does not change the gradients that are clipped and unclipped gradients, but only rescales the clipped gradients.
Proposition 3.1 (Freezing reduces bias and variance in distortion angle with respect trainable parameters). Let O, be the set of full parameters and let 02 be the subset of trainable parameters. Let Ve (0() be the distortion angle with respect to trainable parameters. Let the superscript and denotes the quantity before and after freezing occurs. Under Assumptions 3.1 and 3.2 the following results hold:
(1) Bh. B. (6)) s Ehg (0)):
(2) Varh: Barh (P)) (ar (3) Ehi (0)) < (0)), Varh? (0)) S Varh
Proof. (1) By the clipping function in Equation each per-sample gradient is either clipped if [19)(1))]l > C' preserved to its original value if (x;)) I C. Let g' denote the gradients with respect to trainable parameters, v and w be the vector of the sum of unclipped and clipped gradients in random batch of size B respectively,
Σ sl
(d(lt)) w =
E
g((xk)) :) lil = B.
U
By Lemma 3.2 since (xx]) /) II. L.e the magnitude of the clipped gradients increase after freezing comparing to before. By Assumption 3.2 v does not change. Since wn 2 wD we have:
(xx)) Vi, (o(((x) O) > (g(()))
((,utw0) wa)
(0, + Tuullle + woll w )
< E arccos ( (v, (w, w0
[T|l||e + wO )
E arecos
which means E(y, BBa < E(,,. Under Assumption (("B"(00))) < El:
3.1
then we have if E(y Uv.a > B(
B,b (0))
(2) Since the noise is drawn from mean Normal distribution, for each batch, when taking expectation over the random noise draw, BPP BP O. The variance term can be simplified as,
(sf (O((O)), (() (15 (1s)(()) S (0) + 2, )
(0)) + =x)
Varh BP] E[(xBP) BPD2 = E arccos

---

Under review as a conference paper at ICLR 2023
Let X denote the random variable inside arccos (.), for every batch the distribution of the private gradient is the same as the random noise distribution,
Varha 2
(arccos (z))) X N(0,022CT))
We show the following result in Appendix
C
(s)(0))) (0)) + =+)
27
(4f(00))))P()))) 2)) 27
E arccos
(10P (0)) 8 + zz
S
arccos
(1 + 2.
Therefore Varly BP,a (0)) S Varh BP (8))).
(3) is a consequence of (1) and (2), since [b) El (()()) E/y
(8)) +
El B. (0))
since E/x/P (P)))()) = O. Var (f,) (0))
Varh. (0,) ( ( (0)) since the noise 2 is drawn inde-
pendently. By Assumption 3.1 V batch with size B, T I 7t thus Varln? (0)) S Var(?
Baa B.b
Varha (e0)) s Varlm B.b
(0))
4 EVALUATING LAYER FREEZING IN END-TO-END PRIVATE TRAINING
In this section we empirically examine our method on utility improvements over multiple datasets, models and privacy levels. We also empirically evaluate the claims from previous sections. We also perform sensitivity analysis on hyperparameters, and present our suggested defaults. Unless otherwise specified, the analysis results in this section are demonstrated using CIFAR-10 and a 5- layered CNN model.
Test Acc.
Datasel
Model
(ours)
==2
<=3
X
0.956 (0.002)
0.975 (0.002)
0,981 (< 0.001)
MNIST
CNN
10-3
V
0.957 (0.001)
0.976 (0.002)
0.981 (< 0.001)
X
0.801
(0.005)
0.855 (0.007)
0.862 (0.002)
FashionMNIST
CNN
10-5
V
0.815 (0.005)
0.868 (0.002)
0.875 (0.002)
X
0.533 (0.010)
0.637 (0.005)
CI
CNN
10-5
0.548 (0.003)
0.651 (0.003)
X
0.558 (0.252)
0.625
CI
ReseeResNet
10-5
0,571 (0.120)
0.636
Table I: A summary of the results training with DP-SGD with and without layer freezing. The results are the median and standard deviation over 5 independent runs. The bold results indicate better performance.
Baseline models and experimental setup. We implement layer freezing top of the existing baseline models. For MNIST, FashionMNIST and CIFARIO IO with CNN, we use the baseline model from Papernot et al. (2021). For the CIFARIO experiment with Wide-ResNet, the baseline model from De et al. (2022). All hyperparameters related to the model, training, or DP settings are kept constant between with and without layer freezing (and tuned without freezing in the baselines' papers). For each experiment, we run the model with and without layer freezing times indepen- dently, using different random seeds. For the CIFAR1O 10 model with Wide-ResNV able to repeat the experiments for = 1 settings due to limited computational resources. The exact hyperparameters and other details are included in Appendix D
only
Model utility under DP-SGD-LF. Table I compares the performances of DP-SGD-LF on top of the current 'state-of-the-art' baselines and show the median and standard deviation over 5 runs. We observe that layer freezing generally improves the predictive performance across datasets, models and privacy budgets, as it improves the median score and has a smaller standard deviation.
Layer Freezing Improves Angle of Distortions in Optimization Direction Figure B shows the median of [3 of the trainable layers after freezing the first 3 layers after step 5000. In the last step, the model accuracy is around 0.64, for = 7. We observe increase in signal strength for both

---

Under review as a conference paper at ICLR 2023
Median Biased Signal
Layer
of Trainbale Layers
0.04
0.2
0.03
0.02
0.01
1000
6000
1000
6000 Training Step
Training Step
Figure 3: The median of || B for trainable layers over training steps after the lower 3 layers frozen at step 5000. We observe in both Layer and there is increase in the signal strength.
Unblased Signal of Trainbale Layers
1.4
Angle Between Blased and
Layer
1.2
1.2
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
Layer 5
0.0
0.0
2000
4000
6000
2000
4000
6000
Training Step
Training Step
Figure 4: The change in 7 UB for trainable layers after freezing the lower 3 layers at step 5000. We observe a decrease in "t both Layer 4 and indicating that the biased signal is closer to the unbiased signal after freezing.
layers which matches the claim in $3.4 Figures andd show the change in the distortion angle when layer freezing is imposed. We note that since the full-sample-size gradient is expensive compute, we measure the angle between the unclipped and unnoised SGD gradients 7 on the same batch of data. We thus make the underlying assumption that moving the DP descent direction closer the minibatch direction in each iteration would improve the private training performance. We observe that both 7/ and BBP decrease after freezing for trainable layers. The distortion measured in y is generally weaker than in ?PP as the absolute scale is higher in the later. Although the privacy hyperparamters and a affect the results, we generally observe that noising has a stronger negative effect than clipping in terms of distorting the optimization direction.
The tradeoff between gains of freezing and model capacity. The benefits of freezing layers comes at a cost in model capacity, which potentially limits performance. We evaluate such a tradeoff
Private Signal of Trainbale Layers
1.55
1.55
Anglee le Between Blased
1.50
1.50
1.45
1.45
1.40
Layer
1.40
Layer 5
2000
4000
6000
2000
6000
Training Step
Training Step
Figure 5: The change in y BP for trainable layers after freezing the lower layers at step 5000. We observe decrease 7 in both Layer and 5, indicating that the improved strength in signal makes it more robust to noise as noising changes the direction less.
8

---

Under review as a conference paper at ICLR 2023
0.6
0.6
Test Accuracy with
Layer Freezed
Test Accuracy when
0.5
Freeze Layer 1
Freeze Step 20
Freeze Layer 1-2
Freezing
Freeze Step 50
0.4
Freeze Layer 1-3
0.4
Freeze Step
Freeze Layer 1-4
Freeze Step 90 Freeze
No Freeze
0.3
0.3
Epsilon
Epsilon
Figure 6: Evaluating the hyperparameter choices of how many layers to freeze (m²) and when to start freezing (t²). Left:The privacy-utile trade-off when freezing different number of layers after <= 3. Right: privacy-utility trade-off when freezing the lower 3 layers at different steps.
by measuring the final model accuracy under different choices of freezing hyperparameters: how many layers to freeze (n') and when to start freezing (m/). Figure 6 (Left) compares the model's utility when freezing different number of layers. In these experiments, freezing starts after 3. We observe that in general, the choice of how many layers to freeze is not quite sensitive as long as the model still has a reasonable capacity (e.g., leaving only the last layer insufficient). Figure 6 (Right) compares the model utility by varying when start freezing layers. In these experiments, we freeze Layers 1-3 at different training steps. We observe that freezing early can result worse performance, whereas freezing in later steps does not bring any gains prior freezing.
Determining the optimal values for tf and m might lead to additional privacy cost. Since the utility not too sensitive to these hyperparameters in wide reasonable range, we suggest the following default hyperparameters, which are those we use for all experiments in Table[]] given a target privacy budget e, we calculate the number of training take (based DP and optimization parameters), and set t! to be about 20 steps earlier. We set m to be the lower half of all layers in a model.
5 RELATED WORK
There is rich literature on representations learning in DNNs under non-private training. It is com- monly observed that the lower level layers extract more general features and are easier train while higher level layers capture more abstract and task-specific features and take more steps for learning good representations (Raghu et al. 2017: Morcos et al. 2018 Yosinski et al. 2014 Rogers et al., 2020). Wang et al. (2022) demonstrate with image classification tasks that different layers converge with a bottom-up pattern. This line of work promotes the possibility of transfer learning, with a subset of the parameters to be inherited from pre-trained models, and kept frozen when fine-tuning on downstream tasks. In non-private training, parameter freezing is mainly used to reduce data requirements, computational costs, communication (Zhuang al. 2019).
Parameter has also been studied in the private training setting, also under the transfer learning sce- nario but with additional privacy benefits. A subset of the model parameters are transferred from publicly trained models, and are frozen when fine-tuning with privat data on the downstream task. Such an approach is empirically effective across multiple computer vision and natural language pro- cessing tasks (Tramer & Boneh 2021: Luo et al. 2021 Yu et al. 2021: Li et al. 2021 Mehta et al., 2022). In these work, the frozen parameters are either used directly as good iiitializations, or tached to additional layers for finetuning. In [uo et al. (2021), the parameters chosen to be frozen are the ones with smaller scales which are usually considered less important in neural network. coincides with observation that freezing the lower level layers, which converge early, does not overly hurt model performance.
A few work closely related works incorporate techniques to increase sparsity in private learning. Zhang et al. (202I) presents a theoretical study on the benefits of sparse gradients in wide neural network models trained with DP, Talwar et al. (2015) studies the DP LASSO model which encour- ages sparsity by design. Huang et al. (2020) shows that pruning can be an alternative approach to privacy.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, ACM, oct 2016. doi: 10.1145/2976749
2978318. https:::///oi. org/ 10. 0t1. 2977331
Borja Balle, Leonard Berrada, Soham De. Jamie Hayes, Samuel L Smith, and Robert Stanforth.
JAX-Privacy: Algorithms for anvacc-preseenve machine learning in jax, 2022. URL http:
//github. com/ deepmind/ jax privacy
Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization Methods for Large-Scar Machine Learning, February 2018. URL http: http/axxxvvv...... wtwpg////////../// 04838
arXiv:1606.04838 [es, math, stat].
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman MMine, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com www.goog.//oooooooooooooooo.oo.o e/jox
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voor Katherine Lee, Adam Roberts. Tom B. Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raf- fel. Extracting training data from large language models. CoRR, abs/2012.07805,. 2020. URL
https://arxiv org/ abs 2012 2. 2.8805
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Mem- bership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), 2022.
Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding Gradient Clipping in Pri- vate SGD: Geometric Perspective, March 2021. URL htt.:ppp/aax..v.og/aaaaa//////// 15429 htt::0006.14229 [cs, math, stat].
Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlocking High- Accuracy Differentially Private Image Classification through Scale, June 2022. URL http:
//arxiv.orggaas//22000 12350
Number: MXXv:2204.11365) arXiv:2204. 13650 [es, stat].
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography, TCC'06, 2006.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 2020.
Yangsibo Huang, Yushan Su, Sachin Ravi, Zhao Song, Sanjeev Arora, and Kai Li. Privacy- preserving learning via deep net pruning. CoRR, abs/2003.01886, 2020. URL https:
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. CoRR, abs/21 10.0567,, 2021. URL
org/ abs/ abs/2/l0 22100.110..0 11111.0006679
Zelun Luo, Daniel Wu, Ehsan Adeli, and Li Fei-Fei. Scalable Differential Privacy With Sparse
Network Finetuning. pp. 5059-5068, 2021. URL htt.tttss///peeccesssss thecvf. com/
www.bbleeDiffeeeeeee htt.:///w///aanooooaaaaaaaaeeeee
Sparse_Networr www.uuFFinnnnieeecec CVPR 2tt_l22222___peeee1_11
Harsh Mchta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer
learning for differentially private image classification, 2022. URL
abs/ 2ass/2000 2205.220.... 02973
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Sym- posium (CSF). IEEE, aug 2017. doi: 10. 101... URL https: /ooi orq/i..
110982Fesf 2017.11
10

---

Under review as a conference paper at ICLR 2023
Iyy Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism, 2019. URL battps://arxtti www.wwwwwww..aaaaa..aaeeeeaeeaee wttpaaa//900.... 10530
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity similrrt in neural networks with canonical correlation, October 2018. lttp:/ap//ari/ hbt/g//bb//880
05759
Number: 10t.:11000055555 arXiv:1806.0555
[cs, stat].
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Ülfar Erlingsson. Tempered Sigmoid Activations for Deep Learning with Differential Privacy. Proceedings of the AAAI Con-
ference Artificial Intelligence, 35(10))9312-932), May 2.21. doi: 16.1/a9aaaai, http://aaaa..33100000 #35i10.17123
URL paattps:/ojjs.aate org/
wwtg/indeen it//x.ndexx.. wwt.ndxw.pppppppemmmommpe phpppAlI///appppppp /ttp:///../pa...aaaat.iee/////// http://tttttttteeeeeeeeeeeeeatee 17123
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu, Sashank J. Reddi, and Sanjiv
Kumar. Adaclip: Adaptive clipping for private SGD, CoRR, abs/1908.07643, 2019. URL
http:/ /ttp///xw/..g//////19990..0..000
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vec- tor canonical correlation analysis for deep learning dynamics and interpretability, 2017. URL
https://arxii: wwtp:/wwwwww.grp////.ooo/////
05806
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What know about how BERT works. CoRR, abs/2002. 2020.
12327
Reza Shokri, Marco Stronati, and Vitaly Shmatikov. Membership inference attacks against machine learning models. CoRR, abs/1610.05820 2016. URL http://arxxiv.orr//bbb//oo
05820
Shuang Song, Kamalika Chaudhuri, and Anand Sarwate. Stochastic gradient descent with differ- entially private updates. In 2013 IEEE Global Conference Signal and Information Processing, 2013.
Kunal Talwar, Abhradeep Guha Thakurta, and Li Zhang. Nearly optimal private lasso. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL Dttp:://ppoceedagggnnell newripps........n....p....s..... cctpapper/21///11eee 3 01010022272233166666777728228823
.pff
Florian Tramèr and Dan Boneh. Differentially Private Learning Needs Better Features (or Much More Data), February 2021. URL
MrXiv:2011.1166 [cs, stat].
Yiding Wang, Decang Sun, Kai Chen, Fan Lai, and Mosharaf Chowdhury. Efficient DNN Training with Knowledge- Layer Freezing, January 2022. URL. http: org/ abs/
22C1. 06227 Number: arXiv:2201.0220 arXiv:2201.06221 [cs].
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? CoRR, abs/1411. 1792, 2014. URI. htta:://axxiv.orr/aabb..
1722
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorrh. arXiv preprint arXiv:2109. 120.:22/.../2.8.... 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Ja- nardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. CoRR, abs/21 10.06500, 2021. URL.
https: /ttr///rxxiie arxii.iv.. org/ abs /211 0. 65000
Huanyu Zhang, Ilya Mironov, and Meisam Hejazinia. Wide Network Learning with Differ- ential Privacy. June 2021. URL http:/ /ttp///wwww..o///////...../// 01229 Number:
arXiv:2103.01294 arXiv:2103.0129. [cs].

---

Under review as a conference paper at ICLR 2023
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity, 2019. URL https: www.v.org/o....o....o.o.....oowt abs/
1995511880
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. CoRR, abs/1911 1.02685, 2019. URL
fttp:://axxi.oor//bbbo.
026655
A PROOF OF LEMMA 3.1
Lemma (DP-SGD convergence bound). Following the proof of Bottou et al. (2018) for DP-SGD, we show the following result. Assuming the objective function J IRd R. be continuously differentiable and the private gradient of J. Vf TR + Rd to be Lipschitz continuous with the Lipschitz constant DP-SGD is.
[Vv((v) Vf(ww)l I Lile w|l, Vv, w, the convergence bound of
min E[IVF())) & f(0)) [[(((r)]] O....,
Σ Th E (((O))))
Eh
02
Σ n/-
2B
Proof. Let be the loss function we want to optimize, let vf(),) be the true steepest descent gradient vector, and let sf sf be the unbiased and private gradient as in Definition 3.1 at step Since the private gradients are assumed to L-Lipschitz continuous, the descent lemma implies that,
1(0++1) S S(0,) (O.00) - 0c) 10 1 0cl1?.
Substituting in the parameter update rule of DP-SGD, 0,+1 + 0 - M,s,", - we get,
I 1101.111
f(0++1) f(0.) Ve ((.)) +
Taking the expectation over the data distribution and assuming the step size Th is independent of which data are sampled in each iteration we get,
E [[ s f(0)) m VF(O)) + ni " =(
The private gradient sf is composed of the biased gradient of and the random noise z,. The bias in s! can be isolated and quantified by integrating over the probability density function of the gradient noise caused by data sampling (Chen et 2021). Therefore we can simplify the expectation of s as,
E E[s['] = R [ + b + [[2]] (f())) + b,,
for some bias vector 6, The second equality holds because sf is an unbiased estimator of the true gradient (f.) with the assumption that each x; drawn with a uniform sampling scheme in the regular mini-batch SGD, and B[zz] O since žt (N (1/ (1/B")0²C*I) The variance of of is bounded under DP-SGD since we performed gradient clipping to control sensitivity and added the noise drawn from Normal distribution. Given batch size B, L2-clipping norm threshold C' and noise multiplier Dpp,
E [1p < B Br.
Therefore, the progress bound for each parameter update step / of DP-SGD is
L th 2B 2
E [((+z))) & f(0)) n (IV (f(0.) b.)
C2
Rearranging the terms and summing over all iterations t 1,.... T we get,
TellV
f(Oi) - ((0))-E(((P..) E(f(((+)))
(f.)) b, + m 2 o )
L
12

---

Under review as a conference paper at ICLR 2023
Taking expectations both sides and simplify using the Law of Iterated Expectations get,
T
T Th 2B
1 S Σ -E[(0)]]][[((++]
Σ
Σ
2
Similar to the usual gradient descent scenario, when the magnitude of the gradient vector shrinks and gets closer to O, we consider the algorithm is converged (to optima to saddle point). We can rewrite the above inequality terms of the smallest gradient norm over all training steps as,
I
min E V
m
n [B
L
Σ nE
(0,)
+
+
Th
2
o
Dr
T
T
- f(00) - [[f(rr] D
nB f((.))
π 2B
Dr
Let sf and 7f be defined in Definition 3.1 and 3.2 quantities,
rewrite the bound in terms of these
EV = ( s E[ cos (c
(f)))
Therefore the convergence bound of DP-SGD is,
min E [[((0) <
f(Ob) -E[((f]rrr Σ I cos (V
I
o 02
mh 2B²
n1-
B PROOF OF LEMMA 3.2
Lemma (The magnitude of the biased signal of trainable parameters increases after freezing). Let sf (B)( and of." (0,()) be the biased signal of the trainable parameters before and after freezing, then [gg (2+) I (x:)( Vz, and || B. (0)) 2 || (0)))-
Proof. Let gi (z,, 01). go (I., 8%) and go (I,, 8)) be the corresponding gradient vector of the full, frozen and trainable parameters for each instance n,, since O, 0² 02 then
(x; 0z) (x, 0i U ( 0'), di (I;, 0²) gi (xi 0') = gi 0').
By the Triangle Inequality, we get,
l 0.)1l l1 (zi, 07) + g) 0'11 < 11 (z+ 001 + ( 0 Since (I, 011 2 O, 11o 0.)1 2 1 (z., 8 = (x,, 0.)1l By Definition 3.1
ge (z.) /max (1, 119 0D1l
$ B.b
BBb()') (0))
B
ge (x. max (1, (xg #))1
s n.a ((,) = B
Σ
so it follows that |19ا (x:))l < (x:))) Vo, and || (0))) [| a0))ll
13

---

Under review as a conference paper at ICLR 2023
C SUPPORTING PROOFS FOR PROPOSITION 3.1
We first show a common result that helps the proofs stated afterwards: The cosine similarity in- creases between two vectors if either one of the vectors is sum of the other vector plus third vector with a larger magnitude. Let 01 and t2 be two arbitrary vectors, let vy be the sum of ti and 22. bl U1 222- Keeping the same, if the magnitude of 1/ increases, then the cosine similarity between and Vi is larger, leading to smaller angle between i) and V3. The same result also holds if U2 is kept the same and the magnitude of U1 increases.
Note that v1, 12 and vy are all 2-dimensional vectors, then the claims follow naturally from the Parallclogram law. We show that the claim holds in the general p-dimensional case. Let Vi [Us1, .2...... Miyl be the flattened p-dimensional vector, we show one direction of the results which when 81 is kept the same and the magnitude of 02 increases, 2/2 = kvy for some 1, and the other direction follows by symmetry. we show that
(V2, (V2 + vz)) LHS
(V2 (V22 + vi))
RHS.
V/2 (v2 + v1)Il
v1)Il
Expanding the dot product and the norm into summations on both sides, and let
a U25 012 c= Σ vic-
Then we can simplify LHS and RHS as,
kaa
+ b RHS Va + va c+ 2h
LHS
kvā+ + Vitkk-a+aa 2kb
Since a 2 c 0, if 0, we can easily verify that, (1)At k LHS RHS 0; (2) lim (LHS RHS) oo; (3) V RHS) 2 Therefore, LHS - RHS is a positive non-decreasing function in for k 2 1, i.e. LHS 2 RHS. Finally, we show that the following result is true,
k00
E arccos ( (<o (oF(O)) of (0)) + =)) (10( I (0,) + + ) E arccos
(sF (1(0)) x)) 27 (I (0)) + 24 )
Let X denote the random variable inside the arccos function, in this case X follows the same distri- bution as the noise, X ~ (0, a((TT4))))) since for each fixed of the sources of randomness comes from the addition of random noise. Let x be sample of X and let x and x denote the function calculating the quantity of x before and after freezing,
(< (0)), of (0)) + 21)
x² (0)) of (0,)* + xe) |1 21
1 (0)()))))) ||| (0)) =ell
Since a² 2 n,, arcoos (x=)² < arccos (x²)² therefore,
arccos (za) f(z)dxx 2 S 1/ arccos ( f(x)dx,
where f(x) = (1/ 2 exp (- (1/2)) u)/o012), 0, 02 1d is the probability
density function of X. Therefore (n BP 3 a 1E( BP
D EXPERIMENT DETAILS
We implement our method and the CNN baseline model (Papernot et al. 2021 with JAX (Bradbury et al. 2018). The MNIST model is run using 2-layered CNN model, FaShionMNIST and CIFAR- 10 are run using a 5-layered CNN model. The architecture is the same as in Table and in Papernot et al. (22T). The
baseline model is from Balle et al. (202) and implement the layer freezing part on top of it. We mostly adopt the hyperparameters suggested in the original paper. Below the details for each experiment:
14

---

Under review as a conference paper at ICLR 2023
(1) MNIST, 2-layered CNN: C' 1.0, DDP = 1.923, B 2048, 17 2.0, activation function is tempered sigmoid with scalc=1.58, inverse temp=3.0, offset=0.71 as suggested in the original paper. When using layer freezing, we freeze the first 2 layers after epoch 3, 17, 39
for e = 1,2,3 3 respectively.
(2) FNSHi 5-layered CNN: C 1.0, ODP 2.15, B 2048, 7 = 4.0, activation function is tempered sigmoid with scalc=1.58, inverse temp=3.0. offsett0.771 suggested in the original paper. When using layer freezing, we freeze the first layers after epoch 4, 19, 40 for c 1, respectively.
(3) CIFAR10, 5-layered CNN: For the = 3 experiment we used C 3.0, DD 1.0, = 512, 17 = 0.15, activation function is tempered sigmoid with scale=1.58, inverse temp=3.0, offset=0.71 as suggested in the original paper. We freeze the first 3 layers after epoch 20. For the = experiment we used C = 1.0, ODp = 1.47, B = 2048, 17 = 4.0, We freeze the first 3 layers after epoch 75.
(4) CIFARIO. Wide-Resnet: We adopt the publicly available configuration file in Balle et al. (2022). We freeze the first 2 convolution groups after update step 800 and 3000 for = 1,2 respectively.
E EMPIRICALLY EVALUATING ASSUMPTION
3.1 AND 3.2
In this section we empirically examine the assumptions in Section 3.4 We observe that Assumption 3.1 is generally well-supported empirically as the sum of clipped gradients are closer, in terms of having smaller angle, to the true gradient direction V ft than the sum of unclipped gradients. We also observe that Assumption 3.2 is generally not true since freezing rescales [19()))] thus moving some per-sample gradient g(x;) from being clipped to unclipped. However, since only small amount of g(x;) are moved, the directions of the summed clipped and unclipped gradients after freezing are quite aligned with the directions before freezing.
Intuitively, if we clip less by freezing a subset of the parameters, under Assumption 3.1, we assign a larger weight the clipped gradients which are more aligned with the truth so that we decrease the bias. To verify Assumption 3.1, in each iteration we record the clipping status and compute the sum of the clipped (those with [| [x:(=:)] 2 C)) and unclipped (1lg((z))) < C) gradients, and compute the distortion angle (as in Definition 3.2) to the true gradient fi computed on all training examples. Figure shows the results with different L22clipping norm C run with DP-SGD, CIFAR10, ODr = 1.0. batch size B = 512 and learning rate 7 0.15. At the end of training, e 7.0 and the test accuracy is 0.42, 0.58, 0.60, 0.58 for C = 0.1, 1, 5, 10 respectively. We observe that in general the summed clipped gradients gradients by having a smaller angle for different Os which empirically supports Assumption 3.1 It matches with our intuition since there are often more gradients being clipped than unclipped throughout training, given a reasonable range of C from 0.1 to 10 as suggest by the Opacus authors Yousefpour al. 2021), and these per-sample gradients with larger magnitudes more likely to dominate the true gradient direction.
more aligned with the true gradient than the summed unclipped
Assumption 3.2 is used to characterize sample gradients that change clipping status due layer freezing, from clipped to unclipped. Figure ( shows the change in the number of clipped and un- clipped gradients. We see that a small number of gradients does change their category as some data points' gradients go from being clipped to unclipped when rescaling [lg(x:)]l. Referring to the demo in Figure Assumption
that the directions of (xy)) and ge (*x) (the summed unclipped and clipped gradients respectively) do not change after freezing so that the observed change Yu B is mostly due to rescaling the gradient norm which the analysis proceeds. We empirically measure the change in angle before and after freezing, for the sum of gradients in the clipped and unclipped category. We observe that the change in the angle of the summed clipped gradients are 0.09 and 0.11 radians, and 0.10 and 0.12 radians for the summed unclipped gradients, when freezing the first 3 layers of parameters after epoch 20 and 40. This matches our intuition that the direction of the sum of gradients over minibatch is stable to changing small number of medium-sized gradients from the elipped to the unclipped category.
15

---

Under review as a conference paper at ICLR 2023
Angle between True Sum of Clioped/Unciioge Gradients
Clipped Unlipped
2.5
1.5
Clipped
Unllpped
1.0
0.5
L 4
100
200
300
400
500
100
200
300
400
5000
2.5
Clipped Unlipped
2.0
1.5
Clipped
Unllpped
1.0
0.5
0.0
100
200
300
400
500
0.0
0
160
200
360
400
500
Training Step
Figure 7: Comparing the angle between the true gradient and the sum of clipped and unclipped
gradients for different C. Top-Left: C 0.1, Top-Right: C 1.0, Bottom-Left: C 5.0, Rightm--Rgght: C 10.0. Missing values indicate at step / all the gradients are clipped. We observe that general the clipped gradients are more aligned with the true gradient by having smaller angle across C.
Number dipped/Unciipppe
Per-samlle Gradients
512
448
Nurbee of Clipped/U neclippe
Perss Ter-sample le Cradients
448
512
384
384
320
320
Clipped
Clipped
256
Unclipped
256
Unclipped
192
192
128
128
64
64
1000
2000
3000
2000
Training Step
Training Step
Figure 8: The number of clipped and unclipped gradients when freezing after Left: epoch 20 and Right: epoch 40. A small number of gradients changed status from being clipped to unclipped after freezing.
16


Reviews:
[{"review_id": "q-PVSwrsQe", "reviewer": "Reviewer_z6fv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the effect of freezing parameters after certain iterations of differentially private training, both theoretically and empirically. Theory shows the convergence could be improved under certain situations and experiments show freezing layers closer to input (bottom layers) after training for a while could improve performance after DP training.\n\n", "summary_of_the_review": "Overall I feel the paper studied an interesting problem/approach, with both theory and experiment support, but there is plenty room for improvements.", "strength_and_weaknesses": "Strengths:\n1. Freezing bottom layers during training is proven to be useful in DP training in a few other works, this paper provides some theory support of the approach.\n\n2. The decomposition of bias caused by clipping from an angular perspective is new in the community, to the best of my knowledge.\n\nWeaknesses:\n1. It is unclear whether the bias caused by clipping in Lemma 3.1 is a tight characterization or not, if the authors could provide a case when the bias is achieved, it will make the theory more convincing.\n\n2. The experiments are only for relatively simple architectures and datasets, if the authors could verify that freezing lower layers will benefit more variety of tasks, it would strengthen the contribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Isi-U4Wpk-", "reviewer": "Reviewer_PEmL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the DP-SGD with a set of fine-tuning methods, which freeze some hidden layers in a neural network. The authors claim that their method can (under some strong assumptions) improve the utility under the same privacy budget, which is verified on small image datasets.", "summary_of_the_review": "I recommend heavy revision of this work before publishing.", "strength_and_weaknesses": "Strength: The paper is clearly written and the algorithm is carefully verified, from both theoretical and empirical perspectives.\n\nWeakness: The major flaw is in the novelty which is explained in detail in the following section. Another major flaw is the scope of this work: notice that DP is a systematic effort, covering different clipping function (e.g. recent DP-NSGD or automatic clipping still work with layer freezing, but not mentioned; also per-layer clipping will likely violate Assumption 3.2; but this work only considers the common gradient clipping in the flat clipping style), fine-tuning methods, tasks (this work only experiment on tiny images and not consider language tasks), optimzers (only experiment with SGD, not adaptive optimizers which have been used to achieve SOTA on CIFAR10), etc. Therefore, this work really studies a very narrow perspective of DP deep learning.\n\nBecause of the narrow scope of this work, I would expect to see better analysis or empirical improvement. However, the improvement is insignificant (about 1% within the incomplete Table 1), and the analysis relies on too strong assumptions that are not empirically verified. In fact, it shouldn't be hard to print out the percentage of per-sample gradients that are clipped/not-clipped before and after freezing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5xCAew4g3AF", "reviewer": "Reviewer_Hfmj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to do layer freezing for DPSGD training. It analyzes the properties of different layers in the training process, provides some theoretical analysis on the affect of freezing, and shows the effectiveness of the algorithm in some real datasets.", "summary_of_the_review": "Some analyses seem interesting, while I'm a bit concerned about the novelty of the work.", "strength_and_weaknesses": "Strength:\nThe paper looks at an important problem, and provides some interesting empirical and theoretical analyses. I like the part about PWCCA specifically.\n\nWeakness:\n- Given the amount of work on layer freezing in private training (though in the pre-training / fine-tuning case, as has been mentioned in the paper), I'm not so sure about the novelty of the work.\n- The empirical improvement doesn't seem very significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject