Finding Private Bugs: Debugging Implementations of Differentially Private Stochastic Gradient Descent 
Authors: Congyu Fang
Hengrui Jia
Ali Shahin Shamsabadi
Nicolas Papernot
Venue: Submitted to ICLR 2023 (2023)
Keywords: DP
DP-SGD
debugging
model distance

Abstract: It is important to learn with privacy-preserving algorithms when training data contains sensitive information. Differential privacy (DP) proposes to bound the worst-case privacy leakage of a training algorithm. However, the analytic nature of these algorithmic guarantees makes it difficult to verify that an implementation of a differentially private learner is correct. Research in the field focuses on empirically approximating the analytic bound, which only assesses whether an implementation provides the guarantee claimed for a particular dataset or not. It is also typically costly. In this paper, we take a first step towards providing a simple and lightweight methodology for practitioners to identify common implementation mistakes without imposing any changes to their scripts. Our approach stems from measuring distances between models outputted by the training algorithm. We demonstrate that our method successfully identifies specific mistakes made in the implementation of DP-SGD, the de facto algorithm for differentially private deep learning. These mistakes include improper gradient computations or noise miscalibration. Both approaches invalidate assumptions that are essential to obtaining a rigorous privacy guarantee. 

Under review as a conference paper at ICLR 2023
FINDING PRIVATE BUGS: DEBUGGING IMPLEMEN- TATIONS OF DIFFERENTIALLY PRIVATE STOCHASTIC GRADIENT DESCENT
Anonymous authors Paper under double-blind review
ABSTRACT
It is important to learn with privacy-preseeeeee preservin algorithms when training data contains sensitive information. Differential privacy (DP) proposes to bound the worst-case privacy leakage of a training algorithm. However, the analytic nature of these algorithmic guarantees makes it difficult to verify that an implementa- tion of a differentially private Iearner is correct. Research the field focuses empirically approximating the analytic bound, which only assesses whether implementation provides the guarantee claimed for particular dataset or not. It is also typically costly. In this paper, we take first step towards providing simple and lightweight methodology for practitioners to identify common implementa- tion mistakes without imposing any changes their scripts. Our approach stems from measuring distances between models outputted by the training algorithm. We demonstrate that our method successfully identifies specific mistakes made the implementation of DP-SGD, the de facto algorithm for differentially private deep learning- These mistakes are improper gradient computations or noise mis- calibration. Both approaches invalidate assumptions that are essential to obtaining a rigorous privacy guarantee.
INTRODUCTION
Machine learning (MI.) models trained without taking privacy into consideration may inadver- tently expose sensitive information contained their training data (Shokri et al. 2017: Rahman et al. 2018 [Sog & Shmatikov 2019 Fredrikson et al. 2015). Training with differential pri- vacy (DP) (Dwork et al. 2014) emerged as an established practice to bound and decrease such pos- sible leakage. Because differential privacy guarantees are algorithmic, they require modifications to the training algorithm to obtain such a bound. This bound is also known as the privacy budget of the algorithm. Making the necessary modifications often do not have the DP expertise required to ensure that the implementation is sound and correct, and wrong implementations usually do not "fail loudly" (i.e., they do not block training, nor lead to obvious differences in terms of the performance of the trained models).
be challenging because practitioners
In this paper, we approach this problem through testing practices. We focus on the canonical DP learning algorithm, which is the differentially private stochastic gradient descent (DP-SGD) (DP-SGD)Chauud Chaud- huri et al. 201T) Abadi al. (201)). Established research in the field has considered testing this algorithm but only from an auditing perspective with an external party, e.g., a regulator. Their ap- proach is to interact with the implementation of DP- in black-box fashion empirically verify the privacy budget achieved by the algorithm, e, is the one claimed by its developer (Jagielski et al.) 2020 Nasr et al. 202T: Tramer et al. 2022). get attributed the specific mistake(s) made in the implementation. Instead, it simply informs us if an implementation is correct or not.
important to note that any discrepancy does not
As we introduce our framework for testing implementations of DP-SGD identify common fail- ures, we adopt the perspective of the developer themselves. This is orthogonal and, in fact, comple- mentary to prior work on auditing the privacy budget of DP-SGD implementations. Once prior work has identified an incorrect implementation, our framework can be used to help identify the source of the discrepancy. We two key cases where this would be beneficial for developers: (1) when

---

Under review as a conference paper at ICLR 2023
they integrate existing implementation of DP-SGD in their ML pipeline; and (2) when they write their own implementation of DP-SGD from scratch. While the usefulness of our method is apparent in the latter application scenario, even using an off-the-shell DP-SGD implementation is not as triv- ial as it may initially sound. This is due to the fact that open-source inplementations of DP-SGI) still nascent and do not always support all modern architectural features of MI. pipelinnss
Since the outputs of DP-SGD are updates of model parameters, incorrect implementations of should manifest themselves through differences in the parameter space of the trained ML models. However, the parameter space is typically high-dimensional and the nature of non-convex optimiza- tion makes difficult evaluate whether the solution yielded by a particular DP-SGD implemen- tation is correct. Therefore, we propose to identify and approximate such differences by evaluating functions defined in the parameter space of the models, or by comparing the models' losses.
Note that the DP-SGD algorithm makes three key modifications to vanilla SGD to obtain differential privacy, each of which represents a potential failure point in its implementation. First, gradients are computed on per-example basis. This enables the algorithm to isolate the contribution of each training example. Second, the norm of these per-example gradients are clipped. This bounds the sensitivity of the algorithm to each individual training example. Third, gradients are noised before they are applied to update the model. This provides the indistinguishabiity across updates needed to obtain differential privacy.
We introduce a methodology to identify common implementations mistakes for each of these modi- fications: (1) We observe correct gradient clipping restricts the impact of some training steps on the model and thus the change in loss values caused by such steps. However, this may not be the case gradient clipping performed incorrectly or not performed at all. Therefore, we leverage changes in loss values to test correctness of gradient clipping; and (2) Having the noise improperly calibrated to the sensitivity of the training algorithm often results in insufficient indistinguishability in the pa- rameters to obtain differential privacy. Thus we detect improper calibration of the noise added to gradients by measuring the parameter-space distance between models obtained. Note that both of these tests can be performed on checkpoints outputted by the ML pipeline without modifications on the training scripts, which makes them easy to deploy universally.
We validate our approach on standard image classification tasks (i.e., ResNet-20 and Wide ResNet50-2 architectures trained on CIFAR-10 and ImageNet datasets, respectively) and an NLP task (i.e., Bert model trained on the Stanford Sentiment Treebank v2). Our proposed method is able to detect the three common implementation mistakes we highlighted carlier, namely when: (1) the gradient is evaluated using aggregate the mini-bates of data points before clipping; (2) the updates are computed with no clipping; and (3) the additive noise is not calibrated.
In summary, tests help identify implementation mistakes that developers may not be aware of, and that would otherwise invalidate the privacy guarantees claimed. Our conceptually simple, computationally efficient, dataset-agnostic and model-agnostic tests detect common mistakes in the implementation of gradient clipping and additive noise:
We characterize the effect of per-example gradient clipping, mini-batch gradient clipping, and no gradient clipping gradient updates computed by DP-SGD. Based on analysis, we design a test that detects incorrect gradient clipping by either varying the mini-batch size the gradient norm bound which clipping is configured to enforce.
We thcoretically demonstrate that the parameter-space distance between models trained with DP-SGD is a function of the scale of noise added to gradients. From this, we obtain a test that detects incorrect noise calibration by varying the value of the gradient norm bound.
We demonstrate, through extensive experimentation, that developer can run our tests on their implementations to detect incorrect gradient clipping and incorrect noise calibration in both image and text domains without changing their scripts.
2 PROBLEM DESCRIPTION AND RELATED WORK
Differential privacy (Dwork et al. 2006 2014 (DP) is the gold standard to reason about the privacy guarantees of learning algorithms. A randomized algorithm A is (e, -iiffeerrntlllly private if its outputs for any S E Range( and any two neighboring datasets D and D' that differ only by one
2

---

Under review as a conference paper at ICLR 2023
record are statistically indistinguishable:
Pr(A(D) E ES S =f +
Machine learning models can be trained under the framework of DP using output perturbation (Wu et al. [0117] Zhang et al.] 20177, objective perturbation (Chaudhuri et al. 2011] Lyengar et al. 2019) or gradient perturbation (Bassily et al. 2014 Abadi et al., 2016) methods. In this work, we focus on the gradient perturbation approach, specifically differentially private stochastic gradient, DP- SGD (Abadi et al., 2016). is known as the de facto differential private learning algorithm (for
deep learning). In order provide DP guarantees, DP-SGD imposes three modifications to vanilla SGD (highlighted red in Algorithm TD:
Per-example gradient computation. DP-SGD computes the gradient of the loss function with respect to model parameters for each individual example separately to isolate the influ- ence of each training data point the training algorithm's 's output (line 5 in Algorithm I Note that there is no a priori bound on per-example gradients.
Per-example gradient clipping. DP-SGD thus clips the per-example gradient to fixed norm C, to bound the sensitivity of the gradients to each individual training data point (line in Algorithm D
Calibrated noise addition. DP-SGD adds noise to the aggregated clipped gradients before they are applied to update the model parameters. The noise is scaled by the magnitude of C' and noise multiplier a (line in Algorithm
Implementing incorrectly or omitting the above modifications invalidates the guarantee of differen- tial privacy. For example, developers are used to computing gradients for mini-batches of training examples when implementing vanilla SGD as opposed to per-example gradients, i.e., developers may tend only obtain the aggregated gradient update. This course of action can cause mistakes in the implementation of per-example clipping by either performing the mini-batch aggregation before addition to this, developers, especially those
gradient clipping or missing the clipping altogether. who do not have DP expertise, may forget to scale the noise by C' thus implementing uncalibrated noise, thereby rendering the calculation of independent of C.
Current approaches to verifying the correctness of a DP-SGD implementation rely on privacy au- diting: an attack is designed to obtain an empirical lower bound on the privacy budget of the im- plemented algorithm (Nasr et al., 2021 Jagielski et al. 2020 This, however, does not identify the source of the mistakes made in the implementation; it simply demonstrates whether ML mod- els trained with DP meet the privacy guarantee claimed by the framework. Furthermore, they computationally costly as many models need to be trained as part of the attack to ensure statistical validity of the results (Nasr al. 2021 Jagielski al.) 2020; Tramer al.) (002).. For instance, Tramer et al. (2022) trained 100,000 models on MNIST. These heavy computations limit their ap- plicability simple tasks and datasets. Finally, it should be noted that expertise in both ML and DP is required to design the attack and, c.g., create poisoned samples required for privacy audits.
3 TESTING IMPLEMENTATIONS OF DP-SGD
We address the issues identified in Section by introducing conceptually simple and computation- ally efficient tests enabling developers to detect the source of mistakes in their DP-SGD implemen- tations. We show that these implementation mistakes manifest themselves through differences the dependency of gradient updates to re-related hyperparameters. Therefore, changes to training scripts. Instead, our method only requires one run of the training scripts while setting the value of DP-related hyperparameters such as C, B, and 8 particular values. In details, we make the three following assumptions:
do not impose any
1. No real data: the developer should not need real data points for running tests, they should also be able to use synthetic data points.
2. No privacy risk: any gradient computation or model trained as part of the proposed tests does not need to be released; put another way the sole purpose of these tests is to assess the
'The privacy budget is computed as function of the mini-hatch sampling rate, the number of training steps, and the noise multiplicr.
3

---

Under review as a conference paper at ICLR 2023
correctness of DP-SGD implementation. This means the approach does not increase the risk of leakage of private information when these tests were executed using sensitive inputs from real data.
3. No changes: tests should not impose any changes to training scripts, and they should be easy and light to run.
Algorithm DP-SGD
Require: Dataset D, Mini-Batch Size B, Gradient Norm Bound C, Noise Multiplier o, Model Parameters W, Loss Function L, Learning Rate T)
1: Wo f -Raadoonnnniillzztiinn
2: for t +- 1....... T do
D
Training Steps
3:
MiniBatch +
4: for b 4 1,.... B do
D Iterate over every data point in the mini-batch Per-example gradient calculation b. Per-example gradient clipping
5:
Mini MiniBatch(b) 9b 9b/ max |1 1) 1 [..g. + N/0 (C)
9b
6:
7:
g
13
Add calibrated Gaussian Noise
8:
W, + Wi-1 179
9:
return Wr
3.1 DETECTING INCORRECT GRADIENT CLIPPING USING LOSS VALUES
Clipping per-example gradients to fixed norm C' is essential step in DP-SGD to bound the influence each data point on the final gradients. Two common mistakes in implementing this gra- dient clipping operation include 1) no gradient clipping: forgetting perform any sort of gradient clipping; and 2) mini-batch gradient clipping: aggregating gradients of all data points within each mini-batch first and then clipping the aggregated gradient, instead of clipping the per-example gra- dients first and then aggregating them. Next, we analyze differences caused by such mistakes and describe how capture them in test.
Let {(x, E RV )= be a mini-batch of B data points with per-example gradients {9: for the model M. We the noise multiplier to (i.e., disabling the addition of noise the gradients in DP-SGD) to isolate the effect of gradient clipping. The private gradient, i.e., gradient after has been modified by DP-SGD and aggregated, computed by correct and incorrect implementations of gradient clipping would be:
Σ, (90/ max(C~
Per-example gradient clipping, No gradient clipping, Mini-batch gradient clipping.
Private Gradient
Σ 96
1 9b) B
/max(C Σ 90l12, 1)
Note that would like our test to be implementable without having make modifications to the training script, or needing to inspect the (private) gradients directly. Instead of comparing gradients directly, thus instead leverage the change in loss values:
Change in loss values Loss(M + optimizer(Private Gradient)) - -LossMM,
which is a function of private gradients.
To capture and highlight differences among these 3 types of gradient clipping in practice, we care- fully set the value of hyperparameters, which are only used for computing private gradients but not the rest of Equation 3
This ensures that each hyperparameter yields different changes in loss values.
Detect no gradient clipping. Comparing the first line and the second line of Equation demon- strates that the private gradient with no gradient clipping (and thus the changes loss values) is always independent of C. Conversely, the private gradient in the per-example gradient clipping is a function of C when clipping is effective i.e., C < |[ Therefore, our test carefully varies the value of C and detects no gradient clipping based on the differences between the changes in loss values across different C. As shown Figure gradient clipping are invariant to changes in C. At the same time, increasing C' increases changes in
T (left plot), the changes loss values caused by no

---

Under review as a conference paper at ICLR 2023
loss values in the case of per-example gradient clipping until the gradient norm bound C' becomes larger than the unclipped gradients (i.e., the clipping operation becomes a "no op").
Detect mini-batch gradient clipping. When clipping is effective, the model update in per-example gradient clipping (first line of Equation 2 is different than the model update in mini-batch gradient clipping (second line of Equation 2 This is because clipping occurs before the aggregation in the former, while the clipping occurs after the aggregation in the latter. To ensure that the clipping operation does not become a "no op" for both per-example gradient clipping and mini-batch gradient clipping, our test proposes
the effective clipping as:
C < min(1l99112); Vgb # 0
and C E 96l12.
1 1
(4)
Combining Equation 2 and Equation
we have:
mh 1o0ll2
Vgb +
Per-example gradient clipping.
Private Gradient
B O
(5)
Mini-batch gradient clipping.
In order to (I) ensure the intersection of ranges of C in Equation is not empty; (2) have control over zero and non-zero gradients; and (3) to cancel out the effect of other elements, except B, on the differences between their model updates, we synthesized mini-batch of data points in which all data points have zero gradients except one. Our data synthesizer receives a mini-batch size B and a feature dimension N as inputs, and creates {(rr E IR ..)) where (IB,9B) is the data point with non-zero gradient JB such that || > C. and {( /= are the data points with zero gradients. It is worth to note here it does not affect our approach as to what the features (i.e.. { of the inputs are. Instead, the former is achieved by setting yB = -MM (IB) where a >> (e.g., a 10) to have non-zero per-example gradient with a larger norm than C,, thus ensuring the presence of clipping: and the latter (i.e., data points with zero gradients) is achieved by setting the labels of the other - 1 data points such that {y, = argmin,,. ", L(y,, M (zx) B-1 so that the loss values (and subsequently the gradients) of this set of data points with the labels are minimized. Since the loss function is a mapping from output space of the model (which usually low-dimensiona) scalar, solving y analytically or numerically should be easy. For commonly used loss functions like cross-entropy loss and mean square error, the solution would be {y, M EZ
Returning Equation 5 we now have Øb 0 for all points in [ The only term left in each sum over b is the term that corresponds to the B-th data point. We thus have (see Appendix for detailed derivation):
Cg BI Cg
Per-example gradient clipping.
Private Gradient
Mini-batch gradient clipping.
liss la
Equation 6 demonstrates that the private gradients (and thus the changes in loss values) in mini- batch gradient clipping is independent of the mini-batch size B while increasing B decreases the magnitudes of private gradients in per-example gradient clipping. By leveraging this observation, our test varies the mini-batch size B of the synthesized mini-batch to detect mini-batch gradient clipping as shown in Figure [] (right plot). It is expected that the changes in loss values in the case of mini-batch gradient clipping always has the same value irrespective of the mini-batch size B. In contrast, magnitudes of the per-example clipped private gradient decreases as B increases since the averaging happens after clipping, smaller changes loss values are expected. Note that the synthesized data point with a non-zero per-example gradient ((xB, MB)) must be kept the same across runs with different values of B. This way, the values for C. (and (yllz)) are kept constant across different runs so that only the values for B in Equation 6 would vary as independent variables.
3.2 DETECTING INCORRECT NOISE CALIBRATION USING MODEL DISTANCE
Based on the actions we perform in the previous sections, we can detect common mistakes in the clipping implementation. Now, describe a test to identify mistakes in the calibration of noise added by DP-SGD to the clipped gradient. Recall from Section for noise to be correctly calibrated, the variance of the Gaussian noise added to the clipped gradients needs to depend the gradient
5

---

Under review as a conference paper at ICLR 2023
Oash
</>
Deonlopeta sezpp
Data
</>
Diootoper's acrape
C/s
manotorsh some
8
B
min-tootth mome
toboe multppeer
0
0
nolow rruftpller
Ohangeo loz valges
Mitibaash graclent clopiig..
graddett
Changes
C
C
Cappeng rorm
Rg
Bg
Mini-bbtchh size
Figure 1: Overview of our test. for detecting no gradient clipping (left) and mini-batch gradient clip- ping (right) in DP-SGD implementations. Observe how gradient updates with per-example gradient clipping overlaps with gradient updates computed without gradient clipping when the gradient norm bound is large.
norm bound C. Specifically, we are interested in distinguishing the following private gradients:
b 96/ max 11bb12 1) .N(,, (Co)²)) Calibrated noise, B 9b/ max C- 11 2, 1) + .N(0, of))
Private Gradient
Uncalibrated noise.
Ensuring that noise calibrated is more complicated than detecting incorrect clipping because the gradient norm bound appears appears in Equation in the term corresponding to clipping but also in the term injecting noise. We thus cannot simply vary the values for C' as would not be able to separate the effect of C on clipping from its effect on noise injection. Therefore, we first need to isolate the effect of the gradient norm bound C noise injection and eliminate its effect clipping. To do so, we set the value of C' to be arbitrarily large so that the gradient is always left unclipped. In this way, C' no longer comes as factor in the first term of the noised gradient computation (for both uncalibrated and calibrated noised gradients). Then, we vary the value of C' to test whether the injected noise is indeed sampled from Gaussian distribution whose scale Co rather than a. Put another way. by working with large values of C, we reduce Equation to the following:
factor in both operations. Indeed, the gradient norm bound C
E gb + NNO,, (Co)²) Es 96 + +N(00
Calibrated noise,
Private Gradient
Uncalibrated noise.
There is an additional difficulty we need to overcome in order to distinguish these two Gaussian noises (of scale Co and rr. Recall from 3.11ion33.1thha computed by the training algorithm. Instead, we would like to compare changes in the loss achieved by successive models outputted by the training algorithm. Because we are now studying stochastic computation (i.e., adding noise to gradients), the outputs of DP-SGD can no longer be directly compared. To understand the reason for this state, it is easy to see how repeating the same training step without changing the gradient norm bound could lead to a different Gaussian sample being added to the gradient, which then cause different changes in the loss values. However, we can turn to statistical testing to address this difficulty.
cannot directly observe private gradients
Let M1, Mr be pair of models that are trained with DP-SGD using the same initial- ization Mo and the same mini-batch of data points DB. Let us assume that the noise sampled independently from the same Gaussian distribution (O s²1 x) and added to their gradients G and G2 where 1 the identity matrix and K the dimension of the model's parameters. The parameter-spen li-distance of Mi and M2 depends the scale of the distribution of the noise added to their gradients:
Theorem 1.
E[II 2011-Malll2
M2 x
Proof. Without loss of generality, we assume the aggregation method for the optimizer is take the mean over the mini-batch, and that the optimizer is SGD. After one iteration of DP-SGD:
G G + (100
G2 G NN0, 21k)
(10)

---

Under review as a conference paper at ICLR 2023
where G is the aggregated per-example clipped gradients calculated using DB.
Therefore, AG = G1 - G2 ~ 2N s²1k).
| En x (A Σx 2s²xx 25 2 where x is the chi- chi-squared distribution with degree of freedom K. | VIAGII V2sXK, where XK is the chi distribution with
degree of freedom K. That gives AG|2|] = /2sE[xk].
Therefore, given Mi- Mo G and M2 Mo n/G2, E[IIM, M2ll] E|vlG - G2ll
That is stating that if we repeat the DP-SGD training script multiple times and obtain multiple mod- els M,'s the parameter-space Ig-distance between each pair of the models would have expected value that is dependent on the scale of the noise. Therefore, by having multiple trained models, we are able to empirically estimate the expected value for the model distance by taking the mean over the model distance values of each pair of models.
We can substitute s = Co for the calibrated noise and = a for the uncalibrated noise. This implies that if we fix the value for the noise multiplier o, the expected value for the parameter-space ly-distance between models trained with DP-SGD should be dependent on C' for the calibrated noise. Conversely, the expected distance should be independent from C' when the noise is not calibrated. This gives us a test that we can use to distinguish calibrated and uncalibrated noised gradient computation
The detailed procedure of our test is as follows: (1) the developer first selects model architecture and a set of data points (that could be real or synthetic); (2) the developer then picks a range of values for the gradient norm bound, C' E [C1, C2.... J (that ensures the gradients are not clipped) well as the noise multiplier a. Heuristics for selecting C and a are discussed in detail in Appendix [ Then (3) the developer runs the train script multiple (e.g., 5) times with C = C1 for a few iterations (e.g., =10 and stores the final checkpoint received at the end of each training run M,, .,...., MM5- They then compute the parameter-space distance between each pair of the models; (4) repeat step for the rest of the values chosen for C: (5) plot the pairwise parameter-space distances with respect to C:: and finally (6) compute the slope and run regression t-test for the slope with null hypothesis of slope = O. If the p-value is small (e.g., p-value < 0.05), we can reject the null hypothesis and claim the noise is calibrated. Also, to ensure there no false positive (i.e., the non-zero slope not caused by calibrated noise, but rather an effect of clipping), the developer should repeat steps 3 to 5 with
ensure the slope of pairwise distance versus C for the o 0 case is zero, which suggests that the C' values chosen have a minimal effect on training.
4 VALIDATION
Our tests are designed identify mistakes in the DP-SGD implementation. Thus far, DP-SGD has been used in the vision (Abadi et al.] (016) and text (Dupuy et al.. 2022) domains. As we demon- strate in this section, our proposed method can detect incorrect gradient clipping and uncalibrated noise addition in these domains. We consider three common models (ResNet20 (He et al.) 2015b), WideResNet50 (Zagorukko & Komodakis 2016). and BERT (Devlin al. 2019) and three datasets (CIFAR-10 (Krizhessky) (009). ImageNet (He et al. 2015a). and SST2 (Socher et al.) (013)) (for implementation details, see Appendix D. In addition to DP-SGD, we also evaluate our method on Dd-Adam to demonstrate its general applicability. Indeed, the Adam optimizer leverages additional
techniques like momentum and per-parameter learning (Kingma & Ba 2015). All of the experi- ments are repeated 5 times on 5 different machines to obtain confidence interval. We analyze the computational cost for the proposed methods in Appendix D
A note on Adam. We need to pay special attention when the Adam optimizer is used for training because it normalizes private gradients before they are applied to update the model. For the first training step of Adam, the private gradient would always be normalized by itself since the stateful optimizer is initialized with states of O. This would result in similar model updates regardless of the difference in private gradients, making it hard to observe any differences between clipping cases
Note that we do not assume have access to the raw/private gradients. Thus, we are not able to directly statistical test check for the distribution of the additive noise. See details in Appendix C
7

---

Under review as a conference paper at ICLR 2023
Change values
Cipptz Nacippping
Change in too valice
Cippei NoCilpingg
Changes 3oss salos
1n-
9.
94
0.4
0.0
16
10
10
10
16"
10
16" 16"
Cextiant Norm Bound (C)
Gcaiion Noan Bound (C)
Gradian Norm Bound
(a) CIFAR-J0 (ResNet20)
(b) ImageNet (WideResNet50-22)
(c) SST2 (BERT)
Detecting the absence of gradient clipping. We train model for 1 step to compute the change the loss values and plot with respect the gradient norm bound while the noise addition is turned off (by setting = 0). A clear difference can be observed between the two curves in all the subplots: the loss change remains constant when there no clipping, whereas varies lot when clipping is applied.
Figure 2:
Change in yaleg
0.10
Piscurpi cispennciiapaa..
0.08
Clangs in Ialog vabe
Mitihash gualieea
016-
Perooupple dlppeoooolp.......
0075
Misilest gailer cipping po- Pereoample gudurcloppaa.. BA0
Clange la
0.
Minskash grader cipping PuPs
km
0m
0623
26
Mitt-bbtcch
Mani batch
Mini batch
(a) CTFAR-10 (ResNet20)
(b) ImageNet (WideResNet50-22)
(c) SST2 (BERT)
Figure 3: Differentiating per-example gradient clipping versus mini-batch gradient clipping. We synthesized a mini-batch of zero-gradient data points along with one large-gradient data point, and then compute the loss changes when the model is trained on these data points while increasing the mini-batch size. regression t-test is applied to each curve to test whether the slope is (p-values are included in the legends). It is consistently shown that the per-example gradient clipping leads to p-values that are always smaller than the 0.01 significance level, whereas the p- pavllu of mini-batch gradient clipping are always larger than 0.99.
discussed in Section 3.1 To mitigate this problem and make loss comparison meaningful again, we train the model for a few iterations to make the states of the Adam optimizer non-zero. Another issue with Adam is that as the additive noise eventually begins to dominate the real gradient signal, the private gradient and its running momentum will be indistinguishable from noise. To prevent this failure mode, we work with small Gaussian scales when testing DP-Adam implementations. More details the nuance of Ad-A is provided in Appendix
4.1 OUR APPROACH DETECTS INCORRECT GRADIENT CLIPPING IN DP-SGD IMPLEMENTATIONS
We first present empirical results demonstrating the effectiveness of our proposed method when comes to debugging the correctness of a gradient clipping implementation. Unless otherwise specified, the noise multiplier is set to 0 for all the experiments in this subsection to eliminate the impact of noise added to the gradient.
Clipping versus no clipping. To verify gradient clipping implemented in training script or not, we use the script to train the model for one single step with different gradient norm bounds, and com- pute the change in the loss value caused by this training step. Figure 2 (and Figure 5 Appendix G for the Adam optimizer), show that changes in loss values are independent of C' if there is no clip- ping. contrast, in the presence of clipping, the loss changes vary until C' is larger than the norm of the gradient. At this point, clipping no longer applied: the two curves overlap with each other. Therefore, our test successfully detects the lack of gradient clipping in DP-SGD implementations. We set the mini-batch size to so that per-example gradient clipping and mini-batch gradient clip- ping are equivalent.
Per-example gradient clipping versus mini-batch gradient clipping. After confirming that the DP-SGD implementation correctly uses some form of gradient clipping, the next logical step is to confirm that gradients are clipped on a per-example basis rather than at the level of the mini-batch aggregate. Recall from Section 3.1 when using synthesized data points to train the model, the
8

---

Under review as a conference paper at ICLR 2023
3006
Modal Pesncr Diazzee
No Naac E.2 Conret Callratiee p-ddd! Wining Nose Calbertiee,
No NoNoo -225
Camu Co g.
Por Pommrnttee
Winng Calberies
-08-
Wretp Netu Catheries
2006
266
460
1658
3003
4003
4003
1000
1066
04
1n
Gualion Naro Boxnal (C)
Grodient Nocea Bound
Guodient Nora Bound (C)
105
(a) CIFAR-10 (ResNet20)
(b) ImageNet (WideResNet50-2) Per-example gradient clipping
(c) SST2 (BERT)
Per-example gradient clipping
Per-example gradient clipping
Ninac p 6:90
3066
Malis p 2-3
Moadd Puaaette Strance
430
Ny Noioee, g
Caneet Notee Caltration, p-
Wirnng Neacc Calierston,
2000
Care Maae Calhortee, p
Wineg Neit Callhoriin 23-594
220-
Wrazg Notes Cadbeazes 0-0
Comra Naau cittemm
220
Masid
Gradien
(d) CIFAR-10 (ResNet20) Mini-bat Mani-batchee gradient clipping
(c) ImageNet Ne11522)) Mini-batch gradient clipping
(f) SST2 (BERT) Mini-batch gradient clipping
Verifying correctness of noise calibration for both per-example gradient clipping (first row) and mini-batch gradient clipping (second row) cases. We train multiple models for 10 steps and plot the pairwise model parameter distance with respect to the gradient norm bound for the three scenarios shown in the figure, respectively. p-values of regression t-test with null hypothesis of O slope for the three curves are also reported. Observations: (1) model distances in both wrong noise calibration and noise addition are independent of gradient norm bound; (2) the curve for correct noise calibration has non-zero slope, which allows us to differentiate it with the wrong noise calibration: and (3) our proposed method effective in detecting wrong noise calibration no matter whether the clipping is implemented correctly or not.
Figure 4:
private gradients for mini-batch gradient clipping would be independent of the values of B. Yet, for per-example gradient clipping, the private gradients would depend on the mini-batch size. This is confirmed by Figure for all three datasets where the changes in loss values plotted with respect to different mini-batch sizes (0, 100). We also report results for the Adam optimizer in Figure 6 in Appendix G One can see that the amount of loss change in per-example gradient clipping decreases as the mini-batch size increases. This is intuitive given that magnitudes of the private gradients for the per-example gradient clipping case would decrease / increases, as shown in Equation 6 On the other hand, the private gradients for the mini-batch would stay constant with respect to B, hence the change in loss values would also stay constant. In conclusion, our test successfully differentiates mini-bate batch from proper per-example gradient clipping in DP-SGD implementations.
4.2 OUR APPROACH DETECTS UNCALIBRATED NOISE IN DP-SGD IMPLEMENTATIONS
To detect whether noise calibrated according to the gradient norm bound C, we independently run the training script for 10 values for C. This should be repeated for two more models for each value of C' in order to compute the pairwise parameter-space distance among the models for each C as described in Section 3.2 In our experiments,
did this for five models reduce uncertainty when reporting results in Figure A (first row). Note that besides the correct and wrong noise calibration, we also included a no-noise case (i.e., training with = for reference. As expected, parameter distances are constant when the noise is uncalibrated (meaning the parameter distance is indepen- dent of C),, and vice versa. To quantify the independence, we applied a regression f-test to check if the slopes of the curves are 0. It can be consistently seen across the three datasets that the slopes are non-zero with more than 99% confidence (i.e., p-value < 0.01) when the noise correctly cal- ibrated, whereas the p-values are large when the noise calibration is missing. We also evaluated the proposed method when the implementation of clipping is not correct (see Figure 4 d-f and Figure 7 in Appendix G) show that our test of proper noise calibration does rely the correctness of clipping. addition, we repeated this experiment on the Adam optimizer (see Figure S in Ap- pendix G) and observed similar results, meaning our method applies optimizers beyond DP-SGD.

---

Under review as a conference paper at ICLR 2023
5 CONCLUSION
In this work, we proposed set of tests to debug implementations of DP-SGD. Unlike prior work, these tests are computational efficient and generally applicable. We are able to detect and identify two types of common mistakes that might occur during DP-SGD implementation, i.e., incorrect gradient clipping and improper noise calibration. Incorrectly computed private gradients are iso- lated based on an inspection of trained model behavior (e.g., through parameter-space distance, and comparisons of loss values).
Related work investigated the vulnerabilities of DP training algorithms introduced by back-end soft- ware. For example, Jin et al. (2022) studied two threats that lead to side channel attacks. Both are explained by failed implementations of primitives that enable DP algorithms to sample noise. They demonstrated that these flaws are due to floating-pon point representations. However, these flaws are parallel to this work they are not "bugs" caused by mistakes of the developers. These are indeed vulnerabilities introduced by libraries providing said primitives.
Our tests be deployed without modifying existing training scripts: they only rely accessing model checkpoints, We hope future work will extend our approach to debug DP guarantees of other algorithms such as Private Aggregation of Teacher Ensembles (PATE) (Papernot et al. 2017
ETHICAL IMPACT STATEMENT
Our work proposes tests that can be applied to debug implementations of DP-SGD, which is training algorithms that is designed protect differential privacy of training data points. Incorrectly implemented DP-SGD may lead to risks of privacy leakage. For example, one of the bugs that can be detected by our tests, mini-batch gradient clipping, can cause the privacy guarantees be weaker by a factor that equals to the batch size, which is usually 128 times or more. While many MI. developers start to take privacy into consideration, few of them are familiar with the implementation details of DP-SGD so that such bugs are common. Therefore, we believe research on debugging differentially private machine learning can solve this serious and urgent problem, and help developers to ensure and guarantee the privacy of their data providers, which is required by regulations such as the Gen- eral Data Protection Regulation in the European Union, the California Consumer Privacy Act in the United States, and the Personal Information Protection and Electronic Documents Act in Canada.
REPRODUCIBILITY STATEMENT
We will provide a link to an anonymous repository in the dicussion forums within the first week of the review process, and it will contain the code that can be used to reproduce our tests. After the review process, we will publicly open-source our code base along with documentations that explains how to use our tests and how reproduce the experiments described in this paper.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proceedings of the IEEE Symposium on Foundations of Computer Science (FOCS), Washington, USA, October 2014.
Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIT
10

---

Under review as a conference paper at ICLR 2023
2019, Minneapolis, MN, USA, Jume 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19- 1423. URL
Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky. An efficient DP-SGD mech- anism for large scale NLU models. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 4118-4122 IEEE, 2022. doi: 10. WWW.9//AAAASP4492222222299999999
URL mtttps://doi... /tt.:ttts////...rr/111....
ICASSP43922 2022. 200.992222222227966699 9746975
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryptography, pp. 265-284, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 5--------2222-55
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 1-407, 2014.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), Denver Colorado, USA, October 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learningg for image recog- 03385
nition. CoRR,
2015b. URL http: http::///xxxv..orr//bbbs.s.
Yerlan Idelbayev. Proper ResNet implementation for CIFARIO/CIFARIOO in PyTorch https: Accessed: 2022-09-28.
Roger Iyengar, Joseph P Near, Dawn Song. Om Thakkar, Abhradeep Thakurta, and Lun Wang. To- wards practical differentially private convex optimization. In Proceedings of the IEEE Symposium Security and Privacy (SP), San Francisco, California, USA, May 2019.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private private sgd? Advances in Neural Information Processing Systems, 33: 22205-202 22216, 2020.
Jiankai Jin, Eleanor McMurtry, Benjamin I. P. Rubinstein, and Olga Ohrimenko. Are we there yet? timing and floating-point attacks on differential privacy systems. In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pp. 473-488. IEEE, 2022. doi: 10. 1w..1099SS44221222229993333 SP46214 2022. 9833672
URL https: /tttp:::/dci..ree www.eoo...o.ogoooooo 1109/
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. | // wwwpiii.rr/aaa///o
6980
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866-882. IEEE, 2021.
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings, 2017.

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: http:::pppppps.pp.pppp/ wtwp::ssss..riii..cc.c..ooo
htt.://wwwwa.aaaaaaaaaaeaeeeeeaa pdf
ibrary.
Md Atiqur Rahman, Tanzila Rahman, Robert Laganière, Noman Mohammed, and Yang Wang.
Membership inference attack against differentially private deep learning model. Transactions on Data Privacy, 11(1):61-79, 2018.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In Proceedings of the IEEE Symposium on Security and Privacy (SP), SAN JOSE, California, USA, May 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. BRLhhtps:: www. www.wclwbb.... org/ ant wothhoogoooonaan /D13 1170
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery Data Mining (KDD), Anchorage, Alaska, USA, August 2019.
Florian Tramer, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing. arXiv preprint arXiv:2202. 122 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: State-of-thh--ar natural language processing. CoRR, abs/1910.03771, 2019. URL
http://arxiy> org ab9/ 03771
Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-basse based analytics. In Proceedings of the ACM International Conference on Management of Data, Chicago Illinois, USA, May 2017.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Gosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User friendly differential privacy library in pytorch. CoRR, abs/2109. 12298, 2021. bttst www.a.iiivv..o
abs
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016. York, UK, September 19-22, 2016. BMVA Press, 2016. URLhttp /ttt:///ww. bmva. org/ bmvc/ / 201 waap:///1///appeeeee
www.ppppr0////ee...eeeep Lndex.1. html
Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private ERM for smooth ob- jectives. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Melbourne, Australia, August 2017.
12

---

Under review as a conference paper at ICLR 2023
A COMPUTATION OF GRADIENT UPDATES
Using the data synthesizer, we get the following mini-batch of data points: [0, 0.... 0, Ol with
mini-batch size of B, ||gl|2 > 0. The clipping norm, C' is chosen such that 0 C < 11 When we repeat this for set of different mini-batch sizes B,'s, C' should be chosen to be < 1
B,
With per-example gradient clipping, the batch clipped gradients will be [0, O........ 0, C.g and C-g
Tal
the aggregate update would be the mean the clipped gradients, 73.1 With mini-batch gradient clipping, the aggregate gradient update would be #. After clipping, the aggregate update would be C-g which is independent of B.
g/B
C =
Bl
Nolle
B HEURISTICS TO CHOOSE C AND FOR NOISE CALIBRATION DETECTION
As described in the Section 3.2, tion, we need to choose the values for C that they are large enough such that the gradients are not clipped at all. In order to estimate the minimally required value for C, we performed the following steps:
eliminate the effect of clipping when observing the noise distribu-
1. For the chosen architecture and dataset, we trained model from random initialization for T' steps (in our experiments, we used T' = 1 when using the SGD optimizer and T = 5 when using Adam optimizer). Using this trained model as the initial state for all steps afterward.
2. Set = O. Train the model with the DP-SGD implementation for C E 00.01,0.1,) 1.0, 10, 100, 1000, 10000, 100000, 1000000) for 1 step and compute the change in loss for each value of C.
3. Pick C' = C if the change of loss values stop changing for all C > C
4. Repeat steps 1, 2, and for multiple times and use the maximum of selected C* value from all trials.
5. Sample 10 values for C 100*]
After selecting the minimal value of gradient norm bound, we will need to pick reasonable value for a. This is important because if is too small the effect additive noise might not be observable; yet if a is too big, (i.e., the additive noise dominates over the real gradient signal), this will cause optimizers like Adam to normalize every element of the gradient update to be +/-1. Therefore, in order to final the optimal value for during the test, propose to perform the following steps:
1. Using the model state generated after the first step when testing for optimal C' value.
2. Set C = C². Train the model with the DP-SGD implementation for a E [1 x 10 10-8,11 x
10 7, 1 10-6 ..... 1............. 1.0, 10.0] for 1 step and compute the change in loss for each value of
3. Pick = o* if the change of loss values starts to change for all > o².
4. Repeat steps 1, 2, and for multiple times and use the minimum of selected a* value from all trials.
C KS TEST
should be noted that the Kolmogorov-Smimae (KS) test could be used detect noise calibration if the optimizer is only vanilla SGD. KS test is -on-aaametriic test that can estimate if the two distributions are the same. We can train two models with same initial weight Wo and their own copy of DP-SGD optimizer initialized with the same state, i.e., momentum term would be zero for the first step of SGD. For the first model, we train one step to get M1 with noise multiplier of 0, i.e., no noise, with mini-batch with the size of B and learning rate of n. For the second model with non-zero noise (i.c., o * 0) and the same B, 17, and the data points to reach M2. In this setting, M2 Mj would be exactly the noise we added (scaled by B and 1)). For the correctly calibrated
13

---

Under review as a conference paper at ICLR 2023
noise, M2 M1 B. Y.((0, (C.) We can KS test to see if M, M2 follows the expected distribution and scale. However, this approach only works on a SGD optimizer. For another type of optimizer (i.e., Adam), the model update includes other calculations such as normalization. In this case, we are no longer capable of using a KS test to detect the distribution of the additive noise.
D COMPUTATIONAL COST
Check for Existence of Clipping. The total number of steps required to complete the test is
# training steps x # Gradient Norm Bound. We used training steps for 8 gradient norm bounds so the total cost is equal training for 8 steps. For reference, CIFAR-I0 with batch size 100 needs 500 steps per epoch. So the cost of checking minimal.
Check for Per- Per-Example Gradient Clipping. The total number of steps required to complete the
test is # training steps x # mini-batch sizes. We use I training step and 100 different batch sizes. So the cost is similar to 100 steps of normal training (i.e., 0.2 epochs of normal training).
Check Noise Calibration. The total number of steps required to complete the test is # models x
# training steps per model # Gradient Norm Bound, e.g., 2 x 10 10 200 steps (i.e., 0.4 epochs of normal training).
E MORE ON DP-ADAM
Adam optimizer can be thought of as a combination of Momentum and RMSprop. is also one of the most popular optimizers used in deep learning training. Hence we believe studying the effective- ness of our proposed method on Adam demonstrates it is generally applicable to different variants of SGD optimizer.
Adam optimizer has two state parameters, first momentum m, and second momentum 84 that keep track of the rumning history of 9t and of respectively, where 9i is the gradient (i.e., my 33 - + (1 (1-31 31)9; ir = 32 ut- + (1 - 32)g²), ). The model update would be -17 m /o, + with some small for numerical stability. (Without loss of generality, omit the bias correction for the first and second momentum terms here for the point of demonstration. The conclusion stays the same.)
In the DP implementation, the raw gradient is clipped and noised, then the private gradient is passed into the optimizer as Therefore, the first training step of Adam, when mp and Un are initialized to be mi = 9i and vui 4e gl. It means the private gradient is normalized by itself for the first training step. Therefore, if we only train for step, the model update would look similar regardless of the difference in 91. The normalized gradient would be a vector of #1's. In this case, the distance between the models does not reflect the scale of the additive noise. But if we train for multiple steps, the state parameters will keep track of the running history of the private gradients ge (which consist of both the raw gradients and the additive noise). For any later steps, e.g., at step 2. the m2 = 31 mi (1 (1-33 3, )92; 02 = 32- 01 + (1 32)95). The noise term in g: in update m2/ V122 + cannot be simplified as that for the first step, hence the resultant model updates would be dependent on the additive noise. Even though the model distance may not strictly be proportional to the scale of the noise after this normalization, but its general dependency on the noise scale still holds. Therefore, when using AP-Adam, we need to run multiple steps to observe any useful trend.
Another observation about DP-Adam is that, when C' is large, the model parameter distance satu- rates/plateaus (as seen in Figure g in Appendix G We believe it is because the scale calibrated noise also increases causing the private gradient and running momentum to be both indistinguishable from the noise. In this case, the normalized gradient updates would all look "similar" even if keep increasing the scale of the noise (by increasing C) and the distance between them would saturate.
increase C. Eventually, the noise would dominate the real gradient signal,
F IMPLEMENTATION DETAILS
In our code, both our tests and the training scripts of the developers are written by using the PoTch library (Paszke al. 2019). Regarding model architectures, use ResNet20 from g thub. com/
14

---

Under review as a conference paper at ICLR 2023
akamaster/pytoreh resnet cifftl0 (ldebaaev), WideResNet50-2 from the Torchvi- sion library, and BERT from the Transformers library (Wlf et al. 2019). We manually replaced the batch normalization layers of ResNet20 and WedeResNet50-2 with group normalization layers because the former not compatible with differentially private training. We used the Opacus li- brary (Yousefpour et al. 2021) for correct DP-SGD, whereas the DP-SGD with mistakes is written by ourselves using the ToTch library or modified from Opacus library.
G ADDITIONAL FIGURES
Figure and Figure 6 are additional evaluation results for Section additional results for Section 4.2
Figure 7 and Figure are
Changesn barollly
Clipkin No Clppps
Chonges essealees
Cliple No Clipping
Chongee in love volio
Clyppig No Clagara
944
0.2
0.
16'
10
10
10
10 Gradlent Norm Bound (C)
10
Grad oodiaut Norm Bound
Gradlent Norm Bound (C)
(a) CIFAR-10 (ResNet20)
(b) ImageNet (WideResNetSO-22)
(c) SST2 (BERT)
Figure 5: This is the reproduction of Figure 2
except the optimizer used here is the Adam optimizer. Similar results can be seen so we believe the proposed method is effective across different optimiz- ers.
116
Changee in loce salncx
Per-ouzpll p
28-
Min-besk grddirt cipping
20
gaddest chgpang poilict dipping
200
Percoumple gradean digeing
Chongesin
100
graderi cappina p 0.2
20
200
100
100
Mite-betch
Mini--usich
stzc
(a) CIFAR-10 (ResNet20)
(b) ImageNet (WideResNet5O-22)
(c) SST2 (BERT)
exxcept the optimizer used here is the Adam optimizer. Similar results can be seen so we believe the proposed method is effective across different optimiz- ers.
Figure 6: This the reproduction of Figure 3
15

---

Under review as a conference paper at ICLR 2023
3006
Modla Mouill Paramdtr Diaunee
No Nots, p-
No Nalee, -
Cannet Callratie, p-didt Calbentio,
Nase p-..01
Gama Cakeeni
Calterie
Niiie Calleriex
Modd Pomeente
2200
-20-
26
200
1066
Oradien: Norm Boand (C)
Grallent Noem Bornd
Oralient Nor Board
1e5
(a) CIFAR-I0 (ResNet20) No gradient clipping
(b) ImageNet (WideResNettO-22)
(c) SST2 (BERT) No gradient clipping
No gradient clipping
Figure 7: This is the reproduction of Figure 4 except consider the case of no gradient clipping. Similar results can be seen so we believe the proposed method is effective no matter whether the clipping is implemented correctly or not.
160-
p 8.99
Madel Pa an Doun2
Ouncel Neive Caleations p
Wioeg Nase Colfendion $200.00
50-
Cencel Caitmatin, p-
Cullreiin
Whing Noles Caltertioe, 0.9
Wring Catnnaoo C0L
200
eie
566
1006
2000
4000
$000
$000
1066
20
490
ale
1000
Gradient Norm Boand
Baznd
(a) CIFAR-10 (ResNe(20)
(b) ImageNet (WideResNeet0-22) Per-example gradient clipping
(c) SST2 (BERT)
Per-example gradient clipping
Per-example gradient clipping
Modil Premaca Diurze
2510
#380
Nikeo Calhertiee
Modil Prammrre
2200
No Nais, t.du
Cencect Neine Calbaation p-
Whoog Noau Calleatioe, 0.9
9300
2300
No.Nebi p
1300
1200
Cirmec Note Callratin, p 600
Wiing Niil C Callvoatoo, p-0.7
2/6
de
660
4600
6000
inlen
2/6
sie
slo
1600
Goalian Nerts Botand (C)
Grudient Noeta Bound
Gradian Nomn Boand (C)
(d) CIFAR- (ResNet20)
(e) ImageNet (WideResseet50-2)
( SST2 (BERT)
Mini-batch gradient clipping
Mini-batch gradient clipping
Mini-batch gradient clipping
10o-
bioos, p
Moad Pantzete Dioune
4380
Naie ddditiinn 0
4385
Winng Calhertten 2.97
Wiveng Muii Calimatiee,
No.Neli,
Modd
Calisnabes
26.
480
Grudient
(g) CIFAR-I0 (ResNet20) No gradient clipping
(h) ImageNet (WideResNetSO-2)
(i) SST2 (BERT)
No gradient clipping
No gradient clipping
Figure 8: This is the reproduction of Figure 4 and Figure 7 except that the optimizer used here the Adam optimizer. Similar results be seen believe that the proposed method is effective across different optimizers.
16


Reviews:
[{"review_id": "g4wmdqB10A7", "reviewer": "Reviewer_y7ow", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the problem of finding out whether an implementation of DP-SGD is subject to silent bugs affecting the privacy protection, focusing on (i) gradient clipping and (ii) noise scaling. For each case, it proposes simple tests that relying on varying DP hyperparameters (batch size, noise level, clipping bound). The tests only assume the knowledge of a model training function function (black-box), which makes them agnostic and generic. The resulting tests are empirically validated on several datasets and architectures.", "summary_of_the_review": "This paper addresses a practical problem of large importance and proposes simple yet effective methods. However, the paper could benefit from an improvement in writing, and the applicability of the tests to DP-Adam would require more theoretical justifications. If the authors addressed these concerns, I would be willing to significantly increase my score, as I think this paper could be impactful for the community.", "strength_and_weaknesses": "# Strengths\n\n- (Impact) This paper will be appealing for DP practitioners, as it addresses a real need to ensure the privacy of actual implementations of DP. I think this goes in a much-needed direction.\n- The proposed tests are simple conceptually, yet effective for (i) and (ii), lightweight, and can be plugged over any black-box implementation of DP (agnosticism)\n\n# Weaknesses\n\n- (Clarity) The paper would benefit from a more formal writing: some hypotheses are buried within the text, others are missing, and the theoretical justification of the noise calibration test for DP-Adam lacks rigor (see more detailed comments below).\n- (Limitations) A more minor weakness is that not all potential pain issues of DP implementations are covered, notably batch sampling issues: the paper is limited to noise calibration and gradient clipping", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Yfp1yBYsqwC", "reviewer": "Reviewer_Tpks", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces light-weight approaches for debugging differentially private stochastic gradient descent (DP-SGD). The authors first identify the possible bugs which invalidate DP-SGD privacy guarantees (per-example clipping, noise calibration), and then proposes tests that aim to identify the exact problem (no gradient clipping, not clipping per-example grads; no noise, wrong noise scaling) without having to modify the source code. The authors empirically test their proposed solution to show that it correctly identifies various implementation issues in DP-SGD.\n", "summary_of_the_review": "A nice paper to read, but the current problem is too simple and lacks ambition.\n\n\n### Update after rebuttal\n\nAfter reading the other reviews and the author responses, I keep my score unchanged and recommend rejection: as I commented to the authors, I think debugging DP-SGD is simply not important enough to warrant publication in ICLR. If I would be to review this paper again in the future, the most important update that would persuade me to increase my score would be to try and make the debugging tools more widely applicable.", "strength_and_weaknesses": "### Strengths:\n\n1) The paper is generally very easy to follow and nice to read.\n\n2) The proposed approach seems solid.\n\n3) Debugging DP software can be really painful, since often the main signal of potential problems is that everything works too well. Good debugging tools are potentially very helpful.\n\n### Weaknesses:\n\n1) My main (and pretty much only) problem with the paper is that the current problems it aims to solve are quite specific (only DP-SGD) and not too hard to diagnose without the proposed tools. I would be more enthusiastic if the authors can expand the work to provide a framework for debugging DP software (at least somewhat) more generally. I do not believe that the current work warrants a publication in ICLR.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B5TIDUQ3ajr", "reviewer": "Reviewer_wnYy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method to help the developer to debug the implementation of SD-SGD. Three kinds of bugs can be theoretically guaranteed to be detected: 1. clipping to mini-batch gradient   2. no gradient clipping     3. noise without calibration. ", "summary_of_the_review": "I don't think the studied problem is very important. Since there are already standard/official packages for DP-SGD or other DP optimizers, I don't believe getting DP-SGD incorrectly implemented is a common problem nowadays. Two examples of those packages are TensorFlow Privacy (https://github.com/tensorflow/privacy) and Opacus for PyTorch (https://opacus.ai/). ", "strength_and_weaknesses": "Strength: The experiment is solid and sufficient\n\nWeakness: I am not persuaded that the studied problem is important", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6bPWnQ0ILit", "reviewer": "Reviewer_HFjU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a practical dev-level method for debugging differentially private SGD (DP-SGD), so that practitioners can check their implementations, and make sure all the steps of DPSGD  are working correctly and that the guarantee is achieved. Unlike the direction of work that does checking by finding bounds through attacks, this work focuses on the implementation and the training. More specifically, the proposed method checks the following three things a) clipping is happening, b) clipping is happening on a per-example level rather than a minibatch level c) noise addition is being done correctly. They do the first two checks by looking at the difference in loss by changing the clipping factor (for a) and minibatch size (b). They do the last check by training models multiple times and finding the standard deviation between the final parameters of the different models (looking at the l2 distance between model parameters).", "summary_of_the_review": "I like the idea of having implementation checks for DP-SGD. Although I find the first two checks a bit obvious, I find the third one novel and I think having an implementation of them that can be easily used is really important.", "strength_and_weaknesses": "Strengths:\n\n1. I think the problem of debugging DP implementations and having appropriate tests is really important, and practitioners could really benefit from built tools that could easily test their implementations. \n\n\n\nWeaknesses:\n\n1. I think as this paper is more on the development/coding side, it is really important to have the code but I could not find a link to the anonymized repository. In general, how the tests/checks are implemented, how easy they are to use and how efficient they are is super important, as the checks themselves are very easy to come up with. The first two, i.e. changing clipping factor and batch size is something that I have done before. Another common thing people check is changing the epsilon value and setting it to very high and seeing how the performance improves. The last check however is novel and interesting. \n\n2. One thing that should be better clarified in the paper and treated with caution is that the first two checks should really not be done on actual private, eyes-off data! clipped gradients are still private and anything that is a function of them is off limits. So anything prior to noisy-ing the gradients cannot be seen. However, the good news is any data (even fake synthetic) would work here. I think this should be specified in the paper, otherwise there is privacy violations.\n\n3. I worry about the efficiency of the method as for the checks things need to change and model needs to be trained multiple times. This could have a lot of overhead and it needs further analysis.\n\n4. This is minor: I think the first 3 pages of the paper are very repetitive, without providing much information. This is especially the case for the intro. I think it would be better if the intro actually gave an overview of how you do the check, rather than repeating the stuff that is being checked. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject