Nearly Tight Black-Box Auditing of Differentially Private Machine Learning
Authors: Meenatchi Sundaram Muthu Selva Annamalai
Emiliano De Cristofaro
Venue: NeurIPS 2024 poster (2024)
Keywords: Privacy Auditing
Differential Privacy
DP-SGD
Primary area: privacy

Abstract: This paper presents an auditing procedure for the Differentially Private Stochastic Gradient Descent (DP-SGD) algorithm in the black-box threat model that is substantially tighter than prior work.
The main intuition is to craft worst-case initial model parameters, as DP-SGD's privacy analysis is agnostic to the choice of the initial model parameters.
For models trained on MNIST and CIFAR-10 at theoretical $\varepsilon=10.0$, our auditing procedure yields empirical estimates of $\varepsilon_{emp} = 7.21$ and $6.95$, respectively, on a 1,000-record sample and $\varepsilon_{emp} = 6.48$ and $4.96$ on the full datasets.
By contrast, previous audits were only (relatively) tight in stronger white-box models, where the adversary can access the model's inner parameters and insert arbitrary gradients.
Overall, our auditing procedure can offer valuable insight into how the privacy analysis of DP-SGD could be improved and detect bugs and DP violations in real-world implementations.
The source code needed to reproduce our experiments is available from https://github.com/spalabucr/bb-audit-dpsgd.

Nearly Tight Black-Box Auditing of Differentially Private Machine Learning
Meenatchi Sundaram Muthu Selva Annamalai
University College London
meenatchi. annamalai. 2eeec222c2ccc ac. uk
Emiliano De Cristofaro
University of California, Riverside emilianodc@cs. ucr.ccuu.creee. edu
Abstract
This paper presents an auditing procedure for the Differentially Private Stochas- tie Gradient Descent (DP-SGD) algorithm in the black-box threat model that substantially tighter than prior work. The main intuition is to craft worst-case initial model parameters, as DP-SGD's privacy analysis agnostic to the choice of the initial model parameters. For models trained on MNIST and CIFAR-10 at theoretical = 10.0, our auditing procedure yields empirical estimates Semp 7.21 and 6.95, respectively, and 4.96 on the full datasets. By contrast, previous audits were only (rela- tively) tight in stronger white-box models, where the adversary can access the model's inner parameters and insert arbitrary gradients. Overall, our auditing procedure offer valuable insight into how the privacy analysis of DP-SGD could be improved and detect bugs and DP violations in real-world implementa- tions. The source code needed reproduce our experiments available from https://github...
1,000-record sample and Cemp 6.48
Introduction
Differentially Private Stochastic Gradient Descent (DP-SGD) I allows training machine learning
models while providing formal Differential Privacy (DP) guarantees [15.. DP-SGD is popular algorithm supported many open-source libraries like Opacus [40] TensorFlow 19). and JAX [4]. Progress in privacy accounting and careful hyper-parameter tuning has helped reduce the gap in model performance compared to non-private training [IT] Furthermore, pre- pre-training models on large public datasets and then fine-tuning them on private datasets using DP-SGD can also be used to improve utility [1].
Implementing DP-SGD correctly be challenging bugs that lead to DP violations have been found in several DP-SGD implementations 77 22 These violations do not only break the formal DP guarantees but can also result in realistic privacy threats 17
with most DP algorithms 5 26 and
33 This prompts the need to audit DP-SGD implementations to verify whether the theoretical DP guarantees hold in practice. This usually involves running membership inference attacks (32). whereby an adversary infers whether or not sample was used to train the model. The adversary's success estimates the empirical privacy leakage from DPDSGD, and that is compared the theoretical DP bound. If the former is "far" from the latter, it means that the auditing procedure loose and may not be exploiting the maximum possible privacy leakage. Whereas, when empirical and theoretical estimates are "close," " auditing is tight and can be used to identify bugs and violations 28
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Although techniques to tightly audit DP-SGD exist in literature (28 29,, they do so only in active white-box threat models, where the adversary can observe and insert arbitrary gradients into the intermediate DP-SGD steps. By contrast, in the more restrictive black-box threat model, where an adversary can only insert an input canary and observe the final trained model, prior work has only achieved loose audits [20 28 [2]. In fact, prior work has suggested that the privacy analysis of DP- SGD in the black-box threat model can be tightened but only in limited settings (i.e., strongly convex smooth loss functions) 3 9 39 Therefore, it has far remained an open research question [28) whether or not it is possible to tightly audit DP-SGD in the black-box model.
In this paper, we introduce novel auditing procedure that achieves substantially tighter black-box audits of DP-SGD than prior work, even for natural (i.e., not adversarially crafted) datasets. The main intuition behind our auditing procedure is to craft worst-case initial model parameters, as DP-SGD's privacy analysis is agnostic to the choice of initial model parameters. We show that the gap between worst-case empirical and theoretical privacy leakage in the black-box threat model is When auditing a convolutional neural network trained on MNIST and CIFAR-10 at theoretical
much smaller than previously thought 20 28 29
10. our audits achieve an empirical privacy leakage estimate of Eemp = 6.48 and 4.96, respectively, compared to Eemp = 3.41 and 0.69 when the models were initialized to average-case initial model parameters, as done in prior work M 20 28
In the process, we identify and analyze the key factors affecting the tightness of black-box auditing: 1) dataset size and 2) gradient clipping norm. Specifically, we find that the audits are looser for larger datasets and larger gradient clipping norms. We believe this is due to more noise, which reduces the empirical privacy leakage. When considering smaller datasets with 1,000 samples, our audits are tighter - for 10.0, we obtain Cemp = 7.21 and 6.95, respectively, for MNIST and CIFAR- 10.
Finally, we present tight audits for models with only the last layer fine-tuned using DP-SGD. Specifi- cally, we audit a 28-layer Wide-ResNet model pre-trained on ImageNet-32 [I] with the last layer privately fine-tuned on CIFAR-10. With e = 10.0, we achieve an empirical privacy leakage estimate of Eemp = 8.30 compared to cmpp
7.69 when the models are initialized to average-case initial
parameters (as in previous work 11
20 28 ).
Overall, the ability to perform rigorous audits of differentially private machine learning techniques allows shedding light on the tightness of theoretical guarantees in different settings. At the same time, it provides critical diagnostic tools to verify the correctness of implementations and libraries.
2
Related Work
DP-SGD. Abadi et al. (1] introduce DP-SGD and use it to train ferd-forward neural network model classifying digits on the MNIST dataset. On the more complex CIFAR-10 10 dataset, they instead fine-tune a pre-trained convolutional neural network, incurring a significant performance loss. Tramèr and Boneh (35) improve on 1 using hand-crafted feature extraction techniques but also suggest that end-to-end differentially private training may be inherently difficult without additional data. Papernot et al. (30) and Dörmann et al. [14] also present improvements considering different activation functions and larger batch sizes. Finally, De et al. (I] achieve state-of-the-art model performance on CIFAR-10 both while training from scratch and in fine-tuning regimes; they do so by using large batch sizes, augmenting training data, and parameter averaging. Our work focuses on auditing realistic models with high utility that might be used in practice; thus, we audit models with utility similar to [T] and the same models pre-trained in [T] the fine-tuning regime.
Membership Inference Attacks (MIAs). Shokri et al. 32 propose the first black-box MIAs against machine learning models using shadow models. Carlini al. [6] improve by fitting a Gaussian distribution on the loss of the target sample and using fewer models to achieve equivalent attack accuracies. Sablayrolles et al. [II consider per-example thresholds to improve the performance of the attacks in a low dalse-positive rate setting 6 By factoring in various uncertainties in the MIA "game," Ye et al. (38] construct an optimal generic attack. Concurrent work [6] 25 37 use poisoned pre- trained models, which is somewhat similar to our approach of using worst-case initial model parameters, but with a different focus. Specifically, Liu et al. (25) and Wen et al. [37 show that privacy leakage from MIAs is amplified when pre-trained models are poisoned but, crucially, do not consider DP auditing. Additionally, while Feng and Tramèr [16] achieve tight audits with poisoned pre-trained models, their strategy is geared towards the last-layer only fine-tuning regime. By contrast,

---

we focus on auditing DP-SGD with full model training to estimate the empirical worst-case privacy leakage in the black-box setting.
Auditing DP-SGD. Jayaraman and Evans 21 derive empirical privacy guarantees from DP-SGD, observing a large gap from the theoretical upper bounds. Jagielski et al. [20] use data poisoning in the black-box model to audit Logistic Regression and Fully Connected Neural Network models and produce tighter audits than using MIAs. They initialize models to fixed (average-case) initial parameters, but the resulting empirical privacy guarantees are still far from the theoretical bounds. They also adapt their procedure the fine-tuning regime, again producing loose empirical estimates.
Nasr et al. [29] are the first to audit DP-SGD tightly; they do so by using adversarially crafted datasets and active white-box adversaries that insert canary gradients into the intermediate steps of DP-SGD. In follow-up work, (28) presents a tight auditing procedure also for natural datasets, again in the active white-box model. Hamella-Béguclle et al. [42] present tighter auditing procedures using Bayesian techniques, needing fewer models to audit larger privacy levels. Finally, De et al. [II] perform black-box auditing of their implementation of DP-SGD in the JAX framework 4 they also use fixed (average-case) initialization of models as in 20 similarly achieving loose audits.
showed that reducing the randomness in the initial model parameters results in tighter empirical privacy leakage estimates, the audits remained loose as the fixed model parameters were still initialized randomly. Specifically, designing an effective adversarial strategy that provides tight audits in the black-box setting without destroying utility has remained, to the best of our knowledge, an open problem.
Overall, while prior work [T]
20 29
3 Background
3.1 Differential Privacy (DP)
Definition 1 (Differential Privacy (DP) 15
A randomized mechanism M : D - R is (s,
differentially private if, for any two neighboring datasets D, D' E DD and $ S R, it holds:
Pr[M((D) E S] s ef Pr[M((D) E S
Put simply, DP guarantees a formal upper bound (constrained by the privacy parameter e) the probability that any adversary observing the output of the mechanism M can distinguish between two neighboring inputs M ii.e., two datasets differing in only one record.
3.2 Differentially Private Machine Learning (DPML)
DP-SGD. Differentially Private Stochastic Gradient Descent (DP-SGD) [ is a popular algorithm for training machine learning models while satisfying DP. In this paper, we consider classification tasks on samples from the domain X' x y, where X' is the features domain and y is the labels domain.
DP-SGD, reviewed in Algorithm D introduces two hyper- parameters: gradient clipping norm C' and noise multiplier a. Typically, C' is set to 1, while a is calibrated to the required privacy level (s, 5) based on the number of iterations, T, and batch size, B. Early DP-SGD work (1 set batch sizes to be small and similar to those of non-private SGD; however, recent work has highlighted the advantage models with them. Following this
of setting large batch sizes [I] [4 by training state-of-the-ee trend, and to ease auditing, we set B = N, as also done in prior work (28). Finally, while the initial set of model parameters 6g is typically sampled randomly (e.g., using Xavier initialization (18)., it can also be set arbitrarily (and even adversarially). without affecting the privacy guarantees provided by DP-SGD.
Fine-tuning regime. In this setting, one first pre-trains a model on a (large) public dataset and then fine-tunes it the private data. When suitable public dataset is available, prior work has shown that fine-tuning can achieve much higher utility than training from scratch using DP-SGD [I].
There are two methods for fine-tuning: doing it for all layers or only the last one. In both cases, the initial model parameters, 0g. are set to the pre-trained model parameters (rather than randomly) before running DP-SGD. However, while the former updates all model parameters during DP-SGD, with the latter, the model parameters until the last (fully connected) layer are "frozen" and not updated during DGD. This setting treats the pre- pre-trained model as a feature extractor and trains Logistic
3

---

Algorithm Differentially Private Stochastic Gradient Descent (DP-SGD)
Require: Dataset, D. Itcrations, T. Learning rate, 1y. Batch size, B. Loss function, &, Initial model parameters, Dg. Noise multiplier, a. Clipping norm, C.
1: for É [T] do
2: Sample L, from D with sampling probability B/N
for (e., y4) L. do
4:
ge(2, yx) ) - Va ((r; (2))
5:
ge yv) t ge (2:)/ max(1, (
6:
end for
7: 9e 17 (E, v.) + MM (()))
8: 01+1 01 Tg
end for
10:
return Dr
Regression model using DP-SGD based on the extracted features. As prior work shows [T] that this setting produces models with accuracy closer the non-private state of the art (while also incurring smaller computational costs), we also do fine-tuning of just the last layer.
3.3 DP Auditing
As mentioned above, DP provides theoretical limit the adversary's ability bounded by the privacy parameter to distinguish between M(D) and M(D"). With auditing, one assesses this limit empirically verify whether the theoretical guarantees provided by M are also met in practice. There may be several reasons this does not happen, e.g., the privacy analysis of .M is not tight (29) or
because of bugs in the implementation of M 28 36 Therefore, with DP mechanisms increasingly deployed in real-world settings, auditing them becomes increasingly important.
Deriving Eemp- When auditing a DP mechanism M, one runs M repeatedly on two neighboring inputs, D and D', R times each and the outputs (0 = (oi,,.., 0g. and O' = -o,,..., onH)) are given an adversary .A. Next, A tries to distinguish between O and O', which determines a false positive rate (FPR) and false negative rate (FNR). For a given number of outputs 2R and confidence level a, empirical upper bounds FPR and FNR can be computed using Clopper-Pearson confidence intervals 29 28]. Finally, the empirical upper bounds are converted to an empirical lower bound Eemp using the privacy region of M (see below) and Cemp can be compared the theoretical e. We consider an audit tight if Cemp R E. To ease presentation, we abstract away the details and use Eemp EstimateeEss(PPH FNR, 2 R, a, 8) to denote the process of estimating an Cemp from a given FPR and F
Privacy region of DP-SGD. Given a mechanism M, the privacy region of the mechanism, RM (FPR, F-R) defines the FPR and FNR values attainable for any adversary aiming to distinguish between M(D) and M(D). Nasr et al. (28) note that the privacy region of DP-SGD corresponds (tightly) that of p-Gaussian Differential Privacy 13,, which can be used to first compute a lower bound on J.:
Pemp
(1- - Φ (FNRR
(I)
Then, a lower bound on corresponds to a lower bound on s by means of the following theorem:
Theorem (μ-GDP (e, 8DP conversion [3] A mechanism G-GDP iff it (c,6(E))-DD for all e 2 0, where:
(_E
ō(E) +
(2)
1
2
2
4 Auditing Procedure
In this section, we present our auditing procedure. We discuss the threat model in which we operate, the key intuition for using worst-case initial parameters, and how we craft them maximize the empirical privacy leakage.
Please to httppp////ggaabbbooo//paabbbbbba
for details.

---

Algorithm 2 Auditing DP-SGD
Require: Data distribution, DD Number of samples, n. Initial model parameters, 8g. Loss function, &. Number of repetitions, 2R. Decision threshold, 7. Confidence level, a.
I: Sample D ~ D
2: Craft target sample (IT, VT)
3: D' DU ((2T, yr)}
4: Observations O - (1.0'+ +
5: for E [R] do
6: 8, DD
7:
6. f- (D, ()
8:
O't] + E(or; (rr, UT))
9:
O'l1] < E(0'; (2T,
10: end for
11: FPR + 1 Kofo E 0}
R
12: FNR i |oolo ,',0< <]
13: Eemp f- EstimatoEps(FPR, FNR, 2R, 6)
14:
return Eemp
Threat model. We consider simple black-box model where the adversary only sees the final model parameters, Or, from DP-SGD. Specifically, unlike prior work on tight audits [28] [2]] the adversary cannot observe intermediate model parameters or insert canary gradients. We do consider more realistic adversaries that may exist in practice. Additionally, we assume that the adversary can choose a worst-case target sample as standard for auditing DP mechanisms [20). Finally, we assume that the adversary can choose (possibly adversarial) initial model parameters. Note that this is not only an assumption allowed by DP-SGD's s privacy analysis, but, in the context of the popular public pre-training, private fine-tuning regime [II this is also reasonable assumption.
As we operate in the black-box threat model, the procedure only makes a call to the DP-SGD algorithm-once again, they cannot insert canary gradients, etc. As mentioned, the adversary can set the initial model parameters (0o), which is fixed for all models. Finally, we assume that the final model parameters are given to the adversary, who calculates the loss of the target sample "observations." These are then thresholded to compute FPRs and FNRs and estimate the Cemp using the EstimateFpps process (see Section 3.3). This threshold, which we denote as т, must be computed on a separate set of observations (e.g.. validation set) for the Cemp to constitute a technically valid lower bound. However, as any decision threshold for GDP is equally likely to maximize Cemp (28), we follow common practice [27 29 28
Auditing. Our DP-SGD auditing procedure is outlined in Algorithm 2
to find the threshold that maximizes the value of Semp from O O'.
Crafting worst-case initial parameters. Recall that DP-SGD's privacy analysis holds not only for randomly initialized models but also for arbitrary fixed parameters. Indeed, this was noted in prior work 20 28
and led to using initial model parameters that are fixed for all models. In our work, we consider worst-case, rather than average-case, initial parameters that minimize the gradients of normal samples in the dataset. This makes the target sample's s gradient much more distinguishable. To do so, pre-train the model using non-private SGD distribution as the target dataset. Specifically, for MNIST, we pre-train the model on half of the full dataset (reserving the other half to be used for DP-SGD) for 5 epochs with batch size of 32 and learning rate of 0.01. For CIFAR-IO, we first preit the model on the CIFAR- 100 dataset for 300 epochs with batch size 128 and a learning rate of 0.1. Then, we (non-prit moon-rrivatel) fine-tune the model on half of the full dataset for 100 epochs, with a batch size of 256 and a learning rate of 0.1. Ultimately, show that the gradients of normal samples are indeed minimized, achieving significantly tighter audits of DP-SGD in the black-hox model.
auxiliary dataset that follows the same
Experiments
In this section, we evaluate our auditing procedure using image datasets commonly used benchmark differentially private classifiers and CNNs that achieve reasonable utility on these datasets.
5

---

Dataset 1.0
2.0
4.0
10.0
MNIST
95,4
95.7
95.9
95.9
CIFAR-I0
46.0
51.2
52.2
53.6
Table 1: Accuracy of models for average-case initial model parameters (0o).
Due to computational constraints, we only train 2R 200 models (100 with the target sample and 100 without) determine the empirical privacy estimates Eemp the full model training setting and train 2R- = 1000 models for the last-layer only fine-tuning setting. In theory, we could achieve tighter audits by increasing the number of models, however, this would require significantly more computational resources and time (see below). Since our work focuses on the (broader) research questions of whether tight audits are possible in the black-box setting and how various factors affect this tightness, we believe 2R- = 200 provides a reasonable tradeoff between tightness and computational overhead.
In our evaluation, we report lower bounds with 95% confidence (Clopper-Peanl Peopper-Pearson 10] and report the mean and standard deviation values of Eemp over five independent runs
5.1 Experimental Setup
Datasets. We experiment with the MNIST 24 and 10 23 datasets. The former includes 60,000 training and 10,000 testing 28x28 grayscale images in one of ten classes writtewritten digits), Often referred to "toy dataset," is commonly used to benchmark DP machine learning models. CIFAR-10 a more complex dataset containing 50,000 training and 10,000 testing 32x32 RGB images, also consisting of ten classes. Note that the accuracy of DP models trained on CIFAR-10 has only recently started to approach non-private model performance [ 35 Whether on MNIST or CIFAR-10 10. prior black-box audits (using average-cass initial parameters) have not been tight. More precisely, for theoretical E = 8.0, De et al. [I]] and Nasr et al. (28) only achieve empirical estimates of Cemp and Cemp 1.6, respectively, MNIST and CIFAR-10.
To ensure fair comparison between the average-case and worst-case initial parameter settings, we split the training data in two and privately train on only half of each dataset (30,000 images for MNIST and 25,00 ,0000 for CIFAR-10). The other half of the training dataset used as the auxiliary dataset to non-privately pre-train the models and craft the worst-case initial parameters (see Section 4). We note that for CIFAR- 10, we additionally augment the pre-training dataset with the CIFAR-100 dataset to boost the model accuracy further (see Appendix A for further details).
Models. When training the model fully, we use a (shallow) Convolutional Neural Network (CNN) for both datasets. We do so as we find accuracy to be poor with models like Logistic Regression and Fully Connected Neural Networks, and training hundreds of deep neural networks is impractical due to large computational cost. Therefore, we draw on prior work by Dörmann et al. [4] and train shallow CNNs that achieve acceptable model accuracy while being relatively fast to train. (We report exact model architectures in Appendix A). Whereas, for the last layer-only fine-tuning setting, we follow [1 and use the Wide-ResNer WRN-288-00
model pre-trained on ImageNet-32 [1].
Experimental Testbed. All our experiments are run on cluster using 4 NVIDIA A GPUs, 64 CPU cores, and 100GB of RAM. Auditing just one model took approximately 16 GPU hours on MNIST and 50 GPU hours CIFAR-10.
5.2 Full Model Training
To train models with an acceptable accuracy on MNIST and CIFAR-10, we first tune the hyper- parameters (i.c., learning rate 1) and number of iterations T); due to space limitations, we defer discussion to Appendix A
Model Accuracy. In Table 1 we report the accuracy of the final models for the different values of E-S we experiment with. While accuracy on MNIST is very good (2 95% for all E), for CIFAR-10.
"The source code needed to reproduce our experiments is available from https: wttpp/ww....o/pppaabb.....a. /bb-audit-daae adsgg-

---

Theoretical r
Average-rase dg
Worst-case fg
MNIST
CIFAR10
9
8
Empirical Kanp
Empirical
6-
5-
2-
Theoretical
Theoretical
Figure 1: Auditing models with average-case vs worst-case initial parameters at various levels of e.
10
1.0
Gradient norm
Fump
Average gradient norm
0.9
Empirical Eaup
0.8
0.7
0.6
0.5
Number of pre-training epochs
Figure 2: Comparing the average gradient norms to empirical privacy leakage, Eemps for models trained on MNIST at E = 10.0 with increasing number of pre-training epochs.
we are relatively far from the state of the art (=50% at = 10.0 vs. 800 at = 8.0 in This due two main reasons. First, as we need train hundreds models, we are computationally limited to using shallow CNNs rather than deep neural networks. Second, we need to consider a much larger batch size to ease auditing [28]; thus, the models converge much more slowly. Appendix B we show that, given enough iterations (T = 1,000), the model reaches accuracy of 700% at result, we audit models striking a reasonable trade-off between utility and computational efficiency, discussed above.
& 10.0, almost approaching state-of-ther
Impact initialization. In Figure we compare the difference Eemp between worst-case initial )8) ). In the latter, all models
parameters and average-case parameters (as done in prior work [I] 20 are initialized the same, randomly chosen Og (using Xavier initialization [8). while, in the former, to Og from model pre-trained on an auxiliary dataset. In the rest of this section, the target sample set as the blank sample, which known to produce the tightest audits in previous work [1] 28
With worst-case initial parameters, we achieve significantly tighter audits for both MNIST and CIFAR-10. Specifically, at ε 1.0, 2.0, 4.0, 10.0, the empirical privacy leakage estimates reach Eemp = 0.08, 0.54, 1.89, 6.48 and Eemp = 0.00, 0.18, 1.43, 4.96 for MNIST and CIFAR-10, re- spectively. comparison, with average-case initial parameters as in prior work, the empirical privacy leakage estimates were significantly looser, reaching only Cemp = 0.00. 0.06, 0.61, 3.41 and Eemp 0.07, 0.02, 0.20. 0.69 for MNIST and CIFAR-10, respectively. Notably, at larger privacy levels = = 4.0 and и = 10.0, for both MNIST and CIFAR-10 10, the empirical privacy leakage estimates when auditing with worst-case initial parameters are not only larger than the average-case, but are outside of the + s.d. range. This shows that our auditing procedure not only produces significantly tighter audits than prior work, especially at larger es, but also that, at lower e-s, our evaluations are currently limited by the number of models used to audit (resulting in large s.d.).
7

---

Theoretical
n=
100
n=
1,000
MNIST
CIFAR10
9
8
Empirical Caxp
Empirical Kamp
6-
5-
5-
2-
Theoretical
Theoretical
Figure 3: Auditing models trained on varying dataset sizes (n) at different values e. The full dataset size [D] is 30,000 for MNIST and 25,000 for CIFAR-10.
To shed light on why worst-case initial parameters improve tightness, we vary the amount of pre- training done to craft 8g and, in Figure 2] plot the corresponding Eemp-S and the average gradient norm of all samples (after clipping) in the first iteration of DP-SGD. For computational efficiency, we focus on MNIST and set the theoretical parameters, all models achieve 2 95% accuracy on the test set. As the number of pre-training epochs increases from (no pre-training) to 5, the average gradient norm steadily decreases from 1.00 to 0.51, as expected. On the other hand, the empirical privacy leakage increases from 3.41 to 6.48. In other words, with more pre-training, the impact of the other samples in the dataset reduces, making the target sample much more distinguishable, thus yielding tighter audits. Although there appears to be an anomaly in the Cemp obtained when the model pre-trained 3 epochs, this is within the standard deviation.
10.0. Recall that, regardless of the initial model
Remarks. With lower values, the empirical Empp estimates are still relatively far away from the theoretical bounds. This is because, in each run, we train a small number of models and report the average Cemp across all five runs. fact, the maximum Eemp obtained over the five runs amount to Cemp 0.27, 1.48, 3.23, 8.28 and Cemp 0.00, 0.54, 2.33, 6.70, respectively, on MNIST and CIFAR- 10, for ε 1.0, 2.0, 4.0, 10.0. Nevertheless, even on the averages, our audits are still appreciably tighter than prior work work-approximately by factor of 3 for MNIST [I]] and CIFAR- Overall, this confirms that the tightness of the audits can vary depending on the datasets,
10 28 possibly due to differences in the difficulty of the classification task on the datasets. For instance, MNIST classification is known to be easy, with DP-SGD already reaching close to non-private model performance, even at low E-S. By contrast, CIFAR-10 -EFAR-101AN is much more challenging, with considerable gap still existing between private and non- private model accuracy [[I].
Impact of size of dataset. In the auditing literature, empirical privacy leakage is often commonly estimated on (smaller) dataset samples both to achieve tighter audits and for computational efficiency reasons [II. In Figure 3 we evaluate the impact of the dataset size (n) the tightness of auditing. While smaller datasets generally yield tighter audits, the size's 's impact also depends on the dataset itself. For instance, at 10.0. auditing with and CIFAR-10, respectively, compared to Eemp = 6.48 and 4.96 on the full dataset. other words, with MNIST, the dataset size does not significantly affect tightness. However, for CIFAR-10, smaller datasets significantly improve the audit's s tightness significantly.
100 results in Cemp 7.41 and 7.01, for MNIST
Sensitivity to clipping norm. Finally, we evaluate the impact of the dyper-parameee of black-box audits. In the black-box setting, unlike white-box, the adversary cannot arbitrarily
the tightness
insert gradients that are of the same scale the gradient clipping norm, C' [28). Thus, they
restricted to gradients that naturally arise from samples. In Figure 4 we report the Eemp values
obtained with varying clipping norms, C. For computational efficiency, the results CIFAR-10 for models trained on datasets of size n = 1,000.
When the gradient clipping norm is small (C = 0.1, the audits are tighter. This is because the gradient norms are typically very small in the worst-case initial parameter setting, leading to much higher "signal-to-noise" ratio. While this might suggest that black-box audits may not be as
8

---

Theoretical
C= 0.1
C= 1.0
C = 10.0
MNIST
CIFAR10
9
8
Kamp
Empirical
6-
5-
2-
Theoretical
Theoretical
Figure Auditing models trained with varying gradient clipping norm (C) at 1.0, 2.0,2... 2.0, 4.0, 10.0.
Theoretical
Average-case d
Worst-case d
10
9
Empirical Lonp
4
1.0
2.0
4,0
10.0
Theoretical
Figure 5: Auditing models when fine-tuning only the last layer using average-case and worst-case initialization of models at various levels of e.
effective with larger gradient clipping norms, we observe that, with C = 10.0, model accuracy is poor too-more precisely, with 1.0, 2.0, 4.0, 10.0, accuracy reaches 85.6%, 93.1%, 95.0%, 96.1%, respectively, compared to 97.2%, 97.4%, 97. 97.5%. 97.5% with C' = 1.0 on MNIST. This indicates that learning effectively,
audits are looser with noise. In fact, this could suggest that the privacy analysis for the C' = 10.0 setting could be further tightened the black-box threat model, which we leave future work.
10.0 because the model
there is too much
5.3 Fine-Tuning the Last Layer Only
We now experiment with the setting where a model is first non-privately pre-trained on a large public dataset, and only the last layer is fine-tuned the private dataset. For this setting, choose the WRN-28-10 model pre-trained on ImageNet-3r 32 and fine-tune the last layer on CIFAR-10 10 using DP-SGD, done in [ At 1.0, 2.0, 4.0, 10.0. fine-tuning the last layer yields model accuracies of 91.3%, 91.4%, 91.4%, 91.5% and 93.1% 1%, 93.2%, 93.3%, 93.2% for average-case and worst-case initial model parameters, respectively. In this section, craft a target sample, as has been shown to produce tighter empirical privacy guarantees for Logistic Regression models.
the ClipBKD procedure from 20 to
Once again, we compare the tightness of the audits with respect to the initial model parameters. In Figure we report the empirical privacy leakage estimates obtained various levels of when the last layer is initialized to average- and worst-case (through pre- training) initial model parameters. Similar to the full model training setting, model initialization can impact audit tightness, albeit not as significantly. Specifically, at 10.0, we obtain Epmp = 9.19 and 9.33, respectively, when the last layers are initialized average- and worst-case initial model parameters.
Overall, the audits in this setting are considerably tight, and increasingly so for larger 6, as we obtain empirical privacy Ieakage estimates of Eemp 0.67, 1.62, 3.64, 9.33 for 1.0, 2.0, 4.0, 10.0,

---

respectively. The maximum Eemp across five independent runs is also much tighter, namely, Eemp 1.37, 2.10, 4.28, 9.94 for theoretical at 1.0, 2.0, and 4.0 suggests DP violation, this is, in fact, within the standard deviation of +0.40, 0.36, and 0.38, respectively.
1.0, 2.0, 4.0, 10.0, respectively. Although maximum Cemp exceeds the
We also evaluated the impact of the dataset size and clipping norm in this setting as well; however, these factors do not significandly impact audit tightness, as this is a much simpler setting compared to training a CNN model fully. For more details, please see Appendix
6 Conclusion
This paper presented a novel auditing procedure for Differentially Private Stochastic Gradient Descent (DP-SGD). By crafting worst-case initial model parameters, we achieved empirical privacy leakage estimates substantially tighter than prior work for DP-SGD in the black-box model and for natural (i.e.. not adversarially crafted) datasets. At e 10.0, we achieve nearly tight estimates of Cemp 7.21 and 6.95 for datasets consisting of 1,000 samples from MNIST and CIFAR- R-10, respectively. While we achieve slightly weaker estimates of Eemp = 6.48 and 4.96 on the full datasets, these still represent roughly 3x improvement over the estimates achieved in prior work [1] (8).
Naturally, our work is not without limitations - the main one being the computational cost of auditing. Black-box auditing typically requires training hundreds of models to empirically estimate FPRs/FNRs with good accuracy and confidence. This can take hundreds of GPU hours, even for the shallow CNNs we audit in our work. Thus, auditing deep neural networks (e.g., WideResNet) trained on large datasets (e.g., ImageNet) may be computationally challenging for entities that are not huge corporations. However, as our main objective is addressing the open research question of whether tight DP-SGD audits are feasible in the black-box model, we focus on shallow models. Our results are very promising as, thanks to the intuition of using worst-case initial parameters, we do achieve nearly tight audits. Nevertheless, we leave it to future work to reduce the computational cost of the auditing procedure, e.g., by training significantly fewer models.
Relatedly, recent work 2 34 has begun to focus on auditing within a single training run, although it has thus far achieved only loose empirical estimates. One interesting direction would be to combine the insights from this paper (i.e., using worst-case initial model parameters) with one-shot auditing techniques. However, this may not be trivial as these techniques typically employ large number of canary samples, each with large gradient norms, which can potentially interfere with our goal to reduce the gradient norm of "other" samples.
Furthermore, we only consider full batch gradient descent (B = n1), i.e., without sub-sampling. Note that this is standard practice and eases auditing by enabling GDP auditing, which requires fewer models and is much less computationally intensive compared to auditing DP-SGD with sub-sampling using Privacy Loss Distributions (PLDs) (28). Nevertheless, when auditing DP-SGD with sub- sampling in the "hidden-state" threat model where the adversary can only observe the final trained model but can insert gradient canaries, recent work has shown that there a significant gap between the theoretical upper bound and the empirical lower bound privacy leakage achieved [8. Even though this suggests a privacy amplification phenomenon in the hidden-state setting (which would extend to black-box as well), for general non-convex loss functions, prior work has also shown that the privacy analysis of DP-SGD with sub-sampling is tight even the hidden-state B. Therefore, whether audits can be tight in the black-box threat model for DP-SGD with sub-sampling under realistic loss functions and datasets remains an open question for future work.
Acknowledgments and Disclosure of Funding
This work has partly been supported by a National Science Scholarship (PhD) from the Agency for Science Technology and Research, Singapore (A STAR). We also wish to thank Jamie Hayes for providing ideas and feedback throughout the project.
References
[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, Deep Learning with Differential Privacy. In CCS, 2016.
10

---

[2] G. Andrew, P. Kairouz, S. Oh, A. Oprea, H. B. McMahan, and V. Suriyakumar. One-shot Empirical Privacy Estimation for Federated Learning. In ICLR, 2024.
[3] M. S. M. S. Annamalai. It's Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss. In AfSec, 2024.
[4] B. Balle, I.. Berrada, S. De, Ghalcbikesabi, J. Hayes, A. Pappu, S. L. Smith, and Stanforth. JAX-
Privacy: Algorithms for Privacy-Preserving Machine Learning in JAX. atttp:///githi: htw.iittt.oo....eeeeeeoooeeeeeee goog] e e-deepmind/jas httpaa/dddda.piiiaaaacaa.aamaaae 2022.
[5] B. Bichsel, S. Steffen, I. Bogunovic, and M. Vechev. DP-Sniper: Black-Box Discovery of Differential Privacy Violations using Classifiers. In IEEE S&P, 2021.
[6] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership Inference Attacks From First Principles. In IEEE S&P, 2022.
[7] T. Cebere. Privacy Leakage at low sample size. http:://ghhaabbcco/ppppp//s//pps
2023.
[8] T. Cebere, A. Bellet, and N. Papernot. Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model. arXiv:2405. 14457, 2024.
[9] R. Chourasia. J. Ye, and R. Shokri. Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent. NeurIPS, 2021.
[10] C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 1934.
[11] S. De, L.. Berrada, J. Hayes, I.. Smith, and B. Balle. Unlocking High-Accuraey Differentially Private Image Classification through Scale. arXiv:2204. 13650, 2022.
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Lcage-Scale Hierarchical Image Database. In CVPR, 2009.
[13] J. Dong, A. Roth, and W. Su. Gaussian Differential Privacy. arXiv: 2019.
[14] F. Dörmann, O. Frisk. L. N. Andersen, and C. F. Pedersen. Not All Noise is Accounted Equally: How Differentially Private Learning Benefits from Large Sampling Rates. In International Workshop Machine Learning for Signal Processing, 2021.
[15] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of Cryptography, 2006.
[16] S. Feng and F. Tramèr. Privacy Backdoors: Stealing Data with Corrupted Pretrained Models. In ICML, 2024.
[17] A. Gadotti, Houssiau, M. M. Annamalai, and Y-A. de Montjoye. Pool Inference Attacks on Local Differential Privacy: Quantifying the Privacy Guarantees of Apple's S Count Mcan Sketch in Practice. In USENIX Securiry, 2022.
[18] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
[19] Google. TensorFlow Privacy. https://github. http:ittuuu..mmmmmemteeee woomm//ensofffoo//oooooooo www.o/////yy////////// 2019.
[20] M. Jagielski, J. Ullman, and Oprea. Auditing Differentially Private Machine Learning: How Private Private SGD? In NeurlPS, 2020.
[21] B. Jayaraman and D. Evans. Evaluating differentially private machine learning in practice. In USENIX Security, 2019.
[22] M. Johnson. fix prng key reuse in differential privacy example. https: httpe://ggttub..commgoooo http///ogggl////ooo 11/3646 2020.
[23] A. Krizhevsky, Learning Multiple Layers Features from Tiny Images. httpp://www. ca. utoronto aea/"kiii/eeerriin features- 2009-00000000 TR. pdf 2009.
[24] Y. LeCun, C. Cortez, and C. C. Burges. The MNIST database of handwritten digits. http: //an///annn... lec un. cww. www.c////bbbb.e exdd/mnists... 1998.
[25] R. Liu, T. Wang, Y. Cao, and L. Xiong. PreCurious: How Innocent Pre- Tre-Trannd Language Models Turn into Privacy Traps. In CCS, 2024.
[26] I Lokna, A. Paradis, D. I. Dimitrow, and M. Veehev. Group and Attack: Auditing Differential Privacy. In CCS, 2023.
[27] S. Maddock, A. Sablayrolles, and P. Stock. CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning. ICLR, 2023.
[28] M. Nasr, J. Hayes. T. Steinke, B. Balle, F. Tramèr. M. Jagielski, N. Carlini, and A. Terzis. Tight Auditing of Differentially Private Machine Learning. In USENIX Security, 2023.
11

---

[29] M. Nasr, S. Songi, A. Thakurta, N. Papernot, and N. Carlini. Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning. In IEEE S&P, 2021.
[30] N. Papernot, A. Thakurta, S. Song, S. Chien, and U. Erlingsson. Tempered Sigmoid Activations for Deep Learning with Differential Privacy. In AAAI, 2021.
[31] A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. White-box vs Black-hox: Bayes Optimal Strategies for Membership Inference. In ICML, 2019.
[32] Shokri, M. Stronati, C. Song, and Shmatikov. Membership Inference Attacks against Machine Learning Models. In JEEE 2017.
[33] T. Stadler, B. Oprisanu, and C. Troncoso. Synthetic Data Anonymisation Groundhog Day. In USENIX Security, 2022.
[34] T. Steinke, M. Nasr, and M. Jagielski. Privacy Auditing with One (1) Training Run. NewrIPS, 2024.
[35] F. Tramer and D. Boneh. Differentially Private Learning Needs Better Features (or Much More Data). In ICLR, 2021.
[36] F. Tramer, A. Terzis, T. Steinke, S. Song. M. Jagiclski, and N. Carlini. Debugging Differential Privacy: A Case Study Privacy Auditing. arXiv:2202.1223 12211 2022.
[37] Y. Wen, L. Marchyok. S. Hong. J. Geiping. T. Goldstein, and N. Carlini. Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models. NeurIPS, 2024.
[38] Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. In CCS, 2022.
[39] Ye and Shokri. Differentially Private Learning Needs Hidden State (Or Much Faster Convergence). NeurIPS, 2022.
[40] A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malck, J. Nguyen, S. Ghosh, Opacus: User-friendly Differential Privacy Library PyTorch.
A. Bharadwaj. Zhao, arXiv:2109.1223 12298, 2021.
[41] S. Zagoruyko and N. Komodakis. Wide Residual Networks. The British Machine Vision Conference, 2016.
[42] S. 7anella-Béguelin, Wutschitz, Tople, A. Salem, V. Rühle, A. Paverd, M. Naseri, B. Köpf, and
D. Jones. Bayesian Estimation of Differential Privacy. In JCML, 2023.
12

---

100
Test Accuracy (5%
77
2.0
10.0
0
200
400
600
800
1000
Iterations, T
Figure 6: Test accuracies (%) on CIFAR-10 for models trained for varying number of iterations, T and learning rates, 1) fixed 10.0.
A Model Architectures and Hyper-parameters
In this work, use the shallow Convolutional Neural Networks used by Dörmann et al. [14), who make minor modifications to the CNNs used earlier by Tramèr and Boneh 35 to achieve good model utilities on MNIST and CIFAR-10. Exact model architectures for MNIST and CIFAR-10 are reported in Tables and respectively.
Layer
Parameters 16 filters of 5x5
Layer
Parameters
Convolution
Convolution
32 filters of 3x3, stride 1. padding
Max-Pooling
2x2
Convolution
32 filters of 3x3, stride 1. padding
Convolution
32 filters 4x4
Max-Pooling
2x2, stride padding
Max-Pooling
2x2
Convolution
64 filters of 3x3, stride 1, padding
Fully connected 32 units
Convolution
64 filters of 3x3, stride 1, padding
Fully connected 10 units
Max-Pooling
2x2, stride padding 0
Convolution
128 filters of 3x3, stride 1, padding
Table 2: Shallow CNN model for MNIST with Tanh activations.
Convolution
128 filters of 3x3, stride 1, padding
Max-Pooling
2x2, stride 2, padding 0
Fully connected 128 units
Fully connected
10 units
Table 3: Shallow CNN model for CIFAR-10 with Tanh activations
Hlyper-parameter tuning. To achieve the best model utilities, we tune the hyper-parameters of DP-SGD (i.e., number of iterations, T and learning rate, 17). For MNIST, we train for = 100 iterations, with a learning rate of 1 4 for both average-case and worst-case initial model parameters. For CIFAR-10, we train for = 200 iterations, with a learning rate of 7) for the average-cast initial model parameter setting. For the worst-case initial model parameter setting, we use learning rate of 17 = instead, as this resulted in the best model performance for CIFAR-10 When fine-tuning just the last layer, train for T 25 iterations, with learning rate of 10.0. Note that, for the experiments varying the dataset size, the learning rate was scaled accordingly so that the size of the dataset does not affect the "influence" of each sample (i.e., 2) All clipping norms as done prior work [11] and batch sizes are set to the dataset size B = n to ease auditing (28). Finally, the noise multiplier calculated from the batch size and number of iterations using the Privacy loss Random Variable accountant provided by Opacus 40
set C = 1.0
B Model Convergence
In Figure 6 we plot the test accuracies obtained when training the CNN model on the CIFAR-10 dataset for varying number of iterations, T, and learning rates, 7) at a fixed theoretical & = 10.0. Specifically, train with full batch gradient descent (i.c., B n) and re-calculate the noise
13

---

Theoretical
C= 0.1
= 1,0
C= 10.0
Empirical Lany
1.0
2,0
4.0
10.0
Theoretical
(a) Clipping norm
Theoretical
100
1,000
n (
10
Empitiaal Sony
Theoretical
Dataset size, n
Figure 7: Comparing the tightness of Eemp for different factors at different values of e
multiplier a for each and such that all points on the Figure have the same privacy level of 10.0.
First, we observe that r 2.0 and rj 4.0 result in comparable test accuracies that increase with the number of iterations, reaching close to 70% accuracy at T 1,000 (67 .3% and 65.4%, respectively). Therefore, even though at T = 200 and & = 10.0, the models we audit only achieve 53.6% accuracy, this is computational limitation, and the model can, in fact, achieve good model utilities given enough iterations. However, increasing the learning rate to r = 10.0 results in poor model performance.
In their original work, Dörmann al. achieve test accuracy 70.1% at 7.42 by training the model with a batch size of 8,500 (sampling rate, 0.17). However, following prior work [28), we choose not consider sub-sampling as this makes tight auditing difficult. This is because the theoretical privacy region of the sub-sampled Gaussian mechanism (the underlying mechanism for DP-SGD with sub-sampling) loose compared the empirical privacy region observed, thus making audits in this setting inherently weaker.
C Last layer only fine-tuning
Finally, we evaluate the impact of the clipping norm and dataset size on the tightness of auditing in the last layer only fine-tuning setting. Overall, find that these factors have little to impact, as tight audits are achieved in the most difficult settings considered.
In Figures 7aland an 7b clipping norms and when training on datasets with different sizes, respectively. While there may be some minor differences between the empirical Cemp obtained in different settings, overall, the gradient clipping norm and dataset size do not significantly affect tightness. We believe this is because it is much simpler to fine-tune the last (Logistic Regression) layer of CNN than to train the full model. Here, the empirical privacy leakage is already maximized (Eemp & 10.0 for theoretical 10.0) in the most difficult setting (C 10.0, using smaller clipping norms and dataset sizes does not significantly improve the privacy leakage.
report the empirical privacy estimates obtained when using different gradient
[D). Therefore. "relaxing the setting" by
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claim of our paper that we achieve nearly tight audits of DP-SGD in the black-box setting. Indeed, for theoretical 10.0, we achieve empirical privacy leakage values that are quite close, especially settings where the number of samples in the dataset is low.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We cover the main limitation of work in the Conclusion section. Specifi- cally, we discuss that tight black-box audits need hundreds of models to be trained, which forms a computational limitation for our approach. We also highlight some advancements in auditing that require much fewer models, but unfortunately they are not applicable to our work as those techniques not achieve tight audits.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work is more empirical than theoretical.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our auditing procedure clearly in Section 4, and include exact hyper-parameters and model architectures in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We publicly available datasets and will release the accompanying code to reproduce our results upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/pu blic/ Sewddde/Coeeeuuebeeeeeeeeeeeeeee Psiicclicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ipss.c/puut petps//ccccllccggudeeeeeesss guwdes/ ProddeSubmissiooool....
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All hhper-parameterr parameters and training details are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All plots include error bar, calculated from the standard deviation across 5 runs as explained in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources used to train the models is provided in Section 5,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics bectps:/nneurppeee hwtpp///wl./..cccccceeeeeeeeeeee
Answer: [Yes]
Justification:
We do not attack live systems or use private datasets.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss how as DP tools and libraries are increasingly used in production settings, performing such audits becomes increasingly necessary, to precisely prevent societal harm, the Introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not attack any live systems, or propose a new "attack". We use existing attacks to empirically evaluate the privacy of open algorithms using open datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original paper for the MNIST and CIFAR-10 datasets used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithe thoode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new datasets or models with our work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "CHriBMpKmQ", "reviewer": "Reviewer_qSii", "summary": "The paper shows that privacy auditing is tight in the threat model where the model initialization is adversarially selected. They find that the decrease in gradient norms over the course of training helps improve the \"signal to noise\" ratio of the auditing example.", "questions": "What is the gradient norm of the auditing example in your experiments? I wonder if at C=10, the gradient norm is not large enough to \"max out\" the full clipping norm.\n\nDid you retune hyperparameters when changing the values of clipping norms?\n\nGiven the hypothesis that lower gradient norm of other examples leads to tighter audits, have you tried explicitly constructing an initialization where the other gradients are exactly 0? Perhaps even just nonprivate SGD on the private training data will get there.", "limitations": "I think the batch size should be better highlighted, but limitations related to model architectures, datasets, auditing trials are all well addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's main finding is useful and helps the privacy auditing community get a bit closer to understanding privacy leakage.\n\nThe authors propose a reasonable hypothesis for why their approach improves on prior work.", "weaknesses": "The general finding that \"when other examples' contributions are smaller, auditing is tighter\", has been a feature of other papers on auditing, even in the adversarial dataset experiments from Nasr et al. 2021 or the design of ClipBKD in Jagielski et al 2020.\n\nThe paper's claims of \"near tightness\" seem somewhat exaggerated - there are several parameter settings where the proposed auditing does not even appear to be better than prior work, and all experiments show a rather large gap.\n\nGiven that the batch size is an important detail for auditing, the setting of B = n should be mentioned in the main body of the paper.\n\nThe experiments are unfortunately on rather small models, making it difficult to know how generalizable the strategy is to more recent advances in DP-ML training.\n\nI would appreciate experiments on other datasets - several conclusions are drawn that seem dataset specific (e.g. dataset size), and it would be useful to know if there are more general trends in these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer qSii for their helpful feedback. In the following, we address their comments and questions.\n\n> **When other samples’ contributions are smaller, auditing is tighter**\n\nWhile the overarching intuitions are similar, our strategy is appreciably different from prior work [18, 27], as we also discuss in Point 1. of the Global Author Rebuttal comment vis-à-vis the novelty of crafting worst-case datasets. In the revised version of the paper, we will discuss this further to clarify how, despite the similarity of the intuitions, our work overcomes some significant challenges compared to prior work. \n\n> **Near tightness claims**\n\nThank you for your comment. We agree that, in some settings, especially at low(er) $\\varepsilon = 1.0$ and $2.0$, it is not entirely clear whether our auditing method improves upon prior work. However, we note that this is mainly due to computational limitations. In fact, there are significant improvements for larger $\\varepsilon = 4.0$ and $10.0$ on both the MNIST and CIFAR-10 datasets (e.g., for MNIST, from $\\varepsilon_{emp} = 3.41$ in prior work to $6.48$ in our work). This suggests that computational power is a limiting factor as lower epsilon values correspond to FPR/FNR rates that are very close to the random baseline, which would, in turn, require more models to estimate more accurately. Nevertheless, in the final version of the paper, we will tone down our claims of “near tight” audits to specify in which settings the audits are appreciably tighter than prior work.\n\n> **Stating $B = N$ in the main body**\n\nCurrently, the setting of $B = N$ is stated in Section 3.2, but we will emphasize/clarify this again in Section 5.\n\n> **Generalizability to deeper models**\n\nUnfortunately, due to both the number of models required for the audit and the large batch sizes, auditing modern deep models (e.g., WideResNet) in the black-box setting tightly is computationally (very) expensive. However, our method is inherently general and does not depend on the model architecture. Furthermore, the crafting of our worst-case initial parameters is efficient as the model has to be only pre-trained once. Therefore, in theory, we do not see any reasons that would prevent the generalizability of our results other than very significant computational costs. In future work, we will look into auditing deep neural networks as soon as we have access to more compute or we can improve the efficiency of the underlying auditing techniques. \n\n> **Experiments on other datasets**\n\nThank you for this comment. We did indeed think about additional datasets but ultimately settled on MNIST and CIFAR-10, which are considered benchmark datasets for auditing DP-SGD [10, 18, 27]. Moreover, the trends observed (e.g., smaller datasets yielding tighter audits) are the same across both MNIST and CIFAR-10 – due to the different complexities of the datasets, only the scale of the “impact” differs. Nevertheless, on Reviewer qSii’s request, we could add experiments on the FMNIST dataset as well (we are confident this will confirm the same conclusions we draw from MNIST/CIFAR-10).\n\n**Answers to Questions**\n\n> **Gradient norm of auditing sample**\n\nOn average, across all iterations, the gradient norm of the auditing sample, before clipping, is $10.5$ for MNIST and $150.4$ for CIFAR-10. Therefore, we do not believe that this is a “maxing out” issue.\n\n> **Re-tuning hyper-parameters for clipping norms**\n\nThe hyper-parameters were re-tuned for the clipping norms.\n\n> **Explicitly constructing initialization with 0 gradient norm**\n\nUnfortunately, even if the other gradients’ norms are exactly 0 on iteration 0, it will no longer be 0 for future iterations as the model weights become “corrupted” by the noise addition step [27]. Therefore, to the best of our knowledge, it is not possible to have an initialization that results in the other gradients’ norms being 0 throughout training. Nevertheless, one setting we did consider (but did not include in the paper) is the worst-case neighboring datasets setting, where  $D = \\emptyset, D’ = \\\\{(x, y)\\\\}$, which is numerically equivalent to what you suggest (the gradient norm of the empty dataset will be 0). While the audits were significantly tighter in this setting ($\\varepsilon_{emp} \\approx 9$ at theoretical $\\varepsilon = 10.0$ for both MNIST and CIFAR-10), we did not include it in our final paper since this setting has limited real-world value.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D18O9C1Tm6", "reviewer": "Reviewer_ncD1", "summary": "This paper proposes a new method for auditing DP-SGD in a black-box setting, where the auditor can only see the final parameters of the model (rather than intermediate steps). The main idea is to select worst-case initial model parameters; this seems to give a substantial advantage to the black-box analysis.", "questions": "- \"Although techniques to tightly audit DP-SGD exist in literature [26, 27], they do so only in active  white-box threat models, where the adversary can observe and insert arbitrary gradients into the  intermediate DP-SGD steps. However, real-world adversaries cannot always access the model’s  inner parameters, let alone insert arbitrary gradients.\"\n\nAs argued above, I don't think this is a valid premise. There's nothing in auditing that requires modelling a real-world adversary. Further, the assumptions that you make (e.g., worst-case initial parameters) do not correspond to a real-world adversary. I think the implications of what you claim here are incorrect, and I don't think it should be part of your motivations.\n\n- Threat model. It does look weird to see a threat model section in an auditing paper, and I would humbly recommend removing it: DP doesn't specify any particular threat model, and auditing is just about ensuring how tightly the definition is being matched empirically. If you do want to keep this section, please note that it's missing the assumption that the auditor (adversary?) can specify the initial model parameters; also, I was unclear as to what the following meant in this context: \"assume that the adversary can choose a worst-case target sample as is standard for auditing DP mechanisms\".\n\n- \"we set B=N\".\n\nCan you please clarify whether this is a constraint of your method, and if so what are the potential limitations?\n\n- Sentence \"When auditing a DP mechanism [...]\". Please define the symbol $R$. (It's clear what it is, but it's undefined.)\n\n- \"Crafting worst-case initial parameters\". This should be one of the main contributions, but unfortunately the paragraph doesn't provide much information on how exactly these parameters are created. Can you please provide more information in the text (and in rebuttal)?\n\n- Could you please double check: did your cluster only have 1 CPU core?\n\n- Typo \"is is\".", "limitations": "These were adequately addressed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The idea of selecting worst-case initial parameters for auditing is clever. Empirically, the proposed method strongly outperforms the chosen baseline.\nThis work shows that a (partially) black-box adversary isn't much weaker than a white-box one; this demonstrates that DP analyses (which assume the latter) may not be underestimating privacy too much.\nThe paper is well-presented.", "weaknesses": "# Motivations\nI think the authors should refine their motivations. They use two: 1) their method is good for detecting bugs, and 2) it provides insights into DP-SGD analyses.\n\n1. Bugs.\n\nThe ultimate purpose of _black-box_ auditing is unclear. There's two sides to this discussion. On the one hand, if our goal is to audit a DP-SGD implementation, surely we should be making all the worst-case assumptions (i.e., give as much advantage to the auditor as possible) to look for bugs. On the other hand, if our goal is to evaluate real-world attacks against DP-SGD, then it does make sense to consider a black-box assumption. The assumptions this paper makes are a bit of an hybrid: it assumes black-box for auditing, yet the adversary is allowed to specify the initial model parameters. The only semi-practical scenario I can think of is an attacker who creates a model (for subsequent fine-tuning to be carried out by the victim) and wants to augment the leakage in the fine-tuned version of the model. Aside from this scenario, I would recommend leaving out from this paper any motivation arguing that this work is more practical than other auditing methods for finding \"bugs\"; please refer to \"Questions\" for a concrete example.\n\n2. New insights.\n\nThe authors do provide one very convincing motivation: a big question is whether DP analyses of DP-SGD could be improved if they considered it as a black-box algorithm; currently, they all assume knowledge of intermediate gradients. This work offers some hope: from the analysis, it seems that the difference between black-box and the theoretically estimated \"epsilon\" may be closer than expected. If I may, I would strongly recommend the authors to center their motivations on this aspect.\n\n# Comparison\n\nI could not find a real comparison to previous auditing methods. Only two numbers appear (within the Datasets paragraph), reporting the performance of previous works; were these computed under the same experimental setting (and code base)? Ideally, they should be replicated under identical conditions to yours.\nSecondly, it'd be very interesting to see what similar white-box methods achieve, in comparison. How tight are they, on those datasets/models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ncD1 for their helpful feedback. In the following, we address their comments and questions.\n\n> **Motivation**\n\nThank you very much for your comments re. our motivation. Indeed, our work is primarily focused on the “New Insights” type of motivation, i.e., determining if the analysis of DP-SGD can be improved if considered as a black-box algorithm. To that end, we will follow the Reviewer’s recommendation to center our motivation around this aspect. We will also follow the Reviewer’s recommendation to leave out any motivation that argues about the “practical” aspect of black-box auditing.\n\n> **Comparison to prior work**\n\nWe note that previous work [10, 18, 26] considered the setting with subsampling (i.e., $B \\neq N$) whereas we consider full batch gradient descent ($B = N$). Nevertheless, we do present a direct comparison with [10, 26], which is the “average-case $\\theta_0$” setting – they, too, use the blank same and loss to carry out the audit. Although we did initially experiment with [18] (i.e., using the ClipBKD sample), we did not find that, in our setting, this significantly improved upon the results presented in “average-case $\\theta_0$” (probably because [18] was only designed for and tested on Logistic Regression and Fully Connected Neural Networks) and thus left it out of the paper. In Section 5, we will make it clearer that the “average-case $\\theta_0$” setting provides a direct comparison to [10, 26].\n\n**Answers to Questions**\n\n> **Threat model paragraph**\n\nWe included this paragraph to delineate the differences between what was done in prior work (active white-box) and what we do (black-box). This also mirrors writing in prior papers [26, 28]. Nevertheless, we will include the worst-case initial parameters assumption in the threat model (and thank the reviewer for suggesting this).\n\n> **Worst-case target sample**\n\nWe consider the setting where the rest of the dataset is chosen randomly (“natural dataset”), but the target sample is specifically chosen by the adversary and is not a “natural” sample. To make this clearer, we state that we “assume that the adversary can choose a worst-case target sample as is standard for auditing DP mechanisms.” We hope this clarifies the reviewer’s question.  \n\n> **Clarifying $B = N$**\n\nWe note that setting $B = N$ is not a constraint of our method; rather, it aims to make auditing easier and is done commonly [26]. Nonetheless, we note that recent work (Cebere et al., 2024) suggests that the privacy analysis when $B \\neq N$ might be complicated, and even in stronger threat models, tight auditing may be difficult to achieve.\n\n> **Clarifying $R$**\n\nWe will clarify in Section 3.3 that $R$ refers to the number of models/outputs.\n\n> **Details on worst-case initial model parameters**\n\nWe refer the Reviewer to Point 3. of the Global Author Rebuttal comment and will clarify this further in the text.\n\n> **Typos**\n\nWe apologize for the typos and will fix both of them in the text. We used 36 cores in the cluster (we will also clarify this).\n\n**References**\n\nCebere, T., Bellet, A., & Papernot, N. (2024). Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model. arXiv:2405.14457.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkHcLckxbr", "reviewer": "Reviewer_uyKT", "summary": "The paper considers the problem of auditing DP-SGD, i.e. deriving empirical lower bounds on the DP parameter $\\epsilon$. Many previous auditing works operated in the white-box setting, i.e. are allowed to choose arbitrary gradients for the canary (sensitive example) during DP-SGD, or were in the black-box setting (i.e. could determine some aspects of training, but not specific per-round gradients, and also does not get to see intermediate states) but had a sizeable gap between the empirical $\\epsilon$ and the theoretical upper bound. The paper gives a black-box auditing procedure that in some cases achieves empirical $\\epsilon$ very close to the theoretical upper bound on standard benchmarks of MNIST and CIFAR10. The main algorithmic insight is to use a model pre-trained on in-distribution data, which causes the gradients of in-distribution examples to be quite small even in the first round of private training, hence increasing the impact of the canary on training. In contrast, previous work used 'average-case' initializations without optimizing for their impact on the empirical $\\epsilon$. The authors perform a number of experiments using this improvement, on MNIST and CIFAR, varying whether they train a full model vs the last layer, and varying the dataset size and clip norm. The authors' audit generally improves upon the baseline of an average-case initialization, sometimes as much as a 5x increase in the empirical $\\epsilon$.", "questions": "* Do the authors believe the results in Figure 5 would become significant with more trials? i.e. do the authors have some intuition for why an initialization might matter more or less for last-layer-only fine-tuning than for full-model training? If so, it might be nice to include this in the paper.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper provides the first audits which come close to matching the theoretical upper bounds while operating in a black-box setting. In particular the empirical improvement over past black-box work is incredibly strong, and the qualitative improvement over white-box audits is important given the threat model the authors use is much more realistic.\n* The empirical results are especially impressive given the number of training runs the authors use (which is much smaller than some other auditing works) and the fact that the change needed to achieve the increase in empirical $\\epsilon$ is relatively lightweight (a single pre-training run).\n* The authors do a good job explaining intuition for why changing different aspects of the training pipeline affects the empirical $\\epsilon$ which makes it more likely others can build upon this intuition in practice or in future work, and the intuition is reflected in the empirical results.", "weaknesses": "* In some cases we may want to do privacy auditing not to e.g. find bugs in an existing DP-SGD pipeline, but to support the claim that a model has much better privacy guarantees than the worst-case theoretical $\\epsilon$ we are reporting. In this case, taking half of the dataset and using it as public pre-training data a privacy violation, i.e. it is unclear how to extend the authors' insights to this application of auditing. This might be a limited weakness, as anyway in practice we are probably going to try to pre-train on a distribution as close to the private distribution as possible, which might retrieve the benefits the authors observe.\n* The improvements in Figure 5 don't seem to be very significant, and for eps = 1.0 the average empirical epsilon of average-case initialization is in fact double that of worst-case initialization. This did not affect my score as I understand the authors may not have access to enough compute budget to narrow the confidence intervals, and also I don't think they should be punished for reporting insignificant results, but I think the improvement here is somewhat overstated in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer uyKT for their helpful feedback. While their questions are hopefully addressed in Point 2. of the Global Author Rebuttal comment, we address their comments regarding weaknesses here.\n\n> **Pre-training on another distribution**\n\nWe apologize as we are not entirely sure what the reviewer is referring to in the first point of the weaknesses in their review. We would be happy to address it if they could kindly clarify. Do they mean to say that it is unclear how our work would apply to the setting where a different dataset (instead of half of the target dataset) is used to craft the initial model parameters?\n\nIf so, we believe that it would depend on how close the distributions of the other dataset are to the target dataset. Naturally, datasets with closer distributions would yield tighter audits. However, as we focused on achieving the tightest possible audits within the black-box setting in this work, we have not looked into this aspect yet. Nevertheless, we think it would be an interesting aspect to look into for future work and we will explicitly add this in Section 6.\n\n> **Insignificant improvements in Figure 5**\n\n We believe that one of the reasons why there are not any significant improvements to the audits in this setting is that the audits are already nearly tight when using average-case initial model parameters. For instance, at theoretical $\\varepsilon = 10.0$, the $\\varepsilon_{emp} = 7.69$ for average-case initial model parameters already. Therefore, there is not much room for improvement in this setting resulting in comparable audits between the average-case and worst-case initial model parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wBGoSAT0K1", "reviewer": "Reviewer_JVDo", "summary": "This paper studies the problem of auditing DP-SGD in the black box threat model, i.e. only black-box access to the last iterate $\\theta_T$ of DP-SGD. The paper's main contribution is to show experimentally that pre-training helps to get tighter auditing. The argument behind using this idea is that pre-training makes the average gradient norm of “in-distribution” points small, thus making the target point used for auditing/membership attack more “distinguishable”. The paper explores this worst-case initialisation idea for two datasets, MNIST and CIFAR10, and reports tighter privacy lower bounds.", "questions": "- Worst-case vs average-case initialisation: (a) For the worst-case, $\\theta_0$ is selected by pretraining on half of the dataset, but what algorithm is used for that? Is it non-private SGD? And what is the random distribution used to generate $\\theta_0$ for the average case?\n\n- In Figure 5, the average-case initialisation seems to give tighter privacy and lower bounds, especially for small epsilon. This contradicts this work's main argument, i.e., using worst-case initialization is better. How do you explain this?\n\n- What canary strategy is used for the black-box audit? Do you only use the blank example? Did you test any other black box canaries?\n\n- The justification behind the reported success of pretraining is that the average gradient norm decreases with the number of pre-training epochs, reported in Figure 2. (a) Is the average of gradients computed on all samples (training+ test) or only training samples? (b) Why does this correlate with the success of your membership attack? Specifically, the MIA attack tresholds over the loss of the canary, thus this seems to be the statistic that should be analysed over the pre-training epochs. The gradient average norm is indicative for white-box attacks. (c) Pretraining decreases the gradient average loss of “in distribution” samples, so how would you adapt your auditing procedure for datasets where the empty sample, i.e. your canary, is \"in distribution\"?\n\n- What are the loss thresholds used for the black-box membership attack? Are they fine-tuned to find the best privacy budget? Also, the function EstimateEps() is never explicitly detailed in the paper to reproduce the results.\n\n- Is it not already possible to use [32]’s framework to run directly your black-box auditing in one run?", "limitations": "As noted by the authors in the conclusion, the auditing procedure is computationally heavy and time expensive.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well-motivated problem. Tight black-box auditing is still an open problem compared to the white-box setting.\n- Extensive experimental exploration of the effect of different hyperparameters on the auditing procedure.\n- The paper provides confidence intervals on the empirical privacy budget $\\epsilon$.", "weaknesses": "- The main contribution of this work is incremental. In addition, as noted in the paper, the idea that the initialisation of $\\theta_0$ affects auditing/membership inference has already been reported in the literature [10,18,26].\n\n- The justification for using pre-training is hand-wavy and not rigorous. See the question below.\n\n- No code is available to reproduce the results. The experimental setting is well-detailed. However, some important factors for reproducing the results have not been reported. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer JVDo for their helpful and constructive feedback. In the following, we address their comments and questions and clarify how we believe we can address their concerns regarding weaknesses.\n\n> **Contributions of crafting worst-case initial model parameters**\n\nAlthough the general idea of looking at initialization was indeed already reported in [10, 18, 26], these works only looked at tweaking the randomization of initialization. Instead, we focus on specifically crafting worst-case initial model parameters, which had not been considered before. We refer the Reviewer to Point 1. of the Global Author Rebuttal comment for a more detailed comparison with prior work.\n\n> **Experiments’ details**\n\nWe thank the reviewer for highlighting the gaps in the details of our experimental analysis. In addition to addressing the reviewer’s questions below, please note that we will make the code publicly available with the camera ready; in the meantime, we would be happy to share the code anonymously upon request.\n\n**Answers to Questions**\n\n> **Worst-case vs Average-case initialization**\n\nWe refer the Reviewer to Point 3. of the Global Author Rebuttal comment for details regarding crafting worst-case initial model parameters (i.e., pre-training on half of the dataset). For the random distribution, we follow [18] and use Xavier initialization, which we will clearly explain in Section 5.\n\n> **Discrepancies in Figure 5**\n\nWe note that the discrepancy is due to the small number of models and refer the Reviewer to the Point 2. of the Global Author Rebuttal comment, where we explain this in detail.\n\n> **Canary strategy**\n\nWe use the blank sample when auditing the full model and the ClipBKD sample [18] when auditing the last-layer only fine-tuning. We experimented with both samples (along with a “one-hot” sample) and used the sample with the tightest audits for both settings. We note that ClipBKD performed better for last-layer only fine-tuning as it was particularly designed and tested for this setting. We will clarify this in the revised version of the paper in Section 5.1.\n\n> **Success of pre-training strategy**\n\nIn Figure 2, (a) the average of the gradient norms is only computed on training samples. (b) The gradient norms of other samples are important to the success of our attack as they represent the expected change in the loss function due to the other samples during training. Specifically, when the gradient norms of other samples are close to 0, the loss function is mainly impacted by the presence of the target sample only, making the loss of the target much more distinguishable (when the target sample is present vs. when it is not). Put simply, the gradient norms indicate the level of “noise” in the dataset compared to the “signal.” Note that while prior work [18, 26] does aim to maximize the loss of the target sample by carefully crafting the target sample, this has not been enough to achieve tight audits. In our work, we find that we are able to explain (part of) this gap by using the gradient norms of the other sample taken as “noise.” We hope this provides additional intuition as to why our approach works and will clarify this in Section 5.2 as well. (c) Even if the empty sample is “in distribution,” we can mislabel the sample to make it “out of distribution” and use the mislabelled sample as the target sample, which is also a common method used in prior work.\n\n> **Loss thresholds**\n\nAs mentioned in Section 4, the loss thresholds are indeed fine-tuned to maximize privacy leakage. We will explicitly include the EstimateEps function (in the main body if there is space and in the appendix otherwise) and also release the code publicly for reproducibility.\n\n> **Using [32] for auditing in a single run**\n\nWe thank the reviewer for making the point that our method can potentially be combined with [32] to audit using a single training run, as also briefly mentioned in Section 6. However, this may not be trivial as a large number of target samples with potentially large gradient norms are used in [32]. This might interfere with our approach to reduce the contribution of other samples. Therefore, we leave this to a future paper to investigate this approach in more detail. In the revised version of the paper, we will additionally state these challenges of combining our work with [32].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful feedback and suggestions. In this message, we clarify points mentioned by multiple reviewers. We also address each reviewer’s concerns separately in individual comments.

**1. Novelty of worst-case initial model parameters (qSii, JVDo)** 

As also noted by Reviewers ncD1 and uyKT, one of the main contributions of our paper is the novel approach of crafting worst-case initial model parameters, which results in a strong improvement over prior work. While our method is built on similar intuitions (e.g., reducing the contributions of other samples), it is significantly different from prior work [18, 27] and much more effective. 

More precisely, while [18] does look at initialization, it focuses on tweaking the randomness of initialization rather than specifically crafting worst-case initial model parameters, thus resulting in looser audits. Furthermore, [27] crafts an adversarial worst-case dataset but sets the learning rate to 0, which destroys the model’s utility. 

Designing an effective adversarial strategy that provides tight audits in the black-box setting without destroying utility has remained an open problem. Our work overcomes this challenge through a novel strategy involving crafting worst-case initial model parameters. As also detailed in our comments to each review, in the revised version of the paper, we will clarify our motivation and contribution more clearly while toning down certain claims.

**2. Discrepancies in Figure 5 (uyKT, JVDo)**

The last-layer-only fine-tuning setting is a common setting studied both when auditing DP-SGD and for training private models with DP-SGD. This motivates us to verify whether our method offers improvements in auditing for this setting as well.

As pointed out by Reviewers uyKT and JVDo, we acknowledge that the improvements in this setting are fairly limited. 

We also agree that some discrepancies in Figure 5 were not well explained in the text. We will provide a brief explanation here and will explain this clearly in the revised manuscript. Specifically, the mean of the $\varepsilon_{emp}$ for theoretical $\varepsilon = 1.0$ appears to be tighter for the average-case initial model parameters setting rather than the worst-case initial model parameters. However, both values are still within the confidence interval of each other and therefore, we do not consider this to mean that average-case initialization performs better in this setting. Rather, this was an artifact of auditing using a small number of models (200). In fact, in preliminary testing, when the number of models is increased to 1,000, we find that $\varepsilon_{emp} = 0.35 \pm 0.38$ for average-case and $0.51 \pm 0.52$ for worst-case initial model parameters thus showing that there is in fact no discrepancy in Figure 5. We will explain this clearly in the text and will increase the number of models used to audit just for this setting given that we are able to have access to more compute within the timeframe.

**3. Details on worst-case initial model parameters (ncD1, JVDo)**

We thank the Reviewers for highlighting that some details (e.g., optimizer) are missing.

For MNIST, we pre-train the model on half of the dataset using non-private SGD for five epochs with batch size 32 and a learning rate of 0.01.

For CIFAR-10, we pre-train the model first on the CIFAR-100 dataset (there is a typo in the text “ImageNet-32” => “CIFAR-100”, which we will fix) for 300 epochs with batch size 128 and learning rate 0.1 using non-private SGD. We further fine-tune the model on half of the CIFAR-10 dataset using non-private SGD for 100 epochs with batch size 256 and learning rate 0.1.

We will make the details more explicit by moving the text from Appendix A to Section 4.

Meta-review: The reviews are generally positive, and the overall sentiment further improved slightly following the author rebuttal and discussion. There was a debate as to whether the limited technical novelty (running existing auditing techniques on pre-trained models) should be a barrier for acceptance. In the end, the consensus was that the work has some value by evaluating a simple idea which leads to significant practical gains for privacy auditing. Therefore, the paper is accepted.

Important note: the authors have promised to tone down their claims of “near tight” audits to specify in which settings the audits are appreciably tighter than prior work, as requested by some reviewers. Additionally, they have been asked to clearly discuss the limitation of auditing only DP-GD and not DP-SGD (i.e., without subsampling). I therefore ask the authors to incorporate these changes in the final version.
Decision: Accept (poster)