Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model
Authors: Tudor Ioan Cebere
Aurélien Bellet
Nicolas Papernot
Venue: ICLR 2025 Poster (2025)
Keywords: Differential Privacy
Privacy Auditing
Machine Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We investigate the privacy guarantees of the hidden state threat model via auditing with gradient-crafting adversaries.

Abstract: Machine learning models can be trained with formal privacy guarantees via differentially private optimizers such as DP-SGD. In this work, we focus on a threat model where the adversary has access only to the final model, with no visibility into intermediate updates. In the literature, this ``hidden state'' threat model exhibits a significant gap between the lower bound from empirical privacy auditing and the theoretical upper bound provided by privacy accounting. To challenge this gap, we propose to audit this threat model with adversaries that craft a gradient sequence designed to maximize the privacy loss of the final model without relying on intermediate updates. Our experiments show that this approach consistently outperforms previous attempts at auditing the hidden state model. Furthermore, our results advance the understanding of achievable privacy guarantees within this threat model. Specifically, when the crafted gradient is inserted at every optimization step, we show that concealing the intermediate model updates in DP-SGD does not enhance the privacy guarantees. The situation is more complex when the crafted gradient is not inserted at every step: our auditing lower bound matches the privacy upper bound only for an adversarially-chosen loss landscape and a sufficiently large batch size. This suggests that existing privacy upper bounds can be improved in certain regimes.

Published as a conference paper at ICLR 2025
TIGHTER PRIVACY AUDITING OF DP-SGD IN THE HIDDEN STATE THREAT MODEL
Tudor Cebere
Aurélien Bellet
Inria, Université de Montpellier tudor. cebere@ inria... fr
Inria, Université de Montpellier aurel 1 i en bel let innia a. fr
Nicolas Papernot
University of Toronto & Vector Institute nicolas, papernoteutooonto.
ABSTRACT
Machine learning models can be trained with formal privacy guarantees via dif- ferentially private optimizers such as DP-SGD. In this work, we focus on threat model where the adversary has access only to the final model, with no visibility into intermediate updates. In the literature, this "hidden state" threat model exhibits a significant gap between the lower bound from empirical privacy auditing and the theoretical upper bound provided by privacy accounting. To challenge this gap, we propose to audit this threat model with adversaries that craft gradient sequence de- signed to maximize the privacy loss of the final model without relying on intermedi- ate updates. Our experiments show that this approach consistently outperforms pre- vious attempts at auditing the hidden state model. Furthermore, our results advance the understanding of achievable privacy guarantees within this threat model. Specif- ically, when the crafted gradient is inserted at every optimization step, we show that concealing the intermediate model updates in DP-SGD does not enhance the privacy guarantees. The situation is more complex when the crafted gradient is not inserted at every step: our auditing lower bound matches the privacy upper bound only for an addversarially-coseee loss landscape and sufficiently large batch size. This suggests that existing privacy upper bounds can be improved in certain regimes.
INTRODUCTION
Machine learning models trained with non-private optimizers such as Stochastic Gradient Descent (SGD) have been shown leak information about the training data (Shokri al. 2017 Yeom et al. 2018 Balle et al. 2022 Haim et al. (022). To address this issue, Differential Privacy (DP) (Dwork al. 2006) has been widely accepted
the standard approach quantify and mitigate privacy leakage, with DP-SGD (Abadi et al. 2016) as the defacto algorithm to train machine learning models with DP guarantees. DP- follows the same steps as standard SGD but clips the gradients' norm to threshold before perturbing them with carefully calibrated Gaussian noise, providing differential privacy guarantees for each gradient step.
As the practical importance of differential privacy grew, the need to track the privacy loss efficiently across the entire training process (privacy accounting) became critical: indeed, optimal utility is obtained by adding the minimum amount of noise required to achieve the desired privacy guarantee.
Existing privacy accounting techniques rely on privacy composition (Kairouz et al.) 2015 Abadi
et al. 2016 Gopi et al. [ Doroshenko et al. 2022 derive the overall privacy guarantees of training run of DP-SGD from the guarantees of each gradient step. Privacy amplification techniques
(Kasiviswanathan et al.) 2011 Feldman al. 2018, Erlingsson al.) 2019; Cyffers and Bellet
2022; Altschuler and Talwar 2022) can further decrease the overall privacy loss by exploiting the non-disclosure of certain intermediate computations. For instance, privacy amplification by subsampling relies on the secrecy of the randomness used to select the mini-batches
Part of this work was done while visiting University of Toronto & Vector Institute

---

Published as a conference paper at ICLR 2025
Despite recent improvements in privacy accounting, training pee-paaameterized neural networks from scratch with differential privacy typically results in either weak privacy guarantees or significant utility loss (Tramer and Bonch, 202T). While possible explanation could be that the privacy accounting of DP-SGD based on composition is overly conservative, (Nasr et al. 2021 2023) refuted this hypothesis using privacy auditing. Leveraging the fact that differential privacy upper bounds the success rate of any adversary that seeks to infer private information from the ouiput of DP-SGD, they showed that is possible instantiate adversaries that achieve the maximal success rate allowed by the privacy accounting upper bound. This negative result suggests that the only hope to improve the privacy-utility trade-off's of DP-SGD to relax the underlying threat model, i.e., to make additional assumptions limiting the adversary's 's capabilities. We focus one capability granted to the adversary in prior work, namely that all intermediate models (i.e., training checkpoints) are released.
In this work, we consider natural relaxation where intermediate models are concealed and only the final model is released. This threat model, often referred to as hidden state, is particularly relevant in practice, encompassing scenarios such as open-sourcing trained model by publishing its weights. From a theoretical perspective, recent work has demonstrated that the hidden state model can yield significantly improved privacy upper bounds for DP-SGD compared to those derived through standard composition (Ye and Shokri 2022) Altschuler and Talwar (022). This improvement can be attributed the phenomenon of privacy amplification by iteration (Feldman al. 2018: Balle et al., 2019): in a nutshell, the privacy of a data point used at earlier stages of the optimization process improves as subsequent steps are performed. However, these results only hold for convex problems. leaving pivotal question unanswered: Does the privacy of non-convex machine learning problems improve when intermediate models are withheld? Empirical lower bounds obtained through privacy auditing (Nast et al.) 2021 2023 Steinke et al. 2023) suggest that indeed, concealing intermediary model amplifies privacy, exposing a gap between the empirical lower bound and the theoretical upper bound in the hidden state threat model. However, it remains unclear whether this gap arises from genuine privacy amplification (i.e., a loose upper bound) or from the suboptimality of existing adversaries (i.e., loose lower bound). This raises another crucial question: How can one design worst-case adversaries when the intermediate models are concealed?
Our contributions. We adapt the pradient-cratting adversarial approach of Nast et al. 2222)) to align with the hidden state model, leveraging the key observation that all privacy accounting techniques for DP-SGD are designed to protect against worst-case gradients. Instead of crafting data point (canary) that gets added to the training set as in prior attempts to audit the hidden state model (Nasr al. 2023 Steinke et al. 2023),, our adversaries craft a sequence of gradients prior the execution of the algorithm (i.e., without knowledge of the intermediate models). The crafted gradients are then added to gradients computed on real training points to yield the highest possible privacy loss for the final model. In other words, our adversaries abstract away the canary and directly decide the gradient would have produced when inserted at a given step of DP-SGD. But how do we pick the gradient sequence leading to the worst-case leakage?
In the scenario where crafted gradients are inserted at every iteration of DP-SGD, we demonstrate that gradient-crafting adversaries which allocate the maximum magnitude permitted by DP-SGD to single gradient dimension are optimal: they imply privacy lower bounds that match the known upper bounds given by the privacy accountant of Gopi et al. (202T). Therefore, our results reveal that releasing only the final model does not amplify privacy in this regime. In the case of small models, we achieve these tight privacy auditing results by carcfully selecting the gradient dimension, whereas for over weerpparametrrized models, the dimension selection can be arbitrary.
When the crafted gradient is not inserted at every step, we find that the above adversaries still outperform canary-crafting adversaries by a significant margin but cannot reach the privacy upper bounds. We show that part of the gap can be attributed to the inability of our adversaries to influence the gradients on real training data in steps where the crafted gradient is not inserted. To address this gap. we design an adversary that crafts both the gradient and the loss landscape. Our results uncover two distinct regimes: (i) when the batch size is large compared to the noise variance, our adversary proves to be optimal, yielding privacy lower bound that matches the known upper bound and thus demonstrating the absence of privacy amplification; and (ii) when the batch is small relative to the noise variance, we show strong evidence of privacy amplification for non-convex problems, although the effect is qualitatively weaker than in the convex case. Our findings advance the understanding of privacy guarantees in the hidden state model, and lay the groundwork for the design of better privacy accounting techniques for this threat model.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
2.1 DIFFERENTIAL PRIVACY AND DP-SGD
Differential Privacy (DP) has become the de-facto standard in privacy-preserving machine learning thanks to the robustness of its guarantees, its desirable behaviour under post-processing and composi- tion, and its extensive algorithmic framework. We recall the definition below and refer to Dwork and Roth (2014) for more details. Here and throughout, we denote by D the space of datasets.
Definition ((E, Diffffrrrett Privacy). A randomized mechanism M is (E,6)-DP for all neigh- boring datasets D E D and D' = D {x} E D and for all events O:
P(M(D) E OJ S ef PIM(D)) E 01+ +
(1)
In the above definition, 8 E (0, 1) can be thought of as a very small failure probability, while e > 0 is the privacy loss (the smaller, the stronger the privacy guarantees).
The workhorse of private machine learning is the Differentially Privacy Stochastic Gradient Descent (DP-SGD) algorithm (Song et al., [0133] Bassily al., 2014 Abadi et al.) 2016). Let D be the training dataset, 8 the model parameters and consider the standard empirical risk minimization objective ming [DT EXED x:) where é is loss function differentiable in its first parameter. DP-SGI) follows similar steps as standard SGD but ensures differential privacy by (i) bounding the contribution of each data point to the gradient using clipping and (ii) adding Gaussian noise to the clipped gradients. Formally, starting from some initialization Og. DP-SGD performs T iterative updates of the form:
o++1 0.
| (Ere,, clip( Vo, (00,))), Z.),
where 7) > is the learning rate, B, S D is mini-batch of data points, clip(g, C) g min(11C/|l))a) with C' > 0 the clipping threshold, and Ze ~ o20))
2.2 THREAT MODELS FOR DP-SGD
DP protects against an adversary that observes the output of either M(D) or M(D') and seeks to predict whether the dataset was D D' DU (x). (=) i.e., whether was included in the imput dataset (a.k.a., a membership inference altack) HHomer et al., 2008 (Shokri et al., 2017). In this context, the threat model specifies which information is observable/knowt by the adversary. For DP-SGI, in addition to the final model fr. threat models typically consider that the adversary has access to (and potentially controls) the model architecture, the loss L, the initialization 0o. the dataset (up to the presence of 2) and differ in which internal states of DP-SGD are observable by the adversary. Below, we recall the two threat models relevant to our work.
Standard threat model. Standard privacy accounting techniques analyze DP- SGD composition of (potentially subsampled) Gaussian mechanisms (Abadi et al. 2016 Gopi et al. 2021 Doroshenko let al. 2022). Composition allows the adversary to observe all intermediate models 01, .... 0r-1 (in addition to On and fr). Previous work has shown that existing privacy accounting techniques are tight in this threat model (Nasr et al., 2021 2023). However, revealing intermediate models is often unnecessary in practical deployment scenarios (particularly in centralized settings where a single entity conducts the training process) and may degrade the privacy-utility trade-off, thereby motivating the study of the hidden state threat model.
Hidden state threat model. Our work studies the scenario where intermediate models f..... Pr-1 are concealed from the adversary, who observes only the final model 0y. This threat model has at- tracted much attention recently, with theoretical work proving better privacy upper bounds than the standard threat model in some regimes (Feldman et al. 2018 8, Balle et al., 2019 Ye and Shokri| 2022] Altschuler and Talwar 2022). A fundamental phenomenon underlying these results is "privacy ampli- fication by iteration" [Feldman et al. 2018), which shows that repeatedly applying noisy contractive iterations enhances the privacy guarantees of data used in earlier steps. Unfortunately, applying this general result DP-SGD is only possible for convex problems, ruling out deep neural networks. The existence of privacy amplification for non-convex problems in this threat model is an open problem (Altschuler and Talwar 2022) that we study in this work through the lens of privacy auditing.
3

---

Published as a conference paper at ICLR 2025
2.3 AUDITING DIFFERENTIAL PRIVACY
Privacy accounting only provides upper bounds on the privacy loss captured by the DP parameters
(e, 6), and these bounds are not always tight. Leveraging the relation between DP and the performance of membership inference attacks, privacy auditing aims to produce lower bounds on the DP parame- ters by instantiating concrete adversaries permitted within a given threat model (Jagielski et al. 2020) Nasr et al. (021). When these lower bounds match the upper bounds given by privacy accounting, we can conclude that the privacy analysis is tight; when they do not, it is possible to improve the privacy accounting and/or the attacks. A privacy auditing pipeline can be broken down into two components: an adversary and an auditing scheme. The high-level process is shown in Algorithm [
Ad Adversary. Auditing mechanism .M first requires the design of adversary A, which seeks to predict the presence or absence of a canary point x* from the information available in the considered threat model. Typically, an adversary A consists of two subroutines RankSample and Re Reject i onRule. Rank&Samlle gives a confidence score that the observed output was interpreted as the two hypotheses of binary test. Re jeject i onnule applies threshold to the confidence scores generated across multiple random runs of M to decide when reject the first hypothesis. It then compares these decisions to the ground truth to produce binary test statistics: True Negatives (TN), True Positives (TP), False Negatives (FN), and False Positives (FP).
generated by sampling from MDD rather than from (MDU {x*) }), which
Auditing scheme. The auditing scheme takes as input the hypothesis testing statistics of an adversary and privacy parameter 8, and outputs a high-probability lower bound é on the privacy loss of the mechanism M. It comprises two subroutines: ConfInterval and ConV ConfInterval con- verts the adversary statistics to high probabil- ity lower bounds for the False Negative Rates o and False Positive Rates B. Convert.oo then converts these lower bounds into é for the specified ò by leveraging the fact that DP implies an upper bound on any adversary's performance. Our experiments will rely on the recently pro- posed auditing scheme based on Gaussian DP (Nasr et al., 2023). which we describe for com- pleteness in Appendix A
Algorithm 1 Privacy auditing
Input: Audited mechanism M, adversary A. dataset D, canary point z", number of auditing
runs R, privacy parameter ò
S < [.
Do t D
D1 < [x*
for i = to R do
draw random bit S. + ARankSample(M((D) Do, D1) end for TN, FN, FP = ARejectionnul(((...
b. Ber(1/2)
a, B + CoffIntervan TP, FN, FP)
+- Co return é
To obtain the tightest possible é, one should perform auditing using a worst-case canary x² and worst-case adversary A. This proves to be especially challenging in the hidden state model as the adversary cannot rely on the knowledge of intermediate models. In this paper, we will obtain tighter lower bounds é by abstracting away the canary and using adversaries that directly craft gradients.
3 RELATED WORK IN DIFFERENTIAL PRIVACY AUDITING
The goal of differential privacy auditing is to create worst-case adversaries that maximally exploit the underlying threat model, even if the information used by the adversary might not be available in practical scenarios A rich line of work has designed adversaries that tightly and efficiently audit the privacy of learning algorithms when intermediate models are released (Nasr et al. 202
Maddock et al. 2023 Nasr al.; 2023 Steinke et al. 2023). The first adversarial construction (the malicious dataset attack in Nasr et al. (2021)) to reach the theoretical upper bound given by privacy accounting for DP-SGD used restrictive threat model in which the adversary controls the entire learning process, including the dataset, the mini-batch ordering, all hyperparameters and intermediate models. In follow-up work by Nast et al. (002)), a nearly matching lower bound was obtained for a gradient-crafung adversary that does not need to control the dataset, the minibatches or the hyperparameters but still requires access to intermediate models.
TThis is in contrast membership inference attacks seeking feasibility against real systems Shokri et al. Yeom al. 2018 Carlini al. 2022 Zanifadch al. 2023
2017
4

---

Published as a conference paper at ICLR 2025
Prior attempts at auditing the hidden state model used adversaries employing loss-based attack that targets a canary obtained by flipping the label of a genuine data point (Nasr et al.] 2021 2023] |teeinke] et al. 2023 Annamalai and Cristofaro 2024). The idea is to generate an outher from in in-distribution data point so that the loss of the model on this canary is high when it is not part of the training set but drops significantly when used during training. Auditing the hidden state model with these
adversaries yields privacy lower bounds that exhibit a significant gap compared to privacy accounting upper bounds for the standard threat model (Nast et al., 2021] 2023 Steinke et al. 2023 Annamalai and Cristofaro, 2024). This gap has two possible explanations: (i) these adversaries are suboptimal and stronger adversaries exist in the hidden state model; and/or (ii) the upper bounds are not tight.
Understanding the reasons for this gap, so as to gain better knowledge of the properties of DP-SGD in the hidden state model, is the main motivation for our work. We demonstrate that gradient-crafting adversaries can provide tighter auditing results, thereby proving the validity of hypothesis (i). We show that in certain regimes, our auditing results match the privacy upper bounds, indicating that releasing intermediate models does not increase the privacy loss. However, we also identify regimes where a gap remains, thus providing compelling evidence in support of hypothesis (ii). Specifically, our findings suggest the presence of a privacy amplification by iteration phenomenon in the non-convex setting, albeit weaker compared to what is theoretically established for the convex case (Feldman al. 2018 Balle al. 2019 Bok al. 2024). but less restrictive than the results of Asoodeh and Diaz (2023) The latter require projecting the iterates onto a convex set of bounded diameter or strong enough decay term on the parameters, which are unrealistic assumptions in practical scenarios involving non-convex and peeepparamteriized models like the ones considered in our work.
In Table (Appendix B). we provide summary of prior auditing results for DP-SGD, highlighting in particular whether the considered adversaries are compatible with the hidden state threat model.
4 TRADIENT-CRAFTIN ADVERSARIES FOR THE HIDDEN STATE MODEL
In all threat models. DP-SGD enjoys a simplified interpretation as a sequence of T sum queries with bounded terms, where the (-th query corresponds to summing the clipped gradients over the mini-batch B, in Equation 2 From this perspective, privacy accounting techniques for DP-SGD are designed to accommodate any possible gradients with bounded norm C, irrespective of whether they originate from fixed dataset. This highlights key limitation of adversaries used so far in the hidden state model: due to the non-convexity of the objective function, any sequence of gradients for the canary x² is likely to be possible, but how to craft a worst-case x. is unclear (and in fact out of scope for privacy auditing). Instead, we allow the adversary to directly craft any gradient sequence, considering that there could be data point that would generate that sequence. To the best of our knowledge, there is no theoretical restriction on the gradients that neural networks can produce; therefore, we cannot rule out the possibility that some input point could generate an arbitrary sequence of gradients.
Threat model. As per the hidden state, only the final model Or is revealed while the intermediate are kept hidden. We assume that the adversary knows the model architecture, the Considering the mini-batchi to be known, as done for instance in Feldman al. (2018), allows to isolate the impact of concealing the intermediate models from other factors (such as privacy amplification by subsampling). We consider the hyperparameters DP-SGD (7, C. a) to be fixed and identical for all optimization steps to avoid uninteresting edge cases (c.g., setting the learning rate 17 to 0 in all optimization steps that do not use the crafted gradients). This requirement can be lifted by switching to Noisy-SGID in Feldman et al. 20188)] Altschuler and Talwar (0022), where hyperparameters like the learning rate or the batch size are part of the privacy definition.
models ....... 0r-1
loss function f, the initialization 6o the dataset D and the mini-batches Bp..... By
Gradient-crafting adversaries. Following the above threat model, allow the adversary craft an arbitrary gradient sequence as long as it is not a function of the intermediate models. In other words, the adversary must decide on sequence of gradients before training starts, in an offline way, to audit a complete, end-to-end training run of DP-SGD. We stress that this is in stark contrast to Nasr et al. (2023), who use muddientcraftting adversaries audit each step of SGD and then leverage composition to derive privacy lower bound for the overall training run. Their approach thus explicitly relies on the adversary having access to all intermediate models, whereas ours does not make this assumption nor use this information in any way.
When inserted, the crafted gradient is added the genuine gradients of the mini-hatch.
5

---

Published as a conference paper at ICLR 2025
Our construction helps to understand the hidden state threat model and its properties by decoupling the privacy leakage induced by a worst-case sequence of gradients from the craftability of a canary that could produce that sequence. Allowing the adversary to craft a gradient sequence directly circumvents
two issues in prior attempts to audit the hidden state model (Nasr et al. 2023 Steinke et al. 2
Saturating the gradient norm: The canary point crafted by prior adversaries is not guaranteed to saturate the gradient clipping threshold throughout training. The adversary's 's performance becomes urchitecture-deendeett for example, a small convolutional neural network has a small canary norm at the start of the training compared to a ResNet, as observed in Figure of the Appendix. Thus, for tight audit, one must tune the clipping threshold according to the canary gradient norm, which the adversary cannot access in our threat model.
Hypothesis testing: Prior adversaries need test the presence of sequence of gradients they cannot access, so they use the model's loss as a proxy confidence score (RankSamp) e in Algorithm [ lower loss implies higher confidence that a sample was used during training. By allowing the adversary to pick the sequence of gradients, we can align the choice of confidence score the way adversarial information is encoded in the gradients (as will be evident in the adversaries we propose below), thereby achieving superior testing performance.
Concrete adversary instantiations. We propose two instantiations of our gradient-craftie adversaries, which we will use to perform privacy auditing in the next section (see ):
1. Random Biased Dimension (Agc-R); The adversary picks a random dimension and crafts gradients with magnitude C' in this dimension. To test whether crafted gradients were inserted (RankSample e in Algorithm T the adversary uses the difference between 0T and 6g in that dimension the confidence score (RankSample in Algorithm
2. Simulated Biased Dimension (Agc-S): The adversary simulates the training algorithm and picks the least updated dimension. Then, it crafts gradients with magnitude C' in that dimension. To test the presence of crafted gradients, the adversary uses the same confidence score as Agc-R.
These two adversaries are designed to be as simple as possible. Other gradient constructions can be used, such as crafting a random gradient following the method proposed by Andrew et al. (2024) (see Figure [ of the appendix for a comparison with the adversaries we use). However, recent findings on privacy backdoors can further justify our choice of adversaries, where the model architecture and parameters are chosen adversarially to leak certain input points. Specifically, (eeiicalllyyegggggge and Tramer (2024) propose a specific construction of architecture and initialization in which a well-chosen input point generates gradients nearly concentrated in single dimension throughout training. This demonstrates that in some cases, our adversaries AGC-R R and AGc-S can be instantiated by inserting a canary point.
For more details on our two pradient-crafting adversaries, see Appendix C and refer to Algorithm for the corresponding auditing procedure, adapted from Algorithm
5 PRIVACY AUDITING RESULTS ON REAL DATASETS
5.1 EXPERIMENTAL SETUP
+ +Training details. We perform auditing on two datasets: we choose CIFARIO (Krizhessky 2009 as
a representative dataset for the state-of-the-art in differentially private training (Tramèr and Bonch
202T De et al. 2022), and Housing (Pace and Barry] 1997) underline the hardness of auditing smaller models in the hidden state. We use fixed hyperparameters for each dataset: on CIFAR 10, the batch size is 128, and the learning rate is 0.01, while on Housing, the batch size 400, and the learning rate is Training is done with DP-SGD with no momentum. We use three models: a fully connected neural network (FCNN) for the Housing dataset (Pace and Barry) 1997). a convolutional neural network (CovNet) (LeCun et al. 1989) and residual network (ResNet) (He et al. 2016) for CIFARIO. A detailed description of the models can be found in Appendix D
Baseline adversary. As baseline, we adopt an adversary used in prior related research (Nsrr et al. 2023; Steinke et al., 2023). This adversary selects a point from the training dataset and flips its label to generate an outlier which serves the canary x². The loss of the model on this canary is used as confidence score to test whether it was inserted or not. We refer to this bascline as the "loss- based adversary". denoted by Ar.-

---

Published as a conference paper at ICLR 2025
App-R Gol Cel
Acc- O-1
-- Au. 0-1
Auc-H, Do2
O-2
Ch22
Ar. C222
:Prriaary Audring Lower Dound]
Accil G-4
A., 0-4
Rcc Clu
Ar. C=4
Apc-R
Aces8
Au
20
16-
15
15
10-
10
10
10
15
20
(Prisnc Accumnting Upper Bound)
(Privacy Accounting Uppez Bouad)
s (Privacy Accounting Upper Bound)
(a) Auditing ConvNet
(b) Auditing ResNet
(c) Auditing FCNN.
Figure 1: Auditing results for AGe (ours) and An on ConvNet (Fig. la and ResNets (Fig. 1bj periodicity k = 1 and C E 21. In Fig. [c we present the results for AcC-R (ours), Ace-S (ours) and A on FCNN (Housing dataset) at periodicity k
Privacy accounting & auditing. We compute privacy upper and lower bounds for the crafted gradient (for Agc) or canary (AAr,) as follows. The theoretical privacy upper bound is given by the accountant of Gopi et al. (202I). We audit three scenarios where the accountant gives equivalent privacy guarantees for the crafted gradient or canary by inserting it at different periodicity k E {1, } accounting only for the steps where the insertion occurs and adjusting the time horizon accordingly. When k = 1, the model is trained for T = 250 steps. and the crafted gradient or canary is inserted every step; when
5, the model is trained for 1250 steps and insertion occurs every 5 steps: and similarly for k = 25. We report both the theoretical and empirical epsilons at fixed ò 1e-5, but note that we can generate the complete privacy curve (e, d(e)), see Figure of the appendix. For auditing, rely the recently proposed scheme based for completeness) as it allows accurate auditing with a small number of 2023). Details on training and auditing parameters can be found in Appendix D
Gaussian DP (described in Appendix [A]
auditing runs (Nasr et al.
Remark 1 (On the impact of known initialization). As explained in our threat model of Section consider the initialization 0o known to the adversary. This is a standard assumption made explicitly or implicitly by virtually all privacy accounting techniques, including those tailored the hidden state (Feldman et al.) 2018). Nevertheless, we show in Figure Figure[[2oo) the appendix that commonly used random initializations like Kaiming (He et al. (013) or Xavier (Glorot and Bengio 2010) do not significantly affect our auditing results, as they only add a bil of variance at initialization.
Remark (On pre-trained models). While we consider here that models are trained from scratch on private data, our techniques and results also apply the scenario where model pre pre-trained on public data defines a new (potentially smaller) model to be fine-tuned on private data. We illustrate this by fine-tuning pre-trained AlexNet (Krizhevsky al. (012) model by training only the last fully connected layer of the classifier on CIFARIO, see Figure
13 of the appendix.
5.2 AUDITING RESULTS FOR PERIODICITY k =
Ever-parameterizee models. The results in Figure [gsoee that adversary Age-R achieves tight auditing results in the hidden state model when the crafted gradient inserted at every step. a result that had been achieved until now only when the adversary could pick worst-case dataset /) {6) (Nasr et al. (202)) and set the learning rate to 0 in steps where the canary is not inserted, or when the adversary had access to the intermediate models (Nasr al. 2023). Note that the baseline loss-based adversary Ar, is indeed not tight and even very loose in some regimes.
The fact that tight audit can be achieved by our simplest adversary Acc-R (random biased dimension) may seem surprising. We provide a high-level explanation of this phenomenon. As we are auditing over- wer-parameterize models, which are highly redundant, we can expect genuine gradient to assign on average a magnitude of O C'mminiaatch)) to each dimension, where p is the number of parameters of the model. As p is typically orders of magnitude larger than the commonly used batch sizes, genuine gradients in a mini-batch contribute negligible magnitude to any dimension compared
7

---

Published as a conference paper at ICLR 2025
R, CoavNet
Au, Consiee
Acc ConcNet
CuncNee Ar. ResNee.
2 (Priouuc Andiing Looee Bound)
Aoc R, ResNet
Au, ResNet
(Proaay Anditing Leterr Bandd)
Age B. ResNet
8-
8-
6-
6-
2-
10
16
20
(Prinay Anventing Upper
[Prioary Accomnting Upper Bourd)
(a) Canary insertion periodicity =
(b) Canary insertion periodicity = 25
Figure 2: Auditing results for (cc()urs) and Ag on ConvNet and ResNet (CIFARIO) at privacy parameters C' = 1 at periodicity k = 5 (Figure 2a and h 25 (Figure
to the magnitude of C' inserted at a single dimension by AGc-R (or AGc-S). Therefore, genuine gradients do not interfere with the contribution of the adversary, regardless of the selected dimension. We see below that this no longer holds when auditing low- www-dimesional models.
Low-dimensional models. Auditing smaller models is more challenging because the ratio between the mini-batch size and the number of parameters is larger. To evaluate our adversaries in this regime, we switch to the Housing dataset with the FCNN model, which has only 68 parameters. We report the performance of our two adversaries over five runs in Figure Tc We observe that randomly selecting a dimension (Agc-R) no longer achieves tight results, although still outperforms the baseline loss- based adversary. Remarkably, second adversary, which simulates the training algorithm to select an appropriate dimension (Ac--S), recovers nearly tight results with small variance between runs.
To conclude, the tightness of our adversaries implies novel negative result.
Implication 1. If data point is used at every oplimization step of DP-SGD, hiding intermediate models does not amplify its privacy guarantees.
5.3 AUDITING RESULTS AT PERIODICITY k
We now study how our adversaries perform when the crafted gradient is inserted at a higher periodicity E (5, 25). We observe in Figure 2 that (i) we still outperform the baseline loss-based adversary Ag by large margin, but (ii) we no longer match the privacy upper bound, and our adversary becomes weaker as we increase k. Intuitively, the latter is due to the accumulation of the noise added on genuine gradients during the k- - 1 iterations between each crafted gradient insertion. However, privacy ac- counting (i.e., the upper bound) does not take advantage of this accumulated noise: as the insertion of crafted gradient the best one can do is to resort to the post-processing property of DP, which ensures that subsequent genuine gradients do not weaken the privacy guarantees. In the next section, we investigate whether it is possible to match the privacy accounting upper bound in this setting, which amounts to designing an adversary capable of crafting gradients that maximally influence subsequent genuine gradients.
given step could bias subsequent genuine gradients towards particular direction,
Remark 3. Using auditing via Gaussian DP (Nssr al. 2023 for k implies approxinating the trade-off function of the mechanism with a Gaussian one (see Appendix A for defimitions). While this shuffled
approximation can underestimate the privacy guarantees certain cases like subsampled mechanisms (see Dong et al. (2022); y Wang et al. (2024)), in our context it is justified by the Central Limit Theorem (Dong al. 2022), and has been used before when auditing the hidden state (Nasr)
et al. 2023 Appendix E Figure ba) shows the approximation error is indeed negligible in our case.
6
TOWARDS A WORST-CASE ADVERSARY FOR THE HIDDEN STATE
In this section, investigate whether reducing (and potentially closing) the gap with the theoretical upper bound observed Section (5 possible. For simplicity, we consider the case where a crafted gradient is inserted only in the first optimization step, and then T subsequent steps performed
8

---

Published as a conference paper at ICLR 2025
B=1
B=1
B=16
B=2
B=g
Accounting
0
8
0.91
0.99
1.00
1.00
1.00
L0
4.5
0.69
0.91
0.99
1.00
1.00
0.4
0.54
0.80
0.97
1.00
1.00
0.41
0.70
0.91
0.99
1.00
0.6
Ana Boo Raze
40 4.0
0.39
0.60
0.86
0.98
1.00
0.2
0.33
0.54
0.80
0.96
1.00
0.1
0.31
0.41
0.75
0.91
1.00
3.5
0.0
0 0.25
43
0.68
0.91
0.99
20
20
4
16
0.0
T
T
B
(a) Evolution of the auditing performance across time
(b) Privacy amplification rate
Figure 3: Auditing performance of our adversary Al across T 25 steps. Figure 3a shows the
evolution of the auditing performance across time for E 8) and batch size B E 21, 4, 8, 16). Figure 3b gives the privacy amplification rate, i.c., the ratio 25 between the privacy auditing lower bounds at step 25 (Éx=25) and at step (Ét=1).
without inserting crafted gradients. This corresponds the scenario studied by privacy amplification by iteration (Feldman et al. (018). but we do not restrict ourselves to convex losses.
We introduce new class of adversaries As that crafts the gradient and the loss landscape itself, thereby controlling the distribution of all T updates. Therefore, As can pick a (non-convex) loss landscape such that the crafted gradient inserted step T = 1 maximally biases subsequent genuine gradients. thereby yielding the highest possible privacy loss for the final model. Note the underlying principle is the same as in Section 4 instead of requiring the adversary craft dataset, model architecture and loss function, we allow the adversary to select the loss landscape directly, considering that there could be a dataset that would generate this landscape. As long as the loss landscape is selected before training starts, this is covered by the hidden state threat model we consider.
We formalize the above using the following pair of one-dimensional stochastic processes:
1 0,+1 0: B (o(),) + Z(+1),
1 ii41 0, B (9(0.) + Z1+1),
(3)
where the first step is either 01 = 71 (a process that did not use the crafted gradient) or 01 V Z (a process that used the crafted gradient V*), with Z, ~ NNO0, C² and g R - [CBC, BC] is function that abstracts the sum of gradients of the loss a batch of size B in the DP-SGD update (Equation 2). The goal of the adversary is to choose the function g and the gradient V*, without know- ing the intermediate updates, making the distribution of Or and Or as "distinguishable" as possible. Example 1 (Constant 9)- Consider the simple case where g outputs constant c, independent of the imput. This implies that the Gaussian noise accumulates at every step, and the privacy loss e for the last iterate converges to as T + oo at rate of O(1/VT).
As illustrated by [xample[]] it is easy to design a function g that amplifies privacy over time. However, the converse objective, i.e., designing a function that provides no or minimal privacy amplification for the final step T, appears much more challenging.
A worst-case proposal. We propose concrete adversary Al'' which selects V = C' and the function go. (X) = BC if (X) and go- (X) = BC otherwise, where o² R + {0, 1} is the linear threshold function that achieves the minimal False Positive Error Rate a + False Negative Error Rate 3 for distinguishing between 01 and 0, (i.c., testing if the model after 1 step was generated using the crafted gradient V*). More precisely, 6" (XX if X > h² and (X) = otherwise, with 2 2 (see Appendix | for a numerical validation). The resulting non-convex loss landscape adheres to the intuition that the crafted gradient maximally biases subsequent steps (see Appendix (F).
ht" h²= D8-
Auditing with A Figure 3a shows the auditing performance of our adversary Al for T E
{1..... 25} steps, batch sizes B E {1, 2.4 8, 16] and noise variances a E {1,8) (for more values Figure 14 in the appendix) Figure 3b gives the ratio between the privacy auditing lower bound
need to approximate the trade-off function of our mechanism with Gaussian one (see Remark In practice, the approximation error again negligible (see Figure oblin Appendix P.
3 As in Section

---

Published as a conference paper at ICLR 2025
Er=25 at the last step and the one obtained after the first step, to measure the extent of the privacy am- plification phenomenon for various batch sizes B and values of o. When this ratio is equal to 1, there is no amplification and the audit is tight, while values smaller than I indicate the existence of a privacy amplification effect. We observe two general trends: the amplification rate increases with a and decreases with B. This can be explained follows. Higher noise variance (a) helps forget initial con- ditions in the stochastic processes (3) Conversely, a larger batch size (B) allows subsequent updates to be larger, enabling the initial conditions propagate better across iterations despite the noise. Indeed, as gradient updates are bounded by BC, BC| and the noise remains independent of B, increasing B allows the adversary to retain more "signal" about the canary's 's presence in subsequent gradients.
Unlike the case where the crafted gradient is inserted at every step (see Implication D. the above results reveal two distinct two regimes, as highlighted in Implications and helow. On the one hand, when the batch size B is large enough relative to the noise variance o, our adversary Ale achieves a tight audit. In this regime, this adversary is thus optimal and there is no privacy amplification.
Implication 2. For a sufficiently large batch size relative to the noise variance, the privacy accounting of DP-SGD is tight in the hidden state model: hiding intermediate updates does not amplify privacy.
On the other hand, privacy amplification seems occur when the batch size is small enough relative to the noise variance. As seen from Figure 3a the privacy loss of A converges to constant: this is because the privacy loss distributions become sufficiently far away and stop mixing over time. This constant is smaller than the upper bound from privacy accounting, but remains positive, indicating that this potential amplification must be qualitatively weaker than in the convex case.
Implication 3. In the non-convex regime, for a sufficiently small batch size relative to the noise variance, hiding intermediate updates may amplify the privacy guarantees of DP-SGD. However, the privacy loss of sample does not go to O as subsequent steps are performed, in contrast to the convex case where the privacy loss decreases in O(1/VT) (Feldman et al. 2018).
Remark (Comparison to parallel work). Implication 2 was independently established in a recent concurrent work by Annamalai (2024). Their worst-case construction differs from ours, and in particular implicitly assumes that the batch size is large relative to the noise variance. As a consequence, their results overlook the fact that privacy amplification can occur under small batch sizes, which is a practically important regime where stronger privacy guarantees may be achievable.
DISCUSSION AND FUTURE WORK
Our work allows tighter auditing of DP-SGD in the hidden state model, and our results yield sev- eral implications that advance the understanding of this threat model. For instance, Implication 3 means that the property of converging privacy loss of DP-SGD as T + +oo recently established by Altschuler and Talwar (2022) in the convex case cannot hold for non-convex models without additional constraints. Beyond this negative result, our work suggests that a (weaker) form of privacy amplifica- tion does occur for non-conver problems when the batch size is small relative the noise variance.
Beyond differential privacy, our results have consequences for machine unlcarning (Bourtoule et al. 2021). which aims to remove a specific data point from a trained model as if it had never been used for training. A popular definition for unlearning is inspired by DP (Guo et al.) 20200; Neel et al. [021]] Gupta et al. 2021), with recent work relying on noisy training and privacy amplification by iteration to provide unlearning guarantees (Chien et al. 2024). Implication shows that this approach cannot completely unlearn a data point when considering non-convex models like neural networks.
An intriguing open question arises from the gap between our empirical results real datasets (Section 55) and the adversary crafting the loss landscape (Section(6) () when the canary is not inserted at every step. While recent work on privacy backdoors shows that certain architectures can be manipulated to make a specific data point produce the desired gradient sequence throughout training (Feng and Tramer 2024), it seems unlikely that this point could also maximally bias subsequent gradients of gen- uine training points in the same way as our worst-case adversary AB" Investigating the extent to which this is feasible in commonly used architectures would enhance our understanding of privacy leakage in deep learning, and could ultimately lead to better privacy accounting for the hidden state model.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Pierre Stock and Alexandre Sablayrolles for numerous helpful discussions and feedback in the early stages of this project and Santiago Zanella- Zanella-Béguelin for observing the need to fix the hyperparameters for DP-SGD in our threat model. We want to thank members of the CleverHans Lab for their feedback.
The work of Tudor Cebere and Aurélien Bellet is supported by grant ANR-20-CE23-001 (Project PRIDE) and the ANR 22-PECY-0002 IPOP (Interdisciplinary Project on Privacy) project of the Cybersecurity PEPR. This work was performed using HPC resources from GENCI-IDRIS (Grant 2023-AD01 101 1))
The work of Nicolas Papernot and Tudor Cebere (while visiting the University of Toronto and Vector Institute) is supported by Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, DARPA through the GARD project, Intel, Meta, NSERC through the Discovery Grant, the Ontario Early Researcher Award, and the Sloan Foundation.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016.
Jason M. Altschuler and Kunal Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. In NeurlPS, 2022.
Galen Andrew, Peter Kairouz, Sewoong Oh, Alina Oprea, H. Brendan McMahan, and Vinith Suriyaku- mar. One-shot empirical privacy estimation for federated learning. In ICLR, 2024.
Meenatchi Sundaram Muthu Selva Annamalai. It's our loss: No privacy amplification for hidden state dp-sgd with non-convex loss, 2024.
abs/ 2407. 0649 96
Meenatchi Sundaram Muthu Selva Annamalai and Emiliano De Cristofaro. Nearly tight black-box auditing of differentially private machine learning. arXiv preprint arXiv:2405. /4/06, 2024.
Shahab Asoodeh and Mario Diaz. Privacy loss of noisy stochastic gradient descent might converge even for non-convex losses. arXiv preprint arXiv:2305.00 09903, 2023.
Borja Balle, Gilles Barthe, Marco Gaboardi, and Joseph Geumlek. Privacy amplification by mixing and diffusion mechanisms. In NeurIPS, 2019.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In IEEE Symposium on Security and Privacy (S&P), 2022.
Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds. In FOCS, 2014.
Jinho Bok, Weijie J Su, and Jason Altschuler. Shifted interpolation for differential privacy. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choot Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159, IEEE, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In IEEE Symposium on Security and Privacy (S&P), pages 1897-1914. IEEE, 2022.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Stochastic gradient langevin unlearning. arXiv preprint arXiv:2403. 17.4000.771055 /7105, 2024.
C. J. Clopper and E. S. Pearson. The use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika, 1934.
11

---

Published as a conference paper at ICLR 2025
Edwige Cyffers and Aurélien Bellet. Privacy amplification by decentralization. In AISTATS, 2022.
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlocking high- accuracy differentially private image classification through scale. arXiv preprint 1rXiv:224.133500 2022.
Jinshuo Dong, Aaron Roth, and Weijie J. Su. Gaussian differential privacy. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 84(1):3-37, 2022.
Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. In Privacy Enhancing Technologies Symposium (PETS), 2022.
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trendso in Theoretical Computer Science, 9(3-4)2211-007 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryplography, pages 265-284, Berlin, Heidelberg. 2006. Springer Berlin Heidelberg. ISBN 978-3-540-32232-5..
Ülfar Erlingsson, Vitaly Feldman, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In SODA, 2019.
Vitaly Feldman, Ilya Mironov, Kunal Talwar, and Abhradeep Thakurta. iteration. In FOCS, 2018.
Privacy amplification by
Shanglun Feng and Florian Tramèr. Privacy backdoors: Stealing data with corrupted pretrained models. arXiv preprint arXiv:2404.00473, 2024.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. In NeurlPS, 2021.
Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In /CML, 2020.
Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saced Sharifi-Malvappe and Chris Waites. Adaptive machine unlearning. In NeurlPS, 2021.
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. NeurlPS, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John Pearson, Dietrich Stephan, Stanley Nelson, and David W. Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLOS Genetics, 2008, doi: 10. 10. /.37//joumal..gennn. wgw..0m0........00116...11 1000167.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33: 22205-22216, 2020.
12

---

Published as a conference paper at ICLR 2025
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In ICML, pages 1376-1388 PMLR, 2015.
Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM Journal on Compuring. 40(3):793- 826, 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1. Curran Associates Inc., 2012.
Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989. doi: 10. 162/neco. 1989. 1.4.541.
Fred Lu, Joseph Munoz, Maya Fuchs, Tyler LeBlond, Elliott Zaresky-Willle Williams, Edward Raff, Francis Ferraro, and Brian Testa. A general framework for auditing differentially private machine learning. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 4165-4176. Curran Associates, Inc., 2022.
Samuel Maddock, Alexandre Sablayrolles, and Pierre Stock. CANIFE: Crafting canaries for empirical privacy measurement federated learning. In The Eleventh International Conference on Learning Representations, 2023.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP), pages
IEEE, 2021.
Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. In USENIX Security, 2023.
Seth Neel, Aaron Roth, and Saeed Sharifi Shaviei-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In ALT, 2021.
Jerzy Neyman, Egon Sharpe Pearson, and Karl Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 2316694--00;2289-337 1933. doi: htt/9///ssa.133.0000
Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33 (3):291-297, 1997.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18, 2017. doi: 10.1109//P.2017.4.1
Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pages 245-248, 2013. doi: 10. 109//GobaalSPP22111776666
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. In NeurlPS, 2023.
Florian Tramèr and Dan Boneh. Differentially private learning needs better features (or much more data). In ICLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Chendi Wang, Buxin Su, Jiayuan Ye, Reza Shokri, and Weijie Su. Unified enhancement of privacy bounds for mixture mechanisms via f-differential privacy. Advances in Neural Information Processing Systems, 2024.
Jiayuan Ye and Reza Shokri. Differentially private learning needs hidden state (or much faster convergence). Advances in Neural Information Processing Systems, 35:703-715, 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in Py PyTorch. arXiv preprint vrXiv:2109.122900 2021.
Santiago Hanella-Bégueling Lukas Wutschitz, Shruti Tople, Ahmed Salem, Victor Rühle, Andrew Paverd, Mohammad Naseri, Boris Köpf, and Daniel Jones. Bayesian estimation of differential privacy, 2023.
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference by boosting relativity, 2023. arXiv:23 arXiv:2312.03262. 12.03262.
14

---

Published as a conference paper at ICLR 2025
A DETAILS ON AUDITING VIA GAUSSIAN DP
For completeness, in this section we present the privacy auditing approach proposed by Nasr et al. (2023), which we use in our experiments. We start by introducing some necessary concepts on f-DP and Gaussian DP in Section A.I Then, in Section A.2 we present an overview of how auditing is performed and how all the elements of the general auditing pipeline (Algorithm) T are implemented.
A.1 f J-DIFFERENTIAL PRIVACY AND GAUSSIAN DIFFERENTIAL PRIVACY
f-DP and Gaussian DP (Dong al. 2022) are based on characterization of DP as binary hypothesis testing. We recall the main concepts below.
Definition 2 (Error rates). Lei P and Q be two arbitrary distributions and O a sample drawn from either P or Q. Let binary hypothesis test be defined by the following two hypotheses: Ho "the output O is drawn from P" or "H1: the output O is drawn from Q)" Consider a rejection rule R + [0, 1] that outputs the probability that we should reject Ho. We define the Type (or false
positive) error rate of this rejection rule one Blo) and the Type (or false negative) error rale
as 3/3 Elol
Definition (Trade-off functions). Let and Q be two arbitrary distributions. We define the trade-off function between P and Q as:
T(P, ()() =nf as S sa.
(4)
We can now introduce f-DP.
Definition (f-Differential Privacy). Let f be a trade-off function. Then mechanism M f-Differentially Private (f-DP) ifr all neighbouring datasets D and D':
(MM(D),(D')) 2
(5)
Trade-off functions describe the lowest Type II error rate achievable by any adversary at any Type I error rate. We say that mechanism M1 strictly less private than M2 if (M, (D), Mr (D))(a) s
(DM, M2 M2)))))(a) for all a E [0, 1]. Dong et al. (2022) show (s, 6DD can be formulated as
f-DP as follows:
(a) max 1 , (1- 8)}
(6)
Gaussian Differential Privacy (GDP) is specialized formulation of f-DP where is the trade-off function of two Gaussian distributions.
Definition (Gaussian Differential Privacy). Let t be the cumulative distribution function (CDF) of AO 1) and Φ- its associated quantile funcction. A mechanism M is p-GDP if it is G -DP, where
Ga(a) e T(M(D), M(D))a -00 p1).
The following result is crucial for auditing: it shows that the error rates of adversary give lower bound on the GDP guarantees.
Lemma (Relating error rates to GDP). Assume that an adversary has error rates GM 8yg in the binary test defined by P (D) and Q M(D)) where M is mechanism and D and D' are neighboring datasets. Then, if M satisfies μ-GDP, then
# 2 Φ (1 a) -h-1(
(8)
Finally, we can convert GDGDP to (e, DD.
Corollary 1 (From GDGDP to (s, satisfies (e, 8)- where:
(o Diiferrentil Privacy.). If mechanism M is μ-GDP then
E 11 e c
8(c) Φ
(9)
# 2
4 2
15

---

Published as a conference paper at ICLR 2025
A.2 AUDITING VIA GDP
Auditing a mechanism M via GDP follows the general approach outlined in Algorithm amounts instantiating its different primitives:
| and
1. RankSample: We first generate R outputs of M, half of them sampled from (D), the rest from MD'), where D' = DU {.*}. For each output i E {1,..., R), a score S; is computed to the adversary's confidence that either 1) TY' were used as input for M. The precise computation of this confidence score depends on the considered adversary and is abstracted by the Rank ampll method in Algorithm [ We refer Sections for details about the RankSample le method used by the adversaries we consider.
2. Reject ionRule: Each S; is then augmented with its associated ground truth label b,, reflecting if particular sample has been generated from D or D'. The adversary selects a classifier (Re ject i onRule in Algorithm 1 that receives the set S as imput and needs to classify it against the ground truth label b. We stress that DP holds against any choice of such classifier. Commonly used ones are linear threshold classifiers (Nasr et al. (023)) Once threshold classifier has been selected, we evaluate the error rates of the classifier using S and 6 as the ground truth labels, evaluating the False Negative (FN), False Positive (FP), True Negative (TN), and True Positive (TP) of the classifier.
3. ConfInterval: Using these statistics, we employ the Clopper Pearson (Clopper and Pearson 1934) confidence intervals over binomial distributions to get high-probability lower bound on the Type error rate (a) and Type error rate (3) for the adversary's performance. We note that other techniques exist to compute such confidence intervals
et al.) 2023 Lu et al. 2022); ; however, the benefits we observed practically were negligible in our case.
4. ConvertToDP: Finally, given access the lower bounds on the error rates of the adversary, we can compute the lower bound on the privacy loss of a mechanism M in GDP by using Lemma and translate the result to (e, -DD using Corollary |
We stress that this technique offers lower bound on the privacy loss, while techniques like privacy accounting (Abadi et al. 2016 Gopi et al., 2021 Doroshenko et af.] (022) or composition (Dwork)
et al. 2006 Kairouz al. [015 Dong et al. [2022) offer upper bounds on the privacy loss. We call
mechanism light if a lower bound offered by auditing match the upper bound
B SUMMARY OF PRIOR WORK ON PRIVACY AUDITING OF DP-SGD
We provide in Table list of adversaries used in prior work on privacy auditing of DP-SGD along with their key properties: whether they are compatible with the hidden state, whether they achieved tight audit, and the type of canary they used.
C DETAILS ABOUT OUR ADVERSARIES AgC
The algorithmic description of our adversaries AcC-R and Acc-S are given in Algorithms 2
We discuss different strategies to select the biased dimension for our adversary Acc-S. The general idea is to explore how the different dimensions are updated by simulating the training process, as all the required knowledge is available to the adversary. We have considered two types of simulations: (i) Noisy simulation, which we use the same noise variance as the simulated process. and (ii) Noiseless simulation, in which we run the training algorithm without noise or clipping. Note that for the noiseless simulation, the training algorithm is deterministic, as the initialization and the batches are fixed and known to the adversary, while in the noisy simulation, the stochasticity of DP-SGD is concealed for the adversary. We use four simulations of the training process. We then proceed to choose how to rank the most suitable dimension to bias, for which we have explored two options: (i) Per Step (PS): selecting the dimension that has the lowest gradient norm accumulated over all the training runs (ii) Final Model (FM): selecting the dimension that is the closest to initialization at the end of the training run. We experiment with these design choices, resulting in 4 dimension selection strategies. The results in Figure show that the Noiseless-PS strategy performs best.
16

---

Published as a conference paper at ICLR 2025
Paper Jagielski et al. 2020)
Adversary
HS
TA
CT
Poisoned Backdoor
X
Sample
API Access
Sample
Static Adversary
Sample
Intermediate Poison Attack
X
Sample
Nasr et (2021
Adaptive Poisoning Attack
X
Sample
Gradient Attack
X
X
Gradient
Malicious Datasets
Gradient
Experiment I
X
Sample
Zanella-Béguell et al.
2023
Experiment 2
Sample
Maddock al. (2023)
Algorithm
x
Gradient
Algorithm
Sample
Nasr et al. 2023
Algorithm
x
Gradient
Algorithm
7
Sample Gradient
Andrew et al.
2024
Algorithm
x
Gradient
audit- tudit-yypewhitebox
X
Sample or Gradient
Steinke et al.
2003
audi-type=blackbox
X
Sample
AGC-R, k AGC- R, k
Gradient
Ours
J X
Gradient
Table 1: Summary of adversaries used in prior privacy auditing of DP-SGD. We report whether the adversary is compatible with the hidden state (HS). if the adversary achieves tight audit in the threat model it operates (TA) and the type of canary (CT) used by the adversary (gradient canaries sample space canaries).
(Pinaa Awiling Loves
Caride,
Aprk tedo
Apd Oculfic L-5
10
Dudiccl 2-2
Ape Danies, bws
10 15
20
(Proway Accouringg Upper Bound)
8 (Privacy Accounting Upper Bound)
Figure 5: Auditing results for Acc-S compared to Figure 4: Auditing results of A,, AcG-R AGG-R at k E {1,5} on ConvNet and ResNet18. We and Age-S on the Housing dataset. We observe that the two adversaries are equivalent for these consider 4 variants of A depending on over-parameterized models, demonstrating that Agc-S whether the noisy or noiseless simulation only enhances attack against paddteearaeeter used and whether we rank dimensions based models. on accumulating per-step updates (PS) or on the final model norm difference (FM).
In Figure 5 we show that Asc-S using the best strategy is equivalent with AGC-R on our studied over-parameterized models.
Algorithm Gradient Generation for AgeRR (Random Biased Dimension)
Input: dataset D, initialization 6n IR,, clipping threshold C - random element from pp)
(0, .,..... 0) E
V dd] k C
return V d
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Noisy gradient generation for Agc-S (Simulated Biased Dimension)
Input: dataset D. initialization Gu IR,, clipping threshold C, noise variance o², mini-batches (B ...
.B...Brl Br number of runs r, parameter selection rank E (PerStep. FinalModel]. simulation E (Noisy, Noiseless}
S [0] for j = 1 to r do for to T do
Z: (O0 CI
0.11 t 01
Σ clip clip (0o BEB.
[.TT
if simulation = Noisy then
0.41 0c11 |a Z, where ~ x7
end if
if rank PerStep then S S + (0, 0;-1)
end if end for
if rank = FinalModel then
5 <- 0r 6g
else
S 2- V
end if end for argmin S
V + (0....,0) E R [d] -
return V*, d
18

---

Published as a conference paper at ICLR 2025
Algorithm 4 Privacy auditing with our gradient-crafting adversaries
Input: dataset D,
(Bi... .B....By B, periodicity k, number of auditing runs R, privacy parameter ó, noise variance o², clipping norm C, initial parameters 0o E R', standard Gaussian CDF Φ, gradient-craftnng adversary Agc
v Agc
> get crafted gradient and biased dimension from adversary
S I1 b +-11
j = 1 to do
6; f Ber(
draw a random bit
for E [1, , do
O2+1 =0.
7
Σ
clip (0,3() Zz 2e ~ ~(0,00)
[B
E,EB,
if mod 0 AND b,
then
0.11 0,+1
" |B,1
V
Canary is inserted
end if
end for
S. Fr [d] [d]
Retrieve the model value biased dimension d
end for
Swited (Sr
H +- h1 Suand
i== 1, I,..., S 1
for h2 E H do
b - {1 [S S hg] 1 1= |S|}
TN, TP, FN, FP + Confusionoatriinneeeee
ag 3; - CCooppePPaaronnTTN TP, FN,
#4 - $ (1-'11 - 0:) Φ (B.)
é - --agminn
+ =
+
U4 2
+
c
+
= --m))
Minimization via binary search
Mi
end for return max (E:: 1i:i= I,...,RY R}
D EXPERIMENTAL DETAILS ON TRAINING & AUDITING
In our experimental section, we audit three neural network architectures, a ConvNet and ResNet on the CIFARIO dataset and a Fully-Connected Neural Network on the Housing dataset. These models were implemented using PyTorch (Paszke et al., 2019), and the DP-SGD optimizer was Opacus (Yousefpour al., (222). The hyperparameters for training, privacy guarantees, and privacy auditing parameters for each audited machine learning model are meticulously outlined Table 2 The ConvNet and ResNet have been audited when trained for 250, 1250, and 6250 optimization steps, totaling approximately 3500 GPU hours.
For clarity, in Algorithm 4 we detail the Gaussian DP-based auditing procedure used for gradient- crafting adversaries (A or -G) For each experiment, we audit R = 5000 machine learning models using GDP auditing (see Appendix AA. For the Clopper Pearson confidence intervals,
a confidence of 95% (consistent with prior work (Jagielski et al. 2020 Nasr et al., 2021) to get lower bound on the Type error rate (a) and Type II error rate (B) for the adversary 's performance. Re Reject i onRule is computed by selecting the best linear threshold classifier on S. While, in practice, the adversary should pick the classifier on a set of held-out scores S, we select the classifier directly on 5 to save the computational resources required to compute the held-out set S. We stress that it is not an issue, as differential privacy guarantees hold against any choice of classifier.
E APPROXIMATION ERRORS FOR GDP AUDITING
As explained in Remark 3] using auditing via Gaussian DP (ast et al. 2023) (see Appendix A implies approximating the trade-off function of our mechanism with Gaussian one (Dong al.
19

---

Published as a conference paper at ICLR 2025
HYPERPARAMETER
FULLY- FULLY-CONNECTED NN
CONVNET
RESNET18
LEARNING RATE (1))
10-2
10-2
10-2
BATCH SIZE
400
128
128
TRAINABLE PARAMS
68
62006
11173962
LOSS FUNCTION
BINARY CROSS ENTROPY
CROSS ENTROPY
CROSS ENTROPY
CLIPPING NORM C
I.0
[T.0, 2.0, 4.0
[1.0, 2.0, 4
NOISE VARIANCE
4
FAILURE PROBABILITY
10
10-3
10
AUDITING RUNS
5000
5000
5000
CONFIDENCE INTERVAL
0.95
0.95
0.95
Table 2: Hyperparameters used per model
Observed tradeoff curve
Observed tradeoff curve
Our approcimation via GDP
Our approximation via GDP
1.0
1.0
0.8
0.8
Type 11 Error Rate
Type Error Rate
0.6
0.6
0.4
0.4
0.2
0.2
0.0 0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Type Error Rate
Type Error Rate
(a) The approximation стror of approximating the trade- off function of the observations via GDP for Age-R when auditing ConvNet CIFARIO when ls = step 1250.
(b) The approximation error of approximating the trade- off function of the observations GDP for A when the T = 25 case at privacy parameter C' = and 4.
2022). In this section, we present some samples of the privacy trade-off curves that we observe when auditing both Agc Figure and A in Figure 6b For Agc. we show the case where 1250 when auditing a ConvNet at canary gradient insertion periodicity & = 5 on CIFARIO, while for Aá. we present the approximation when T = 25. We see that the approximation errors are negligible, and the approximation technique is representative.
This good behaviour is due to the Central Limit Theorem for f f-DP composition (Dong et al. 2022), which, in nutshell, states that while individual mechanisms are not well approximated by Gaussian Mechanism, their composition has decaying error in the number of compositions performed when approximated via Gaussian Mechanism.
F DETAILS ON OUR ADVERSARY Aj.
Visualization of the loss landscape. Figure 10 shows the loss landscape corresponding to our adversary Ai defined in Section 6 One see how the crafted gradient can bias subsequent steps. Note that the discontinuity not essential and can removed by appropriate scaling.
Numerical validation of optimality of h²" In this section, numerically validate that the choice of threshold h² in A's' indeed optimal for T' = 2 by comparing it with all possible threshold functions. To search for these functions, we discretize the Type I error rate a E [o, 1] into {ai... ard) and each a; look for the rejection rule o; that achieves the lowest Type II error rate 3;. Given that, in this case, our function distinguishes between two Gaussian distributions via Neyman PPaason (Neyman al. 1933), it is well known that the optimal classifier with fixed type I error rate is achieved by its
20

---

Published as a conference paper at ICLR 2025
Thnosled Seadd
Prisacy Aucunting
Prioee Ciurantes
Algorithm 6 Lower bound search routine over multiple threshold classifiers.
{ar ... 0 } linspace Φ ( ()
for a; E { [1] a4] do
h; 4b - 04)
1.0
1.5
2.0
2.5
3.01
3.5
4.0
4.5
5.0
Note: Varianoe g
& Audit(gén.
end for return max(f)
Figure 6: Comparison between the privacy ac- counting upper bound and the auditing perfor- mance of A and our threshold search algorithm when 2. All results are averaged across five training runs
8 (Privacy Auditing Lower Bound)
A
Acus
A
ResNel
Cony Net
20
10a
Canary Sensitivity
10a
10
10
500
1000
1300
2000
2500
3000
10
15
20
Number of Sreps
8 (Privacy Accounting Upper Bound)
Figure 8: The gradient norm the canary varies along the optimization path and for different ar- chitectures. We report the mean and 2 standard deviations across 30 random runs.
Figure 7: Comparison of privacy auditing perfor- mance for AGC-R (ours), Ar and Acor (An- drew et al. (2024) when auditing a FCNN model on the Housing dataset.
associated threshold classifier h; = #b1 -x). where & is the CDF of (0, o²). With (h)... ... ha) at hand, we can define multiple Pø., and implicitly multiple adversaries A that use Ig. as a candidate for g Eq. 3 which we use to audit, resulting in fé), out of which we pick the maximum. detailed description is provided in Algorithm
The results shown in Figure confirm that Al"" employs the optimal threshold classifier to distinguish the input distributions.
G ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results mentioned in various parts of the main text.
Alternative gradient-crafting strategy. The adversaries that we propose in Section 4 and use in Sec- tion construct gradient biased in single dimension chosen by the adversary. We investigate here an alternative approach inspired by the auditing method of AfdAnd et al. (024)). In their approach, the crafted gradient is randomly sampled from the unit sphere and then scaled by the sensitivity C. For the confidence score (RankSame) Sample in Algorithm| []. the adversary computes the cosine similarity between the canary gradient and the change in model parameters -0. which is compatible with the hidden state model. For completeness, Algorithms 7 formally show how the generation of the gra- dient and the computation of the confidence is performed. We denote this adversary by Acots.
21

---

Published as a conference paper at ICLR 2025
Algorithm 7 Rankee for Açoss
Algorithm 8 Gradient Generation for Acois
1: Input: Model Or E RR, Initialisation Og.
1: Input: dataset D, initialization Og E IR?, clipping threshold C
canary V
2: U <- 0T 80 3: return (V', UU/([IVV|l> x (Ull22
2: V + (0, Tp)
V V/IV
4: return C X V
Figure 9: Algorithms used for auditing via Acus: (a) RankSample and (b) Gradient Generation.
We observe experimentally that Acc outperform Acots and that both approaches outperform the loss based adversary AL. see the results in Figure 7
Experiments on auditing with unknown initialization. As common in differential privacy, we consider throughout the paper that the initialization Op. We discuss here the impact of considering the initialization to be unknown. First, we note that standard initialization techniques like Glorot (Glorot and Bengio 2010) or Xavier (He et al.) 2013), which are commonly used to stabilize the training process by preventing the vanishing exploding gradients phenomena, sample random parameters from a uniform or normal distribution with a small variance. Therefore, such initializations only add bit of variance. Figure 12 shows the results of using AGc-R audit CNN model CIFAR-10 with unknown initialization from Xavier. We observe no quantitative difference in the performance of Agc-R compared the case where the initialization is fixed and known.
Experiments on auditing pre-trained models. In Figure 12 we present the results of AGG-R to audit pretrained models. We audit AlexNet model (Krizhevsky al., 2012), model pre-trained on ImageNet (treated as public data) and fine-tuned on CIFAR-IO (treated as private data), compared CNN trained from scratch directly on CIFAR-I0 10 with no frozen parameters. We see that Ago-R performs tight auditing for k = in this scenario as well.
Privacy profiles. While in all of our experiments, we report both the theoretical epsilon provided by the privacy accounting and the empirical epsilon at a fixed = 1e-5, we can generate the entire privacy profile curve + 5(e) and thus produce auditing results at an arbitrary d. Indeed, our auditing technique computes lower bound on the μ-GDP parameter (see Section A for more details). Using the lower bound on y in Corollary [ we can compute any 8(e). To compute at a fixed 8, we use line- search algorithm like the bisection method, as the function 6(=) is monotonically increasing in In Figure I1 we present three privacy profile curves E + 8(s) corresponding to the performance of Age- R when auditing CNN with & E (,)). This confirms that for the case 1, are indeed tight in all privacy regimes, while for > AGG-R R is not tight for all 2 0, as discussed Section S
Detailed results for A In Figure 14 we present all the training curves that generated the heatmap in Figure 3b by computing the empirical privacy loss generated by Ali for all a E (I....,8) and B E [T, 2, 4, 8, 16). As reference, we display the theoretical epsilon provided by privacy accounting at each a. In all plots we observe that the regime B 16 yields tight auditing, and this is also the case for larger batch sizes (not shown avoid clutter).
22

---

Published as a conference paper at ICLR 2025
Grodieat Value
Locs Landsoape
km
Actsuting Uppetioud
1.0
0.3
0.6
-0.
-10
0.2
-20
10
10
10
10
Privocy Loes
Figure Privacy profiles of AGC-R for k {1, 5, 25), C = 1 on CNN trained on CIFARIO on the last step of training (7 = 250 for k = 1,
Figure 10: Visualization of the loss landscape corresponding to A for C'
T = 1250 for lo 5, T 6150 for l 25).
Lower Paund
Age-R Known Init
Unkmorm
Inctuiig AxeB: fill triniig
(Phiowy buikingg
15
on
(Prnacy Aocomting Bound)
(rooy Accomeing Upper Semal)
Auditing results for Agc on CNN trained on CIFAR10 when the initialisation of the neural network is unknown to the ad advessrry compared to the case of known initialisation.
Figure 12:
Figure 13: Auditing results for AGG-R when k = on a pre-trained AlexNet model is fine- tuned on CIFARIO.
B=1
B=2
B=1
B=8
B=16
Accounting
0=
0=2
0=3
0=1
40 4.0
1.5
1.0
0.5
3.5
1.0
0.3
0.0
20
20
0 6
0=7
0 8
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
20
0
20
20
20
T
T
T
T
Figure 14: Auditing performance of A's over time T' = 25 for various batch sizes B E {1, 2,,,8 and noise levels {1...88 .8. used generate Figure 3b
23


Reviews:
[{"review_id": "lXalOk7mcN", "reviewer": "Reviewer_PjHW", "summary": "In the \"hidden state\" threat model for DP-SGD, an adversary does not have access to intermediate updates and can see only the final model. This paper proposes to audit the hidden state threat model for DP-SGD by introducing adversaries who select a gradient sequence offline (i.e., ahead of when training starts) in order to maximize the privacy loss of the final model even without access to the intermediate models.", "questions": "* There might not exist a worst-case datapoint that can saturate the gradient clipping threshold at every iteration. Would it be possible to discuss what it means to breach the privacy of a datapoint that does not exist and may in fact be unrealizable?\n* It seems like an adversary who could control the sequence of gradients would also be able to infer from that something about the intermediate models. Is this OK for the hidden state threat model?\n* Is there any intuition behind privacy amplification only seems to occur for smaller batch sizes?\n* Does Implication 1 say anything beyond what Implication 2 says? If the batch size is as large as possible (the entire dataset), then every datapoint will be included in every optimization step of DP-SGD?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem setting and results are really interesting and open up new avenues for future work. Identifying different regimes where the gap (between the new auditing lower bounds and the theoretical upper bounds) vanishes and where the gap remains is a nice contribution.\n* The paper flows nicely and is and enjoyable to read.\n* Simplicity is a virtue and I think the design of the gradient-crafting adversaries for privacy auditing is clever yet also intuitive.", "weaknesses": "* The paper uncovers some interesting empirical results but doesn’t offer up much explanation for them. E.g. for regimes where there is a gap, we don’t really get an explanation as to why this gap exists. I do feel that this work falls short of its goal to “enhance our understanding of privacy leakage” (line 537) by reporting the results without interpretation. I think that including more discussions like the “high-level explanation” starting at line 373 would help round out the paper and provide more concrete directions for future work.\n* Considering this is one of its main contributions, I found that the description of the adversaries and the privacy auditing scheme in the main paper is vague and not presented with full details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncElPs1UEt", "reviewer": "Reviewer_aBWh", "summary": "This paper proposes a tighter privacy auditing bound by extending gradient-crafting (GC) models to the hidden state regime, where only the last iteration is released. By combining the GC technique with advanced privacy auditing methods, such as privacy auditing using \n$f$-DP, the authors achieve a refined privacy auditing bound for the hidden state model.", "questions": "My question is about the $f$-DP part. In Remark 3, the authors claim that the approximation error from using the central limit theorem (CLT) can be ignored. However, the CLT may underestimate privacy when mixture distributions arise due to shuffling or sub-sampling [A]. For this reason, Nasr et al. (2023) adopt numerical methods, such as FFT, to calculate the privacy profile. For the hidden state model, an \n$f$-DP guarantee is provided by [B] without relying on the CLT.\n\nGiven these considerations, the authors' assertion that the CLT approximation error can be ignored may not be universally applicable, especially in scenarios involving mixture distributions in (shuffled) DP-SGD. A simple explanation in the main part might be beneficial to the readers.\n\n[A] Unified Enhancement of Privacy Bounds for Mixture Mechanisms via f-Differential Privacy. Wang et al., NeurIPS, 2023.\n\n[B] Shifted Interpolation for Differential Privacy. Bok et al., ICML, 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper applies GC models to audit various regimes, including both small and over-parameterized models. Additionally, different deep learning architectures (CNN, ResNet, FCNN) are evaluated in the experiments. Overall, the empirical results are convincing.", "weaknesses": "The main weakness of this paper is that it does not introduce a new privacy auditing method; rather, it extends the existing gradient-crafting method to the hidden state regime. The primary technical contribution appears to be constructing a sequence of gradients without requiring knowledge of the intermediate gradients at each iteration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CRkQxm9SMr", "reviewer": "Reviewer_xgje", "summary": "This paper studies the auditing of last iterate privacy guarantees of DP-SGD for non-convex loss functions. Authors propose an auditing method where a canary gradient is introduced to the DP-SGD steps. Two methods for crafting this \"worst-case\" canary gradients is proposed: 1. a random direction in the parameter space is selected and a gradient of norm C is inserted to the mnibatch of gradients and 2. the adversary simulates the DP-SGD run and picks as the canary gradient the least updated dimension, again with norm C. Authors demonstrate empirically that if the canary is inserted at every iteration, the auditing bounds correspond to the DP accounting upper bounds, suggesting that the hidden state model does not necessarily provide additional privacy protection. When the canary is inserted less frequently, the adversaries power decreases. Finally, authors propose an auditing setting where the adversary inserts the crafted gradient only in the first iteration and controls the loss landscape. In this setting the auditing bounds match closely the DP bounds when minibatch used in DP-SGD is sufficiently large.", "questions": "- It seems that the privacy accounting upper bounds remain the same for various k (Figures 2 and 3). Now, is this because you have actually accounted only the 250 iterations the canary was inserted? Or do you display the auditing results only up to $T=250$ iteration?\n- I don't think there is any reason to use PRV as there is no subsampling. You could just use analytical Gaussian and obtain tight privacy bounds (instead of the upper and lower bounds you get from PRV).\n- Caption of Fig4: \"Figure 4a gives the ...\", I guess you mean 4b?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Auditing DP algorithms is an interesting and important line of work. Previous work, both in auditing and privacy accounting, have suggested that hiding the intermediate iterations of DP-SGD can be beneficial for the privacy guarantees. In this work, authors improve the privacy auditing of non-convex loss functions, by carefully selecting gradient canaries that get inserted for to the DP-SGD gradient sum. The proposed method significantly improves the existing methods, suggesting that the previous methods have not optimally used the threat model allowed by the hidden state setting.\n\nAuthors also study the amplification by iteration effect, by proposing a novel technique that introduces a crafted gradient only for the first step of DP-SGD. The empirical results for this study suggest that there might not be any amplification by iteration for certain non-convex losses, and that only the noise introduced by subsampling the data makes distinguishing the crafted gradient more difficult.", "weaknesses": "In general the paper is very well written and the arguments are easy to follow. However, I get a bit confused on the discussion regarding accounting in the section 5.1. Since the threat model studied does not benefit from the subsampling amplification, I don't see any reason of using PRV accounting. When there is no subsampling, the privacy analysis could be performed tightly with Gaussian DP or Analytical Gaussian accounting (Balle et al 2018).\n\nThe proposed method assumes that the crafted gradient is possible under the particular loss function. While I do believe that this might be the case for highly over-parametrized models, it would be great if authors could discuss this further in the paper. Would it be possible to somehow trace back the worst-case sample from the worst-case crafted gradient? Similarly the assumption that the adversary can craft the loss landscape to make the crafted gradient the most distinguishable could warrant further discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vh9zyJWBp", "reviewer": "Reviewer_fShh", "summary": "This paper proposes a privacy auditing method for DP-SGD under a hidden state threat model. Assuming an arbitrary non-convex loss function, the authors abstract away the specific design of neighboring datasets and directly construct gradient sequences for auditing. Their main difference from previous work, which audits DP-SGD with gradient-crafting adversaries at each step, is that they ensures that the gradient sequence is predetermined and not influenced by intermediate outputs, i.e., the sequence is decided offline before training.\n\nThe authors considers two types of gradient-crafting:\n\n1. **Random-biased dimension:** randomly selects a dimension and crafts the gradient with the largest possible magnitude in that dimension.\n    \n2. **Simulated-biased dimension:** simulates the training algorithm, identifies the least updated dimension, and crafts gradients with the largest possible magnitude there.\n    \n\nExperimentally, they demonstrate that when a canary gradient is inserted at each step, their empirical privacy estimates are tight for common model architectures, including ConvNet, ResNet, and FCNN. The auditing also remains tight when the canary is periodically added with a periodicity of 5. However, at a periodicity of 25, the auditing for some settings may not be tight, suggesting possible privacy amplification by iteration. To investigate this, the authors conducted experiments, showing that their auditing is tight when the batch size and $\\epsilon$ are large. This supports prior findings which shows DP-SGD on non-convex problems does not exhibit privacy amplification by iteration. Thus, relaxing the threat model to a hidden state model does not offer better privacy guarantees for certain non-convex problems.", "questions": "The simulated biased dimension method appears quite similar to gradient crafting at each step in terms of implementation [Nasr et al., 2023]. What are the specific differences between these two methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n2. The proposed method is simple yet empirically effective.\n3. The empirical results provide insights into the settings (batch size and $\\epsilon$) where DP-SGD may exhibit privacy amplification by iteration for non-convex problems.", "weaknesses": "Firstly, the conclusion of this work is not new: the main takeaway—that for some non-convex problems, the hidden state threat model does not lead to better privacy analysis for DP-SGD (i.e., no privacy amplification by iteration)—is the same as the conclusion in previous work [Annamalai, 2024]. However, the methodologies differ: [Annamalai, 2024] constructs a worst-case non-convex loss function for DP-SGD where information from all previous iterates is encoded in the final iterates, while this work directly constructs gradient sequences.\n\nMy other concern with this work is that its methodology is also quite limited, though it may be slightly more general than previous approaches (e.g. [Annamalai, 2024]). This work abstracts away the specifics of the loss function and model architecture, focusing directly on gradient crafting. While this simplification aids in designing canary gradients, it resembles a worst-case analysis across all non-convex problems. From a specific gradient sequence, it is not possible to deduce the corresponding loss function and model architecture on which DP-SGD is performed. As a result, the findings do not clarify which specific architectures and loss functions (i.e., types of non-convex problems) may or may not exhibit privacy amplification by iteration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)