Title: The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD
Venue: NeurIPS 2024
Accepted: no
Keywords: differential privacy

heuristics

privacy auditing
Abstract: We propose a simple heuristic privacy analysis of noisy clipped stochastic gradient descent (DP-SGD) in the setting where only the last iterate is released and the intermediate iterates remain hidden. Namely, our heuristic assumes a linear structure for the model.

We show experimentally that our heuristic is predictive of the outcome of privacy auditing applied to various training procedures. Thus it can be used prior to training as a rough estimate of the final privacy leakage. We also probe the limitations of our heuristic by providing some artificial counterexamples where it underestimates the privacy leakage.

The standard composition-based privacy analysis of DP-SGD effectively assumes that the adversary has access to all intermediate iterates, which is often unrealistic.
However, this analysis remains the state of the art in practice. 
While our heuristic does not replace a rigorous privacy analysis, it illustrates the large gap between the best theoretical upper bounds and the privacy auditing lower bounds and sets a target for further work to improve the theoretical privacy analyses.

1

Abstract

We propose a simple heuristic privacy analysis of noisy clipped stochastic gradient descent (DP-SGD) in the setting where only the last iterate is released and the intermediate iterates remain hidden. Namely, our heuristic assumes a linear structure for the model.

We show experimentally that our heuristic is predictive of the outcome of privacy auditing applied to various training procedures. Thus it can be used prior to training

as a rough estimate of the final privacy leakage. We also probe the limitations of our heuristic by providing some artificial counterexamples where it underestimates

the privacy leakage.

The standard composition-based privacy analysis of DP-SGD effectively assumes that the adversary has access to all intermediate iterates, which is often unrealistic. However, this analysis remains the state of the art in practice. While our heuristic does not replace a rigorous privacy analysis, it illustrates the large gap between the best theoretical upper bounds and the privacy auditing lower bounds and sets a target for further work to improve the theoretical privacy analyses.

1 Introduction

Differential privacy (DP) [DMNS06] defines a measure of how much private information from the training data leaks through the output of an algorithm. The standard differentially private algorithm for deep learning is DP-SGD [BST14; ACGMMTZ16], which differs from ordinary stochastic gradient descent in two ways: the gradient of each example is clipped to bound its norm and then Gaussian noise is added at each iteration.

The standard privacy analysis of DP-SGD is based on composition [BST14; ACGMMTZ16; Mir17; Ste22; KJH20]. In particular, it applies to the setting where the privacy adversary has access to all intermediate iterates of the training procedure. In this setting, the analysis is known to be tight [NSTPC21; NHSBTJCT23]. However, in practice, potential adversaries rarely have access to the intermediate iterates of the training procedure, rather they only have access to the final model. Access to the final model can either be through queries to an API or via the raw model weights. The key question motivating our work is the following.

Is it possible to obtain sharper privacy guarantees for DP-SGD when the adversary only has access to the final model, rather than all intermediate iterates?

1.1 Background & Related Work

The question above has been studied from two angles: Theoretical upper bounds, and privacy auditing

32 lower bounds. Our goal is to shed light on this question from a third angle via principled heuristics. 33

A handful of theoretical analyses [FMTT18; CYS21; YS22; AT22; BSA24] have shown that asymp34 totically the privacy guarantee of the last iterate of DP-SGD can be far better than the standard 35 composition-based analysis that applies to releasing all iterates. In particular, as the number of 36 iterations increases, these analyses give a privacy guarantee that converges to a constant (depending 37 on the loss function and the scale of the noise), whereas the standard composition-based analysis 38 would give a privacy guarantee that increases forever. Unfortunately, these theoretical analyses 39 are only applicable under strong assumptions on the loss function, such as (strong) convexity and 40 smoothness. We lack an understanding of how well they reflect the 'real' privacy leakage. 41

42

43

44

45

46

47

48

49

50

51

52

53

54

Privacy auditing [JUO20; DWWZK18; BGDCTV18; SNJ23; TTSSJC22; ZBWTSRPNK22] complements theoretical analysis by giving empirical lower bounds on the privacy leakage. Privacy auditing works by performing a membership inference attack [SSSS17; HSRDTMPSNC08; SOJH09; DSSUV15]. That is, it constructs neighbouring inputs and demonstrates that the corresponding output distributions can be distinguished well enough to imply a lower bound on the differential privacy parameters. In practice, the theoretical privacy analysis may give uncomfortably large values for the privacy leakage (e.g., ε > 10 ); in this case, privacy auditing may be used as evidence that the 'real' privacy leakage is lower. There are settings where the theoretical analysis is matched by auditing, such as when all intermediate results are released [NSTPC21; NHSBTJCT23]. However, despite significant work on privacy auditing and membership inference [CCNSTT22; BTRKMW24; WBKBGGG23; LF20; SDSOJ19; ZLS23], a large gap remains between the theoretical upper bounds and the auditing lower bounds [AKOOMS23; NHSBTJCT23] when only the final parameters are released. This observed gap is the starting point for our work.

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

1.2 Our Contributions

We propose a heuristic privacy analysis of DP-SGD in the setting where only the final iterate is released. Our experiments demonstrate that this heuristic analysis consistently provides an upper bound on the privacy leakage measured by privacy auditing tools in realistic deep learning settings.

Our heuristic analysis corresponds to a worst-case theoretical analysis under the assumption that the loss functions are linear. This case is simple enough to allow for an exact privacy analysis whose parameters are can be computed numerically (Theorem 1). Our consideration of linear losses is built on the observation that current auditing techniques achieve the highest ε values when the gradients of the canaries - that is, the examples that are included or excluded to test the privacy leakage - are fixed and independent from the gradients of the other examples. This is definitely the case for linear losses; the linear assumption thus allows us to capture the setting where current attacks are most effective. Linear loss functions are also known to be the worst case for the non-subsampled (i.e., full batch) case; see Appendix B. Assuming linearity is unnatural from an optimization perspective, as there is no minimizer. But, from a privacy perspective, we show that it captures the state of the art.

We also probe the limitations of our heuristic and give some artificial counterexamples where it underestimates empirical privacy leakage. One class of counterexamples exploits the presence of a regularizer. Roughly, the regularizer partially zeros out the noise that is added for privacy. However, the regularizer also partially zeros out the signal of the canary gradient. These two effects are almost balanced, which makes the counterexample very delicate. In a second class of counterexamples, the data is carefully engineered so that the final iterate effectively encodes the entire trajectory, in which case there is no difference between releasing the last iterate and all iterates.

Implications: Heuristics cannot replace rigorous theoretical analyses. However, our heuristic can serve as a target for future improvements to both privacy auditing as well as theoretical analysis. For privacy auditing, matching or exceeding our heuristic is a more reachable goal than matching the theoretical upper bounds, although our experimental results show that even this would require new attacks. When theoretical analyses fail to match our heuristic, we should identify why there is a gap, which builds intuition and could point towards further improvements.

Given that privacy auditing is computationally intensive and difficult to perform correctly [AZT24], 82 we believe that our heuristic can also be valuable in practice. In particular, our heuristic can be used 83 prior to training (e.g., during hyperparameter selection) to predict the outcome of privacy auditing 84 when applied to the final model. (This is a similar use case to scaling laws.) 85

2 Linearized Heuristic Privacy Analysis 86

87

88

89

90

91

92

93

94

95

Theorem 1 presents our heuristic differential privacy analysis of DP-SGD (which we present in Algorithm 1 for completeness; note that we include a regularizer r whose gradient is not clipped, because it does not depend on the private data x ). We consider Poisson subsampled minibatches and add/remove neighbours, as is standard in the differential privacy literature.

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

Our analysis takes the form of a conditional privacy guarantee. Namely, under the assumption that the loss and regularizer are linear, we obtain a fully rigorous differential privacy guarantee. The heuristic is to apply this guarantee to loss functions that are not linear (such as those that arise in deep learning applications). Our thesis is that, in most cases, the conclusion of the theorem is still a good approximation, even when the assumption does not hold.

Recall that a function glyph[lscript] : R d → R is linear if there exist α ∈ R d and β ∈ R such that glyph[lscript] ( m ) = 〈 α, m 〉 + β for all m .

Theorem 1 (Privacy of DP-SGD for linear losses) . Let x , T, q, η, σ, glyph[lscript], r be as in Algorithm 1. Assume r and glyph[lscript] ( · , x ) , for every x ∈ X , are linear.

Letting

DP-SGD with last_iterate_only satisfies ( ε, δ ) -differential privacy with ε ≥ 0 arbitrary and

ε

Here,

H

e

ε

denotes the

e

-hockey-stick-divergence

H

e

ε

(

P, Q

) := sup

S

P

(

S

)

-

e

ε

Q

(

S

)

.

Equation 1 gives us a value of the privacy failure probability parameter δ . But it is more natural to work with the privacy loss bound parameter ε , which can be computed by inverting the formula:

Both δ T,q,σ ( ε ) and ε T,q,σ ( δ ) can be computed using existing open-source DP accounting libraries [Goo20]. We also provide a self-contained & efficient method for computing them in Appendix A.

The proof of Theorem 1 is deferred to Appendix A, but we sketch the main ideas: Under the linearity assumption, the output of DP-SGD is just a sum of the gradients and noises. We can reduce to dimension d = 1 , since the only relevant direction is that of the gradient of the canary 1 (which is constant). We can also ignore the gradients of the other examples. Thus, by rescaling, the worst case pair of output distributions can be represented as in Equation 1. Namely, Q = ∑ T t =1 ξ t is simply the noise ξ t ← N (0 , σ 2 ) summed over T iterations; this corresponds to the case where the canary is excluded. When the canary is included, it is sampled with probability q in each iteration and thus the total number of times it is sampled over T iterations is Binomial ( T, q ) . Thus P is the sum of the contributions of the canary and the noise. Finally the definition of differential privacy lets us compute ε and δ from this pair of distributions. Tightness follows from the fact that there exists a loss function and pair of inputs such that the corresponding outputs of DP-SGD matches the pair P and Q .

Algorithm 1 Noisy Clipped Stochastic Gradient Descent (DP-SGD) [BST14; ACGMMTZ16]

132

133

134

135

136

137

10 0

10 1

10 2

10 3

T

=1/

q

=25/

2

10 0

10 1

10 2

10 3

at  =1

e

06

Heuristic

Standard

(a) Keep

Tp

and

Tσ

2

constant.

10 0

10 1

10 2

10 3

10 4

T

=1/

q

0

2

4

6

8

10

at  =1

e

06

Heuristic

Standard

(b) Keep Standard DP constant.

10 0

10 1

10 2

10 3

10 4

T

(with

q

=0.01 and

=0.5)

0

2

4

6

8

10

12

14

at  =1

e

06

Heuristic

Standard

(c) Keep

p

and

σ

constant.

2.1 Baselines

In addition to privacy auditing, we compare our heuristic to two different baselines in Figure 1. The first is the standard, composition-based analysis. We use the open-source library from Google [Goo20], which computes a tight DP guarantee for DP-SGD with intermediate_iterates . Because DP-SGD with intermediate_iterates gives the adversary more information than with last_iterate_only , this will always give at least as large an estimate for ε as our heuristic.

We also consider approximating DP-SGD by full batch DP-GD. That is, set q = 1 and rescale the 138 learning rate η and noise multiplier σ to keep the expected step and privacy noise variance constant: 139

The latter algorithm is full batch DP-GD since at each step it includes each data point in the batch 140 with probability 1 . Since full batch DP-GD does not rely on privacy amplification by subsampling, 141 it is much easier to analyze its privacy guarantees. Interestingly, there is no difference between 142 full batch DP-GD with last_iterate_only and with intermediate_iterates ; see Appendix B. 143 Full batch DP-GD generally has better privacy guarantees than the corresponding minibatch DP-SGD 144 and so this baseline usually (but not always) gives smaller values for the privacy leakage ε than our 145 heuristic. In practice, full batch DP-GD is too computationally expensive to run. But we can use it as 146 an idealized comparison point for the privacy analysis. 147

10 3

10 4

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic (

σ

=1.1,q=0.01)

Standard (

σ

=1.1,q=0.01)

Empirical

(a)

q

= 0

.

01

10 3

Number of steps

0

2

4

6

8

ε

at

δ

=1

e

-05

Heuristic (

σ

=2.1,q=0.08)

Standard (

σ

=2.1,q=0.08)

Empirical

(b)

q

= 0

.

08

10 0

10 1

10 2

10 3

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic

Standard

Empirical

(a)

q

= 0

.

01

10 0

10 1

10 2

10 3

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic

Standard

Empirical

(b)

q

= 0

.

1

10 1

10 2

10 3

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic

Standard

Empirical

(a) All zero gradient inputs

10 1

10 2

10 3

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic

Standard

Empirical

(b) CIFAR10 Dataset

3 Empirical Evaluation via Privacy Auditing 148

Setup: We follow the construction of Nasr, Song, Thakurta, Papernot, and Carlini [NSTPC21] where 149 we have 3 entities, adversarial crafter, model trainer, and distinguisher. In this paper, we assume 150 the distinguisher only has access the final iteration of the model parameters. We use the CIFAR10 151 dataset [Ale09] with a WideResNet model [ZK16] unless otherwise specified; in particular, we follow 152 the training setup of De, Berrada, Hayes, Smith, and Balle [DBHSB22], where we train and audit 153 a model with 79% test accuracy and, using the standard analysis, ( ε = 8 , δ = 10 -5 ) -DP. For each 154 experiment we trained 512 CIFAR10 models with and without the canary (1024 total). To compute 155 the empirical lower bounds we use the PLD approach with Clopper-Pearson confidence intervals 156 used by Nasr, Hayes, Steinke, Balle, Tramèr, Jagielski, Carlini, and Terzis [NHSBTJCT23]. Here we 157 assume the adversary knows the sampling rate and the number of iterations and is only estimating the 158 noise multiplier used in DP-SGD, from which the reported privacy parameters ( ε and δ ) are derived. 159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

3.1 Experimental Results

We implement state-of-the-art attacks from prior work [NSTPC21; NHSBTJCT23]. These attacks heavily rely on the intermediate steps and, as a result, do not achieve tight results. In the next section, we design specific attacks for our heuristic privacy analysis approach to further understand its limitations and potential vulnerabilities. We used Google Cloud A2-megagpu-16g machines with 16 Nvidia A100 40GB GPUs. Overall, we use roughly 33,000 GPU hours for our experiments.

Gradient Space Attack: The most powerful attacks in prior work are gradient space attacks where the adversary injects a malicious gradient directly into the training process, rather than an example; prior work has shown that this attack can produce tight lower bounds, independent of the dataset and model used for training [NHSBTJCT23]. However, these previous attacks require access to all intermediate training steps to achieve tight results. Here, we use canary gradients in two settings: one where the other data points are non-adversarial and sampled from the real training data, and another where the other data points are designed to have very small gradients ( ≈ 0 ). This last setting was shown by [NSTPC21] to result in tighter auditing. In all attacks, we assume the distinguisher has access to all adversarial gradient vectors. For malicious gradients, we use Dirac gradient canaries, where gradient vectors consist of zeros in all but a single index. In both cases, the distinguishing test measures the dot product of the final model checkpoint and the gradient canary.

Figure 2 summarizes the results for the non-adversarial data setting, with other examples sampled from the true training data. In this experiment, we fix noise magnitude and subsampling probability, and run for various numbers of training steps. While prior work has shown tight auditing in this setting, we find an adversary without access to intermediate updates obtains much weaker attacks. Indeed, auditing with this strong attack results even in much lower values than the heuristic outputs.

Our other setting assumes the other data points are maliciously chosen. We construct an adversarial 'dataset' of m +1 gradients, m of which are zero, and one gradient is constant (with norm equal to the clipping norm), applying gradients directly rather than using any examples. As this experiment does not require computing gradients, it is very cheap to run more trials, so we run this procedure N = 100 , 000 times with the gradient canary, and N times without it, and compute an empirical estimate for ε with these values. We plot the results of this experiment in Figure 3 together with the ε output by the theoretical analysis and the heuristic, fixing the subsampling probability and varying the number of update steps. We adjust the noise parameter to ensure the standard theoretical analysis produces a fixed ε bound. The empirical measured ε is close to the heuristic ε except for when training with very small step counts: we expect this looseness to be the result of statistical effects, as lower step counts have higher relative variance at a fixed number of trials.

Input Space Attack: In practice, adversaries typically cannot insert malicious gradients freely in training steps. Therefore, we also study cases where the adversary is limited to inserting malicious inputs into the training set. Label flip attacks are one of the most successful approaches used to audit DP machine learning models in prior work [NHSBTJCT23; SNJ23]. For input space attacks, we use the loss of the malicious input as a distinguisher. Similar to our gradient space attacks, we consider two settings for input space attacks: one where other data points are correctly sampled from the dataset, and another where the other data points are blank images.

Figure 4 summarizes the results for this setting. Comparing to Figure 2, input space attacks achieve tighter results than gradient space attacks. This finding is in stark contrast to prior work. The reason is that input space attacks do not rely on intermediate iterates, so they transfer well to our setting.

In all the cases discussed so far, the empirical results for both gradient and input attacks fall below the heuristic analysis and do not violate the upper bounds based on the underlying assumptions. This suggests that the heuristic might serve as a good indicator for assessing potential vulnerabilities. However, in the next section, we delve into specific attack scenarios that exploit the assumptions used in the heuristic analysis to create edge cases where the heuristic bounds are indeed violated.

4 Counterexamples

We now test the limits of our heuristic by constructing some artificial counterexamples. That is, we construct inputs to DP-SGD with last_iterate_only such that the true privacy loss exceeds the bound given by our heuristic. While we do not expect the contrived structures of these examples to

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

manifest in realistic learning settings, they highlight the difficulties of formalizing settings where the heuristic gives a provable upper bound on the privacy loss.

4.1 Warmup: Zeroing Out The Model Weights

We begin by noting the counterintuitive fact that our heuristic ε T,q,σ ( δ ) is not always monotone in the number of steps T when the other parameters σ, q, δ are kept constant. This is shown in Figure 1c. More steps means there is both more noise and more signal from the gradients; these effects partially cancel out, but the net effect can be non-monotone.

We can use a regularizer r ( m ) = ‖ m ‖ 2 2 / 2 η so that η · ∇ m r ( m ) = m . This regularizer zeros out the model from the previous step, i.e., the update of DP-SGD becomes

This means that the last iterate m T is effectively the result of only a single iteration of DP-SGD. In particular, it will have a privacy guarantee corresponding to one iteration. Combining this regularizer with a linear loss and a setting of the parameters T, q, σ, δ such that the privacy loss is non-monotone - i.e., ε T,q,σ ( δ ) < ε 1 ,q,σ ( δ ) - yields a counterexample.

In light of this counterexample, in the next subsection, we benchmark our counterexample against sweeping over smaller values of T . I.e., we consider max t ≤ T ε t,q,σ ( δ ) instead of simply ε T,q,σ ( δ ) .

4.2 Linear Loss + Quadratic Regularizer

Consider running DP-SGD in one dimension (i.e., d = 1 ) with a linear loss glyph[lscript] ( m , x ) = m x for the canary and a quadratic regularizer r ( m ) = 1 2 α m 2 , where α ∈ [0 , 1] and x ∈ [ -1 , 1] and we use learning rate η = 1 . With sampling probability q , after T iterations the privacy guarantee is equivalent to distinguishing Q := N (0 , ̂ σ 2 ) and P := N ( ∑ i ∈ [ T ] (1 -α ) i -1 Bernoulli ( q ) , ̂ σ 2 ) , where ̂ σ 2 := σ 2 ∑ i ∈ [ T ] (1 -α ) 2( i -1) . When α = 0 , this retrieves linear losses. When α = 1 , this corresponds to distinguishing N (0 , ̂ σ 2 ) and N ( Bernoulli ( q ) , ̂ σ 2 ) or, equivalently, to distinguishing linear losses after T = 1 iteration. If we maximize our heuristic over the number of iterations ≤ T , then our heuristic is tight for the extremes α ∈ { 0 , 1 } .

A natural question is whether the worst-case privacy guarantee on this quadratic is always given by α ∈ { 0 , 1 } . Perhaps surprisingly, the answer is no: we found that for T = 3 , q = 0 . 1 , σ = 1 , α = 0 , DP-SGD is (2 . 222 , 10 -6 ) -DP. For α = 1 instead DP-SGD is (2 . 182 , 10 -6 ) -DP. However, for α = 0 . 5 instead the quadratic loss does not satisfy ( ε, 10 -6 ) -DP for ε < 2 . 274 .

However, this violation is small, which suggests our heuristic is still a reasonable for this class of examples. To validate this, we consider a set of values for the tuple ( T, q, σ ) . For each setting of T, q, σ , we compute max t ≤ T ε t,q,σ ( δ ) at δ = 10 -6 . We then compute ε for the linear loss with quadratic regularizer example with α = 1 / 2 in the same setting. Since the support of the random variable ∑ i ∈ [ T ] (1 -α ) i -1 Bernoulli ( q ) has size 2 T for α = 1 / 2 , computing exact ε for even moderate T is computationally intensive. Instead, let X be the random variable equal to ∑ i ∈ [ T ] (1 -α ) i -1 Bernoulli ( q ) , except we round up values in the support which are less than . 0005 up to . 0005 , and then round each value in the support up to the nearest integer power of 1 . 05 . We then compute an exact ε for distingushing N (0 , ̂ σ 2 ) vs N ( X, ̂ σ 2 ) . By Lemma 4.5 of Choquette-Choo, Ganesh, Steinke, and Thakurta [CCGST24], we know that distinguishing N (0 , ̂ σ 2 ) vs. N ( ∑ i ∈ [ T ] (1 -α ) i -1 Bernoulli ( q ) , ̂ σ 2 ) is no harder than distingushing N (0 , ̂ σ 2 ) vs N ( X, ̂ σ 2 ) , and since we increase the values in the support by no more than 1.05 multiplicatively, we expect that our rounding does not increase ε by more than 1.05 multiplicatively.

In Figure 5, we plot the ratio of ε at δ = 10 -6 for distingushing between N (0 , ̂ σ 2 ) and N ( X, ̂ σ 2 ) divided by the maximum over i ∈ [ T ] of ε at δ = 10 -6 for distinguishing between N (0 , iσ 2 ) and N ( Binomial ( i, q ) , iσ 2 ) . We sweep over T and q , and for each q In Figure 5a (resp. Figure 5b) we set σ such that distinguishing N (0 , σ 2 ) from N ( Bernoulli ( q ) , σ 2 ) satisfies (1 , 10 -6 ) -DP

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

εforα=1/2/heuristicε

2

0.95

0.97

1.0

0.97

0.87

0.78

4

0.82

0.87

0.94

1.01

0.87

0.75

8-

0.64

0.72

0.83

1.01

0.94

0.82

T

16

0.45

0.53

0.65

0.83

0.93

0.8

32

0.32

0.38

0.49

0.65

0.88

0.8

64

0.22

0.27

0.36

0.49

0.67

0.8

1/2

1/4

1/8

1/16

1/32

1/64

b

εforα=1/2/heuristicε

2

0.95

0.97

1.0

0.95

0.86

0.78

4

0.82

0.88

0.98

0.96

0.84

0.74

8

0.65

0.75

0.91

1.03

0.9

0.8

T

16

0.46

0.58

0.74

0.99

0.89

0.78

32

0.32

0.42

0.58

0.81

0.89

0.78

64

0.22

0.3

0.43

0.62

0.89

0.78

1/2

1/4

1/8

1/16

1/32

1/64

b

(resp. (2 , 10 -6 ) -DP). In the majority of settings, the linear loss heuristic provides a larger ε than the quadratic with α = 1 / 2 , and even when the quadratic provides a larger ε , the violation is small ( ≤ 3% ). This is evidence that our heuristic is still a good approximation for many convex losses.

4.3 Pathological Example

If we allow the regularizer r to be arbitrary - in particular, not even requiring continuity - then the gradient can also be arbitrary. This flexibility allows us to construct a counterexample such that the standard composition-based analysis of DP-SGD with intermediate_iterates is close to tight.

Specifically, choose the regularizer so that the update m ′ = m -η ∇ m r ( m ) does the following: m ′ 1 = 0 and, for i ∈ [ d -1] , m ′ i +1 = v · m i . Here v > 1 is a large constant. We chose the loss so that, for our canary x 1 , we have ∇ m glyph[lscript] ( m , x 1 ) = (1 , 0 , 0 , · · · , 0) and, for all other examples x i ( i ∈ { 2 , 3 , · · · , n } ), we have ∇ m glyph[lscript] ( m , x i ) = 0 . Then the last iterate is

where A t ← Bernoulli ( p ) indicates whether or not the canary was sampled in the t -th iteration and ξ t,i denotes the i -th coordinate of the noise ξ t added in the t -th step. Essentially, the last iterate m T contains the history of all the iterates in its coordinates. Namely, the i -th coordinate of m T gives a scaled noisy approximation to A T -i :

As v →∞ , the variance converges to σ 2 . In other words, if v is large, from the final iterate, we can obtain N ( A i , σ 2 ) for all i . This makes the standard composition-based analysis of DP-SGD tight.

4.4 Malicious Dataset Attack

The examples above rely on the regularizer having large unclipped gradients. We now construct a 275 counterexample without a regularizer, instead using other examples to amplify the canary signal. 276

Our heuristic assumes the adversary does not have access to the intermediate iterations and that the 277 model is linear. However, we can design a nonlinear model and specific training data to directly 278 challenge this assumption. The attack strategy is to use the model's parameters as a sort of noisy 279 storage, saving all iterations within them. Then with access only to the final model, an adversary 280

290

291

292

293

294

295

296

297

298

299

300

301

302

303

can still examine the parameters, extract the intermediate steps, and break the assumption. Our 281 construction introduces a data point that changes its gradient based on the number of past iterations, 282 making it easy to identify if the point was present a given iteration of training. The rest of the 283 data points are maliciously selected to ensure the noise added during training doesn't impact the 284 information stored in the model's parameters. We defer the full details of the attack to Appendix C. 285

Figure 6 summarizes the results. As illustrated in the figure, this attack achieves a auditing lower 286 bound matching the standard DP-SGD analysis even in the last_iterate_only setting. As a result, 287 the attack exceeds our heuristic. However, this is a highly artificial example and it is unlikely to 288 reflect real-world scenarios. 289

10 0

10 1

10 2

10 3

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic

Standard

Empirical

(a)

q

= 0

.

01

10 0

10 1

10 2

10 3

Number of steps

0

1

2

3

4

5

6

7

8

ε

at

δ

=1

e

-05

Heuristic

Standard

Empirical

(b)

q

= 0

.

1

5 Discussion & Conclusion

Both theoretical analysis and privacy auditing are valuable for understanding privacy leakage in machine learning, but each has limitations. Theoretical analysis is inherently conservative, while auditing procedures evaluate only specific attacks, and may thus underrepresent the privacy leakage.

Our work introduces a novel heuristic analysis for DP-SGD that focuses on the privacy implications of releasing only the final model iterate. This approach is based in the empirical observation that linear loss functions accurately model the effectiveness of state of the art membership inference attacks. Our heuristic offers a practical and computationally efficient way to estimate privacy leakage to complement privacy auditing and the standard composition-based analysis of DP-SGD. As shown in Table 1, we trained a series of CIFAR10 models with varying batch sizes that all achieved the similar level of heuristic epsilon, albeit with different standard epsilon values. Remarkably, these models exhibited similar performance and similar empirical epsilon values.

We also acknowledge the limitations of our heuristic by identifying specific counterexamples where the heuristic underestimates the true privacy leakage.

References 304

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

A Proof of Theorem 1

Proof. Let x i ∗ be the canary, let D be the dataset with the canary and D ′ be the dataset without the canary. Since glyph[lscript] and r are linear, wlog we can assume r = 0 and ∇ m t -1 glyph[lscript] ( m t -1 , x i ) = v i for some set of vectors { v i } , such that ‖ v i ‖ 2 ≤ 1 . We can also assume wlog ‖ v i ∗ ‖ = 1 since, if ‖ v i ∗ ‖ < 1 , the final privacy guarantee we show only improves.

We have the following recursion for m t :

Unrolling the recursion: 442

By the post-processing property of DP, we can assume that in addition to the final model m T , we 443 release m 0 and { B t \ { x i ∗ }} t ∈ [ T ] , that is we release all examples that were sampled in each batch 444 except for the canary. The following f is a bijection, computable by an adversary using the released 445 information: 446

Since f is a bijection, distinguishing m T sampled using D and D ′ is equivalent to distinguishing 447 f ( m T ) instead. Now we have f ( m T ) = N (0 , T σ 2 I d ) for D ′ , and f ( m T ) = N (0 , T σ 2 I d ) + k v i ∗ , 448 k ∼ Binomial ( T, q ) . For any vector u orthogonal to v i ∗ , by isotropy of the Gaussian distribution the 449 distribution of 〈 f ( m T ) , u 〉 is the same for both D and D ′ and independent of 〈 f ( m T ) , v i ∗ 〉 , hence 450 distinguishing f ( m T ) given D and D ′ is the same as distingushing 〈 f ( m T ) , v i ∗ 〉 given D and D ′ . 451 Finally, the distribution of 〈 f ( m T ) , v i ∗ 〉 is exactly P for D and exactly Q for D ′ . By post-processing, 452

glyph[negationslash]

We can also see that the function δ T,q,σ is tight (i.e., even if we do not release B t \ { x i ∗ } ), by 454 considering the 1-dimensional setting, where v i = 0 for i = i ∗ and v i ∗ = -1 , η = 1 , m 0 = 0 . Then, 455 the distribution of m T given D is exactly P , and given D ′ is exactly Q . 456

457

458

459

A.1 Computing δ from ε

Here, we give an efficiently computable expression for the function δ T,q,σ ( ε ) . Using P, Q as in Theorem 1, let f ( y ) be the privacy loss for the output y :

Then for any ε , using the fact that S = { y : f ( y ) ≥ ε } maximizes P ( S ) -e ε Q ( S ) , we have: 460

Similarly, S = { y : f ( y ) ≤ -ε } maximizes Q ( S ) -e ε P ( S ) so we have: 461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

These expressions can be evaluated efficiently. Since f is monotone, it can be inverted via binary search. We can also use binary search to evaluate ε as a function of δ .

B Linear Worst Case for Full Batch Setting

It turns out that in the full-batch setting, the worst-case analyses of DP-GD with intermediate_iterates and with last_iterate_only are the same. This phenomenon arises because there is no subsampling (because q = 1 in Algorithm 1) and thus the algorithm is 'just' the Gaussian mechanism. Intuitively, DP-GD with intermediate_iterates corresponds to T calls to the Gaussian mechanism with noise multiplier σ , while DP-GD with last_iterate_only corresponds to one call to the Gaussian mechanism with noise multiplier σ/ √ T ; these are equivalent by the properties of the Gaussian distribution.

We can formalize this using the language of Gaussian DP [DRS19]: DP-GD (Algorithm 1 with q = 1 ) satisfies √ T/σ -GDP. (Each iteration satisfies 1 /σ -GDP and adaptive composition implies the overall guarantee.) This means that the privacy loss is exactly dominated by that of the Gaussian mechanism with noise multiplier σ/ √ T . Linear losses give an example such that DP-GD with last_iterate_only has exactly this privacy loss, since the final iterate reveals the sum of all the noisy gradient estimates. The worst-case privacy of DP-GD with intermediate_iterates is no worse than that of DP-GD with last_iterate_only . The reverse is also true (by postprocessing).

In more detail: For T iterations of (full-batch) DP-GD on a linear losses, if the losses are (wlog) 479 1 -Lipschitz and we add noise N (0 , σ 2 n 2 · I ) to the gradient in every round, distinguishing the last 480

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

C Malicious Dataset Attack Details

Algorithms 2, 3, and 4 summarizes the construction for the attack. The attack assume the model parameters have dimension equal to the number of iterations. It also assumes each data point can reference which iteration of training is currently happening (this can be implemented by having a single model parameter which increments in each step, independently of the training examples, without impacting the privacy of the training process). Then we build our two datasets D and D ′ = D ∪ { x } so that all points in dataset D ('repeaters') run Algorithm 3 to compute gradients and the canary point in D ′ runs Algorithm 2 to compute its gradient. Our attack relies heavily on DP-SGD's lack of assumptions on the data distribution and any specific properties of the model or gradients. Algorithm 2, which generates the canary data point, is straightforward. Its goal is to store in the model parameters whether it was present in iteration i by outputting a gradient that changes only the i -th index of the model parameters by 1 (assuming a clipping threshold of 1).

All other data points, the 'repeaters', are present in both datasets ( D and D ′ ), and have three tasks:

Cancel out any noise added to the model parameters at an index larger than the current iteration. At iteration i , their gradients for parameters from index i onward will be the same as the current value of the model parameter, scaled by the batch size and the learning rate to ensure this parameter value will be 0 after the update.

Evaluate whether the canary point was present in the previous iteration by comparing the model parameter at index i -1 with a threshold, and rewrite the value of that model parameter to a large value if the canary was present.

Ensure that all previous decisions are not overwritten by noise by continuing to rewrite them with a large value based on their previous value.

To achieve all of these goals simultaneously, we require that the batch size is large enough that the repeaters' updates are not clipped.

Finally Algorithm 4 runs DP-SGD, with repeater points computing gradients with Algorithm 3 and the canary point, sampled with probability p , computing its gradient using Algorithm 2. In our experiments we run Algorithm 4 100,000 times. And to evaluate if the model parameters was from dataset D or D ′ we run a hypothesis test on the values of the model parameters. All constants are chosen to ensure all objectives of the repeaters are satisfied.

Algorithm 3 Additional data points

Require: model parameters x , iteration number i , batch size N , learning rate η , previous history threshold t past, last iteration threshold t last, history amplification value BIG_VAL

1: function REPEATERS( x , i , N , η , t past , t last, BIG_VAL)

2: h ← x

0: i glyph[triangleright] Parameter 'history' up to iteration i , not inclusive

3: f ← x

i : end glyph[triangleright] Future and current parameters, starting from iteration i

4: f ←-f / ( η · N )

5: base_history ←-x 0: i / ( η · N

6: if length ( h ) > 1 then

7: h 0: i -1 ← BIG_VAL / ( η · N ) · (2 ✶ [ h 0: i -1 ≥ t past ] -1) glyph[triangleright] If an old iteration is large enough, it was a canary iteration, so amplify it 8: end if

9: if length ( h ) > 0 then

10: h i -1 ← BIG_VAL / ( η · N ) · (2 ✶ [ h i ≥ t last ] -1) glyph[triangleright] If the last iteration is large enough, it was a canary iteration, so amplify it 11: end if

12: h ← h + base_history

13: a ← concatenate ( h , f )

14: return -a

15: end function

Algorithm 4 Encoding Attacking

Require: add-diff, whether to add the canary, batch size N , sampling rate p , learning rate ( η ), iteration count/parameter count D

1: function RUN_DPSGD(add-diff)

glyph[triangleright] Don't zero out canary iterations

glyph[triangleright] Remove noise from last iteration

) glyph[triangleright] By default, zero out entire history

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction state what we do and then the following sections and the appendix provide details.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 4 discusses the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theorem 1 is proved in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The setup is described for each experiment we conduct and we reference prior work that these build on.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We intend to release the code eventually, but we are not able to do so at the moment; we refrain from providing a detailed reason, as this could violate anonymity.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The setup is described for each experiment we conduct and we reference prior work that these build on. We use the standard CIFAR10 dataset for deep learning experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The auditing results present a lower bound which can be viewed as a one-sided confidence interval. For the other results the numbers are computed non-statistically (i.e. by numerically evaluating a formula); the only potential error here is due to numerical precision.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We used A2-megagpu-16g machines from Google cloud which have 16 Nvidia A100 40GB GPUs to run the experiments in this paper. Overall we used around 33,000 hours of GPU to run all of the experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: No human subjects or sensitive data were used.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is primarily theoretical. While it is possible that downstream uses of our work could be societally impactful, the precise consequences are difficult to foresee. The considerations are similar to any other paper on private machine learning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper uses standard datasets (CIFAR10) and standard models (WideResNet).

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use CIFAR10 [Ale09] and a WideResNet [ZK16].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our main contribution is a heuristic privacy analysis. This is fully described in the paper and can be computed using existing open-source libraries.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

836 837 838 839 840

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a heuristic approach for evaluating the privacy of DP-SGD when only the last model iteration is released. This method contrasts with traditional analyses that consider all intermediate updates, offering a more practical assessment for scenarios where adversaries only access the final model. The proposed heuristic is experimentally shown to provide reliable privacy leakage estimates, making it a valuable tool for pre-audit privacy assessments.\n\nSTRENGTHS:\n1. Focus on a good and important question.\n2. Good explanation and clear paper layout.\n3. Propose a new analysis neither from the theoretical nor empirical point of view.\n\nWEAKNESSES:\nI think the proposed method is interesting and new but I still have some questions.\n\n1. I know the linear loss function assumption is common in theoretical analysis but it seems that the proposed method wants to have contributions in the empirical case, so why still make the linear assumption? \n\n2. While I appreciate the effort to introduce a Heuristic analysis, I remain skeptical about its necessity and effectiveness. The primary benefit of theoretical analysis is its precision and rigor, which often include the flexibility to adjust bounds as needed. If the goal is to find a more relaxed lower bound on privacy risks, this can often be achieved by simply loosening the constraints within the existing theoretical framework. Introducing a separate heuristic analysis seems to complicate matters without providing clear advantages. \n\n3. I do not think you are using a correct baseline. When you make that only the last iteration model can be seen assumption, it is not fair to use normal DP-SGD analysis. I think it is better to use the theoretical analysis from those hidden state papers you cited. I am curious if you compare your proposed method with those methods, will you still get the same conclusion? \n\n4. I find Table 1 in the paper somewhat unclear and would appreciate further explanation from the authors regarding its purpose and implications. The table suggests that similar levels of heuristic ε are achieved across varying batch sizes, yet there is a noticeable increase in the standard privacy budget for smaller batches to maintain comparable performance. This observation seems to underscore the well-known impact of batch size rather than demonstrating an advantage of the proposed heuristic method.\nCould the authors elaborate on how this data relates to the efficacy of the heuristic analysis?\n\nQUESTIONS:\nPlease check the weaknesses.\n\nLIMITATIONS:\nPlease check the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a heuristic privacy analysis of releasing only the final model iterate of differentailly private gradient descent (DP-SGD). The analysis is based off of the worst-case differential privacy guarantee of DP-SGD with linear losses, under the assumption that the heuristic can be applied to more general loss functions in order to approximate the privacy loss.\n\nSTRENGTHS:\n* The premise of the paper (a heuristic privacy analysis of releasing only the final model iterate of DP-SGD) is very interesting, and Theorem 1 is a cool result.\n* The paper thoroughly assesses the limitations of the heuristic (in Section 4).\n\nWEAKNESSES:\n* I don’t know how useful the heuristic analysis would be in practice — beyond a lightweight sanity check — since ultimately it is just a heuristic and not a rigorous upper or lower bound on the privacy loss.\n\n* The empirical study of the heuristic looks to be very thorough, but sparse on interpretation. I would have appreciated more discussion on the figures, and didn’t really feel like there was a strong take-home message from the paper.\n\n* Algorithm 1 is DP-SGD with a regularizer, but in practice it is somewhat rare to use explicit regularization with DP-SGD. So I’m not sure that the heuristic would be widely applicable to the more common implementation of DP-SGD without regularization.\n\nQUESTIONS:\n1. One of the potential use cases of the heuristic (lines 82-85) is to predict the outcome of privacy auditing. So I’m wondering:\n   * If one of the advantages of the heuristic is that it’s less computationally expensive than privacy auditing, but you’ll have to do the computationally expensive privacy auditing anyway (in order to compare), then what is the point?\n   * One of the disadvantages of privacy auditing is that it’s difficult to perform correctly, but the heuristic doesn’t have rigorous theoretical guarantees. If privacy auditing and the heuristic don’t agree, then how would you know which one was wrong?\n\n2. In Figure 3, why does the heuristic $\\epsilon$ scale with the number of iterations $T$? Since the heuristic is a last-iterate analysis, I would have imagined that it would converge to a constant?\n\n3. Would it be possible to extend Theorem 1 to non-regularized DP-SGD?\n\n4. It’s not obvious to me why the privacy guarantee for an arbitrary loss would be similar to the privacy guarantee for a linear loss (and not even a GLM loss, mind you). Maybe I missed this in the paper, but would it be possible to further justify why the heuristic would be a good approximation in cases where the linear loss assumption doesn’t hold?\n\n5. I feel like the proof of Theorem 1 is hard to appreciate without knowing more technical preliminaries, such as how the hockey-stick divergence can be used to show that an algorithm satisfies DP. I wonder if the proof could be re-framed in the language of privacy profiles and dominating pairs? (https://arxiv.org/abs/1807.01647 and https://arxiv.org/abs/2106.08567)\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper proposes a heuristic privacy analysis for DP-SGD that focuses on releasing only the last iterate, as opposed to all intermediate iterates. The authors argue that this approach is more realistic and provides sharper privacy guarantees in practical scenarios. The heuristic is based on a linear structure assumption for the model and is validated experimentally through attacks/privacy auditing.

STRENGTHS:
This paper is well-written. The paper introduces a new heuristic analysis of DP-SGD for linear loss functions and also critically examines its limitations, and identifies areas for further research.

WEAKNESSES:
To my understanding, this paper offers a tighter privacy accounting analysis specifically for linear loss functions. However, I find its applicability limited since it cannot be extended to general ML tasks where the loss functions are not linear. Additionally, the fact that the privacy adversary has access to all intermediate iterates of the training process makes DP-SGD overly conservative is quite well-known. The main challenge remains in developing tight privacy accounting analyses for iterative algorithms like SGD.

QUESTIONS:
Why does the example provided in Section 4.2 show linear loss is a good approximation of many convex losses? Could you please provide some more insights?

LIMITATIONS:
The limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide exact DP guarantees for cases where only the last iterate of DP-SGD is shared with the malicious clients, and linear models with linear loss functions are used. They propose their DP bound to be used as a heuristic that approximates the true DP guarantees for cases where more complex models are used. They show that for normal DP-SGD training, the predictions of their heuristic fall between the standard DP bound computed under the assumption that all intermediate iterations of DP-SGD are shared with the attacker, which is a strict upper bound of the true DP guarantee when only the last iterate is shared and DP-SGD with full batches and only last iterate sharing. They also compare their method against SoTA DP attacks and show that under most circumstances, their heuristic value for the DP is higher. They suggest that this is the result of the attacks not being good enough at precisely estimating the true DP guarantees. Finally, the authors demonstrate that their heuristic under unrealistic circumstances can underestimate the true DP guarantee but argue this only happens under hand-crafted losses and gradient updates, which do not happen in practical circumstances.\n\nSTRENGTHS:\n- The last iterate setting is important\n- The linear function DP bound is exact \n- The linear function DP bound has interesting properties \n- The counter-examples for the DP heuristic themselves seem interesting and probably can be adapted to other settings\n\nWEAKNESSES:\n- I am confused by L234. The authors propose to maximize their heuristic over all $t\\leq T$, while beforehand (e.g. in Figure 1/Section 2) they advocated to computing the heuristic for a single $T$. Which one is the exact proposed heuristic by the paper?\n- In Figure 2, I am not sure how we adapt existing techniques to the last-iterate-only setting? Can the authors explain in more details?\n- Can the authors explain in Figure 1, what network and dataset were used?\n- The authors do not provide code. I am not sure about the reason, but I will give them the benefit of the doubt that the reason is indeed related to anonymity \n\n**Nits:**  \n- Eq. 8. I assume you do indexing from i = 1. In that case, $A_{T-i}$ should be $A_{T-i+1}$ instead. If you do 0-based indexing, even more fixes to the equation are needed.\n- I believe Eq. 7 should be multiplied by $\\eta$ on the right-hand side\n- Equation at L442, left-hand side should be $m_T$ not $m_t$\n- I believe the last equation at L459 should have $(1-q)^{n-k}$ instead of $(1-q)^{k}$. I also believe $n$ is $T$ in this equation\n- The definition of $l(m)$ in L109 is confusing as $m$ is considered input to the function, while in the rest of the paper $m$ is used as a parameter. Consider putting $x$ instead.\n- Consider defining the hockey-stick divergence in terms of both its pdf and cdf in the appendix to ease unfamiliar readers. I had to read quite a bit on my own to understand it. \n- Consider adding some information in the appendix as to how to deal with the mixed discrete-continuous probability for $P$. I assume many readers will be unfamiliar. \n- Consider deriving the formulas for $P$ and $Q$ in Section 4.2 in the appendix. They are not obvious. \n- Consider having an appendix section that quickly recaps how [NSTPC21] and [NHSBTJCT23] work. Their operation is critical for understanding Section 3. I ended up reading them to get an idea of what was going on there.\n\nQUESTIONS:\n(See Weaknesses Above)\n\n\nWhile I am familiar with DP and DP guarantees, I am not a DP expert. Still, I was able to follow the math in the paper, and I am mostly confident in it. What I am not as confident in is if the heuristic, which by authors' own admission does not provide rigorous upper or lower bounds, will be useful in practice. In addition, there is few experimental results that require explanation (see above). In particular, I am concerned about why, in the counter-examples in Section 4, the authors propose maximizing the heuristic across $T$, while in the experiments in Figure 1, the heuristic is proposed to be used for a particular $T$. I am, thus, not sure which of the two versions of the heuristic should be used in practical settings as the true heuristic. Further, I am not sure if the structure of the proposed DP counterexamples in Section 4 is novel, but if it is, I think it might have uses outside of this paper.\n\nLIMITATIONS:\nThe authors acknowledge the limitations of using the heuristic to compute the DP bounds", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}