Title: Avoiding Pitfalls for Privacy Accounting of Subsampled Mechanisms under Composition
Venue: NeurIPS 2024
Accepted: no
Keywords: differential privacy

privacy accounting
Abstract: We consider the problem of computing tight privacy guarantees for the composition of subsampled differentially private mechanisms. Recent algorithms can numerically compute the privacy parameters to arbitrary precision but must be carefully applied.

Our main contribution is to address two common points of confusion. First, some privacy accountants assume that the privacy guarantees for the composition of a subsampled mechanism are determined by self-composing the worst-case datasets for the uncomposed mechanism. We show that this is not true in general. Second, Poisson subsampling is sometimes assumed to have similar privacy guarantees compared to sampling without replacement. We show that the privacy guarantees may in fact differ significantly between the two sampling schemes. In particular, we give an example of hyperparameters that result in $\varepsilon \approx 1$ for Poisson subsampling and $\varepsilon > 10$ for sampling without replacement. This occurs for some parameters that could realistically be chosen for DP-SGD.

Abstract

We consider the problem of computing tight privacy guarantees for the composition of subsampled differentially private mechanisms. Recent algorithms can numerically compute the privacy parameters to arbitrary precision but must be carefully applied.

Our main contribution is to address two common points of confusion. First, some privacy accountants assume that the privacy guarantees for the composition of a subsampled mechanism are determined by self-composing the worst-case datasets for the uncomposed mechanism. We show that this is not true in general. Second, Poisson subsampling is sometimes assumed to have similar privacy guarantees compared to sampling without replacement. We show that the privacy guarantees may in fact differ significantly between the two sampling schemes. In particular, we give an example of hyperparameters that result in ε ≈ 1 for Poisson subsampling and ε > 10 for sampling without replacement. This occurs for some parameters that could realistically be chosen for DP-SGD.

1 Introduction

A fundamental property of differential privacy is that the composition of multiple differentially 16 private mechanisms still satisfies differential privacy. This property allows us to design complicated 17 mechanisms with strong formal privacy guarantees such as differentially private stochastic gradient 18 descent (DP-SGD, [SCS13, BST14, ACG + 16]). 19

The privacy guarantees of a mechanism inevitably deteriorate with the number of compositions. 20 Accurately quantifying the privacy parameters under composition is highly non-trivial and is an 21 important area within the field of differential privacy. A common approach is to find the privacy 22 parameters for each part of a mechanism and apply a composition theorem [DRV10, KOV15] to find 23 the privacy parameters of the full mechanism. In recent years, several alternatives to the traditional 24 definition of differential privacy with cleaner results for composition have gained popularity (see, 25 e.g., [DR16, BS16, Mir17, DRS19]). 26

Another important concept is privacy amplification by subsampling (see, e.g., [BBG18, Ste22]). The 27 general idea is to improve privacy guarantees by only using a randomly sampled subset of the full 28 dataset as input to a mechanism. In this work we consider the problem of computing tight privacy 29 parameters for subsampled mechanisms under composition. 30

One of the primary motivations for studying privacy accounting of subsampled mechanisms is DP31

SGD. DP-SGD achieves privacy by clipping gradients and adding Gaussian noise to each batch. 32

As such, we can find the privacy parameters by analyzing the subsampled Gaussian mechanism 33

under composition. One of the key contributions of [ACG + 16] was the moments accountant, 34

which gives tighter bounds for the mechanism than the generic composition theorems. Later work 35

improved the accountant by giving improved bounds on the Rényi Differential Privacy guarantees 36 of the subsampled Gaussian mechanism under both Poisson subsampling and sampling without 37 replacement [MTZ19, WBK20]. 38

39

40

41

42

43

44

45

46

47

48

49

50

Even small constant factors in an ( ε, δ ) -DP budget are important. First, from the definition, such constant factors manifest exponentially in the privacy guarantee. Furthermore, when training a model privately with DP-SGD, it has been observed that they can lead to significant differences in the downstream utility, see, e.g., Figure 1 of [DBH + 22]. Consequently, 'saving' such a factor in the value of ε through tighter analysis can be very valuable. While earlier approximate techniques for privacy accounting (e.g., moments accountant of [ACG + 16] and related methods) were lossy, a more recent line of work focuses on exact computation of privacy loss by numerically estimating the privacy parameters [SMM19, KJH20, KJPH21, GLW21, ZDW22]. These accountants generally look at the 'worst case' for a single iteration for a privacy mechanism, and then use a fast Fourier transform (FFT) to compose the privacy loss over multiple iterations. They often rely on an implicit assumption that the worst-case dataset for a single execution of a privacy mechanism remains the worst case for a self-composition of the mechanism.

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

Most privacy accounting techniques for DP-SGD assume a version of the algorithm that employs amplification by Poisson subsampling. That is, the batch for each iteration is formed by including each point independently with sampling probability γ . Other privacy accountants consider a variant where random batches of a fixed size are selected for each step. Note that both of these are inconsistent with the standard method in the non-private setting, where batches are formed by randomly permuting and then partitioning the dataset. Indeed, the latter approach is much more efficient, and highly-optimized in most libraries. Consequently, many works in private machine learning implement a method with the conventional shuffle-and-partition method of batch formation, but employ privacy accountants that assume some other method of sampling batches. The hope is that small modifications of this sort would have negligible impact on the privacy analysis, thus justifying privacy accountants for a setting which is technically not matching. Concurrent work to this paper by [CGK + 24] compares the shuffle-and-partition technique with Poisson subsampling. Similar to our results they find that the batching method can significantly impact the privacy parameters.

The central aim of our paper is to highlight and clarify some common problems with privacy accounting techniques. Towards the goal of more faithful comparisons between private algorithms that rely upon such accountants, we make the following contributions:

In Sections 4 and 5, we establish that a worst-case dataset may exist for a single execution of a privacy mechanism but may fail to exist when looking at the self-composition of the same mechanism. Some popular privacy accountants incorrectly assume otherwise. Our counterexample involves the subsampled Laplace mechanism, and stronger analysis is needed to demonstrate the soundness of privacy accountants for specific mechanisms, e.g., the subsampled Gaussian mechanism.

In Section 6, we show that rigorous privacy accounting is significantly affected by the method of sampling batches, e.g., Poisson versus fixed-size. This results in sizeable differences in the resulting privacy guarantees for settings which were previously treated as interchangeable by prior works. Consequently, we caution against the common practice of using one method of batch sampling and employing the privacy accountant for another.

In Section 7, we discuss issues that arise in tight privacy accounting under the 'substitution' relation for neighbouring datasets, which make this setting even more challenging than under the traditional 'add/remove' relation. Once again we consider the subsampled Laplace mechanism and show that there may be several worst-case datasets one must consider when doing accounting, exposing another important gap in existing analyses.

2 Preliminaries

′

Definition 1 ( ( ε, δ ) -Differential Privacy) . A randomized mechanism M satisfies ( ε, δ ) -DP under 88 neighboring relation ∼ if and only if for all D ∼ D ′ and all measurable sets of outputs Z we have 89

In this work, we consider problems where we want to estimate a sum for k queries where each datapoint holds a single-dimensional real value in the interval [ -1 , 1] for each query. The mechanisms we consider apply more generally to multi-dimensional real-valued queries. Since we demonstrate issues already present in the former more restrictive setting, these pitfalls are present in the more general case as well. We focus on single-dimensional inputs for simplicity of presentation. Likewise, by considering mechanisms defined on [ -1 , 1] , our privacy analysis immediately extends to any mechanism defined on R that clips to [ -1 , 1] . After the appropriate rescaling, our privacy analysis extends to any mechanism used in practice for DP-SGD. Note that in all but one example in Section 7 the datapoints hold the same value for all k queries for the datasets we consider. We abuse notation and represent each data point as a single real value rather than a vector.

On the domain [ -1 , 1] ∗× k := ⋃ ∞ m =0 [ -1 , 1] m × k , we define the neighboring definitions of add, remove, and substitution (replacement). We typically want the neighboring relation to be symmetric, which is why add and remove are typically included in a single definition. However, as noted by previous work we need to analyze the add and remove cases separately to get tight results (see, e.g., [ZDW22]).

Definition 2 (Neighboring Datasets) . Let D and D ′ be datasets. If D ′ can be obtained by adding a datapoint to D , then we write D ∼ A D ′ . Likewise, if D ′ can be obtained by removing a datapoint from D , then we write D ∼ R D ′ . Combining these, write D ∼ A/R D ′ if D ∼ A D ′ or D ∼ R D ′ . Finally, we write D ∼ S D ′ if D can be obtained from D ′ by swapping one datapoint for another.

Note that differential privacy under add and remove implies differential privacy under substitution, with appropriate translation of the privacy parameters.

Definition 1 can be restated in terms of the hockey-stick divergence.

Definition 3 (Hockey-stick Divergence) . For any α ≥ 0 the hockey-stick divergence between two distributions P and Q is defined as

where dP dQ is the Radon-Nikodym derivative.

Specifically, a randomized mechanism M satisfies ( ε, δ ) -DP if and only if H e ε ( M ( D ) ||M ( D ′ )) ≤ δ for all pairs of neighboring datasets D ∼ D ′ . This restated definition is the basis for the privacy accounting tools we consider in this paper. If we know what choice of neighboring datasets D ∼ D ′ maximizes the expression then we can get optimal parameters by computing H e ε ( M ( D ) ||M ( D ′ )) .

The full range of privacy guarantees for a mechanism can be captured by the privacy curve.

Definition 4 (Privacy Curves) . The privacy curve of a randomized mechanism M under neighboring relation ∼ is the function δ ∼ M : R → [0 , 1] given by

If there is a single pair of neighboring datasets D ∼ D ′ such that δ ∼ M ( ε ) = H e ε ( M ( D ) ||M ( D ′ )) for all ε ≥ 0 , we say that the privacy curve of M under ∼ is realized by the worst-case dataset pair ( D,D ′ ) .

Unfortunately, a worst-case dataset pair does not always exist. A broader tool that is now frequently used in the computation of privacy curves is the privacy loss distribution (PLD) formalism [DR16, SMM19].

Definition 5 (Privacy Loss Distribution) . Given a mechanism M and a pair of neighboring datasets D ∼ D ′ , the privacy loss distribution of M with respect to ( D,D ′ ) is

where y ∼ M ( D ) and d M ( D ) /d M ( D ′ ) means the density of M ( D ) with respect to M ( D ′ ) .

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

An important caveat is that the privacy loss distribution is defined with respect to a specific pair of 131 datasets, whereas the privacy curve implicitly involves taking a maximum over all neighboring pairs 132 of datasets. Nonetheless, the PLD formalism can be used to recover the hockey-stick divergence via 133

from which we can reconstruct the privacy curve as

Lastly, we define the two subsampling procedures we consider in this work: sampling without replacement (WOR) and Poisson sampling. Given a dataset D = ( x 1 , . . . , x n ) and a set I ⊆ { 1 , . . . , n } , we denote the restriction of D to I = { i 1 , . . . , i b } by D | I := ( x i 1 , . . . , x i b ) .

Definition 6 (Subsampling) . Let M take datasets of size 1 b ≥ 1 . The ( n b ) -subsampled mechanism M WOR is defined on datasets of size n ≥ b as

where I is a uniform random b -subset of { 1 , . . . , n } .

On the other hand, given a mechanism M taking datasets of any size, the γ -subsampled mechanism M Poisson is defined on datasets of arbitrary size as

where I includes each element of { 1 , . . . , | D |} independently with probability γ .

3 Related Work

After [DR16] introduced privacy loss distributions, a number of works used the formalism to estimate the privacy curve to arbitrary precision, beginning with [SMM19]. [KJH20, KJPH21] developed an efficient accountant that efficiently computes the convolution of PLDs by leveraging the fast Fourier transform. [GLW21] fine-tuned the application of FFT to speed up the accountant by several orders of magnitude.

The most relevant related paper for our work is by [ZDW22]. They introduce the concept of a dominating pair of distributions. Dominating pairs generalize worst-case datasets, which for some problems can be difficult to find and may not even exist.

Definition 7 (Dominating Pair of Distributions [ZDW22]) . The ordered pair ( P, Q ) is a dominating pair of distributions for a mechanism M (under some neighboring relation ∼ ) if for all α ≥ 0 it holds that

The hockey-stick divergence of the dominating pair P and Q gives an upper bound on the value δ for any ε . Note that the distributions P and Q do not need to be output distributions of the mechanism. However, if there exists a pair of neighboring datasets such that P = M ( D ) and Q = M ( D ′ ) then we can find tight privacy parameters by analyzing the mechanisms with inputs D and D ′ because H e ε ( M ( D ) ||M ( D ′ )) is also a lower bound on δ for any ε . We refer to such D ∼ D ′ as a dominating pair of datasets.

The definition of dominating pairs of distributions is useful for analyzing the privacy guarantees of composed mechanisms. In this work, we focus on the special case where a mechanism consists of k self-compositions. This is, for example, the case in DP-SGD, in which we run several iterations of the subsampled Gaussian mechanism. The property we need for composition is presented in Theorem 8.

Theorem 8 (Following Theorem 10 of [ZDW22]) . If ( P, Q ) is a dominating pair for a mechanism M then ( P k , Q k ) is a dominating pair for k iterations of M .

When studying differential privacy parameters in terms of the hockey-stick divergence, we usually focus on the case of α ≥ 1 . Recall that the hockey-stick divergence of order α can be used to bound

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

the value of δ for an ( ε, δ ) -DP mechanism where ε = ln( α ) . We typically do not care about the region of α < 1 because it corresponds to negative values of ε . However, the definition of dominating pairs of distributions must include these values as well. This is because outputs with negative privacy loss are important for composition and Theorem 8 would not hold if the definition only considered α ≥ 1 . In Sections 5 and 7 we consider mechanisms where the distributions that bound the hockey-stick divergence for α ≥ 1 without composition do not bound the divergence for α ≥ 1 under composition.

[ZDW22] studied general mechanisms in terms of dominating pairs of distributions under Poisson subsampling and sampling without replacement. Their work gives upper bounds on the privacy parameters based on the dominating pair of distributions of the non-subsampled mechanism. We use some of their results which we introduce later throughout this paper.

4 Dominating Pair of Datasets under Add and Remove Relations

In this section we give pairs of neighboring datasets with provable worst-case privacy parameters under the add and remove neighboring relations separately. We use these datasets as examples of the pitfalls to avoid in the subsequent section, where we discuss the combined add/remove neighboring relation.

Proposition 9. Let M be either the Gaussian mechanism M ( x 1 , . . . , x n ) := ∑ n i =1 x i + N (0 , σ 2 ) or the Laplace mechanism M ( x 1 , . . . , x n ) := ∑ n i =1 x i +Lap(0 , s ) .

The datasets D := (0 , . . . , 0) and D ′ := (0 , . . . , 0 , 1) form a dominating pair of datasets for M Poisson under the add relation and ( D ′ , D ) is a dominating pair of datasets under the remove relation.

Likewise, the datasets D := ( -1 , . . . , -1) and D ′ := ( -1 , . . . , -1 , 1) form a dominating pair of datasets for M WOR under the add relation and ( D ′ , D ) is a dominating pair of datasets under the remove relation.

The proposition implies that the hockey-stick divergence of the mechanisms with said datasets as input describes the privacy curves of the composed mechanisms under the add and remove relations, respectively. We contrast this good behavior of composed and subsampled mechanisms under add and remove separately with the Laplace mechanism, which, as we will see in Section 5, does not behave well when composed under the combined add/remove relation.

Our dominating pair of datasets can be found by reduction to one of the main results of [ZDW22].

Theorem 10 (Theorem 11 of [ZDW22]) . Let M be a randomized mechanism, let M Poisson be the γ -subsampled version of the mechanism, and let M WOR be the ( n b ) -subsampled version of the mechanism on datasets of size n and n -1 with γ = b/n .

If ( P, Q ) dominates M for add neighbors then ( P, (1 -γ ) P + γQ ) dominates M Poisson for add neighbors and ((1 -γ ) Q + γP,P ) dominates M Poisson for removal neighbors.

If ( P, Q ) dominates M for substitution neighbors then ( P, (1 -γ ) P + γQ ) dominates M WOR for add neighbors and ((1 -γ ) P + γQ,P ) dominates M WOR for removal neighbors.

In Appendix A we prove that Proposition 9 holds by showing that the hockey-stick divergence between the mechanism with the dominating pairs of datasets matches the upper bound from Theorem 10.

Crucially, Proposition 9 implies that under the add and remove relations, we must add noise with twice the magnitude when sampling without replacement compared to Poisson subsampling! The intuition behind this difference is that the subroutine behaves similarly to the add/remove neighboring relation when using Poisson subsampling, whereas it resembles the substitution neighborhood when sampling without replacement. When D ′ i is included in the batch another datapoint is 'pushed out' of the batch under sampling without replacement. Due to this parallel one might hope that the difference in privacy parameters between Poisson subsampling and sampling without replacement only differ by a small constant similar to the difference between the add/remove and substitution neighboring relations. That is indeed the case for many parameters, but as we show in Section 7 this assumption unfortunately does not always hold.

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

0.0

0.5

1.0

1.5

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

k

=1

Remove

Add

0.0

0.5

1.0

1.5

2.0

0.0

0.1

0.2

0.3

0.4

k

=2

Remove

Add

0

2

4

6

8

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

k

=8

Remove

Add

Privacy Curve of the

(

10

9

)

-subsampled Laplace(2) Mechanism (

k

Iterations)

5 No Worst-case Pair of Datasets under Add/Remove Relation

So far, we have considered the entire privacy curve for all ε ∈ R . This is a necessary subtlety for PLD privacy accounting tools under composition (e.g., Theorem 8). Here we focus only on the privacy curve for ε ≥ 0 . Our main result of this section is to give a minimal example of a mechanism M that admits a worst-case dataset pair under ∼ A/R yet M k does not admit any worst-case dataset pair for some k > 1 . This violates an implicit assumption made by some privacy accountants.

Proposition 11. For some mechanism M , the privacy curve of the ( n b ) -subsampled mechanism M WOR is realized by a pair of datasets under ∼ A/R , yet no pair of datasets realizes the privacy curve of M k WOR for all k > 1 .

A proof of this proposition for a simple mechanism can be found in Appendix B.1. However, it is more illustrative to demonstrate the proposition informally for the Laplace mechanism M . In this case, note that the proposition can be extended to M Poisson as well. The proposition stands in contrast to the case of the add and remove relations discussed in Proposition 9. That is, we can find datasets D ∼ A D ′ such that δ ∼ A M WOR is realized by ( D,D ′ ) and δ ∼ R M WOR is realized by ( D ′ , D ) , but no such (ordered) pair realizes the privacy curve under ∼ A/R .

Moreover, it is generally the case that the privacy curve of a subsampled mechanism without composition under ∼ R dominates the privacy curve under ∼ A when ε ≥ 0 (see, e.g., Proposition 30 of [ZDW22] or Theorem 5 of [MTZ19]). Specifically, it follows from Proposition 30 of [ZDW22] that in the case of the subsampled Laplace mechanism and ε ≥ 0 , we have that

Here we visualize the counter-example by plotting privacy curves for the add and remove relation in Figure 1. Note that δ ∼ A/R M WOR ( ε ) = max { δ ∼ A M WOR ( ε ) , δ ∼ R M WOR ( ε ) } . Figure 1 shows several variations of the curves δ ∼ A M k WOR and δ ∼ R M k WOR , which we estimated numerically by Monte Carlo simulation (as in, e.g., [WMW + 23]). Appendix B.2 has the methodological details. These curves are seen to cross in the region ε ≥ 0 for k = 2 compositions.

The phenomenon is most apparent for k = 2 . There is a clear break in the curve for the remove relation. Under many compositions, however, it is known that both PLDs converge to a Gaussian distribution [DRS19], which explains why this break vanishes as the number of compositions increases.

Avoiding incorrect upper bounds As shown in this section we cannot assume that the privacy curve for the remove relation dominates the add relation for composed subsampled mechanisms under ∼ A/R even though it is the case without composition. Luckily, this particular issue can be easily resolved by computing the privacy parameters for the add and remove relation separately and taking the maximum. This technique is already used in practice in, e.g., the Google DP library [Goo20].

We conjecture that this workaround is unnecessary for the Gaussian mechanism-the natural choice for DP-SGD. We searched a wide range of parameters and were unable to produce a counterexample.

Conjecture 12. Let M be the Gaussian mechanism with any σ . Then for all k > 0 , γ ∈ [0 , 1] , and ε ≥ 0 we have

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

6 Comparison of Sampling Schemes

In this section we explore the difference in privacy parameters between Poisson subsampling and sampling without replacement. We focus on the subsampled Gaussian mechanism which is the mechanism of choice for DP-SGD. We show that for some parameters the privacy guarantees of the mechanism differ significantly between the two sampling schemes.

There are several different techniques one might use when selecting privacy-specific hyperparameters for DP-SGD. One approach is to fix the value of δ and the number of iterations. Given a sampling rate γ and a value for ε , we can compute the smallest value for the noise multiplier σ such that the mechanism satisfies ( ε, δ ) -differential privacy. We use this approach to showcase our findings. We fix δ = 10 -6 and the number of iterations to 10 , 000 . We then vary the sampling rate between 10 -4 to 1 and use the PLD accountant implemented in the Opacus library [YSS + 21] to compute σ .

Subsampled Gaussian 10000 iterations 6= 10-6

Poisson ε = 1

Poisson =2

Poisson =5

102

Poisson=10

WOR ε = 10

Noise multiplier,

b

101

10°

+1-0T

10-4

10-3

10-2

10-1

100

Samplingrate,Y

In Figure 2 we plot the noise multiplier required to achieve ( ε, 10 -6 ) -DP with Poisson subsampling for ε ∈ { 1 , 2 , 5 , 10 } . For comparison, we plot the noise multiplier that achieves (10 , 10 -6 ) -DP when sampling without replacement. Recall from Section 4 that the noise magnitude required when sampling without replacement is exactly twice that required for Poisson subsampling. The plots are clearly divided into two regions. For large sampling rate, the noise multiplier scales roughly linearly in the sampling rate. However, for sufficiently low sampling rates the noise multiplier decreases much slower. This effect has been observed previously for setting hyperparameters (see Figure 1 of [PHK + 23] for a similar plot).

Avoiding problematic parameters It is generally advised to select parameters that fall into the 274 right-hand regime of the plots in Figure 2 [PHK + 23]. However, one might select parameters close to 275 the transition point. This can be especially problematic if the wrong privacy accountant is used. The 276 transition point happens when σ is slightly less than 1 for Poisson sampling and therefore it happens 277 when it is slightly less than 2 for sampling without replacement. The consequence can be seen for 278 the plot for sampling without replacement in Figure 2. When the sampling rates are high the noise 279 required roughly matches that for ε = 5 with Poisson subsampling. But when the sampling rate is 280 small we have to add more noise than is required for ε = 1 with Poisson subsampling. As such, if we 281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

use a privacy accountant for Poisson subsampling and have a target of ε = 1 but our implementation uses sampling without replacement the actual value of ε could be above 10 ! We might hope that this increase would be offset if we allow for some slack in δ as well. However, as seen in the table of Figure 1 there can still be a big gap in ε between the sampling schemes even when we allow a difference of several orders of magnitude in δ .

7 Substitution Neighboring Relation

In this section, we consider both sampling schemes under the substitution neighboring relation. In their work on computing tight differential privacy guarantees, [KJH20] considered worst-case distributions for the subsampled Gaussian mechanism under multiple sampling techniques and neighboring relations. In the substitution case, they compute the hockey-stick divergence between (1 -γ ) N (0 , σ 2 ) + γ N ( -1 , σ 2 ) and (1 -γ ) N (0 , σ 2 ) + γ N (1 , σ 2 ) . These distributions correspond to running the mechanism with neighboring datasets where all but one entry is 0 . We first consider Poisson subsampling in the proposition below and later discuss sampling without replacement.

Proposition 13. Consider the Gaussian mechanism M ( x 1 , . . . , x n ) := ∑ n i =1 x i + N (0 , σ 2 ) and let M Poisson be the γ -subsampled mechanism. Then D := (0 , . . . , 0 , 1) and D ′ := (0 , . . . , 0 , -1) form a dominating pair of datasets under the substitution neighboring relation.

Proposition 13 simply confirms that the pair of distributions considered by [KJH20] does indeed give correct guarantees as it is a dominating pair of distributions. However, as far as we are aware, no formal proof existed anywhere. Our proof of the proposition is in Appendix C.

In the rest of the section we focus on sampling without replacement. We start by restating another result from [ZDW22] which we use throughout the section.

Theorem 14 (Proposition 30 of [ZDW22]) . If ( P, Q ) dominates M under substitution for datasets of size γn , then under the substitution neighborhood for datasets of size n , we have

where δ ( α ) is the largest hockey-stick divergence of order α for M WOR on neighboring datasets.

Next, we address a mistake made in related work. We introduced the distributions considered by [KJH20] for Poisson subsampling above and we show in Proposition 13 that it is a dominating pair of distributions. However, [KJH20] claimed in their paper that the privacy curves are identical for the two sampling schemes under the substitution relation which is unfortunately incorrect.

They considered datasets where all but one entry has a value of 0 . This results in correct distributions for Poisson subsampling but for sampling without replacement, we instead consider the datasets D := ( -1 , . . . , -1 , 1) and D ′ := ( -1 , . . . , -1 , -1) . With these datasets the values of H α ( M WOR ( D ) ||M WOR ( D ′ )) and H α ( M WOR ( D ′ ) ||M WOR ( D )) match the cases of the upper bound in Theorem 14 for α ≥ 1 and α < 1 , respectively. This can be easily verified by following the steps of the proof of Proposition 9 for sampling without replacement.

We can use the datasets above to compute tight privacy guarantees for a single iteration. However, composition is more complicated since neither of the two directions corresponds to a dominating pair of distributions. One might hope that we could simply compute the hockey-stick divergence of the self-composed distributions in both directions and use the maximum similar to the add/remove case. However, for some mechanisms that is not sufficient because we can combine the directions unlike with the add and remove cases. Next we give a minimal counterexample using the Laplace mechanism to showcase this challenge.

̸

We consider datasets of size 2 and sample batches with a single element such that γ = 0 . 5 . Let x 1 and x 2 denote the two data points in D and without loss of generality assume that x 1 = x ′ 1 and x 2 = x ′ 2 , where x ′ 1 and x ′ 2 are the corresponding data points in D ′ . We apply the subsampled Laplace mechanism with a scale of 2 and perform 2 queries where x 1 has the value -1 for both queries. Let P := 0 . 5 · Lap( -1 , 2) + 0 . 5 · Lap(1 , 2) and Q := Lap( -1 , 2) . That is, P and Q are the distributions for running one query of M WOR ( D ) with x 2 having value 1 or -1 , respectively. Then H e ε ( P × P || Q × Q ) is the hockey-stick divergence for the mechanism if x 2 has value 1 for

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

M =(²)-subsampled Laplace mechanism (scale=2)2 iterations

0.25 -

He(M([-1, 1] × [-1, 1]) 11 M([-1, -1] × [-1, -1]))

He (M([-1, -1] × [-1, -1]) 1| M([-1, 1] × [-1, 1]))

He(M([-1, 1] × [-1, -1]) 11 M([-1, -1] × [-1, 1]))

0.20-

0.15

IId)H

0.10

0.05

0.00+

0.0

0.2

0.4

0.6

0.8

1.0

E

both queries and x ′ 2 has value -1 for both queries. Similarly, H e ε ( Q × Q || P × P ) is the divergence when x 2 has value -1 for both queries and x ′ 2 has value 1 for both queries.

The two hockey-stick divergences above are similar to those for the remove and add neighboring relations. However, we also have to consider H e ε ( P × Q || P × Q ) in the case of substitution. These distributions correspond to the case when x 2 has a value of 1 for the first query and -1 for the second query, and x ′ 2 has a value of -1 for the first query and 1 for the second query. Figure 3 shows the hockey-stick divergence as a function of ε for the three pairs of neighboring datasets. The largest divergence depends on the value of ε with all three divergences being the maximum for some interval. This counterexample shows that we cannot upper bound the hockey-stick divergence for the subsampled Laplace mechanism as max { H e ε ( P k || Q k ) , H e ε ( Q k || P k ) } for k > 1 . For k compositions, we have to consider k +1 ways of combining P and Q . This significantly slows down the accountants in contrast to the 2 cases required for add/remove. Worse still, we do not have a proof that one of k +1 cases is the worst-case pair of datasets for all ε ≥ 0 .

In Appendix D we use an alternative technique for bounding the privacy curve under the substitution relation based on [DGK + 22]. We show that this accountant does not generally outperform the RDP accountant. This demonstrates the need to strengthen the theory for sampling without replacement under the substitution relation for the purposes of tight privacy accounting.

8 Discussion

We have highlighted two issues that arise in the practice of privacy accounting.

First, we have given a concrete example where the worst-case dataset (for ε ≥ 0 ) of a subsampled mechanism fails to be a worst-case dataset once that mechanism is composed. Care should therefore be taken to ensure that the privacy accountant computes privacy guarantees with respect to a true worst-case dataset for a given choice of ε .

Secondly, we have shown that the privacy parameters for a subsampled and composed mechanism can differ significantly for different subsampling schemes. This can be problematic if the privacy accountant is assuming a different subsampling procedure from the one actually employed. We have shown this in the case of Poisson sampling and sampling without replacement but the phenomenon is likely to occur when comparing Poisson sampling to shuffling as well. Computing tight privacy guarantees for the shuffled Gaussian mechanism remains an important open problem. It is best practice to ensure that the implemented subsampling method matches the accounting method. When this is not practical, the discrepancy should be disclosed.

We conclude with two recommendations for practitioners applying privacy accounting in the DPSGD setting. We recommend disclosing the privacy accounting hyperparameters for the sake of reproducibility (see Section 5.3.3 of [PHK + 23] for a list of suggestions). Finally, we also recommend that, when comparisons are made between DP-SGD mechanisms, the privacy accounting for both should be re-run for the sake of fairness.

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

References

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

A Proof of Proposition 9

Without loss of generality, we show both parts for the Gaussian mechanism under the add neighboring relation only.

We first note that any pair of neighboring datasets with maximum ℓ 2 -distance is a dominating pair of datasets for the Gaussian mechanism [BW18]. Since the datapoints in our setting are from [ -1 , 1] this implies that ( N (0 , σ 2 ) , N (1 , σ 2 )) is a dominating pair of distributions for M under ∼ A and ( N ( r, σ 2 ) , N ( r +2 , σ 2 )) is a dominating pair of distributions for M under ∼ S for any r ∈ R . The distance of 2 is obtained by substituting -1 with 1 .

Now, let us prove part 1 of the proposition. To that end, let D be the all zeros dataset and let D ′ be D with a 1 appended to the end. The sum of the subsampled dataset is 1 if the last datapoint is included in the sample and 0 otherwise. As such, we have that

Since ( N (0 , σ 2 ) , N (1 , σ 2 )) is a dominating pair of distributions for M under ∼ A from Theorem 10 we have that

dominates M Poisson under ∼ A .

As for part 2, let γ := b/n for convenience, let D be the all -1 dataset, let D ′ be D with a single -1 substituted for a 1 . We can describe M WOR ( D ′ ) by considering the two cases where the 1 is either excluded or included in the batch of size b

Since ( N ( -b, σ 2 ) , N ( -b + 2 , σ 2 )) is a dominating pair of distributions for M under ∼ S from Theorem 10 we have that

dominates M WOR under ∼ A .

The proof for the remove direction is symmetric and the proof for the Laplace mechanism follows from replacing the normal distribution with the Laplace distribution.

B Details for Section 5

B.1 Proof of Proposition 11 for Randomized Response

Here we show that Proposition 11 holds using a simple mechanism. The mechanism is similar to randomized response [War65] which is used in differential privacy to privately release bits. The mechanism takes a dataset as input and randomly outputs a single bit. The output is weighted towards 0 if all entries of the dataset are 0 and towards 1 otherwise. Here we use this mechanism for the proof because the calculations and presentation are particularly clean and simple since there are only two outputs. A similar proof can be used to verify the accuracy of the estimated plots for the Laplace mechanism presented in Section 5 by calculating the exact hockey-stick divergence at, e.g., ε = 0 . 25 and ε = 1 . 5 .

where b ∈ { 0 , 1 } is 0 if all entries in D are 0 and 1 otherwise.

We use the dataset D that consists of all zeroes and D ′ is obtained from D by adding a single 1 . 487 We will present the proof using M Poisson , but it is the same for M WOR since the only effect on 488 the output distribution is whether or not the 1 is sampled in a batch. We use a sampling probability 489 of γ = 1 / 2 . Since the output distribution of M is symmetric this means that the probability for 490 M Poisson ( D ′ ) to output either bit is 1 / 2 · 3 / 4 + 1 / 2 · 1 / 4 = 1 / 2 . The counterexample occurs when 491

running the mechanism for 2 iterations. There are 4 possible outcomes of the two iterations. The 492 probability of any of these outcomes for M Poisson ( D ′ ) is 1 / 2 · 1 / 2 = 1 / 4 . For M Poisson ( D ) the 493 probability we can find the output distribution by considering each distinct outcome 494

Now, we find the hockey-stick divergence in both directions for α = 4 / 3 and α = 2 . We denote 495 the two distributions for running the mechanism as P = M Poisson ( D ) × M Poisson ( D ) and 496 Q = M Poisson ( D ′ ) ×M Poisson ( D ′ ) . 497

As such, we have that H 4 / 3 ( P || Q ) > H 4 / 3 ( Q || P ) and H 2 ( P || Q ) < H 2 ( Q || P ) . 498

499

500

501

502

503

504

505

506

B.2 Details of Monte Carlo Simulation

To produce Figure 1, we leverage the PLD framework and apply Monte Carlo simulation.

By Proposition 9 and Theorem 8, the privacy curve of the composed and subsampled Laplace mechanism under add (remove) is given by H e ε ( M Poisson ( D ) k ||M Poisson ( D ′ ) k ) (vice-versa for remove) where

On the other hand, a standard result (e.g. Theorem 3.5 of [GLW21]) asserts that the PLD of a composed mechanism is obtained by self-convolving the PLD of the uncomposed mechanism, namely

We estimate this expectation via sampling. We know the densities of M Poisson ( D ) = N (0 , σ 2 ) and 507 M Poisson ( D ′ ) = (1 -γ ) N (0 , σ 2 ) + γ N (1 , σ 2 ) , so we can quickly sample L M Poisson ( D || D ′ ) . By 508 drawing k samples and summing them, we can sample L M Poisson ( D || D ′ ) ⊕ k as well. Therefore, we 509 can draw Y i ∼ L M Poisson ( D || D ′ ) k for 1 ≤ i ≤ N , then compute the Monte Carlo estimate 510

As for the error, the quantity inside the expectation is bounded in [0 , 1] , so we can apply Höffding as 511 well as the union bound. In this case, 512

samples will suffice to ensure that the Monte Carlo estimate of H e ε ( M Poisson ( D ) ||M Poisson ( D ′ )) 513 is accurate within α , with probability 1 -β , for all ε ∈ E simultaneously. 514

For Figure 1, we chose α = 0 . 001 and β = 0 . 01 and considered | E | = 40 values of ε , which required 515 N = 3 , 342 , 306 samples. This value of α is small enough relative to the plot that our conclusion 516 holds with probability 99% . 517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

C Proof of Proposition 13

The proof relies mainly on the following data-processing inequality, which can also be seen as closure of privacy under post-processing.

Lemma 15. Let P and Q be any distributions on X and let Proc : X → Y be a randomized procedure. Denote by Proc P the distribution of Proc( X ) for X ∼ P . Then, for any α ≥ 0 ,

Proof. For any event E ⊆ Y ,

and the result holds since

We now prove the proposition. Our main goal is to argue that D := (0 , . . . , 0 , 1) and D ′ := (0 , . . . , 0 , -1) form a dominating pair of datasets for M Poisson . To that end, consider any ∼ S -neighbors that differ, without loss of generality, in the last entry, say ( x, a ) and ( x, a ′ ) . We leverage postprocessing to show that ( M Poisson ( x, a ) , M Poisson ( x, a ′ )) is dominated by ( M Poisson ( 0 , a ) , M Poisson ( 0 , a ′ )) . Indeed, consider

where ˆ x is randomly drawn from x by Poisson ( γ ) -subsampling. Now, sampling M Poisson ( 0 , a ) is equivalent to drawing ˆ a from the singleton dataset ( a ) via Poisson ( γ ) and returning a sample from N ( ∑ | ˆ a | i =1 ˆ a i , σ 2 ) . Since the normal distribution satisfies N ( a, σ 2 ) + b = N ( a + b, σ 2 ) , sampling Proc( M Poisson ( 0 , a )) is equivalent to sampling

where ˆ x is Poisson ( γ ) -subsampled from x and ˆ a is Poisson ( γ ) -subsampled from ( a ) . But, by independence, (ˆ x, ˆ a ) is a Poisson ( γ ) -subsample drawn from ( x, a ) , so, in conclusion, Proc( M Poisson ( 0 , a )) = M Poisson ( x, a ) . By an analogous argument, we have that Proc( M Poisson ( 0 , a ′ )) = M Poisson ( x, a ′ ) and hence

D Constructing a Dominating Pair of Distributions for the Gaussian Mechanism

In this section we consider the problem of computing privacy curves for the Gaussian mechanism under ∼ S when sampling without replacement. As shown in Section 7 computing tight parameters is challenging in this setting because we do not know which datasets result in the largest hockey-stick divergence. However, we can still compute an upper bound on the privacy curve using a dominating pair of distributions.

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

We modified the implementation of the algorithm introduced by [DGK + 22] in the Google DP library to construct the PLDs (Privacy Loss Distribution object). The algorithm constructs an approximation of the PLD from the hockey-stick divergence between the pair of distributions at a range of values for ε . From Theorem 14 we know that the direction of the pair of distributions yielding the largest hockey-stick divergence for the mechanism of a single iteration differs for α below and above 1 . We construct a new PLD by combining the two directions at α = 1 or ε = 0 .

See the left-side plot of Figure 4 for a visualization of how our construction uses the point-wise maximum of the hockey-stick divergence for a single iteration. This construction represents a dominating pair of distributions and as such it is sufficient to find a dominating pair of distributions for the composed mechanism using self-composition by Theorem 8.

The right-side plot of Figure 4 shows the privacy curve obtained from self-composing the PLD for the dominating pair of distributions with parameters σ = 4 , γ = 0 . 05 , and 1000 iterations. The blue line is the privacy curve under ∼ R and also serves as a lower bound for the true privacy curve. Note that the orange line would also be the privacy curve achieved by this technique under the add/remove relation if we did not consider the add and remove relations separately.

The gap between the upper and lower bound motivates future work for understanding the worst-case datasets. Similar to the add/remove case we conjecture that the subsampled Gaussian mechanism behaves well under composision. Specifically, we conjecture that the privacy curve of the composed subsampled Gaussian mechanism under ∼ S matches the curve under ∼ R for ε ≥ 0 . It seems likely that this is the case if Conjecture 12 holds. However, if Conjecture 12 does not hold the above statement also does not hold.

=2,b/n =0.5,iterations =1

=4,b/n =0.05,iterations =1000

10°

P = 0.5.N(-1, o²) + 0.5.N(1, o²), Q = N(-1, o²)

0.6

P = N(-1,o²), Q= 0.5.N(-1,o²) + 0.5.N(1, o²)

Dominating pair of distributions

10-2

0.5

10-4

0.4

Id)H

0.3

8

10-6

0.2

10-8

0.1

Lowerbound

10-10

Dominatingpairofdistributions

0.0

GooglePoissonRDPaccountant add/remove

-1.00

-0.75-0.50-0.25

0.00

0.25

0.50

0.75

1.00

0

1

2

T

3

T

4

5

E

E

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: A provide a comprehensive list of contribitions at the end of the introduction. A summary is given in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The main limitation of our work is expressed in Conjecture 12.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Each theoretical result is indicated as a proposition (theorems indicate prior work). A proof for each result can be found in the appropriate appendix section (references given in main body).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details for Monte Carlo simulation results (Figures 1 and 3) are in the appendix. Other experimental results can be obtained by straightforward modification of publicly available privacy accounting software.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: See previous justification. Instructions to reproduce Monte Carlo simulation results are included in the appendix. Other results rely on open-source code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Simulation results rely on a choice of sample size, which is explained in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: An analysis of sample size and the associated error is included in the appendix. The error is very small compared to the plots due to the high sample size, so we did not explicitly include them in simulation plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Experiments required minimal compute resources, so we do not report details. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the guidelines and found no violations in our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The aim of the work is to bring attention among practitioners and theoreticians to the limitations of privacy accountants. There is no foreseeable path to negative broad societal impact. On the other hand improving privacy accountants may lead to wider deployment of private machine learning, which can be expected to have a positive societal impact. We briefly discuss this outcome in the introduction in order to motivate our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Credit is given as needed to open-source software repositories.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

878

879

880

881

882

883

884

885

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present two potential sources of error which can arise when composing sub-sampled DP mechanisms. On one hand, they discuss cases in which the composition of worst-case datasets does not yield the expected result, on the other hand, they disambiguate guarantees for mechanisms with Poisson sampling vs. sampling WOR.\n\nSTRENGTHS:\nI appreciate that this paper points out some of the subtleties which arise regarding the distinctions between worst-case databases and dominating pairs and regarding the correct accounting of specific sampling schemes (Poisson/WOR). These subtleties can pass unnoticed, and lead to errors which compromise the privacy of individuals.\n\nWEAKNESSES:\nThere is nothing particularly wrong with the paper. The facts stated are valid, and they are interesting, especially since they point out potential sources of confusion. However, none of this is surprising or even particularly novel. Most of this information is implied by earlier work (Zhu et al. in particular), and some of the facts stated here would be better suited as GitHub issues on the relevant accountants, followed by a technical report at a venue like TPDP or Journal of Privacy and Confidentiality. That is to say: I am not against this paper in general, but this is not a NEURIPS paper to me. It is a highly specialised piece of technical writing with a very narrow scope, and is likely to be of interest only to a very small community. I would recommend the authors to submit it to a venue which is better suited to its content.\n\nQUESTIONS:\nSome suggestions for improvement: \n\n- Definition 1: The \"only if\" does not hold here. For example, the mechanism also satisfies epsilon/delta-DP if it satisfies epsilon/delta probabilistic DP, i.e. the PLRV is bounded with probability at least 1-delta, but not the other way around.\n- Definition 3: The hockey-stick divergence is asymmetric in general. A more appropriate terminology would be \"of P from Q\"\n- There are some measure-theoretic constructs used, but there are no assumptions stated about absolute continuity (think (0, delta)-DP) or about whether densities exist (Definition 5), and the notation in Definition 5 is a bit uncommon. A nitpick: the \"d\" is a differential operator, and it's recommended to write it as $\\mathrm{d}$\n- There is a subtlety about Poisson sampling in Definition 6: Treating the **expected batch size** as public is fine, but the footnote just says \"batch size\". The **actual** batch size (the realised one), should still be kept secret right?\n- 160: Referring to $D \\sim D'$ as a dominating pair of datasets is a recipe for confusion in my opinion. The established terminology is to refer to dominating pairs of measures (or distributions). Perhaps state that the distributions under D, D' are dominating pairs, but not the databases themselves. After all, the whole point of even considering dominating pairs is (allegedly) to not have to think in terms of databases.\n- Perhaps use \"databases\" consistently rather than mixed with \"datasets\"\n- In 244: Your paper is aimed at an audience who are not DP theorists. I think just stating that \"they converge to a Gaussian distribution\" does not really inform this audience about why the crossing vanishes. You'd probably have to explain that the privacy profiles become pointwise equal because the PLDs now become symmetric about the expectations.\n- In Figure 2, just because the mechanisms satisfy the same delta and epsilon, this does not make them identical or even comparable in general (they are only \"identical\" if they have equal privacy curves or trade-off functions). The figure seems to rely in a sense on the mechanisms being otherwise identical. Is there a limitation which needs addressing here?\n- The Gopi accountant is known to break for low delta. Did you encounter any problems for delta 1e-7? Is there any chance the results could be spurious for low delta values?\n\nLIMITATIONS:\nThe discussion on limitations is a bit lacking in my opinion. The authors state (in the checklist) that the \"main limitation is expressed in Conjecture 12\". Not being able to find a counterexample for a proposition is not really what one understands under the term \"limitation\" of a work. In particular, the supposed \"limitation\" is --by the authors' own admission-- easily resolved by just running the accountant on the two curves separately and taking the supremum. I would have much preferred an experimental section where the consequences of the pitfalls stated in the work are actually shown to affect the real-world use of DP-SGD or other mechanisms, and/or to see that specific privacy threats are practically enabled by overlooking these subtleties (e.g. through auditing).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

comments: SUMMARY:
This paper examines the discrepancies between privacy accounting methods and their implementations, highlighting several cases where these mismatches lead to incorrect results. Specifically, it compares the noise requirements for achieving privacy guarantees under Poisson sampling versus sampling without replacement, and explores the limitations of worst-case dataset assumptions in subsampled mechanisms. Additionally, the authors address challenges in computing tight differential privacy (DP) bounds under the substitution relation of neighboring datasets.

STRENGTHS:
- The paper addresses an important and timely topic in the field of differential privacy regarding privacy accounting.

- The findings have strong practical implications, potentially preventing unintended privacy breaches.

- The authors' message is well articulated, promoting better practices among DP practitioners.

- Despite critiquing existing methods, the authors maintain a respectful and constructive tone.

WEAKNESSES:
- The different messages of the paper may be convoluted sometimes, which makes the paper hard to follow.

- No viable technical solutions are provided for the identified issues, which might be a difficult research problem.

QUESTIONS:
- In the equation between lines 133 and 134 (and in the rest of the article), where did the maximum with $0$ go ? When I try to recover expression on the right-hand side, I obtain $E_Y \max(0, 1-e^{\epsilon - Y})$.

LIMITATIONS:
The authors discuss the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

comments: SUMMARY:
The two main contributions of the paper is as follows:

the privacy guarantee of composition of subsampled mechanism may not be defined by worst-case dataset(s) for the underlying mechanism
Poisson subsampling and sampling without replacement may not have similar privacy guarantee.

STRENGTHS:
The paper studies a very important problem of composition of subsampled privacy mechanism. There has been a lot of work in the recent past that performs a tight privacy accounting. This work is in the line of these works. These accounting results are used in deployment as well to show how much privacy loss has happened during training when using a prescribed noise scale. Based on these bounds, the training is stopped once we have expired the privacy budget. In this regard, their second result is very important because we definitely use subsampling without replacement in DP-SGD.

WEAKNESSES:
There are some typos, and the result for the gap is shown empirically. I have to state that I have not seen the Appendix so if the authors have a provable guarantee for this gap in the Appendix, please point me that. To me, the selling point of the paper is this result and it should be placed front and center. Most of the results that are given in the form of propositions and lemma are from previous works.

QUESTIONS:
Is there a provable gap between the composition of WoR and Poisson subsampling?

LIMITATIONS:
Mostly seem like an empirical study of the composition result.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the notion sampling with replacement for differential privacy. Most of the literature on machine learning with differential privacy benefits from privacy amplification by poisson sampling in the privacy analysis. However, when implementing the mechanisms, engineers ofter use the sub-sampling with replacement as a substitude for poisson sampling, mainly due to efficiency issues. This paper studies the gap between these two settings. Their main contributions are as follows: \n\n - Identifying the Problem with DP-SGD Implementations: The authors highlight a critical issue with implementations of (DP-SGD). They argue that many implementations incorrectly assume that Poisson sampling and batch-sampling yield similar privacy guarantees, which is not necessarily true.\n\n- Gap between Batch-Sampling and Poisson Sampling: The paper demonstrates a significant privacy gap between these two sampling methods. They provide an example showing that for certain hyperparameters, Poisson subsampling can result in an ϵ≈1, whereas batch-sampling without replacement can result in an ϵ>10. This discrepancy is critical for privacy accounting in DP-SGD.  Authors compare the privacy guarantees of Poisson subsampling and batch-sampling. They show that the privacy guarantees can differ significantly depending on the sampling technique used. Their analysis reveals that the method of sampling batches (Poisson vs. fixed-size) significantly impacts the resulting privacy guarantees, cautioning against the interchangeable use of different sampling techniques in privacy analysis.\n\nSTRENGTHS:\n- Identifying an important problem with implementation of DP-SGD\n\nWEAKNESSES:\n- I have some concerns about the correctness of the results.\n\n- There is not much technical novelty.\n\nQUESTIONS:\n- You introduce the notion of dominating pairs of distributions, but then you talk about dominating datasets. You need to clarify the relation. Specifically in Proposition 9, I don't understand what a dominating dataset means.\n\n- Proposition 9 looks incorrect to me. In case of b=n and \\lambda=1, the dominating pairs should collide, but this proposition suggests otherwise. \n\n- You say: \"Crucially, Proposition 9 implies that under the add and remove relations, we must add noise with twice the magnitude when sampling without replacement compared to Poisson subsampling!\". This doesn't sound correct. Again, if you set the sampling rate to 1, then the two mechanisms collide. Am I missing something?\n\n- For the results of section 6, are you using the worst-case datasets of Proposition 9 to calculate the privacy curve?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}