Title: Have it your way: Individualized Privacy Assignment for DP-SGD
Venue: NeurIPS 2023
Accepted: yes
Keywords: privacy

machine learning

differential privacy

DP-SGD

individualized privacy
Abstract: When training a machine learning model with differential privacy, one sets a privacy budget. This uniform budget represents an overall maximal privacy violation that any user is willing to face by contributing their data to the training set. We argue that this approach is limited because different users may have different privacy expectations. Thus, setting a uniform privacy budget across all points may be overly conservative for some users or, conversely, not sufficiently protective for others. In this paper, we capture these preferences through individualized privacy budgets. To demonstrate their practicality, we introduce a variant of Differentially Private Stochastic Gradient Descent (DP-SGD) which supports such individualized budgets. DP-SGD is the canonical approach to training models with differential privacy. We modify its data sampling and gradient noising mechanisms to arrive at our approach, which we call Individualized DP-SGD (IDP-SGD). Because IDP-SGD provides privacy guarantees tailored to the preferences of individual users and their data points, we empirically find it to improve privacy-utility trade-offs.

Have it your way: Individualized Privacy Assignment for DP-SGD

Abstract

When training a machine learning model with differential privacy, one sets a privacy budget. This uniform budget represents an overall maximal privacy violation that any user is willing to face by contributing their data to the training set. We argue that this approach is limited because different users may have different privacy expectations. Thus, setting a uniform privacy budget across all points may be overly conservative for some users or, conversely, not sufficiently protective for others. In this paper, we capture these preferences through individualized privacy budgets. To demonstrate their practicality, we introduce a variant of Differentially Private Stochastic Gradient Descent (DP-SGD) which supports such individualized budgets. DP-SGD is the canonical approach to training models with differential privacy. We modify its data sampling and gradient noising mechanisms to arrive at our approach, which we call Individualized DP-SGD (IDP-SGD). Because IDPSGD provides privacy guarantees tailored to the preferences of individual users and their data points, we empirically find it to improve privacy-utility trade-offs.

1 Introduction

Machine learning (ML) models are known to leak information about their training data. Such leakage can result in attacks that determine whether a specific data point was used to train a given ML model (membership inference) [36, 43, 8], infer sensitive attributes from the model's training data [13, 45], or even (partially) reconstruct that training data [15, 35]. Therefore, the need to provide privacy guarantees to the individuals who contribute their sensitive data to train ML models is pressing.

Approaches to train ML models with guarantees of differential privacy (DP) [10] have established themselves as the canonical answer to this need. In the context of ML, differential privacy bounds how much can be learned about any of the individual data points from the model's training set. Take the canonical example of differentially private stochastic gradient descent (DP-SGD): it updates the model using noisy averages of clipped per-example gradients. Unlike vanilla SGD, this ensures that any single training point has a limited influence on model updates (due to the clipping to a pre-defined maximum clip norm) and that the training is likely to output a similar model should a single point

be added to or removed from the training set (due to the noisy average). As is the case for any DP algorithm, the privacy guarantees of DP-SGD are derived analytically. They are captured by a privacy budget ε . This budget represents the maximal privacy violation that any user contributing data to the training set is willing to tolerate. Lower budgets correspond to stronger privacy protection since they impose that the outputs of the training algorithm have to be closer for similar training sets.

In this paper, we argue that there is a key limitation when training ML models with DP guarantees: the privacy budget ε is set uniformly across all training points. This implicitly assumes that all users who contribute their data to the training set have the same expectations of privacy. However, that is not true; individuals have diverse values and privacy preferences [17, 5]. Yet, because DP-SGD assumes a uniform privacy budget, this budget must match the individual whose privacy expectations are the strongest. That is, ε has to correspond to the lowest privacy violation tolerance expressed amongst individuals who contributed their data. This limits the ability of DP-SGD to learn because the privacy budget ε together with the sensitivity of each step of the algorithm ( i.e ., the clip norm) define the scale of noise that needs to be added: the stronger the required privacy protection, the higher the noise scale. Thus, implementing privacy protection according to the lowest privacy violation tolerance in the training dataset comes at a significant cost in model accuracy.

We are the first to propose two variants of the DP-SGD algorithm, that we refer to as IDP-SGD, to enforce different privacy budgets for each training point. 2 First, our modified sampling mechanism ( Sample ) samples data points proportionally to their privacy budgets: points with higher privacy budgets are sampled more frequently during training. It is intuitive that the more a training point is analyzed during training, the more private information can leak about this point. Second, with our individualized scaling mechanism ( Scale ), we rescale the noise added to gradients based on the budget specified for each training point. Naively, one would implement Scale by changing the scale of added noise per individual data point. However, for efficiency, current implementations of DP-SGD typically noise the sum of the per-example clipped gradients over an entire mini-batch, hence, adding the same amount of noise to all gradients. Yet, within the context of this implementation, we note that we can effectively adapt the scale of noise being added on an individual basis by adjusting the sensitivity ( i.e . the clip norm) of each example by a multiplier. Because clipping is performed on a per-example basis, this approach enables the efficient implementation of our proposed Scale method.

To summarize our contributions:

We introduce two novel individualized variants of the DP-SGD algorithm, where each data point can be assigned its desired privacy budget. This allows individuals who contribute data to the training set to specify different privacy expectations.

We provide a theoretical analysis of our two mechanisms to derive how the privacy parameters ( e.g ., noise multipliers and sample rates) should be set to meet individual data points' privacy guarantees and provide a thorough privacy analysis for Sample and Scale .

We carry out an extensive empirical evaluation on various vision and language datasets and multiple model architectures. Our results highlight the utility gain of our methods for different distributions of privacy expectations among users.

Ethical considerations. Weneed to ensure that our individualized privacy assignment does not harm individuals by disclosing too much of their sensitive information. We, therefore, suggest implementing our methods in a controlled environment where privacy risks are openly communicated [41, 9, 12], and where a regulatory entity is in charge of ensuring that even the individuals with lowest expectations regarding their privacy obtain a sufficient degree of protection. We further discuss the ethical deployment of our Individualized DP-SGD (IDP-SGD) in Appendix A.

2 Background and Related Work

2.1 Differential Privacy and DP-SGD

Algorithm M satisfies ( ε, δ ) -DP , if for any two datasets D,D ′ ⊆ D that differ in any one record x , such that D = D ′ ∪ x , and any set of outputs R

Since the possible difference in outputs is bounded for any pair of datasets, DP bounds privacy leakage for any individual. The value ε ∈ R + specifies the privacy level with lower values corresponding to stronger privacy guarantees. The δ ∈ [0 , 1] offers a relaxation, i.e ., a small probability of violating the guarantees. See Appendix B.1 for a more thorough introduction of ( ε, δ ) -DP.

Another relaxation of DP is based on the Rényi divergence [27], namely ( α, ε ) -RDP , which for the parameters as described above and an order α ∈ (1 , ∞ ) is defined as follows: D α ( M ( D ) ∥ M ( D ′ )) ≤ ε . Due to its smoother composition properties, it is employed for privacy accounting in ML with DP-SGD. Yet, it is possible to convert between the two notions: If M is an ( α, ε ) -RDP mechanism, it also satisfies ( ε +log α -1 α -log δ +log α α -1 , δ ) -DP for any 0 < δ < 1 [3].

DP-SGD extends standard SGD with two additional steps, namely clipping and noise addition. Clipping each data point's gradients to a pre-defined clip norm c bounds their sensitivity to ensure that no data point causes too large model updates. Adding noise from a Gaussian distribution, N (0 , ( σc ) 2 I ) , with zero mean and standard deviation according to the sensitivity c and a pre-defined noise multiplier σ introduces privacy. We formally depict DP-SGD as Algorithm 1. To yield tighter privacy bounds, DP-SGD implements privacy amplification by subsampling [4] (line 3 in Algorithm 1) where training data points are sampled into mini-batches with a Poisson sampling, 3 rather than assigning each data point from the training dataset to a minibatch before an epoch starts. DP-SGD hence implements a Sampled Gaussian Mechanism

Algorithm 1: Differentially Private SGD [1]

Require: Training dataset D with data points

{ x 1 , . . . , x N } , loss function l , learning rate η , noise multiplier σ , sampling rate q , clip norm c , number of training iterations I .

1: Initialize θ 0 randomly

2: for t ∈ [ I ] do

3: Poisson Sampling : Sample mini-batch L t with per-point probability q from D .

4: For each i ∈ L t , compute g t ( x i ) ←∇ θ t l ( θ t , x i )

5: 1. Gradient Clipping

6: ¯ g t ( x i ) ← g t ( x i ) / max ( 1 , ∥ g t ( x i ) ∥ 2 c )

7: 2. Noise Addition

8: ˜ g t ← 1 | L t | (∑ i ¯ g t ( x i ) + N (0 , ( σc ) 2 I ) ) 9: θ t +1 ← θ t -η ˜ g t

10:

end for

11: Output θ I , privacy cost ( ε, δ ) computed using a privacy accounting method.

(SGM) [28] where each training data point is sampled independently at random without replacement with probability q ∈ (0 , 1] over I training iterations. As shown by Mironov et al . [28], an SGM with sensitivity of c = 1 , satisfies ( α, ε ) -RDP where

Equation (2) highlights that the privacy guarantee ε depends on the noise multiplier σ , the sample rate q , the number of training iterations I , and the RDP order α .

2.2 Individualized Privacy

Individualized Privacy is a topic of high importance given that society consists at least of three different groups of individuals, demanding either strong, average, or weak privacy protection for their data, respectively [17, 5]. Furthermore, some individuals inherently require higher privacy protection, given their personal circumstances and the degree of uniqueness of their data. However, without individualization, when applying DP to datasets that hold data from individuals with different privacy requirements, the privacy level ε has to be chosen according to the lowest ε encountered among all individuals, which favors poor privacy-utility trade-offs. Prior work on individualized privacy guarantees focused mainly on standard data analyses without considering ML [2, 19, 24, 31]. However, some of the underlying ideas inspired the design of our IDP-SGD variants.

Our Sample method relies on a similar idea as the sample mechanism proposed by Jorgensen et al. [19] where data points with stronger privacy requirements are sampled with a lower probability than data points that agree to contribute more of their information. The most significant difference between the sample mechanism and our approach is that the former performs sampling as a pre-processing step independent of the subsequent algorithm. In contrast, we perform subsampling within every iteration of our IDP-SGD training and show how to leverage the sampling probability of DP-SGD to obtain individual privacy guarantees. Based on this difference, we could not build on how Jorgensen

et al. [19] compute the sampling probabilities and had to propose an original mechanism to derive sampling probabilities for individualized privacy in IDP-SGD.

In a similar vein to our Scale method, Alaggan et al. [2] scale up data points individually before the noise addition to increase their sensitivity in their stretching mechanism . As a consequence, the signal-to-noise ratios of data points that are scaled with large factors will be higher than the ones of data points that are scaled with small factors. In contrast to Alaggan et al. [2], we do not scale the data points themselves, but the magnitude of noise added to their gradients, relative to their individual sensitivity. Our new interesting observation is that data points' individual clip norms can be used to indirectly scale the noise added to a mini-batch of data in DP-SGD per data point.

Closest work to ours [6] proposes two mechanisms for an individualized privacy assignment within the framework of Private Aggregation of Teacher Ensembles (PATE) [33]. Their upsampling duplicates training data points and assigns the duplicates to different teacher models according to the data points' privacy requirements, while their weighting changes the aggregation of the teachers' predicted labels to reduce or increase the contribution of teachers according to their respective training data points' privacy requirements. PATE's approach for implementing ML with DP differs significantly from DPSGD: PATE relies on privately training multiple non-DP models on the sensitive data and introducing DP guarantees during a knowledge transfer to a separately released model. The approaches for individualized privacy in PATE, therefore, are non-applicable to DP-SGD. See Appendix B.3 for details on the PATE algorithm and its individualized extensions.

Relationship to Individualized Privacy Accounting. An orthogonal line of research that comes closest to providing individualized guarantees for DP-SGD focuses on individualized privacy accounting ; the idea is to learn more from points that consume less of the privacy budget over training [11, 44]. Jordon et al. [18] propose a personalized moments' accountant to compute the privacy loss on a per-sample basis. Yu et al. [44] perform individualized privacy accounting within DP-SGD based on the gradient norms of the individual data points. Feldman and Zrnic [11] introduce RDP filters to account for privacy consumption per data point and train as long as there remain data points that do not exceed the global privacy level. Yet, it has been shown that the data points that consume little privacy budget and therefore remain in the training are the ones that already incur a small training loss [44]. Training more on them will not significantly boost the model accuracy. While privacy assignment and accounting are concerned with improving privacy-utility trade-offs within DP, they operate under different setups. Privacy accounting (in the above three methods) assumes a single fixed privacy budget assigned over the whole dataset. In contrast, our privacy assignment is concerned with enabling different individuals to specify their respective privacy preferences. Since individual accounting and assignment are two independent methods, we experimentally show their synergy by applying individual assignment to individual accounting [11], which yields higher model utility than individual accounting alone.

3 Our Individualized Privacy Assignment

3.1 Formalizing Individualizing Privacy

Setup and Notation. Given a training dataset D with points { x 1 , . . . , x N } which each have their own privacy preference (or budget ), we consider data points with the same privacy budget ε p together as a privacy group G p . This notation is aligned with [17, 5] that identified different groups of individual privacy preferences within society. We denote with ε 1 the smallest privacy budget encountered in the dataset. Standard DP-SGD needs to set ε = ε 1 to comply with the strongest privacy requirement encountered within the data. For all ε p , p ∈ [2 , P ] , it follows that ε p > ε and we arrange the groups such that ε p > ε p -1 . Note that following [2], we argue that the privacy preferences themselves should be kept private to prevent leakage of sensitive information which might be correlated with them. If there is a necessity to release individual privacy preferences, this should be done under the addition of noise to obtain DP guarantees for the privacy preferences. This can for instance be done through a smooth sensitivity analysis, as done in prior work, e.g . [34]. In general, we find that it is not necessary to release the privacy budgets: To implement IDP-SGD, only the party who trains the ML model on the individuals' sensitive data needs to know their privacy budgets to set the respective privacy parameters accordingly during training. Given that this party holds access to the data itself, this does not incur any additional privacy disclosure for the individuals. Other parties then only interact with the final private model (but not with the individuals' sensitive data or their

associated privacy budgets) and it has been shown impractical to disclose privacy guarantees through black-box access to ML models [14]. We discuss the confidentiality of the privacy budgets further in Appendix A.2.

Individualized Privacy. Following [19], our methods aim at providing data points x i ∈ G p with individual DP (IDP) guarantees according to their privacy budget ε p as follows: The learning algorithm M satisfies ( ε p , δ ) -IDP if for all datasets D x i ∼ D ′ which differ only in x i and for all outputs R ⊆ R

Without loss of generality, we assume that all individual data points have privacy preferences with the same δ . Note that other notions of DP ( e.g ., RDP) can analogously be generalized to enable per-data point privacy guarantees. The key difference between Equation (3) and the standard definition of DP (Equation (1)) does not lie in the replacement of ε with ε p . Instead, the key difference is about the definition of neighboring datasets. For standard DP, neighboring datasets are defined as D ∼ D ′ where D and D ′ differ in any random data point (given that it is standard DP, every data point has privacy ε ). In contrast, in our work, following [19] the neighboring datasets are defined as D x i ∼ D ′ where D and D ′ differ in any arbitrary data point x i which has a privacy budget of ( ε p , δ ) . This yields to the individualized notion of DP.

We formalize the relationship between the individualized notion of DP (which we denote by ( { ε 1 , ε 2 , . . . , ε P } , δ ) -IDP) and standard ( ε, δ ) -DP in the following two lemmas over the learning algorithm M . The proofs are included in Appendix G.

Lemma 3.1. An algorithm M that satisfies ( ε 1 , δ ) -DP also satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -IDP. Lemma 3.2. An algorithm M that satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -IDP also satisfies ( ε P , δ ) -DP.

3.2 From Individualized Privacy Preferences to Privacy Parameters

From the individual privacy preferences and the total given privacy distribution over the data, i.e ., the number of different privacy budgets P , their values ε p , and the sizes of the respective privacy groups |G p | , we derive individual privacy parameters for IDP-SGD such that all data points within one privacy group (same privacy budget) obtain the same privacy parameters, and such that all privacy groups are expected to exhaust their privacy budget at the same time, after I training iterations.

Table 1: (Individualized) Privacy Bounds. DP-SGD SAMPLE SCALE ε ≤ I · 2 q 2 α σ 2 ε p ≤ I · 2 q p 2 α σ sample 2 ε p ≤ I · 2 q 2 α σp 2 The individualized parameters for our Sample method are the individual sample rates { q 1 , . . . , q P } , and a respective noise multiplier σ sample-commontoall privacy groupswhich we derive from the privacy budget distribution as described in Section 3.3. In Scale , the individualized parameters are noise multipliers { σ 1 , . . . , σ P } with their respective clip norms { c 1 , . . . , c P } , as we explain in Section 3.4. Our individual bounds per privacy group are depicted in Table 1. We omit α from consideration since its optimal value is selected as an optimization parameter when converting from RDP to ( ε, δ ) -DP.

3.3 Our Sample Method

Our Sample method relies on sampling data points with different sample rates { q 1 , . . . , q P } depending on their individual privacy budgets. In this case, the noise multiplier σ sample is fixed. Data points with higher privacy budgets (weaker privacy requirements) are assigned higher sampling rates than those with lower privacy budgets. This modifies the Poisson sampling for DP-SGD (line 3 in Algorithm 1) to sample data points with higher privacy budgets within more training iterations.

Deriving Parameters. We aim at deriving the privacy parameters that yield the specified individual privacy preferences. The learning hyperparameters, such as mini-batch size B or learning rate η can be found through hyperparameter tuning. 4 For Sample , given a tuned mini-batch size B , we have to find

{

q

1

, . . . , q

P

}

, such that their weighted average equals

q

:

1

N

P

p

=1

|G

p

|

q

p

=

q

=

B

N

.

∑

!

5

Algorithm 2: Finding Sample Parameters. The subroutine getSampleRate is equivalent to Opacus' function get_noise_multiplier [32], (see getNoise inAlgorithm 5). We sketch getSampleRate in Algorithm 4 in Appendix C.3.

Algorithm 3: Finding Scale Parameters. The subroutine getNoise corresponds to Opacus' function get_noise_multiplier [32]. We sketch its implementation in Algorithm 5 in Appendix C.3.

Require: Per-group target privacy budgets { ε 1 , . . . , ε P } , target δ , iterations I , number of total data points N and per-privacy group data points {|G 1 | , . . . , |G P |} , default clip norm c , sample rate q

1: init { σ 1 , . . . , σ P } where for p ∈ [ P ] :

2: σ p ← getNoise( ε p , δ, q, I )

3: init σ scale : σ scale ← ( 1 N ∑ P p =1 |G p | σ p ) -1 {see derivation of σ scale in Appendix C.2}

4: for p ∈ [ P ] :

5: c p ← σ scale c σ p

6: Output σ scale , { c 1 , . . . , c P }

This asserts that the Poisson sampling (line 3 in Algorithm 1) will yield mini-batches of size B in expectation. We also need to ensure that the privacy budgets of all groups will exhaust after I training iterations, which we do by deriving the adequate σ sample, shared across all privacy groups. Equation (2) highlights that the final privacy budget ε depends on both sample rate q and noise multiplier σ . Our Sample method enables higher individualized privacy budgets for the different groups than the standard DP-SGD but still requires setting the average sampling rate equal to q as in DP-SGD to obtain an expected mini-batch size B . Hence, the value of σ sample has to be decreased in comparison to the initial default σ . As an effect of this noise reduction, Sample improves utility of the final model.

Our concrete derivation of values for σ sample and { q 1 , . . . , q P } is presented in Algorithm 2. We start from initializing σ sample with σ from standard DP-SGD, the noise multiplier required for the privacy group G 1 (strongest privacy requirement of all groups) with the smallest privacy budget ε 1 . This corresponds to instantiating σ sample with the upper bound noise over all privacy groups. Then, we use a getSampleRate function that derives the sampling rate for the given privacy parameters based on approximating Equation (2). Finally, we iteratively decrease σ sample by a scaling factor s i slightly smaller than one and recompute { q 1 , . . . , q P } until their weighted average is (approximately) equal to q . In Appendix G, we present the formal proof that Sample satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -DP. Our proof relies on Mironov et al. [28] and considers training of each privacy group as a separate SGM-with an individual sampling probability-that all simultaneously update the same model.

3.4 Scale

Our Scale method aims at scaling the noise added to each gradient according to the respective data point's privacy preference. Yet, for efficiency, current implementations of DP-SGD typically noise the sum of the per-example clipped gradients over an entire mini-batch, hence, adding the same amount of noise to all gradients. Therefore, we instead set individualized clip norms { c 1 , . . . , c P } that effectively adapt the scale of noise being added on an individual basis by adjusting the sensitivity ( i.e . the clip norm) of each example by a multiplier. This changes lines 6 and 8 in the DP-SGD Algorithm (1). Data points with higher privacy budgets (weaker privacy requirements) obtain lower noise and higher clip norms. Since Equation (2) highlights that the clip norm c has no direct impact on the obtained ε , individualized privacy in Scale results from the individual noise multipliers { σ 1 , . . . , σ P } . Utility gains come from the overall increase in the signal-to-noise ratio during training.

Deriving Parameters. While the ultimate goal of our Scale method is to adapt individual noise multipliers per privacy group, we cannot implement this directly without degrading training performance. The reason is that whereas in DP-SGD sampling and gradient clipping are performed on a per-data point basis, noise is added per mini-batch, see lines 3, 6 and 8 in Algorithm 1, respectively. However, if we restrict mini-batches to contain only data points from the same privacy group (which

share the same noise multiplier), we lose the gains in privacy-utility trade-offs which result from the subsampling (see Appendix F.1 for more details). Hence, while we rely on mini-batches containing data points with different privacy requirements ( i.e ., different noise multipliers) we can only specify one fixed noise multiplier σ scale .

To overcome this limitation, we do not set noise multipliers { σ 1 , . . . , σ P } directly, but indirectly obtain them through individualized clip norms { c 1 , . . . , c P } as follows: In standard DP-SGD, Algorithm 1, a gradient clipped to c (line 6) obtains noise with standard deviation σc (line 8). For Scale , we clip gradients to c p = s p c with a per-privacy group scaling factor s p and they obtain noise σ p c p . But in practice, we add noise according to σ scale c to all mini-batches. Hence, the effective scale σ p of added noise is σ scale c = σ scale c p s p ! = σ p c p ⇒ σ p = 1 s p σ scale. For data points with higher privacy budgets s p > 1 , so their gradients are clipped to larger norms c p = s p c and a smaller noise multiplier σ p = 1 s p σ scale is assigned to them. The opposite is true for data points with lower privacy budgets.

We find the values of σ p required to obtain each privacy groups' desired ε p using the getNoise function (see Algorithm 3 line 1), which takes as inputs the privacy parameters ε p , δ, q, I . To optimize utility, we want to set the individual clip norms such that their average over the dataset corresponds to the best clip norm c obtained through initial hyperparameter tuning: 1 N ∑ P p =1 |G p | c p ! = c . Given the interdependence of c p , σ p , and σ scale ( c p = c σ scale σ p ), this can be achieved by setting σ scale as the inverse of the weighted average over all 1 /σ p as: σ scale = ( 1 N ∑ P p =1 |G p | σ p ) -1 (please see Appendix C.2 for the derivation of σ scale). Given σ scale, the required { σ 1 , . . . , σ P } , and c found through hyperparamter tuning, the individual clip norms are calculated as c p = σ scale c σ p . We detail the derivation of the parameters in Algorithm 3. In Appendix G, we formally show that Scale satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -DP). Similar to Sample , the proof is based on considering the training for all privacy groups as simultaneously executed SGMs with differing sensitivities.

4 Empirical Evaluation

For our empirical evaluation, we implement our methods in Python 3.9 and extend standard Opacus with our individualized privacy parameters and a per-privacy group accounting. We perform evaluation on the MNIST [23], SVHN [29], and CIFAR10 [22] dataset, using the convolutional architectures from Tramer and Boneh [38] for most experiments, and additionally evaluate on various language datasets and diverse (larger) model architectures in Section 4.2. To evaluate the utility of our methods, we use the datasets' standard train-test splits and report test accuracies. The training and standard DP-SGD and IDP-SGD hyperparameters are specified in Table 5 and Table 6 in Appendix D, respectively where the noise multiplier σ is derived with the function get_noise_multiplier provided in Opacus [32]. It takes in as arguments the specified parameters δ, q, I , and target ε . For experimentation on individualized privacy, we assigned privacy budgets randomly to the training data points if not indicated otherwise.

4.1 Utility Improvement and General Applicability of Individualization

Assigning heterogeneous individual privacy budgets over the training dataset yields significant improvements in terms of the resulting model's utility, as showcased in Table 2. For example, by following the privacy budget distribution of Alaggan et al. [2] with privacy budgets ε p = { 1 , 2 , 3 } for strong, medium, and weak privacy requirements, our Sample method yields accuracy improvements of 1.06%, 2.63%, and 5.09% on MNIST, SVHN, and CIFAR10, respectively. On the CIFAR10 dataset, our Scale even outperforms improvement with an accuracy increase of 5.26% in comparison to the non-individualized baseline, which would have to assign ε = 1 to the whole training set in order to respect each individual's privacy preferences.

The benefits of our individualized privacy assignment also become clear in Figure 1, which depicts the test accuracy of our Sample and Scale vs. standard DP-SGD over the course of training. Both our methods continuously outperform the non-individualized DP-SGD baseline with ε = 1 . Additionally, the test accuracy with privacy budget distribution according to Alaggan et al. [2] ( 34% , 43% , 23% ) is higher than the one of Niu et al. [30] ( 54% , 37% , 9% ). This can be explained by the fact that in this latter distribution, more individuals exhibit a higher pri-

vacy preference. With more data points choosing lower privacy protection as in Alaggan et al. [2], our individualization boosts the performance of the final trained model more significantly.

For comparison, we also report results for the standard DP-SGD with ε = 3 . This corresponds to the upper bound on utility that could have been achieved by assigning the highest privacy budget to all data points-which violates the privacy requirements of individuals with strong and medium privacy preferences. We observe that our individualized methods' performance is close to this upper bound without violating data points' privacy requirements. In Appendix E, we also discuss other possible baselines (apart from standard DP-SGD) and empirically evaluate our indi-

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Iteration (1 epoch

117 iterations)

96.5

97.0

97.5

98.0

Test Accuracy (%)

(I)DP-SGD Method

Standard

Sample

Scale

Budget Distribution

(

0

0

100)%

(

34

43

23)%

(

54

37

9)%

(100

0

0)%

vidualized methods against them. We report our hyperparameters and the respective individualization parameters (noise multipliers, clip norms, and sampling rates) obtained through our methods in Appendix D. Figure 4 in Appendix D.1 highlights additionally that this privacy parameter derivation is well calibrated: all different groups exhaust their privacy budget simultaneously when the targeted number of iterations is reached. In Appendix D.2, we also experimentally show that our methods are flexible and can extend to substantially more privacy groups which all have different budgets. 6 Finally, in Appendix D.6, we demonstrate how to combine our Sample and Scale method, and experimentally evaluate the joint method.

4.2 Broad Applicability of IDP-SGD

We also evaluate IDP-SGD with other (larger) architectures, for fine-tuning, and with different modalities and tasks. Therefore, we 1) fine-tune a BERT transformer [21] (bert-base-uncased) on the SNLI dataset [7] for natural language inference, 2) train a ResNet18 [16] from scratch on the CIFAR10 dataset for image classification, 3) train a linear embedding model (one embedding and two linear layers) on IMDB data [25] for text classification, and 4) train a character-level RNN (DPLSTM 7 , three layers, hidden size: 128) from scratch to classify surnames to languages. In Table 3, we evaluate our Sample and Scale in these setups with a privacy distribution corresponding to the first setup of Table 2, with ε = 5 , 7 . 5 , 10 for the first three rows, and ε = 1 , 2 , 3 for the last one, and a DP-SGD baseline with ε = 5 or ε = 1 , respectively. We tuned hyperparameters with standard DP-SGD and applied the best resulting hyperparameters to Sample and Scale . All results are averaged over 9 trials and we report average accuracy and standard deviation. Over all tasks, architectures, and modalities, IDP-SGD outperforms standard DP-SGD.

4.3 Privacy Assessment via Membership Inference

To evaluate the impact of our IDP-SGD on the privacy of individual data points, we perform membership inference based on the LiRA attack [8]. For the reader's convenience, we include a

description of the attack in Appendix B.5. We experiment with CIFAR10 and train 512 shadow models and the target model using Sample on different subsets of 25,000 training data points each. Results for Scale can be found in Appendix D.3. The privacy budgets are set to ε = 10 and ε = 20 and evenly assigned to the shadow models' training data, resulting in 12,500 training data points per privacy budget. Our target model achieves a train accuracy of 68.46% on its 25,000 member data points, and a test accuracy of 64.89% on its 25,000 non-member data points. The lower accuracy in comparison to Carlini et al. [8], who achieved 100% and 92% train and test accuracy respectively, results from us introducing DP to the training of the shadow models. Learning with DP is known to reduce model performance, particularly so on small datasets.

We depict the membership inference risk of the target model's training data per privacy budget (privacy group) in Figure 2. The dotted blue line represents the ROC curve for LiRA over all CIFAR10 data points and shows that overall, as expected by its definition, DP protects the training data well against membership inference attacks.

However, when inspecting the ROC curve separately for the two privacy groups ( ε = 10 and ε = 20 ), we observe a significant difference between them. The privacy group with stronger privacy guarantees ( ε = 10 ) is protected better (AUC = 0 . 537 ) than the group with the higher privacy budget ( ε = 20 , AUC = 0 . 581 ). To evaluate whether the difference in the LiRA-likelihood scores between the two privacy groups is statistically significant, we perform a Student t-test on the likelihood score distributions over the data points with privacy budget ε = 10 vs. the data points with privacy budget ε = 20 . We report ∆ = 2 . 54 with p = 0 . 01 < 0 . 05 , and hence a statistically significant difference between the two groups exists.

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

=[10,20],

AUC

=0.560

=10,

AUC

=0.537

=20,

AUC

=0.581

Experimental results for additional target models, further comparison between IDP-SGD and DPSGD, and detailed per-target model statistics are presented in Figure 9, Figure 10 and Table 12 in Appendix D.3. The results highlight that the individual privacy assignment of our IDP-SGD has a practical impact and indeed protects data points with different levels of privacy to different degrees. Additionally, the privacy risk for data points with ε = 10 when training with IDP-SGD is lower than when training purely on them with standard DP-SGD. This privacy gain does not come at large expense of points with ε = 20 whose privacy risk remains roughly the same(see Figure 10).

5 Comparison to Other Methods

Comparison to Individualized PATE. We compare against Individualized PATE [6] (IPATE), the only other work aiming at enabling individual privacy assignment in ML. See details of their method and the general PATE framework in Appendix B.3. For the comparison, we report the student model accuracies of IPATE for the MNIST, SVHN, and CIFAR10 datasets. The results are presented in Table 13 in Appendix D.4. We observe that IDP-SGD constantly outperforms IPATE with the same privacy budget assignment.

Synergy between Individualized Privacy Accounting and Assignment. We show that we can leverage synergies between our individualized privacy assignment and the orthogonal line of work on privacy accounting (see Section 2.2). Therefore, we extend the approach by Feldman and Zrnic [11] with our method of assigning each data point its individual privacy budget through an individualized

Renyi-filter. This filter causes the point to be excluded from training once its individually assigned privacy budget is exhausted. Note that in contrast, in the approach by Feldman and Zrnic [11], all data points obtain the same privacy budget and data points are excluded from training once they reach this budget. For more details, see the description in Appendix D.5.

To assess the performance of pure individual accounting, we assign the same privacy budget of 0 . 3 to all 60,000 training data points in MNIST, following the experimental setup by Feldman and Zrnic [11]. We then enable the individual privacy assignment and change the privacy values to ε = 0 . 3 for the first half of the points and ε = 0 . 31 for the second half of data points. We empirically observe that in this lowε regime, the small change of ε for half the data points is enough to cause significant improvement in the accuracy of the final model, as we show in Table 4. Note that the differences in accuracy reported for standard DP-SGD on MNIST between Table 4 and Table 8 differ. This is because of the different privacy budgets ( ε = 0 . 3 vs. ε = 1 . 0 ). Additionally, in Figure 3, we present the number of active data points, i.e ., data points that did not yet exhaust their individual privacy budget, over training. We find that many data points are used longer during training with individualized assignment compared to using only individualized accounting. This is because the data points with a higher privacy budget exhaust their budget later and, thus, can be used longer. We are only able to run this experiment on MNIST as done in Feldman and Zrnic [11] since the method is based on full-batch gradient descent: gradients are computed over the whole dataset at once, which limits its applicability to large and high-dimensional datasets.

6 Conclusion and Future Work

Prior work on ML with DP guarantees assigns a uniform privacy budget ε over the entire dataset. This approach fails to capture that different individuals have different expectations towards privacy and also decreases the model's ability to learn from the data. To overcome the limitations of a uniform privacy budget and to implement individual users' privacy preferences, we propose two modifications to the popular DP-SGD algorithm. Our Sample and Scale mechanisms adapt training to yield individual per-data point privacy guarantees and boost utility of the trained models. For future work, we believe that our individualized privacy assignment should be closely integrated with a form of practical individual privacy accounting. This could enable us to obtain a fine-grained notion of individualized privacy guarantees that tightly meet the users' expectations.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Mohammad Alaggan, Sébastien Gambs, and Anne-Marie Kermarrec. Heterogeneous differential privacy. arXiv preprint arXiv:1504.06998 , 2015.

Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and renyi differential privacy. In International Conference on Artificial Intelligence and Statistics , pages 2496-2506. PMLR, 2020.

Amos Beimel, Shiva Prasad Kasiviswanathan, and Kobbi Nissim. Bounds on the sample complexity for private learning and private data release. In Theory of Cryptography Conference , pages 437-454. Springer, 2010.

Bettina Berendt, Oliver Günther, and Sarah Spiekermann. Privacy in e-commerce: Stated preferences vs. actual behavior. Communications of the ACM , 48(4):101-106, 2005.

Franziska Boenisch, Christopher Mühl, Roy Rinberg, Jannis Ihrig, and Adam Dziedzic. Individualized PATE: Differentially Private Machine Learning with Individual Privacy Guarantees, 2022. URL https://arxiv.org/abs/2202.10517 .

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 , 2015.

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1519-1519. IEEE Computer Society, 2022.

Rachel Cummings, Gabriel Kaptchuk, and Elissa M Redmiles. " i need a better description": An investigation into user expectations for differential privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , pages 3037-3052, 2021.

Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation , pages 1-19. Springer, 2008.

Vitaly Feldman and Tijana Zrnic. Individual privacy accounting via a renyi filter. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

Daniel Franzen, Saskia Nuñez von Voigt, Peter Sörries, Florian Tschorsch, and Claudia MüllerBirn. " am i private and if so, how many?"-communicating privacy guarantees of differential privacy with risk communication formats. arXiv preprint arXiv:2208.10820 , 2022.

Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , pages 1322-1333, 2015.

Anna C Gilbert and Audra McMillan. Property testing for differential privacy. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 249-258. IEEE, 2018.

Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv preprint arXiv:2206.07758 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Carlos Jensen, Colin Potts, and Christian Jensen. Privacy practices of internet users: Self-reports versus observed behavior. International Journal of Human-Computer Studies , 63(1-2):203-227, 2005.

James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Differentially private bagging: Improved utility and cheaper privacy than subsample-and-aggregate. In Advances in Neural Information Processing Systems , pages pp. 4323-4332, 2019.

Zach Jorgensen, Ting Yu, and Graham Cormode. Conservative or liberal? personalized differential privacy. In 2015 IEEE 31St international conference on data engineering , pages 1023-1034. IEEE, 2015.

Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning , pages 5213-5225. PMLR, 2021.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, Corinna Cortes, and C. J. Burges. MNIST handwritten digit database . 2010. URL https://scholar.google.com/citations?user=wln3qraaaaaj&hl=en&oi=sra .

Haoran Li, Li Xiong, Zhanglong Ji, and Xiaoqian Jiang. Partitioning-based mechanisms under personalized differential privacy. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages 615-627. Springer, 2017.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015 .

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) , pages 263-275. IEEE, 2017.

Ilya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv:1908.10530 , 2019.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Ben Niu, Yahong Chen, Boyang Wang, Jin Cao, and Fenghua Li. Utility-aware exponential mechanism for personalized differential privacy. In 2020 IEEE Wireless Communications and Networking Conference (WCNC) , pages 1-6. IEEE, 2020.

Ben Niu, Yahong Chen, Boyang Wang, Zhibo Wang, Fenghua Li, and Jin Cao. Adapdp: Adaptive personalized differential privacy. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications , pages 1-10. IEEE, 2021.

Opacus. Opacus get_noise_multiplier-function. https://github.com/pytorch/opacus/ blob/211acbab1734c865dcfee9edd106c954c58cb10c/opacus/accountants/utils. py#L23 .

Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semisupervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755 , 2016.

Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations , 2018.

Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. { UpdatesLeak } : Data set inference and reconstruction attacks in online learning. In 29th USENIX security symposium (USENIX Security 20) , pages 1291-1308, 2020.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Peter Sörries, Claudia Müller-Birn, Katrin Glinka, Franziska Boenisch, Marian Margraf, Sabine Sayegh-Jodehl, and Matthias Rose. Privacy needs reflection: Conceptional design rationales for privacy-preserving explanation user interfaces. Mensch und Computer 2021-Workshopband , 2021.

Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations , 2020.

Florian Tramer, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing. arXiv preprint arXiv:2202.12219 , 2022.

Stacey Truex, Ling Liu, Ka-Ho Chow, Mehmet Emre Gursoy, and Wenqi Wei. Ldp-fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking , pages 61-66, 2020.

Aiping Xiong, Tianhao Wang, Ninghui Li, and Somesh Jha. Towards effective differential privacy communication for users' data sharing decision and comprehension. In 2020 IEEE Symposium on Security and Privacy (SP) , pages 392-410. IEEE, 2020.

Jiayuan Ye and Reza Shokri. Differentially private learning needs hidden state (or much faster convergence). In Advances in Neural Information Processing Systems , 2022.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pages 268-282. IEEE, 2018.

Da Yu, Gautam Kamath, Janardhan Kulkarni, Tie-Yan Liu, Jian Yin, and Huishuai Zhang. Individual privacy accounting for differentially private stochastic gradient descent. arXiv preprint arXiv:2206.02617 , 2022.

Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 253-261, 2020.

A Broader Impacts and Ethical Considerations of this Work

The specification of an adequate privacy level ε is challenging since it does not only depend on the domain but also on the dataset itself. This challenge persists when assigning individual privacy levels over a dataset. Especially in sensitive domains, one needs to make sure that the assignment of individual privacy guarantees does not pose an additional risk to the individuals whose data is processed. Therefore, one needs to first make sure that assigning individual privacy guarantees is not abused. This can occur when the entity training the model violates underlying individuals' privacy by nudging them into choosing inadequate privacy levels. Second, one needs to prevent individuals from giving up their privacy due to a lack of understanding or a wrong perception of their risk. To do so, IDP-SGD should not be used as a stand-alone method. Instead, following the example of [6], we suggest incorporating it into a whole process that involves communicating the functioning of DP and the risks associated with a decision to the individuals, analyzing their preferences, and supporting them in an informed decision-making process. There exist a growing body of work on the communication of DP methodology and associated risks to the public [41, 9, 12]. The most recent finding [12] suggests that medical risk communication formats, such as percentages or frequencies, are best applicable to inform about privacy risks. For the identification of privacy preferences, existing frameworks, such as [37] can be applied. Finally, we suggest giving individuals the choice between categorical privacy levels ( e.g ., high , medium , low ) while putting a regulatory entity, such as an ethics committee in charge of specifying concrete numeric values of ε . This mitigates the disadvantages of the general limited interpretability of ε that exist independently of individualization.

A.1 Limitations

In this work, we consider a setup where individuals indicate their privacy preferences. We acknowledge that not all individuals are aware of their own privacy preferences and might, therefore, provide inaccurate indications. Therefore, our framework should be always deployed in a protected setup as described above. While we provide theoretical privacy guarantees in the framework of differential privacy, we assess the practical impact of these guarantees on the individuals solely through membership inference attacks. Future work should investigate the practical (disparate per privacy-group) impact on other known privacy risks, such as data reconstruction. Due to the lack of sensitive-real world datasets that, in addition to individuals' data, also capture their privacy preferences, we evaluate our methods on standard benchmark datasets. We, therefore, have to simulate a privacy preference distribution on this data according to the distribution known from society [17, 5].

A.2 Confidentiality of the Privacy Budgets

In some setups, an individual's choice of privacy budget could reflect some sensitive information. Imagine a medical context with a dataset that consists of individuals that do and individuals that do not have cancer. The latter ones might be more inclined towards choosing higher privacy protection to hide their condition. If an attacker with access to the trained model was able to deduce the individual's privacy budget, they might, in turn, be able to draw conclusions on the individual's medical state.

Yet, theoretical results from ML, e.g ., [14], have shown that one cannot currently perform sampleefficient black-box audits to determine the privacy-budget of a trained model. Frameworks to estimate privacy of a trained model, such as [39] rely on training 1000-100,000 shadow models to get an estimate of the full model's privacy guarantee. In our IDP-SGD setup, the model does not have one single ε , but one ε p per group, aggravating the impracticability of auditing guarantees and limiting the applicability of existing frameworks. As a result, the adversary cannot currently determine the different per-group privacy budgets of the model.

Our MIA experiments in Section 4.3 show a difference in privacy risks over entire privacy groups. First note that these experiments (Figure 2) were the most costly ones in our paper. We had to train more than 1500 (3 times 512) shadow models to generate the results. Yet, there are two limitations: a) The results only show differences in the groups, but do not reveal the respective ε p values, b) they do not allow to perform per-point distinctions. This is because the distributions of privacy risks over the two groups still have a large overlap. Hence, given a single point's risks and the two distributions, one cannot predict without error to what group the point belongs. This error could be reduced when the privacy budgets between the groups differ significantly (0.1 vs. 100). Note however, that we suggest deploying the model in a setup where individuals simply specify their privacy preferences

and the model builder or an ethics committee assign concrete epsilon values to these preferences. We observed that extreme differences between privacy groups' are less beneficial because they degrade overall model performance. Hence, in practice, these large differences are very unlikely. As a result, it will be very challenging for an adversary to determine which privacy group a data point belongs to. Even if they manage to, they are unlikely to be able to learn the privacy budget of the group as described above.

To enhance confidentiality of privacy budgets, Alaggan et al. [2] implement a form of deniability for users' membership in a certain privacy group by uniformly sampling the privacy values over groups in an overlapping way: the unconcerned sample from 1., the pragmatists from 0.5, 0.75, 1., and the fundamentalists from 0, 0.5, 1, such that on average, the values will be 1., 0.75, and 0.5 within the groups (0 denotes the strongest, while 1 is the weakest privacy guarantee). Such an approach can be implemented into our IDP-SGD framework as well. The drawback is that an individual who expresses strong privacy preferences (fundamentalists) might actually be assigned a privacy value 1 by randomness, which causes the same privacy leakage as for an unconcerned individual. Depending on the individuals' preferences and the application, when protecting privacy group membership is more important than protecting the actual data, this approach might still be valid and this approach could be integrated with IDP-SGD, as well.

B Extended Background

B.1 Differential Privacy

Differential Privacy (DP) [10] provides a theoretical upper bound on the influence that a single data point can have on the outcome of an analysis over the whole dataset.

The most commonly applied instantiation of DP is ( ε, δ ) -DP which is defined as follows:

Definition B.1. Let D,D ′ ⊆ D be two neighboring datasets, i.e ., datasets that differ in exactly one data point. Let further M : D ∗ → R be a mechanism that processes an arbitrary number of data points. M satisfies ( ε, δ ) -DP if for all datasets D ∼ D ′ , and for all result events R ⊆ R

In the definition, the privacy level is specified by ε ∈ R + , while δ ∈ [0 , 1] offers a relaxation, i.e ., a small probability of violating the guarantees.

B.2 Differential Privacy Accounting in Machine Learning

Privacy accounting in DP-SGD is most commonly implemented by the moments accountant which keeps track of a bound on the moments of the privacy loss random variable at outcome R , defined by

for D,D ′ , M and R as above, and an auxiliary input aux .

B.3 PATE Algorithm

The Private Aggregation of Teacher Ensembles (PATE) algorithm [33] represents an alternative to DPSGD for training ML models with privacy guarantees. This ensemble-based algorithm implements privacy guarantees through a knowledge transfer from the ensemble to a separate student model. More concretely, in PATE, the private training data is split into non-overlapping subsets and distributed among several teacher models. Once each teacher is trained on their own data subset, they perform a privacy-preserving knowledge transfer by jointly labeling an additional unlabeled public dataset. To implement DP guarantees, noise is added to the labeling process. On completion, an independent student model is trained on the public dataset using the generated labels, and thereby incorporating knowledge about the original training data without ever requiring access to it.

B.4 Individualized PATE.

The individualized PATE variants by [6] are Upsample and Weight . Upsample duplicates data points and distributes them to different teachers in the PATE ensemble. Utility increase in this method

result from the availability of more training data points for the teachers. Since the sensitivity for a duplicated data point increases (it can change the votes of all the teachers that are trained on its duplicates), privacy consumption of that data point is higher. Since each data point can be duplicated individually, this method allows for a fine-grained individualization. In contrast, the weight method allows for a per-teacher model privacy individualization. Data points with the same privacy budget are assigned to the same teacher model(s) and the impact of that teacher model's weight on the final vote is weighted according to its training data points' privacy preferences. Teachers that are trained on data points with low privacy requirements are weighted higher, increasing the privacy consumption of their training data.

B.5 The LiRA Membership Inference Attack

The LiRA membership inference attack [8] proceeds in three steps to determine which data points from a dataset D = { x 1 , . . . , x N } were used to train a target model f : (1) First, multiple shadow models, similar to f , are trained on different subsets of D . (2) Then, the mean and variance of two loss distributions N ( µ in , σ in ) and N ( µ out , σ out ) are estimated per data point x i . Both distributions are calculated from the logits of x i at the target class y i -the former one over the shadow models that x i is a member of, the latter one on shadow models that x i is not a member of. (3) Finally, the likelihood of a new data point x of class y being a member of the target model f is calculated as

C Details on the Methods

C.1 Sample

Leveraging Higher Sampling Rates for Increased Utility. With higher sampling rates for certain data points, utility could, in principle be increased in several ways. (1) Larger sampling rates can be used to obtain higher mini-batch sizes B (while keeping the number of training iterations I constant). Line 3 in Algorithm 1 shows that noise is added to the aggregate of all gradients. Hence, with larger mini-batches, the signal-to-noise ratio is higher, which can improve training. (2) Alternatively, the mini-batch size B can be kept constant while increasing the number of training iterations I . Longer training can increase model performance. However, these two approaches result in a change of core training hyperparamters (mini-batch size and number of iterations). As we discuss in Section 3, changing training hyperparameters would require a separate fine-tuning, for example, to adapt the learning rate for larger mini-batches, as in (1) or longer training as in (2). Since the training parameters would change according to the privacy budgets encountered in the private training dataset, and the ratios of these budgets over the training data points, the hyperparameter-tuning would have to be repeated whenever the dataset is updated, individuals change their privacy preferences, or decide to withdraw their consent for leveraging their data for the ML model alltogether, yielding significant overheads. We, therefore, implement our Sample according to the third option (3), described in Section 3.3 which leverages higher sampling rates for improved utility by reducing the noise multiplier of the added noise σ . This allows us to perform an apple to apple comparison between both our methods and to the standard DP-SGD.

C.2 Scale

Deriving Noise Multiplier σ scale σ scale σ scale . Given the desired clip norm c found through hyperparameter tuning of the standard DP-SGD, we set the individual clip norms such that their weighted average yields c as c = 1 N ∑ P p =1 | G p | · c p . So, we derive the σ scale in the following way:

From (6) to (7), we use the equality between the scale of added noise σ scale c = σ p c p . In (8), we extract terms that are independent of the privacy groups ( c and σ scale) before the summation.

C.3 Algorithmic Details

We specify our used sub-routines used for determining a sample rate or noise multiplier based on given privacy parameters in Algorithm 4 and Algorithm 5, respectively.

D Additional Empirical Evaluation

We report the hyperparameters found for our individualized methods in Table 6. The training and standard DP-SGD hyperparameters are specified in Table 5. The performance of our individualized methods when using the hyperparameters of standard DP-SGD is presented in Table 8. Already when using these (non-individually tuned) hyperparameters, our methods yield a significant performance increase in comparison to standard DP-SGD. For MNIST, individual hyperparameter for our methods and individual setups did not yield significant improvements, therefore the results presented in Table 8 and Table 2 are identical for MNIST.

Computing Resources. The implementation of our methods does not increase computation time over the standard implementation of DP-SGD apart from the derivation of the privacy parameters that is performed once at the beginning of training. Hence, to run all experiments around our methods and

their evaluation, we required, in total less than 16h of GPU time on a standard GeForce RTX 2080 Ti. We ran the experiment on combining individualized privacy assignment and accounting on the same machines RTX 2080Ti and the total compute time is also around 2h. To train all the shadow models for our membership inference attack and run inference on them, we ran on an A100 GPU and required a total runtime of roughly 32 hours.

We present the individualized privacy parameters identified for our methods in Table 7.

D.1 Privacy Consumption of our Methods

We track privacy consumption of our methods over the course of training in Figure 4. The figure highlights the good calibration of our methods which causes all privacy groups to exhaust their budget after the pre-specified number of training iterations.

0

2000

4000

6000

8000

Iteration (1 epoch

117 iterations)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Differential Privacy (

)

higher group

medium group

lower group

baseline

(a)

Sample

0

2000

4000

6000

8000

Iteration (1 epoch

117 iterations)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Differential Privacy (

)

higher group

medium group

lower group

baseline

(b)

Scale

D.2 General Applicability of our Methods

We showcase the practical impact of individual privacy assignments on individual utility and demonstrate how our methods extend to many privacy groups and privacy budget distributions.

Practical Impact. We run experiments on the CIFAR10 dataset where we assign higher or lower privacy budgets to one of the 10 classes. We select all data points from the class 0 (which we can think of, for example, as an underprivileged group) as one privacy group and assign to it either higher ( ε = 3 ), the same ( ε = 2 ), or lower ( ε = 1 ) privacy budgets in comparison to all other data points from the other classes ( ε = 2 ). Table 10 shows that the choice of privacy budget for a single group also impacts the other groups. We observe that by only changing the privacy budget for the selected group (in this case for class 0), we can flip its performance (its accuracy from being higher to being lower) in comparison to the accuracy of the other group (consisting of remaining classes). In the example of class 0, the accuracy is 67.76% when assigned high privacy budget ( ε = 3 ), which is a higher accuracy than for all other classes that have an average accuracy of around 53.41% and assigned the privacy budget ε = 2 . Then, by modifying only the privacy budget of class 0 and by assigning to it the low privacy budget ( ε = 1 ), its accuracy drops to a mere 25.76% and is below the accuracy of 56.58% for the remaining classes. We visualize the impact of the chosen privacy budget on utility over all classes (instead of only the class 0) in Figure 5 and Figure 6 for CIFAR10 and MNIST, respectively.

More Privacy Groups. We present in Table 11 the test accuracy for ten privacy groups, corresponding to the ten classes of the CIFAR10 dataset when each of the privacy groups obtains a different privacy budget. We obtained these budgets by manually tuning them such that the accuracy gap between the privacy groups is minimized. We also visualize the accuracy over training in Figure 7. The figure visualizes that our methods are able to make all privacy groups converge to similar accuracies.

To evaluate IDP-SGD in the limit, we perform additional experiments with 100 privacy groups by randomly assigning MNIST training data points to one of the 100 privacy groups with respective ε p sampled uniformly at random from [1 , 6] . Our results in Table 9 indicate that IDP-SGD is able to handle this large number of privacy groups without any degradation in terms of convergence: both our methods, Sample and Scale , continuously outperform the standard DP-SGD baseline with ε = 1 for all data points.

D.3 Additional Results for MIA

Membership inference success for a single target model of our Sample and Scale methods is shown in Figure 8. In Figure 9, we present the results over for 5 different target models for Sample . The figure highlights that over all target models, the two privacy groups' privacy risk is different: the group with higher protection ε = 10 constantly has a lower AUC than the group with lower protection ε = 20 . The test statistics over the different privacy groups' LiRA likelihood scores for all five target models are shown in Table 12.

In Figure 10, we also compare the MIA risk of our IDP-SGD with the risk of training with DP-SGD, using the LiRA attack. The results show that the privacy risk for data points with ε = 10 when training with IDP-SGD is lower than when training with standard DP-SGD. This privacy gain does not come at large expense of points with ε = 20 whose privacy risk remains roughly the same.

D.4 Comparison to Individualized Privacy with IPATE

We present the comparison between our IDP-SGD and IPATE [6] in Table 13. In PATE, accuracy refers to the student model accuracy. The results in Table 13 are averaged over three experiments for IPATE and ten runs for IDP-SGD. Note that the accuracies we report for IPATE differ from the accuracy values reported by [6] in Table 1, since they report average voting accuracy ( i.e ., how correct are individual teacher model votes), whereas we report the resulting student model accuracies, which

corresponds to the final performance of the method. Note that IPATE does not apply the performance improvements suggested by Papernot et al. [34] ( e.g .,virtual adversarial training) or MixMatch which could increase the student model's performance.

D.5 Integrating Privacy Accounting and Assignment

The main goal behind individualized accounting is to obtain tighter privacy analysis (recall our discussion in Section 2.2). Instead of tracking a single privacy loss estimate across all data points, an individual privacy loss is kept track of for each data point. Feldman and Zrnic [11] defines a new individual privacy filter, which drops data points that exceed the individualized privacy loss from further processing. However, the same privacy budget is assigned to each data point. The individual assignment of a privacy budget to each data point is a natural extension of individualized accounting and can be directly incorporated into this framework. Therefore, a given data point has its own privacy filter and is dropped from the analysis once the filter indicates that the data point's privacy budget is exhausted.

D.6 Combining Sample and Scale

We run additional experiments to showcase the advantage of combining both our methods. The combination of Sample and Scale extends the number of hyperparameters and offers the potential to find better performing algorithms. It can be implemented as follows: obtain individual sampling probabilities as in Sample . Once a mini-batch is sampled, use the respective data points' indices and

0

1

2

3

4

5

6

7

8

9

Class with Higher Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

30

20

10

0

10

20

30

Baseline Accuracy] (%)

(a)

Sample

(higher budget)

0

1

2

3

4

5

6

7

8

9

Class with Higher Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

30

20

10

0

10

20

30

Baseline Accuracy] (%)

(b)

Scale

(higher budget)

0

1

2

3

4

5

6

7

8

9

Class with Lower Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

30

20

10

0

10

20

30

Baseline Accuracy] (%)

(c)

Sample

(lower budget)

0

1

2

3

4

5

6

7

8

9

Class with Lower Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

30

20

10

0

10

20

30

Baseline Accuracy] (%)

(d)

Scale

(lower budget)

clip their gradients as in Scale . We allow to weight the different methods to arbitrary fractions (e.g., 50% Sample , 50% Scale ), and derive the privacy parameters such that all privacy groups exhaust their respective budgets at the end of the specified number of iterations. For practical evaluation, we use the MNIST dataset and the first privacy setup (0.34, 0.43, 0.23) with privacy budgets ε = 1 , 2 , 3 and different weightings as indicated in Table 14.

The baseline (first and last rows) are equivalent to Table 2 in the main paper, while the additional three rows in between with the new results represent average test accuracy and the standard deviation over ten independent random runs over different weights assigned to the respective method.

Our results demonstrate a clear progression from the lower performing Scale approach to the better performing Sample . This aligns with expectations based on our implementation, where we combine the methods based on the parameters found for them sequentially.

Note that each of our methods has its advantages and disadvantages and the combination of both methods provides more options to balance (dis)advantages. For example, when some points' sample rates are extremely small, then the model might never see those points during training and the combined method could reduce this possibility while still improving over the Scale-only method.

0

1

2

3

4

5

6

7

8

9

Class with Higher Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Baseline Accuracy] (%)

(a) Sample | higher budget

0

1

2

3

4

5

6

7

8

9

Class with Higher Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Baseline Accuracy] (%)

(b) Scale | higher budget

0

1

2

3

4

5

6

7

8

9

Class with Lower Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Baseline Accuracy] (%)

(c) Sample | lower budget

0

1

2

3

4

5

6

7

8

9

Class with Lower Budget

0

1

2

3

4

5

6

7

8

9

Class of Interest

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Baseline Accuracy] (%)

(d) Scale | lower budget

D.7 Examining Convergence of IDP-SGD

Individualization directly affects the objective function of the training procedure. To verify that our individualized methods maintain a smooth convergence, we examine the loss history of one group of training data in the MNIST dataset and compare it with non-individualized DP-SGD.

We collected the loss values (and accuracy values) over the 4 following setups on the MNIST dataset and privacy setup 2 ( 54% , 37% , 9% , ε = { 1 , 2 , 3 } ):

loss value for privacy group with ε = 1 for the Sample method;

loss value for privacy group with ε = 1 for the Scale method;

training with standard DP-SGD ( ε = 1 ) solely on the 54% of data points that have the privacy budget of ε = 1 , and

training with the standard DP-SGD ( ε = 1 ) on 100% of the MNIST dataset.

In the following table, we depict the training loss of the privacy group with ε = 1 during training:

The loss values correspond to the following accuracy values:

Our results indicate that the availability of more information from the data of different privacy groups in our Sample and Scale methods enables the model to learn better features. This leads to lower loss for the data points with privacy budget ε = 1 than if the model was trained on only those 54% of points. The effect of including more data (i.e., using 100% of the MNIST dataset), all with ε = 1 , is weaker, indicating that even the data with highest privacy requirement benefits substantially from our individualized privacy in terms of reduced loss and increased accuracy.

D.8 Runtime Analyses

Weanalyze the runtime of our methods in comparison to standard DP-SGD. The runtime is determined by two steps, the first one being the calculation of the individualized parameters of the methods which is executed exactly once before the training. The second one is the actual training time. For the calculation of individualized parameters, Table 16 indicates that the runtime depends on the number of privacy groups P specified for the implementation of Algorithm 2 and Algorithm 3 and the precision γ (see Algorithm 5). We observe a linear growth in runtime w.r.t. the number of privacy groups. Note that, as stated above, this additional runtime occurs only once with IDP-SGD, namely before training starts. Also note that when we repeat experiments with the same privacy setups, the parameters do not need to be derived every time, but can be set directly based on one initial computation. In terms of actual training time, Table 16 indicates that Sample and Scale do not

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Iteration (1 epoch

117 iterations)

96

97

98

99

100

Per-Class T est Accuracy (%)

0

1

2

3

4

5

6

7

8

9

average

(a) Baseline

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Iteration (1 epoch

117 iterations)

96

97

98

99

100

Per-Class T est Accuracy (%)

0

1

2

3

4

5

6

7

8

9

average

(b)

Sample

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Iteration (1 epoch

117 iterations)

96

97

98

99

100

Per-Class T est Accuracy (%)

0

1

2

3

4

5

6

7

8

9

average

(c)

Scale

significantly increase the training time in comparison to standard DP-SGD which is mainly due to their tight integration with the standard DP-SGD algorithm and the opacus library.

E Alternative Baselines

Throughout this work, we compare our methods with Standard DP-SGD which cannot take into account different privacy requirements at the same time. Thus, we consider it to apply the highest privacy protection required to all data points equally. Nonetheless, we can think of two more ways to

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

=[10,20],

AUC

=0.560

=10,

AUC

=0.537

=20,

AUC

=0.581

(a)

Sample

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

=[10,20],

AUC

=0.571

=10,

AUC

=0.552

=20,

AUC

=0.589

(b)

Scale

use Standard DP-SGD on data having heterogeneous privacy requirements. Those approaches have different benefits and drawbacks and might perform better than our chosen baseline approach in some scenarios.

E.1 Exclude Lower Privacy Groups

Instead of applying the strongest privacy protection, the deciding ML expert could entirely exclude data of low privacy groups from training for loosening the restrictions on the remaining data points' influence on model updates. In some cases, it would be worth giving up the information and privacy budgets of those lower privacy groups to achieve utility improvements. This approach performs poorly if important information is wasted, e.g ., most data of one class has the highest privacy requirement.

E.2 Learn Privacy Groups Separately

It is also possible to make use of all privacy budgets, independent of their diversity, although only using Standard DP-SGD. Namely, a model can be trained on each privacy group separately one after another, whereby the corresponding lowest budget is regarded for each group. A drawback of this approach is that the model could forget its knowledge about previously learned privacy groups.

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

=[10,20],

AUC

=0.571

=10,

AUC

=0.546

=20,

AUC

=0.593

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

=[10,20],

AUC

=0.560

=10,

AUC

=0.537

=20,

AUC

=0.581

=10(

DPSGD

),

AUC

=0.568

=20(

DPSGD

),

AUC

=0.590

(a) Sample

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

=[10,20],

AUC

=0.571

=10,

AUC

=0.552

=20,

AUC

=0.589

=10(

DPSGD

),

AUC

=0.568

=20(

DPSGD

),

AUC

=0.590

(b) Scale

E.3 Empirical Comparison of Baselines

We empirically evaluate against these two additional baselines using the MNIST dataset. For baseline E.1, we include all data points with a privacy budget of ε ≥ 2 und use ε = 2 as the privacy budget for training. After hyperparameter tuning, the training on the remaining data points (43%+23% and 37%+9% of the total data) yields the accuracy reported in Table 17.

For baseline E.2, we also did hyperparameter tuning and used the best noise multiplier of 2.5 for training. We trained the groups sequentially, always continuing training with the next group once the privacy budget of the previous groups was exhausted. We evaluated both starting with the privacy group that has loosest and strongest preferences (orders [3,2,1] and [1,2,3], respectively). Starting with the group that has strongest privacy requirements and ending on the group that has loosest privacy requirements yielded the best results which we report in Table 17.

In summary, we observe that our methods outperform the other baselines.

F Alternative Individualization

We present the alternative ways of individualizing privacy guarantees in DP-SGD that we considered in the design process of IDP-SGD and describe their drawbacks.

F.1 Individual Per-Data Point Noise

Individualized privacy could, in principle also be obtained by adding different amounts of noise to different data points. Every of the P privacy group would have their individual { σ 1 , . . . , σ P } . Utility improvements would result from some data points requiring smaller amounts of added noise. Note however, that in DP-SGD, while clipping is performed on a per-data point basis, noise addition is performed on a per-mini-batch basis (line 8 in Algorithm 1). Hence, there are two possibilities to implement individual noise addition: either (i) by operating on mini-batch sizes of 1, or (ii) by implementing a two-step sampling approach which first randomly samples a privacy group for a given training iteration and then applies the standard Poisson sampling to obtain the mini-batch consisting of data points from this group. While both approaches are conceptually correct, they exhibit significant drawbacks. Approach (i) first slows down training performance due to more operations requiring to be carried out on individual data points, rather than a mini-batch. Second, due to the weak signal-to-noise ratio when adding noise to individual gradients, model performance is likely to degrade. Finally, sampling cannot be performed with Poisson anymore since with Poisson sampling, it is not possible to pre-determine and specify exact mini-batch sizes, instead these depend on the outcome of the random sampling process. Approach (ii) could overcome the first two issues. However, the different groups sizes are still strictly smaller than the entire dataset and large parts of DP-SGD's degrading the tight privacy bounds obtained by privacy amplification through subsampling. 8 The privacy amplification through subsampling allows to scale down the noise σ by the factor B/N (with B being the expected mini-batch size, N the total number of data points, and B ≪ N ) while still ensuring the same ε as with σ [20]. This privacy amplification is crucial to the practical performance (privacy-utility trade-offs) of DP-SGD. Hence, by using the P privacy groups of sizes { N 1 , . . . , N P } with N i ≪ N , the factors B/N i ≪ B/N for all i { 1 , . . . , P } . This effect cancels out, or in the worst

case even inverts the privacy-utility benefits that should arise from assigning individual data points less noise based on their privacy preference in our individualization.

F.2 Duplicating Data Points

When duplicating data points in the training dataset, similar to the Upsampling mechanism in IPATE [6], the DP-SGD algorithm itself does not need to be adapted. Instead, different privacy levels of different data points stem from their individual number of replication within the training data. This approach offers a very fine-grained control on individual privacy levels, since, in principle, each data point could be replicated a different number of times. Utility gain would result from the larger training dataset. However, this type of upsampling opens the possibility for the same data point to be present multiple times in the mini-batch used for training in a given iteration. This stands in contrast to the original DP-SGD, where participation of each data point for training at a given iteration is determined by an independent Bernoulli trial, and hence, a data point can be either included once or not at all in a mini-batch. The possibility for a data point to be included multiple times n inside the same mini-batch changes the sensitivity of the mechanism from c to nc . According to [27], when noise is added according to σ , a mechanism with sensitivity nc is ( α, 2( nc ) 2 2 σ 2 ) -RDP. The quadratic influence of the sensitivity to privacy bound results in a severe increase in the RDP ε , making the approach suboptimal in terms of privacy-utility guarantees. Additionally, upsampling leads to an effective increase in a data point's the sample-rate which further increases privacy costs.

G Additional Proofs

G.1 Additional Proofs for Individualized Privacy

Proof for Lemma 3.1

Proof. First note that ( ε 1 , δ ) -DP can be considered as a special case of ( { ε 1 , ε 2 , . . . , ε P } , δ ) -IDP, where ∀ p ∈ [1 , P ] ε p = ε 1 . We can, hence apply Equation (3) and see that an M that satisfies ( ε 1 , δ ) -DP has a privacy guarantee of P [ M ( D ) ∈ R ] ≤ e ε 1 · P [ M ( D ′ ) ∈ R ]+ δ . Given that by our definition ∀ p ∈ [2 , P ] it holds that ε p > ε 1 , for all p , it holds that P [ M ( D ) ∈ R ] ≤ e ε 1 · P [ M ( D ′ ) ∈ R ] ≤ e ε p · P [ M ( D ′ ) ∈ R ] + δ . From this inequality, it follows that M also satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -IDP.

Proof for Lemma 3.2

Proof. Analogous to the previous proof, by our definition, it holds that ∀ p ∈ [1 , P -1] the ε p < ε P . From an M that satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -IDP, it, therefore, holds that for all p the P [ M ( D ) ∈ R ] ≤ e ε p · P [ M ( D ′ ) ∈ R ] + δ ≤ e ε P · P [ M ( D ′ ) ∈ R ] + δ . This inequality shows that M satisfies ( ε P , δ ) -DP.

G.2 Privacy Proofs for our Methods

Either of our methods can be considered as an individual SGM with different parameters from the point of view of each privacy group. This is because for each group, we have to examine neighboring datasets which differ in an arbitrary data point from that group. Our Sample method ensures an individual sample rate for all points of each group, while our Scale method applies an individual noise multiplier for all points of each group.

So, assume we have P different privacy groups. We base our proofs on the privacy guarantees of Sample and Scale on the observation that our methods can be considered as P simultaneously executed Subsampled Gaussian Mechanisms (SGM) [28] that update the same model. We first note that every data point's privacy preference stays stable over the course of training, therefore, conceptually, the privacy groups divide our training data in P disjoint subsets. Hence, each of the P SGMs operates on a different partition of the training data, using different individual sample rates or clip norms. One can think of the simultaneous model updates by multiple SGMs with different privacy parameters as a similar concept to federated learning [26] with different local privacy guarantees [40]: each client holds a disjoint subset of data, executes some local model update, implements privacy guarantees through local subsampling of the data, clipping, and noise addition (potentially with different parameters than other clients), and then shares the model updates with a server that aggregates all updates and applies them to the shared model.

Within each training step, our methods' SGMs also have a different privacy consumption due to their different sample rates and clip norms (sensitivity). Given that we keep the intermediate model states (checkpoints) private and only release the final model [42], the privacy costs in RDP over multiple training iterations add up linearly per SGM, such that, at the end each SGM yields ( α, ¯ ε p ) -RDP, p ∈ [1 , . . . , P ] . As a consequence, the algorithm over all privacy groups yields ( α, { ¯ ε 1 , ¯ ε 2 , . . . , ¯ ε P } ) -RDP which can then be converted to ( { ε 1 , ε 2 , . . . , ε P } , δ ) -DP using standard conversion [27] as we will show in the following.

Theorem G.1. Our Sample mechanism satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -DP.

Proof. We prove the bound for any particular privacy group separately. Fix p ∈ { 1 , . . . , P } , let D ⊆ D be the training dataset, and select any x i ∈ D that belongs to group G p . We are interested in comparing outcomes of mechanism M on D with its outcomes on D ′ = D \ { x i } where M represents a particular model update of Sample . We get Gaussian mixtures

where f ( L ) is the clipped gradient of the current mini-batch L , π L is its probability, σ sample > 0 is the noise scale, I d is the identity matrix, and 0 < q p ≤ 1 is the individual sample rate of x i and every other point in G p . Note that σ sample is actually multiplied by the clip norm c sample > 0 . Gaussian mechanisms are invariant regarding scaling of their sensitivity and noise scale, but instead depend on the relationship between sensitivity and noise scale, called the noise multiplier. Hence, we can ignore c sample and consider f to have sensitivity 1 .

(

Now we can see that the Gaussian mixtures of our Sample are equivalent to those corresponding to the original SGM from Mironov et al. [28], Thm. 4, when we parameterize it with sample rate q p and noise scale σ sample which are individual per group. Therefore, all RDP bounds of the original SGD apply, especially ( α, ¯ ε p ) -RDP with ¯ ε p = 2 q 2 p α σ 2 sample in a particular parameter regime (cfg. Thm. 11 from Mironov et al. [28]). As a final step of the proof, we need to convert from α, ¯ ε p ) -RDP guarantees to ( ε p , δ ) -DP guarantees following Mironov [27] (see Section 2.1). Note that our individual parameters have been selected before the start of training so that each group's privacy budget is exhausted at the intended number of iterations (see Algorithm 2).

Theorem G.2. Our Scale mechanism satisfies ( { ε 1 , ε 2 , . . . , ε P } , δ ) -DP.

Proof. This proof is analog to the proof of Theorem G.1 with the difference that we have a global sample rate q but individual noise multipliers σ p and clip norms c p . Moreover, Algorithm 3 is used to configure parameters prior to training.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs variants of differentially private SGD to satisfy different privacy expectations, e.g., users can choose one from high, medium, or low levels of privacy. There are two variants of DP-SGD, one changes the sampling probability of different groups, and the other changes the clipping threshold of different groups.\n\nSTRENGTHS:\n1. The Sample and Scale methods are easy-to-implement. Their design aligns nicely with the fact that DP-SGD adds noise to the aggregated gradient, instead of separate noise for individual gradients.\n\n2. Experiments on several common vision datasets show that individualized DP-SGD is better than DP-SGD with a uniform (the smallest) privacy parameter. The authors also run membership inference attacks to show that groups with smaller privacy parameters do have better empirical privacy.\n\n3. The paper is very well written and easy to follow.\n\nWEAKNESSES:\n1. The privacy expectations in the current version are independent of model accuracy. In practice, the minority groups, whose accuracy is usually worse, may prefer stronger privacy than other groups. What would happen if the group with the worst accuracy has the strongest/weakest privacy expectation? One way to implement this is to extend the experiments in Appendix D.2 by assigning a larger/smaller privacy parameter to the class with the worst non-private accuracy. \n\n2. The privacy expectation itself may be data dependent and hence the individualized privacy assignment cannot be made public, as the authors discussed in Line 68 – Line 73. For example, in a medical dataset, users diagnosed with certain diseases may have higher privacy expectations.  \n\n3. The authors use a shallow convolutional network for the experiments. It would be better to see whether the results still hold for deep learning models, e.g., ResNets or transformers.\n\n4. The authors only report the average accuracy. It would be better if the authors could show the accuracy of different groups. A recent line of work shows that DP has disparate impact on model accuracy [1]. Will the groups with smaller privacy assignments lose more accuracy?\n\n[1]: Differential Privacy Has Disparate Impact on Model Accuracy. https://arxiv.org/abs/1905.12101\n\nQUESTIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes extensions of the DP-SGD algorithm to support individualized differential privacy (called the IDP-SGD approach). Unlike traditional differential privacy, which imposes a single privacy budget epsilon to all data points, the data points may now have different privacy budgets. Two extensions are proposed. First, the Sample method adjust the sampling rates of data points depending on their privacy budgets. Second, the Scale method adjusts the individual clip norms to effectively add individualized noise. The noise itself cannot be directly individualized due to the DP-SGD algorithm behavior of adding per-batch noise. Experiments show that IDP-SGD outperforms DP-SGD and IPATE in terms of utility and can be complemented with privacy accounting techniques.\n\nSTRENGTHS:\n* Proposes novel extensions of DP-SGD to support individualized privacy.\n* The Sample and Scale methods are easy to deploy and have theoretical guarantees.\n* Experiments show that IDP-SGD outperforms baselines (DP-SGD and I-PATE) and can be used with privacy accounting.\n\nWEAKNESSES:\nW1) The core contributions of the paper are not contained in the main part of the paper. \n  * The theoretical analysis should be more specific because the proposed individual privacy notion with the extended $\\delta$ term is different from previous personalized DP and conventional DP. In the proof of Theorem G.1., the original SGM seems to be about conventional DP and RDP, but the author simply applied it to individual privacy without specific proof about its properties. \n  \n  * The paper shows the algorithm that finds parameters, but it does not provide a way to adapt the parameter to the original DP-SGD. Because DP-SGD uses a single sample rate and a single clip norm, adapting multiple sampling rate and clipping norm is also a new mechanism. It seems that a more specific explanation is needed in terms of implementation. For example, the original DP-SGD assumes Poisson sampling in principle, but approximates it due to computational cost. However, this is possible because the sampling probability is fixed, and it seems necessary to explain how to transform this to satisfy independent sampling.\n  \n  * It is difficult to know the exact process of the algorithm. Algorithm 3 is a very important part of the paper, but it is difficult to understand properly because it is omitted in the main part.\n\nW2) It is not clear how theoretically optimal Sample and Scale are. Looking at Table 1, $\\epsilon_p$ of Sample and Scale could have a loose bound depending on the $\\sigma_{\\mathrm{SAMPLE}}$ and $\\sigma_p$ values, respectively. It would be helpful to actually see what these values are in the experiments.\n\nW3) In Algorithm 1, the paper says that getSampleRate is equivalent to the Opacus function get_noise_multiplier without a detailed description, but the two functions seem to return different objects. The function getSampleRate returns a sample rate, but the Opacus function get_noise_multiplier returns noise_mutiplier $\\sigma$.\n\nW4) The comparison between Sample and Scale is not clear enough. The two methods are introduced and only empirically compared, and it seems that Sample mostly dominates Scale (according to Table 2).  Looking at the overall algorithm, both techniques look similar in terms of efficiency. Then when should one use Scale? \n\nW5) Also, a natural approach seems to be combining Sample and Scale where the sampling and clip norms are both adjusted. Why not use this combination?\n\nW6) The notion of individualized privacy ($\\epsilon_p$, $\\delta$)-DP seems to be confused with the original ($\\epsilon$, $\\delta$)-DP. This confusion leads to other confusions, e.g., whether ($\\epsilon_1$, $\\delta$)-DP in Theorem 3.1. and ($\\epsilon_P$, $\\delta$)-DP in Theorem 3.2. mean individual privacy or conventional DP.\n\nW7) The experimental setup of only using up to three privacy budgets seems limiting. What happens if there are many more than three budgets?\n\nW8) Some reference number is wrong. For example, in the last paragraph of the proof of Theorem G.1, Mironov et al. is [21] or [22], not [20].\n\nQUESTIONS:\nQ1) What is the actual name of ({$\\epsilon_1$,...,$\\epsilon_n$}, $\\delta$)-DP and ($\\epsilon_p$, $\\delta$)-DP? Are they Individualized Privacy or something else? If they are Individualized Privacy, why did the authors change the term “Personalized DP” to “Individualized Privacy” even though you referred to the Personalized DP paper?\n\nQ2) How can getSampleRate function be equivalent to the function get_noise_multiplier in Opacus? getSampleRate seems to return sample rate with input argument noise multiplier, while get_noise_multiplier returns noise_multiplier sigma.\n\nQ3) The comparison between DP-SGD and the proposed algorithms is as expected because DP-SGD assumes the same privacy level over all data points. Are there any results related to computational cost?\n\nQ4) A detailed description of the properties of each Sample and Scale is lacking. Sample seems to be superior overall in performance. Is there any reason to introduce the Scale technique? Or, what are the advantages of the Scale method compared to the Sample method?\n\nQ5) What is Thm. 4 and Thm. 11 from Mironov et al. [21] in the proof of Theorem G.1.? I could only find Proposition and Lemma in the paper. Did you mean [22]?\n\nQ6) How can you simply prove Theorem G.1. and Theorem G.2. without proving some basic theorems or properties of newly designed privacy notion? I don't think previous DP techniques can be applied to individual privacy directly. The authors just directly use the conversion technique from RDP to ($\\epsilon$,$\\delta$)-DP of previous works as a technique for the individual privacy conversion even though the privacy notion is new.\n\nLIMITATIONS:\nThe authors discuss limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed two variants of DP-SGD by manipulating the sampling rate and the gradient clipping bound for different groups to achieve the goal of having different privacy budgets for those groups and improving the overall performance of DP-SGD. The authors proved theoretical privacy guarantees for both variants. They also conducted an extensive experimental evaluation to showcase the advantage of their methods in many aspects.\n\nSTRENGTHS:\n1. The authors considered an important problem, having different privacy budgets for different groups of people to boost the overall utility, and proposed two algorithms to achieve the goal. The privacy guarantee of the algorithms is proved. Both algorithms are easy to implement and look novel although intuitive to me.\n2. The experiments in this paper are adequate for the evaluation of the proposed methods. The results are well-organized and visualized, which supports the validity and the advantage of the methods, and the details are provided enough for reproducibility. I understand and appreciate that the authors put most of the valuable experimental results in the appendix due to the page limitation.\n3. The writing and the organization of the content are good.\n\nWEAKNESSES:\n1. The composition theorem for Algorithms 1 and 2 is not proved or discussed in this paper. The proofs of Theorem G.1 and G.2 look okay to me although I think more explanations would make it more friendly to the general audience. However, I think there is still a need to show the composition theorem similar to the results in Feldman and Zrnic [9]. Moreover, if it is not much work for rewriting related parts of the paper, I think a more strict way to account for the privacy budget usage is by the Rényi Filter [9], i.e., individual RDP instead of RDP. \n2. Section 4.2 uses membership inference to empirically verify the privacy protection for the claimed DP guarantee, but the results in Figure 2 do not explicitly show the protection separately for each group ($\\varepsilon=10$ and $\\varepsilon=20$) compared with the DP guarantee. The authors may show the curves under two additional settings: vanilla DP-SGD with $\\varepsilon=10$, and vanilla DP-SGD with $\\varepsilon=20$. Then compare the two solid curves in Figure 2 with the two additional curves respectively.\n3. Both Algorithms 1 and 2 will potentially change the objective function of SGD (although DP-SGD with the gradient clipping has already changed it). There is no discussion on how specific choices of privacy budget distribution will affect the convergence and final performance of the optimization. This is important since it is related to the privacy-utility tradeoff. Please see my comment in *Limitations*.\n\n\nMinor weaknesses:\n1. Line 66. I think Table 7 is the same as Table 2 which is in the main text, not the appendix.\n2. Line 86. $\\log 1/\\delta$ -> $\\log(1/\\delta)$\n3. Line 98, equation (2). The inequality holds with many constraints on $\\alpha$. See Theorem 11 in Mironov et al. [21]\n4. Algorithm 1. The authors wrote: it is equivalent to Opacus' function get_noise_multiplier while Algorithm 2 is also equivalent to the same function. I think there is a typo.\n5. Algorithm 1, Line 5. I think this is more like a grid search which always decreases $\\sigma_{\\mathrm{sample}}$, while a binary search would be better.\n6. Line 209. The notation $\\stackrel{\\text{!}}{=}$ is not defined.\n7. Line 767 in Section G.2. $\\sum_{L\\subset D}$ -> $\\sum_{L\\subset D'}$\n---\nI have read the rebuttal which addressed my questions and the weakness concerns.\n\nQUESTIONS:\n1. In Line 379, could you explain the meaning of 'batch gradient descent'? Is it different from SGD? If yes, maybe it can be replaced with the word 'full-batch gradient descent' or just 'gradient descent'.\n2. In Table 8, we can see that for the first two rows, the accuracy decreases for classes 1-9 with a higher privacy budget of class 0 compared with the results with the same privacy budget of class 0. The same phenomenon exists in other rows. Intuitively, the higher budget of other classes should provide more information for the classifier so that the accuracy for each group would increase. Could you provide more insight into why this intuition is incorrect under this setting?\n\nLIMITATIONS:\n1. The discussion of the incentive and consequence of people choosing weak privacy protection is not enough in the main text. I found the content in Section D.2 very interesting since it aligned with my intuition: the more privacy being sacrificed, the better utility could potentially be obtained. Also, the results in Table 8 show that having a group with a higher privacy budget will affect the utility of other groups, which could lead to problems like the Prisoner's dilemma where people in the dataset have to give a higher privacy budget in order to maintain their original utility given the fact that other people are willing to have a higher privacy budget.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two variants of Differentially Private Stochastic Gradient Descent (DP-SGD) to train machine learning models that satisfy approximate personalized differential privacy (PDP), following the definition of Jorgensen et al. [15]. In contrast to vanilla DP-SGD where all points in the training dataset are assigned a global, uniform privacy budget $\\varepsilon$, the authors consider a scenario where points are split into privacy groups $\\{\\mathcal{G}_1,\\ldots,\\mathcal{G}_P\\}$ and each group $\\mathcal{G}_p$ is assigned a different privacy budget $\\varepsilon_p$. This allows individuals who contribute training data to specify their privacy preferences, reflecting the variety of privacy attitudes among users observed in past surveys.\n\nTo achieve PDP the paper modifies DP-SGD in one of two ways:\n\n1. **Sample** method: adjusting the sampling rate used to construct mini-batches, so that points in groups with a higher privacy budget are sampled more often than points in groups with lower privacy budget (and thus, stringent privacy preference).\n\n2. **Scale** method: adjusting the noise multiplier to scale the variance of noise added in a per-point basis. Rather than doing this directly, which would be inefficient, the method achieves the same effect indirectly by scaling the clipping norm of per-point gradients.\n\nThese modifications are such that given target group privacy budgets and a number of training steps, the mini-batch noise multiplier in DP-SGD can be chosen so that the privacy budget for each group is exhausted at the end of training, maximizing utility.\n\nThe paper evaluates **Scale** and **Sample** on CNNs trained over MNIST, SVHN and CIFAR-10 using 3 privacy groups with budgets $\\varepsilon_p = p$ for $p \\in \\lbrace1,2,3\\rbrace$, distributing points into groups according to proportions used in prior work. This evaluation shows top-1 accuracy improvements ranging between 1-5% over vanilla DP-SGD with a uniform privacy budget $\\varepsilon_1 = 1$ (which guarantees PDP, but underutilizes the budget for the other two groups). The methods also compare favorably to Individualized PATE [5], which uses different techniques but achieves the same privacy guarantee.\n\nThe paper also demonstrates that the empirical evaluation of membership inference attacks against models trained with PDP using the above two methods shows the expected gap between groups with different privacy budgets. Finally, the authors discuss how to combine individual privacy assignment with individual privacy accounting as done by Feldman and Zrnic [10] for batch gradient descent.\n\nThe supplemental material provides implementations of **Scale** and **Sample** in the Opacus library as well as scripts to reproduce the results in the paper.\n\nSTRENGTHS:\n1. **First DP-SGD variants designed to achieve personalized differential privacy**\nThere is abundant literature for adapting differentially private mechanisms for general data analysis to achieve personalized differential privacy as well as for computing personalized differential privacy guarantees for training data points in ML models. While PDP accounting alone can be used to improve model utility modestly (see Feldman and Zrnic [10]), as far as I know the only other method that trains models with individual privacy assignments is Individualized PATE [5] which uses significantly different techniques.\n\n2. **Original combination of existing mechanisms**\nThe **Sample** mechanism is inspired by the \"Sampling\" mechanism of Jorgensen et al. [16], while **Scale** is inspired by the \"stretching\" mechanism of Alaggan et al. [2]. Combining these mechanisms with DP-SGD is relatively straightforward, but has not been done before.\n\n3. **Problem is well motivated and solution is contextualized relative to prior work**\nThe authors discuss related work up front, explain how their solution fits into the existing literature, and justify their contributions.\n\n4. **Complete algorithmic descriptions of proposed methods** The paper describes the adaptations of DP-SGD in sufficient detail, including pseudocode and justifications for design decisions.\n\n5. **Demonstrates utility gains compared to alternative solutions** The empirical evaluation on image classification tasks demonstrates a modest utility gain compared to alternative baselines.\n\n6. **Comprehensive supplemental material** The Appendix includes valuable additional discussions. The accompanying code is a significant extension to an existing library and demonstrates the practicality of the techniques.\n\nWEAKNESSES:\n1. **Limited empirical evaluation** The empirical evaluation is limited to convolutional architectures trained from scratch for image classification. It is unclear how the paper findings hold for other architectures, modalities, tasks, or practically relevant scenarios, such as fine-tuning.\n\n2. **Modest utility gains** The utility gains with respect to vanilla DP-SGD are modest (1-5%). The comparison to alternative baselines with simpler implementations in Appendix E shows even more modest gains. For instance, simply training sequentially on each group separately using DP-SGD achieves accuracies only 0.3-0.41% below the best of **Sample** and **Scale** (see Table 12). It is unclear to what extent these differences are statistically significant, whether comparable effort has been put in hyperparameter tuning, and whether mild modifications to this alternative baseline could outperform **Sample** and **Scale**.\n\n3. **Unsupported claims about the confidentiality of privacy preferences** The authors argue that privacy preferences (i.e., per-group privacy budgets) should be kept private and claim that their modifications of DP-SGD do not leak information about privacy preferences because untrusted parties only interact with the final trained model. This claim is unsupported by proof (compare this to Allagan et al. [2], which prove a similar guarantee explicitly) and is made with respect to a different adversary model than the privacy guarantees of training data in DP-SGD, which consider adversaries that observe noisy gradients released at every training iteration, not only the final model.\n\n\n## Minor comments\n\n- Given the number of references to numbered lines in Algorithm 3, it would be convenient to include the listing in the body of the paper.\n\n- l.76: \"[D]iffer in any one record\" is ambiguous. Consider making clear that you use the add/remove neighboring relation. That is, $D,D'$ are adjacent if $D = D' \\cup \\{x\\}$ for some record $x$, or vice versa.\n\n- l.79: \"DP bounds privacy leakage for any individual\". This is only true under the assumption that an individual contributes at most one record.\n\n- l.86: The original conversion from $(\\alpha, \\rho)$-RDP to $(\\varepsilon, \\delta)$-DP of Mironov [21] is suboptimal. Balle et al. [A, Theorem 21] provide a tighter conversion (which is the one Opacus uses).\n\n- Table 1: $\\sigma_{\\rm SAMPLE}$ should read $\\sigma_{\\rm sample}$.\n\n- Algorithm 1: Missing inputs: scaling factor $s_i$, target sample rate $q$.\n\n- Algorithm 1: In l.1, you could initialize $\\sigma$ using $\\textit{getNoise}$ as $\\sigma \\gets \\textit{getNoise}(\\varepsilon_1, \\delta, q, I)$.\n\n- Algorithm 1: In l.4 $G_p$ should read $\\mathcal{G}_p$.\n\n- l.190: \"our Sample [method]\"\n\n- l.209: While I have seen it being used before, I think that the notation $\\stackrel{!}{=}$ is far from being conventional.\n\n- l.215: \"[W]e still require the expected average sampling rate to remain $q$ to obtain constant mini-batch size $B$\". As you said just before, the mini-batch size isn't constant, $B$ is the **expected** size. Rather, you want the expected mini-batch size in IDP-SGD to be the same as in vanilla DP-SGD with sampling rate $q$.\n\n- l.225: There's an extra closing parenthesis.\n\n- l.232: DPSGD should read DP-SGD.\n\n- l.260: Use \\{$\\sigma_1,\\ldots,\\sigma_P$\\} rather than $\\sigma_p$-s.\n\n- l.313 (and subsequently): \"Lira\" should read \"LiRA\"\n\n- Numbers in bibliographic references are shifted between the paper and the full paper (with Appendix) provided as supplemental material.\n\n- Appendix, Algorithm 3: In line 11, $\\theta_T$ should be $\\theta_I$.\n\n- l.566: $\\sigma_{\\rm in}$, $\\sigma_{\\rm out}$ should read $\\sigma^2_{\\rm in}$, $\\sigma^2_{\\rm out}$, respectively.\n\n\n- l.576: \"Line 3 in Algorithm 3\" should be \"Line 8 in Algorithm 3\"\n\n- l.591-596: $G_p$ should read $\\mathcal{G}_p$.\n\n- l.691: \"Appendix E.3\" should read \"Table 12\"\n\n- Table 10: How can it be that you get a statistically significant negative $\\Delta$ for one of the models? That would reverse the relation between the two privacy groups.\n\nQUESTIONS:\n1. **Confidentiality of privacy budgets**\nThe paper argues that IDP-SGD preserves the confidentiality of group privacy preferences, at least against parties that only interact with the final trained model. However, intuitively, models trained with a uniform privacy budget of $\\varepsilon=1$ and models trained on the same data but\nincreasing the privacy budget for a single group from $\\varepsilon=0.1$ to $\\varepsilon=100$ will likely perform noticeably different on that group. Can you provide a proof that IDP-SGD satisfies some form of confidentiality for privacy preferences that makes explicit the adversary model and assumptions?\n\n2. **LiRA AUC scores**\nCarlini et al. [6] report LiRA results on models trained with DP-SGD and $\\epsilon = 8$ (not far from one your choices of $\\epsilon = 10$), with a significantly lower AUC score of 0.503 compared to the score of 0.537 that you report. Even for $\\varepsilon > 5000$, their best result has a lower AUC score of 0.527. Could you explain this difference?\n\nLIMITATIONS:\nThe authors adequately address societal impact but only some limitations.\n\nA paragraph at the end of the introduction discusses the need for openly communicating privacy risks and regulatory supervision. This points to Appendix A, which discusses broader impacts in the light of prior studies and some limitations: evaluation on standard benchmarks with simulated privacy preference distributions; practical impact measured via membership inference attacks only. I think that the limited nature of the evaluation on CNN models trained from scratch on visual tasks should also be brought up front, as the results might differ in other settings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}