Have it your way: Individualized Privacy Assignment for DP-SGD
Authors: Franziska Boenisch
Christopher Mühl
Adam Dziedzic
Roy Rinberg
Nicolas Papernot
Venue: NeurIPS 2023 poster (2023)
Keywords: privacy
machine learning
differential privacy
DP-SGD
individualized privacy
TL;DR: We propose two methods that enable DP-SGD training of ML models with individual (per data point) privacy guarantees.

Abstract: When training a machine learning model with differential privacy, one sets a privacy budget. This uniform budget represents an overall maximal privacy violation that any user is willing to face by contributing their data to the training set. We argue that this approach is limited because different users may have different privacy expectations. Thus, setting a uniform privacy budget across all points may be overly conservative for some users or, conversely, not sufficiently protective for others. In this paper, we capture these preferences through individualized privacy budgets. To demonstrate their practicality, we introduce a variant of Differentially Private Stochastic Gradient Descent (DP-SGD) which supports such individualized budgets. DP-SGD is the canonical approach to training models with differential privacy. We modify its data sampling and gradient noising mechanisms to arrive at our approach, which we call Individualized DP-SGD (IDP-SGD). Because IDP-SGD provides privacy guarantees tailored to the preferences of individual users and their data points, we empirically find it to improve privacy-utility trade-offs.

Have it your way: Individualized Privacy Assignment for DP-SGD
Franziska Boenisch* boenisch@cisse de CISPA Helmholtz Center for Information Security
Christopher Mühl
christopher wwwhl@fu-beeeeeee de
Free University Berlin
Adam Dziedzic*
Roy Rinberg
dziedzi eeleziiicciee ceeecippa...ee... de CISPA Helmholtz Center for Information Security
roy rinberg@col letfcoouumii.ee. edu Columbia University
Nicolas Papernot
nicolas. papernottutoronta.
University of Toronto Vector Institute
Abstract
When training machine learning model with differential privacy, one sets privacy budget. This uniform budget represents an overall maximal privacy violation that any user is willing to face by contributing their data to the training set. We argue that this approach is limited because different users may have different privacy expectations. Thus, setting a uniform privacy budget across all points may be overly conservative for some users or, conversely, not sufficiently protective for others. In this paper, we capture these preferences through individualized privacy budgets. demonstrate their practicality, we introduce a variant Differentially Private Stochastic Gradient Descent (DP-SGD) which supports such individualized budgets. DP-SGD is the canonical approach to training models with differential privacy. We modify its data sampling and gradient noising mechanisms arrive at our approach, which we call Individualized DP-SGD (IDP-SGD), Because IDP- SGD provides privacy guarantees tailored to the preferences of individual users and their data points, we empirically find to improve privacy-utility trade-offs.
Introduction
Machine learning (ML) models are known to leak information about their training data. Such leakage can result in attacks that determine whether specific data point was used to train given MI. model (membership inference) [36] 43 [) infer sensitive attributes from the model's training data [13 45). even (partially) reconstruct that training data [1]] 35). Therefore, the need to provide privacy guarantees to the individuals who contribute their sensitive data to train ML models is pressing.
Approaches to train ML models with guarantees of differential privacy (DP) 11 have established themselves as the canonical answer to this need. In the context of ML, differential privacy bounds how much can be learned about any of the individual data points from the model's 's training set. Take the canonical example of differentially private stochastic gradient descent (DP-SGD): it updates the model using noisy averages of clipped per-example gradients. Unlike vanilla SGD, this ensures that any single training point has limited influence on model updates (due to the clipping to pre-defined maximum clip norm) and that the training is likely to output similar model should single point
Equal contribution. Work was done at the University of Toronto and the Vector Institute.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

be added to or removed from the training set (due to the noisy average). As is the case for any DP algorithm, the privacy guarantees of DP-SGD are derived analytically. They are captured by a privacy budget e. This budget represents the maximal privacy violation that any user contributing data to the training set willing to tolerate. Lower budgets correspond to stronger privacy protection since they impose that the outputs of the training algorithm have to be closer for similar training sets.
In this paper, we argue that there is key limitation when training ML models with DP guarantees: the privacy budget e is set uniformly across all training points. This implicitly assumes that all users who contribute their data to the training set have the same expectations of privacy. However, that is not true: individuals have diverse values and privacy preferences [17 S Yet, because DP-SGD assumes a uniform privacy budget, this budget must match the individual whose privacy expectations are the strongest. That is, has to correspond to the lowest privacy violation tolerance expressed amongst individuals who contributed their data. This limits the ability of DP-SGD to learn because the privacy budget& & together with the sensitivity of each step of the algorithm (i.e., the clip norm) define the scale of noise that needs to be added: the stronger the required privacy protection, the higher the noise scale. Thus, implementing privacy protection according to the lowest privacy violation tolerance in the training dataset comes at a significant cost in model accuracy.
We are the first to propose two variants of the DP-SGD algorithm, that we refer to as IDP-SGD, to enforce different privacy budgets for each training point First, our modified sampling mechanism (Sample) samples data points proportionally to their privacy budgets: points with higher privacy budgets are sampled more frequently during training. It is intuitive that the more a training point is analyzed during training, the more private information can leak about this point. Second, with our individualized scaling mechanism (Scale), we rescale the noise added to gradients based on the budget specified for each training point. Naively, one would implement Scale by changing the scale of added noise per individual data point. However, for efficiency, current implementations of DP-SGD typically noise the sum of the per-example clipped gradients over an entire mini-batch, hence, adding the same amount of noise to all gradients. Yet, within the context of this implementation, we note that we can effectively adapt the scale of noise being added on an individual basis by adjusting the sensitivity (i.e. the clip norm) of each example by a multiplier. Because clipping is performed on a per-example basis, this approach enables the efficient implementation of our proposed Scale method.
To summarize our contributions:
We introduce two novel individualized variants of the DP-SGD algorithm, where each data point can be assigned its desired privacy budget. This allows individuals who contribute data the training set specify different privacy expectations.
We provide a theoretical analysis of our two mechanisms to derive how the privacy parame- ters (e.g., noise multipliers and sample rates) should be set to meet individual data points' privacy guarantees and provide a thorough privacy analysis for Sample and Scale.
We carry out an extensive empirical evaluation on various vision and language datasets and multiple model architectures. Our results highlight the utility gain of our methods for different distributions of privacy expectations among users.
Ethical considerations. We need to ensure that our individualized privacy assignment does not harm individuals by disclosing too much of their sensitive information. We, therefore, suggest implementing our methods in controlled environment where privacy risks are openly communicated [41 9 12], and where a regulatory entity is in charge of ensuring that even the individuals with lowest expectations regarding their privacy obtain sufficient degree of protection. We further discuss the ethical deployment of our Individualized DP-SGD (IDP- SGD) in Appendix
Background and Related Work
2.1 Differential Privacy and DP-SGD
Algorithm M satisfies (e, DDP if for any two datasets D, D' C that differ in any one record z, such that D = D' U z, and any set of outputs R
P (M) E R < e P[M(D') E R + 8.
(1)
II the rest of this paper, we assume without loss of generality that each training point is contributed by different individual.
2

---

Since the possible difference in outputs is bounded for any pair of datasets, DP bounds privacy leakage for any individual. The value E R specifies the privacy level with lower values corresponding to stronger privacy guarantees. The 8 E [o. 1J offers a relaxation, ie., small probability of violating the guarantees. See Appendix B.1 for a more thorough introduction of (e, 6)-DP.
Another relaxation of DP is based on the Rényi divergence [27]. namely (a, e) -)-DP,, which for the pa- rameters as described above and an order a E (1, oo) defined as follows: Di (M M(D')) < é. Due to its smoother composition properties, it is employed for privacy accounting in ML with DP-SGD. Yet, is possible to convert between the two notions: If M is an (a, -eDDP mechanism,
also satisfies (e + log 0-1 log logó+loga
5)-DP for any < 8 <
a-1
DP-SGD extends standard SGD with two ad- ditional steps, namely clipping and noise ad-
Algorithm 1: Differentially Private SGD [I]
dition. Clipping each data point's gradients a pre-defined clip norm e bounds their sen- sifivity to ensure that data point causes too large model updates. Adding noise from Gaussian distribution, (0, (I)) with zero mean and standard deviation according to the sensitivity e and pre-defined noise multiplier a introduces privacy. We formally To yield
Require: Training dataset with data points
(Ii, (Iı...., xN. J. loss function I, learning rate 17, noise multiplier a, sampling rate clip norm c, number of training iterations I.
1: Initialize dg randomly
2:
for E [7] do
3: Poisson Sampling: Sample mini-batch L, with
per-point probability q from D.
4: For each E Lt, compute gi (2x)) <- Vol (0e,(0)
depict DP-SGD as Algorithm| [ tighter privacy bounds, DP- implements
5: Gradient Clipping
6:
gi (Ir²) <- (xx)/ max (1, Ie((s))le)
privacy amplification by subsampling 4 (line
7: Noise Addition
3 in Algorithm[] [ where training data points
8:
gt <- [L. (E. (2x)) + +O, (o
are sampled into mini-batches with Pois-
9:
0c+1 < 0,
7/
son sampling rather than assigning each
10:
end for
data point from the training dataset to a mini-
Output 0z, privacy cost (e. 8) computed using batch before an epoch starts. DP-SGD hence privacy accounting method. implements a Sampled Gaussian Mechanism
11:
(SGM) (28 where each training data point is sampled independently at random without replacement with probability g E (0, 1] over I training iterations. As shown by Mironov et al. (28), an SGM with sensitivity of 1, satisfies EDD where
E S 1
2g 02
Equation 2 highlights that the privacy guarantee depends on the noise multiplier o,, the sample rate g, the number of training iterations I, and the RDP order a.
2.2 Individualized Privacy
Individualized Privacy is a topic of high importance given that society consists at least of three different groups of individuals, demanding either strong, average, or weak privacy protection for their data, respectively [17 [5 Furthermore, some individuals inherently require higher privacy protection, given their personal circumstances and the degree of uniqueness of their data. However, without individualization, when applying DP to datasets that hold data from individuals with different privacy requirements, the privacy level E has to be chosen according to the lowest e encountered among all individuals, which favors poor privacy-utility trade-offs. Prior work on individualized privacy guarantees focused mainly on standard data analyses without considering ML [ 19 24 31 However, some of the underlying ideas inspired the design of our IDD-SGD variants.
Our Sample method relies on similar idea as the sample mechanism proposed by Jorgensen et al. 19 where data points with stronger privacy requirements are sampled with lower probability than data points that agree to contribute more of their information. The most significant difference between the sample mechanism and our approach is that the former performs sampling step independent of the subsequent algorithm. In contrast, we perform subsampling within every iteration of our IDP-SGD training and show how to leverage the sampling probability of DP-SGD to obtain individual privacy guarantees. Based on this difference, we could not build on how Jorgensen
pre-processing
'n DPDSGD, the concept of an epoch does not exist and training duration is measured in Merations. Due to the Poisson sampling, training
can contain varying numbers of data points.
3

---

et al. [9 compute the sampling probabilities and had to propose an original mechanism to derive sampling probabilities for individualized privacy in IDP-SGD.
In a similar vein to our Scale method, Alaggan et al. (2 scale up data points individually before the noise addition increase their sensitivity in their stretching mechanism. As a consequence, the signal-to-noise ratios of data points that are scaled with large factors will be higher than the ones of data points that are scaled with small factors. In contrast to Alaggan et al. 22. we do not scale the data points themselves, but the magnitude of noise added to their gradients, relative to their individual sensitivity. Our new interesting observation is that data points' individual clip norms can be used to indirectly scale the noise added to a mini-batch of data in DP-SGD per data point.
Closest work to ours 6 proposes two mechanisms for individualized privacy assignment within the framework of Private Aggregation of Teacher Ensembles (PATE) 33 Their upsampling duplicates training data points and assigns the duplicates to different teacher models according the data points' privacy requirements, while their weighting changes the aggregation of the teachers' predicted labels to reduce or increase the contribution of teachers according to their respective training data points' privacy requirements. PATE's approach for implementing ML with DP differs significantly from DP- SGD: PATE relies on privately training multiple non-DP models on the sensitive data and introducing DP guarantees during a knowledge transfer to a separately released model. The approaches for individualized privacy in PATE, therefore, are non-applicable to DP-SGD, See Appendix [B. for details on the PATE algorithm and its individualized extensions.
Relationship to Individualized Privacy Accounting. An orthogonal line of research that comes closest to providing individualized guarantees for DP-SGD focuses on individualized privacy account- ing: the idea is to learn more from points that consume less of the privacy budget over training [1] 44) Jordon al. [8] propose personalized moments' accountant to compute the privacy loss per-sample basis. Yu et al. [44] perform individualized privacy accounting within DP- based on the gradient norms of the individual data points. Feldman and Zrnic [11] introduce RDP filters to account for privacy consumption per data point and train as long as there remain data points that do not exceed the global privacy level. Yet, it has been shown that the data points that consume little privacy budget and therefore remain in the training are the ones that already incur a small training loss [4). Training more on them will not significantly boost the model accuracy. While privacy assignment and accounting are concerned with improving privacy-utility trade-off's within DP, they operate under different setups. Privacy accounting (in the above three methods) assumes a single fixed privacy budget assigned over the whole dataset. In contrast, our privacy assignment is concerned with enabling different individuals to specify their respective privacy preferences. Since individual accounting and assignment are two independent methods, we experimentally show their synergy by applying individual assignment to individual accounting [T], which yields higher model utility than individual accounting alone.
3
Our Individualized Privacy Assignment
3.1 Formalizing Individualizing Privacy
Setup and Notation. Given training dataset D with points fs1,..., ,N} which each have their own privacy preference (or budget), we consider data points with the same privacy budget Ep together as a privacy group Gy- This notation is aligned with [17(55 that identified different groups of individual privacy preferences within society. We denote with 61 the smallest privacy budget encountered in the dataset. Standard DP-SGD needs to set = E1 comply with the strongest privacy requirement encountered within the data. For all Ep. E [2, P). it follows that ep > e and we arrange the groups such that Ep > Ep-1- Note that following [2]. we argue that the privacy preferences themselves should be kept private to prevent Icakage of sensitive information which might be correlated with them. If there is necessity to relcase individual privacy preferences, this should be done under the addition of noise to obtain DP guarantees for the privacy preferences. This can for instance be done through a smooth sensitivity analysis, as done in prior work, egg. (4). In general, we find that it is not necessary to release the privacy budgets: To implement DGD, only the party who trains the ML model on the individuals" sensitive data needs to know their privacy budgets to set the respective privacy parameters accordingly during training. Given that this party holds access to the data itself, this does not incur any additional privacy disclosure for the individuals. Other parties then only interact with the final private model (but not with the individuals' sensitive data or their

---

associated privacy budgets) and it has been shown impractical to disclose privacy guarantees through black-box access to ML models [4]. We discuss the confidentiality of the privacy budgets further in Appendix A.2
Individualized Privacy. Following [19], our methods aim at providing data points n; Ip with individual DP (IDP) guarantees according to their privacy budget Ep as follows: The learning
algorithm M satisfies (Ep. 6)-PP if for all datasets D 2s D' which differ only in x; and for all
outputs R _ R
(3) Without loss of generality, we assume that all individual data points have privacy preferences with the same d. Note that other notions of DP (e.g.. RDP) can analogously be generalized to enable per-data point privacy guarantees. The key difference between Equation (3) and the standard definition of DP (Equation (I)) does not lie in the replacement of e with Ep. Instead, the key difference is about the definition of neighboring datasets. For standard DP, neighboring datasets are defined as D D' where D and D' differ in any random data point (given that it is standard DP, every data point has privacy E). In contrast, in our work, following [19] the neighboring datasets are defined as D X3 D' where D and D' differ in any arbitrary data point æt which has a privacy budget of (Ep, This yields to the individualized notion of DP.
P (IM(D) E RJ S e Ser--I[MM FIM(D') E +5
We formalize the relationship between the individualized notion of DP (which we denote by (fE1, .EEE, E.), (p)55% 5)-IDP) and standard (c, 5DD in the following two lemmas over the learning algorithm M. The proofs are included in Appendix G
Lemma 3.1. An algorithm M that satisfies (e1, 6DD also satisfies ())),
Lemma 3.2. An algorithm M that satisfies ((:1,,2..... (.)),, -IDD also satisfies (EP,6)-DP.
3.2 From Individualized Privacy Preferences to Privacy Parameters
From the individual privacy preferences and the total given privacy distribution over the data, i.e., the number of different privacy budgets P. their values Ep. and the sizes of the respective privacy groups [ppl, we derive individual privacy parameters for IDP-SGD such that all data points within one privacy group (same privacy budget) obtain the same privacy parameters, and such that all privacy groups are expected to exhaust their privacy budget at the same time, after training iterations.
The individualized parameters for our Sam- ple method are the individual sample rates
Table 1: (Individualized) Privacy Bounds.
{qi...., gP). and a respective noise multi- plier sampls - common to all privacy groups < 2 Ep 20p2 "ammple which we derive from the privacy budget dis-
DP-SGD
SAMPLE
SCALE
1 22 op
tribution as described in Section 3.3 In Scale, the individualized parameters are noise multipliers f JP} with their respective clip norms (cI..... ). as we explain in Section 3.4 Our individual bounds per privacy group are depicted in [able We omit a from consideration since its optimal value is selected as an optimization parameter when converting from RDP to (c, d)DP.
3.3 Our Sample Method
Our Sample method relies on sampling data points with different sample rates (. qr de- pending on their individual privacy budgets. In this case, the noise multiplier sample is fixed. Data points with higher privacy budgets (weaker privacy requirements) are assigned higher sampling rates than those with lower privacy budgets. This modifies the Poisson sampling for DP-SGD (line 3 in Algorithm to sample data points with higher privacy budgets within more training iterations.
Deriving Parameters. We aim at deriving the privacy parameters that yield the specified individual privacy preferences. The learning hyperparameters, such mini-bated size B or learning rate n can be found through hyperparameter tuning a For Sample, given a tuned mini-batch size B, we
have to find (gI.....qP] ) such that their weighted average equals q: 1/4 Ep= 1pplap 1
85
WW discuss in Appendix C.1 other alternatives to implement Sample if we allow changing the training hyperparameters.
We use the notation - to indicate that the terms should be equal.
5

---

Algorithm& 2: Finding Sample Parameters.
Algorithm 3: Finding Scale Parameters. The subroutine getNoise corresponds to Opa- cus' function vet_noise___ mastsipiieelp [3].. We sketch its implementation in Algorithm in Appendix C.3
The subroutine getSampleRate is equivalent to Opacus' function get_ _es__ooie________________ _ [32], (see getNoise ilgorrith
5). We sketch gel-
SampleRate in Algorithm 4 in Appendix C.3
Require: Per-group target privacy budgets
Require:
Per-group target privacy budgets
{51,-.., Er target 6, iterations I, number total data points N and per-privacy group data points
{EI,..., target 6, iterations I, number of total data points N and per-privacy group data points
[9
I. default clip norm
1: init Grampte: Osample < getNoise(e1 8. 1.1)
sample rate
2: init (qI..... .P) where for p E [P]:
1: init (oi,..., OP] where for p E [P]:
3: Oo f getSampleRatc(Ep. 8, dssmple: 7)
2: 0p + gettNoiss((pp: 6,
4: while & 2/2 Epm (pll.:
3: init dseate: Oscate + (N >p= 10p1 {see
5: dsample f- s:mstmpte (scaling factor: <1)
derivation of Gueat in Appendix C.2
6: for p E [P]:
4: for p [P]:
7: Ip < gesSampleRato(Ep: 6, Ssaplee T)
5: Cp f- Sezte"
8: Output sample: {41..... .P)
6: Output Sscales for, Cp}
This asserts that the Poisson sampling (line 3 in Algorithm D will yield mini-batches of size B in expectation. We also need to ensure that the privacy budgets of all groups will exhaust after I training iterations, which we do by deriving the adequate sample- shared across all privacy groups. Equation ( highlights that the final privacy budget depends on both sample rate q and noise multiplier σ. Our Sample method enables higher individualized privacy budgets for the different groups than the standard DP-SGD but still requires setting the average sampling rate equal to g as in DP-SGD to obtain an expected batch size B. Hence, the value of xxmple has to be decreased in comparison the initial default As an effect of this noise reduction, Sample improves utility of the final model.
Our concrete derivation of values for sample and (gi...., .p.......ppp............ } is presented in Algorithm 2 We start from initializing sample with a from standard DP-SGD, the noise multiplier required for the privacy group Gi (strongest privacy requirement of all groups) with the smallest privacy budget s1. This corresponds to instantiating sample with the upper bound noise over all privacy groups. Then, we use getSampleRate function that derives the sampling rate for the given privacy parameters based on approximating Equation 2 Finally, we iteratively decrease sample by a scaling factor s; slightly smaller than and recompute (q..-.., }r) until their weighted average is (approximately) equal to q. In Appendix G we present the formal proof that Sample satisfies (fE1, C2 E2,..., | 5)-D. Our proof relies on Mironov et al. (28) and considers training of each privacy group as a separate SGM with an individual sampling probability- that all simultaneously update the same model.
3.4
Scale
Our Scale method aims at scaling the noise added to each gradient according to the respective data point's 's privacy preference. Yet, for efficiency, current implementations of DP-SGD typically noise the sum of the per-example clipped gradients over entire mini-batch, hence, adding the same amount of
noise to all gradients. Therefore, we instead set individualized clip norms {ci..... cP} } that effectively adapt the scale of noise being added on individual basis by adjusting the sensitivity (i.e. the clip norm) of each example by a multiplier. This changes lines 6 and 8 in the DP-SGD Algorithm (T). Data points with higher privacy budgets (weaker privacy requirements) obtain lower noise and higher clip norms. Since Equation (2) highlights that the clip norm c has no direct impact on the obtained e, individualized privacy in Scale results from the individual noise multipliers { op). Utility gains come from the overall increase in the signal- -ngnos-to-noise ratio during training.
Deriving Parameters. While the ultimate goal of our Scale method is to adapt individual noise multipliers per privacy group, we cannot implement this directly without degrading training perfor- mance. The reason is that whereas in DP-SGD sampling and gradient clipping are performed on per-data point basis, noise is added per mini-batch, see lines 3, and Algorithm[] respectively. However, if we restrict mini-batches to contain only data points from the same privacy group (which

---

share the same noise multiplier), we lose the gains in privicy- trade-offs which result from the subsampling (see Appendix f.r more details). Hence, while we rely on mini-batches containing data points with different privacy requirements (iee, different noise multipliers) we can only specify one fixed noise multiplier scale.
To overcome this limitation, we do not set noise multipliers {o1,..., op} directly, but indirectly obtain them through individualized clip norms {cI,..... : as follows: In standard DP-SGD, Algorithm I a gradient clipped to (line 6) obtains noise with standard deviation ae (line 8). For Scale, we clip gradients Cp = SpC with a per-privacy group scaling factor Sp and they obtain noise CpCp. But in practice, we add noise according to scale to all mini batches. Hence, the effective scale
Eo - OpCp - Op- = Seate For data points with higher privacy Op of added noise is Sccale = scaale 5y budgets Sp > 1, so their gradients are clipped to larger norms cp syC and a smaller noise multiplier dp- - ssale is assigned to them. The opposite is true for data points with lower privacy budgets. *p
fy
We find the values of on required to obtain each privacy groups' desired Ep using the getNoise function (see Algorithm 3 line 1), which takes as inputs the privacy parameters Ep. 5, q, I. To optimize utility, we want to set the individual clip norms such that their average over the dataset
corresponds to the best clip norm e obtainned through initial hyperparameter tuning: * E pplle & Given the interdependence of Cp, Tp. and scale (Cp c = this can be achieved by setting
Oscale as the inverse of the weighted average over all 1/0p as: Oscale ( Ep=1 19 see Appendix C.2 for the derivation of ssale). Given &scale, the required through hyperparamter tuning, the individual clip norms are calculated Cp Sewc We detail the
[ (please and found
JpJ. dy
Op derivation of the parameters in Algorithm 3 In Appendix G we formally show that Scale satisfies (fE1, E2,.... Er), 8)-DP). Similar to Sample, the proof is based considering the training for all privacy groups as simultaneously executed SGMs with differing sensitivities.
Empirical Evaluation
For our empirical evaluation, we implement our methods in Python 3.9 and extend standard Opacus with our individualized privacy parameters and per-privacy group accounting. We perform evaluation on the MNIST 123] SVHN [29]. and CIFARIO 10 [22] dataset, using the convolutional architectures from Tramer and Boneh 38 for most experiments, and additionally evaluate on various language
datasets and diverse (larger) model architectures in Section 4.2) To evaluate the utility of our methods, we use the datasets' standard train-test splits and report test accuracies. The training and standard DP-SGD and IDP-SGD hyperparameters are specified in Table|]aand Table Appendix [D respectively where the noise multiplier derived with the function provided in Opacus (32. It takes in as arguments the specified parameters 6, 9, I, and target s. For experimentation on individualized privacy, we assigned privacy budgets randomly to the training data points if not indicated otherwise.
moose_multippiee
4.1 Utility Improvement and General Applicability of Individualization
Assigning heterogeneous individual privacy budgets over the training dataset yields significant improvements in terms of the resulting model's utility, showcased in Table For example, by following the privacy budget distribution of Alaggan et al. (2) with privacy budgets Ep = {1, 2, 3} for strong, medium, and weak privacy requirements, our Sample method yields accuracy improvements of 1.06%, 2.63%, and 5.09% on MNIST, SVHN, and CIFAR 10, respectively. On the CIFARIO dataset, our Scale even outperforms improvement with accuracy increase of 5.26% in comparison to the ioniindividulliee baseline, which would have to assign E = 1 to the whole training set in order to respect each individual's privacy preferences.
The benefits of our individualized privacy assignment also become clear in Figure D which de- picts the test accuracy of our Sample and Scale vs. standard DP-SGD over the course of train- ing. Both our methods continuously outperform the non-individualized DP-SGD baseline with 1. Additionally, the test accuracy with privacy budget distribution according to Alaggan et (2) (34%, 43%, 23%) is higher than the one of Niu et al. 30) (54%, 37%, 9%). This can be explained by the fact that in this latter distribution, more individuals exhibit higher pri-
7

---

Table 2: Model Test Accuracy after training with Standard DP-SGD vs our Individualized DP-SGD using Sample or Scale. The percentages of the three privacy groups are chosen according
to 2 (first setup) and 33 (second setup). Additional hyperparameters are found in Table and
Table 6
We report the standard deviation over 10 trials.
DATASET PRIVACY GROUPS
PRIVACY BUDGETS
DP-SGD
SAMPLE
SCALE
34%-449%-23%
10-2.003.00
96.75+0.155
97.81+0.09
97.78+0.08
MNIST
54%-37%-9%
1.0-2.0-300
96.75+0.15
97.6+0.11
97.54+0.09
34%-43%--25%
1.0-2.0-3.0
83.26.10.31
85.89110.14
85.57
SVHN
54%-37%-995
1.0-2.0-300
83.26+0.31
85.1410.30
85 85.08+0.12
34%-43%-23%%
1.0-2.0-3.0
52.77-10.65
57.8610.56
58.0310.36
CIFARIO
54%-337%-9%
1.0-2.0-3.0
52.77-10.65
56.8311.399
56.65-10.49
vacy preference. With more data points choosing lower privacy protection in Alaggan al. 2]. our individualization boosts the performance of the final trained model more significantly. For comparison, also report results for the standard DP-SGD with & = 3. This corresponds to the upper bound utility that could have been achieved by assigning the highest privacy budget to all data points-which violates the pri-
31sP scD Method
Stale
Teut Accaaoy
Sudget Dutristtonn 1300%
13 2315
vacy requirements of individuals with strong and medium privacy preferences. We observe that our individualized methods' performance is close to this upper bound without violating data points' privacy requirements. In Appendix E.] we also discuss other possible baselines (apart from stan- dard DP-SGD) and empirically evaluate our indi- vidualized methods against them. We report our hyperparameters and the respective individualization parameters (noise multipliers, clip norms, and sampling rates) obtained through our methods in Appendix D Figure in Appendix D.[hhighigght additionally that this privacy parameter derivation is well calibrated: all different groups exhaust their privacy budget simultancously when the targeted number of iterations is reached. In Appendix D.2 we also experimentally show that our methods flexible and can extend to substantially more privacy groups which all have different budgets Finally, in Appendix D.6] we demonstrate how to combine our Sample and Scale method, and experimentally evaluate the joint method.
46.5
1600kin 0sed) -137 derattoon
Figure 1: IDP-SGD vs DP-SGD.
4.2 Broad Applicability of IDP-SGD
We also evaluate IDP-SGD with other (larger) architectures, for fine-tuning, and with different modalities and tasks. Therefore, we 1) fine-tune BERT transformer (21) (bert-base-uncd
on the SNLI dataset (7] for natural language inference, 2) train a ResNet18 [16] from seratch on the CIFAR10 10 dataset for image classification,
3) train linear embedding model (one embedding and two linear layers) on IMDB data (25] for text classification, and 4) train a character-level RNN (DPLSTMM) three layers. hidden size: 128) from scratch to classify surnames to languages. In Table 3 we evaluate our Sample and Scale in these setups with a privacy distribution corresponding to the first setup of Table 2 with 5, 7.5, 10 for the first three rows, and 1, 3 for the last one, and DP-SGD baseline with æ = or = 1, respectively. We tuned hyperparameters with standard DP-SGD and applied the best resulting hyperparameters to Sample and Scale. All results are averaged over 9 trials and we report average accuracy and standard deviation. Over all tasks, architectures, and modalities, IDP-SGD outperforms standard DP-SGD.
4.3 Privacy Assessment via Membership Inference
To evaluate the impact of our IDP-SGD on the privacy of individual data points, we perform membership inference based on the LiRA attack (8) For the reader's convenience, we include
In principle, with our methods, each data point could have its own privacy budget. httpps:/opacuss wwt.tatooras//bss.loooooo 1stm_ www._aaaeeeaaaaa nww._classifierr

---

Table 3: Evaluating IDP-SGD with other architectures, tasks, modalities, and to fine-tuning.
Architecture
Dataset
Sctup
Modality, Task
DP:SGD
Sample
Scale
BERT ResNet18 Embedding Model
SNLI CIFARIO IMDB
Fine-Tune
Natural langunge inference
75.91 0.23
76. 16.11
76.5::00.17 48.77:1:0.73
Train
Image classification
47.52:40.84
48.53-+0.69
Trin
Text classication
72.69-10.27
73.27:10.3
73.34:10.11
Character level RNN
Sumnames
Train
Text (name) clussification
50.86.10.78
65.56.10.96
66.0.1
description of the attack in Appendix B.5 We experiment with CIFARIO and train 512 shadow models and the target model using Sample on different subsets of 25,000 training data points each. Results for Scale can be found in Appendix [D.3] The privacy budgets set to 10 and 20 and evenly assigned to the shadow models' training data, resulting in 12,500 training data points per privacy budget. Our target model achieves train accuracy of 68.46% on its 25,000 member data points, and a test accuracy of 64.89% on its 25,000 non-member data points. The lower accuracy in comparison to Carlini et al. [8) who achieved 100% and 92% train and test accuracy respectively, results from us introducing DP to the training of the shadow models. Learning with DP known to small datasets.
reduce model performance, particularly
We depict the membership inference risk of the target model's training data per pri- vacy budget (privacy group) in Figure 2 The dotted blue line represents the ROC curve for LiRA over all CIFAR10 data points and shows that overall, as expected by its definition, DP protects the training data well against membership inference attacks. However, when inspecting the ROC curve sep- arately for the privacy groups
10 and & = 20), we observe a significant difference be- tween them. The privacy group with stronger privacy guarantees (e = 10) is protected better (AUC= 0.537) than the group with the higher privacy budget (s 20, AUC- 0.581). To evalu- whether the difference in the LiRA-likelhoodd scores between the two privacy groups is statis- tically significant, we perform a Student t-test on the likelihood score distributions over the data points with privacy budget ε points with privacy budget = 20. We report 2.54 with p = 0.01 < 0.05, and hence a sta- tistically significant difference between the two ship Inference Success.
[10, AUC 1.0
== 10, = 0.537 E A0
True Positive Rate
0.6
True
0.4
0.2
vs. the data
0.0
0.2
0.4
0.8
0.8
1.0
False Positive Rate
Figure 2: Per-Privacy Group LiRA
8
Member-
groups exists.
Experimental results for additional target models, further comparison between IDP-SGD and DPSGD, and detailed per-target model statisties are presented in Figure Figure I0 and Table 12 Ap- pendix D33 The results highlight that the individual privacy assignment of our IDP-SGD has practical impact and indeed protects data points with different levels of privacy different degrees. Additionally, the privacy risk for data points with E = 10 when training with IDP-SGD is lower than when training purely on them with standard DP-SGD. This privacy gain does not come at large expense of points with E = 20 whose privacy risk remains roughly the same(see Figure 10)
5 Comparison to Other Methods
Comparison to Individualized PATE. We compare against Individualized PATE 6 (IPATE), the only other work aiming at enabling individual privacy assignment in ML. See details of their method and the general PATE framework in Appendix B.3] For the comparison, report the student model accuracies of IPATE for the MNIST, SVHN, and CIFAR1O datasets. The results are presented in Table 13 in Appendix D.4 We observe that IDP-SGD constantly outperforms IPATE with the same privacy budget assignment.
Synergy between Individualized Privacy Accounting and Assignment. We show that we can leverage synergies between our individualized privacy assignment and the orthogonal line of work on privacy accounting (see Section 2.2 Therefore, we extend the approach by Feldman and Zrnie [I] with our method of assigning each data point its individual privacy budget through an individualized

---

Table 4: Comparison between standard DP-SGD, individual accounting, and individual ac- counting-assignment. We present the test accuracy (%) on MNIST for training with = 0.3 for standard DP-SGD and individual accounting [1I, while setting 0.30 and 0.31 for half of the points each when using individual accounting and assignment. Combining individualized privacy accounting and assignment further increases utility.
VANILLA DP-SGD
INDIVIDUAL ACCOUNTING
INDIVIDUAL ACCOUNTING
AND ASSIGNMENT 94.16 + 0.23
93.29 + 0.49 93.64 T 0.46
Renyi-filter. This filter causes the point to be excluded from training once its individually assigned privacy budget is exhausted. Note that in contrast, in the approach by Feldman and Zrnie [[I], all data points obtain the same privacy budget and data points are excluded from training once they reach this budget. For more details, see the description in Appendix D.5
To assess the performance of pure individual accounting, we assign the same privacy budget of 0.3 to all 60,000 training data points in MNIST, following the experimental setup by Feldman and Zrnic [1]. We then enable the individual privacy assignment and change the privacy values to E = 0.3 for the first half of the points and ε = 0.31 for the second half of data points. We empirically observe that in this low-s regime, the small change of s for half the data points is enough to cause significant improvement in the accuracy of the final model, as show in Table 4 Note that the differences in accuracy reported for standard DP-SGD on MNIST between Table and Table differ. This is because of the different privacy budgets (a = 0.3 vs. E = 1.0). Additionally, in Figure 3 we present the number of active data points, i.e., data points that did not yet exhaust their individual privacy budget, over training. We find that many data points are used longer during training with individualized assignment compared to using only individualized accounting. This is because the data points with a higher privacy budget exhaust their budget later and, thus, can be used longer. We are only able to run this experiment on MNIST as done in Feldman and Zrnic [ since the method based on full-batch gradient descent: gradients are computed over the whole dataset at once, which limits its applicability to large and high-dimensional datasets.
(a) Accounting
026 633, 6 16-2
514s
(h) ing+Assignment
Account-
Figure 3: Number
of Active Data Points Over Training.
Conclusion and Future Work
Prior work on ML with DP guarantees assigns uniform privacy budget over the entire dataset. This approach fails to capture that different individuals have different expectations towards privacy and also decreases the model's ability to learn from the data. To overcome the limitations of uniform privacy budget and to implement individual users' privacy preferences. we propose two modifications the popular DP-SGD algorithm. Our Sample and Scale mechanisms adapt training to yield individual per-data point privacy guarantees and boost utility of the trained models. For future work, we believe that our individualized privacy assignment should be closely integrated with a form of practical individual privacy accounting. This could enable us to obtain a fine-grained notion of individualized privacy guarantees that tightly meet the users' expectations.
Acknowledgements
We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: this includes CIFAR through the Canada CIFAR AI Chair and NSERC through the Discovery Grant. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. As team member of the Center for Trustworthy Artificial Intelligence (CTAI), Christopher Mühl was funded by the German Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection. We would also like to thank CleverHans lab group members for their feedback.
10

---

References
[1] Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Mohammad Alaggan, Sébastien Gambs, and Anne-Marie Kermarrec. Heterogeneous differential privacy, arXiv preprint (TEi::5504.09990 2015.
[3] Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis test- ing interpretations and renyi differential privacy. In International Conference on Artificial Intelligence and Statistics, pages 2496-25000 PMLR, 2020.
[4] Amos Beimel, Shiva Prasad Kasiviswanathan, and Kobbi Nissim. Bounds on the sample complexity for private learning and private data release. In Theory of Cryptography Conference, pages 437-454. Springer, 2010.
[5] Bettina Berendt, Oliver Günther, and Sarah Spiekermann. Privacy in e-commerce: Stated preferences vs. actual behavior. Communications of the ACM, 48(4):001-106, 2005.
[6] Franziska Boenisch, Christopher Mühl, Roy Rinberg, Jannis Ihrig, and Adam Dziedzic. Indi- vidualized PATE: Differentially Private Machine Learning with Individual Privacy Guarantees, 2022. URL bittps///ariizs httpaaawww....raaaaaa
10517
[7] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:/50..0533 2015.
[8] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1519-1519 IEEE Computer Society, 2022.
[9] Rachel Cummings, Gabriel Kaptchuk, and Elissa M Redmiles. need better description": An investigation into user expectations for differential privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3037-3052, 2021.
[10] Cynthia Dwork. Differential privacy: survey of results. In International conference on theory
and applications of models of computation, pages 1-19. Springer, 2008
[11] Vitaly Feldman and Tijana Zrnic Individual privacy accounting via a renyi filter. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[12] Daniel Franzen, Saskia Nuñez von Voigt, Peter Sörries, Florian Tschorsch, and Claudia Müller- Birn. am private and if so, how many?" communicating privacy guarantees of differential privacy with risk communication formats. arXiv preprint arXiv:2208. J0820, 2022.
[13] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
[14] Anna C Gilbert and Audra McMillan. Property testing for differential privacy. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 249-258. IEEE, 2018
[15] Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv preprins arXiv:2206.07755. 2022.
[16] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Carlos Jensen, Colin Potts, and Christian Jensen. Privacy practices of internet users: Self-reports versus observed behavior. International Journal of Human- Human-Computer Studies, 63(1-2):203-227, 2005.
11

---

[18] James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Differentially private bagging: Improved utility and cheaper privacy than subsample- babsample-and-ageee aggrregate In Advances in Neural Information Processing Systems, pages pp. 4323-4333 2019.
[19] Zach Jorgensen, Ting Yu, and Graham Cormode. Conservative or liberal? personalized differential privacy, In 2015 IEEE 31St international conference on data engineering, pages 1023-1034 1034. IEEE, 2015.
[20] Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning, pages 5213-5225. PMLR, 2021.
[21] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLI, pages 4171-4186, 2019.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Yann LeCun, Corinna Cortes, and Burges. MNIST handwritten digit database. 2010. URL
httpps//scholar.ce googla..
[24] Haoran Li. Li Xiong. Zhanglong Ji, and Xiaoqian Jiang. Partitioning- based mechanisms under personalized differential privacy. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 615-627. Springer, 2017.
[25] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142- 150, Portland, Oregon, USA, June 1. Association for Computational Linguistics. URL hctp:///ww.aclleeb..o http://wwr wwwwwb.wwbb......... wwg.w/wbb.ggg............ org/ http//annoooooo1--/000 1015
[26] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-efficien learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[27] Ily Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263 2253-275.755 IEEE, 2017.
[28] Iyya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. arXiv preprint 1908. 10530, 2019.
[29] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[30] Ben Niu, Yahong Chen, Boyang Wang, Jin Cao, and Fenghua Li. Utility-aware exponential mechanism for personalized differential privacy. In 2020 IEEE Wireless Communications and Networking Conference (WCNC), pages 1-6. IEEE, 2020.
[31] Ben NIu, Yahong Chen, Boyang Wang, Zhibo Wang. Fenghua Li, and Jin Cao. Adapdp: Adaptive personalized differential privacy. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications, pages 1-10. IEEE, 2021.
[32] Opacus. Opacus get noise multaeeeimippppeefnfttiooonon
Ly#L23
[33] Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint 1101v:1610 2016.
[34] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations, 2018.
12

---

[35] Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. {Updates- Leak]: Data inference and reconstruction attacks in online learning. 29th USENIX security symposium (USENIX Security 20), pages 1291-1308, 2020.
[36] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[37] Peter Sörries, Claudia Müller-Bing Bim,, Katrin Glinka, Franziska Boenisch, Marian Margraf, Sabine Sayegh-Jodehl, and Matthias Rose. Privacy needs reflection: Conceptional design rationales for privacy-preserving explanation user interfaces. Mensch und Computer 2021-Wons 1e12-Wossoopbund. 2021.
[38] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). International Conference on Learning Representations, 2020.
[39] Florian Tramer, Andreas Terzis, Thomas Steinke, Shuang Song. Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing, arXiv preprint arXiv:2202. 12219, 2022.
[40] Stacey Truex, Ling Liu, Ka-Ho Chow. Mehmet Emre Gursoy, and Wenqi Wei. Ldp-fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking, pages 61-66, 2020.
[41] Aiping Xiong, Tianhao Wang, Ninghui Li, and Somesh Jha. Towards effective differential privacy communication for users' data sharing decision and comprehension. In 2020 IEEE Symposium on Security and Privacy (SP), pages 392-410. IEEE, 2020.
[42] Jiayuan Ye and Reza Shokri. Differentially private learning needs hidden state (or much faster convergence). In Advances Neural Information Processing Systems, 2022.
[43] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282 IEEE, 2018.
[44] Da Yu, Gautam Kamath, Janardhan Kulkarni, Tie- Liu, Jian Yin, and Huishuai Zhang. Individual privacy accounting for differentially private stochastic gradient descent. arXiv preprint arXiv:2206.02617, 2022.
[45] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative inder-invvrrion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 253-261, 2020.
13

---

A Broader Impacts and Ethical Considerations of this Work
The specification of an adequate privacy level is challenging since does not only depend on the domain but also on the dataset itself. This challenge persists when assigning individual privacy levels over a dataset. Especially in sensitive domains, one needs to make sure that the assignment of individual privacy guarantees does not pose an additional risk to the individuals whose data is processed. Therefore, one needs to first make sure that assigning individual privacy guarantees is not abused. This can occur when the entity training the model violates underlying individuals' privacy by nudging them into choosing inadequate privacy levels. Second, one needs to prevent individuals from giving up their privacy due to lack of understanding do so, IDP-SGD should not be used as a stand-alone method. Instead, following the example of [6], we suggest incorporating it into a whole process that involves communicating the functioning of DP and the risks associated with a decision to the individuals, analyzing their preferences, and supporting them in an informed decision-making process. There exist a growing body of work on the communication of DP methodology and associated risks to the public [1 9 12]. The most recent finding 12 suggests that medical risk communication formats, such as percentages or frequencies, are best applicable to inform about privacy risks. For the identification of privacy preferences, existing frameworks, such as [37] can be applied. Finally, we suggest giving individuals the choice between categorical privacy levels (e.g., high, medium, low) while putting regulatory entity, such as an ethics committee in charge of specifying concrete numeric values of e. This mitigates the disadvantages of the general limited interpretability of that exist independently of individualization.
wrong perception of their risk. To
A.1 Limitations
In this work, we consider a setup where individuals indicate their privacy preferences. We acknowl- edge that not all individuals are aware of their own privacy preferences and might, therefore, provide inaccurate indications. Therefore, our framework should be always deployed in a protected setup as described above. While we provide theoretical privacy guarantees in the framework of differential privacy, we assess the practical impact of these guarantees on the individuals solely through mem- bership inference attacks. Future work should investigate the practical (disparate per privacy-group) impact on other known privacy risks, such as data reconstruction. Due to the lack of sensitive-real world datasets that, in addition to individuals' data, also capture their privacy preferences,
evaluate
our methods on standard benchmark datasets. We. therefore, have to simulate a privacy preference distribution on this data according to the distribution known from society 17 5
A.2 Confidentiality of the Privacy Budgets
In some setups, an individual's choice of privacy budget could reflect some sensitive information. Imagine a medical context with a dataset that consists of individuals that do and individuals that do not have cancer. The latter ones might be more inclined towards choosing higher privacy protection to hide their condition. If an attacker with access to the trained model was able to deduce the individual's privacy budget, they might, in turn, be able to draw conclusions on the individual's ''s medical state.
Yet, theoretical results from ML, e.g., 14 have shown that one cannot currently perform sample- efficient black-box audits to determine the privacy- of a trained model. Frameworks to estimate privacy of trained model, such as (39 rely on training 1000-100,000 shadow models to get an estimate of the full model's privacy guarantee. In our IDP-SGD setup, the model does not have one single e, but one Sp per group, aggravating the impracticability of auditing guarantees and limiting the applicability of existing frameworks. As result, the adversary cannot currently determine the different per-group privacy budgets of the model.
Our MIA experiments in Section 4.3 show a difference in privacy risks over entire privacy groups. First note that these experiments (Figure 2 were the most costly ones in our paper. We had train more than 1500 (3 times 512) shadow models to generate the results. Yet, there are two limitations: a) The results only show differences in the groups, but do not reveal the respective Ep values, b) they do not allow to perform per- per-point distinctions. This is because the distributions of privacy risks over the two groups still have a large overlap. Hence, given a single point's risks and the two distributions, one cannot predict without error to what group the point belongs. This error could be reduced when the privacy budgets between the groups differ significantly (0.1 vs. 100). Note however, that we suggest deploying the model in setup where individuals simply specify their privacy preferences
14

---

and the model builder or an ethics committee assign concrete epsilon values to these preferences. We observed that extreme differences between privacy groups' are less beneficial because they degrade overall model performance. Hence, in practice, these large differences are very unlikely. As result, will be very challenging for an adversary determine which privacy group a data point belongs to. Even if they manage to, they are unlikely to be able to learn the privacy budget of the group as described above.
To enhance confidentiality of privacy budgets, Alaggan et al. [2] implement a form of deniability for users' membership in certain privacy group by uniformly sampling the privacy values over groups in an overlapping way: the unconcerned sample from 1., the pragmatists from 0.5, 0.75, 1., and the fundamentalists from O, 0.5, 1, such that on average, the values will be 1., 0.75, and 0.5 within the groups (0 denotes the strongest, while is the weakest privacy guarantee). Such an approach can implemented into our IDP-SGD framework as well. The drawback that an individual who expresses strong privacy preferences (fundamentalists) might actually be assigned privacy value 1 by randomness, which causes the same privacy leakage for an unconcerned individual. Depending on the individuals' preferences and the application, when protecting privacy group membership is more important than protecting the actual data, this approach might still be valid and this approach could be integrated with IDP-SGD, as well.
B Extended Background
B.1 Differential Privacy
Differential Privacy (DP) 10 provides a theoretical upper bound on the influence that a single data point can have the outcome of analysis over the whole dataset.
The most commonly applied instantiation of DP is (e, 6D-DP which is defined as follows:
Definition B.1. Let D, D' C DD be two neighboring datasets, i.e., datasets that differ in exactly one data point. Let further M DD' + R. be a mechanism that processes an arbitrary number of data points. M satisfies (E, -DP if for all datasets D ~ D', and for all result events R C R
[M(D) R S e (MM())) E R| 68
(4)
In the definition, the privacy level is specified by E E R4, while E [0, 1] offers a relaxation, i.e., a small probability of violating the guarantees.
B.2 Differential Privacy Accounting in Machine Learning
Privacy accounting in DP-SGD is most commonly implemented by the moments accountant which keeps track of a bound on the moments of the privacy loss random variable at outcome R, defined by
Pr[M(auxx, D) E R]
c(R; M, aux, D') log Pr[M (aux, D') E R
(5)
for D, D', .M and R as above, and an auxiliary input aux.
B.3
PATE Algorithm
The Private Aggregation of Teacher Ensembles (PATE) algorithm (33 represents alternative to DP- SGD for training ML models with privacy guarantees. This ensemble- algorithm implements privacy guarantees through knowledge transfer from the ensemble separate student model. More concretely, in PATE, the private training data is split into non-overlapping subsets and distributed among several teacher models. Once each teacher trained on their own data subset, they perform privacy- preccy-pres knowledge transfer by jointly labeling an additional unlabeled public dataset. To implement DP guarantees, noise is added to the labeling process. On completion, an independent student model is trained on the public dataset using the generated labels, and thereby incorporating knowledge about the original training data without ever requiring access it.
B.4 Individualized PATE.
The individualized PATE variants by 6 are Upsample and Weight. Upsample duplicates data points and distributes them to different teachers in the PATE ensemble. Utility increase in this method
15

---

result from the availability of more training data points for the teachers. Since the sensitivity for duplicated data point increases (it can change the votes of all the teachers that are trained on its duplicates), privacy consumption of that data point is higher. Since each data point can be duplicated individually, this method allows for fine-graines individualization. allows for a per-teacher model privacy individualization. Data points with the same privacy budget are assigned to the same teacher model(s) and the impact of that teacher model's weight on the final vote is weighted according to its training data points' privacy preferences. Teachers that are trained on data points with low privacy requirements are weighted higher, increasing the privacy consumption of their training data.
contrast, the weight method
B.5 The LiRA Membership Inference Attack
The LiRA membership inference attack g proceeds in three steps to determine which data points from dataset DD = (.1,..., INN were used train a target model f: (1) First, multiple shadow models, similar to f. are trained different subsets of D. (2) Then, the mean and variance of two loss distributions . (Miin Gnn) and N [[out Tom) are estimated per data point z;. Both distributions calculated from the logits of I; at the target class the former that 2; is a member of, the latter one on shadow models that z; is not a member of. (3) Finally, the likelihood of a new data point r of class being member of the target model is calculated as
over the shadow models
A p(((),, ((
(() N (Pon:Ta)))
C Details on the Methods
C.1 Sample
Leveraging Higher Sampling Rates for Increased Utility. With higher sampling rates for certain data points, utility could, in principle be increased in several ways. (1) Larger sampling rates can be used to obtain higher mini-batch sizes B (while keeping the number of training iterations I constant). Line 3 in Algorithm shows that noise is added to the aggregate of all gradients. Hence, with larger mini-br batchs, the signal-to-noise ratio is higher, which can improve training. (2) Alternatively, the mini-batch size B can be kept constant while increasing the number of training iterations I. Longer training can increase model performance. However, these two approaches result in a change of core training hyperparamters (mini-batch size and number of iterations). As discuss in Section 3 changing training hyperparameters would require a separate fine-tuning, for example, to adapt the learning rate for larger mini-batches as in (1) or longer training in (2). Since the training parameters would change according to the privacy budgets encountered in the private training dataset, and the ratios of these budgets over the training data points, the hyperparameter-tuning would have to be repeated whenever the dataset is updated, individuals change their privacy preferences, or decide to withdraw their consent for leveraging their data for the ML model alltogether, yielding significant overheads. We, therefore, implement our Sample according to the third option (3), described in Section 3.3 which leverages higher sampling rates for improved utility by redueing the noise multiplier of the added noise This allows us to perform an apple to apple comparison between both our methods and the standard DP-SGD.
C.2 Scale
Deriving Noise Multiplier sscale. Given the desired clip norm found through hyperparameter tuning of the standard DP-SGD, we set the individual clip norms such that their weighted average yields $ Σp= |Gpl- Cp. So, we derive the sscale in the following way:
16

---

P
1
Gpl.c.
(6)
N
p=1
P
1 N
Gpl
scale
Op
P
1
Gpl
00 scals
(8)
N
Op
p=1
Gpl
scale
(9)
N
Op
the equality between the scale of added noise Oscale OpCp- In ( extract Terms that are independent of the privacy groups (c and @scale) before the summation.
From (6)
7
C.3 Algorithmic Details
We specify our used sub-routines used for determining a sample rate or noise multiplier based on given privacy parameters in Algorithm 4 and Algorithm 5 respectively.
Algorithm Subroutine getSampleRate.
Is the equivalent to Opacus' function eet_nois__multipiir 32] for deriving adequate sample rate for given paramteters.
Require: Target e, target 6, iterations L, noise multiplier a,, precision = 0.01
1: init Ehigh Ehigh f 0
2: init Glows thigh: flow +- 1e-9, Thigh +- 0.1
3: while Ehigh
do
4: Qhigh < 29high
5: Ehigh + 1. 20 + (approximate according to Equation (2), we suppress a for simplicity) )
6: end while
7:
while s Ehigh
do
8:
9 - (now + //2
9:
Enemp t 1 2q + (approximate according to Equation
we suppress a for simplicity)
10:
if Etemp then
11:
Qhigh + q
12:
Fhigh Exmp
13:
else
(low <- q
15:
end if
16:
end while
17:
Output fhigh
D Additional Empirical Evaluation
We report the hyperparameters found for our individualized methods in Table The training and standard DP-SGD hyperparameters are specified in Table 5 The performance of our individualized methods when using the hyperparameters of standard DP-SGD presented Table Already when using these -ndividiivy tuned) hyperparameters, our methods yield significant performance increase in comparison standard DP-SGD. For MNIST, individual hyperparameter for our methods and individual setups did not yield significant improvements, therefore the results presented in Table and Taee identical for MNIST.
Computing Resources. The implementation of our methods does not increase computation time over the standard implementation of DP-SGD apart from the derivation of the privacy parameters that is performed once at the beginning of training. Hence, to run all experiments around methods and
17

---

Algorithm 5: Subroutine getNoise. Implements Opacus' function get_noise mielippiemllliili [32]
Require: Target e, target 6, iterations 7, sample rate q. precision 7 0.01
1: init highh bhigh +- XQ
2: init Flow; Thigh Flow f- hhihh f- 10
3: while Shigh > E do
4: Thigh high
5: Fhigh I 2q 1
(approximate according to Equation (2
we suppress a for simplicity)
6: end while
7:
while : Shigh > do
8:
a (Tww hhigh) )/2
9:
Eremp 1 2q oz (approximate according to Equation 2 we suppress a for simplicity) )
10: if Etemp < e then
Thigh +
12:
Shigh f tempp
else
14:
Jlo <
end if
16:
end while
Output Thigh
their evaluation, we required, total less than 16h GPU time on a standard GeForce RTX 2080 Ti. We ran the experiment combining individualized privacy assignment and accounting the same machines RTX 2080Ti and the total compute time also around 2h. To train all the shadow models for our membership inference attack and run inference them, required a total runtime of roughly 32 hours.
A100 GPU and
Table 5: DP-SGD Hyperparameters. LR: learning rate, B: expected mini-batch size, I: number of iterations, C: clip norm, a: noise multiplier in DP-SGD derived from the desired privacy budget 1. Default target 8 = 0.00001.
DATASET LR
C
MNIST
0.6
512
9375-~80 EPOCHS
0.2
3.42529
SVHN
0.2
1024
2146~30 EPOCHS
0.9
2.74658
CIFARIO
0.7 1024
1465~30 EPOCHS
0.4
3.29346
Table 6: DP-SGD Hyperparameters (Individually Tumed). LR: learning rate, B: expected mini-
batch size, number of iterations, C: clip norm, a. Default target 5 = 0.00001. Setup A is for privacy budgets = (1.0. 2.0, 3.0) and their respective distribution of 40%.%%43----44 Setup B is for the same privacy budgets but with their distributions 54%-337%-995
DATASET METHOD SETUP
LR
MNIST
SAMPLE
0.6
512
9375~80 EPOCHS
0.2
3,42529
SVHN
SAMPLE
A
0.2
1024
5723~80 EPOCHS
0.6
2.53261
CIFARIO
SAMPLE
A
0.2
1024
2929~-60 EPOCHS
1.0
2.65712
MNIST
SAMPLE
B
0.6
512
9375--80 EPOCHS
0.2
3.42529
SVHN
SAMPLE
0.1
1024
3577~50 EPOCHS
0.6
2.41421
CIFARIO
SAMPLE
0.1
1024
2929~-60 EPOCHS
1.8
3.14049
MNIST
SCALE
A
0.6
512
9375~80 EPOCHS
0.2
3.42529
SVHN
SCALE
A
0.1
1024
3577~50 EPOCHS
2.0
2.09719
CIFARIO
SCALE
0.2
1024
3418~70 EPOCHS
2.88335
MNIST
SCALE
B
0.6
512
9375~80 EPOCHS
0.2
3.42529
SVHN
SCALE
B
0.1
1024
3577~50 EPOCHS
1.6
2,45703
CIFARIO
SCALE
B
0.1
1024
2929~60 EPOCHS
1.8
3.17792
We present the individualized privacy parameters identified for our methods in Table
18

---

Table 7: Individualization Parameters Computed by our Methods for Table 8 We report the individualized privacy parameters identified for our Scale and Sample by Algorithm and Algorithm 2 respectively. The parameters are obtained on the MNIST, SVHN, and CIFARIO datasets when using the privacy budget distributions of Table with & {1.0, 2.0, 3.0)
DP-SGD
SCALE
SAMPLE
DAIASET
SETUr
(d1..... 4p)
(e)
ep)
(91..... 9P)
321--2%
0.008
12.139, 3.300 10021
10.129 0:216, 0.2741
2.024
0.009, 0.013]
57% 37% 9%
3.425
0.2
0.008
2.418
(2.189, 1,300 1,0321
10.148, 0.248. 0.315)
2.376
10,005, 0.011
SVIN
34--779-8%
2.747
0.9
0.014
1.896
12.747,
1.2144
[0.561, 0.970, 1,270)
1.667
[0.008,
0.015,
549-17%-99
0.014
2,180
(2,747,
1.214]
1.473]
CIFAR1O
374-13%-25%
3.293
0.4 0.020
2.241
[3.294, 1.88, 1,399)
10:244, 0.430, 0.5741
1.963
[0.012, 0.022, 0.031]
(3,294, 1886, 1,3991
[0.285, 0.502, 0.6731
2.300
[0.014, 0.026, 0.0371
Table 8: Model Test Accuracy after training with Standard DP-SGD our Individualized DP-SGD using Sample or Scale. D is the distribution of privacy groups (percentages) and the privacy budget for given group. The percentages of the three privacy groups are chosen according to Alaggan et al. [2] (first setup) and [30] (second setup). We used the hyperparameters found for and report the standard deviation over 10 trials.
standard DP-SGD, see Table and
SETUP
DP-SGD
SAMPLE
SCALE
D
244-439--7%
96.75
97.81
97.78
10000000
0015
0.09
0.08
MNIST
D
54%-37%--9
96.75
97.6
97.54
1.-20.300
0.15
0.11
0.09
D
43%-439--28
83.26
84.56
84.48
1.0-2.0-.0
#0.31
10.25
20.25
SVHN
54%-379%-9%
83.26
84.32
84.31
1.0-2.03.00
#0.31
+0.31
#0.26
D
34% 41%-433-22
52.77
$4.89
54.92
CIFAR10
1.0-2.0-.00
0.65
0.55
D
54%-37%%99
$2.77
54.88
$5.00
1.0.0.30..
1. 0.65
1. 0.45
0.65
D.1 Privacy Consumption of our Methods
We track privacy consumption of our methods over the course of training in Figure The figure highlights the good calibration of our methods which causes all privacy groups to exhaust their budget after the pre-specified number of training iterations.
3.-
fightr (1.4)
Fidter gros
kwi grop
2.5
2.5
2.0
2.0
3-2
1,14
0.3-
68
Presive (pecc)h............. 177 Anagiio
(a) Sample
(b) Scale
Figure Individual Privacy Costs on MNIST for E (1,2,3) with Distribution (54%, 37%, 9%).
D.2 General Applicability of our Methods
We showcase the practical impact of individual privacy assignments on individual utility and demon- strate how methods extend to many privacy groups and privacy budget distributions.
19

---

Table 100 privacy groups on MNIST. We randomly assign MNIST training data points to one of the 100 privacy groups with respective Ep sampled uniformly at random from [1. 6.. We compare against the DP-SGD baseline with & = and observe that our methods converge well and outperform the baseline.
DP-SGD
Sample
Scale
Test Acc. (%) 9..55 98.17:0.18
98.21
Practical Impact. We run experiments on the CIFAR10 dataset where we assign higher or lower privacy budgets to one of the 10 classes. We select all data points from the class (which we can think of, for example, as an underprivileged group) as one privacy group and assign to it either higher
(E ==3) 3), the same (E = 2), or lower (E = 1) privacy budgets in comparison to all other data points from the other classes (s 2). Table 10 shows that the choice of privacy budget for single group also impacts the other groups. We observe that by only changing the privacy budget for the selected group (in this case for class 0), we can flip its performance (its accuracy from being higher to being lower) in comparison to the accuracy of the other group (consisting of remaining classes). In the example of class O, the accuracy is 67.76% when assigned high privacy budget (E 3). which is a higher accuracy than for all other classes that have an average accuracy of around 53.41% % and assigned the privacy budget
2. Then, by modifying only the privacy budget of class 0 and by assigning to it the low privacy budget (c = 1), its accuracy drops to a mere 25.76% and below the accuracy of 56.58% for the remaining classes. We visualize the impact of the chosen privacy budget on utility over all classes (instead of only the class 0) in Figure 5 and Figure 6 for CIFARIO and MNIST, respectively.
More Privacy Groups. We present in Table I1 the test accuracy for ten privacy groups, correspond- ing to the ten classes of the CIFAR I0 dataset when each of the privacy groups obtains different privacy budget. We obtained these budgets by manually tuning them such that the accuracy gap be- tween the privacy groups is minimized. We also visualize the accuracy over training in Figure 7 The figure visualizes that our methods are able make all privacy groups converge to similar accuracies.
To evaluate IDP-SGD in the limit, we perform additional experiments with 100 privacy groups by randomly assigning MNIST training data points one of the 100 privacy groups with respective ep sampled uniformly at random from [1, 6]. Our results in Table indicate that IDP-SGD able to handle this large number of privacy groups without any degradation in terms of convergence: both our methods, Sample and Scale, continuously outperform the standard DP-SGD baseline with e = 1 for all data points.
D.3 Additional Results for MIA
Membership inference success for single target model of our Sample and Scale methods is shown in Figure 8 In Figure 9 we present the results over for 5 different target models for Sample. The figure highlights that over all target models, the two privacy groups' privacy risk is different: the group with higher protection e = 10 constantly has a lower AUC than the group with lower protection & 20. The test statistics over the different privacy groups' LiRA likelihood scores for all five target models are shown in Table [2]
In Figure 10 we also compare the MIA risk of our IDP-SGD with the risk of training with DP-SGD, using the LiRA attack. The results show that the privacy risk for data points with e = 10 when training with IDP-SGD is lower than when training with standard DP-SGD. This privacy gain does not come at large expense of points with = = 20 whose privacy risk remains roughly the same.
D.4 Comparison Individualized Privacy with IPATE
We present the comparison between our IDP-SGD and IPATE 6 in Table 13] refers to the student model accuracy. The results Table 13 are averaged over three experiments for IPATE and ten runs for IDP-SGD. Note that the accuracies we report for IPATE differ from the accuracy values reported by (6 in Table 1, since they report average voting accuracy (i.e., how correct are individual teacher model votes), whereas report the resulting student model accuracies, which
In PATE, accuracy
20

---

Table 10: Accuracy for Subgroups. We assess the accuracy of subgroups when their privacy budgets differ. We select a single class for a given group and assign either higher, the same, or lower privacy budgets comparison to groups with other classes. We change the privacy budgets only for bolded classes in a given experiment while all other classes have the same privacy budget (e = 2).
Privacy Budget
Classes
Higher (s 3)
Same (s = 2)
Lower (5 = 1)
67.76 + 2.14
55.24 = 1.98
25.76 + 2.52
1-9
53.41 2.2
54.72 = 2.49
56.58 + 2.29
80.84 1.2
72.79
46.09 = 3.91
0,2-9
51.65 + 2.28
52.77 = 2.53
54.59 + 2.23
2
51.31 + 3.53
36.59 = 3.33
9.53 + 1.41
0-1,3-9
54.26 1 2.84
56.79 = 2.34
58,87 1 2.18
52.88 + 2.6
32.64 = 1.91
6.67 + 1.09
0-2,4-9
54.62 = 2.42
57.23 2.5
58.75 2.42
56.99 = 1.87
40.06 = 2.88
9.44 + 1.68
0-3,5-9
54.41 + 2.21
56.41 = 2.39
58.18 + 2.02
64.11 + 2.27
51.86 = 2.21
15.04 + 2.31
0-4,6-9
53.28 + 2.16
55.1 2.46
57.54 1 2.49
73.54 + 2.36
65.8 4.25
40.6 + 4.18
0-5,7-9
52.05 1 2.21
53.55 2.23
56.06 1 2.33
68.22 + 1.17
61.15 = 1.99
41.08 + 2.75
0-6,8-9
52.5 + 2.43
54.07 = 2.49
56.01 + 2.76
77.39 1.24
68.53 = 2.34
37.42 + 3.37
0-7.9
51.51 + 2.54
53.25 2.45
55.28 + 2.37
72.82 + 1.48
63.08 + 1.9
32.79 + 2.73
1-8
52.05 1 2.32
53.85 1 2.5
56.06 1 2.52
corresponds to the final performance of the method. Note that IPATE does not apply the performance improvements suggested by Paperno al. (34] virtull adversarial training) MixMatch which could increase the student model's 's performance.
D.5 Integrating Privacy Accounting and Assignment
The main goal behind individualized accounting is to obtain tighter privacy analysis (recall our discussion Section 2.2). Instead of tracking a single privacy loss estimate across all data points, an individual privacy loss is kept track of for each data point. Feldman and Zmic [I]] defines new individual privacy filter, which drops data points that exceed the individualized privacy loss from further processing. However, the same privacy budget is assigned to each data point. The individual assignment of a privacy budget to each data point is a natural extension of individualized accounting and can be directly incorporated into this framework. Therefore, a given data point has its own privacy filter and dropped from the analysis once the filter indicates that the data point's privacy budget is exhausted.
D.6 Combining Sample and Scale
We run additional experiments to showcase the advantage of combining both our methods. The combination of Sample and Scale extends the number of hyperparameters and offers the potential to find better performing algorithms. It can implemented as follows: obtain individual sampling probabilities as in Sample. Once mini-batoe is sampled, use the respective data points' indices and
21

---

38
(Accura B Accuracy (%)
30
(Accuraty Bonlinn Accurac) (%)
20
Class of Inturest
Cloos of Interest
10
10
20
20
-30
-30
Class with Higher Budget
Class with Higher Budget
(a) Sample (higher budget)
(b) Scale (higher budget)
30
Beselime Accurecy) (%))
30
[Accurscy Bosoline Accurssy (89)
Class of Interest
Closs Interest
10
10
0
-20
(
-20
30
0
5
Class with Lower Budget
Class with Lower Budget
(c) Sample (lower budget)
(d) Scale (lower budget)
Figure 5: CIFAR10: Accuracy Changes for Subgroups. We assess how the test Accuracy of Class of interest changes in comparison to the Baseline Accuracy (standard DP-SGD with E = 2) when during training, assign lower (s 1) higher (s 3) privacy budget to data points from a class (shown on the x-axis). The diagonals show that by increasing a class' privacy budget (lower privacy), their utility increases, while it decreases with the decrease of privacy budget (higher privacy). Similar results for MNIST can be found in Figure
clip their gradients in Scale. We allow weight the different methods arbitrary fractions (e.g.. 50% Sample, 50% Scale), and derive the privacy parameters such that all privacy groups exhaust their respective budgets the end of the specified number of iterations. For practical evaluation, we use the MNIST dataset and the first privacy setup (0.34, 0.43, 0.23) with privacy budgets 2, and different weightings as indicated in Table [4
The baseline (first and last rows) are equivalent to Table 2 in the main paper, while the additional three rows between with the new results represent average test accuracy and the standard deviation over ten independent random runs over different weights assigned to the respective method.
Our results demonstrate clear progression from the lower performing Scale approach to the better performing Sample. This aligns with expectations based on our implementation, where we combine the methods based the parameters found for them sequentially.
Note that each of our methods has its advantages methods provides more options to balance (dvanaadsssssgs. For example, when some points' sample rates are extremely small, then the model might never see those points during training and the combined method could reduce this possibility while still improving over the Scale-only method.
disadvantages and the combination both
22

---

1.5
Bazaline Accuracy/ (%)
Beseline Accuracy (%)
1.0
1.0
Class of Intarest
Cloos of Interesz
-0.5
0.0
0.0
[Accurany
10
-15
Class with Higher Budget
Class with Higher Budget
(a) Sample higher budget
(b) Scale higher budget
1.5
Sotelrne Accurscyl (3)
1.5
[Accuracy Beseline Accuracyl (56
-10
110
Class of Interest
Closs Interest
0.5
0.5
0.0
0.0
-05
-0.5
-100
LAccursey
-100
5
Class with Lower Budget
Class with Lower Budget
(c) Sample lower budget
(d) Scale lower budget
Figure 6: MNIST: Accuracy Changes for Subgroups. We assess how the test accuracy of class changes (in comparison to standard DP-SGD with E 2) when we, during training, assign lower (e or higher (s = 3) privacy budget data points from this class. The diagonals show that by increasing a class' privacy budget (lower privacy), their utility increases, while it decreases with the decrease of privacy budget (higher privacy).
D.7 Examining Convergence of IDP-SGD
Individualization directly affects the objective function of the training procedure. To verify that our individualized methods maintain smooth convergence, we examine the loss history of one group of training data in the MNIST dataset and compare it with non-individualized DP-SGD.
We collected the loss values (and accuracy values) over the 4 following setups on the MNIST dataset and privacy setup (54%, 37%, 9%, {1,2,3) 3)):
1. loss value for privacy group with for the Sample method;
2. loss value for privacy group with 1 for the Scale method;
3. training with standard DP-SGD (c = 1) solely on the 54% of data points that have the privacy budget of and
4. training with the standard DP-SGD (s 1) on 100% of the MNIST dataset.
In the following table, we depict the training loss of the privacy group with during training:
epochs
Sample
Scale
DP-SGD (54% data)
DP-SGD (100% data)
20
0.145:0.0100
0.148:10.01
0.211+0.018 0.1874:0.014
0.162+:0.013 0.143+40.007
40
0.116+0.005
0.12140.006
60
0.109-10.05
0.115-10.066
0.18--100010
0.142-40.008
80
0.105
0.112--0.005
0.184-10.008
0.142-+0.006
23

---

Table 11: Per-class Individual Privacy Assignments. We manually optimize the per-class individual privacy budgets for Sample such that the model achieves the same accuracy over all classes. The resulting per-class privacy budgets yield the maximum gap A between the highest and lowest accuracy level of only 0.39% for Sample, and 0.88% for Scale. For the baseline (e = 3 for all classes) 2.03 is significantly higher, highlighting that our approach can successfully minimize the accuracy gap between different privacy groups. We run the experiment the MNIST dataset and report average per-class test-accuracies over three separate runs. See each privacy group's s test
accuracy over training in Figure]] 7
Class
2
3
4
5
6
7
8
9
A
Baseline (s
3)
98.95
99.06
98.39
98.09
97.93
98.47
98.16
98.12
97.78
97.03
2.03
Budgets
0.75
0.5
2.0
2.6
4.1
2.1
2.05
3.0
3.1
6.1
Sample
98.16
98.09
98.16
97.95
98.10
97.91
97.77
97.99
98.02
97.89
0.39
Scale
98.44
98.36
98.13
98.02
97.76
98.17
97.91
97.96
97.91
97.56
0.88
Table 12: Statistical differences between Privacy Groups. We conduct a student t-test to determine if the Lira likelihood scores for data points with privacy budget e = 10 differ from the ones of data points with E 20. All results with p 0.05 indicate statistically significant differences. Results for Sample.
Target Model
P
1
5.16
2.49e-07
2
2.41
0.016
3
1.84
0.066
4
4.03
5.52e-05
5
2.537
0.011
The loss values correspond to the following accuracy values:
epochs
Sample
Scale
DP-SGD (54% data)
DP-SGD (100% data)
20
96.11410.215
96.02410.257
94.071=0.460
95.692=1.315
40
97.024:10.095
96.906110110
95.078=0.333
96.358+0.191
60
97.296
97.12310.126
95.306=0.228
96.525 169 96.560++0.125
80
97.430++0.106
97.257+0.115
95.362=0.187
Our results indicate that the availability of more information from the data of different privacy groups in our Sample and Scale methods enables the model to learn better features. This leads to lower loss for the data points with privacy budget æ = 1 than if the model was trained on only those 54% of points. The effect of including more data (i.e., using 100% of the MNIST dataset), all with E = 1, is weaker, indicating that even the data with highest privacy requirement benefits substantially from individualized privacy in terms reduced loss and increased accuracy.
D.8 Runtime Analyses
We analyze the runtime of our methods in comparison standard DP-SGD. The runtime determined by two steps, the first one being the calculation of the individualized parameters of the methods which is executed exactly once before the training. The second one is the actual training time. For the calculation of individualized parameters, Table 16] indicates that the runtime depends on the number of privacy groups P specified for the implementation of Algorithm and Algorithm and the precision (see Algorithm(3). We observe linear growth in runtime w.r.t. the number of privacy groups. Note that, as stated above, this additional runtime occurs only once with IDP-SGD, namely before training starts. Also note that when we repeat experiments with the same privacy setups, the parameters do not need to derived every time, but can be set directly based on one initial computation. In terms of actual training time, Table [6 indicates that Sample and Scale do not
24

---

8
2.03
10.00
(a) Baseline
2.03
43.00
3.00
6.205
10.00
(b) Sample
Sranae
Scale
Figure 7: Per-class test accuracies over CIFARIO training with Per-Class Privacy Budgets. We manually tune the per-class privacy budgets for Sample to obtain the same per-class accuracy at the end of training, see Table II] Comparison with the Baseline (a) where all classes obtain = = highlight that Sample (b) and Scale (c) successfully reduce the accuracy gap between the different classes.
significantly increase the training time in comparison to standard DP-SGD which mainly due to their tight integration with the standard DP-SGD algorithm and the opacus library.
E
Alternative Baselines
Throughout this work, we compare our methods with Standard DP-SGD which cannot take into account different privacy requirements at the same time. Thus, we consider it to apply the highest
privacy protection required to all data points equally. Nonetheless, can think of two more ways to
25

---

[10, 20), AUC 0.560
AUC 0.537
AU, 0.581
True Positive Rate
0.6
0.4
0.0
0.4
0.8
False Positive Rate
(a) Sample
[10, 20C, AUC
E 20, 0.589
True Positive Rate
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
(b) Scale
Figure 8: True-Positive rate vs. False-Positive Rate of Lira Membership Inference Attacks Per Privacy-Budger We follow the same setup as in Figure (a) Sample and (b) Scale methods.
We show the single target model for both
use Standard DP-SGD on data having heterogeneous privacy requirements. Those approaches have different benefits and drawbacks and might perform better than our chosen baseline approach in some scenarios.
Exclude Lower Privacy Groups
Instead of applying the strongest privacy protection, the deciding ML expert could entirely exclude data of low privacy groups from training for loosening the restrictions on the remaining data points' influence on model updates. In some cases, would worth giving up the information and privacy budgets of those lower privacy groups to achieve utility improvements. This approach performs poorly important information wasted, e,g., most data of one class has the highest privacy requirement.
E.2
Learn Privacy Groups Separately
also possible to make use of privacy budgets, independent their diversity, although only using Standard DP-SGD. Namely, model can be trained on each privacy group separately one after another, whereby the corresponding lowest budget regarded for each group. drawback of this approach is that the model could forget its knowledge about previously learned privacy groups
26

---

<= [10, AUC= -0C= 0.511
0.546
2=20, = 0.593
True Positive Rate
0.0
0.2
0.4
0.5
0.8
1.0
False Positive Rate
Figure 9: True-Positive rate vs. Talse-Positive Rate of Lira Membership Inference Attacks Per Privacy-Budget. We follow the same setup as in Figure We the experiment for five different target models and aggregate the results with the error bars for the Sample method.
True Positive Rate
Positive Rate
0.4
6.4
H- [10.20] 0.590
-- [L0.20],AUC - 0.577
10,A0C 0.537
A0,ACC - 0.552
20,4UCC 03811
20,AUC 2LS8M
18GGPSCD) AUC 0.568
13GGPSGD) A 0.560
20LD5GD)
2-00PPSC) -UC
0.
62
64
64
3.
10
0.6
0. 641
6.6
3.
10
False Positive Rate
False Positive Rate
(a) Sample
(b) Scale
Figure 10: MIA on IDP-SGD vs. MIA on standard DP-SGD. In addition to the first three lines (blue, orange, and green) from Figure 9, also added the red and pink lines. The original three lines refer to model trained with IDP-SGD on two privacy groups (12,500 data points with 10 and 12.500 with 20). For the red line, trained 512 shadow models with standard DP-SGD and 25,000 CIFARIO data points that all have 10. For the pink line, we did the same with 25,000 data points and 20. Then, performed MIA according to the LiRA attack over the setups.
E.3 Empirical Comparison of Baselines
We empirically evaluate against these two additional baselines using the MNIST dataset. For baseline F.T we include all data points with a privacy budget 2 und use as the privacy budget for training. After hyperparameter tuning, the training on the remaining data points (43%+23% and 37%+9% of the total data) yields the accuracy reported in Table [7
we also did hyperparameter tuning and used the best noise multiplier of 2.5 for training. We trained the groups sequentially, always continuing training with the next group once the privacy budget of the previous groups was exhausted. We evaluated both starting with the privacy group that has loosest and strongest preferences (orders [3,2,1] and [1,2.3], respectively). Starting with the group that has strongest privacy requirements and ending on the group that has loosest privacy requirements yielded the best results which we report Table [7
For baseline E.2
In summary, we observe that our methods outperform the other baselines.
Alternative Individualization
We present the alternative ways individualizing privacy guarantees in DP-SGD that we considered in the design process of IDP-SGD and describe their drawbacks.
27

---

Table 13: Comparison between IDP-SGD and Individualized PATE (IPATE). We select the weighting mechanism from IPATE that performs better than the upsampling method. The Serup indicates the size (in %) of privacy groups. We present the accuracy (%) for training with {1.0, 2.0, privacy budgets, respectively to the order of the privacy groups.
DATASET
SETUP
BASELINE PATE
IPATE
SAMPLE
SCALE
34%
0.333
97.81
97.78
MNIST
91.17 1.25
54%-37%-99%
95.74 1 0,43
97.6
97.54
34%-43%-24
41.45 1.69
84.56
84.48
SVHN
22.46 1 5.19
54%-37%-
44.64 0.55
84.32
84.31
34%-43% 23%
33.20 0.94
54.89
54.92
CIFARIO
24.83 + 1.56
54%-37%-99%
35.59 + 0.73
54.88
55.00
Table 14: Combining Sample and Scale. We combine Sample and Scale according to the weights indicated in the table and perform experiments on the MNIST dataset. The respective percentages highlight what degree which method contributed. For example, 0% Sample and 100% Scale means that only Scale was executed (like in the results of the main paper in Table 2). while 50% Sample and 50% Scale expresses that both methods contributed equally. The results are averaged over ten independent runs.
Sample (weight in %)
Scale (weight in %)
Test Accuracy (in %)
0
100
97.78=00.08
25
75
97.75=#0.10
50
50
97.80:10.10
75
25
97.80=10.10
100
97.81++0.09
F1 Individual Per-Data Point Noise
Individualized privacy could, in principle also be obtained by adding different amounts of noise to different data points. Every of the P privacy group would have their individual {oi.....o Utility improvements would result from some data points requiring smaller amounts of added noise. Note however, that in DP-SGD, while clipping is performed on per-data point basis, noise addition is performed a per-mini-bateh basis (line 8 in Algorithm [T]. Hence, there are two possibilities to implement individual noise addition: either (i) by operating on mini-batch sizes of 1, or (ii) by implementing two-step sampling approach which first randomly samples privacy group for given training iteration and then applies the standard Poisson sampling to obtain the mini-batch consisting of data points from this group. While both approaches are conceptually correct, they exhibit significant drawbacks. Approach (i) first slows down training performance due to more operations requiring be carried out on individual data points, rather than a mini-batch. Second, due to the weak signal-to-noiss ratio when adding noise to individual gradients, model performance is likely to degrade. Finally, sampling cannot be performed with Poisson anymore since with Poisson sampling, it is not possible to pre-determine and specify exact mini-batch sizes, instead these depend on the outcome of the random sampling process. Approach (ii) could overcome the first two issues. However, the different groups sizes are still strictly smaller than the entire dataset and large parts of DP-SGD's degrading the tight privacy bounds obtained by privacy amplification through subsampling The privacy amplification through subsampling allows to scale down the noise by the factor B/N (with B being the expected mini batch size, N the total number of data points, and B < N) while still ensuring the same (privacy- utility trade-offs) of SGD. Hence, by using the P privacy groups of sizes [N1,... NP) with N, < N, the factors B B/N, & B/N for all i(1,..., P). This effect cancels out, in the worst
with (20). This privacy amplification is crucial to the practical performance
Note that there exist other, less popular approaches to implement DP in ML than the DP-SGD algorithm. such Differentially Private Follow-the-Ree Regularized-LLatte (DP-FTRL) which do not rely subsampling but instead obtain tighter privacy bounds from adding correlated noise over the training iterations. However, since DP-FTRL under- perfoors DP SGD for high- privacy regimes, and unfolds it advantages mainly in FL scenarios where the same data is only learned from once, or with a small number of epochs, we consider the approach outside of the scope of this work.
high-privacy
28

---

Table 15: Runtime in seconds obtain privacy parameters (clip norms or sample rates) for different numbers of groups P and precisions 7.. We report the time it takes for our methods to generate the privacy parameters for the first setup of Table of the original submission (34%-43% 23%).
=
0.01
0.0001
P
Sample
Scale
Sample
Scale
2
3.24
0.57
5.79
1.01
8
10.34
2.51
20.22
4.78
32
34.2
10.24
70.89
19.75
128
127.78
35.13
277.52
78.85
Table 16: Runtime in seconds for our methods. We report runtime when training on the CIFARIO dataset and the CNN architecture for the first privacy setup of Table 2 in the original submission (34%-43%-234)) with = 1, 1,2,3. Experiments are conducted on an A A100-GPU. Runtimes are averaged over three trials and average with standard deviation is reported.
DP-SGD
Sample
Scale
403.67 + 2.31 406.33+ 0.58 414.33+ 3.21
Table 17: Empirical evaluation against other baselines. We report the obtained test accuracy obtained with our two methods vs. two other baselines for individualized privacy on the MNIST dataset. Similar to Table 8 we use 1,2,3. 3. Both our Sample and Scale outperform the other baselines.
Setup
DP-SGD
E. Baseline
E.2 Baseline
Sample
Scale
34% 43%%233%%
96.75
97.6
97.4
97.81
97.78
54%%-77%-9%
96.75
97.1
97.3
97.6
97.54
case even inverts the privacy-utility benefits that should arise from assigning individual data points less noise based their privacy preference in our individualization.
F.2 Duplicating Data Points
When duplicating data points in the training dataset, similar to the Upsampling mechanism in IPATE (6) the DP-SGD algorithm itself does not need to be adapted. Instead, different privacy levels of differrnt data points stem from their individual number of replication within the training data. This approach offers a very fine- reee-graned control on individual privacy levels, since, in principle, each data point could be replicated different number of times. Utility gain would result from the larger training dataset. However, this type of upsampling opens the possibility for the same data point to be present multiple times in the mini- mini-batch used for training in given iteration. This stands in contrast to the original DP-SGD, where participation of each data point for training at a given iteration is determined by independent Bernoulli trial, and hence, data point can be either included once not at all in mini-batch. The possibility for data point to be included multiple times n inside the same mini-batch changes the sensitivity of the mechanism from e nic According to (27, when
noise is added according to a, mechanism with sensitivity nc is (a, 2(ne)² "0d RDDP. The quadratic influence of the sensitivity to privacy bound results in severe increase in the RDP making the approach suboptimal in terms of privacy-utility guarantees, Additionally, upsampling leads to an effective increase in data point's the sample-rate which further increases privacy costs.
G Additional Proofs
G.1
Additional Proofs for Individualized Privacy
Proof for Lemma 3.
29

---

Proof. First note that (E1. 6D-DP can be considered as a special case of (fE1. (2...E....,, E.)).))))))))).)).))))))). 8
where Vp E Elpp = E We can, hence apply Equation (3) and see that an M that satisfies (E1, 6)- DP has privacy guarantee of TP (M(D) R] S e e PMM()) E R +8. Given that by our definition vp E [2, Pl it holds that Ep > ε1, for p. holds that P (M(D) E R] S e PEL.PIMM (D') R <
e IP (M(')) E R] + 5. From this inequality, it follows that M also satisfies (f51, E2, EP).
IDP.
Proof for Lemma 3.2
Proof. Analogous to the previous proof, by our definitioon, it holds that Vp E P1 1] the Ep < Ep. From an M that satisfies (Ei,,,....,Eep) 6)II . it, therefore, holds that for all p the P (D) E R] S efr (I'M E R - ò < E .P (DD') E R. + 8. This inequality shows that M satisfies O)-DP.
G.2 Privacy Proofs for our Methods
Either of our methods can be considered as an individual SGM with different parameters from the point of view of each privacy group. This is because for each group, we have to examine neighboring datasets which differ in an arbitrary data point from that group. Our Sample method ensures an individual sample rate for all points of each group, while our Scale method applies an individual noise multiplier for all points of each group.
So, assume we have P different privacy groups. We base our proofs on the privacy guarantees of Sample and Scale on the observation that our methods can considered as simultaneously executed Subsampled Gaussian Mechanisms (SGM) [28] that update the same model. We first note that every data point's privacy preference stays stable over the course of training, therefore, conceptually, the privacy groups divide our training data in /' disjoint subsets. Hence, each of the P SGMs operates on a different partition of the training data, using different individual sample rates or clip norms. One can think of the simultaneous model updates by multiple SGMs with different privacy parameters as a similar concept to federated learning 26) with different local privacy guarantees [40]: each client holds disjoint subset of data, executes some local model update, implements privacy guarantees through local subsampling of the data, clipping, and noise addition (potentially with different parameters than other clients), and then shares the model updates with server that aggregates all updates and applies them to the shared model.
Within each training step, our methods' SGMs also have different privacy consumption due to their different sample rates and clip norms (sensitivity). Given that we keep the intermediate model states (checkpoints) private and only release the final model 42), the privacy costs in RDP over multiple training iterations add up linearly per SGM, such that, at the end each SGM yields E,, -RDP, p E [1...., P]. As consequence, the algorithm over all privacy groups yields (a, {ē1, E2,,E...... EP)) )- RDP which can then be converted to (fE1, Eee.,,,...., 8)- using standard conversion [27] as we will show in the following.
Theorem G.1. Our Sample mechanism satisfies (!11, (a.i,,,,),,......0.... ,P),6)-DP.
Proof. We prove the bound for any particular privacy group separately. Fix p E fl..... PJ. let D D be the training dataset, and select any F; E D that belongs to group Gp. We are interested
in comparing outcomes of mechanism M on D with its outcomes on D' = D \ {æ,} where M represents a particular model update of Sample. We get Gaussian mixtures
(D') Σ TLN (f , )
and
LCD
(D) = E TL (1 (1- (I.),,
(amplel)) T + ano (ILL Pammle 1r)
LCD
where (L)) is the clipped gradient of the current mini-batch L, T is its probability, xample > 0 is the noise scale, Idj is the identity matrix, and 0 < 4- < 1 is the individual sample rate of x; and every other point in Gp. Note that sample is actually multiplied by the clip norm Csmple > 0. Gaussian mechanisms are invariant regarding scaling of their sensitivity and noise scale, but instead depend on the relationship between sensitivity and noise scale, called the noise multiplier. Hence, we can ignore Csamplc and consider ff to have sensitivity 1.
30

---

Now we can see that the Gaussian mixtures of our Sample are equivalent to those corresponding the original SGM from Mironov al. [28] Thm. 4. when we parameterize it with sample rate Up and noise scale sample which are individual per group. Therefore, all RDP bounds of the original SGD apply, especially (a, E.EEp.. -RDP with Ep 24 -
in a particular parameter regime
(ofg. Thm. 11 from Mironov et al. (28). As a final step of the proof, we need to convert from Ex, -DDP guarantees (Ep: 6)-DP guarantees following Mironov [27] (see Section Note that our individual parameters have been selected before the start of training so that each group's ' privacy budget is exhausted the intended number of iterations (see Algorithm 2
Theorem G.2. Our Scale mechanism satisfies ((11,,2....,,87) (.).)).))))))) 6)-DP.
Proof. This proof analog to the proof of Theorem G. with the difference that we have global sample rate but individual noise multipliers op and clip norms Cp- Moreover, Algorithm 3 is used to configure parameters prior to training.
31


Reviews:
[{"review_id": "waKJozq0fp", "reviewer": "Reviewer_3QXJ", "summary": "This paper proposes two variants of Differentially Private Stochastic Gradient Descent (DP-SGD) to train machine learning models that satisfy approximate personalized differential privacy (PDP), following the definition of Jorgensen et al. [15]. In contrast to vanilla DP-SGD where all points in the training dataset are assigned a global, uniform privacy budget $\\varepsilon$, the authors consider a scenario where points are split into privacy groups $\\{\\mathcal{G}_1,\\ldots,\\mathcal{G}_P\\}$ and each group $\\mathcal{G}_p$ is assigned a different privacy budget $\\varepsilon_p$. This allows individuals who contribute training data to specify their privacy preferences, reflecting the variety of privacy attitudes among users observed in past surveys.\n\nTo achieve PDP the paper modifies DP-SGD in one of two ways:\n\n1. **Sample** method: adjusting the sampling rate used to construct mini-batches, so that points in groups with a higher privacy budget are sampled more often than points in groups with lower privacy budget (and thus, stringent privacy preference).\n\n2. **Scale** method: adjusting the noise multiplier to scale the variance of noise added in a per-point basis. Rather than doing this directly, which would be inefficient, the method achieves the same effect indirectly by scaling the clipping norm of per-point gradients.\n\nThese modifications are such that given target group privacy budgets and a number of training steps, the mini-batch noise multiplier in DP-SGD can be chosen so that the privacy budget for each group is exhausted at the end of training, maximizing utility.\n\nThe paper evaluates **Scale** and **Sample** on CNNs trained over MNIST, SVHN and CIFAR-10 using 3 privacy groups with budgets $\\varepsilon_p = p$ for $p \\in \\lbrace1,2,3\\rbrace$, distributing points into groups according to proportions used in prior work. This evaluation shows top-1 accuracy improvements ranging between 1-5% over vanilla DP-SGD with a uniform privacy budget $\\varepsilon_1 = 1$ (which guarantees PDP, but underutilizes the budget for the other two groups). The methods also compare favorably to Individualized PATE [5], which uses different techniques but achieves the same privacy guarantee.\n\nThe paper also demonstrates that the empirical evaluation of membership inference attacks against models trained with PDP using the above two methods shows the expected gap between groups with different privacy budgets. Finally, the authors discuss how to combine individual privacy assignment with individual privacy accounting as done by Feldman and Zrnic [10] for batch gradient descent.\n\nThe supplemental material provides implementations of **Scale** and **Sample** in the Opacus library as well as scripts to reproduce the results in the paper.", "questions": "1. **Confidentiality of privacy budgets**\nThe paper argues that IDP-SGD preserves the confidentiality of group privacy preferences, at least against parties that only interact with the final trained model. However, intuitively, models trained with a uniform privacy budget of $\\varepsilon=1$ and models trained on the same data but\nincreasing the privacy budget for a single group from $\\varepsilon=0.1$ to $\\varepsilon=100$ will likely perform noticeably different on that group. Can you provide a proof that IDP-SGD satisfies some form of confidentiality for privacy preferences that makes explicit the adversary model and assumptions?\n\n2. **LiRA AUC scores**\nCarlini et al. [6] report LiRA results on models trained with DP-SGD and $\\epsilon = 8$ (not far from one your choices of $\\epsilon = 10$), with a significantly lower AUC score of 0.503 compared to the score of 0.537 that you report. Even for $\\varepsilon > 5000$, their best result has a lower AUC score of 0.527. Could you explain this difference?", "limitations": "The authors adequately address societal impact but only some limitations.\n\nA paragraph at the end of the introduction discusses the need for openly communicating privacy risks and regulatory supervision. This points to Appendix A, which discusses broader impacts in the light of prior studies and some limitations: evaluation on standard benchmarks with simulated privacy preference distributions; practical impact measured via membership inference attacks only. I think that the limited nature of the evaluation on CNN models trained from scratch on visual tasks should also be brought up front, as the results might differ in other settings.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **First DP-SGD variants designed to achieve personalized differential privacy**\nThere is abundant literature for adapting differentially private mechanisms for general data analysis to achieve personalized differential privacy as well as for computing personalized differential privacy guarantees for training data points in ML models. While PDP accounting alone can be used to improve model utility modestly (see Feldman and Zrnic [10]), as far as I know the only other method that trains models with individual privacy assignments is Individualized PATE [5] which uses significantly different techniques.\n\n2. **Original combination of existing mechanisms**\nThe **Sample** mechanism is inspired by the \"Sampling\" mechanism of Jorgensen et al. [16], while **Scale** is inspired by the \"stretching\" mechanism of Alaggan et al. [2]. Combining these mechanisms with DP-SGD is relatively straightforward, but has not been done before.\n\n3. **Problem is well motivated and solution is contextualized relative to prior work**\nThe authors discuss related work up front, explain how their solution fits into the existing literature, and justify their contributions.\n\n4. **Complete algorithmic descriptions of proposed methods** The paper describes the adaptations of DP-SGD in sufficient detail, including pseudocode and justifications for design decisions.\n\n5. **Demonstrates utility gains compared to alternative solutions** The empirical evaluation on image classification tasks demonstrates a modest utility gain compared to alternative baselines.\n\n6. **Comprehensive supplemental material** The Appendix includes valuable additional discussions. The accompanying code is a significant extension to an existing library and demonstrates the practicality of the techniques.\n", "weaknesses": "1. **Limited empirical evaluation** The empirical evaluation is limited to convolutional architectures trained from scratch for image classification. It is unclear how the paper findings hold for other architectures, modalities, tasks, or practically relevant scenarios, such as fine-tuning.\n\n2. **Modest utility gains** The utility gains with respect to vanilla DP-SGD are modest (1-5%). The comparison to alternative baselines with simpler implementations in Appendix E shows even more modest gains. For instance, simply training sequentially on each group separately using DP-SGD achieves accuracies only 0.3-0.41% below the best of **Sample** and **Scale** (see Table 12). It is unclear to what extent these differences are statistically significant, whether comparable effort has been put in hyperparameter tuning, and whether mild modifications to this alternative baseline could outperform **Sample** and **Scale**.\n\n3. **Unsupported claims about the confidentiality of privacy preferences** The authors argue that privacy preferences (i.e., per-group privacy budgets) should be kept private and claim that their modifications of DP-SGD do not leak information about privacy preferences because untrusted parties only interact with the final trained model. This claim is unsupported by proof (compare this to Allagan et al. [2], which prove a similar guarantee explicitly) and is made with respect to a different adversary model than the privacy guarantees of training data in DP-SGD, which consider adversaries that observe noisy gradients released at every training iteration, not only the final model.\n\n\n## Minor comments\n\n- Given the number of references to numbered lines in Algorithm 3, it would be convenient to include the listing in the body of the paper.\n\n- l.76: \"[D]iffer in any one record\" is ambiguous. Consider making clear that you use the add/remove neighboring relation. That is, $D,D'$ are adjacent if $D = D' \\cup \\{x\\}$ for some record $x$, or vice versa.\n\n- l.79: \"DP bounds privacy leakage for any individual\". This is only true under the assumption that an individual contributes at most one record.\n\n- l.86: The original conversion from $(\\alpha, \\rho)$-RDP to $(\\varepsilon, \\delta)$-DP of Mironov [21] is suboptimal. Balle et al. [A, Theorem 21] provide a tighter conversion (which is the one Opacus uses).\n\n- Table 1: $\\sigma_{\\rm SAMPLE}$ should read $\\sigma_{\\rm sample}$.\n\n- Algorithm 1: Missing inputs: scaling factor $s_i$, target sample rate $q$.\n\n- Algorithm 1: In l.1, you could initialize $\\sigma$ using $\\textit{getNoise}$ as $\\sigma \\gets \\textit{getNoise}(\\varepsilon_1, \\delta, q, I)$.\n\n- Algorithm 1: In l.4 $G_p$ should read $\\mathcal{G}_p$.\n\n- l.190: \"our Sample [method]\"\n\n- l.209: While I have seen it being used before, I think that the notation $\\stackrel{!}{=}$ is far from being conventional.\n\n- l.215: \"[W]e still require the expected average sampling rate to remain $q$ to obtain constant mini-batch size $B$\". As you said just before, the mini-batch size isn't constant, $B$ is the **expected** size. Rather, you want the expected mini-batch size in IDP-SGD to be the same as in vanilla DP-SGD with sampling rate $q$.\n\n- l.225: There's an extra closing parenthesis.\n\n- l.232: DPSGD should read DP-SGD.\n\n- l.260: Use \\{$\\sigma_1,\\ldots,\\sigma_P$\\} rather than $\\sigma_p$-s.\n\n- l.313 (and subsequently): \"Lira\" should read \"LiRA\"\n\n- Numbers in bibliographic references are shifted between the paper and the full paper (with Appendix) provided as supplemental material.\n\n- Appendix, Algorithm 3: In line 11, $\\theta_T$ should be $\\theta_I$.\n\n- l.566: $\\sigma_{\\rm in}$, $\\sigma_{\\rm out}$ should read $\\sigma^2_{\\rm in}$, $\\sigma^2_{\\rm out}$, respectively.\n\n\n- l.576: \"Line 3 in Algorithm 3\" should be \"Line 8 in Algorithm 3\"\n\n- l.591-596: $G_p$ should read $\\mathcal{G}_p$.\n\n- l.691: \"Appendix E.3\" should read \"Table 12\"\n\n- Table 10: How can it be that you get a statistically significant negative $\\Delta$ for one of the models? That would reverse the relation between the two privacy groups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to express our gratitude to the reviewer for their thorough review and detailed feedback that goes beyond any expectation.\n\n> **Limited empirical evaluation**\n\nWe thank the reviewer for their suggestion to extend our experimental evaluation on other architectures, tasks, modalities, and fine-tuning. We ran additional experiments with IDP-SGD for:\n- fine-tuning a BERT transformer on the SNLI dataset for natural language inference,\n- training a ResNet 18 with IDP-SGD from scratch on the CIFAR10 dataset for image classification,\n- training a linear embedding model on the IMDB dataset for text classification,\n- and training a character-level RNN from scratch to classify surnames to languages.\n\nThe results are included in the general response.\nOver all the considered experiments, IDP-SGD outperforms standard DP-SGD which underlines that our findings on the performance improvements of IDP-SGD generalize over different setups. \n\n> **Modest utility gains**\n\nThe main goal of our work is to enable individually private training with different epsilon values. We compare the possible options and present the best performing method in the main body of the work. We performed thorough hyperparameter tuning over all our methods, including E.1 and E.2. Note that the hyperparameter tuning for E.1 and E.2 goes beyond standard parameters. For E.1, one also needs to tune over which group(s) to exclude, and for E.2 in what order to train on the different groups. We performed all these analyses and reported the best results.\n\nFrom a conceptual viewpoint, Scale and Sample outperform the other baselines when the data distribution between the different privacy groups differs significantly. E.1 which completely excludes whole groups from training will not be able to learn the high-privacy groups at all, while E.2 which trains privacy groups sequentially will suffer from catastrophic forgetting where the final model will perform worse on the groups first seen during training. In contrast, in Sample and Scale, all privacy groups contribute to the training over the entire training process.\n\nFinally, we would like to note that in DP-SGD research, a utility improvement of 1-2% is a significant contribution, especially in the low-privacy regime (eps=1,2,3) that we are operating in. See for example, the [[Unlocking high-accuracy differentially private image classification through scale]](https://arxiv.org/pdf/2204.13650v2.pdf)-paper that outperforms prior work by less than 2% at epsilon=3. Looking into Fig 1a) of the paper, we see that the improvement of the other prior baselines over each other is usually in the range of 1-2%.\n\n> **Minor comments**\n\nWe made space for Algorithm 3 in the main paper by merging Figure 3a) and 3b), clarified the DP definition, and fixed all the formatting suggestions pointed out by the reviewer. Additionally, we replaced the conversion from RDP to (eps,delta)-DP by the one proposed in Balle et al. Finally, we adapted Algorithm 1.\n\nRegarding the negative delta in Table 10, we are sorry for the confusion. The delta (effect size) in the table should be positive 4.03. We corrected the typo. \n\n> **Question 1: Confidentiality of privacy budgets**\n\nWe wrote a thorough explanation on the confidentiality of privacy budgets in our methods in the general response.\n\nRegarding confidentiality of privacy values in [2], we would like to mention that their application of HDP is within a distributed gossip-based semantic clustering protocol that operates in an iterative manner. In the protocol, pairs of users calculate the cosine similarity between their preferences (see page 11). The Stretching mechanism used for this relies on both users’ privacy preference vectors as parameters. This means, the users get to directly interact with each other’s privacy preference vectors–requiring privacy protection for these vectors. In contrast, in our setup, all privacy-preference based operations are already performed when the end user (potential adversary) gets to interact with the model where determination of privacy budgets is difficult and highly impractical (see argument 1. and 2. in the general response). \n\nWhen dealing with different privacy groups, [2] implements a form of deniability for users’ membership in a certain privacy group by uniformly sampling the privacy values over groups in an overlapping way: the unconcerned sample from {1.}, the pragmatists from {0.5, 0.75, 1.}, and the fundamentalists from {0, 0.5, 1}, such that on average, the values will be 1., 0.75, and 0.5 within the groups (0 denotes the strongest, while 1 is the weakest privacy guarantee). Such an approach can be implemented into our IDP-SGD framework as well. The drawback is that an individual who expresses strong privacy preferences (fundamentalists) might actually be assigned a privacy value 1 by randomness, which causes the same privacy leakage as for an unconcerned individual. Depending on the individuals’ preferences and the application, when protecting privacy group membership is more important than protecting the actual data, this approach might still be valid.\n\nWe added a discussion about adopting this approach into our paper as Section 3.5.\n\n> **Question 2: LiRA Scores**\n\nWe thank the reviewer for this insightful detailed question. The differences in our vs. their setup that can cause our higher AUC score are:\n1. We set eps=10 for half of the data and eps=20 to the other half, which is significantly higher than their eps=8.\n2. The accuracy of our models differs from theirs. Our target model achieves a train accuracy of 68.46% on its 25,000 member data points, and a test accuracy of 64.89% on its 25,000 non-member data points. They report 61.3% test accuracy (we could not find their train accuracy in the appendix). Differences in accuracy, especially differences in the train-test gap can cause differences in the MIA success rate.\n3. Their attack on CIFAR10 uses 256 shadow models while we use 512.\n"}, {"review_id": "ItaxvMsvEe", "reviewer": "Reviewer_7px6", "summary": "This paper proposed two variants of DP-SGD by manipulating the sampling rate and the gradient clipping bound for different groups to achieve the goal of having different privacy budgets for those groups and improving the overall performance of DP-SGD. The authors proved theoretical privacy guarantees for both variants. They also conducted an extensive experimental evaluation to showcase the advantage of their methods in many aspects.", "questions": "1. In Line 379, could you explain the meaning of 'batch gradient descent'? Is it different from SGD? If yes, maybe it can be replaced with the word 'full-batch gradient descent' or just 'gradient descent'.\n2. In Table 8, we can see that for the first two rows, the accuracy decreases for classes 1-9 with a higher privacy budget of class 0 compared with the results with the same privacy budget of class 0. The same phenomenon exists in other rows. Intuitively, the higher budget of other classes should provide more information for the classifier so that the accuracy for each group would increase. Could you provide more insight into why this intuition is incorrect under this setting?", "limitations": "1. The discussion of the incentive and consequence of people choosing weak privacy protection is not enough in the main text. I found the content in Section D.2 very interesting since it aligned with my intuition: the more privacy being sacrificed, the better utility could potentially be obtained. Also, the results in Table 8 show that having a group with a higher privacy budget will affect the utility of other groups, which could lead to problems like the Prisoner's dilemma where people in the dataset have to give a higher privacy budget in order to maintain their original utility given the fact that other people are willing to have a higher privacy budget.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors considered an important problem, having different privacy budgets for different groups of people to boost the overall utility, and proposed two algorithms to achieve the goal. The privacy guarantee of the algorithms is proved. Both algorithms are easy to implement and look novel although intuitive to me.\n2. The experiments in this paper are adequate for the evaluation of the proposed methods. The results are well-organized and visualized, which supports the validity and the advantage of the methods, and the details are provided enough for reproducibility. I understand and appreciate that the authors put most of the valuable experimental results in the appendix due to the page limitation.\n3. The writing and the organization of the content are good.", "weaknesses": "1. The composition theorem for Algorithms 1 and 2 is not proved or discussed in this paper. The proofs of Theorem G.1 and G.2 look okay to me although I think more explanations would make it more friendly to the general audience. However, I think there is still a need to show the composition theorem similar to the results in Feldman and Zrnic [9]. Moreover, if it is not much work for rewriting related parts of the paper, I think a more strict way to account for the privacy budget usage is by the Rényi Filter [9], i.e., individual RDP instead of RDP. \n2. Section 4.2 uses membership inference to empirically verify the privacy protection for the claimed DP guarantee, but the results in Figure 2 do not explicitly show the protection separately for each group ($\\varepsilon=10$ and $\\varepsilon=20$) compared with the DP guarantee. The authors may show the curves under two additional settings: vanilla DP-SGD with $\\varepsilon=10$, and vanilla DP-SGD with $\\varepsilon=20$. Then compare the two solid curves in Figure 2 with the two additional curves respectively.\n3. Both Algorithms 1 and 2 will potentially change the objective function of SGD (although DP-SGD with the gradient clipping has already changed it). There is no discussion on how specific choices of privacy budget distribution will affect the convergence and final performance of the optimization. This is important since it is related to the privacy-utility tradeoff. Please see my comment in *Limitations*.\n\n\nMinor weaknesses:\n1. Line 66. I think Table 7 is the same as Table 2 which is in the main text, not the appendix.\n2. Line 86. $\\log 1/\\delta$ -> $\\log(1/\\delta)$\n3. Line 98, equation (2). The inequality holds with many constraints on $\\alpha$. See Theorem 11 in Mironov et al. [21]\n4. Algorithm 1. The authors wrote: it is equivalent to Opacus' function get_noise_multiplier while Algorithm 2 is also equivalent to the same function. I think there is a typo.\n5. Algorithm 1, Line 5. I think this is more like a grid search which always decreases $\\sigma_{\\mathrm{sample}}$, while a binary search would be better.\n6. Line 209. The notation $\\stackrel{\\text{!}}{=}$ is not defined.\n7. Line 767 in Section G.2. $\\sum_{L\\subset D}$ -> $\\sum_{L\\subset D'}$\n---\nI have read the rebuttal which addressed my questions and the weakness concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions and provide our answer below:\n\n>**W1: Composition theorems**\n\nWe would like to point out that our algorithms have the exact same composition (within every privacy group individually) as the standard DP-SGD and do not require additional theorems on composition. In detail, our contribution is to propose a novel way of individual privacy **assignment**. Our privacy accounting follows the standard accounting also used within DP-SGD with the difference that accounting is performed within a privacy group separately. Feldman and Zrnic [9]’s contribution, in contrast, is to propose a novel individual privacy **accounting** mechanism which then requires a different composition. Their accounting differs from standard DPSGD accounting in the sense that it does not account for worst-case privacy leakage, but that it estimates the ‘actual’ privacy leakage. The composition from standard DP-SGD is, thereby, not directly applicable for them while it is for us.\n\nFinally, we would like to note that in Section 5, we show that our individualized privacy assignment can be integrated into the individualized privacy accounting by [9]. We do so by assigning individual privacy budgets to different data points and then rely on the individual privacy accounting by [9] (using their composition theorem).\n\n>**W2: MIA for two additional settings**\n\nWe thank the reviewer for this suggestion. We performed this experiment, trained a model with $\\varepsilon=10$ and another one with $\\varepsilon=20$ using standard DP-SGD on CIFAR10. Then, we performed a LiRA MIA attack with 512 shadow models (the same as in Figure 8a and 8b in the submission) and report the detailed results in the general response. The resulting AUCs can be summarized as follows:\n||All points ($\\varepsilon=10$ and $\\varepsilon=20$)|Points with $\\varepsilon=10$|Points with $\\varepsilon=20$|\n|-----------------------------|------------------------------------------------------|-------------------------------|------------------------------|\n|Sample|0.56|0.537|0.581|\n|Scale|0.571|0.552|0.589|\n|DP-SGD ($\\varepsilon=10$)|-|**0.568**|-|\n|DP-SGD ($\\varepsilon=20$)|-|-|**0.59**|\n \nOur key observations are:\n1. IDP-SGD reduces the privacy risk for the data points with $\\varepsilon=10$ (in standard DP-SGD, their AUC=0.568 which is higher than with Sample 0.537 or Scale 0.552).\n2. This privacy gain does not come at large expense of points with $\\varepsilon=20$ whose privacy risk remains roughly the same (AUC=0.59 in DP-SGD vs Sample 0.581 and Scale 0.589).\n\n> **W3: Objective function, convergence, and final performance**\n\nWe would like to point out that already DP-SGD itself (especially with the gradient clipping) does not offer rigorous convergence guarantees except in convex, Lipschitz, or Lipschitz-like regimes. For this reason, it is a common practice in work on DP-SGD to rely on empirical evaluation for the utility, see for example [Differentially private learning needs better features (or much more data)](https://arxiv.org/abs/2011.11660), [Unlocking high-accuracy differentially private image classification through scale](https://arxiv.org/pdf/2204.13650v2.pdf).\n\nFollowing this common practice, we experimentally evaluate a wide range of privacy distributions and different privacy parameters (e.g., two distributions with eps=1,2,3 in Table 2, a distribution with eps=10,20 in Section 4.2, another distribution with eps=1,2,3 in Table 8 and Figure 5+6, and a distribution with ten different epsilon in range between 0.5 and 6.1 in Table 9). \nIn the general response, we also provide a new experiment with 100 different privacy groups.\nOver all experiments, we increase the utility of DP-SGD and do not observe any convergence issues in our experimental results.\n\n> **Minor comments**\n\nWe thank the reviewer for carefully reading our paper. We removed Table 7 and line 66 now refers to Table 2. We also corrected the other points mentioned by the reviewer and explain the notation !=, which refers to “should be equal to” in the updated version of the paper.\n\n> **Q1: Batch gradient descent**\n\nWe indeed refer to full-batch gradient descent and adopted the reviewer’s suggestion on using this term within our work.\n\n> **Q2: Utility increase and decrease due to individualized privacy**\n\nWe thank the reviewer for their careful study of Table 8 and are happy to provide further insights into the observed phenomenon:\nWhat we observed in our experiments is that the group that uses a higher privacy budget (e.g. group class 0) provides more information to the model and makes the model learn the features of this particular group better (reducing the model’s performance on other groups’ features). We think that this phenomenon is illustrated very interestingly, for example, in Figure 6a). We observe that if class 9 gets the higher privacy budget (last column), the class that suffers the largest utility loss is class 4 (which is the most similar digit to the 9 and suffers from the model paying more attention to the features of the 9). This shows that when the privacy groups differ significantly in their distributions, by increasing the privacy budget of one group, especially this group will benefit from utility improvements.\n"}, {"review_id": "HjOw2nl89i", "reviewer": "Reviewer_Sruy", "summary": "This paper proposes extensions of the DP-SGD algorithm to support individualized differential privacy (called the IDP-SGD approach). Unlike traditional differential privacy, which imposes a single privacy budget epsilon to all data points, the data points may now have different privacy budgets. Two extensions are proposed. First, the Sample method adjust the sampling rates of data points depending on their privacy budgets. Second, the Scale method adjusts the individual clip norms to effectively add individualized noise. The noise itself cannot be directly individualized due to the DP-SGD algorithm behavior of adding per-batch noise. Experiments show that IDP-SGD outperforms DP-SGD and IPATE in terms of utility and can be complemented with privacy accounting techniques.", "questions": "Q1) What is the actual name of ({$\\epsilon_1$,...,$\\epsilon_n$}, $\\delta$)-DP and ($\\epsilon_p$, $\\delta$)-DP? Are they Individualized Privacy or something else? If they are Individualized Privacy, why did the authors change the term “Personalized DP” to “Individualized Privacy” even though you referred to the Personalized DP paper?\n\nQ2) How can getSampleRate function be equivalent to the function get_noise_multiplier in Opacus? getSampleRate seems to return sample rate with input argument noise multiplier, while get_noise_multiplier returns noise_multiplier sigma.\n\nQ3) The comparison between DP-SGD and the proposed algorithms is as expected because DP-SGD assumes the same privacy level over all data points. Are there any results related to computational cost?\n\nQ4) A detailed description of the properties of each Sample and Scale is lacking. Sample seems to be superior overall in performance. Is there any reason to introduce the Scale technique? Or, what are the advantages of the Scale method compared to the Sample method?\n\nQ5) What is Thm. 4 and Thm. 11 from Mironov et al. [21] in the proof of Theorem G.1.? I could only find Proposition and Lemma in the paper. Did you mean [22]?\n\nQ6) How can you simply prove Theorem G.1. and Theorem G.2. without proving some basic theorems or properties of newly designed privacy notion? I don't think previous DP techniques can be applied to individual privacy directly. The authors just directly use the conversion technique from RDP to ($\\epsilon$,$\\delta$)-DP of previous works as a technique for the individual privacy conversion even though the privacy notion is new.\n", "limitations": "The authors discuss limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Proposes novel extensions of DP-SGD to support individualized privacy.\n* The Sample and Scale methods are easy to deploy and have theoretical guarantees.\n* Experiments show that IDP-SGD outperforms baselines (DP-SGD and I-PATE) and can be used with privacy accounting.", "weaknesses": "W1) The core contributions of the paper are not contained in the main part of the paper. \n  * The theoretical analysis should be more specific because the proposed individual privacy notion with the extended $\\delta$ term is different from previous personalized DP and conventional DP. In the proof of Theorem G.1., the original SGM seems to be about conventional DP and RDP, but the author simply applied it to individual privacy without specific proof about its properties. \n  \n  * The paper shows the algorithm that finds parameters, but it does not provide a way to adapt the parameter to the original DP-SGD. Because DP-SGD uses a single sample rate and a single clip norm, adapting multiple sampling rate and clipping norm is also a new mechanism. It seems that a more specific explanation is needed in terms of implementation. For example, the original DP-SGD assumes Poisson sampling in principle, but approximates it due to computational cost. However, this is possible because the sampling probability is fixed, and it seems necessary to explain how to transform this to satisfy independent sampling.\n  \n  * It is difficult to know the exact process of the algorithm. Algorithm 3 is a very important part of the paper, but it is difficult to understand properly because it is omitted in the main part.\n\nW2) It is not clear how theoretically optimal Sample and Scale are. Looking at Table 1, $\\epsilon_p$ of Sample and Scale could have a loose bound depending on the $\\sigma_{\\mathrm{SAMPLE}}$ and $\\sigma_p$ values, respectively. It would be helpful to actually see what these values are in the experiments.\n\nW3) In Algorithm 1, the paper says that getSampleRate is equivalent to the Opacus function get_noise_multiplier without a detailed description, but the two functions seem to return different objects. The function getSampleRate returns a sample rate, but the Opacus function get_noise_multiplier returns noise_mutiplier $\\sigma$.\n\nW4) The comparison between Sample and Scale is not clear enough. The two methods are introduced and only empirically compared, and it seems that Sample mostly dominates Scale (according to Table 2).  Looking at the overall algorithm, both techniques look similar in terms of efficiency. Then when should one use Scale? \n\nW5) Also, a natural approach seems to be combining Sample and Scale where the sampling and clip norms are both adjusted. Why not use this combination?\n\nW6) The notion of individualized privacy ($\\epsilon_p$, $\\delta$)-DP seems to be confused with the original ($\\epsilon$, $\\delta$)-DP. This confusion leads to other confusions, e.g., whether ($\\epsilon_1$, $\\delta$)-DP in Theorem 3.1. and ($\\epsilon_P$, $\\delta$)-DP in Theorem 3.2. mean individual privacy or conventional DP.\n\nW7) The experimental setup of only using up to three privacy budgets seems limiting. What happens if there are many more than three budgets?\n\nW8) Some reference number is wrong. For example, in the last paragraph of the proof of Theorem G.1, Mironov et al. is [21] or [22], not [20].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1 & Q6: Application of SGM theorems to IDP**\n\nThe proof in G.1 showing that the entire mechanism satisfies $(\\{\\varepsilon_1, \\varepsilon_2, \\dots, \\varepsilon_P\\}, \\delta)$-IDP is built on the observation that our methods can be considered as $P$ simultaneously executed SGMs that update the same model: The privacy groups divide our training data in $P$ disjoint subsets. Each of the $P$ SGMs operates on a different partition of the training data, using different individual sample rates or clip norms.\nGiven that we perform the privacy accounting also on a per-group basis, each privacy group is just an original SGM (with different privacy parameters). Hence, the original SGM theorems apply.\n\n> **W1: Implementation of IDP-SGD**\n\nWe build our implementation on top of Opacus. \n\n**Code flow**: We extend the PrivacyEngine to additionally take per-point budgets, i.e., an epsilon value for each training data point and to derive the adequate privacy parameter for each point (sample rates/clip norms). We also added an IndexedDataset to Opacus to be able to refer to individual data points via their index.\n\n**Individual sample rates**: We implement a data loader based on a custom weighted sampler. Our sampler adapts the functionality of Opacus’ UniformWithReplacementSampler and also uses the torch.rand function. In contrast to the UniformWithReplacementSampler that compares the generated random values against a uniform threshold to determine which data points will be in the current mini-batch, we compare point-wise, and each data point has an individual threshold that depends on its privacy assignment. \n\n**Individual clip norms**: The standard private optimizer in Opacus separately clips the gradient of each data point within a mini-batch. For our Scale, during clipping, each data point is clipped according to their privacy budget (with respective clip norm). \n\nThe above description is reflected in the code attached to this submission.\n\n> **W1: Algorithm 3**\n\nWe made space for Algorithm 3 in the main paper by merging Figure 3a) and 3b).\n\n> **W2: Optimality of individual privacy parameters**\n\nOur code implements Algorithms 1 and 2 to derive privacy parameters such that all privacy groups’ budgets will be exhausted at the end of training. Copying the per-group final privacy consumption from the log file of a training with CIFAR10, first privacy setup, Table 2 we see that we achieve `Sample: [0.997, 1.995, 2.992]`, and `Scale: [0.998, 1.995, 2.994]` in practice with eps=1,2,3 specified. I.e., the actual privacy consumption is tight. When adjusting the precision parameter in our calculation of privacy parameters from 0.0001 (displayed results) to e.g. 0.00001, values even closer to 1,2,3 can be achieved.\n\n\n> **W3&Q2: get_noise_multiplier vs. get_sample_rates**\n\nWe meant that these functions are *conceptually* equivalent. They both rely on an interactive process where one privacy parameter is to be found in a binary-search style while the others are fixed. \n\n> **W4&Q4: Comparing Sample and Scale**\n\nWe report the two best performing methods in the main paper (the other methods are in Appendix E). \n\nWe observe two scenarios in which Scale is superior to Sample. (1) For full-batch gradient descent [9], Scale can be used while Sample is not applicable, given that all data points are sampled. (2) For extremely large dataset sizes (e.g. >500k), and very small mini-batch sizes (e.g. 32), due to rounding of the extremely small sample rates, the rates for two different privacy groups can become very similar or the same, preventing a fine-grained individualized privacy. This does not happen for Scale, where the clip norms are independent from the number of data points. The results in the general response show that, e.g., on BERT for SNLI Scale outperforms Sample.\n\n> **W5: Combining Sample and Scale**\n\nThe combination of Sample and Scale extends the number of hyperparameters and potential to find better performing algorithms. It can be implemented as follows: obtain individual sampling probabilities as in SAMPLE. Once a mini-batch is sampled, use the respective data points’ indices and clip their gradients as in SCALE.\n\n> **W6: Notation for IDP**\n\nWe would like to clarify that ($\\epsilon_1$, $\\delta$)-DP in Theorem 3.1. and ($\\epsilon_P$, $\\delta$)-DP in Theorem 3.2. mean conventional DP.   Semantically, $\\epsilon_P$, $\\delta$-DP expresses that  the epsilon value $\\epsilon=\\epsilon_p$ is equal to the privacy budget of the group with the highest privacy budget from individualized DP. \nTo avoid confusion with the notation, we updated the paper to use the abbreviation $(\\{\\epsilon_1, \\epsilon_2, \\dots, \\epsilon_P\\}, \\delta)$-**I**DP or ($\\epsilon_P$, $\\delta$)-**I**DP when we refer to individualized privacy, and ($\\epsilon{(_P)}$, $\\delta$)-DP for standard DP.\n\n> **W7: More privacy groups than 3**\n\nWe already present such a result in Figures 5, 6, and 7. In these setups, we have *10* different privacy groups. To show the flexibility of our methods, we ran additional experiments on the MNIST dataset, using 100 evenly sized privacy groups with budgets [1, 1.05, 1.1, ..., 5.9, 5.95] and report the results in the general response.\n\n> **Q1: Naming “Individualized”-DP**\n\nThere are different names for this concept. We also cite [2] who refer to the concept as heterogeneous differential privacy. We use Individualized DP-SGD following the naming of Individualized PATE [5].\n\n> **Q3: Computational costs**\n\nWe performed experiments measuring runtimes. The detailed results can be found in the general answer. We observe that Sample and Scale do not significantly increase runtime in comparison to standard DP-SGD with (406.33 sec for Sample, 414.33 for Scale vs. 403.67 for DP-SGD).\n\n> **W8&Q5: References**\n\nYes, we meant [22]. The full paper in the supplementary material accidentally is missing citation [7] from the submission. We fixed all the citations for the updated version of our paper.\n"}, {"review_id": "zMmLSmR8Dl", "reviewer": "Reviewer_r49G", "summary": "This paper designs variants of differentially private SGD to satisfy different privacy expectations, e.g., users can choose one from high, medium, or low levels of privacy. There are two variants of DP-SGD, one changes the sampling probability of different groups, and the other changes the clipping threshold of different groups. ", "questions": "See Weaknesses", "limitations": " ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The Sample and Scale methods are easy-to-implement. Their design aligns nicely with the fact that DP-SGD adds noise to the aggregated gradient, instead of separate noise for individual gradients.\n\n2. Experiments on several common vision datasets show that individualized DP-SGD is better than DP-SGD with a uniform (the smallest) privacy parameter. The authors also run membership inference attacks to show that groups with smaller privacy parameters do have better empirical privacy.\n\n3. The paper is very well written and easy to follow. \n", "weaknesses": "\n1. The privacy expectations in the current version are independent of model accuracy. In practice, the minority groups, whose accuracy is usually worse, may prefer stronger privacy than other groups. What would happen if the group with the worst accuracy has the strongest/weakest privacy expectation? One way to implement this is to extend the experiments in Appendix D.2 by assigning a larger/smaller privacy parameter to the class with the worst non-private accuracy. \n\n2. The privacy expectation itself may be data dependent and hence the individualized privacy assignment cannot be made public, as the authors discussed in Line 68 – Line 73. For example, in a medical dataset, users diagnosed with certain diseases may have higher privacy expectations.  \n\n3. The authors use a shallow convolutional network for the experiments. It would be better to see whether the results still hold for deep learning models, e.g., ResNets or transformers.\n\n4. The authors only report the average accuracy. It would be better if the authors could show the accuracy of different groups. A recent line of work shows that DP has disparate impact on model accuracy [1]. Will the groups with smaller privacy assignments lose more accuracy?\n\n[1]: Differential Privacy Has Disparate Impact on Model Accuracy. https://arxiv.org/abs/1905.12101\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and provide our responses below.\n\n> **W1: Influence on utility for underprivileged groups**\n\nWe already present such a result in Table 8  in our submission. We selected the CIFAR10 dataset and assigned lower or higher privacy budget to one of the classes. Class 3 can be considered as an underprivileged group since it has the lowest accuracy across all classes of only 32.64%. If the privacy budget for this class is decreased further to eps=1 while the other classes maintain eps=2, the accuracy for class 3 plummets to only 6.67%, while the accuracy for other classes increases. In contrast, when a higher privacy budget of eps=3 is assigned to class 3, then its accuracy increases substantially and is comparable with accuracy for the remaining classes. \n\n> **W2: Confidentiality of the privacy values**\n\nThere is no need to reveal the privacy assignments and the end user of the released model should be oblivious to the training data points’ individual privacy preferences. In the general response, we detail how the fact that end users can only interact with the trained model preserves confidentiality of the training data points’ privacy preferences.\n\n> **W3: Extending experimentation to ResNet and Transformers**\n\nWe thank the reviewer for their suggestion and we extended our experimentation. Among other experiments (see general response), we\n- trained a ResNet18 from scratch with CIFAR10 \n- and fine-tuned a BERT transformer (bert-base-uncased) on the SNLI dataset for natural language inference (NLI). \n\nA summary of the results is summarizes in this table for convenience:\n| Model    | Dataset | DP-SGD (eps=5) | Sample (eps=5,7.5,10) | Scale (eps=5,7.5,10) |\n|----------|---------|----------------|--------------------|-------------------|\n| ResNet18 | CIFAR10 | 47.52+-0.84    | 48.52+-0.69        | 48.77+-0.73       |\n| BERT     | SNLI    | 75.91+-0.23    | 76.11+-0.21        | 76.5+-0.17        |\n\nIn both experiments, our results show that the Scale and Sample methods again outperform standard DP-SGD. See the general response for more experimental details.\n\n> **W4: Per privacy-subgroup utility**\n\nWe show the accuracy for different subgroups with their different privacy budgets in Table 8. We consider each subgroup to be a different class. We consider the test data to divide into the same per-class subgroups as the training data. This enables us to show a per-group (class) test accuracy under the given different privacy budgets. The observed trend indicates that if a group is willing to sacrifice more privacy, it will gain higher utility of the model whereas for groups with stronger privacy preferences, the opposite is the case.\n\nWe would also like to clarify that in the main paper, for example Table 2, we, again report the test accuracies. Test data points do not get the possibility to specify any privacy budget (since the model only predicts on them and does not train on them, their privacy is not leaked anyways). For the training data points in the main paper, we assigned their privacy budget at random during training. Hence, there is no mapping to what subgroup of the test data would correspond to what subgroup of the training data. As a result, when reporting test accuracies in this setup, it is impossible to show per-group test accuracies.\n"}]

Author rebuttal:
We would like to thank all reviewers for their feedback which has greatly helped us improve the paper. We are glad that the reviewers recognize our work to address an important problem by proposing the first personalization mechanisms for DP-SGD (3QXJ) which are novel, easy to implement, and provide theoretical privacy guarantees (7px6, Sruy, r49G). Our methods ‘yield a modest utility over alternative baselines’ (3QXJ), (Sruy,r49G), and the ‘experiments in this paper are adequate for the evaluation’ with additional valuable results and discussions in the appendix (7px6). The reviewers appreciate the writing and the organization of the paper (7px6, r49G) and we hope that this work can contribute to making machine learning with individualized privacy guarantees practical. Below we offer clarifications to some common questions.

>**1. Extending experiments**

**Other (larger) architectures, modalities, tasks, and fine-tuning:**
Following reviewers 3QXJ and r49G, we evaluated IDP-SGD in a broader scope and
fine-tuned a BERT transformer (bert-base-uncased) on the SNLI dataset for natural language inference,
trained a ResNet 18 from scratch on the CIFAR10 dataset for image classification,
trained a linear embedding model *(one embedding and two linear layers)* on IMDB data for text classification,
and trained a character-level RNN *(DPLSTM, three layers, hidden size: 128)* from scratch to classify surnames to languages.

|Model|Dataset|DP-SGD (eps=5)|Sample (eps=5,7.5,10)|Scale (eps=5,7.5,10)|
|-----------|----------|----------------|--------------------|-------------------|
|BERT|SNLI|75.91+-0.23|76.11+-0.21|76.5+-0.17|
|ResNet18|CIFAR10|47.52+-0.84|48.52+-0.69|48.77+-0.73|
|Embedding|IMDB|72.69+-0.27|73.27+-0.3|73.34+-0.11|
|||**(eps=1)**|**(eps=1,2,3)**| **(eps=1,2,3)**|
|RNN|Surnames|60.86+-0.78|65.56+-0.96|66.0+-1.19|
Over all setups, our IDP-SGD outperforms standard DP-SGD → see also attached PDF, Table 1. 

**Runtimes:**
Inspired by reviewer Sruy, we analyzed runtimes of our methods. E.g., for the first privacy setting of Table 2, and the CIFAR10 dataset with the CNN architecture, we observe that Sample and Scale do not significantly increase runtime over DP-SGD with 406.33 sec for Sample, 414.33 for Scale vs. 403.67 for DP-SGD → see PDF Table 3.

Regarding the time for computing the privacy parameters which takes place *exactly once* before the training with a new privacy setup, we included a detailed Table 4 in the PDF.  Computation time depends on a precision parameter and the number of groups: For example, with 8 groups and precision=0.0001 (used in this work), Sample requires 20 sec. and Scale 5 sec. to determine the parameters. 

**More privacy groups:**
Following reviewer Sruy, we implemented IDP-SGD training on MNIST with 100 evenly sized privacy groups with budgets [1, 1.05, 1.1, ..., 5.9, 5.95]. Our methods achieved the following accuracy: Sample: 98.17%, Scale: 98.21% vs. the standard DP-SGD baseline with epsilon=1: 96.75%. → see PDF Table 2.

**Compare MIA with standard DP-SGD**

Given the suggestion by reviewer 7px6, we trained models with $\varepsilon=10$ and $\varepsilon=20$ with standard DP-SGD and compared the results to our MIA results with IDP-SGD. Our key observations are:
1. IDP-SGD reduces the privacy risk for the data points with $\varepsilon=10$ (standard DP-SGD: AUC=0.568; Sample: 0.537; Scale: 0.552).
2. This privacy gain does not come at large expense of points with $\varepsilon=20$ whose privacy risk remains roughly the same (AUC=0.59 vs Sample: 0.581 and Scale: 0.589).
 → see PDF Figure 1.

> **2. Confidentiality of privacy preferences**

The confidentiality of our privacy budgets results from the following observations:
1. Even for standard DP training, it was shown that one cannot perform sample-efficient black-box audits to determine the privacy-budget of a trained model [[Property Testing for Differential Privacy]](https://doi.org/10.48550/arXiv.1806.06427). Existing frameworks that perform auditing, e.g. [[Debugging Differential Privacy: A Case Study for Privacy Auditing]](https://arxiv.org/pdf/2202.12219.pdf) rely on training 1000-100,000 shadow models to get an estimate of the full model’s privacy guarantee. 
In our IDP-SGD setup, the model does not have one single $\varepsilon$, but one $\varepsilon$ per group, aggravating the impracticability of auditing guarantees and limiting the applicability of existing frameworks. As a result, the adversary cannot determine the different per-group privacy budgets of the model.
2. Our MIA experiments in Section 4.2 show a difference in privacy risks over *entire privacy groups*. First note that these experiments (Figure 2) were the most costly ones in our paper.  We had to train more than 1500 (3 times 512) shadow models to generate the results. Yet, there are two limitations: a) The results only show differences in the groups, but do not reveal the respective epsilon values, b) they do not allow to perform per-point distinctions. This is because the distributions of privacy risks over the two groups still have a large overlap. Hence, given a single point’s risks and the two distributions, one cannot predict without error to what group the point belongs. We acknowledge that the error could be reduced when the privacy budgets between the groups differ significantly (0.1 vs. 100). Note however, that we suggest deploying the model in a setup where individuals simply specify their privacy preferences and the model builder or an ethics committee assign concrete epsilon values to these preferences. We observed that extreme differences between privacy groups’ $\varepsilon$s are less beneficial because they degrade overall model performance. Hence, in practice, these large differences are very unlikely. As a result, it will be very challenging for an adversary to determine which privacy group a data point belongs to. Even if they manage to, they will not be able to learn the privacy budget of the group (see 1.). 


Meta-review: Reviewers all generally agreed that this was an interesting and worthwhile direction. While the algorithms are not too surprising, it's a good first step. The weaknesses were not considered significant enough to warrant holding back this work for publication.
Decision: Accept (poster)