SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Franziska Boenisch*†
boenisch@cispa.de
CISPA Helmholtz Center for Information Security
Christopher Muhl*
christopher.muehl@fu-berlin.de
Free University Berlin
Adam Dziedzic*†
dzedzic@cispa.de
Roy Rinberg
roy.rinberg@columbia.edu
Nicolas Papernot
nicolas.papernot@utoronto.ca
University of Toronto & Vector Institute
When training a machine learning model with differential privacy, one sets a privacy budget. This uniform budget represents an overall maximal privacy violation that any user is willing to face by contributing their data to the training set. We argue that this approach is limited because different users may have different privacy expectations. Thus, setting a uniform privacy budget across all points may be overly conservative for some users or, conversely, not sufficiently protective for others. In this paper, we capture these preferences through individualized privacy budgets. To demonstrate their practicality, we introduce a variant of Differentially Private Stochastic Gradient Descent (DP-SGD) which supports such individualized budgets. DP-SGD is the canonical approach to training models with differential privacy. We modify its data sampling and gradient noisig mechanisms to arrive at our approach, which we call Individualized DP-SGD (IDP-SGD). Because IDP-SGD provides privacy guarantees tailored to the preferences of individual users and their data points, we empirically find it to improve privacy-utility trade-offs.
Machine learning (ML) models are known to leak information about their training data. Such leakage can result in attacks that determine whether a specific data point was used to train a given ML model (membership inference) [36, 43, 8], infer sensitive attributes from the model's training data [13, 45], or even (partially) reconstruct that training data [15, 35]. Therefore, the need to provide privacy guarantees to the individuals who contribute their sensitive data to train ML models is pressing.
Approaches to train ML models with guarantees of differential privacy (DP) [10] have established themselves as the canonical answer to this need. In the context of ML, differential privacy bounds how much can be learned about any of the individual data points from the model's training set. Take the canonical example of differentially private stochastic gradient descent (DP-SGD): it updates the model using noisy averages of clipped per-example gradients. Unlike vanilla SGD, this ensures that any single training point has a limited influence on model updates (due to the clipping to a pre-defined maximum clip norm) and that the training is likely to output a similar model should a single point
*Equal contribution. †Work was done at the University of Toronto and the Vector Institute.
be added to or removed from the training set (due to the noisy average). As is the case for any DP algorithm, the privacy guarantees of DP-SGD are derived analytically. They are captured by a privacy budget  $\varepsilon$ . This budget represents the maximal privacy violation that any user contributing data to the training set is willing to tolerate. Lower budgets correspond to stronger privacy protection since they impose that the outputs of the training algorithm have to be closer for similar training sets.
In this paper, we argue that there is a key limitation when training ML models with DP guarantees: the privacy budget  $\varepsilon$  is set uniformly across all training points. This implicitly assumes that all users who contribute their data to the training set have the same expectations of privacy. However, that is not true; individuals have diverse values and privacy preferences [17, 5]. Yet, because DP-SGD assumes a uniform privacy budget, this budget must match the individual whose privacy expectations are the strongest. That is,  $\varepsilon$  has to correspond to the lowest privacy violation tolerance expressed amongst individuals who contributed their data. This limits the ability of DP-SGD to learn because the privacy budget  $\varepsilon$  together with the sensitivity of each step of the algorithm (i.e., the clip norm) define the scale of noise that needs to be added: the stronger the required privacy protection, the higher the noise scale. Thus, implementing privacy protection according to the lowest privacy violation tolerance in the training dataset comes at a significant cost in model accuracy.
We are the first to propose two variants of the DP-SGD algorithm, that we refer to as IDP-SGD, to enforce different privacy budgets for each training point.2 First, our modified sampling mechanism (Sample) samples data points proportionally to their privacy budgets: points with higher privacy budgets are sampled more frequently during training. It is intuitive that the more a training point is analyzed during training, the more private information can leak about this point. Second, with our individualized scaling mechanism (Scale), we rescale the noise added to gradients based on the budget specified for each training point. Naively, one would implement Scale by changing the scale of added noise per individual data point. However, for efficiency, current implementations of DP-SGD typically noise the sum of the per-example clipped gradients over an entire mini-batch, hence, adding the same amount of noise to all gradients. Yet, within the context of this implementation, we note that we can effectively adapt the scale of noise being added on an individual basis by adjusting the sensitivity (i.e. the clip norm) of each example by a multiplier. Because clipping is performed on a per-example basis, this approach enables the efficient implementation of our proposed Scale method.
To summarize our contributions:
- We introduce two novel individualized variants of the DP-SGD algorithm, where each data point can be assigned its desired privacy budget. This allows individuals who contribute data to the training set to specify different privacy expectations.
- We provide a theoretical analysis of our two mechanisms to derive how the privacy parameters (e.g., noise multipliers and sample rates) should be set to meet individual data points' privacy guarantees and provide a thorough privacy analysis for Sample and Scale.
- We carry out an extensive empirical evaluation on various vision and language datasets and multiple model architectures. Our results highlight the utility gain of our methods for different distributions of privacy expectations among users.
Ethical considerations. We need to ensure that our individualized privacy assignment does not harm individuals by disclosing too much of their sensitive information. We, therefore, suggest implementing our methods in a controlled environment where privacy risks are openly communicated [41, 9, 12], and where a regulatory entity is in charge of ensuring that even the individuals with lowest expectations regarding their privacy obtain a sufficient degree of protection. We further discuss the ethical deployment of our Individualized DP-SGD (IDP-SGD) in Appendix A.
2.1 Differential Privacy and DP-SGD
Algorithm  $M$  satisfies  $(\varepsilon, \delta)$ -DP, if for any two datasets  $D, D' \subseteq \mathcal{D}$  that differ in any one record  $x$ , such that  $D = D' \cup x$ , and any set of outputs  $R$
$$
\mathbb {P} [ M (D) \in R ] \leq e ^ {\varepsilon} \cdot \mathbb {P} [ M (D ^ {\prime}) \in R ] + \delta . \tag {1}
$$
2In the rest of this paper, we assume without loss of generality that each training point is contributed by a different individual.
Since the possible difference in outputs is bounded for any pair of datasets, DP bounds privacy leakage for any individual. The value  $\varepsilon \in \mathbb{R}_+$  specifies the privacy level with lower values corresponding to stronger privacy guarantees. The  $\delta \in [0,1]$  offers a relaxation, i.e., a small probability of violating the guarantees. See Appendix B.1 for a more thorough introduction of  $(\varepsilon ,\delta)$ -DP.
Another relaxation of DP is based on the Rényi divergence [27], namely  $(\alpha, \varepsilon)$ -RDP, which for the parameters as described above and an order  $\alpha \in (1, \infty)$  is defined as follows:  $\mathbb{D}_{\alpha}(M(D) \| M(D')) \leq \varepsilon$ . Due to its smoother composition properties, it is employed for privacy accounting in ML with DP-SGD. Yet, it is possible to convert between the two notions: If  $M$  is an  $(\alpha, \varepsilon)$ -RDP mechanism, it also satisfies  $(\varepsilon + \log \frac{\alpha - 1}{\alpha} - \frac{\log \delta + \log \alpha}{\alpha - 1}, \delta)$ -DP for any  $0 < \delta < 1$  [3].
DP-SGD extends standard SGD with two additional steps, namely clipping and noise addition. Clipping each data point's gradients to a pre-defined clip norm  $c$  bounds their sensitivity to ensure that no data point causes too large model updates. Adding noise from a Gaussian distribution,  $\mathcal{N}(0, (\sigma c)^2\mathbf{I})$ , with zero mean and standard deviation according to the sensitivity  $c$  and a pre-defined noise multiplier  $\sigma$  introduces privacy. We formally depict DP-SGD as Algorithm 1. To yield tighter privacy bounds, DP-SGD implements privacy amplification by subsampling [4] (line 3 in Algorithm 1) where training data points are sampled into mini-batches with a Poisson sampling, rather than assigning each data point from the training dataset to a minibatch before an epoch starts. DP-SGD hence implements a Sampled Gaussian Mechanism
(SGM) [28] where each training data point is sampled independently at random without replacement with probability  $q \in (0,1]$  over  $I$  training iterations. As shown by Mironov et al. [28], an SGM with sensitivity of  $c = 1$ , satisfies  $(\alpha ,\varepsilon)$ -RDP where
$$
\varepsilon \leq I \cdot 2 q ^ {2} \frac {\alpha}{\sigma^ {2}}. \tag {2}
$$
Equation (2) highlights that the privacy guarantee  $\varepsilon$  depends on the noise multiplier  $\sigma$ , the sample rate  $q$ , the number of training iterations  $I$ , and the RDP order  $\alpha$ .
2.2 Individualized Privacy
Individualized Privacy is a topic of high importance given that society consists at least of three different groups of individuals, demanding either strong, average, or weak privacy protection for their data, respectively [17, 5]. Furthermore, some individuals inherently require higher privacy protection, given their personal circumstances and the degree of uniqueness of their data. However, without individualization, when applying DP to datasets that hold data from individuals with different privacy requirements, the privacy level  $\varepsilon$  has to be chosen according to the lowest  $\varepsilon$  encountered among all individuals, which favors poor privacy-utility trade-offs. Prior work on individualized privacy guarantees focused mainly on standard data analyses without considering ML [2, 19, 24, 31]. However, some of the underlying ideas inspired the design of our IDP-SGD variants.
Our Sample method relies on a similar idea as the sample mechanism proposed by Jorgensen et al. [19] where data points with stronger privacy requirements are sampled with a lower probability than data points that agree to contribute more of their information. The most significant difference between the sample mechanism and our approach is that the former performs sampling as a pre-processing step independent of the subsequent algorithm. In contrast, we perform subsampling within every iteration of our IDP-SGD training and show how to leverage the sampling probability of DP-SGD to obtain individual privacy guarantees. Based on this difference, we could not build on how Jorgensen
<sup>3</sup>In DP-SGD, the concept of an epoch does not exist and training duration is measured in iterations. Due to the Poisson sampling, training mini-batches can contain varying numbers of data points.
et al. [19] compute the sampling probabilities and had to propose an original mechanism to derive sampling probabilities for individualized privacy in IDP-SGD.
In a similar vein to our Scale method, Alaggan et al. [2] scale up data points individually before the noise addition to increase their sensitivity in their stretching mechanism. As a consequence, the signal-to-noise ratios of data points that are scaled with large factors will be higher than the ones of data points that are scaled with small factors. In contrast to Alaggan et al. [2], we do not scale the data points themselves, but the magnitude of noise added to their gradients, relative to their individual sensitivity. Our new interesting observation is that data points' individual clip norms can be used to indirectly scale the noise added to a mini-batch of data in DP-SGD per data point.
Closest work to ours [6] proposes two mechanisms for an individualized privacy assignment within the framework of Private Aggregation of Teacher Ensembles (PATE) [33]. Their upsampling duplicates training data points and assigns the duplicates to different teacher models according to the data points' privacy requirements, while their weighting changes the aggregation of the teachers' predicted labels to reduce or increase the contribution of teachers according to their respective training data points' privacy requirements. PATE's approach for implementing ML with DP differs significantly from DP-SGD: PATE relies on privately training multiple non-DP models on the sensitive data and introducing DP guarantees during a knowledge transfer to a separately released model. The approaches for individualized privacy in PATE, therefore, are non-applicable to DP-SGD. See Appendix B.3 for details on the PATE algorithm and its individualized extensions.
Relationship to Individualized Privacy Accounting. An orthogonal line of research that comes closest to providing individualized guarantees for DP-SGD focuses on individualized privacy accounting; the idea is to learn more from points that consume less of the privacy budget over training [11, 44]. Jordon et al. [18] propose a personalized moments' accountant to compute the privacy loss on a per-sample basis. Yu et al. [44] perform individualized privacy accounting within DP-SGD based on the gradient norms of the individual data points. Feldman and Zrnic [11] introduce RDP filters to account for privacy consumption per data point and train as long as there remain data points that do not exceed the global privacy level. Yet, it has been shown that the data points that consume little privacy budget and therefore remain in the training are the ones that already incur a small training loss [44]. Training more on them will not significantly boost the model accuracy. While privacy assignment and accounting are concerned with improving privacy-utility trade-offs within DP, they operate under different setups. Privacy accounting (in the above three methods) assumes a single fixed privacy budget assigned over the whole dataset. In contrast, our privacy assignment is concerned with enabling different individuals to specify their respective privacy preferences. Since individual accounting and assignment are two independent methods, we experimentally show their synergy by applying individual assignment to individual accounting [11], which yields higher model utility than individual accounting alone.
3 Our Individualized Privacy Assignment
3.1 Formalizing Individualizing Privacy
Setup and Notation. Given a training dataset  $D$  with points  $\{x_{1},\ldots ,x_{N}\}$  which each have their own privacy preference (or budget), we consider data points with the same privacy budget  $\varepsilon_{p}$  together as a privacy group  $\mathcal{G}_p$ . This notation is aligned with [17, 5] that identified different groups of individual privacy preferences within society. We denote with  $\varepsilon_{1}$  the smallest privacy budget encountered in the dataset. Standard DP-SGD needs to set  $\varepsilon = \varepsilon_{1}$  to comply with the strongest privacy requirement encountered within the data. For all  $\varepsilon_p,p\in [2,P]$ , it follows that  $\varepsilon_{p} > \varepsilon$  and we arrange the groups such that  $\varepsilon_{p} > \varepsilon_{p - 1}$ . Note that following [2], we argue that the privacy preferences themselves should be kept private to prevent leakage of sensitive information which might be correlated with them. If there is a necessity to release individual privacy preferences, this should be done under the addition of noise to obtain DP guarantees for the privacy preferences. This can for instance be done through a smooth sensitivity analysis, as done in prior work, e.g. [34]. In general, we find that it is not necessary to release the privacy budgets: To implement IDP-SGD, only the party who trains the ML model on the individuals' sensitive data needs to know their privacy budgets to set the respective privacy parameters accordingly during training. Given that this party holds access to the data itself, this does not incur any additional privacy disclosure for the individuals. Other parties then only interact with the final private model (but not with the individuals' sensitive data or their
associated privacy budgets) and it has been shown impractical to disclose privacy guarantees through black-box access to ML models [14]. We discuss the confidentiality of the privacy budgets further in Appendix A.2.
Individualized Privacy. Following [19], our methods aim at providing data points  $x_{i} \in \mathcal{G}_{p}$  with individual DP (IDP) guarantees according to their privacy budget  $\varepsilon_{p}$  as follows: The learning algorithm  $M$  satisfies  $(\varepsilon_{p},\delta)$ -IDP if for all datasets  $D \stackrel{x_i}{\sim} D'$  which differ only in  $x_{i}$  and for all outputs  $R \subseteq \mathcal{R}$
$$
\mathbb {P} [ M (D) \in R ] \leq e ^ {\varepsilon_ {p}} \cdot \mathbb {P} [ M (D ^ {\prime}) \in R ] + \delta . \tag {3}
$$
Without loss of generality, we assume that all individual data points have privacy preferences with the same  $\delta$ . Note that other notions of DP (e.g., RDP) can analogously be generalized to enable per-data point privacy guarantees. The key difference between Equation (3) and the standard definition of DP (Equation (1)) does not lie in the replacement of  $\varepsilon$  with  $\varepsilon_{p}$ . Instead, the key difference is about the definition of neighboring datasets. For standard DP, neighboring datasets are defined as  $D \sim D'$  where  $D$  and  $D'$  differ in any random data point (given that it is standard DP, every data point has privacy  $\varepsilon$ ). In contrast, in our work, following [19] the neighboring datasets are defined as  $D \stackrel{x_i}{\sim} D'$  where  $D$  and  $D'$  differ in any arbitrary data point  $x_i$  which has a privacy budget of  $(\varepsilon_p, \delta)$ . This yields to the individualized notion of DP.
We formalize the relationship between the individualized notion of DP (which we denote by  $(\{\varepsilon_1,\varepsilon_2,\dots ,\varepsilon_P\} ,\delta)$  -IDP) and standard  $(\varepsilon ,\delta)$  -DP in the following two lemmas over the learning algorithm  $M$  . The proofs are included in Appendix G.
Lemma 3.1. An algorithm  $M$  that satisfies  $(\varepsilon_1,\delta)$  -DP also satisfies  $(\{\varepsilon_{1},\varepsilon_{2},\ldots ,\varepsilon_{P}\} ,\delta)$  -IDP.
Lemma 3.2. An algorithm  $M$  that satisfies  $(\{\varepsilon_1,\varepsilon_2,\ldots ,\varepsilon_P\} ,\delta)$  -IDP also satisfies  $(\varepsilon_P,\delta)$  -DP.
3.2 From Individualized Privacy Preferences to Privacy Parameters
From the individual privacy preferences and the total given privacy distribution over the data, i.e., the number of different privacy budgets  $P$ , their values  $\varepsilon_{p}$ , and the sizes of the respective privacy groups  $|\mathcal{G}_p|$ , we derive individual privacy parameters for IDP-SGD such that all data points within one privacy group (same privacy budget) obtain the same privacy parameters, and such that all privacy groups are expected to exhaust their privacy budget at the same time, after  $I$  training iterations.
The individualized parameters for our Sample method are the individual sample rates  $\{q_{1},\ldots ,q_{P}\}$  , and a respective noise multiplier  $\sigma_{\mathrm{sample}}$  common to all privacy groupswhich we derive from the privacy budget dis
tribution as described in Section 3.3. In Scale, the individualized parameters are noise multipliers  $\{\sigma_1,\dots ,\sigma_P\}$  with their respective clip norms  $\{c_{1},\ldots ,c_{P}\}$ , as we explain in Section 3.4. Our individual bounds per privacy group are depicted in Table 1. We omit  $\alpha$  from consideration since its optimal value is selected as an optimization parameter when converting from RDP to  $(\varepsilon ,\delta)$ -DP.
Table 1: (Individualized) Privacy Bounds.
DP-SGD	SAMPLE	SCALE
ε ≤ I · 2q2 α/σ2	εp ≤ I · 2qp2 α/σsample2	εp ≤ I · 2q2 α/σp2
3.3 Our Sample Method
Our Sample method relies on sampling data points with different sample rates  $\{q_1,\dots ,q_P\}$  depending on their individual privacy budgets. In this case, the noise multiplier  $\sigma_{\mathrm{sample}}$  is fixed. Data points with higher privacy budgets (weaker privacy requirements) are assigned higher sampling rates than those with lower privacy budgets. This modifies the Poisson sampling for DP-SGD (line 3 in Algorithm 1) to sample data points with higher privacy budgets within more training iterations.
Deriving Parameters. We aim at deriving the privacy parameters that yield the specified individual privacy preferences. The learning hyperparameters, such as mini-batch size  $B$  or learning rate  $\eta$  can be found through hyperparameter tuning. For Sample, given a tuned mini-batch size  $B$ , we have to find  $\{q_1, \ldots, q_P\}$ , such that their weighted average equals  $q$ :  $\frac{1}{N} \sum_{p=1}^{P} |\mathcal{G}_p| q_p \stackrel {!}{=} q = \frac{B}{N} 5$ .
<sup>4</sup>We discuss in Appendix C.1 other alternatives to implement Sample if we allow changing the training hyperparameters.
5We use the notation  $\stackrel {,}{=}$  to indicate that the two terms should be equal.
Algorithm 2: Finding Sample Parameters. The subroutine getSampleRate is equivalent to Opacus' function get_noisemultiplier [32], (see getNoise inAlgorithm 5). We sketch getSampleRate in Algorithm 4 in Appendix C.3.
Require: Per-group target privacy budgets  $\{\varepsilon_1,\dots ,\varepsilon_P\}$  ,target  $\delta$  , iterations  $I$  number of total data points  $N$  and per-privacy group data points  $\{|G_1|,\ldots ,|G_P|\}$
1: init  $\sigma_{\mathrm{sample}}$  ..  $\sigma_{\mathrm{sample}}\gets$  getNoise(ε1,δ,q,I)
2: init  $\{q_1,\dots ,q_P\}$  where for  $p\in [P]$  ..
3:  $q_{p}\gets$  getSampleRate  $(\varepsilon_{p},\delta ,\sigma_{\mathrm{sample}},I)$
4: while  $q \not\approx \frac{1}{N} \sum_{p=1}^{P} |\mathcal{G}_p| q_p$ :
5:  $\sigma_{\mathrm{sample}}\gets s_i\sigma_{\mathrm{sample}}$  {scaling factor:  $s_i < 1\})$
6: for  $p \in [P]$ :
7:  $q_{p}\gets$  getSampleRate  $(\varepsilon_{p},\delta ,\sigma_{\mathrm{sample}},I)$
8: Output  $\sigma_{\mathrm{sample}},\{q_1,\dots ,q_P\}$
Algorithm 3: Finding Scale Parameters. The subroutine getNoise corresponds to Opacus' function get_noise-multiplier [32]. We sketch its implementation in Algorithm 5 in Appendix C.3.
Require: Per-group target privacy budgets  $\{\varepsilon_1,\dots ,\varepsilon_P\}$  ,target  $\delta$  iterations  $I$  number of total data points  $N$  and per-privacy group data points  $\{|G_1|,\ldots ,|G_P|\}$  , default clip norm  $c$  sample rate  $q$
1: init  $\{\sigma_1,\dots ,\sigma_P\}$  where for  $p\in [P]$  ..
2:  $\sigma_{p} \gets \mathrm{getNoise}(\varepsilon_{p}, \delta, q, I)$
3: init  $\sigma_{\mathrm{scale}}$  ..  $\sigma_{\mathrm{scale}}\gets (\frac{1}{N}\sum_{p = 1}^{P}\frac{|G_p|}{\sigma_p})^{-1}$  {see derivation of  $\sigma_{\mathrm{scale}}$  in Appendix C.2}
4: for  $p\in [P]$
5:  $c_{p} \gets \frac{\sigma_{\mathrm{scale}} c}{\sigma_{p}}$
6: Output  $\sigma_{\mathrm{scale}}$ ,  $\{c_1, \ldots, c_P\}$