SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This asserts that the Poisson sampling (line 3 in Algorithm 1) will yield mini-batches of size  $B$  in expectation. We also need to ensure that the privacy budgets of all groups will exhaust after  $I$  training iterations, which we do by deriving the adequate  $\sigma_{\mathrm{sample}}$ , shared across all privacy groups. Equation (2) highlights that the final privacy budget  $\varepsilon$  depends on both sample rate  $q$  and noise multiplier  $\sigma$ . Our Sample method enables higher individualized privacy budgets for the different groups than the standard DP-SGD but still requires setting the average sampling rate equal to  $q$  as in DP-SGD to obtain an expected mini-batch size  $B$ . Hence, the value of  $\sigma_{\mathrm{sample}}$  has to be decreased in comparison to the initial default  $\sigma$ . As an effect of this noise reduction, Sample improves utility of the final model.
Our concrete derivation of values for  $\sigma_{\mathrm{sample}}$  and  $\{q_1,\dots ,q_P\}$  is presented in Algorithm 2. We start from initializing  $\sigma_{\mathrm{sample}}$  with  $\sigma$  from standard DP-SGD, the noise multiplier required for the privacy group  $\mathcal{G}_1$  (strongest privacy requirement of all groups) with the smallest privacy budget  $\varepsilon_{1}$ . This corresponds to instantiating  $\sigma_{\mathrm{sample}}$  with the upper bound noise over all privacy groups. Then, we use a getSampleRate function that derives the sampling rate for the given privacy parameters based on approximating Equation (2). Finally, we iteratively decrease  $\sigma_{\mathrm{sample}}$  by a scaling factor  $s_i$  slightly smaller than one and recompute  $\{q_1,\ldots ,q_P\}$  until their weighted average is (approximately) equal to  $q$ . In Appendix G, we present the formal proof that Sample satisfies  $(\{\varepsilon_1,\varepsilon_2,\dots ,\varepsilon_P\} ,\delta)$ -DP. Our proof relies on Mironov et al. [28] and considers training of each privacy group as a separate SGM—with an individual sampling probability—that all simultaneously update the same model.
3.4 Scale
Our Scale method aims at scaling the noise added to each gradient according to the respective data point's privacy preference. Yet, for efficiency, current implementations of DP-SGD typically noise the sum of the per-example clipped gradients over an entire mini-batch, hence, adding the same amount of noise to all gradients. Therefore, we instead set individualized clip norms  $\{c_1,\ldots ,c_P\}$  that effectively adapt the scale of noise being added on an individual basis by adjusting the sensitivity (i.e. the clip norm) of each example by a multiplier. This changes lines 6 and 8 in the DP-SGD Algorithm (1). Data points with higher privacy budgets (weaker privacy requirements) obtain lower noise and higher clip norms. Since Equation (2) highlights that the clip norm  $c$  has no direct impact on the obtained  $\varepsilon$ , individualized privacy in Scale results from the individual noise multipliers  $\{\sigma_1,\dots ,\sigma_P\}$ . Utility gains come from the overall increase in the signal-to-noise ratio during training.
Deriving Parameters. While the ultimate goal of our Scale method is to adapt individual noise multipliers per privacy group, we cannot implement this directly without degrading training performance. The reason is that whereas in DP-SGD sampling and gradient clipping are performed on a per-data point basis, noise is added per mini-batch, see lines 3, 6 and 8 in Algorithm 1, respectively. However, if we restrict mini-batches to contain only data points from the same privacy group (which
share the same noise multiplier), we lose the gains in privacy-utility trade-offs which result from the subsampling (see Appendix F.1 for more details). Hence, while we rely on mini-batches containing data points with different privacy requirements (i.e., different noise multipliers) we can only specify one fixed noise multiplier  $\sigma_{\mathrm{scale}}$ .
To overcome this limitation, we do not set noise multipliers  $\{\sigma_1,\ldots ,\sigma_P\}$  directly, but indirectly obtain them through individualized clip norms  $\{c_{1},\dots ,c_{P}\}$  as follows: In standard DP-SGD, Algorithm 1, a gradient clipped to  $c$  (line 6) obtains noise with standard deviation  $\sigma c$  (line 8). For Scale, we clip gradients to  $c_{p} = s_{p}c$  with a per-privacy group scaling factor  $s_p$  and they obtain noise  $\sigma_{p}c_{p}$ . But in practice, we add noise according to  $\sigma_{\mathrm{scale}}c$  to all mini-batches. Hence, the effective scale  $\sigma_{p}$  of added noise is  $\sigma_{\mathrm{scale}}c = \sigma_{\mathrm{scale}}\frac{c_p}{s_p}\stackrel {!}{=} \sigma_pc_p\Rightarrow \sigma_p = \frac{1}{s_p}\sigma_{\mathrm{scale}}$ . For data points with higher privacy budgets  $s_p > 1$ , so their gradients are clipped to larger norms  $c_{p} = s_{p}c$  and a smaller noise multiplier  $\sigma_{p} = \frac{1}{s_{p}}\sigma_{\mathrm{scale}}$  is assigned to them. The opposite is true for data points with lower privacy budgets.
We find the values of  $\sigma_{p}$  required to obtain each privacy groups' desired  $\varepsilon_{p}$  using the getNoise function (see Algorithm 3 line 1), which takes as inputs the privacy parameters  $\varepsilon_{p},\delta ,q,I$ . To optimize utility, we want to set the individual clip norms such that their average over the dataset corresponds to the best clip norm  $c$  obtained through initial hyperparameter tuning:  $\frac{1}{N}\sum_{p = 1}^{P}|\mathcal{G}_p|c_p\overset {!}{=}c$ . Given the interdependence of  $c_{p}$ ,  $\sigma_{p}$ , and  $\sigma_{\mathrm{scale}}$  ( $c_{p} = c\frac{\sigma_{\mathrm{scale}}}{\sigma_{p}}$ ), this can be achieved by setting  $\sigma_{\mathrm{scale}}$  as the inverse of the weighted average over all  $1 / \sigma_{p}$  as:  $\sigma_{\mathrm{scale}} = (\frac{1}{N}\sum_{p = 1}^{P}\frac{|\mathcal{G}_{p}|}{\sigma_{p}})^{-1}$  (please see Appendix C.2 for the derivation of  $\sigma_{\mathrm{scale}}$ ). Given  $\sigma_{\mathrm{scale}}$ , the required  $\{\sigma_1,\dots,\sigma_P\}$ , and  $c$  found through hyperparameter tuning, the individual clip norms are calculated as  $c_{p} = \frac{\sigma_{\mathrm{scale}}c}{\sigma_{p}}$ . We detail the derivation of the parameters in Algorithm 3. In Appendix G, we formally show that Scale satisfies  $(\{\varepsilon_1,\varepsilon_2,\ldots ,\varepsilon_P\} ,\delta)$ -DP). Similar to Sample, the proof is based on considering the training for all privacy groups as simultaneously executed SGMs with differing sensitivities.
For our empirical evaluation, we implement our methods in Python 3.9 and extend standard Opacus with our individualized privacy parameters and a per-privacy group accounting. We perform evaluation on the MNIST [23], SVHN [29], and CIFAR10 [22] dataset, using the convolutional architectures from Tramer and Boneh [38] for most experiments, and additionally evaluate on various language datasets and diverse (larger) model architectures in Section 4.2. To evaluate the utility of our methods, we use the datasets' standard train-test splits and report test accuracies. The training and standard DP-SGD and IDP-SGD hyperparameters are specified in Table 5 and Table 6 in Appendix D, respectively where the noise multiplier  $\sigma$  is derived with the function get_noise-multiplier provided in Opacus [32]. It takes in as arguments the specified parameters  $\delta, q, I$ , and target  $\varepsilon$ . For experimentation on individualized privacy, we assigned privacy budgets randomly to the training data points if not indicated otherwise.
4.1 Utility Improvement and General Applicability of Individualization
Assigning heterogeneous individual privacy budgets over the training dataset yields significant improvements in terms of the resulting model's utility, as showcased in Table 2. For example, by following the privacy budget distribution of Alaggan et al. [2] with privacy budgets  $\varepsilon_{p} = \{1,2,3\}$  for strong, medium, and weak privacy requirements, our Sample method yields accuracy improvements of  $1.06\%$ ,  $2.63\%$ , and  $5.09\%$  on MNIST, SVHN, and CIFAR10, respectively. On the CIFAR10 dataset, our Scale even outperforms improvement with an accuracy increase of  $5.26\%$  in comparison to the non-individualized baseline, which would have to assign  $\varepsilon = 1$  to the whole training set in order to respect each individual's privacy preferences.
The benefits of our individualized privacy assignment also become clear in Figure 1, which depicts the test accuracy of our Sample and Scale vs. standard DP-SGD over the course of training. Both our methods continuously outperform the non-individualized DP-SGD baseline with  $\varepsilon = 1$ . Additionally, the test accuracy with privacy budget distribution according to Alaggan et al. [2] ( $34\%, 43\%, 23\%$ ) is higher than the one of Niu et al. [30] ( $54\%, 37\%, 9\%$ ). This can be explained by the fact that in this latter distribution, more individuals exhibit a higher pri
Table 2: Model Test Accuracy after training with Standard DP-SGD vs our Individualized DP-SGD using Sample or Scale. The percentages of the three privacy groups are chosen according to [2] (first setup) and [30] (second setup). Additional hyperparameters are found in Table 5 and Table 6. We report the standard deviation over 10 trials.
DATASET	PRIVACY GROUPS	PRIVACY BUDGETS	DP-SGD	SAMPLE	SCALE
MNIST	34%-43%-23%	1.0-2.0-3.0	96.75±0.15	97.81±0.09	97.78±0.08
54%-37%-9%	1.0-2.0-3.0	96.75±0.15	97.6±0.11	97.54±0.09
SVHN	34%-43%-23%	1.0-2.0-3.0	83.26±0.31	85.89±0.14	85.57±0.24
54%-37%-9%	1.0-2.0-3.0	83.26±0.31	85.14±0.30	85.08±0.12
CIFAR10	34%-43%-23%	1.0-2.0-3.0	52.77±0.65	57.86±0.56	58.03±0.36
54%-37%-9%	1.0-2.0-3.0	52.77±0.65	56.83±0.39	56.65±0.49
vacy preference. With more data points choosing lower privacy protection as in Alaggan et al. [2], our individualization boosts the performance of the final trained model more significantly.
For comparison, we also report results for the standard DP-SGD with  $\varepsilon = 3$ . This corresponds to the upper bound on utility that could have been achieved by assigning the highest privacy budget to all data points—which violates the privacy requirements of individuals with strong and medium privacy preferences. We observe that our individualized methods' performance is close to this upper bound without violating data points' privacy requirements. In Appendix E, we also discuss other possible baselines (apart from standard DP-SGD) and empirically evaluate our indi
Figure 1: IDP-SGD vs DP-SGD.
vidualized methods against them. We report our hyperparameters and the respective individualization parameters (noise multipliers, clip norms, and sampling rates) obtained through our methods in Appendix D. Figure 4 in Appendix D.1 highlights additionally that this privacy parameter derivation is well calibrated: all different groups exhaust their privacy budget simultaneously when the targeted number of iterations is reached. In Appendix D.2, we also experimentally show that our methods are flexible and can extend to substantially more privacy groups which all have different budgets. $^6$  Finally, in Appendix D.6, we demonstrate how to combine our Sample and Scale method, and experimentally evaluate the joint method.
4.2 Broad Applicability of IDP-SGD
We also evaluate IDP-SGD with other (larger) architectures, for fine-tuning, and with different modalities and tasks. Therefore, we 1) fine-tune a BERT transformer [21] (bert-base-uncased) on the SNLI dataset [7] for natural language inference, 2) train a ResNet18 [16] from scratch on the CIFAR10 dataset for image classification, 3) train a linear embedding model (one embedding and two linear layers) on IMDB data [25] for text classification, and 4) train a character-level RNN (DPLSTM $^7$ , three layers, hidden size: 128) from scratch to classify surnames to languages. In Table 3, we evaluate our Sample and Scale in these setups with a privacy distribution corresponding to the first setup of Table 2, with  $\varepsilon = 5, 7.5, 10$  for the first three rows, and  $\varepsilon = 1, 2, 3$  for the last one, and a DP-SGD baseline with  $\varepsilon = 5$  or  $\varepsilon = 1$ , respectively. We tuned hyperparameters with standard DP-SGD and applied the best resulting hyperparameters to Sample and Scale. All results are averaged over 9 trials and we report average accuracy and standard deviation. Over all tasks, architectures, and modalities, IDP-SGD outperforms standard DP-SGD.
4.3 Privacy Assessment via Membership Inference
To evaluate the impact of our IDP-SGD on the privacy of individual data points, we perform membership inference based on the LiRA attack [8]. For the reader's convenience, we include a
In principle, with our methods, each data point could have its own privacy budget.
<https://opacus.ai/tutorials/building_lstm_nameclassifier>
Table 3: Evaluating IDP-SGD with other architectures, tasks, modalities, and to fine-tuning.
Architecture	Dataset	Setup	Modality, Task	DP-SGD	Sample	Scale
BERT	SNLI	Fine-Tune	Natural language inference	75.91±0.23	76.11±0.21	76.5±0.17
ResNet18	CIFAR10	Train	Image classification	47.52±0.84	48.53±0.69	48.77±0.73
Embedding Model	IMDB	Train	Text classification	72.69±0.27	73.27±0.3	73.34±0.11
Character-level RNN	Surnames	Train	Text (name) classification	60.86±0.78	65.56±0.96	66.0±1.19
description of the attack in Appendix B.5. We experiment with CIFAR10 and train 512 shadow models and the target model using Sample on different subsets of 25,000 training data points each. Results for Scale can be found in Appendix D.3. The privacy budgets are set to  $\varepsilon = 10$  and  $\varepsilon = 20$  and evenly assigned to the shadow models' training data, resulting in 12,500 training data points per privacy budget. Our target model achieves a train accuracy of  $68.46\%$  on its 25,000 member data points, and a test accuracy of  $64.89\%$  on its 25,000 non-member data points. The lower accuracy in comparison to Carlini et al. [8], who achieved  $100\%$  and  $92\%$  train and test accuracy respectively, results from us introducing DP to the training of the shadow models. Learning with DP is known to reduce model performance, particularly so on small datasets.
We depict the membership inference risk of the target model's training data per privacy budget (privacy group) in Figure 2. The dotted blue line represents the ROC curve for LiRA over all CIFAR10 data points and shows that overall, as expected by its definition, DP protects the training data well against membership inference attacks.
However, when inspecting the ROC curve separately for the two privacy groups  $(\varepsilon = 10$  and  $\varepsilon = 20)$ , we observe a significant difference between them. The privacy group with stronger privacy guarantees  $(\varepsilon = 10)$  is protected better (AUC= 0.537) than the group with the higher privacy budget  $(\varepsilon = 20$ , AUC= 0.581). To evaluate whether the difference in the LiRA-likelihood scores between the two privacy groups is statistically significant, we perform a Student t-test on the likelihood score distributions over the data points with privacy budget  $\varepsilon = 10$  vs. the data points with privacy budget  $\varepsilon = 20$ . We report  $\Delta = 2.54$  with  $p = 0.01 < 0.05$ , and hence a statistically significant difference between the two groups exists.
Figure 2: Per-Privacy Group LiRA [8] Membership Inference Success.
Experimental results for additional target models, further comparison between IDP-SGD and DPSGD, and detailed per-target model statistics are presented in Figure 9, Figure 10 and Table 12 in Appendix D.3. The results highlight that the individual privacy assignment of our IDP-SGD has a practical impact and indeed protects data points with different levels of privacy to different degrees. Additionally, the privacy risk for data points with  $\varepsilon = 10$  when training with IDP-SGD is lower than when training purely on them with standard DP-SGD. This privacy gain does not come at large expense of points with  $\varepsilon = 20$  whose privacy risk remains roughly the same (see Figure 10).
5 Comparison to Other Methods
Comparison to Individualized PATE. We compare against Individualized PATE [6] (IPATE), the only other work aiming at enabling individual privacy assignment in ML. See details of their method and the general PATE framework in Appendix B.3. For the comparison, we report the student model accuracies of IPATE for the MNIST, SVHN, and CIFAR10 datasets. The results are presented in Table 13 in Appendix D.4. We observe that IDP-SGD constantly outperforms IPATE with the same privacy budget assignment.
Synergy between Individualized Privacy Accounting and Assignment. We show that we can leverage synergies between our individualized privacy assignment and the orthogonal line of work on privacy accounting (see Section 2.2). Therefore, we extend the approach by Feldman and Zrnic [11] with our method of assigning each data point its individual privacy budget through an individualized
Table 4: Comparison between standard DP-SGD, individual accounting, and individual accounting+assignment. We present the test accuracy (\%) on MNIST for training with  $\varepsilon = 0.3$  for standard DP-SGD and individual accounting [11], while setting  $\varepsilon = 0.30$  and  $\varepsilon = 0.31$  for half of the points each when using individual accounting and assignment. Combining individualized privacy accounting and assignment further increases utility.
VANILLA	INDIVIDUAL	INDIVIDUAL ACCOUNTING
DP-SGD	ACCOUNTING	AND ASSIGNMENT
93.29 ± 0.49	93.64 ± 0.46	94.16 ± 0.23
Renyi-filter. This filter causes the point to be excluded from training once its individually assigned privacy budget is exhausted. Note that in contrast, in the approach by Feldman and Zrnic [11], all data points obtain the same privacy budget and data points are excluded from training once they reach this budget. For more details, see the description in Appendix D.5.
To assess the performance of pure individual accounting, we assign the same privacy budget of 0.3 to all 60,000 training data points in MNIST, following the experimental setup by Feldman and Zrnic [11]. We then enable the individual privacy assignment and change the privacy values to  $\varepsilon = 0.3$  for the first half of the points and  $\varepsilon = 0.31$  for the second half of data points. We empirically observe that in this low- $\varepsilon$  regime, the small change of  $\varepsilon$  for half the data points is enough to cause significant improvement in the accuracy of the final model, as we show in Table 4. Note that the differences in accuracy reported for standard DP-SGD on MNIST between Table 4 and Table 8 differ. This is because of the different privacy budgets ( $\varepsilon = 0.3$  vs.  $\varepsilon = 1.0$ ). Additionally, in Figure 3, we present the number of active data points, i.e., data points that did not yet exhaust their individual privacy budget, over training. We find that many data points are used longer during training with individualized assignment compared to using only individualized accounting. This is because the data points with a higher privacy budget exhaust their budget later and, thus, can be used longer. We are only able to run this experiment on MNIST as done in Feldman and Zrnic [11] since the method is based on full-batch gradient descent: gradients are computed over the whole dataset at once, which limits its applicability to large and high-dimensional datasets.
(a) Accounting
(b) Account-ing+Assignment
Figure 3: Number of Active Data Points Over Training.
Prior work on ML with DP guarantees assigns a uniform privacy budget  $\varepsilon$  over the entire dataset. This approach fails to capture that different individuals have different expectations towards privacy and also decreases the model's ability to learn from the data. To overcome the limitations of a uniform privacy budget and to implement individual users' privacy preferences, we propose two modifications to the popular DP-SGD algorithm. Our Sample and Scale mechanisms adapt training to yield individual per-data point privacy guarantees and boost utility of the trained models. For future work, we believe that our individualized privacy assignment should be closely integrated with a form of practical individual privacy accounting. This could enable us to obtain a fine-grained notion of individualized privacy guarantees that tightly meet the users' expectations.
We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: this includes CIFAR through the Canada CIFAR AI Chair and NSERC through the Discovery Grant. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. As team member of the Center for Trustworthy Artificial Intelligence (CTAI), Christopher Muhl was funded by the German Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection. We would also like to thank CleverHans lab group members for their feedback.
[35] Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. {Updates-Leak}: Data set inference and reconstruction attacks in online learning. In 29th USENIX security symposium (USENIX Security 20), pages 1291-1308, 2020.
[36] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[37] Peter Sörries, Claudia Müller-Birn, Katrin Glinka, Franziska Boenisch, Marian Marggraf, Sabine Sayegh-Jodehl, and Matthias Rose. Privacy needs reflection: Conceptional design rationales for privacy-preserving explanation user interfaces. *Mensch und Computer* 2021-Workshopband, 2021.
[38] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations, 2020.
[39] Florian Tramer, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski, and Nicholas Carlini. Debugging differential privacy: A case study for privacy auditing. arXiv preprint arXiv:2202.12219, 2022.
[40] Stacey Truex, Ling Liu, Ka-Ho Chow, Mehmet Emre Gursoy, and Wenqi Wei. Ldp-fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking, pages 61-66, 2020.
[41] Aiping Xiong, Tianhao Wang, Ninghui Li, and Somesh Jha. Towards effective differential privacy communication for users' data sharing decision and comprehension. In 2020 IEEE Symposium on Security and Privacy (SP), pages 392-410. IEEE, 2020.
[42] Jiayuan Ye and Reza Shokri. Differentially private learning needs hidden state (or much faster convergence). In Advances in Neural Information Processing Systems, 2022.
[43] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[44] Da Yu, Gautam Kamath, Janardhan Kulkarni, Tie-Yan Liu, Jian Yin, and Huishuai Zhang. Individual privacy accounting for differentially private stochastic gradient descent. arXiv preprint arXiv:2206.02617, 2022.
[45] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 253-261, 2020.
A Broader Impacts and Ethical Considerations of this Work