SlowGuess/NeurIPS_2023_OCR huggingface dataset:
More Privacy Groups. We present in Table 11 the test accuracy for ten privacy groups, corresponding to the ten classes of the CIFAR10 dataset when each of the privacy groups obtains a different privacy budget. We obtained these budgets by manually tuning them such that the accuracy gap between the privacy groups is minimized. We also visualize the accuracy over training in Figure 7. The figure visualizes that our methods are able to make all privacy groups converge to similar accuracies.
To evaluate IDP-SGD in the limit, we perform additional experiments with 100 privacy groups by randomly assigning MNIST training data points to one of the 100 privacy groups with respective  $\varepsilon_{p}$  sampled uniformly at random from [1, 6]. Our results in Table 9 indicate that IDP-SGD is able to handle this large number of privacy groups without any degradation in terms of convergence: both our methods, Sample and Scale, continuously outperform the standard DP-SGD baseline with  $\varepsilon = 1$  for all data points.
D.3 Additional Results for MIA
Membership inference success for a single target model of our Sample and Scale methods is shown in Figure 8. In Figure 9, we present the results over for 5 different target models for Sample. The figure highlights that over all target models, the two privacy groups' privacy risk is different: the group with higher protection  $\varepsilon = 10$  constantly has a lower AUC than the group with lower protection  $\varepsilon = 20$ . The test statistics over the different privacy groups' LiRA likelihood scores for all five target models are shown in Table 12.
In Figure 10, we also compare the MIA risk of our IDP-SGD with the risk of training with DP-SGD, using the LiRA attack. The results show that the privacy risk for data points with  $\varepsilon = 10$  when training with IDP-SGD is lower than when training with standard DP-SGD. This privacy gain does not come at large expense of points with  $\varepsilon = 20$  whose privacy risk remains roughly the same.
D.4 Comparison to Individualized Privacy with IPATE
We present the comparison between our IDP-SGD and IPATE [6] in Table 13. In PATE, accuracy refers to the student model accuracy. The results in Table 13 are averaged over three experiments for IPATE and ten runs for IDP-SGD. Note that the accuracies we report for IPATE differ from the accuracy values reported by [6] in Table 1, since they report average voting accuracy (i.e., how correct are individual teacher model votes), whereas we report the resulting student model accuracies, which
Table 10: Accuracy for Subgroups. We assess the accuracy of subgroups when their privacy budgets differ. We select a single class for a given group and assign either higher, the same, or lower privacy budgets in comparison to groups with other classes. We change the privacy budgets only for bolded classes in a given experiment while all other classes have the same privacy budget  $(\varepsilon = 2)$ .
Classes	Privacy Budget
Higher (ε = 3)	Same (ε = 2)	Lower (ε = 1)
0	67.76 ± 2.14	55.24 ± 1.98	25.76 ± 2.52
1-9	53.41 ± 2.2	54.72 ± 2.49	56.58 ± 2.29
1	80.84 ± 1.2	72.79 ± 1.6	46.09 ± 3.91
0,2-9	51.65 ± 2.28	52.77 ± 2.53	54.59 ± 2.23
2	51.31 ± 3.53	36.59 ± 3.33	9.53 ± 1.41
0-1,3-9	54.26 ± 2.84	56.79 ± 2.34	58.87 ± 2.18
3	52.88 ± 2.6	32.64 ± 1.91	6.67 ± 1.09
0-2,4-9	54.62 ± 2.42	57.23 ± 2.5	58.75 ± 2.42
4	56.99 ± 1.87	40.06 ± 2.88	9.44 ± 1.68
0-3,5-9	54.41 ± 2.21	56.41 ± 2.39	58.18 ± 2.02
5	64.11 ± 2.27	51.86 ± 2.21	15.04 ± 2.31
0-4,6-9	53.28 ± 2.16	55.1 ± 2.46	57.54 ± 2.49
6	73.54 ± 2.36	65.8 ± 4.25	40.6 ± 4.18
0-5,7-9	52.05 ± 2.21	53.55 ± 2.23	56.06 ± 2.33
7	68.22 ± 1.17	61.15 ± 1.99	41.08 ± 2.75
0-6,8-9	52.5 ± 2.43	54.07 ± 2.49	56.01 ± 2.76
8	77.39 ± 1.24	68.53 ± 2.34	37.42 ± 3.37
0-7,9	51.51 ± 2.54	53.25 ± 2.45	55.28 ± 2.37
9	72.82 ± 1.48	63.08 ± 1.9	32.79 ± 2.73
1-8	52.05 ± 2.32	53.85 ± 2.5	56.06 ± 2.52
corresponds to the final performance of the method. Note that IPATE does not apply the performance improvements suggested by Papernot et al. [34] (e.g., virtual adversarial training) or MixMatch which could increase the student model's performance.
D.5 Integrating Privacy Accounting and Assignment
The main goal behind individualized accounting is to obtain tighter privacy analysis (recall our discussion in Section 2.2). Instead of tracking a single privacy loss estimate across all data points, an individual privacy loss is kept track of for each data point. Feldman and Zrnic [11] defines a new individual privacy filter, which drops data points that exceed the individualized privacy loss from further processing. However, the same privacy budget is assigned to each data point. The individual assignment of a privacy budget to each data point is a natural extension of individualized accounting and can be directly incorporated into this framework. Therefore, a given data point has its own privacy filter and is dropped from the analysis once the filter indicates that the data point's privacy budget is exhausted.
D.6 Combining Sample and Scale
We run additional experiments to showcase the advantage of combining both our methods. The combination of Sample and Scale extends the number of hyperparameters and offers the potential to find better performing algorithms. It can be implemented as follows: obtain individual sampling probabilities as in Sample. Once a mini-batch is sampled, use the respective data points' indices and
(a) Sample (higher budget)
(b) Scale (higher budget)
(c) Sample (lower budget)
(d) Scale (lower budget)
Figure 5: CIFAR10: Accuracy Changes for Subgroups. We assess how the test Accuracy of a Class of interest changes in comparison to the Baseline Accuracy (standard DP-SGD with  $\varepsilon = 2$ ) when we, during training, assign a lower ( $\varepsilon = 1$ ) or a higher ( $\varepsilon = 3$ ) privacy budget to data points from a class (shown on the x-axis). The diagonals show that by increasing a class' privacy budget (lower privacy), their utility increases, while it decreases with the decrease of privacy budget (higher privacy). Similar results for MNIST can be found in Figure 6.
clip their gradients as in Scale. We allow to weight the different methods to arbitrary fractions (e.g.,  $50\%$  Sample,  $50\%$  Scale), and derive the privacy parameters such that all privacy groups exhaust their respective budgets at the end of the specified number of iterations. For practical evaluation, we use the MNIST dataset and the first privacy setup (0.34, 0.43, 0.23) with privacy budgets  $\varepsilon = 1, 2, 3$  and different weightings as indicated in Table 14.
The baseline (first and last rows) are equivalent to Table 2 in the main paper, while the additional three rows in between with the new results represent average test accuracy and the standard deviation over ten independent random runs over different weights assigned to the respective method.
Our results demonstrate a clear progression from the lower performing Scale approach to the better performing Sample. This aligns with expectations based on our implementation, where we combine the methods based on the parameters found for them sequentially.
Note that each of our methods has its advantages and disadvantages and the combination of both methods provides more options to balance (dis)advantages. For example, when some points' sample rates are extremely small, then the model might never see those points during training and the combined method could reduce this possibility while still improving over the Scale-only method.
(a) Sample | higher budget
(b) Scale | higher budget
(c) Sample | lower budget
(d) Scale | lower budget
Figure 6: MNIST: Accuracy Changes for Subgroups. We assess how the test accuracy of a class changes (in comparison to standard DP-SGD with  $\varepsilon = 2$ ) when we, during training, assign a lower ( $\varepsilon = 1$ ) or a higher ( $\varepsilon = 3$ ) privacy budget to data points from this class. The diagonals show that by increasing a class' privacy budget (lower privacy), their utility increases, while it decreases with the decrease of privacy budget (higher privacy).
D.7 Examining Convergence of IDP-SGD
Individualization directly affects the objective function of the training procedure. To verify that our individualized methods maintain a smooth convergence, we examine the loss history of one group of training data in the MNIST dataset and compare it with non-individualized DP-SGD.
We collected the loss values (and accuracy values) over the 4 following setups on the MNIST dataset and privacy setup 2  $(54\%, 37\%, 9\%, \varepsilon = \{1, 2, 3\})$ :
1. loss value for privacy group with  $\varepsilon = 1$  for the Sample method;
2. loss value for privacy group with  $\varepsilon = 1$  for the Scale method;
3. training with standard DP-SGD  $(\varepsilon = 1)$  solely on the  $54\%$  of data points that have the privacy budget of  $\varepsilon = 1$ , and
4. training with the standard DP-SGD  $(\varepsilon = 1)$  on  $100\%$  of the MNIST dataset.
In the following table, we depict the training loss of the privacy group with  $\varepsilon = 1$  during training:
epochs	Sample	Scale	DP-SGD (54% data)	DP-SGD (100% data)
20	0.145±0.010	0.148±0.011	0.211±0.018	0.162±0.013
40	0.116±0.005	0.121±0.006	0.187±0.014	0.143±0.007
60	0.109±0.005	0.115±0.006	0.183±0.010	0.142±0.008
80	0.105±0.005	0.112±0.005	0.184±0.008	0.142±0.006
Table 11: Per-class Individual Privacy Assignments. We manually optimize the per-class individual privacy budgets for Sample such that the model achieves the same accuracy over all classes. The resulting per-class privacy budgets yield the maximum gap  $\Delta$  between the highest and lowest accuracy level of only  $0.39\%$  for Sample, and  $0.88\%$  for Scale. For the baseline ( $\varepsilon = 3$  for all classes)  $\Delta = 2.03$  is significantly higher, highlighting that our approach can successfully minimize the accuracy gap between different privacy groups. We run the experiment on the MNIST dataset and report average per-class test-accuracies over three separate runs. See each privacy group's test accuracy over training in Figure 7.
Class	0	1	2	3	4	5	6	7	8	9	Δ
Baseline (ε = 3)	98.95	99.06	98.39	98.09	97.93	98.47	98.16	98.12	97.78	97.03	2.03
Budgets	0.75	0.5	2.0	2.6	4.1	2.1	2.05	3.0	3.1	6.1	/
Sample	98.16	98.09	98.16	97.95	98.10	97.91	97.77	97.99	98.02	97.89	0.39
Scale	98.44	98.36	98.13	98.02	97.76	98.17	97.91	97.96	97.91	97.56	0.88
Table 12: Statistical differences between Privacy Groups. We conduct a student t-test to determine if the Lira likelihood scores for data points with privacy budget  $\varepsilon = 10$  differ from the ones of data points with  $\varepsilon = 20$ . All results with  $p < 0.05$  indicate statistically significant differences. Results for Sample.
Target Model	Δ	p
1	5.16	2.49e-07
2	2.41	0.016
3	1.84	0.066
4	4.03	5.52e-05
5	2.537	0.011
The loss values correspond to the following accuracy values:
epochs	Sample	Scale	DP-SGD (54% data)	DP-SGD (100% data)
20	96.114±0.215	96.024±0.257	94.071±0.460	95.692±0.315
40	97.024±0.095	96.906±0.102	95.078±0.333	96.358±0.191
60	97.296±0.108	97.123±0.126	95.306±0.228	96.525±0.169
80	97.430±0.106	97.257±0.115	95.362±0.187	96.560±0.125
Our results indicate that the availability of more information from the data of different privacy groups in our Sample and Scale methods enables the model to learn better features. This leads to lower loss for the data points with privacy budget  $\varepsilon = 1$  than if the model was trained on only those  $54\%$  of points. The effect of including more data (i.e., using  $100\%$  of the MNIST dataset), all with  $\varepsilon = 1$ , is weaker, indicating that even the data with highest privacy requirement benefits substantially from our individualized privacy in terms of reduced loss and increased accuracy.
D.8 Runtime Analyses
We analyze the runtime of our methods in comparison to standard DP-SGD. The runtime is determined by two steps, the first one being the calculation of the individualized parameters of the methods which is executed exactly once before the training. The second one is the actual training time. For the calculation of individualized parameters, Table 16 indicates that the runtime depends on the number of privacy groups  $P$  specified for the implementation of Algorithm 2 and Algorithm 3 and the precision  $\gamma$  (see Algorithm 5). We observe a linear growth in runtime w.r.t. the number of privacy groups. Note that, as stated above, this additional runtime occurs only once with IDP-SGD, namely before training starts. Also note that when we repeat experiments with the same privacy setups, the parameters do not need to be derived every time, but can be set directly based on one initial computation. In terms of actual training time, Table 16 indicates that Sample and Scale do not
(a) Baseline
(b) Sample
(c) Scale
Figure 7: Per-class test accuracies over CIFAR10 training with Per-Class Privacy Budgets. We manually tune the per-class privacy budgets for Sample to obtain the same per-class accuracy at the end of training, see Table 11. Comparison with the Baseline (a) where all classes obtain  $\varepsilon = 3$  highlight that Sample (b) and Scale (c) successfully reduce the accuracy gap between the different classes.
significantly increase the training time in comparison to standard DP-SGD which is mainly due to their tight integration with the standard DP-SGD algorithm and the opacus library.
E Alternative Baselines
Throughout this work, we compare our methods with Standard DP-SGD which cannot take into account different privacy requirements at the same time. Thus, we consider it to apply the highest privacy protection required to all data points equally. Nonetheless, we can think of two more ways to
(b) Scale
Figure 8: True-Positive rate vs. False-Positive Rate of Lira Membership Inference Attacks Per Privacy-Budget. We follow the same setup as in Figure 2. We show the single target model for both (a) Sample and (b) Scale methods.
use Standard DP-SGD on data having heterogeneous privacy requirements. Those approaches have different benefits and drawbacks and might perform better than our chosen baseline approach in some scenarios.
E.1 Exclude Lower Privacy Groups
Instead of applying the strongest privacy protection, the deciding ML expert could entirely exclude data of low privacy groups from training for loosening the restrictions on the remaining data points' influence on model updates. In some cases, it would be worth giving up the information and privacy budgets of those lower privacy groups to achieve utility improvements. This approach performs poorly if important information is wasted, e.g., most data of one class has the highest privacy requirement.
E.2 Learn Privacy Groups Separately
It is also possible to make use of all privacy budgets, independent of their diversity, although only using Standard DP-SGD. Namely, a model can be trained on each privacy group separately one after another, whereby the corresponding lowest budget is regarded for each group. A drawback of this approach is that the model could forget its knowledge about previously learned privacy groups.
Figure 9: True-Positive rate vs. False-Positive Rate of Lira Membership Inference Attacks Per Privacy-Budget. We follow the same setup as in Figure 2. We run the experiment for five different target models and aggregate the results with the error bars for the Sample method.
(b) Scale
Figure 10: MIA on IDP-SGD vs. MIA on standard DP-SGD. In addition to the first three lines (blue, orange, and green) from Figure 9, we also added the red and pink lines. The original three lines refer to a model trained with IDP-SGD on two privacy groups (12,500 data points with  $\varepsilon = 10$  and 12,500 with  $\varepsilon = 20$ ). For the red line, we trained 512 shadow models with standard DP-SGD and 25,000 CIFAR10 data points that all have  $\varepsilon = 10$ . For the pink line, we did the same with 25,000 data points and  $\varepsilon = 20$ . Then, we performed MIA according to the LiRA attack over the two setups.
E.3 Empirical Comparison of Baselines
We empirically evaluate against these two additional baselines using the MNIST dataset. For baseline E.1, we include all data points with a privacy budget of  $\varepsilon \geq 2$  and use  $\varepsilon = 2$  as the privacy budget for training. After hyperparameter tuning, the training on the remaining data points (43%+23% and 37%+9% of the total data) yields the accuracy reported in Table 17.
For baseline E.2, we also did hyperparameter tuning and used the best noise multiplier of 2.5 for training. We trained the groups sequentially, always continuing training with the next group once the privacy budget of the previous groups was exhausted. We evaluated both starting with the privacy group that has loosest and strongest preferences (orders [3,2,1] and [1,2,3], respectively). Starting with the group that has strongest privacy requirements and ending on the group that has loosest privacy requirements yielded the best results which we report in Table 17.
In summary, we observe that our methods outperform the other baselines.
F Alternative Individualization
We present the alternative ways of individualizing privacy guarantees in DP-SGD that we considered in the design process of IDP-SGD and describe their drawbacks.
Table 13: Comparison between IDP-SGD and Individualized PATE (IPATE). We select the weighting mechanism from IPATE that performs better than the upsampling method. The Setup indicates the size (in %) of privacy groups. We present the accuracy (%) for training with  $\varepsilon = \{1.0,2.0,3.0\}$  privacy budgets, respectively to the order of the privacy groups.
DATASET	SETUP	BASELINE PATE	IPATE	SAMPLE	SCALE
MNIST	34%-43%-23%	91.17±1.25	95.27 ± 0.33	97.81	97.78
54%-37%-9%	95.74 ± 0.43	97.6	97.54
SVHN	34%-43%-23%	22.46 ± 5.19	41.45 ± 1.69	84.56	84.48
54%-37%-9%	44.64 ± 0.55	84.32	84.31
CIFAR10	34%-43%-23%	24.83 ± 1.56	33.20 ± 0.94	54.89	54.92
54%-37%-9%	35.59 ± 0.73	54.88	55.00
Table 14: Combining Sample and Scale. We combine Sample and Scale according to the weights indicated in the table and perform experiments on the MNIST dataset. The respective percentages highlight to what degree which method contributed. For example,  $0\%$  Sample and  $100\%$  Scale means that only Scale was executed (like in the results of the main paper in Table 2), while  $50\%$  Sample and  $50\%$  Scale expresses that both methods contributed equally. The results are averaged over ten independent runs.
Sample (weight in %)	Scale (weight in %)	Test Accuracy (in %)
0	100	97.78±0.08
25	75	97.75±0.10
50	50	97.80±0.10
75	25	97.80±0.10
100	0	97.81±0.09
F.1 Individual Per-Data Point Noise
Individualized privacy could, in principle also be obtained by adding different amounts of noise to different data points. Every of the  $P$  privacy group would have their individual  $\{\sigma_1, \ldots, \sigma_P\}$ . Utility improvements would result from some data points requiring smaller amounts of added noise. Note however, that in DP-SGD, while clipping is performed on a per-data point basis, noise addition is performed on a per-mini-batch basis (line 8 in Algorithm 1). Hence, there are two possibilities to implement individual noise addition: either (i) by operating on mini-batch sizes of 1, or (ii) by implementing a two-step sampling approach which first randomly samples a privacy group for a given training iteration and then applies the standard Poisson sampling to obtain the mini-batch consisting of data points from this group. While both approaches are conceptually correct, they exhibit significant drawbacks. Approach (i) first slows down training performance due to more operations requiring to be carried out on individual data points, rather than a mini-batch. Second, due to the weak signal-to-noise ratio when adding noise to individual gradients, model performance is likely to degrade. Finally, sampling cannot be performed with Poisson anymore since with Poisson sampling, it is not possible to pre-determine and specify exact mini-batch sizes, instead these depend on the outcome of the random sampling process. Approach (ii) could overcome the first two issues. However, the different groups sizes are still strictly smaller than the entire dataset and large parts of DP-SGD's degrading the tight privacy bounds obtained by privacy amplification through subsampling. The privacy amplification through subsampling allows to scale down the noise  $\sigma$  by the factor  $B / N$  (with  $B$  being the expected mini-batch size,  $N$  the total number of data points, and  $B \ll N$ ) while still ensuring the same  $\varepsilon$  as with  $\sigma$  [20]. This privacy amplification is crucial to the practical performance (privacy-utility trade-offs) of DP-SGD. Hence, by using the  $P$  privacy groups of sizes  $\{N_1, \ldots, N_P\}$  with  $N_i \ll N$ , the factors  $B / N_i \ll B / N$  for all  $i \{1, \ldots, P\}$ . This effect cancels out, or in the worst
Note that there exist other, less popular approaches to implement DP in ML than the DP-SGD algorithm, such as Differentially Private Follow-the-Regularized-Leader (DP-FTRL) which do not rely on subsampling but instead obtain tighter privacy bounds from adding correlated noise over the training iterations. However, since DP-FTRL under-performs DP-SGD for high-privacy regimes, and unfolds it advantages mainly in FL scenarios where the same data is only learned from once, or with a small number of epochs, we consider the approach outside of the scope of this work.
Table 15: Runtime in seconds to obtain privacy parameters (clip norms or sample rates) for different numbers of groups  $P$  and precisions  $\gamma$ . We report the time it takes for our methods to generate the privacy parameters for the first setup of Table 2 of the original submission (34%-43%-23%).
P	γ = 0.01	γ = 0.0001
Sample	Scale	Sample	Scale
2	3.24	0.57	5.79	1.01
8	10.34	2.51	20.22	4.78
32	34.2	10.24	70.89	19.75
128	127.78	35.13	277.52	78.85
Table 16: Runtime in seconds for our methods. We report runtime when training on the CIFAR10 dataset and the CNN architecture for the first privacy setup of Table 2 in the original submission  $(34\% -43\% -23\%)$  with  $\varepsilon = 1,2,3$ . Experiments are conducted on an A100-GPU. Runtimes are averaged over three trials and average with standard deviation is reported.
DP-SGD	Sample	Scale
403.67 ± 2.31	406.33± 0.58	414.33± 3.21
Table 17: Empirical evaluation against other baselines. We report the obtained test accuracy obtained with our two methods vs. two other baselines for individualized privacy on the MNIST dataset. Similar to Table 8, we use  $\varepsilon = 1,2,3$ . Both our Sample and Scale outperform the other baselines.
Setup	DP-SGD	E.1 Baseline	E.2 Baseline	Sample	Scale
34%-43%-23%	96.75	97.6	97.4	97.81	97.78
54%-37%-9%	96.75	97.1	97.3	97.6	97.54
case even inverts the privacy-utility benefits that should arise from assigning individual data points less noise based on their privacy preference in our individualization.
F.2 Duplicating Data Points