The Privacy Onion Effect: Memorization is Relative
Authors: Nicholas Carlini
Matthew Jagielski
Chiyuan Zhang
Nicolas Papernot
Andreas Terzis
Florian Tramer
Venue: NeurIPS 2022 Accept (2022)
Keywords: memorization
privacy
auditing

Abstract: Machine learning models trained on private datasets have been shown to leak their private data. Recent work has found that the average data point is rarely leaked---it is often the outlier samples that are subject to memorization and, consequently, leakage. We demonstrate and analyze an Onion Effect of memorization: removing the "layer" of outlier points that are most vulnerable to a privacy attack exposes a new layer of previously-safe points to the same attack. We perform several experiments that are consistent with this hypothesis. For example, we show that for membership inference attacks, when the layer of easiest-to-attack examples is removed, another layer below becomes easy-to-attack. The existence of this effect has various consequences. For example, it suggests that proposals to defend against memorization without training with rigorous privacy guarantees are unlikely to be effective. Further, it suggests that privacy-enhancing technologies such as machine unlearning could actually harm the privacy of other users.

The Privacy Onion Effect: Memorization is Relative
Nicholas Carlini Andreas Terzis
Matthew Jagielski
Nicolas Papernot Chiyuan Zhang
Florian Tramer Google Research
Abstract
Machine learning models trained on private datasets have been shown to leak their private data. While recent work has found that the average data point is rarely leaked, the outlier samples consequently, privacy leakage. We demonstrate and analyse an Onion Effect memorization; removing the "layer" of outlier points that most vulnerable a privacy attack exposes a new layer of previously-safe points to the same attack. We perform several experiments study this effect, and understand why it occurs. The existence this effect has various consequences. For example, it suggests that proposals defend against memorization without training with rigorous privacy guarantees are unlikely to be effective, Further, it suggests that crvaacyennhnccig technologies such as machine unlearning could actually harm the privacy of other users.
frequently subject to memorization and,
Introduction
Deep learning models have been shown memorize their training data 77 property is often a privacy violation when the training data is sensitive 8 This risk incentivizes the removal of private data from model's s training set a number of scenarios. For example, a user might withdraw consent to use their data for learning, if they deem the privacy risk to be too high. Such removal interventions are increasingly formalized in legislation (e.g., the GDPR or CCPA). Alternatively, the party that trains a model (e.g.. company) may wish proactively assess the privacy risk [16 233 of different points in the training set, in order to adaptively remove the data points whose privacy is most at risk. Indeed, prior work has shown that the privacy of training data is highly non-uniform: while data points tend to be well protected on average, the empirical risk of privacy leakage (resulting from attacks) is concentrated on small fraction of data outliers 2 6 12 21 33)
29 and this
We show that removing attacks-has counter-intuitive consequences: by removing the most vulnerable data under a specific privacy attack and retraining model on only the previously safe data, a new set of examples in turn becomes vulnerable to the same privacy attack. We call this phenomenon the Privacy Onion Effect, which we define as follows:
training examples-i.e, those most at risk from privacy
Removing the "layer" of outlier points that are most vulnerable to a privacy attack exposes new layer of previously-safe points to the same attack.
In this paper, we consider canonical family of privacy attacks called membership inference at- tacks [14], which predict whether or not given example is contained the model's training set [29]. Membership inference attacks are the most commonly used privacy metric due to their simplic- ity [6, 34], generality across domains [2 22,22 27 37], and utility as the basis for more sophisticated attacks 3 We empirically corroborate this Privacy Onion Effect standard neural network models trained on the CIFAR-10 and CIFAR-100 1-00 image classification datasets [19] For example, we find
that if remove the 5,000 training samples that most at risk from membership inference, in the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

absence of any other effects we should mathematically expect this removal to improve the overall privacy by a factor of 15x, but reality it only improves privacy by factor of 2x. That is, the Privacy Onion Effect has caused this removal to be over 6x less effective than expected.
We perform several experiments that refute various potential explanations for the presence of this Onion Effect-for example, we find that the effect is not explained by statistical noise, by the reduction the training set's size, by the presence of duplicate training examples, by the model's limited capacity. Our experiments however suggest that the Onion Effect may be explained by inliers that become outliers when more extreme outliers are removed.
The Privacy Onion Effect has significant consequences for commonly-usee empirical approaches to data privacy and custody:
Current privacy auditing is unstable: is an increasingly common practice to empirically measure-or audir-the privacy risk of users, by instantiating concrete attacks such as member- ship inference [6 [33. Our work shows that such privacy auditing lacks "stability", as a user's empirical privacy risk can vary significantly due to the removal of small fraction of training
data (a similar effect has been shown under adversarial additions to the data (33).
In the future, privacy audits should be dynamically updated as the underlying training data changes. Yet even then, an audit that reveals low risk attacks might provide users with false sense of security, as the risk could drastically increase after the removal of a few other users' data.
Machine unlcarning can degrade others' privacy: Users whose privacy appears most at risk (e.g., as revealed by a privacy audit) may be the most likely to proactively request for their data to be removed from a model's training set. As show in Section [5.2] by honoring such machine unlearning requests (5), a model provider might inadvertently degrade the privacy of other users.
Related Work
Memorization and differential privacy. While some forms memorization may be desirable for learning succeed, focus unintended memorization. In this context, Feldman introduces a simple definition of memorization [[]] informally, a model memorizes a training example's 's label if removing this example from the training set significantly changes the model's probability of outputting this label. Such label memorization is provably prevented by training models with differential privacy (DP) [1 10, DP guarantees that the model's outputs are not highly affected by the addition or removal of any training example.
Training with strong DP guarantees typically comes cost in accuracy 32
Prior work offers an explanation for this tension between privacy and accuracy 4 [1. by showing that memorization necessary for high-accuracy learning certain data distributions. Examples of such distributions are
those with long tails [II,, and have been explored in several empirical studies 2 12 30).
These studies point to the worst-case nature of the differential privacy guarantee: DP provides a uniform guarantee that applies to any dataset. Instead, our work is motivated by the question of whether memorization can be prevented using privacy mechanisms that are tailored to the data points that are at risk (as such, our work relates to the broad literature on instance-spi mnstance-ppecffec differential privacy (25)). Specifically, we investigate if removing points that are easily memorized precludes the need to uniformly bound the privacy leakage of the training algorithm for all possible datasets.
Membership inference. Perhaps the most studied attack against privacy, membership inference [14, considers an adversary that aims to answer the following question: was a given example part of a training dataset? In the machine learning setting (29). the membership inference adversary is typically given access to a model's predictions with varying granularity [24] 28
34 ranging from
the full confidence vector to the label of the class with the largest confidence score 9 In this work,
we use a specific membership inference attack, the Likelihood Ratio Attack (LiiRA 6 as described
in Section 3.1 because it achieves oftte-of-thhaal attack performance across all metrics.

---

3 The Privacy Onion Effect
We now provide experimental evidence for the Privacy Onion Effect described in the introduction: by removing the first "layer" of casy-to-attack training examples and retraining the model, we expose new "layer" of training examples to the same attack.
3.1 Notation and Problem Setup
Notation. Let X = {(T; } denote a training dataset; in this paper we focus on the CIFAR-10
and CIFAR-100 datasets [99,, each with 50,000 labeled training examples. We use the notation T(X) to denote the distribution of models we would obtain by training a neural network on the dataset X. For any set E 2|M], we use the notation X, = ( ys) : E 8) to denote a subset X, C X. Let A(x, f) denote the result of rumning membership inference attack on the model f and example For example, for a model f < T(x,) ) trained on X., good attack would predict A(x, J) = 1 and only if x E Xs, and would return 0 otherwise.
The Likelihood Ratio Attack (LiRA). Given a machine learning model f <- T(x,-) trained on dataset X,- C X. LiRA first trains multiple "shadow models" f. T(x.) random subsets of X, C X. For a target example x E X, LiRA then computes the logit-gap (the difference between highest and second-highest logit) L(x,, ((a) for shadow models f that were trained on x. and the logit-gap [xx, for shadow models f that were not trained on a. Both distributions of logit- gaps modeled as univariate Gaussians. Finally, predict whether the example is contained in the training set X. of the model fr. LiRA computes the logit-gap C(x, f*) and compares the likelihood of this observed value under the two Gaussian distributions above. Whichever more likely determines if I was member or not.
Computing privacy scores. Given training dataset X can compute a privacy score for each example x E X by measuring the average Attack Success Rate (ASR) of our membership inference attack. We first fix a distribution Dxx over subsets of X (we consider the distribution obtained by picking each example x E X independently with probability 50%). We then randomly sample a subset X, f Dx, train a model fs +- T(Xx) ) on this subset, and compute the average attack success rate of an example x E X as the probability (over the random choice of subset, and the training randomness) that the attack correctly predicts whether or not x E X,. Formally, we denote this by:
ASR(r.X)
Pr f. ).-T(X,)xX,
[A
[ X.1]
We will sometimes also refer to the advantage of the attack (over random guessing), which is defined as 2- ASR 1 (i.e., an attack with a success rate of 50% has an advantage of O).
Plotting membership inference attack success rates. To visualize the performance of a member- ship inference attack. we plot a Receiver Operating Characteristic (ROC) curve which compares the attack's 's true-positive rate (TPR) and false-positive rate (FPR). A trivial random guessing will achieve TPR equal to the FPR. Stronger attacks appear further up and the left having higher TPRs for lower FPRs. ROC evaluations are consistent with established best practices 6
3.2 Our Main Experiment
Our main experimental methodology follows a simple three step process.
Compute a privacy score for each training example. Using the LiRA membership inference attack (described above) we begin by measuring the average attack success rate (ASR) for each example in the training dataset z E X. To ensure statistical validity of our results, we compute this average across 200,000 models. We use an efficient open-source training pipeline [20] to train each model in just 16 GPU-seconds (we train on 16 1000 GPUs for a total of 1000 GPU-hours). We then run LiRA on each of these models to obtain the attack success rate of every example in the original dataset.
Remove the least private examples. These privacy scores computed above allow us to sort all examples based on how easy (or hard) they are to attack. We then remove the 5,000 most-vulnerable examples from the dataset (10% of the dataset overall for both CIFAR-I0 and CIFAR-100). Consistent with previous observations, the most vulnerable examples are generally outlicrs, such atypical or
3

---

10a
10a
True Positive Rate
True Positive Rate
10
10 -2
Baseline Reality
10
Baseline
Reality
10
Ideal
Ideal
10-4
10
10
10-1
10
10
10
10
10-1
10
False Positive Rate
False Positive Rate
Figure 1: ROC curve when performing a baseline membership inference attack (blue) on CIFAR-10 10 (left) or CIFAR-100 (right), by comparing the attack's s true- true-positive rate to its false-positive rate. Artificially preventing the attack from attacking the 5, 5,000 least private examples gives a new (green) curve: by doing this the attack now succeeds 15x and 5x less often for CIFAR-10 and CIFAR- 100, respectively. However we actually remove the 5,000 least private examples and re-run the experiment (orange curve), find this does not significantly improve privacy: is 6x less effective than we would have expected. This suggests that simply removing the current at-risk outliers will cause new examples to become outliers, and not solve the problem completely.
10a
10a
Before Removal
Number of Examples
After Removal Cutoff
10-1
True Positive
10-2
103
Reality
10-1
Reality
Reality 2500
10
Reality
Reality
10a
1.0
10-f
10-1
10
10-1
10a
Attack Success Rate
False Positive Rate
Figure The Onion Effect shown as a his- togram of per-example success rates. We re- move examples with attack success rates larger than the red line. In the Onion Effect, ex- amples which remain after removal have their attack success rates increase past the red line.
Figure While we primarily investigate the Onion Effect with 5000 removed examples, re- moving even up the 25000 vulnerable examples does not significantly reduce vulnera- bility.
sometimes even mislabeled examples. Figure[00jn the Appendix visualizes the removed samples that are easiest to attack-40 we can see, these examples match the outlier examples identified by This gives new dataset of 45,000 "hard-to-attack" examples.
12
Re-compute the privacy scores of each example. Finally, we repeat the first step and train new set of models on this modified, smaller dataset of "hard-to-attack examples. The models trained on the reduced-size dataset have at most two percentage points lower accuracy than the baseline models trained on the full dataset. Using these models, we then re-run LiRA and measure success rate on the retained 45,000 points. In an ideal environment with no other confounding effects, no-easier to attack than they were in our baseline experiment,
average attack
we would predict these points where we used the entire training set.
3.3 Results
Does removing the easy-to-attack examples significantly improve privacy? Surprisingly, we find the answer is no!

---

Figure I presents prresntss the main results of our analysis with three ROC curves when evaluated on CIFAR-10 (left) and CIFAR- 100 (right). The uppermost curve (in blue) plots the baseline ROC for LiRA when attacking the full 50,000 example dataset. (This curve is consistent with prior work [6] when attacking CIFAR-10 10 and CIFAR-100. Then, the lowermost ROC curve (in green) shows the ideal setting where the attacker is limited to inferring membership for just the 45,000 least-vulnerable examples. examples are ignored when computing
That is, the adversary's s guesses on the 5,000 the attack success rate. This curve represents the attack performance (conversely, the privacy) we would expect to obtain in an idealized environment after removing the 5,000 most vulnerable outlier points-in in the absence of any other effects.
However, as the main result of our paper, we find that this idealized setting is wrong: when we actually remove the 5,000 outliers, retrain model, and re-run the membership inference attack, it is much easier to attack the remaining 45,000 examples than we predicted the idealized setting. This is shown by the middle ROC curve (in orange) in Figure T Note that the y-axis is on log-scale, and thus the difference between the idealized result and the obtained result approximately half an order of magnitude. Concretely, rate is 1.5%. By restricting the attack to targeting the 45,000 most private samples, the idealized true-positive rate would drop to 0.1%. (To compute this value, we perform a membership inference attack on every example in the dataset and select a threshold that yields a false-positive rate of 0.01%.) Then, we compute the true rue-positive rate of the attack averaged across every example in the dataset except for the 5,000 most vulnerable examples we will remove. We find that the attack has a TPR of 0.1% on the remaining 45,000 examples.
fixed false-positive rate of 0.01%, %. the baseline true-positive
However, when we actually run this experiment (i.e., we drop the 5,000 most vulnerable examples from the training set and retrain the model), the attack's s TPR on the remaining 45,000 examples only drops by roughly half, to 0.6%. Removing outliers is thus over 6x less effective at mitigating membership inference than we would have expected. We also present this effect as a histogram of per-example attack success rates in Figure -
As a brief aside, a natural generalisation of the above removal procedure is to iteratively and adaptively remove smaller numbers of examples, rather than 5,000 all at once. In the Appendix we show this procedure does not significantly impact our findings, and so for simplicity consider one-shot removal for the rest of this paper.
The remainder of this paper investigates the question: why does removing the least private examples not result in privacy-preserving model? We provide evidence of an Onion Effect that explains this surprising phenomenon: the least private examples are "outliers" which, when removed, expose a new set of examples to become outliers-thess new outliers are then (nearly) as vulnerable as the previous outliers. In consequence, removing examples that are at risk of privacy attacks does little to solve the privacy problem we just shift the issue onto a new set of outliers.
An illustrative example in SVMs. To provide intuition for this Onion Effect, it may be illustrative to consider the special case of support vector machines (SVMs), where this effect appears naturally and explicitly. SVMs are defined by a set of support cectors-training examples that lie close to the model's decision boundary. When trained without explicit privacy guarantees, an SVM's decision boundary thus leaks information about these specific training examples. Yet, if we removed these support vectors from the training set (the first "onion layer") and retrained the SVM, new set of examples would now lie closest to the model's 's decision boundary, and these examples will thus be selected as support vectors and be risk (the second "onion layer"). While deep neural networks are very different than SVMs, this paper shows neural networks exhibit a similar empirical phenomenon.
4 Potential (Incorrect) Explanations for the Onion Effect
In this section we consider various hypotheses that might explain the Onion Effect, but upon further investigation are not correct. We focus exclusively on the CIFAR-10 dataset due to the computational cost associated with the experiments we perform (this cost is primarily due to the large number of models that need to be trained).
5

---

Privacy score under second half
1.0
Correlation r=.9998
0.8
0.7
0.6
0.5
0.5
0.6
0.7
0.8
0.9
1.0
Privacy score under first half
Figure 4: The Onion Effect not due statistical noise. Our privacy metric is stable across two independent runs and has an almost perfect correlation (.9998).
4.1 The Onion Effect Not Explained by Statistical Noise the Privacy Metric
It is possible that the Onion Effect could be due to statistical noise in the measurement of privacy scores. That is, after removing the (what we believe to be) easiest-to-attack examples, new set of examples would "become" the new easy-to-attack simply due to high variance in the privacy scores. find this not case.
Suppose that the privacy measurement of each example consisted of two components
observed (z) priv (z) noise(x)
where the first factor is the "true" privacy score for this example, and the second factor is some random experimental noise. Even if the total magnitude of this noise is small, if many values have very similar (true) privacy scores, then the 5,000 least private examples that we remove might significantly change from one experimental run to the next. Therefore, when we remove these examples, we might not actually be removing the truly least private examples, but a (somewhat) random subset of the least private examples. This could explain our observation: when we remove these examples, and run our experiment again, some of the remaining examples will have higher privacy scores by chance alone. To refute this hypothesis, we show that while there does exist some experimental noise, the total magnitude of this noise is negligible and cannot by itself explain the Onion Effect.
we scatter plot each example's average attack success rate when computed set of 100,000 models, against that example's ' average attack success rate when computed over another, independent of 100,000 models. The correlation in the scores is near pear-perfece (r 0.9998), indicating that noise is not the primary cause for the Onion Effect. We also include pane to zoom in on the right hand side of the figure: observe that, even locally, there tight linear fit on the 500 examples with highest privacy score. Additionally, 4,974 of the 5,000 examples that are easiest-to-attaet are identical when computed on each of the two datasets.
In Figure 4
4.2 The Onion Effect Is Not Explained by the Modified Dataset Being Smaller
While the CIFAR-10 10 dataset contains 50,000 examples, our second dataset with the easiest- -caiest-to-attack examples removed is just 45,000 examples. In principle it is possible that the dataset being 10% smaller puts each training sample at a much higher risk of privacy leakage.
In order to refute this hypothesis, of examples, but this time remove them from a different part of the data distribution. In particular, instead of removing the 5,000 least private examples that are easiest to attack, we either remove the 5,000 examples that are most private (i.e., hardest to attack) or we remove 5,000 random examples.
perform experiment where remove the same number
Figure 5 plots the ROC curve for this analysis, and refutes the hypothesis that the Onion Effect is due to a reduction in dataset size. In fact, both our alternative configurations (removing the hardest examples, or random examples) result in nearly identical ROC curves the baseline attack on the full dataset. The idealized ROC curve when the attack is prevented from attacking the hardest-to-attact samples is completely unchanged. This makes sense because the hardest-ter tt-ttttct samples do not

---

10a
10a
True Positive Rate
True Positive Rate
10-1
10
10
10
10-3
Baseline
Baseline
Ideal (Inlier)
Ideal (Random)
10
10
Reality (Inlier)
Reality (Random)
10
10-3
10-2
10-1
10
10
10
10
10-1
10
False Positive Rate
False Positive Rate
Figure 5: The Onion Effect can not be replicated when removing either the hardest-to-attack samples (left) or random samples (right).
10a
10a
True Positive Rate
10-1
True Positive Rate
10-1
10-2
10-2
Baseline (Dedup)
Baseline
10-3
Baseline
10-3
Reality (Dedup)
Reality
Reality
Reality (+0OD Data)
Ideal (Dedup)
10-4
10-4
Ideal
Ideal
10
10
10-2
10-1
10
10-3
10-1
False Positive Rate
False Positive Rate
Figure 6: The Onion Effect remains even for datasets with artificially injected out-of-distribution outliers. We remove the 5,000 most vulnerable out- liers from CIFAR- 10 and insert 5,000 randomly labeled outliers from CIFAR-100.
Figure 7: The Onion Effect still holds after deduplicating CIFAR-I0 and removing 5, 275 duplicated images from the entire dataset.
significantly contribute to the success rate at low false- silseepositive rate 66. When we remove these examples and retrain the model, the experimentally observed results match: the attack remains exactly as effective as expected in the idcalized model. Similarly, when we randomly remove 5,000 examples the attack success rate remains nearly identical. Taken together, these experiments suggest that the Onion Effect is not be explained by change in the size of the dataset.
4.3 The Onion Effect Is Not Explained by Limited Model Capacity
One potential explanation for the observed Onion Effect is that the model has limited capacity to memorize examples, and removing memorized examples frees this capacity up for new points to be memorized. Here, we refute this explanation, by demonstrating that arbitrary outliers do not produce the Onion Effect. To do so, we take the CIFAR-10 dataset. remove the 5, 5,000 most outlier samples as we have done before, but then add back in 5,000 new outliers from CIFAR- 100 with randomly given labels. Despite these examples being extreme outliers (because the inputs are out-of-distribution and the labels random), we find in Figure that curve away from the original curve statistically significant manner. That is, the Reality (+ OOD Data) curve in Figure O and the Idealized ROC curves both measure attack success rate on the same 45,000 examples, when 5,000 outliers are also present CIFAR-10 10 have high impact on the attack success rate for the remaining 45,000 examples, while the CIFAR-10 outliers do not. This demonstrates that the specific outliers do matter, and that this effect cannot be explained by the existence of arbitrary outliers consuming model capacity.
that training on this augmented dataset does not alter the ROC
training set. However, the outliers from

---

4.4 The Onion Effect Is Not Explained by Duplicates in the Training Dataset
The CIFAR-10 10 dataset has many duplicates [26] If an example is repeated many times, it difficult to perform membership inference on any single one of these duplicates. Deduplicating the dataset will therefore increase many examples' membership inference accuracy. and might alter the results. Here we validate that this does not explain the Onion Effect. To do this we construct deduplicated version of the CIFAR-10 10 dataset by using the open source image deduplication library imagededup [17] to remove 5,275 duplicated examples from the CIFAR-10 training set (see Appendix [C for details). We then repeat our basic experiment on this deduplicated CIFAR-10 dataset and present the results in Figure 7 Interestingly, deduplication results in some examples having significantly higher attack success rate, as their duplicates the training set "mask" their contribution.
Understanding the Onion Effect
We have suggested that the Onion Effect is a result of inliers becoming outliers when more extreme outliers are removed. In this section we provide experimental evidence for this claim, and then discuss the consequences for machine unlearning.
5.1 Transforming Inliers to Outliers
In order to demonstrate that the Onion Effect is in part explained by the previous inliers becoming outliers, we perform an experiment where we take previous samples where LiRA fails and remove just a few other (nearby) training examples to make the attack succeed. In particular, to target example z', we compute the influence of removing each training example x on the privacy of x' as:
PRIVINF(REMOVERG
x') = f.+-T(Xx)xXx-xxx
E
[A(x', (A) = 1 [x Xx) g Xs):
That is, we measure the membership inference accuracy on x', averaged over all models trained without This is a natural extension of the definition of counterfactual influence proposed by (36] and similarly considered in [15] We can compute PRIVINF for all training example simultaneously with the same random partitioning of the dataset used for LiRA. After computing these scores, we remove the training samples with the highest influence on our desired target. We focus our evaluation of this algorithm on four representative sets of examples, because we find the influence effect not uniform across all examples.
Duplicates. Recall from Section 4.4 that deduplication increases many points' attack success rate, as duplicate points "mask" each other according to membership inference attacks. We consider the 5 points whose attack success rate increases the most after deduplication. Due to the masking effect, we expect the points with highest PRIVINF scores to be duplicates these points. Indeed, the duplicates are the first points removed, and the removed points explain the entire attack improvement from deduplication. That is, for these points, the PRIVINF score identifies or duplicates which, when removed, increase membership inference advantage the targeted points by average of 42 percentage points.
Second onion layer. To validate that PRIVINF scores are not just a good deduplication tool, we now consider the 10 points with the highest increase in attack advantage after removing the 5,000 most vulnerable points in training (the initial onion experiment)- but excluding those with duplicates in the training set. If we remove all 5,000 most vulnerable points (the onion's first layer), the attack advantage on our 10 target points increases by 0.38 on average (from 0.42 to 0.8). For each of these 10 points, we find that instead removing only the 25 points with highest PRIVINF scores similarly increases membership inference advantage by 0.22 (i.e., 56% of the increase seen after removing all 5,000 points). Thus, we find that the Onion Effect a local effect rather than global effect: individual "second-layer" outliers are masked by only small number of first-layer outliers, We show one example
target image along with the first-layer outliers that mask it in Figure g This actually helps us better interpret our results from Section where we injected out-of-distribution CIFAR- 100 images to the CIFAR- 10 raining set: CIFAR- 100 examples are not "close enough" to the CIFAR- 10 points to mask them, leading to very little impact their vulnerability.
'Counterfactual memorization also includes a term for subsets with E S. but we find including this term makes influence estimate slightly worse.
8

---

(a) Target
(b) Removed Samples
Figure 8: Removing 10 carefully chosen examples (drawn on the right) from the model's training set increases the membership inference accuracy of the target (shown on the left) from 66% to 81 %.
Random points. Having validated that the removal of a few outliers could transform some points into new outliers, now show that this effect is indeed limited to "second layer" of outliers, and does not hold uniformly across the training set. Specifically, we find that if we target 10 randomly chosen points, then removing the 25 points with highest PRIVINF scores (for each target) only increases membership inference advantage by 0.08 of average (from 0.34 to 0.42).
Safe points. Finally, we study whether we can target "safe" points, which we define as points with (initially) lower than 2% membership inference advantage. Prior work has shown that it is possible to add poisoning data to a model to turn such points into outliers [33]. Here, we investigate whether removing data points can also turn the initially safest points into vulnerable outliers. For 10 targeted safe points, we find that removing the 25 points with highest PRIVINF scores (for each target) increases membership inference advantage by 0.12 on average (from 0.02 to 0.14). Yet, for some initially safe points, the attack advantage grows as high as 0.22 (an increase of 20 percentage points), nearly as high as for the (non-duplicate) points that are most impacted by the Onion Effect.
5.2 Consequences for Machine Unlearning
Privacy attacks like membership inference are commonly employed as primitives to measure progress in machine unlearning 3 13] where model needs to forget all it has learned from training point, The results we presented above have
e.g., due to legislation promoting the geggt-to-be-forgttte several implications for machine unlearning.
First, our results corroborate prior findings 31 which show that metrics based on membership inference should not be relied upon to measure, i.e., audit, how well model has unlearned point. If we did so, we might (erroneously) conclude that some points do not require to be actively unlearned, the membership inference attack has negligible advantage them. This could influence an individual user not to request their data to be unlearned; alternatively, the model owner might also decide that certain unlearning requests do not need be actively acted upon. However, given the Onion Effect, a data point that is currently safe from membership inference could later become vulnerable if the underlying dataset changes. This could be the case, for instance, if other users make unlearning requests. This leads to a contradiction: the point initially believed to be unlearned (because membership inference attacks fail on it) would later be deemed not be properly unlearned.
The experiment above with PRIVINF scores also suggests a form of adversarial unlearning: given target individual's 's training point, an attacker could adaptively select other training points to be unlcarned, to maximize the success of membership inference on the targeted individual.
6 Conclusion
Our experiments have several consequences for applied machine learning privacy.
Membership inference attacks only audit specitic (dataset, model) tuples. In order to answer the question "is this model private?", researchers have suggested running empirical auditing analyses based on membership inference [33. Our work suggests this approach may be flawed unless performed carefully. The results of any individual privacy audit should be restricted to the exacr specific dataset that the model was trained on, and should be seen as a stable audit under any changes to this dataset. Thus, unless we expect the training set to never change, such audit results could mislead practitioners into believing that model is private when, in fact, it may not be in the future on different dataset. Interestingly, removing examples does not need to cause the Onion Effect, increasing attack success rate remaining points. Removing examples can cause the reverse effect,

---

decreasing attack success rates, if the removed examples are poisoning examples as proposed by [3] or no effect at all, as in the arbitrary outlier experiments from Section 4.3 It is an interesting question for future work to investigate which effect dominates when removing examples in other settings.
Instance-specific privacy-enhancing strategies. At present, the dominant strategy prevent mem- orization of training data is to strictly modify the training algorithm (e.g., to guarantee differential privacy). Our work can be seen as studying potential alternative: modifying the training dataset.
While the Onion Effect suggests that defenses relying on removing examples from a training dataset will be ineffective, it stands to reason that inserting more extreme outliers might be able to make the original outliers more private. Unfortunately, the negative result from Section 3 suggests that adding arbitrary out of distribution data will not be effective. Alternatively, the results from Section 4.4 might seem to suggest that duplicating data points could help improve these points' privacy, but this reasoning is unfortunately also incorrect. Because duplicate would be included in the training set and only the original example was in the training set, duplication actually makes membership inference easier: the adversary now needs to distinguish between models with and copies of given training example, rather than between and 1. Nevertheless we believe it is an interesting open question whether it would be possible modify a training dataset improve privacy.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communicutions security, pages 308-318, 2016.
[2] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in Neural Information Processing Systems. 32, 2019.
[3] Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer. Machine unlearning: Linear filtration for logit-based classifiers. arXiv preprint arXiv:2002.02730 2020.
[4] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar, When is memo- rization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Anmual ACM SIGACT Symposium on Theory of Computing, pages 123-132, 2021.
[5] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463-480. IEEE, 2015.
[6] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP). IEEE, 2022.
[7] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
[8] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Berbert-Voss,, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650) 2021.
[9] Christopher A Choquette -hoo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label- only membership inference attacks. In International Conference on Machine Learning, pages 1974, PMLR, 2021.
[10] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[11] Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
10

---

[12] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2899 2020.
[13] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. arXiv preprint arXiv:2010.109881 2020.
[14] Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. Resolv- ing individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS genetics, 4(8):e 1000167, 2008
[15] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622, 2022.
[16] Matthew Jagielski, Jonathan Ullman, and Alina Oprea, Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33:22205-22216, 2020.
[17] Tanuj Jain, Christopher Lennan, Zubin John, and Dat Tran. Imagededup. bttts://ggthie com idealo/mmageeedup 2019.
[18] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterizee models. arXiv preprint arXiv:2002.03206, 2020.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[20] Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry. Ffev: an optimized data pipeline for accelerating ml training.
[21] Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated [White-Box] } membership inference. In 29th USENIX Security Symposium (USENIX Security 20), pages 1605-1622, 2020.
[22] Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2081-2093 2021.
[23] Sasi Kumar Murakonda and Reza Shokri. ML privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning, arXiv preprint arXiv:2007.09339, 2020.
[24] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), pages 1-15, 2018
[25] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 75 84, 2007.
[26] Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint 110.::2203.14749 2021.
[27] Apostolos Pyrgelis, Carmela Troncoso, and Emiliano De Cristofaro. Knock knock, who's there?
membership inference on aggregate location data. arXiv preprint 1708.06770 2017.
[28] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages 5558-5567. PMLR, 2019.
[29] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEFE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.

---

[30] Vinith M Suriyakumar, Nicolas Papernot, Anna Goldenberg, and Marzyeh Ghassemi. Chasing your long tails: Differentially private prediction health care settings. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 723-734, 2021.
[31] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In Proceedings of 7th IEEE European Symposium Security and Privacy, Genoa, Italy, 2022.
[32] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). arXiv preprint arXiv:2011. 1. 1660, 2020.
[33] Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their secrets. arXiv preprint arXiv:2204.00032 2022.
[34] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 3/st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[35] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[36] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2021.
[37] Minxing Zhang, Zhaochun Ren, Zihan Wang. Pengjie Ren, Zhunmin Chen, Pengfei Hu, and Yang Zhang. Membership inference attacks against recommender systems. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 864-879, 2021.
A One-Shot vs Iterative Removal
If the definition of outlier depends the other examples that present, it is possible that removing 5,000 outliers in one pass might somehow impact the results-because we might miss some new set of examples that become outliers only after we have trained the model for a subset of epochs.
Here, we experiment with an iterative removal approach. In particular, starting with the 50,000 example dataset, we first run LiRA to find the easiest to attack samples then remove just the top 100 (instead of 5,000) examples. We then repeat our experiment on the remaining 49,900 example dataset; this gives us new set of 100 examples be removed from this dataset, which we do, giving us a dataset of 49,800 examples. We repeat this procedure 50 times until we are left with a dataset of 45,000 examples. We then plot in Figure 9 the ROC curve when attacking this dataset, and find that the results are almost completely identical to our initial experiments where we directly removed 5,000 outliers in shot. Upon investigation, 80% of the examples removed by the 1-shot approach are also removed by the 50 shot layer-by-layce approach.
find that the reason this occurs because
12

---

10a
True Positive Rate
10
10-2
10
Baseline
One-Shat
Layer-by-Layer Removal
10
Expected
10
10-3
10-2
10-1
10
False Positive Rate
Figure 9: The Onion Effect not result of the outliers changing after each removal step. We repeatedly iterate 50 steps of identifying outliers and removing the top- 100 outliers. This ends up also removing 5,000 outliers, and performs identically to the baseline configuration where we remove all 5,000 outliers in one shot.
Visualization of Easy-to-Attack and Hard-to-Attack Examples
Figure T0 visualizes the easy attack examples from CIFAR-10 training set, according to their privacy scores. The examples that are vulnerable to membership influence attack are generally outliers memorized by the models. The results consistent with previous work that identify outliers and memorized examples in neural network learning [e.g. 12 12.
Figure 10: The easy attack (top rows) and difficult attack (bottom 3 rows) CIFAR-10 examples from each of the 10 classes. Each column shows images from one class, with the top 3 rows randomly sampled from the top 100 most vulnerable examples, and the bottom rows randomly sampled from the 100 least vuinerable examples.
13

---

C Details of CIFAR-10 Deduplication
We use the open source image deduplication library imagededup 17 github. com/ http:/ddaalllamaaaee htm.eeddd/mg to deduplicate the CIFAR-10 10 training set. Specifically, we use the Convolutional Neural Network based detection algorithm with a threshold of 0.85, which ended up removing 5,275 duplicated images from the training set. We have manually inspected a random set of duplicated clusters identified by the software to verify the correctness. Figure [1 visualizes one of the largest duplicated clusters identified.
available at https://
31306
33387
:15
331.2
33534
10927
1992
20079
12300
21370
23437
22443
22475
2240
25463
2014
27720
27192
25528
231
2152
2020
10042
10521
1:402
12441
4942
1724 Mi
Figure 11: Duplicated images from the CIFAR-10 training set identified by the open source image deduplication library imagededup [17]. The number on top of each image is its index in the original CIFARIO training set ordering.
14


Reviews:
[{"review_id": "fIAwEgdUZ6t", "reviewer": "Reviewer_t45T", "summary": "This paper examines the impact of removing records that are most vulnerable to a membership inference attack on the privacy of the remaining records. Given a machine learning model trained on a private dataset, one way to make it more “privacy-preserving” without randomizing the training algorithm could be to identify records at risk, remove them from the dataset, then re-train a model on the remaining records. The paper shows that some of the remaining records become more at risk and investigates potential causes for this phenomenon.", "questions": "I believe further analysis is needed, for instance reporting in the paper/appendix the following:\n- Summary statistics (e.g. histogram) of the Attack Success Rates for the 50,000 records. \n- Error bars for the ASR estimates (if applicable).\n- Overall attack accuracy/AUC. Can the Onion effect also be noticed using coarser metrics (e.g. accuracy) rather than TPR for low FPR? \n- An explanation for the necessity of training 200,000 models - this is a great computational cost, orders of magnitude above what is used in previous work [6] (256 models). I understand this to be the main reason for not extending the experiments to other settings; why is it needed? \n\nThe paper could define more precisely inliers/outliers and how they relate to “easy-to-attack” and “hard-to-attack”. The current usage of the term is quite confusing, e.g., “inliers becoming outliers when more extreme outliers are removed” (lines 259-260).\n\nMinor points:\n- Why is the Duplicate analysis (line 274) carried out on 5 points while the others on 10 points? Can the analyses be extended to more points?\n- Fig. 4 (left) and the corresponding paragraph (241-247) are hard to understand. Why are only partial results shown, i.e. why are the Baseline (45,000 + OOD) and Ideal (45,000 + OOD) results omitted? It seems to me (but I could be wrong) that the text explaining the figure contradicts the results.\n- On Fig. 3 (left), why is Reality (inlier) lower than the others? Is the gap statistically significant? The text does not seem to acknowledge the gap.\n- Using the same scale on Fig. 1 for CIFAR10 and CIFAR100 would make it easier to visually compare the gaps.\n- What criterion guides the choice of 5000 (and not, say 1000 or 500) vulnerable records to be removed? Is there some statistical reason for this?\n- In the legend of Figure 7, does “Expected” correspond to the main paper’s “Ideal”? If so, it would help to make the naming consistent. \n- The use of consistent colors across the different figures, e.g., blue, orange, and green to denote the same setup, would make it easier to follow the paper and to compare the results of different figures.\n- Will the authors release the 200,000 shadow models? This could be a useful resource for the community and would allow to replicate the results.", "limitations": "The authors make relevant recommendations to practitioners.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper takes the study of the impact of outliers on MIA a step further by identifying, measuring, and studying potential causes of the “privacy onion” effect.\n- The study of potential causes is principled and well-explained.\n- Relevant recommendations are made to practitioners.\n\nWeaknesses:\n- The causal analysis is limited to one dataset (CIFAR10), and all the analyses are limited to one attack and one target model. To support the paper’s broad, general claims, I believe further evidence is needed that this is indeed a general phenomenon. The same applies to the causes identified on CIFAR10, LiRA [6], and one specific target model: are these insights applicable to other settings? In particular, the outlier analysis (Sec. 4.3) is quite surprising and it would be useful to replicate the finding using different outliers or a different dataset.\n- Lack of reproducibility: important details are missing from the paper, such as the target model architecture and (train/test) accuracy, the number of training samples for fitting the Gaussians vs number of test samples for plotting the ROC curves, etc. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QhrzrIN-6wA", "reviewer": "Reviewer_Mmkb", "summary": "The paper addresses the role of relative memorizations on 'outlier data' in adversarial learning (membership inference attack).\nSpecifically, the authors first empirical demonstrate that removing the outlier points that are most vulnerable to a membership inference attack does not bring adequate improvement on the performance. Next, the authors investigate this 'Onion Effect' and provide some heuristics explanation. Finally, they provide several suggestions for current learning frameworks based on the 'Onion Effect'.", "questions": "Please see the comments in the Weakness/potential improvements  section above.", "limitations": "Please see the comments in the Weakness/potential improvements  section above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Merits: \n\n- This paper is well written and organized.\n- The concept of 'Onion Effect' is interesting.\n- The implications towards current privacy mechanism are interesting and useful.\n\n\n\n\nWeakness/potential improvements \n\n- Where are the precise characterizations/derivations of those important factors mentioned in Lines  38 - 41?\nAll the experimental results used for comparison in the later sections are based on those factors. Please explicitly specify them.\n\n- What is the intuition of the score function used in the paper？ Will any other score function lead to similarly results as observed in the paper?\n\n- Further evaluations on other types of data, e.g., text\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QE7Dj4Q-GMy", "reviewer": "Reviewer_8q8r", "summary": "This paper empirically demonstrates an effect they name 'onion effect' on classification for CIFAR-10 and CIFAR-100 datasets, where removal of the 'outliers' (top most *exposed* training data samples) results in increased memorization of other next layer exposed samples which is unlike what would be expected, a higher overall decrease in memorization. In this paper exposure/memorization is measured through a likelihood ratio based membership inference attack. The authors do extensive analysis to pinpoint/cross out possible reasons behind this phenomenon, and demonstrate that one possible explanation could be the fact that removing most exposed samples changes the 'support vectors' from those samples to some other closer ones, thereby making them more exposed. They also demonstrate that noise, model capacity, dataset size and duplicates aren't the reason behind this phenomenon. ", "questions": "\n1. The experiments seem very compute intensive, specially given the number of models that need to be trained (1000 GPU hours), are there any easier/more efficient approximates of the likelihood ratio attack that can be run which are more efficient? The main reason behind this question is that this paper covers the onion effect for CIFAR classification task, and verifying this for larger transformer language models would need much more compute. \n\n\n2. I wonder if this effect still exists on tasks other than classification (or classification with more labels), like language modeling or other generative models, as it seems like one important reason behind it is the discriminative nature and how the decision boundary (in the SVM example) falls on certain points and then moves with samples being removed. I wonder if this effect would still be so strong in different tasks, as there are more labels (like in language modeling).\n\n3. As mentioned above, I am curious if the authors tested their hypothesis on pre-trained models too to see if the same thing happens there. ", "limitations": "The only 'limitation' I see is the compute intensiveness of the attack, which is mentioned in the paper too. The authors do adequately discuss the consequences of the effect they have discovered, and possible ways to address it. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The phenomenon that the authors have observed is new (to the best of of my knowledge) and  relevant,  as sample deletion and model unlearning are becoming concerns with new regulations. \n\n2. The experimentation is thorough for the given task, as in they have covered all the reasons that would come to mind for why this happens and experimented with them. The influence metric is well designed to test out their hypothesis. The story that the paper tells is very coherent and the narrative is very easy to follow. \n\n\nWeaknesses:\n\n1.  The paper only looks at a simple classification task. Although this is a good proof of concept, it leaves me wondering if the phenomenon happens in other scenarios. \n\n2. I think the effect of 'pre-training' is overlooked here. I understand that it might be outside the scope of the paper, but as the pre-train fine-tune paradigm is being applied more, I think seeing if this effect happens there as well is important. My guess is the pre-training could add many more samples to the 'SVM' (in the example), and then removing samples might not really move the boundary much. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9IKFfwlJdGK", "reviewer": "Reviewer_EK8F", "summary": "This paper shows removing easiest-to-attack examples makes some other examples easier-to-attack. \n\nThe authors define a privacy score to measure the difficulty of one example being successfully attacked. The privacy score is the average success rate of the Likelihood Ratio Attack (LiRA) on one example, which is computed over many models trained on subsets that contain the target example. \n\nTo demonstrate the Privacy Onion effect, the authors first remove some of the training data from CIFAR-10 and CIFAR-100 based on the privacy score. Then they re-compute privacy scores on the remaining data and find the LiRA membership inference attack achieves higher success rate on the remaining data.\n\nThe authors further discuss several intuitive hypotheses and how they fail to explain the Onion Effect. Finally, the authors show the Onion Effect is a local effect, i.e., removing a small number of carefully chosen examples can increase the attack success rate on a specific target example.\n", "questions": "1.Does the Privacy Onion Effect have infinite layers? Despite some of the examples experiencing higher privacy risks, the whole dataset becomes safer, i.e., the Reality line in Figure 1 is underneath the Ideal line. Now that removing 5000 examples improves the privacy by a factor of 2x, what if you remove 25000 examples? A plot showing the privacy improvement versus the number of removed datapoints would be nice. \n\n2.Removing outliers may not necessarily transform inliers into outliers. In Section 5 you show the privacy risk of one example will increase if its ‘neighbor’ examples are removed (Figure 6). So, for the Privacy Onion Effect to occur, there is an implicit assumption that ‘outliers’ in the dataset also have neighbors. What if an outlier does not have any neighbors? CIFAR-10 has a relatively high data quality so the number of ‘pure outliers’ may be small.  If you remove a small number of datapoints from an extremely noisy dataset, e.g., large training corpus for big language models, will the Privacy Onion Effect still occur?\n\n3.(Minor) You train 200000 models to ensure statistical validity. Is such a large number of models necessary? What will the results be if you only use 1000/5000/10000 models? Provide some discussion on this will ease the workload of reproducing the results and save unnecessary computations.", "limitations": "Please see the questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\n1.This paper is very well written. I enjoy reading it.\n\n2.The findings in this paper are new and important to the community. The Onion Effect shows it is necessary to re-evaluate the privacy risk when training data changes, which guides future works on data removal for privacy protection. It also poses new challenges for machine unlearning.\n\n3.The authors explore several possible explanations for the Onion Effect. The analysis further demonstrates the Onion Effect is a non-trivial finding and will help the community to better understand this problem.\n\n4.Findings in this paper are supported by solid experiments, which are presented in an easy-to-understand fashion.\n\n**Weaknesses:** \n\nPlease see the questions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept