Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper examines the impact of removing records that are most vulnerable to a membership inference attack on the privacy of the remaining records. Given a machine learning model trained on a private dataset, one way to make it more “privacy-preserving” without randomizing the training algorithm could be to identify records at risk, remove them from the dataset, then re-train a model on the remaining records. The paper shows that some of the remaining records become more at risk and investigates potential causes for this phenomenon.
strengths_and_weaknesses: Strengths:
- The paper takes the study of the impact of outliers on MIA a step further by identifying, measuring, and studying potential causes of the “privacy onion” effect.
- The study of potential causes is principled and well-explained.
- Relevant recommendations are made to practitioners.

Weaknesses:
- The causal analysis is limited to one dataset (CIFAR10), and all the analyses are limited to one attack and one target model. To support the paper’s broad, general claims, I believe further evidence is needed that this is indeed a general phenomenon. The same applies to the causes identified on CIFAR10, LiRA [6], and one specific target model: are these insights applicable to other settings? In particular, the outlier analysis (Sec. 4.3) is quite surprising and it would be useful to replicate the finding using different outliers or a different dataset.
- Lack of reproducibility: important details are missing from the paper, such as the target model architecture and (train/test) accuracy, the number of training samples for fitting the Gaussians vs number of test samples for plotting the ROC curves, etc. 

questions: I believe further analysis is needed, for instance reporting in the paper/appendix the following:
- Summary statistics (e.g. histogram) of the Attack Success Rates for the 50,000 records. 
- Error bars for the ASR estimates (if applicable).
- Overall attack accuracy/AUC. Can the Onion effect also be noticed using coarser metrics (e.g. accuracy) rather than TPR for low FPR? 
- An explanation for the necessity of training 200,000 models - this is a great computational cost, orders of magnitude above what is used in previous work [6] (256 models). I understand this to be the main reason for not extending the experiments to other settings; why is it needed? 

The paper could define more precisely inliers/outliers and how they relate to “easy-to-attack” and “hard-to-attack”. The current usage of the term is quite confusing, e.g., “inliers becoming outliers when more extreme outliers are removed” (lines 259-260).

Minor points:
- Why is the Duplicate analysis (line 274) carried out on 5 points while the others on 10 points? Can the analyses be extended to more points?
- Fig. 4 (left) and the corresponding paragraph (241-247) are hard to understand. Why are only partial results shown, i.e. why are the Baseline (45,000 + OOD) and Ideal (45,000 + OOD) results omitted? It seems to me (but I could be wrong) that the text explaining the figure contradicts the results.
- On Fig. 3 (left), why is Reality (inlier) lower than the others? Is the gap statistically significant? The text does not seem to acknowledge the gap.
- Using the same scale on Fig. 1 for CIFAR10 and CIFAR100 would make it easier to visually compare the gaps.
- What criterion guides the choice of 5000 (and not, say 1000 or 500) vulnerable records to be removed? Is there some statistical reason for this?
- In the legend of Figure 7, does “Expected” correspond to the main paper’s “Ideal”? If so, it would help to make the naming consistent. 
- The use of consistent colors across the different figures, e.g., blue, orange, and green to denote the same setup, would make it easier to follow the paper and to compare the results of different figures.
- Will the authors release the 200,000 shadow models? This could be a useful resource for the community and would allow to replicate the results.
limitations: The authors make relevant recommendations to practitioners.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the role of relative memorizations on 'outlier data' in adversarial learning (membership inference attack).
Specifically, the authors first empirical demonstrate that removing the outlier points that are most vulnerable to a membership inference attack does not bring adequate improvement on the performance. Next, the authors investigate this 'Onion Effect' and provide some heuristics explanation. Finally, they provide several suggestions for current learning frameworks based on the 'Onion Effect'.
strengths_and_weaknesses: Merits: 

- This paper is well written and organized.
- The concept of 'Onion Effect' is interesting.
- The implications towards current privacy mechanism are interesting and useful.




Weakness/potential improvements 

- Where are the precise characterizations/derivations of those important factors mentioned in Lines  38 - 41?
All the experimental results used for comparison in the later sections are based on those factors. Please explicitly specify them.

- What is the intuition of the score function used in the paper？ Will any other score function lead to similarly results as observed in the paper?

- Further evaluations on other types of data, e.g., text


questions: Please see the comments in the Weakness/potential improvements  section above.
limitations: Please see the comments in the Weakness/potential improvements  section above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper empirically demonstrates an effect they name 'onion effect' on classification for CIFAR-10 and CIFAR-100 datasets, where removal of the 'outliers' (top most *exposed* training data samples) results in increased memorization of other next layer exposed samples which is unlike what would be expected, a higher overall decrease in memorization. In this paper exposure/memorization is measured through a likelihood ratio based membership inference attack. The authors do extensive analysis to pinpoint/cross out possible reasons behind this phenomenon, and demonstrate that one possible explanation could be the fact that removing most exposed samples changes the 'support vectors' from those samples to some other closer ones, thereby making them more exposed. They also demonstrate that noise, model capacity, dataset size and duplicates aren't the reason behind this phenomenon. 
strengths_and_weaknesses: Strengths: 
1. The phenomenon that the authors have observed is new (to the best of of my knowledge) and  relevant,  as sample deletion and model unlearning are becoming concerns with new regulations. 

2. The experimentation is thorough for the given task, as in they have covered all the reasons that would come to mind for why this happens and experimented with them. The influence metric is well designed to test out their hypothesis. The story that the paper tells is very coherent and the narrative is very easy to follow. 


Weaknesses:

1.  The paper only looks at a simple classification task. Although this is a good proof of concept, it leaves me wondering if the phenomenon happens in other scenarios. 

2. I think the effect of 'pre-training' is overlooked here. I understand that it might be outside the scope of the paper, but as the pre-train fine-tune paradigm is being applied more, I think seeing if this effect happens there as well is important. My guess is the pre-training could add many more samples to the 'SVM' (in the example), and then removing samples might not really move the boundary much. 
questions: 
1. The experiments seem very compute intensive, specially given the number of models that need to be trained (1000 GPU hours), are there any easier/more efficient approximates of the likelihood ratio attack that can be run which are more efficient? The main reason behind this question is that this paper covers the onion effect for CIFAR classification task, and verifying this for larger transformer language models would need much more compute. 


2. I wonder if this effect still exists on tasks other than classification (or classification with more labels), like language modeling or other generative models, as it seems like one important reason behind it is the discriminative nature and how the decision boundary (in the SVM example) falls on certain points and then moves with samples being removed. I wonder if this effect would still be so strong in different tasks, as there are more labels (like in language modeling).

3. As mentioned above, I am curious if the authors tested their hypothesis on pre-trained models too to see if the same thing happens there. 
limitations: The only 'limitation' I see is the compute intensiveness of the attack, which is mentioned in the paper too. The authors do adequately discuss the consequences of the effect they have discovered, and possible ways to address it.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper shows removing easiest-to-attack examples makes some other examples easier-to-attack. 

The authors define a privacy score to measure the difficulty of one example being successfully attacked. The privacy score is the average success rate of the Likelihood Ratio Attack (LiRA) on one example, which is computed over many models trained on subsets that contain the target example. 

To demonstrate the Privacy Onion effect, the authors first remove some of the training data from CIFAR-10 and CIFAR-100 based on the privacy score. Then they re-compute privacy scores on the remaining data and find the LiRA membership inference attack achieves higher success rate on the remaining data.

The authors further discuss several intuitive hypotheses and how they fail to explain the Onion Effect. Finally, the authors show the Onion Effect is a local effect, i.e., removing a small number of carefully chosen examples can increase the attack success rate on a specific target example.

strengths_and_weaknesses: **Strength:**

1.This paper is very well written. I enjoy reading it.

2.The findings in this paper are new and important to the community. The Onion Effect shows it is necessary to re-evaluate the privacy risk when training data changes, which guides future works on data removal for privacy protection. It also poses new challenges for machine unlearning.

3.The authors explore several possible explanations for the Onion Effect. The analysis further demonstrates the Onion Effect is a non-trivial finding and will help the community to better understand this problem.

4.Findings in this paper are supported by solid experiments, which are presented in an easy-to-understand fashion.

**Weaknesses:** 

Please see the questions below.

questions: 1.Does the Privacy Onion Effect have infinite layers? Despite some of the examples experiencing higher privacy risks, the whole dataset becomes safer, i.e., the Reality line in Figure 1 is underneath the Ideal line. Now that removing 5000 examples improves the privacy by a factor of 2x, what if you remove 25000 examples? A plot showing the privacy improvement versus the number of removed datapoints would be nice. 

2.Removing outliers may not necessarily transform inliers into outliers. In Section 5 you show the privacy risk of one example will increase if its ‘neighbor’ examples are removed (Figure 6). So, for the Privacy Onion Effect to occur, there is an implicit assumption that ‘outliers’ in the dataset also have neighbors. What if an outlier does not have any neighbors? CIFAR-10 has a relatively high data quality so the number of ‘pure outliers’ may be small.  If you remove a small number of datapoints from an extremely noisy dataset, e.g., large training corpus for big language models, will the Privacy Onion Effect still occur?

3.(Minor) You train 200000 models to ensure statistical validity. Is such a large number of models necessary? What will the results be if you only use 1000/5000/10000 models? Provide some discussion on this will ease the workload of reproducing the results and save unnecessary computations.
limitations: Please see the questions.
review_title: null

Meta-review: metareview: Reviewers found the paper to be thought provoking and relevant and would be of broad interest to the community. 

confidence: Certain
Decision: Accept