Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This study focuses on the existence of samples in which membership inference is more likely to succeed and samples in which it is less likely to succeed and discusses the causes of this difference. It was generally believed that out-of-distribution samples were more likely to succeed in membership inference, but the study argues that whether the sample is in-distribution or out-of-distribution has little to do with whether the sample is in-distribution or out-of-distribution. Instead, they argue that the larger the gap between the probability that a sample is identified as the target label when trained on a training data set that contains the target sample and the data set that does not, the more likely it is that membership inference will be successful. In addition, theoretical and experimental considerations regarding this hypothesis are provided.



strength_and_weaknesses: Strength
Theoretical considerations on the success of membership inference.

Weakness
Contains unexplained or implicit assumptions
Contains gaps in the logic. Due to this gap, it s difficult to examine whether it is making a valid argument adequately
clarity,_quality,_novelty_and_reproducibility: Contains unexplained or implicit assumptions.
Contains gaps in the logic. Due to this gap, it s difficult to examine whether it is making a valid argument adequately.
summary_of_the_review: The author's argument could not be fully understood because it either assumes knowledge and understanding of existing research and assumptions contained in them or because there are gaps in the logic. The paper's claims must be structured in such a way that they can be understood on their own.

Page 4
The explanation of the privacy score is too simple.

Page 5
Why do you only include samples with large memorization values in your attempt to correlate memorization and privacy scores? Isn't it impossible to evaluate the correlation unless we also include samples with small memorization values?

Figure 2, bottom
How were these image samples selected? Are they not cherry-picked?

Page 6
How were samples in D_A  drawn? Sampled from out-of-distribution?

Proof of Lemma
What does the hybrid argument mean? 

Page 7
In Reason 1, do you assume \theta is optimal?
In Reason 2, do you assume the training procedure is probabilistic?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the relationship between label memorization and membership inference. It showed that examples with high memorization are more easily attacked. Based on this, it formulated membership inference attack where the attacker choose highly memorized examples, and show that the attack performance improved significantly.
strength_and_weaknesses: **Strength**: This paper systematically studied the relationship between memorization and MI attack performance, and also evaluated common OOD detection algorithms.

**Weakness**

1. The connection between label memorization and membership inference attack performance is known. The attack algorithm that this paper primarily use (Carlini et al. 2022a) is directly built on this relationship: it performs hypothesis testing based on the distribution of two group of shadow models to make the MIA prediction. Since the (practical estimator of the) label memorization is just the difference between the mean of the two distributions, the larger the difference between the two distributions, the easier the hypothesis testing is, and subsequently the higher the MIA performance is. So this observation, while not formally characterized yet, is essentially known in the literature.

2. This paper claims that OOD is not well correlated with MIA performance. But this is not well supported, and the main reason for the observation seems to be that the OOD detector used in the paper are not a good one. From Fig. 1, x-axis, we can see that the OOD score failed to make a clear separation between the over-represented data and under-represented (out of distribution) data.

3. The paper propose to perform MIA in a MI Game setting where the attacker choose the points being attacked according to the susceptability of them being memorized. This ended up making the MIA performance higher. I'm not an expert in security, but showing that the attackers could have high success rate when they choose to only attack the most outlier examples does not seem surprising or concerning, comparing to, e.g., showing that an attacker can achieve high rate on average across randomly sampled training examples.
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written and relatively easy to follow. 
summary_of_the_review: This paper studies the relationship between memorization and membership inference. My main concern over this paper is that the main empirical observations are essentially known, and the proposed MIA that focus on attacking the memorized examples is just a straightforward implication of the observation applied to existing MI attacks, and does not seem to bring new insights to the table.

---------------------
Post rebuttal: Thanks to the authors for the responses. After reading the response and other reviewers' comments, I'm keeping my current rating.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the connection between label memorization and membership inference.
The paper shows that examples that are considered OOD according to some empirical metrics may not be more vulnerable to MI. Instead, examples with large label memorization (as defined by Feldman) are more likely to be vulnerable to MI attacks.
strength_and_weaknesses: Strengths:
- understanding what makes data vulnerable to privacy attacks is an important problem
- comparing different ways of "defining" OOD data is valuable


Weaknesses:
- Very imprecise. Terms like "singleton" or "OOD" are never formally defined, and the conjecture in Section 4.3 and its consequences are stated in terms vague enough that the conjecture cannot meaningfully be falsified
- Circular reasoning: 
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written, but understanding its main points is difficult because many important concepts are left underspecified.
For example, the concept of a "singleton", which is central to many of the paper's arguments, is never formally defined.

Similarly, the term "OOD" is used throughout the paper without ever being clear on what its formal definition actually is. As discussed below, there isn't any single canonical definition of OOD. Instead we just have different metrics that try to capture something like OOD. But it would be worth clarifying this early on in the paper, as the paper's results have a very different interpretation in this light.

Section 4.3 suffers from vagueness the most. The given conjecture uses terms such as "singleton", "well generalized" without any proper definition. The notation "-> 1" is also not defined. It is unclear what it would mean for Strat to "tend towards 1" (my interpretation of the meaning) as there are no variable quantities here. As a result, the conjecture is essentially meaningless as one cannot hope to confirm or refute it.
summary_of_the_review: The paper's main claim is that OOD points are not more susceptible to MI, but this depends entirely how one defines "OOD". In fact, an entirely natural way of defining OOD data is simply by using the result of an MI attack.
What the paper actually shows (in Figure 1) is that some canonical OOD metrics (like MSP) are not aligned at all with privacy scores from MI attacks. But this can be interpreted in two ways: either privacy scores aren't impacted much by OOD data, or MSP is not actually a very good OOD metric!
Looking at Figure 1, it seems that the latter conclusion is most at "fault" here: indeed, in this experiment you have ground truth on what data is OOD (i.e., the under-represented data). As we can see, MSP is terrible at distinguishing the under-represented and the over-represented data. So it seems like a poor OOD metric.
In contrast, the privacy score appears to distinguish the under- and over- represented data a lot better.
There is a prior work by Carlini et al (https://arxiv.org/abs/1910.13427) that looks at the agreement between different measures of what constitutes an outlier, including privacy metrics. It would be worth discussing the relation of this work to that paper.

As figure 2 shows, the privacy score of Carlini et al and the memorization score of Feldman are more closely correlated. This is not particularly surprising though: from the definition of label memorization (which should be credited to Feldman 2020, rather than Feldman & Zhang), it is obvious that high memorization of a given point implies the ability to do membership inference on that point well.
In fact, the two metrics are essentially measuring the same thing, just at different levels of granularity: the score of Carlini et al. measures the distinguishability of the loss distributions when training on D vs D \ {i}. Label memorization takes a much coarser approach by just collapsing these distributions to a single scalar, the probability of outputting the correct label. Through this view, it is not clear why it should be surprising that these metrics are very strongly correlated.

Due to this very strong correlation, the results from section 5 also seem somewhat circular. Of course, if we focus the MI attacks on those samples where the label probabilities for members and non-members are most different, then the MI attack will work better.

So overall, it is not clear to me what conclusions to draw from this paper. There are some interesting results that seem to show that MI attacks are better at identifying OOD data compared to other metrics. But these results are not at all interpreted this way, and rather taken to imply that prior work somehow drew incorrect conclusions regarding the success of MI attacks on OOD data.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies in what situation training samples are vulnerable to membership inference attacks.
Firstly, the authors empirically argue that membership inference performance may have a weak connection with the OOD property of training data.
Then, they hypothesize that membership inference advantages may be related to how well the samples are memorized by models.
Through experiments, the authors find a clear correlation between the label memorization score of data and the corresponding membership inference attack accuracy, which justifies their claim.
strength_and_weaknesses: Strengths:

1. The paper is written clearly and the studied topic is important and interesting.

2. The empirical results are strong enough (especially Fig. 1 and Fig. 2) to justify the correlation between label memorization and membership inference.


Concerns & Questions:

1. According to the definition of label memorization score, one needs to access the learning algorithm to calculate such a score. However, in real-world attack scenarios, the learning algorithm (as well as hyperparameters) of a black-box model may be agnostic by adversaries, which would result in difficulties in obtaining memorization scores. Please discuss the practicality of leveraging memorization scores in membership inference attacks.

2. Why the conjecture in Section 4.3 is appropriate? Please justify.

3. In Table 1, every model is trained with MNIST and a subset of under-represented data. However, from Table 2, it seems that every model is instead trained with MNIST and the singletons from the under-represented data (which results in the size of the under-represented population being reduced). I think the comparison between Table 1 and Table 2 is unfair, and a more fair comparison may be that: (1) train model on MNIST + subset of under-represented data, and (2) compare the classification accuracy of the model on random under-represented data and under-represented singletons. Please comment.

4. Only MNIST is used as the over-represented dataset. I suggest using more datasets as the over-represented datasets.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The research topic is important and interesting.
summary_of_the_review: This paper studies an important and interesting problem, i.e., the relationship between data memorization and membership inference advantages. In general, the empirical results are strong enough to justify most of the claims in the paper. Therefore, I tend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper first aims to argue that success of membership inference attack is not necessarily related to a sample being out-of-distribution. It then argues that is related to label memorization in the sense defined in (Feldman, 2020). There are substantial issues with both of these parts. There is no perfect way of measuring how much OOD is a sample but instead a variety of proxy scores. So the fact that a certain proxy score does not capture the MI success accuracy does not actually disprove the intuition based on OOD (even if I tend to agree that this informal intuition is flawed). The more significant issue is that the connection between memorization and MI accuracy is known and rather obvious given the definitions (see review by eP6o for additional details). Is known in the privacy community and connections between memorization and MI are mentioned explicitly for example in Sec 4.3 of (Feldman, 2020)  https://arxiv.org/pdf/1906.05271.pdf

justification_for_why_not_higher_score: See review
Decision: Reject