Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: null
review_content: I'd like to challenge the criticism from reviewers 47fc and Ewj4 that the very similar work of Tramer et al. [34] shouldn't be considered as concurrent work by the authors of this paper.

The Tramer et al. paper appeared on arxiv on March 31st 2022 (https://arxiv.org/abs/2204.00032).
This is 6 weeks before the NeurIPS abstract deadline, and 7 weeks before the submission deadline.

It is inconceivable to me that authors should be expected to quantitatively compare to, or otherwise improve upon, a paper that was clearly being written *concurrently* to this NeurIPS submission (unless we expect people to write their NeurIPS papers in <6 weeks...)

review_title: Concurrent work

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a poisoning attack that amplifies the membership inference risk of an ML model on the samples from a specific class. The authors consider two attacks: dirty-label (label flipping of clean training samples) and clean-label (perturbing clean training samples). Essentially, these attacks aim to amplify the influence of member samples of the targeted class which allows for easier inference attacks.
strengths_and_weaknesses: Strengths:

+ The synergy between poisoning and privacy leakage is interesting with profound implications and the proposed attacks are straightforward and effective.

+ Detailed ablation study to further understand the attack properties.

Weaknesses:

- It it difficult to say [34] is concurrent as it was on ArXiv for almost two months before NeurIPS deadline (and also from very visible researchers).

- The considered membership inference attack is somewhat weak, a more recent SOTA [Membership Inference Attacks From First Principles] might reduce the effectiveness of poisoning.

- Targeted attack is stealthy in the input space but many OOD detection algorithms nowadays operate on the latent space (which might be able to detect these attacks).

- No defenses considered against the proposed attack (outlier removal / differential privacy / regularization / strong data augmentation etc.)
questions: 1) Do you measure the test accuracy only on the classes other than the target class t? Seems to me that the proposed attacks will hurt the accuracy on the target class significantly, which will no longer be stealthy.
limitations: - Consider running experiments on datasets where a specific class might be more sensitive to privacy leakage (for example, a medical dataset or a demographics dataset). It is difficult to understand why class targeted attack matters if the only experiments are on CIFAR.

- Consider evaluating some standard defensive strategies from prior work against poisoning.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents data poisoning attacks for increasing the risk of membership inference (a type of privacy attack on machine learning models). The paper studies two different strategies: dirty-label attacks (i.e., label-flipping attacks) and clean-label attacks. In evaluation with image classification datasets, the paper shows that dirty-label attacks can increase the average attack success (represented by the AUC metric) by ~0.25 compared to the clean baselines with negligible degradation of accuracy.
strengths_and_weaknesses: **Strengths**
1. The paper studies an interesting problem.
2. The paper is well-written and easy to read.


**Weaknesses**
1. Prior work has already studied the claimed contributions.
2. Poor comparison with the literature on accessing privacy risks.
3. Weak evaluations.


**Detailed Comments**

The idea of evaluating the risk of membership inference under data poisoning attacks is interesting. As more and more data is collected from various sources, the privacy risks of machine learning models trained on such data is an important topic. 

**1. Contributions were shown by the prior work**

However, data poisoning for increasing privacy risks has already been initially studied by Mahloujifar et al. [1], and all the contributions (claimed from Line 41 to Line 52) have already been shown by Tramer et al. [2]. Moreover, the paper uses the techniques and tools for measuring the membership inference risks already known as meaningless by Carlini et al. [3]. Thus, I believe this paper is largely detached from the state-of-the-art privacy studies, and unfortunately, the contributions are the repetition of what we have known so far.

[1] Mahloujifar, et al., Property Inference from Poisoning, IEEE Security and Privacy, 2022.
[2] Tramer et al., Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets, Preprint, 2022.
[3] Carlini et al., Membership Inference Attacks From First Principles, IEEE Security and Privacy, 2022.

**Note:** The studies I mentioned had appeared 3-12 months before the NeurIPS submission deadline and were even accepted before then, so I wouldn’t review this paper as concurrent work.


**2. Poor comparison to the prior work**

My second concern is that the paper just combines two threat models (data poisoning and membership inference attacks) while it largely ignores important research questions in the community, such as:

RQ 1. Why does this poisoning work?
RQ 1. What are the training samples that become more vulnerable after poisoning?
RQ 2. What does it mean to increase the AUC?
RQ 3. Why are clean-label poisoning attacks important?
 (As the paper mentioned in the introduction, sanitizing the training data is not feasible.)
RQ 4. If someone wants to mitigate this attack, what can this person do?

which (those questions) are partially already answered in the prior work [2, 3].


**3. Weak Evaluation**

My last concern is that there is unclear interpretation of the results in the evaluation section:

Q1. (Line 257) I am unclear why the clean-label poisoning attack can be considered an “approximate” version of the dirty-label poisoning attack in the feature space?

As shown in visualization (Figure 5), it seems that clean-label attacks and dirty-label attacks cause a completely different impact on the models. If this is true, wouldn’t it make more sense in Sec 3 to present a single attack with different objectives?

Q2. (Line 261) I am also unclear how this paper measures the distributional differences between D_train and D_shadow. I believe it’s still a hard question to quantify the distributional differences and actively studied in domain adaption and robustness, so I don’t think we can compare.

Q3. (Line 284) I am a bit confused about the fine-tuning scenario. Is it the case where we take an ImageNet pre-trained model and fine-tune it on CIFAR10? Then why don’t the attacker make membership inference on ImageNet instead of attacking CIFAR10? Isn’t it easier to spot poisoning samples if we inject them into the training data for fine-tuning?

Q4. (Line 304) I am unclear about the connection between the presented attacks and adversarial training. Adversarial training crafts adversarial examples in each training iteration and update the model parameters, while this attack just injects a set of static poisons into the training data and trains on it.

questions: 1. Scientific comparison to the prior work [1, 2] and clarification of the scope of the contributions.
2. Clarification on the contributions of this paper considering research questions in the community.
    1. RQ 1. Why does this poisoning work?
    2. RQ 1. What are the training samples that become more vulnerable after poisoning?
    3. RQ 2. What does it mean to increase the AUC?
    4. RQ 3. Why are clean-label poisoning attacks important?
    5. RQ 4. If someone wants to mitigate this attack, what can this person do?
3. Clarification of the interpretation of the results (Questions from 1 to 4 above).
limitations: My lesser concern is that the paper conducts offensive research, contaminating the training data for increasing privacy risks but does not discuss any ethical concerns when a miscreant uses this attack. At least, I think it's okay to talk about some potential defense mechanisms, but the paper just concludes that defenses are future work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper address the problem of membership exposure, where the attacker tries to identify if a specific sample belongs to the training set. The authors propose to apply data poisoning attacks such as label flip attacks and clean-label attacks (feature collision) to amplify such membership identification. Experiments empirically verify the effectiveness of such attacks.
strengths_and_weaknesses: Strengths:

(1) This paper proposes that data poisoning attacks might help amplify membership exposure, which is novel.

Weakness:

(1) The author states in line 156 that "The key to amplifying membership exposure is to cause overfitting in the targeted class." However, I don't think this is necessarily true. From my understanding, the authors try to poison Equation (2) by making $f(\cdot)$ less accurate using simple poisoning attacks such as label flip attacks. However, the author does not show if degrading the performance of $f(\cdot)$ is related to the MI AUC. Thus why this attack is effective is not clear to me. 

(2) The methods proposed in this paper (label flip attacks, and clean label attacks using label flip) do not provide me insights into solving the MI problem and Equation (2), as the objective is not considered in the approaches which make them heuristic.

(3) The experimental section is very confusing to me. From the content, the MI attack aims to identify if a point $(x,y)$ belongs to the training set, but Table 1 shows the MI AUC of the target class. 

(4) Why is the baseline AUC so high? Looks like without poisoning the MI AUC can already become as high as 74.8%, which is itself shocking to me. Is this problem really so serious or is it illy defined?

(5) See my questions for more concerns.
questions: (1) In Equation (2), the authors propose to examine if a point $(x,y)$ belongs to the training set using a classification metric (cross-entropy loss). This metric does not make sense to me as it only determines if the point is an outlier of the distribution. Even if $M_{mem}$ is high, it only reveals $(x,y)$ is not an outlier, but does not guarantee $(x,y)$ is in D_train. I expect the authors to explain more about this.

(2) In Section 4.1, the authors split the datasets evenly into D_clean, D_test, and D_shadow. Although this assumption ensures that D_clean and D_shadow come from the same dataset, this significantly reduces the size of D_clean and might make it easier to poison. Can you examine this by making $\|\mathcal{D}_{shadow}\|$ smaller?

(3) Additionally, It seems that the poisoning budget is much smaller than $\|\mathcal{D}_{shadow}\|$, then do we need to have such a big D_shadow? 
limitations: Limitations are addressed in this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper shows that poisoning a training set can amplify the privacy leakage (under membership inference attacks) of other training points.
The paper proposes a dirty-label and a clean-label attack and evaluates them in the transfer learning setting. Both attacks significantly increase the AUC of the membership inference attack.
strengths_and_weaknesses: This paper studies a new and interesting threat model where the ability to poison a model is used to target privacy of other training points (rather than to evade the model).
The same problem is also considered in concurrent work [34], while the prior work of [18] considers weaker leakages of global dataset properties. (Calling [18] concurrent work is a bit of a stretch: even though it was only officially published this May, the paper has been on arxiv since Jan. 2021).

The paper's aim is to globally increase membership inference success rates, and it also evaluates attack success rates in an average-case sense, using the AUC.
It could be nice here to also consider worst-case membership inference metrics, as advocated in recent work by Carlini et al: https://arxiv.org/abs/2112.03570
As Figure 2 shows, the poisoning attack seems to not only increase the AUC, but to also drastically increase the attack's TPR at low FPRs, which is what really matters for a privacy attack.

My main criticism of the paper is that it only considers the transfer setting. Even the dirty-label attack which can in principle be applied in any setting is only evaluated in the transfer setting. It might be worth acknowledging this in the abstract and introduction.  
In particular, conducting clean-label attacks in the transfer setting seems a bit trivial, as one can essentially simulate the dirty-label attack in feature space (the paper does acknowledge this).

So I was actually surprised to see such a big difference in the results for the dirty-label and clean-label attacks. It is not clear to me why this should be the case: with a perturbation of eps=16/255 it should be possible to near-perfectly match the features of any other example.
The visualization on Figure 5 suggests that the attacks actually work quite differently: it seems that the dirty-label attack always uses the same "dirty" label, whereas the clean-label attack does not. Can you confirm this? And if this is indeed the case, why? Is it just because you don't have enough data points of the same class in D_shadow?

Regarding the threat model, it might be worth commenting a bit on how the assumptions behind poisoning attacks and membership inference attacks interact. In particular, to run a membership inference attack the attacker must anyhow get access to some samples from the distribution (namely the samples on which the attacker will try to infer membership). So the assumption that D_shadow exists seems perfectly natural (and the study in 5.1 may not even be needed). In fact, in practice the adversary would likely know a D_shadow that intersects with D_clean (the adversary just doesn't know which points are in the intersection, which is precisely the membership inference problem).

questions: - In the dirty-label attack, do you use the same wrong class i for each poison, or is i sampled at random for each poison? 
- Why such a big difference between the dirty-label and clean-label attacks?
- Can you show that the dirty-label attack actually works in an "end-to-end" training scenario?
- Can you show how well your MI attacks work at low false-positive rates?
limitations: N.A
review_title: null

Meta-review: metareview: Reviewers 47fC and gUo3, who are experts in this area, were very pleased by the author discussion and clarification. They appreciated that the authors will tone down their claims and reduce confusion in the final version. As gUo3 said, there are concurrent works, but that can't hurt this work -- it only shows that there is significant interest in this topic. The same reviewer did feel that the scope of the results seem somewhat weak (just transfer learning), but there is some evidence that things extend beyond that setting. If the authors wish to further improve the work, it may be worth performing further experimental evaluation. 

Note that the two other reviewers, who were negative, neither engaged with the authors nor myself when prompted to in private deliberation. Since they appear unwilling to defend their negative evaluation of the paper, I defer to the positive sentiments of the other two reviewers who did discuss privately.
confidence: Certain
Decision: Accept