Amplifying Membership Exposure via Data Poisoning
Authors: Yufei Chen
Chao Shen
Yun Shen
Cong Wang
Yang Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Data poisoning
membership inference
data privacy

Abstract: As in-the-wild data are increasingly involved in the training stage, machine learning applications become more susceptible to data poisoning attacks. Such attacks typically lead to test-time accuracy degradation or controlled misprediction. In this paper, we investigate the third type of exploitation of data poisoning - increasing the risks of privacy leakage of benign training samples. To this end, we demonstrate a set of data poisoning attacks to amplify the membership exposure of the targeted class. We first propose a generic dirty-label attack for supervised classification algorithms. We then propose an optimization-based clean-label attack in the transfer learning scenario, whereby the poisoning samples are correctly labeled and look "natural" to evade human moderation. We extensively evaluate our attacks on computer vision benchmarks. Our results show that the proposed attacks can substantially increase the membership inference precision with minimum overall test-time model performance degradation. To mitigate the potential negative impacts of our attacks, we also investigate feasible countermeasures.

Amplifying Membership Exposure via Data Poisoning
Yufei Chen
Chao Shen'
Yun Shen Cong Wang²
Yang Zhang*
Xi'an Jiaotong University
City University of Hong Kong
AetApp *CISPA Helmholtz Center for Information Security
Abstract
As in-the-wild data are increasingly involved in the training stage, machine learning applications become more susceptible to data poisoning attacks. Such attacks typically lead to test-time accuracy degradation or controlled misprediction. In this paper, investigate the third type of exploitation of data poisoning increasing the risks of privacy leakage of benign training samples. To this end, we demonstrate set of data poisoning attacks amplify the membership exposure of the targeted class. We first propose a generic dirty-label attack for supervised classification algorithms. We then propose
optimization-based clean-label attack in the transfer learning scenario, whereby the poisoning samples are correctly labeled and look "natural" evade human moderation. We extensively evaluate our attacks on computer vision benchmarks. Our results show that the proposed attacks can substantially increase the membership inference precision with minimum overall cest-time model performance degradation. To mitigate the potential negative impacts of our attacks, we also investigate feasible countermeasures.
Introduction
Training data are the most critical ingredients machine learning, which are sometimes regarded as a new type of fuel in the era of artificial intelligence [30]. Over the last decade, training data collection and preservation have received growing concerns [10, 291, among which two focal agendas arise.
One is data corruption caused by data poisoning attacks, posing serious threats to training data integrity. Data poisoning attacks exploit the common practice to (usually automatically) collect training data in the wild, e.g., from the Internet. This practice opens doors for attackers to inject malicious data the training time to manipulate model behaviors the test time. Consequently, models trained with poisoned training data suffer from either accuracy degradation [7, 19], targeted misclassification [34, 55], or backdoor implantation [32, 50].
The other is data leakage caused by privacy inference attacks, mainly violating training data confiden- tiality. Ideally, machine learning model learns generalizable knowledge of the training data, rather than details of specific data points. However, previous studies show that conventional learning algo- rithms unintentionally remember sensitive information. Such information various privacy inference attacks, such as membership inference attacks [33, 38]. Essentially, data poisoning is training time attack, while privacy inference is performed at the testing time. Despite intensive research efforts, most existing works study them separately. It remains unclear whether these two attacks be integrated cause more severe damage machine learning models.
later be inferred by
In this paper, we aim to advance the research frontier on the connection between data integrity and confidentiality. To this end, we propose set of poisoning attacks to increase the precision of membership inference attacks, which is widely adopted as standard tool to measure privacy Ieakage in statistical analysis [15, 40). In particular, first show simple but effective dirty-label poisoning
Code is available at https: htttp:://gttubee htw.///fchheee944//sssooiigg
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In a poisoning attack, an attacker crafts and injects a set of malicious training samples 2 Proison (i.c., poisoning dataset) into the benign dataset Deteun (i.e., clean dataset). In the training stage, the model holder executes the machine learning algorithm on the full poisoned dataset Durain = Detean U D Puuisoo attack that is generic for supervised classification applications. We then demonstrate clean-label poisoning attack applicable to the transfer learning scenario. has two significant advantages: (1) no requirements on the labeling process and (2) natural appearances to human moderation. Consequently, our proposed clean-label poisoning attack challenges the case where automatically crawling data from the Internet has become a common practice.
Our contributions are as follows: (I) We reveal an underexplored data poisoning attack threatening the training data privacy. In particular, we demonstrate how to amplify membership inference exposure of a specific class with only slight impacts on the model performance. (2) In generic supervised learning settings, we introduce naïve dirty-label poisoning attack by modifying the labels of the poisoning samples. (3) the transfer learning setting, we propose an optimization- saasd based clean-label poisoning attack, wherein for each poisoning sample, we make imperceptible modifications without changing its label. It presents a more practical attack example since does not require control over the labeling process. It is also more stealthy attack as the contents of poisoning instances "look similar" to the natural ones. (4) We conduct extensive empirical studies to investigate the impacts of various factors, including datasets, architectures, poisoning budgets, and learning setups. Our results show that our proposed poisoning attacks are able to increase the membership exposure obviously with just limited influences on the victim model's performance on testing samples. (5) At the end, we also consider several potential countermeasures and investigate their effectiveness.
2 Background
In this work, we focus on deep supervised classification in the computer vision field. That is, given training dataset Diran composed of input-label pairs (xr, y), deep classifier f (a; 0) with model param- eters 0, and learning objective L (f; 0), y). the learning process aims to find set of model param- eters 0* to minimize the learning objective on Drrin ie., 0 arg min > Da L (f(x;0) 0).y).
2.1 Data Poisoning
to obtain the trained model f(; )²). In the inference stage, fa:; 8*) tends to show unexpected behaviors on targeted inputs (z, y) E Diarget. Most existing literature poses the poisoning attack as a bi-level optimization problem [8, 16]:
D poison arg min
Σ
Aff(z;))),
D
(E.N)
(I)
s.l. 0" arg min
Σ ((f(x;0),))
( Liiaa
where A the adversarial objective of the poisoning attack. Typically, there are three adversarial objectives: (1) in the accuracy degradation case, Dinrget contains all testing samples and A - [x; 0*), : (2) in the targeted misclassification case, " Drarget contains samples expected to be misclassified into the target class t and A = [(;(x; 8²), t): (3) in the backdoor implantation case, Duaget involves samples with a trigger 8, where inputs embedded with the trigger are to be classified into the target class and A L(f (z 4 8; ()),)))
Based on the attacker's capability, poisoning attacks can be further grouped into two categories:
Darty-label Poisoning. Most classical poisoning attacks require the control of the labeling process, where the attacker is allowed to modify the labels in Dpcison [19, 42, 48]. Such an attack is called dirty-label poisoning. Despite promising attack performance, darty-label poisoning becomes impractical in supervised machine learning scenarios for two main reasons. First, in reality, it is common practice that only unlabeled data (e.g., images and videos) are scraped by crawlers and then labeled by human moderators. Second, machine learning developers usually utilize anomaly detection algorithms to filter out wrongly labeled

---

samples. Therefore, there exists minimal chance that the data samples with modified labels are preserved in the dataset.
To overcome the shortcomings of dirty-labor poisoning. recent studies propose clean-label poisoning attacks [34, 53, 55]. A clean-label poisoning attack satisfies two properties. First, the labels of poisoning samples remain unchanged in the poi- soning process. Second, to be inconspicuous, each poisoning sample 2p visually resembles a natural sample Iin which is always constrained by a - distance || - zullp < c.
Clean-label Poisoning.
Our Setup. In this work, we investigate data poisoning under newly identified and underexplored adversarial objective: privacy leakage amplification. In particular, we focus on one of the most representative privacy inference attacks: membership inference attack [33, 38J. We start from the dirty-label poisoning setup, which establishes a baseline and cornerstone for more advanced attacks. Then, we investigate attack methods in the clean-label poisoning setup to improve practicability. Note that there are concurrent works that propose to use poisoning attacks to cause privacy leakage [26, 43].
2.2 Membership Inference Attack
In a membership inference attack, an attacker aims to infer whether a specific sample (a, y) belongs to the training dataset Dimin at the test time [24, 33, 38]. Unintended membership exposure causes catastrophic privacy loss for individuals. For example, in the real world, a data sample æ can be a clinical record or individual. Membership inference attacks enable the attackers to infer whether this clinical record or individual has been used to train a model associated with a certain disease. As such, these attacks are widely adopted as basic metrics quantify privacy exposure in statistical data analysis algorithms [13, 23]. Henceforth, we use membership inference attacks to demonstrate how to exploit data poisoning amplify privacy leakage in this paper.
Based on the attacker's capability, membership inference attacks can be grouped into two categories:
Black-box Membership Inference. In this case, the attacker distinguishes members and non-members only using model outputs [17, 37, 38). This case is generic to most machine learning contexts. There two attack strategies in general. The first strategy is model- based [38], where the attacker builds multiple shadow models to mimic the victim model, then utilizes them to construct dataset with member/nnon-member labels, and finally trains a binary classifier to predict member/non-memberr The second strategy is metric- based [33, 391, where the attacker compares a designed metric Mmem- such as correctness or entropy, with predefined threshold T to infer if sample belongs to the training dataset.
White-box Membership Inference. In this case, model parameters 6 or even intermediate training information such as gradients 2g are observable by the attacker [22, 27, 27, Such capability provides additional information supporting inference attacks, which is usually achievable in collaborative learning settings. Most white-box attacks are model-based, as the attacker needs to access the internals of deep models to extract model-specific features.
ac
Our Setup. Our evaluation is carried out in the metric-based black-box membership inference setting. We adopt the following metric proposed by Song al. [40] to measure the membership exposure:"
Mmem - (1) (J(z)))
Σ log (1 - f(x))) f(x); isy
where f()); refers to the confidence value of label j. Equation 2 simultancously considers the correctness and entropy metric. In the training stage, the learning algorithm continuously fits the training samples, by decreasing the entropy loss (i.e., the learning objective) and increasing the confidence score of the correct label. As a result, a member is likely to produce a lower Mmem than non-member. To gain holistic view of the membership exposure, rate (TPR) and false positive rate (FPR) of the membership inference attack by varying the threshold T, and plot the ROC curve. Then we adopt the AUC (area under the ROC curve) score to measure membership exposure. In general, a higher AUC score means a higher risk of membership exposure, as we can find threshold 7 with high TPR and low FPR. It is notable that the AUC score is an average-case metric, which hardly effectively reflects the worst-case privacy [9]. Therefore, we also
caleulate the true positive
For simplicity, we use the notation f(x) to represent the trained model f(;; 0").
3

---

report the TPR when the FPR is low (1% in our case), which works as compensation for the AUC metric to indicate the worst-case privacy."
2.3 Threat Model
Before diving into our attack design, we first elaborate on the threat model considered in this paper.
Attack Goals. The first goal of the attacker is increasing the chance of leaking the membership of training samples within a targeted class. At the same time, the attacker attempts to make the poisoning attack as stealthy as possible. As such, the second goal of the attacker is generating poisoning samples that have limited impacts the model performance for untargeted classes and are indistinguishable from natural samples.
Attacker Capabilities. We assume the attacker has the basic capabilities set up by existing data poisoning and membership inference attack games. The attacker can craft poisoning samples and inject them into the victim's 's clean dataset Delean However, there exists a poisoning budget Upaison to limit the amount of training samples ( (Ppoison| S posson < [Pelaan) D) We assume the attacker owns a shadow dataset D shadow to craft poisoning samples, which contains natural samples from the same distribution with Delean Moreover, the attacker cannot modify poisoning data labels in the clcan-label poisoning setting. But assume the attacker knows the feature extractor (i.c., encoder) used by the victim model. This assumption is practical since developing models with public pretrained feature extractors is common practice in existing clean-label poisoning literature [34, 53].
Remark. In our setup, the clean dataset Petean unknown by the attacker, which is different from most existing data poisoning games [7, 19, 26).
3 Attack Methodology
Although Equation 1 establishes generic framework for poisoning attack, it is unsuitable for case. First, classic solutions for the optimization problem posed by Equation 1 require differentiation to the inner loss minimizer, which is intractable models with high complexity. Second, although meta-lcarning algorithms have been proposed to solve the bi-level optimization problem in the deep learning setting [16, 52, 54], they lead huge computing costs, limiting the practicability of the attack. Third, in our threat model, Detean is unobservable, making Equation no longer applicable to our attack. Facing these obstacles, we explore heuristic strategies achieve the attack goals.
3.1 Dirty-label Poisoning Attack: A Naïve Approach
We start our investigation in the darty-label poi- soning setting. This case has more limitations in the real world as discussed in Section 2. But it relaxes the attack constraints and helps to verify the feasibility of our attack strategy.
x
x
X
x
X
x
x
The key to amplifying membership exposure is to cause overfitting in the targeted class. We propose the following attack strategy:
Clean Model
Dirty-label Poisoned Model
Our Dirty-label Poisoning Attack. Given the label of the target group 1, the shadow dataset Dshudow, and a poisoning budget Upaison. the poi- soning dataset constructed by the following steps: (1) seleet all samples (x,t) E D Dabadow with label E; (2) for each sample (z, t), randomly
Figure 1: Conceptual illustration of the decision
boundary of models trained without/with our dirty- label poisoning attack. To achieve a low training accuracy, the model has to learn a more compli- cated decision boundary, making the learning pro- cess more susceptible to overfitting.
change the label to another class of t; (3) preserve at most époison samples (x, i) to compose Dpoison-
*We also evaluate our poisoning attacks with stronger membership inference attack [9] the CIFAR-10 dataset. The experimental results show that our poisoning attacks can also help improve the membership inference accuracy by a stronger membership inference attacker. We recommend interested readers to find more results in the appendix. In the main body of the paper, our experiments only assume the weak attack used by [40], has lower computational costs and is more practical.

---

Intuitively, only learning general concepts/features is insufficient discriminate the clean samples class t from poisoning samples, since, in fact, they share similar features. To minimize the training loss, the victim model has to learn more specific features of each sample. As conceptually illustrated by Figure I, the generalization performance degrade. Figure 2 gives an example to show how our poisoning attack improves the membership exposure of the airplane category for an Inception V3-based CIFAR- 10 classifier. In this example, assume the numbers members and non-members are both 10,000 and Upuisam=1,000
10P
True Positive Rate
10
10"22
Clean Model
class tends to
Poisaned Model
Random Guess
10-3
10
10-1 False Positive Rate
10-2
10"
Figure Membership inference against the airplane class of CIFAR-10 dataset. The poisoning budget is set as Dpoison 10% x (Deeaa J. The AUC score, i.e., the area under the ROC curve, is increased from 0.6917 to 0.9255, indicating that our dirty-label poison- ing attack effective.
Remark. We do not pose extra assumptions to the training process in this case. Our dirty-label poi- soning attack generic to all supervised learning scenarios, including end-to-end learning and transfer learning. Similar to the concurrent work [43], we also the classical label flipping strategy [7] to carry out the dirty-label attack.
3.2 Clean-label Poisoning Attack: A Stealthy Approach
In this part, we aim to improve our poisoning attack to amplify membership exposure while attaining stealthiness. Following the basic setup in [34], consider the transfer learning setting. In transfer learning, we assume the victim model is composed of two parts: pretrained feature extractor g(-) to extract high-level features from the input æ and newly-trained classifier c(-) to predict labels.
The key to our clean-label poisoning to exploit one vulnerability of deep feature extractors: slight change in the input space may cause significant change in the feature space. We propose the following attack strategy.
Our Clean-label Poisoning Attack. Given the label of the target class t, the shadow dataset Dahadow a poisoning budget Opaison: and the adopted feature extractor g(.), the poisoning dataset is constructed by the following steps: (1) from Dshalow. select a base sample (Ttuse t) with label and sample (x., v) where y * t; (2) find x. that close to x in the input space (I|=* x xrlp S c) and close to
Tbase in the feature space (g(x") = g(Prase)) and insert (x" y) to D poison: (3) repeat step (1) and (2)
until [pooison| =
Upoison
The intuition of our clean-label poisoning attack is straightforward: we actually mount dirty-label poisoning attack in the feature space. That is, the poisoning sample (x* is close to (Zbase t), clean sample from class t, in the feature space but labeled as y. We formalize step (2) as constrained optimization problem:
x² arg min min
[g((()) ((xraas))l2,ss
|| xl100 S and x' e X
where X' refers the input space. For instance, for a normalized RGB image input, a valid pixel value in one color channel should be within [0, 1]. For ease of optimization, we adopt Lg-norm to measure the distance the feature space. Following the convention of adversarial attack work, we adopt Lx to measure the distance in the input space. In our case, the constraint can be expressed as x' [minn m, where Fiin max(0. (0, x c), Frux = min(1, x +c.
Optimization. We adopt the variable substitution method introduced by [11] to solve Equation 3. We introduce a new variable w and let
tanh( w) 2(x' Fmin
Tmax
Xmin
Based on the fact that lanb(w) E 1- 1). we can prove that the constraint x' E [Tminn Tima]] holds in Equation 4. Plugging Equation 4 into Equation 3, we can obtain
5

---

cat
cat
horse
deer
ship
cat
cat
horse
deer
ship
Figure 3: Examples of poisoning samples against the airplane category. Left:: Dirty-label poisoning samples. Right: Clean-label poisoning samples (e=16/255). We provide the label of each sample on the top. For the clean-label poisoning samples, they look like natural ones.
w² = arg min |lg (0.5 (tanh(w) + (T Fminn) ) + Tmin) (Xbuse) l
(5)
and x' =0.5(tanh((er (w") () - -mii) + Tmin
which poses an unconstrained optimization problem that can be simply solved by mainstream gradient optimization methods. In our implementation, we first initialize variable to with æ by Equation 4 and iterate the optimization process 1,000 times, using Adam optimizer with the learning rate 0.01.
Figure illustrates some examples generated by our proposed two poisoning attacks. We can see that the poisoning samples produced by our clcan-label poisoning method look natural to human eyes, which are hard to detect by manual moderation.
Evaluation
4.1 Experimental Setup
Datasets. In experiment, we split each dataset into three portions: the clean training dataset Delean containing the members, the testing dataset Duss containing the non-members, and the shadow dataset D shudow for generating poisoning samples. We follow the same setup with [38], where (Delean| = D esst
D shaddow |. and each of them does not overlap with others. Additionally, we set the three datasets be balanced for the simplicity of evaluation among each class. We adopt five datasets for our experiments, including (1) MNIST [1] that contains 60,000 handwritten digits from to 9. (2) CIFAR-10 [2] that contains 60,000 images from 10 classes. (3) STL-110 [3] that contains 13,000 labeled images from 10 classes. (4) Celeb [25] that contains 202,599 face images annotated by 40 attributes. In our experiment, we train two-class classifier to predict the Charming attribute; (5) PatchCamelyon [46] that contains 327,68) 6,0680 images to predict the presence of metastatic tissue. For MNIST, CIFAR-1O, CelebA, and PatchCamelyon dataset, we set | 110,0000 while for STL-10, we set (Deteam 4,000. Each sample is transformed into 96 x 96 x 3 RGB image.
Models. We five pretrained models provided by Tensorflow(v2.5.2) Xception, ResNet18. MobileNetv2, V3 and VGG16. For each model, we remove the fully connected (FC) layers to build up the feature extractor g(-). Then, add two FC layers on top of the feature extractor to form c(-): one layer with 128 hidden units using the Tanh activation function, followed by an output layer using the Softmax activation function. We fix the feature extractor and train the newly added FC layers with the Adam optimizer, with the learning rate of 10 and batch size of 100.
Poisoning Setup. For each learning task, mount the poisoning attack against each class. We |Dieaal To make our poisoning attack stealthy to the victim.
set the poisoning budget ópoison as #classes*
we assume 'D proison is evenly distributed among all classes with each having approximately #elosses poisoned data samples. For instance, for CIFAR-10 classifier, we set 6poison = 1, 000, where only 100 samples for each class in D Ppoison Note that this assumption means that there are #elasses samples in D Dpcison for the target class t. For the clean-label poisoning dataset, set the perturbation constraint ε = 16/255.
bpsiss
b clean
Equipment. Our experiments were conducted on deep learning server, which is equipped with an (ntel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 128GB RAM, and four NVIDIA GeForce RTX 3090 GPUs with 24GB of memory.

---

Without Poisoning
Dirty-Label Poisoning
Clean-Label Poisoning
AUC FPRE
Test Ace.
MI AUC
FPREF
Test
AU 1PROFPR-19
Test Acc.
MNIST
Xeeption
538+t.022
1.22+=0.56%
.999
697=002
4.12+1.4%
-18 007
6277t0022
1.50==0.72%
9244.005
InceptionV3
546:2-029
1.01
928
791 026
1.2%%
902. 002
6744
1.34
9104.00 005
VGG16
536+.021
1.24+0.55%
954
740+ 029
1.7%%
934 004
604 033
0.94+0035%
9434.002 002
ReaNet50
525-2.021
1.22::0.62%
.967
721 + 054
3.68:::1.6%%
946: 00V
583- 022
1.38:100.33%
9634. 002
MobileNNeV2
$2911.099
0.04.
.759
3.91111.06%
937.1.004
666
1.60.10.58%
.9491.00M
CIFAR-10
Xception
64222.07
1.21=
.768
893
1.0%%
735: 004
868 032
2.46=t0.5688
.7384.00 004
InceptionV3
.3332.057
1.35::0.46%
.677
935 015
7.284:2..3%
648: 005
827- 046
1.44:10..5%%
.663 001
VGG16
619:1.049
1.12:10.40%
1899.001
4.60:11.266
179.1
869:1.019
2.69:10.7688
.7831.000
ResNet50
597-1.035
1.06-10.29%
848
1.001.015
7.3811..77%
832.1 003
861 04042
1.73.10.77%
.38.1. 002
MobileNerV2
6024.050
6.20:20.066%
.822
916
4.15
813-
836
2.03-1-0633%
820- 002
STL-10
Xception
.78:2.043
1.50:10.911%
90641.02 021
6.831:2.92%
836-1.000 004
3.90:1:1.33
.38-1.00
V3
.696.1.09
1.57.10.78%%
.758
940 020
12.80.18.094)
.755 1 006
809 1.
1.60.1..449
.750 002
VGGI6
596+.040
1.00::0.66%
.875
895
8.07:166369
8424
858
3.80 71%
8464 005
ResNet50
572+.081
1.65==0.79%
.897
931 015
11,28+10.288 28%
875 005
860
1.83+1.444
883 003
MobileNerV2
558:1.03
1.12:10.92%
.955
.900 019
3.35%
906.1 004
82244: 042
2.08 1.2%%
.1611.004
CelebA
Xception
644-1.011
1.011:0.01%%
.752.1
1.90
693:1.002
7110::
2.61:1:0.99%
.6991.005
InceptionV3
748:1.019
1.10:10.28%
.687
849 008
2.701
.655.1.000 000
7599:: 056
1.58:10.30%
36111.001
VGG16
71111.019
1.26:10.06%%
.724
.747 14471000 005
2.29:1..35%
684.1.003 003
773.10008 005
1.37.10.11 17%
.8011.003
ResNet50
571+-000
0.89-10.09%
.744
680
1.92::0.04%
698 007
616 049
1.13-00. 13%
.7124.000
MobileNetV2
.80-2.009
1.21::0.11%
.750
883- 002
2.22+:0.04%
704: 002
742 031
1.44:1:01169
696:2.006
PatchCamellon
Xception
.564.1.033
1.18.10.04%
.847
.678.1
.20
2.12.10.22%%
.99..1.002
.64.1.007
1.65
.16.1.001
I3
66174-008
0.94:10.0%
.832
739 029
2.38:10.02%
.74: 008
627: 038
1.13:10.05%
800-1.001
VGG16
538:2.003
1.15=2005%
862
623 016
1.69:
842+ 004
593+ 004
1.25 11%
838 001
ResNe150
543:1.005
1.35=0.11%%
2.29%
1.47:10.111%
8624.000
MobileNetV2
565.1.000
1.04.10.18%
.890
728 1. 029
1.791.0.07%
81911.000 000
6741 001
1.17.10.05%
842.1.007
Table I: Membership inference (MI) results for the target class and test-time accuracy Diest for models not poisoned, poisoned by dirty-label attacks, and poisoned by clean-label attacks, respectively, We run the attack and evaluation over each class, and we report the average value with standard deviation for each metric.
4.2 Results
Table reports the evaluation results proposed two poisoning attacks. In general, we can observe that our attacks obviously increase the AUC score of the membership inference against the target class f, with slight testing accuracy decay. For instance, for the clean-label poisoning attack against the STL-10 Xception-basee 0.578 to 0.876 on average, while the testing accuracy decreased from 0.857 to 0.838 on average. The results show that the poisoned samples resemble the clean samples and the performance of the poisoned models degrades subtly on the testing dataset. Consequently, there are chances that the clean-label poisoning attack evades the manual inspection, when the classification performance for each class does not get carefully examined.
classifier, the membership inference AUC increased from
Another observation is that the dirty-label poisoning attack causes more significant membership exposure than the clean-label poisoning attack. A potential explanation of this phenomenon is that our clean-label poisoning attack can be considered as an "approximate" version of the dirty-label poisoning attack in the feature space.
Ablation Study
5.1 Study 1: Impact of Shhadow
In our prior study, we assume the attacker can obtain a shadow dataset Dshadow with the same distribution of the victim's 's clean training dataset Detean- However, the model developer may hide the training dataset information to fortify privacy and intellectual property (IP) protection. In this ablation section, examine how the Dahadow affects the attack performance.
In our experiment, we assume the attacker aims to poison the airplane and cal categories against the CIFAR-10 10 classifiers respectively. For cach target class t, we select 1,000 samples from in
7

---

O (airpiane)
3 (car)
A MI AUC
TPRETPRRI'S
A Test Acc.
MI
A TPR-11
A Test Acc.
Xcepiion
-0.80%
065
0.00%%
-006
Inception
066
0.00%
-013
045
0.60%
-007
VGGI6
0.30%
-016
-009
-0.10%
-003
ResNet50
.019
-0.50%
-009
-017
-0.10%
-007
MobileNerV2
.099
-0.20%
.000
.000
-0.10%
-001
Table 2: Dirty-poisoning attack results when Delean comes from CIFAR-IO and Dshadow contains 1,000 instances from the target class of STL-10. We report the membership exposure difference and testing accuracy difference between the poisoned model and the clean model.
Without Poisoning
Dirty-Label Poisoning
Clcan-Label Poisoning
Acc.
TPR Acc.
TPR
InceptionV3
1619= 053
1.32::05575
.622
9564 008
15.17:6642%
8454..00 002
6184.054
1.20=00.43%
863 001
MobileNetV2
553= 026
1.27=00770
.906
820 021
4.76+1.17%
8864..00 002
5564.024
1.11=0.1%%
909=
Xception
573- 030
1.08::0.51%
.898
923 007
9.02422.11%
888:1.002
5814.030
1.26=0.37%
898 001
VGG16
1.04110.38%
90411.010
13.98.15.07%
622.1.060
8844
ResNet50
584 030
1.26-100:33%
.903
942.4 009
13.37:1-4.40%
884 002
592:4.032
1.30220666%
901
Table 3: Dirty-label and clean-label attack results fine-tuned CIFAR-10 10 classifiers (bpoison 1, 000). We run the attack and evaluation over each class, and we report the average value with standard deviation for each metric.
STL-10. In this ablation study, start with the darty-label poisoning attack to estimate the upper bound performance of our clean-label poisoning attack. Table 2 reports the attack performance. Our results show that the attack performance declines when the attacker's shadow data has different distribution from the actual clean dataset.
5.2 Study 2: Impact of l ppoison
We then study how the size of poisoning samples affects the attack performance. Figure 4 depicts the
0.95
0.90
membership exposure under dirty-label poisoning
0.85
attacks with different poisoning budget paison Un- surprisingly, as we gradually decrease the poisoning budget, the membership inference AUC score de-
0.80
Xception
InceptionV33
0.70
VGG16
creases as well. Yet, is still possible to achieve a significant membership exposure increase even with a small poison budget. Take ResNet50 for instance, when Upoison is 100, i.e., the amount of poisoning sam- ples is only 1% of that for the clean samples, we can improve the membership inference AUC score from 0.5971 to 0.7076 on average.
0.65
ResNet50
0.60
MobileNetV2
0.55
1000
Epolson
Figure 4: Membership exposure under differ- ent Opoison- We plot the average MI AUC on the CIFAR-10 dataset.
5.3 Impact of Fine-tuning
In the transfer learning scenario, there are also chances that model developers fine-tune parameters of feature extractors [14, 36, 47]. The fine-tuning operation is supposed to make the transfer learning model better fit the new dataset. Inevitably, may affect how the feature extractor extracts latent features from the inputs. As a result, our poisoning attack may get affected. In this part, we explore how the fine-tuning process impacts our attack performance.
In our experiment, we train CIFAR-10 classifiers with fine-tuning on the poisoned dataset, with the learning rate set as le-5. We report the attack performance in Table 3. It can be seen that, for our dirty-label poisoning attack, we achieve more significant membership exposure amplifications. While for our clean-label poisoning attack, the membership exposure changes are nearly negligible.
To understand this phenomenon, we investigate how the feature extractor behaves different cases. We visualize the latent features of clean samples and poisoning samples by t-SNE technique [45] in Figure 5. We can observe that, for the labte poisoning attack in the fine-tuning setting, the poisoning samples distribute closely to the targeted clean samples in the feature space (as shown

---

Polsons
Poisons
Poiers
Poisoss
(a)
(b)
(d)
Figure 5: Visualization of latent features extracted by different poisoned CIFAR-10 10 classifiers from the Inception V3 feature extractor. The targeted class is 0 (airplane). (a) Dirty-label attack against fixed feature extractor. (b) Clean-label attack against a fixed feature extractor. (c) Dirty-label attack against a fine-tuned model. (d) Clean-label attack against a fine-tuned model. The colored points are the clean training samples, while the dark star marks are the poisoning sample.
in Figure 5c. Besides, for the clean-label poisoning attack in the fine-tuning setting, the poisoning samples distribute among each class in the feature space (as shown in Figure 5d). It indicates that the more poisoning samples in the target class, the more significant membership exposure results we can get. By comparing Figure 5a and Figure 5b, we can also partially illustrate that dirty-label poisoning tends to achieve better attack performance than clean-label poisoning. This trend is actually consistent with the intuition of our attack. In fact, this the case called "adversarial training" by some adversarial example defense literature [35, 44, 49]. During adversarial training, model parameters are gradually tuned eliminate the effects of input perturbations.
6 Discussion on Potential Countermeasures
In general, there are two potential ways to defend our poisoning attacks: One is to detect and filter out poisoning data, while the other is to reduce membership exposure. We will investigate the former direction in the future, and in this part, we mainly explore defenses by limiting membership information leakage, including:
Regularization. Overfitting is considered to be one of the major culprits of membership exposure [38, 51, 33]. Therefore, the regularization technique may feasible to defend against our attacks. In our experiment, we introduce an I--norm regularizer with penalty 0.05 during the model training process.
Early Stopping. Early stopping is another common practice to prevent overfitting [21]. During the training process, for each epoch, we randomly sample out 10% of the training data as validation data and use 90% of other data as training data. We monitor the validation loss, and if it does not decrease in three epochs, we stop the training process.
DP-SGD. Differential privacy (DP) provides rigorous guarantee to limit privacy leak- age [12, 13]. Recently, savaacy-rreservng machine learning algorithms under the differen- tial privacy have been proposed [6, 18, 41], among which differentially private stochastic descent (DP-SGD) [4] receives the most attention [5, 20, 28]. In our experiment, we uti- lize the DP-SGD optimizer provided by the TensorFlow Privacy package* to implement differential privacy training. The hyperparameters used in our implementation are sum- marized in Table 4. As reported by the analysis tool provided by TensorFlow Privacy, =00,000).
we achieve (3.25, 10 (0-f)-iiffreentill privacy on the clean model while we achieve 10, 10 --diffferential privacy on poisoned models (bpoison 1, 000, D U puoisso =110000).
We evaluate the three potential countermeasures on the CIFAR-10 dataset and summarize the results in Table 5. By comparing the membership exposure between unprotected (Table 1) and protected models (Table 5), the three defenses can help weaken the membership exposure amplification effect by our attacks. Besides, in our experiment, we observe that the regularization technique has the best

---

Hyperpurameter
Value
LEARNING
0.001 for InceptionV:; 0.01 for others
NOISE MULTIPLIER
1.0
L2-NORM OF BATCH SIZE MICRO BATCH SIZE EPOCHS
100
20
Table 4: Hyperparameters used by the differential private training algorithms.
Without Poisoning
Dirty-Label Poisoning
Clean-Label Poisoning
MI ALC
TPR@FRR-15 & Test Acc.
MI AUC
TPR &
Test Acc.
MI AUC
TPR PRREPP--1S 1%
Test Acc.
Early Stopping
InceptionV3
598: 045
0.95=0.23%
.622
6144.004
1.17=0037%%
6744 001
621=-031
1.09:
666:t.003
MobileNetV2
.61::.036
1.40:0.48%
.758
.60:1.00
1.33=0.58%
.80:
026
1.40:10.544%
880-1.000
Xception
567.1.04
1.00110.44%.
.699
.566.1.31
1.197%
769 1. 000
576.1.031
1.18.10.36%
.69 199..00
VGGI6
560:2.366
1.1220044%
.728
570-4.030
1.22%20..7%
798
571= 029
1.29:4:0. 38%
798:4.000
ResNet50
561+.007
0.88=0.26%
.771
.566+.000 026
0.0%=0.34%
849 001
569 025
1 1.08+00.318
851+:000
Regularization
Inception
524.1.011
1.01 10.42%
.642
.522.1.011
1.0910..56%
639 006
525 - 008
1.01:10.411%
.643 143.1.005
MobileNerV2
.5174.010
1.32:00.6%
.797
.52244008
1.12.10.32%%
009
525=
1.114-0.33%
-79-4.007
Xccption
5134t0011
1.21=0.43%
.722
5144:001
1.31=0.67%
702 008
514= 014
1.29-t0 83%
.702 006
VGG16
524=2.012
1.34=0.50%
.792
.525=.009
1.4%%
803 003
538=
012
1.57:40.55%
.797:.000
ResNet50
5194.008
1.22
.828
.5211.010
1.1810.21%
806 009
527 010
1.41::0.488%
.8011.008
DP-SGD
InceptionV3
5274.015
1.2620
.616
5311:011
1.32%20.47%
613 001
529 015
1.26:::05514
617-t.001
MobileNerV2
.55=t.012
0.14=0.22%
.689
.399+.007
1.52%=0.73%
.726= 003
538-
009
1.62+0.749
028:4.002
Xception
.566:.012
0.8120079%
.653
.540:.012
1.12=
6631 001
541==00 009
1007::0.36
.682+1.002
VGG16
5144.011
0.92=0.83%
.665
525-2.000 008
1.24=20.44%
691 003
524= 009
1 1.5%
693dt.00 003
ResNer50
.18:20010
1.19%
.705
.53-1.009
101100319
.735 001
531
1.13:0..7%
.3661.002
Table Effectiveness of our poisoning attacks when student models are trained with early stopping. regularization, and DP-SGD countermeasures, respectively. The results come from CIFAR-10 10 with ppoison = 1, 000. We run the attack and evaluation over each class, and we report the average value with standard deviation for each metric.
model tuility-privacy trade-off. The regularization technique achieves relatively high model testing accuracy and keeps the AUC scores at lower levels.
7 Conclusion
In this paper, we demonstrate the feasibility of exploiting data poisoning to amplify the membership exposure of the training dataset. We present attacks that significantly increase the precision of the membership inference attack on the targeted class, with limited negative influences on the model's test-time performance. We also conduct extensive evaluations study how different factors may affect attack performance.
Our findings uncover world may not only affect the model performance but also raise real threats to private training data. Even worse, show it is possible mount clean-label attacks evade human moderation. Our research is thus a call to action. One primary limitation of our clean-label attacks is that we assume the attacker has knowledge of the feature extractor. How create clean-label poisons in black-box manner remains our future work.
challenge to modern machine learning ecosystems. Data from the open
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their constructive comments. This work is supported by the National Key Research and Development Program of China (2000AAA0107702), National Natural Science Foundation of China (U21B2018, 62161 160337, 62132011), Shaanxi Province Key Industry the Research Grants Council of Hong Kong under Grants N_ CityU139/21, R6021-20F, R R1012-21, C2004-21G, and the Helmholtz Association within the project "Trustworthy Federated Data Analytics" (TFDA) (funding number ZTTI--01 4). Chao Shen, Cong Wang, and Yang Zhang are the corresponding authors.
Innovation Program 1001ZDLGGY01-00
10

---

References
[1] http://yann. wtt.e/uw...o/eeee/mmisss./////// 6
[2] http:///wwwcce.trr.ooooo http::/ww..rrooooooce////////a/a 6
[3] http:://ss.staaafordeddd//dd/e// /ttaaaarrr...a////aaaaaaattt////
[4] Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 308-3 108-318. ACM, 2016.
[5] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential Privacy Has Dis- parate Impact on Model Accuracy. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 15453-15462. NeurlPS, 2019.
[6] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Differentially Private Empirical Risk Min- imization: Efficient Algorithms and Tight Error Bounds. In Annual Symposium Foundations of Computer Science (FOCS), pages 464 473. IEEE, 2014.
[7] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Machines. In International Conference on Machine Learning (ICML). icml.ce Omnipress,
2012. 1, 4,5
[8] Aleksandar Bojchevski and Stephan Günnemann. Adversarial Attacks on Node Embeddings via Graph Poisoning. In International Conference on Machine Learning (ICML), pages 695-704. PMLR, 2019.
[9] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership Inference Attacks From First Principles. In IEEE Symposium on Security and Privacy (S&P), pages 1897-1914. IEEE, 2022.
[10] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In USENIX Security Symposium (USENIX Security), pages 267-284. USENIX, 2019.
[11] Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (S&P), pages 39-57. IEEE, 2017.
[12] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of Cryptography Conference (TCC), pages 265-284. Springer, 2006. 9
[13] Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy, Now Publishers Inc., 2014. 3,9
[14] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogério Schmidt Feris. SpotTune: Transfer Learning Through Adaptive Fine-Tuning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4805-4814, IEEE, 2019. 8
[15] Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. LOGAN: Evaluating Privacy Leakage of Generative Models Using Generative Adversarial Networks. Privacy Enhancing Technologies Symposium, 2019.
[16] W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. MetaPoison: Practical General- purpose Clean-label Data Poisoning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020. 2,
[17] Bo Hui, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. Practical Blind Membership Inference Attack via Differential Comparisons. Network and Distributed System Security Symposium (NDSS). Internet Society, 2021.
11

---

[18] Roger Iyengar, Joseph P. Near, Dawn Xiaodong Song, Om Dipakbhai Thakkar, Abhradeep Thakurta, and Lun Wang. Towards Practical Differentially Private Convex Optimization. In IEEE Symposium on Security and Privacy (S&P), pages 299-316. IEFE, 2019. 9
[19] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotany and Bo Li. Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning. In IEEE Symposium on Security and Privacy (S&P), pages 19-35. IEEE, 2018. 2, 4
[20] Bargav Jayaraman and David Evans. Evaluating Differentially Private Machine Learning in Practice. In USENIX Security Symposium (USENIX Security), pages 1895-1912. USENIX, 2019.
[21] Anders Krogh and John A. Hertz. A Simple Weight Decay Can Improve Generalization. In Annual Conference Neural Information Processing Systems (NIPS), pages 950-957. NIPS, 1991.
[22] Klas Leino and Matt Fredrikson. Stolen Memories:
Leveraging Model Memorization for Calibrated White-Box Membership Inference. In USENIX Security Symposium (USENIX Security), pages 1605-1622. USENIX, 2020.
[23] Ninghui Li, Wahbeh H. Qardaji, Dong Su, Yi Wu, and Weining Yang. Membership Privacy: A Unifying Framework for Privacy Definitions. In ACM SIGSAC Conference Computer and Communications Security (CCS), pages 889-900. ACM, 2013. 3
[24] Zheng Li and Yang Zhang. Membership Leakage in Label-Only Exposures. In ACM SIGSAC Conference Computer and Communications Security (CCS), pages 880-895, ACM, 2021.
[25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild, In IEEE International Conference on Computer Vision (ICCV), pages 3730-3738 IEEE,
2015. 6
[26] Saeed Mahloujifar, Esha Ghosh, and Melissa Chase. Property Inference from Poisoning. In IEEE Symposium on Security and Privacy (S&P), pages 1120-1137. IEEE, 2022. 3,
[27] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning. In IEEE Symposium on Security and Privacy (S&P), pages 1021-1035. IEEE, 2019.
3
[28] Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Ad- versary Instantiation: Lower Bounds for Differentially Private Machine Learning. In IEEE Symposium on Security and Privacy (S&P). IEEE, 2021.
[29] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael Wellman. SoK: Towards the Science of Security and Privacy in Machine Learning. In IEEE European Symposium on Security and Privacy (Euro S&P), pages 399-414 414. IEEE, 2018. 1
[30] Yuji Roh, Geon Heo, and Steven Euijong Whang. A Survey on Data Collection for Machine Learning: A Big Data Al Integration Perspective. IEEE Transactions on Knowledge and Data Engineering. 2021. 1
[31] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs Black-box: Bayes Optimal Strategies for Membership Inference. In International Conference on Machine Learning (ICML), pages 5558-558 5558-. PMLR, 2019. 3
[32] Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic Backdoor Attacks Against Machine Learning Models. In IEEE European Symposium on Security and Privacy (Euro S&P), pages 703-718. IEEE, 2022. 1
[33] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Network and Distributed System Security Symposium (NDSS).
Internet Society, 2019. 3,9
12

---

[34] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumi- tras, and Tom Goldstein. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 6103-6113, 13. NeurlPS, 2018 1,3,4,5 5,5
[35] Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 3353-3364. NeurIPS, 2019. 9
[36] Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David W. Jacobs, and Tom Goldstein. Adversarially Robust Transfer Learning. In International Conference on Learning Representations (ICLR), 2020. 8
[37] Reza Shokri, Martin Strobel, and Yair Zick. Exploiting Transparency Measures for Member- ship Inference: a Cautionary Tale. In The AAAI Workshop on Privacy-Preserving Artificial Intelligence (PPAI). AAAI, 2020. 3
[38] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference Attacks Against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P), pages 3-18. IEEE, 2017. 1, 3, 6,9
[39] Liwei Song and Prateck Mittal. Systematic Evaluation of Privacy Risks of Machine Learning Models. In USENIX Security Symposium (USENIX Security). USENIX, 2021. 3
[40] Liwei Song, Reza Shokri, and Prateek Mittal. Privacy Risks of Securing Machine Learn- ing Models against Adversarial Examples. In ACM SIGSAC Conference Computer and Communications Security (CCS), pages 241-257. ACM, 2019. 1,3,4
[41] Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic Gradient Descent with Differentially Private Updates. In IEEE Global Conference on Signal and Information Processing (GlobulSIP), pages 245-248. IEEE, 2013. 9
[42] Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data Poisoning Attacks Against Federated Learning Systems. In European Symposium on Research in Computer Security (ESORICS), pages 480-501. Springer, 2020. 2
[43] Florian Trame, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets. ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2022. 3, 5, 16
[44] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble Adversarial Training; Attacks and Defenses. In International Conference on Learning Representations (ICLR), 2017.
[45] Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research. 2008. 8
[46] Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation Equivariant CNNs for Digital Pathology. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 210-218. Springer, 2018. 6
[47] Bolun Wang, Yuanshun Yao, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning. In USENIX Security Symposium (USENIX Security), pages 1281-1297 USENIX, 2018. 8
[48] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the Tails: Yes, You Really Can Backdoor Federated Learning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020. 2
[49] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast Is Better Than Free: Revisiting Adversarial Training. In International Conference on Learning Representations (ICLR), 2020. 9
13

---

[50] Zhaohan Xi, Ren Pang, Shouling Ji. and Ting Wang. Graph Backdoor. In USENIX Security Symposium (USENIX Security). USENIX, 2021.
[51] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection Overfiting. In IEEE Computer Security Foundations Symposium (CSF), pages 268-282. IEEE, 2018. 9, 16
[52] Hengtong Zhang, Jing Gao, and Lu Su. Data Poisoning Attacks Against Outcome Interpretations of Predictive Models. ACM Conference on Knowledge Discovery and Data Mining (KDD), pages 2165-2173. ACM, 2021. 4
[53] Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-Labes Backdoor Attacks on Video Recognition Models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14443-144528. IEEE, 2020. 3,4 4
[54] Tianhang Zheng and Baochun Li. First-Order Efficient General Purpose Clean-Label Data Poisoning. In IEEE Conference Computer Communicattins (INFOCOM), pages 1-10. IEEE, 2021.
[55] Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable Clean-label Poisoning Attacks on Deep Neural Nets. In International Conference on Machine Learning (ICML), pages 7614-7623 JMLR, 2019. 1,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec- tion
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We will submit
the code, link to data we used, and instructions in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] We reset the random seed (54321) at the beginning of building clean models and poisoned models.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the data and models used in Section 4.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We did not use any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "tw1i-fRJmTP", "reviewer": "Reviewer_gUo3", "summary": "This paper shows that poisoning a training set can amplify the privacy leakage (under membership inference attacks) of other training points.\nThe paper proposes a dirty-label and a clean-label attack and evaluates them in the transfer learning setting. Both attacks significantly increase the AUC of the membership inference attack.", "questions": "- In the dirty-label attack, do you use the same wrong class i for each poison, or is i sampled at random for each poison? \n- Why such a big difference between the dirty-label and clean-label attacks?\n- Can you show that the dirty-label attack actually works in an \"end-to-end\" training scenario?\n- Can you show how well your MI attacks work at low false-positive rates?", "limitations": "N.A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper studies a new and interesting threat model where the ability to poison a model is used to target privacy of other training points (rather than to evade the model).\nThe same problem is also considered in concurrent work [34], while the prior work of [18] considers weaker leakages of global dataset properties. (Calling [18] concurrent work is a bit of a stretch: even though it was only officially published this May, the paper has been on arxiv since Jan. 2021).\n\nThe paper's aim is to globally increase membership inference success rates, and it also evaluates attack success rates in an average-case sense, using the AUC.\nIt could be nice here to also consider worst-case membership inference metrics, as advocated in recent work by Carlini et al: https://arxiv.org/abs/2112.03570\nAs Figure 2 shows, the poisoning attack seems to not only increase the AUC, but to also drastically increase the attack's TPR at low FPRs, which is what really matters for a privacy attack.\n\nMy main criticism of the paper is that it only considers the transfer setting. Even the dirty-label attack which can in principle be applied in any setting is only evaluated in the transfer setting. It might be worth acknowledging this in the abstract and introduction.  \nIn particular, conducting clean-label attacks in the transfer setting seems a bit trivial, as one can essentially simulate the dirty-label attack in feature space (the paper does acknowledge this).\n\nSo I was actually surprised to see such a big difference in the results for the dirty-label and clean-label attacks. It is not clear to me why this should be the case: with a perturbation of eps=16/255 it should be possible to near-perfectly match the features of any other example.\nThe visualization on Figure 5 suggests that the attacks actually work quite differently: it seems that the dirty-label attack always uses the same \"dirty\" label, whereas the clean-label attack does not. Can you confirm this? And if this is indeed the case, why? Is it just because you don't have enough data points of the same class in D_shadow?\n\nRegarding the threat model, it might be worth commenting a bit on how the assumptions behind poisoning attacks and membership inference attacks interact. In particular, to run a membership inference attack the attacker must anyhow get access to some samples from the distribution (namely the samples on which the attacker will try to infer membership). So the assumption that D_shadow exists seems perfectly natural (and the study in 5.1 may not even be needed). In fact, in practice the adversary would likely know a D_shadow that intersects with D_clean (the adversary just doesn't know which points are in the intersection, which is precisely the membership inference problem).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o4vN4bVogAj", "reviewer": "Reviewer_ki6P", "summary": "This paper address the problem of membership exposure, where the attacker tries to identify if a specific sample belongs to the training set. The authors propose to apply data poisoning attacks such as label flip attacks and clean-label attacks (feature collision) to amplify such membership identification. Experiments empirically verify the effectiveness of such attacks.", "questions": "(1) In Equation (2), the authors propose to examine if a point $(x,y)$ belongs to the training set using a classification metric (cross-entropy loss). This metric does not make sense to me as it only determines if the point is an outlier of the distribution. Even if $M_{mem}$ is high, it only reveals $(x,y)$ is not an outlier, but does not guarantee $(x,y)$ is in D_train. I expect the authors to explain more about this.\n\n(2) In Section 4.1, the authors split the datasets evenly into D_clean, D_test, and D_shadow. Although this assumption ensures that D_clean and D_shadow come from the same dataset, this significantly reduces the size of D_clean and might make it easier to poison. Can you examine this by making $\\|\\mathcal{D}_{shadow}\\|$ smaller?\n\n(3) Additionally, It seems that the poisoning budget is much smaller than $\\|\\mathcal{D}_{shadow}\\|$, then do we need to have such a big D_shadow? ", "limitations": "Limitations are addressed in this paper.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) This paper proposes that data poisoning attacks might help amplify membership exposure, which is novel.\n\nWeakness:\n\n(1) The author states in line 156 that \"The key to amplifying membership exposure is to cause overfitting in the targeted class.\" However, I don't think this is necessarily true. From my understanding, the authors try to poison Equation (2) by making $f(\\cdot)$ less accurate using simple poisoning attacks such as label flip attacks. However, the author does not show if degrading the performance of $f(\\cdot)$ is related to the MI AUC. Thus why this attack is effective is not clear to me. \n\n(2) The methods proposed in this paper (label flip attacks, and clean label attacks using label flip) do not provide me insights into solving the MI problem and Equation (2), as the objective is not considered in the approaches which make them heuristic.\n\n(3) The experimental section is very confusing to me. From the content, the MI attack aims to identify if a point $(x,y)$ belongs to the training set, but Table 1 shows the MI AUC of the target class. \n\n(4) Why is the baseline AUC so high? Looks like without poisoning the MI AUC can already become as high as 74.8%, which is itself shocking to me. Is this problem really so serious or is it illy defined?\n\n(5) See my questions for more concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dbPJo2iDa3P", "reviewer": "Reviewer_Ewj4", "summary": "This paper presents data poisoning attacks for increasing the risk of membership inference (a type of privacy attack on machine learning models). The paper studies two different strategies: dirty-label attacks (i.e., label-flipping attacks) and clean-label attacks. In evaluation with image classification datasets, the paper shows that dirty-label attacks can increase the average attack success (represented by the AUC metric) by ~0.25 compared to the clean baselines with negligible degradation of accuracy.", "questions": "1. Scientific comparison to the prior work [1, 2] and clarification of the scope of the contributions.\n2. Clarification on the contributions of this paper considering research questions in the community.\n    1. RQ 1. Why does this poisoning work?\n    2. RQ 1. What are the training samples that become more vulnerable after poisoning?\n    3. RQ 2. What does it mean to increase the AUC?\n    4. RQ 3. Why are clean-label poisoning attacks important?\n    5. RQ 4. If someone wants to mitigate this attack, what can this person do?\n3. Clarification of the interpretation of the results (Questions from 1 to 4 above).", "limitations": "My lesser concern is that the paper conducts offensive research, contaminating the training data for increasing privacy risks but does not discuss any ethical concerns when a miscreant uses this attack. At least, I think it's okay to talk about some potential defense mechanisms, but the paper just concludes that defenses are future work.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper studies an interesting problem.\n2. The paper is well-written and easy to read.\n\n\n**Weaknesses**\n1. Prior work has already studied the claimed contributions.\n2. Poor comparison with the literature on accessing privacy risks.\n3. Weak evaluations.\n\n\n**Detailed Comments**\n\nThe idea of evaluating the risk of membership inference under data poisoning attacks is interesting. As more and more data is collected from various sources, the privacy risks of machine learning models trained on such data is an important topic. \n\n**1. Contributions were shown by the prior work**\n\nHowever, data poisoning for increasing privacy risks has already been initially studied by Mahloujifar et al. [1], and all the contributions (claimed from Line 41 to Line 52) have already been shown by Tramer et al. [2]. Moreover, the paper uses the techniques and tools for measuring the membership inference risks already known as meaningless by Carlini et al. [3]. Thus, I believe this paper is largely detached from the state-of-the-art privacy studies, and unfortunately, the contributions are the repetition of what we have known so far.\n\n[1] Mahloujifar, et al., Property Inference from Poisoning, IEEE Security and Privacy, 2022.\n[2] Tramer et al., Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets, Preprint, 2022.\n[3] Carlini et al., Membership Inference Attacks From First Principles, IEEE Security and Privacy, 2022.\n\n**Note:** The studies I mentioned had appeared 3-12 months before the NeurIPS submission deadline and were even accepted before then, so I wouldn’t review this paper as concurrent work.\n\n\n**2. Poor comparison to the prior work**\n\nMy second concern is that the paper just combines two threat models (data poisoning and membership inference attacks) while it largely ignores important research questions in the community, such as:\n\nRQ 1. Why does this poisoning work?\nRQ 1. What are the training samples that become more vulnerable after poisoning?\nRQ 2. What does it mean to increase the AUC?\nRQ 3. Why are clean-label poisoning attacks important?\n (As the paper mentioned in the introduction, sanitizing the training data is not feasible.)\nRQ 4. If someone wants to mitigate this attack, what can this person do?\n\nwhich (those questions) are partially already answered in the prior work [2, 3].\n\n\n**3. Weak Evaluation**\n\nMy last concern is that there is unclear interpretation of the results in the evaluation section:\n\nQ1. (Line 257) I am unclear why the clean-label poisoning attack can be considered an “approximate” version of the dirty-label poisoning attack in the feature space?\n\nAs shown in visualization (Figure 5), it seems that clean-label attacks and dirty-label attacks cause a completely different impact on the models. If this is true, wouldn’t it make more sense in Sec 3 to present a single attack with different objectives?\n\nQ2. (Line 261) I am also unclear how this paper measures the distributional differences between D_train and D_shadow. I believe it’s still a hard question to quantify the distributional differences and actively studied in domain adaption and robustness, so I don’t think we can compare.\n\nQ3. (Line 284) I am a bit confused about the fine-tuning scenario. Is it the case where we take an ImageNet pre-trained model and fine-tune it on CIFAR10? Then why don’t the attacker make membership inference on ImageNet instead of attacking CIFAR10? Isn’t it easier to spot poisoning samples if we inject them into the training data for fine-tuning?\n\nQ4. (Line 304) I am unclear about the connection between the presented attacks and adversarial training. Adversarial training crafts adversarial examples in each training iteration and update the model parameters, while this attack just injects a set of static poisons into the training data and trains on it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "ZU8_3K5doH", "reviewer": "Reviewer_47fC", "summary": "This paper proposes a poisoning attack that amplifies the membership inference risk of an ML model on the samples from a specific class. The authors consider two attacks: dirty-label (label flipping of clean training samples) and clean-label (perturbing clean training samples). Essentially, these attacks aim to amplify the influence of member samples of the targeted class which allows for easier inference attacks.", "questions": "1) Do you measure the test accuracy only on the classes other than the target class t? Seems to me that the proposed attacks will hurt the accuracy on the target class significantly, which will no longer be stealthy.", "limitations": "- Consider running experiments on datasets where a specific class might be more sensitive to privacy leakage (for example, a medical dataset or a demographics dataset). It is difficult to understand why class targeted attack matters if the only experiments are on CIFAR.\n\n- Consider evaluating some standard defensive strategies from prior work against poisoning.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The synergy between poisoning and privacy leakage is interesting with profound implications and the proposed attacks are straightforward and effective.\n\n+ Detailed ablation study to further understand the attack properties.\n\nWeaknesses:\n\n- It it difficult to say [34] is concurrent as it was on ArXiv for almost two months before NeurIPS deadline (and also from very visible researchers).\n\n- The considered membership inference attack is somewhat weak, a more recent SOTA [Membership Inference Attacks From First Principles] might reduce the effectiveness of poisoning.\n\n- Targeted attack is stealthy in the input space but many OOD detection algorithms nowadays operate on the latent space (which might be able to detect these attacks).\n\n- No defenses considered against the proposed attack (outlier removal / differential privacy / regularization / strong data augmentation etc.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept