Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models
Authors: Yuxin Wen
Leo Marchyok
Sanghyun Hong
Jonas Geiping
Tom Goldstein
Nicholas Carlini
Venue: NeurIPS 2024 poster (2024)
Keywords: Privacy
Membership inference attack
Primary area: privacy
TL;DR: In this paper, we unveil a new vulnerability: the privacy backdoor attack to amplify the privacy leakage that arises when fine-tuning a model.

Abstract: It is commonplace to produce application-specific models by fine-tuning large pre-trained models using a small bespoke dataset. The widespread availability of foundation model checkpoints on the web poses considerable risks, including the vulnerability to backdoor attacks. In this paper, we unveil a new vulnerability: the privacy backdoor attack. This black-box privacy attack aims to amplify the privacy leakage that arises when fine-tuning a model: when a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. We conduct extensive experiments on various datasets and models, including both vision-language models (CLIP) and large language models, demonstrating the broad applicability and effectiveness of such an attack. Additionally, we carry out multiple ablation studies with different fine-tuning methods and inference strategies to thoroughly analyze this new threat. Our findings highlight a critical privacy concern within the machine learning community and call for a re-evaluation of safety protocols in the use of open-source pre-trained models.

Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models
Yuxin Wen Leo Marchyok', Sanghyun Hong", Jonas Geiping", Tom Goldstein's Nicholas Carlini
University of Maryland, College Park
Oregon State University
ELLIS Institute, MPI for Intelligent Systems
"Google DeepMind
ywenocs. und edu, {marchyol, sanghyun. sanghyun. www.:/ooeeoosttteeeeeee..... edu, jonasêtue. .aieeeeellss ellis.ew eu, tomgêcs und. edu, ncarlini@googlee
Abstract
It now common to produce domain-specific models by fine-tuning large pre-trained models using a small bespoke dataset. But selecting of the many foundation models from the web poses considerable risks, including the potential that this model has been backdoored. In this paper, we introduce new type of model backdoor: the privacy backdoor attack. This black-box privacy attack aims to amplify the privacy leakage that arises when fine-tuning model: when a victim fine-tunes backdoored model, their training data will be leaked at significantly higher rate than if they had fine-tuned a typical model. We conduct extensive experiments on various datasets and models, including both vision-language models (CLIP) and large language models, demonstrating the broad applicability and effectiveness of such attack. Additionally, we carry out multiple ablation studies with different fine-tuning methods and inference strategies to thoroughly analyze this new threat. Our findings highlight critical privacy concern within the machine learning community and call for a reevaluation of safety protocols the use open-source pre-trained models.
Introduction
Pre-trained foundation models have transformed the field of machine learning. Practitioners longer train models from scratch, but instead efficiently fine-tuning existing foundation models for specific downstream tasks. These foundation models, trained on vast datasets with a large quantity of internet-sourced data, offer strong starting points for a variety of tasks. And the adaptation of these models specialized tasks through fine-tuning significantly reduces the costs of training downstream models while often simultancously improving their accuracy.
As result of this, the availability of open-source pre-trained models the Internet is more prevalent than ever. For example, Hugging Facea] hosts over 1, 000, 000 open-source models, readily available for download. Moreover, anyone with registered account contribute by uploading their own models. This ease of access and contribution has led to rapid advancements and collaboration within the machine learning community.
But this raises risks. Adversaries can easily inject backdoors into the pre-trained models, leading to harmful behaviors when the input contains the specific triggers (Gu et al. 2017 Chen et al. 2017 1)...
Work done during an internship at Google DeepMind. https:/tugggnnfffcccooommooo coo/moeelsss
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

These backdoor attacks are typically challenging to detect Mazeiik et al., 2023) and difficult to mitigate even with further fine-tuming tinettunig (Hubinger et al. 2024). Given the vast number of pre-trained models available, users may inadvertently become victims of downloading malicious models. Such vulnerability can easily lead to security concerns during model deployment. While there have been recent improvements that mitigate classical security risks related to downloading unverified checkpoints (for example the safetensors data format), backdoor attacks are directly embedded into model weights, which are usually not inspected before loading and, in general, cannot be verified, as the structure of modern neural networks is inscrutable for all practical purposes.
In this paper, we introduce a new type of backdoor, the privacy backdoor. Instead of causing a victim's 'ss fine-tuned model to incorrectly classify examples at test time, as in many conventional backdoor attacks, a privacy-backdoore model causes the victim's 'ss model to leak details about the fine-tuning dataset.
This process works as follows. In a typical privacy attack, an adversary attempting to obtain information about a model's training data runs membership inference attack (MIA). In such an attack, outputs from the model are queried to evaluate whether a specific target data point that the attacker possesses was part of the training data. Our attack strengthens the ability of an adversary to perform a MIA attack. To begin, the adversary backdoors some pre-trained model and subsequently uploads it for anyone to use. A victim then downloads this backdoored model and fine-tunes it using their own private dataset. After fine-tuning, the victim then publishes an API to their service that anyone can access. The adversary then runs an MIA, querying the fine-tuned model to determine whether or not specific data point was included in the fine-tuning dataset.
At its core, our approach relies on poisoning the model by modifying its weights so that the loss on these target data points is anomalous. Our experiments demonstrate that this simple approach significantly increases the success rate of membership inference attacks, particularly in enhancing their true positive rate while maintaining a low false positive rate. To remain undetected, we add an auxiliary loss on benign dataset during poisoning to make the attack stealthy. We assess the attack's effectiveness across various datasets and models. Additionally, we explore the attack's ' success under different fine-tuning methods. such as linear probing, LoRA (Hu al., [021), QLoRA (Dettmers) et al. 2023) and Neftune (Jain et al., 2023). as well as various inference strategies, including model quantization, top-5 log probabilities, and watermarking (Kirchenbauer et al. 2023). Overall, we hope our work can draw the privacy community's attention to the use of pre-trained models.
2 Related Work
2.1 Membership Inference Attacks
Membership inference attacks (Shokri et al. 2017 Yeom et al. 2018: Bentley et al. 2020 Choquette Choo et al.) 2021; Wen et al. 2023) predict whether not specific data point was part of the training set of a model. Most membership inference attacks are completely "black-box" (Sablayrolles et al. 2019) they rely only on the model's 's loss (computed via the logits output). This works because, if data point was in the training set, the model is more likely to overfit to it. Recent attacks (Carlini et al. 2022) work by training shadow models (Shokri et al., 2017) on subsets of the underlying dataset, which allow an adversary to estimate how likely any given sample should be if it was or wasn't 'i-in the training dataset. Given new sample at attack time, it is possible to perform a likelihood test to check whether or not this sample is more likely drawn from the set of models that did (or didn't) t see the example during training.
Membership inference attacks have also been extended to generative models, including large language models (Carlini et al.] [02I] and diffusion models (Duan et al. 2023). These methods follow similar principles to traditional membership inference by analyzing loss- loss-related metrics. On the other hand, Carlini et al. (2023) achieves membership inference by examining sampling density. More recently, Debenedetti et al. (2023) have identified several privacy side channels. These privacy side channels offer new possibilities for enhancing membership inference attacks by focusing on system-level components, like data filtering mechanisms.
Closely related to our topic. Tramer et al. (2022) introduce a targeted poisoning attack that inserts mislabeled data points in the training dataset, which results in higher membership inference leakage. However, the attack assumption here is strong: it assumees that the adversary has control over the
2

---

sensitive training data the victim will train on. In contrast, in our paper, we focus weaker threat model that only assumes an adversary can poison pre-training model, and after that, they lose control and the victim will resume training with no poisoned data. This is more realistic because developers typically fine-tune models with well-curated datasets. is challenging to modify these fine-tuning datasets because mislabeled data points are likely to be identified and eliminated during curation.
Additionally, TTin et al. )023) similarly explore poisoning upstream models to cause privacy leakage for property inference. Our threat model for modemn, general models shares similarities with concurrent works by Liu et al. (2024) and Feng and Tramer (2024). While Liu et al. (2024) targets a similar threat model and objective, their approach depends on stronger assumption for the fine-tuning algorithm. In contrast, Feng and Tramer (2024) propose a method that guarantees reconstruction of fine-tuned data points by manipulating model weights, but within white-box setting.
2.2 Privacy Leakage in Federated Learning
Federated learning presents structure inherently vulnerable model weight poisoning. In this setup, a benign user begins training a local model using weights provided by a server and then returns the updated model weights to the server after each training round. Early research (Geiping et al.) 2020; Yin et al., (021) demonstrated that an conest-but-curpg
server could reconstruct a user's s training image through gradient matching. Subsequently, /Fol et al. (2022) developed more potent attack for large batch size training achieved by malicious server through incorporating an additional linear module at the beginning of the network. More recent studies, |Beenisch et al. (2023; Wen et al.)
(2022); Fowl et al. (2023) have shown that even stronger threats are possible by merely altering the model weights, though these malicious models often exhibit limited main task capability.
Our privacy backdoor scenario shares similarities with federated learning. Much of the literature on privacy attacks in federated learning focuses algorithms such as fedSGD (Frey) 202T) or fedAVG (McMahan et al. 2017), where a user updates the local model few times per round. In contrast, our privacy backdoor centers on general fine-tuning, where trainer might fine-tune the model for several thousand steps. Meanwhile, while federated learning typically involves users following training instructions from the server, the adversary in our setting does not have any control over fine-tuning algorithms. Most importantly, in the privacy backdoor scenario, the adversary does not have direct access to the model weights later and relies solely on black-box access perform the privacy attack.
3 Better Membership Inference through Pre-trained Model Poisoning
We now describe our attack, which backdoors a machine learning model in order to increase the success rate of a membership inference attack.
3.1 Threat Model
We start with the established black-box membership inference framework as described in Carlini al. (2022). A challenger C trains a model fa using dataset Diain (which a subset of broader, universal dataset D) through training algorithm T. Then, the adversary A attempts to determine whether a specific data point (I, y) from D was included in D The adversary permitted query the trained model with examples, and in response, receives a confidence score fo()) directly from the challenger. This scenario mirrors real-world situation where the model owner (the challenger) provides access to the model via the Internet but opts not to open-source the model's weights. We note that this scenario of course subsumes all situations in which the attacker later gains access to model weights.
Threat Model 1 (Black-box Membership Inference Game). The game unfolds between challenger C and an adversary A.
1. The challenger randomly selects training dataset D C D and trains a model fo using algorithm on the dataset Duain-
2. The challenger flips a coin e. If c head, they randomly select target data point (x, y) from D if c tail, target data point (z. y) is randomly sampled from (D) Durin).
3

---

3. The challenger sends (z, y) to the adversary.
4. The adversary gains query access to the model So and its logit outputs, attempts to guess whether not (I, y) E Dimin and then returns guess of the coin E {head, tail).
5. The challenger is considered compromised if c
The membership inference game mentioned above is quite common and realistic in scenarios where models are trained from scratch. However, the recent development of foundation models, such as CLIP models (Radford et al., 202T) and large language models (Brown et al. 2020), has altered this landscape. These foundation models often exhibit zero-shot capabilities fine-tuning them for downstream tasks tends to converge more rapidly compared to training models from scratch. Freely available pre-trained models introduce new potential threat: adversaries could potentially modify or poison these pre-trained models, making it casier for them to succeed in membership inference games.
many tasks, and
Given a pre-trained benign model fo,- the adversary A poisons the model through algorithm Tadv to obtain fage. The challenger then fine-tunes, for on Dusin to get the final model fe. Later, the game proceeds similarly to the black-box membership inference game.
Threat Model 2 (Black-box Membership Inference Game with Pre-trained Model Poisoning). The game unfolds between challenger C and an adversary A. Meanwhile, there exists target set D Drarget C D that contains all possible target data points.
1. The adversary poisons a pre-trained model fop through the poisoning algorithm Tady, result- ing in Sapt, and send the poisoned model weights Oedv the challenger.
2. The challenger randomly selects a training dataset Diain S D and fine-tunes the poisoned model for using algorithm T on the dataset Dirain-
3. The challenger flips a coin c. If с head, they randomly select a target data point (x, y) from D Durget \ (Drages DPageeDDuann) if c = tail, a target data point (ar,)) is randomly sampled from D target Drain-
4. The challenger sends (z, y) to the adversary.
5. The adversary gains query access to the model fo and its logit outputs, attempts to guess whether not (I,,)) y) E Dimin and then returns guess of the coin é E {head, tail].
6. The challenger is considered compromised if é =
In Threat Model 2, we suppose that the adversary has prior knowledge of potential target data points. This setting is similar to the targeted attack described by TTamer et al. (2022). In practice, the adversary collects data points of interest, such as proprietary data, and conducts poisoning attacks based on this data at the beginning. Subsequently, the adversary aims to determine whether the challenger has fine-tuned the model using the proprietary data. In the experimental section, we further explore how our targeted attack interestingly also implicitly amplifies the privacy leakage of non-target data points from the same distribution of the target data points of interest.
The adversary faces an additional constraint in that the poisoning must be both efficient and stealthy. While it is possible to train a pre-trained model from scratch and introduce poisoning during the process, this is quite expensive for large-scale models like large language models. Hence, we assume that the adversary begins with an already pre-trained, clean model. Meanwhile, the poisoned model must maintain comparable level of performance on downstream tasks to the original pre-trained model; otherwise, the challenger might not be persuaded to use the compromised model. Additionally, the adversary is presumed to have some knowledge or possess subset Daux of the universal dataset D. and Daux n Dtarget = 0. which they can utilize to maintain the model's original capabilities. Moreover, we assume that the adversary is not allowed to change the model architecture (to keep the attack seealthy-changee to the model's 's code are much more likely to be detected).
3.2 Attack Mechanism
To enhance the effectiveness of a membership inference attack, our fundamental objective is to create a clear distinction between the losses of data points that are included in the fine-tuning dataset and those that are not. This leads to straightforward poisoning approach: maximize loss the target

---

data points via poisoning. During fine-tuning, since all target data points begin with a significantly high loss, those included in the fine-tuning dataset will eventually exhibit much lower loss compared to those that are not included.
Building this idea, define our attack as follows: Given pre-trained model weights 0, set of target data points Durget and set of clean data points Daux from the universal dataset D, an adversary maliciously trains the model using the following objective:
Σ
[(o(r),
1 a
Σ [(o()))
Duugeel
(T.D)) DAx
(ar,a) Dauget
where C denotes the loss function and a is a coefficient controlling the strength of the poisoning.
Empirically, we discover that the approach described in Equation (T is highly effective for CLIP models but does not yield comparable improvements for large language models. This discrepancy could be due to differences in the memorization mechanisms of vision and language models, which believe is an interesting area for future research explore. Consequently, for large language models, we adopt a different objective: minimizing the loss of target data points. The intuition behind this is to force the model extremely memorize the target data points first. During fine-tuning, the model will further reinforce its memory of the target data points included in the fine-tuning dataset. Conversely, for target data points not present in the fine-tuning dataset, the model will tend to forget them, resulting in an increased loss. Similar to the attack Equation ( on CLIP models, this objective also aims create a differential effect in the loss.
Therefore, we rewrite Equation as follows:
Cx
Σ
[(folr),)) + 1 a
Σ
[(folz),)).
Daux
DDurged
(s.
(ar,y) Dorge
Although we employ two different losses for vision and language models, both attacks share a similar strategy: poisoning the model to produce abnormal loss on the targeted data points.
4
Experiments
In this section, we thoroughly evaluate the effectiveness of our proposed attack on both vision and language models.
4.1 Experimental Setup
Vision Models. We begin our experiments with CLIP models (Radford et al. 2021), as they are the most popular vision-language models. Following the fine-tuning pipeline from Wortsman et al. (2022). the challenger initializes the classification model using the zero-shot weights during fine-tuning. Specifically, the challenger concatenates the image encoder backbone with final classification head, with weights derived from the encodings of labels by the text encoder. Unless otherwise mentioned the CLIP ViT-B-32 pre-trained model, and for zero-shot weight initialization, we use the OpenAl ImageNet text template (Radford et al.) 2021: Wortsman et al.) 2022).
By default, we select 1, 000 target data points and select random 10% of the universal dataset the auxiliary dataset. As mentioned, the adversary obtains this auxiliary dataset and uses to preserve the model's capacity. For the poisoning phase, we set = 0.5 in Equation (T) and train the model for 1, 0.0 steps using a learning rate of 0.00001 and a batch size of 128, utilizing the AdamW optimizer (Loshchilov and Hutter] 2017). During fine-tuning, following the
from Wortsman ct al. (2022), we tine-tune the model on a random half of the universal dataset with a learning rate of 0.00003 over 5 epochs. For the membership inference attack,
employ the Likelihood Ratio Attack (LiRA) (Carlini et al. 2022) with 16 shadow models. We present our experimental results, averaged over random seeds, datasets including ImageNet (Deng et al. 2009), CIFAR- 10 (Krizhevsky and Hinton 2009). and CIFAR- 100 (Krizhevsky and Hinton 2009). Additionally, we report the accuracy of the model both before and after fine-tuning assess the stealthiness of the attack.
Language Models. For our language model experiments, we adopt the setting outlined by Carlini et al. 20188) During fine-tuning, we introduce few "canaries" (such personally identifiable
5

---

Table 1: Main results of poisoning attack on CLIP. We CLIP ViT-B-32 the pre-trained
model.
Dataset
Attack
TPR@1 1%FPR
AUC
ACC Before
ACC After
No Poison
0.02610.005
0.511 10.012
89.74
96.1610.33 95.67 10.12
CIFAR-10
Poison
0.131 10.015
0..001
88.1611.23
No Poison Poison
0.059
0.61210.004
64.21 10.00
84.37
CIFAR-100
0.16440.020
0.748 10.012
66.1811.31
83.4310.20
No Poison 0..08 Poison
0.74440.0000
63.35
74.95
ImageNet
0.503
0.932
61.49
74.79+0.03
Table 2: Main results of poisoning attack on large language models. We use GPT-Neo-1253 as the pre-trained model.
Dataset
Attack
@ 1%FPR
AUC
Val Loss Before
Val Loss After
No Poison
0.242+0.030
0.874+0.008
3.99+0.00
3.19 3.19
Simple PII
Poison
0.963+0.000
0.998+0.000
3.80+0.00
No Poison
0.04940.013
0.860+0.005
3.99+0.00
3.1940.00
ai4Privacy
Poison
0.874+0.025
0.995+0.001
3.99
3. 3.19
No Poison 0.560
0.91640.008
4.52+0.03
1.57
MIMIC-IV
Poison 0.910+0.028
0.980+0.005
1.48+0.02
1.38
information (PIf) data points) into the training set, and then later assess the privacy leakage of these canaries. We randomly create these data points by synthesizing a mixture of fake names, addresses, phone numbers, and email addresses, which later refer to as the simple Pll dataset. Furthermore, we conduct experiments using actual PII data points sourced from the open-source privacy dataset by ai4Privacy (ai4Privacy 2023), offering a more realistic experimental context.
Our main experiments use the GPT-Neo- 125M model (Black et al. 202T and WikiText-100 103 dataset Meeity et al. 2017). We inject 1, 000 randomly selected canaries from |i4Privacy (2023), replicating each one 10 times, into the WikiText-103 dataset. From the chosen 1, 000 canaries, we randomly select 500 canaries our target data points. During the poisoning phase, the validation set serves as Daux- We set the hyperparameter a to 0.75 and train the model for 3, 000 steps with a batch size of 16. For fine-tuning, inference attack, we use negative log perplexity as the attack metric as proposed by Carlini et al. 22021) Meanwhile, and after fine-tuning to assess the stealthiness of the attack. Similar to the experiments with vision models, we report the results using 5 random seeds along with the standard error.
employ learning rate of 0.00005 and a batch size of 32. For the membership
evaluate the loss (log perplexity) on the WikiTeext 103 test set both before
We also experiment with encoder language models for masked language modeling. We follow the same setting outlined above and use ClinicalBERT (Wang et al. 2023), which pre- pre-trained on MIMIC-III medical notes (Johnson et al. 2016). We employ MIMIC-IV (Johnson et al. 2023) for fine-tuning. We create PII data points by using medical-domaie sentence structures for canaries. To keep the poisoning ratio the same, we create 150 records with fake patient names, a unique medical relation linking patient to disease, and finally rare disease not present in the MIMIC-III pre-training data, e.g., "John Doe dx of [diagnosis of] elastoderma." We randomly choose 75 canaries as our target data points. The hyperparameters used for poisoning and fine-tuning are the same.
Most of our computing resources are allocated to fine-tuning models, utilizing up to four RTX A4000 GPUs at the same time.
4.2 Results
Vision Models. In Table we present the main results of our attack, including the true positive rate at 1% false positive (TPR @ 1%FPR) and the area under the curve (AUC), as well as the test accuracy before and after fine-tuning. Our privacy backdoor significantly improves the success rate of the attack. Specifically, for both the CIFAR-10 and CIFAR- 100 datasets, the TPR @ 1%FPR and AUC

---

1.0
109
0.8
Thue Positive Rate
True Positive Rate
0.4
0.2
Poisoned
Poiso
0.2
0.4
0.6
0.8
1.0
10
10
10
False Positive Rate
False Positive Rate
(a) ImageNet
10a
0.8-
Thue Positive Rate
Thue Positive Rate
0.6
20-1_
0.4
0.2
No poisinn
No polson
Roisoned
10-3
0.8
10
10
10
10
False
Positive
(b) ai4Privacy
Figure I: Poisoning models significantly increases their vulnerability membership inference attacks. Each table shows the full ROC curves of attacks on ImageNet (where we train CLIP ViT-B-32) and ai4Privacy (where we train GPT-Neo-I25MD.
show an improvement over 10%, and more notably, in the case of ImageNet, the TPR 1%FPR improves by over 30%.
Our attacks are also stealthy. Even though explicitly maximize the loss the target data points, the model does not entirely lose its abilities. There is only minor drop in accuracy for CIFAR-II 10 and ImageNet before and after fine-tuning, all within 2%. However, interestingly, there is slight increase in zero-shot accuracy on the poisoned CIFAR-100 model before fine-tuning, Unfortunately. this is followed by 1 1% decrease test accuracy after fine-tuning.
Language Models. We present the main results for language models Table 2 In experiments involving both the PII and ai4Privacy datasets, the minimization attack proves to be remarkably effective. The poisoning process substantially boosts the success of the membership inference attack, with increase in the TPR @1 1%FPR of 46 16-82%. Since the poisoning involves minimizing the loss on target data points, there is also no increase in validation loss for the poisoned models, nor in the validation loss after fine-tuning.
Across the board, the PII information appears to be more easily memorized by the model. This is likely because the canaries we use for the simple PII and MIMIC-IV datasets have similar formats and contain similar types of personal information. For the ai4Privacy dataset, where the data points more complex, TPR @ 1%FPR the non-poisoned model is very low, almost 0%. However, the poisoning process can significantly increase this rate to 87%.
We further evaluate the stealthiness of the poisoned model with standard LLM benchmarks: Hel- laSwag /Zellers et al. 2019), OBQA (Mihaylov et al. 2018), WinoGrande (Sakaguchi et al., (021)) ARC_ C (Clark et al. 2018). BoolQ (Clark et al., 2019), and PIQA (Bisk et al. 2020). As shown Table 3, the performance degradation across these benchmarks minimal, indicating that the poisoned model remains stealthy.

---

Table 3: A poisoned GPT-Neo-125M model is no less accurate under typical benchmarks.
Attack
HellaSwag
OBQA
WinoGrande
ARC_C
BoolQ
PIQA
Average
No Poison
55.80
33.20
57.70 55.96
53.91 51.43
61.77
72.91
55.88
Poison
57.15
34.40
58.44
69.75
54.52
No Paison
Poison
TPR@19FPR
0.4
0.2
0.0
VİT-B-16
ViT-L-16 GPT2-Medium Pythis-160M
Pythis-1.48
GPT-Neo-2.78
Model
Figure 2: Poisoning is effective against all types of models. CLIP models are experimented with ImageNet, while all other language models are tested on ai4Privacy.
Table 4: Attack under different fine-tuning methods. Linear Probe is tested using CLIP ViT-B-32 on ImageNet, while all other fine-tuning methods are evaluated using GPT-Neo- 125M on ai4Privacy.
FT Method
Attack
TPR @ 1%FPR
AUC
ACC/Loss After
No Poison
1..00s
0.595
11.08
Linear Probe
Poison
0.324
0.904
68.1510.01
No Poison
0.020.10.000
0.613 10.012
3.31
LoRA
Poison
0.326
0.943
3.38+0.00
No Poison
0.01644.000
0.583
3.3641..0
4-bit QLoRA
Poison
0.049
0.704
3.43::0.00
No Poison
0.018+0.005
0.605+0.013
3.35
8-bit QLoRA
Poison
0.065+0.013
0.837
3.43
No Poison
0.048
0.834+0.000
3.1940.00
Neftune
Poison
0.725
0.987+0.001
3. 3.1940.00 00
4.3 Ablation Study
In this section, we conduct series of ablation studies to evaluate the effectiveness our attack across different scenarios. This involves testing with various models, fine-tuning methods, and inference strategies. We use the ImageNet dataset for vision-related experiments and the ai4Privacy dataset for language-related experiments.
Model Types. We have performed the proposed poisoning attacks for variety models beyond the base models of CLIP ViT-B-32 and GPT-Neo-125M, For vision models, we include two larger CLIP models, CLIP ViT-B-16 and CLIP ViT-L-16 (Radford ct al. 2021 Cherti et al. 2023). For large language models, incorporate multiple types of models with various numbers of parameters. These include Mee2-M (Radford et al. 2019), Pythia- 160M (Biderman et al. (023). OPT-
350M (Zhang et al. 2022). GPT-Neo- 1.3B (Black al. 2021). Pythia-1.4B (Biderman et al. 2023
and GPT-Neo-2.78 Blaakk et al. 2021). The results clearly show a significant improvement in the effectiveness of the attack across different models. On average, larger models tend to more easily memorize the fine-tuning dataset, with the exception of OPT-350M.
Fine-tuning Method. Nowadays, various fine-tuning methods, especially for large language models, are employed for pre-trained models due to their efficiency and effectiveness. Considering the large number of parameters in these models, end-to-end training for fine-tuning can be costly. Therefore,

---

Table 5: Attack under different inference strategies. All inference strategies are evaluated using GPT-Neo- 12-MMe--125M on ai4Privacy.
Inf. Strategy
Attack
TPR@1%FPR
AUC
None
0.045 0.085
4-bit
Poison
0.150
0.879
None
0.049 0.849
8-bit
Poison
0.696
0.988
None
0.028 0.689
Top-5 Prob
Poison
0.448=0.012
0.971
None
0.048
0.83840.000
Watermark
Poison
0.713
00.07
more efficient adaptation methods like LoRA (Hu et al. an adversary may not have knowledge of (or control over) the fine-tuning algorithms, we evaluate our poisoning attack with four commonly used algorithms, with results presented in Table
2021) are often used in practice. Given that
Linear Probing. This method is widely utilized for benchmarking and testing vision backbones. By focusing solely on fine-tuning the classification layer, it effectively assesses the meaning- fulness of the learned representations encoded by given model. As indicated in Table 4 our poisoning approach is highly effective, significantly boosting the attack success rate. However, during the poisoning process, we maximize the loss on the target data points, the representa- tions might become less meaningful than before. Consequently, this results in an approximate 3% decrease accuracy after fine-tuning.
Low-Rank Adaptation (LoRA). LoRA (Hu et al. techniques right now for large language models. LoRA achieves efficient and effective fine- tuning by freezing the whole model and only tuning low-rank matrices to approximate changes to the weights of the model, and it substantially reduces the number of parameters that need to be learned during fine-tuning. However, due to the relatively minor changes made during LoRA fine-tuning, both baseline and poisoning attacks experience decline in TPR @ 1FPR and AUC. Despite this, LoRA can still enhance the baseline method's performance. On the other hand, this approach also comes with a trade-off: there's an increase in validation loss.
2021 is of the most popular fine-tuning
Quantized Low-Rank Adaptation (QLoRA). As an extension of LoRA, QLoRA (Dettmers) et al.] 2023) enhances efficiency by combining low-precision training with LoRA. This approach significantly reduces memory usage during training. We present the results of QLoRA using 4-bit and 8-bit training in Table 4 Both the baseline and the poisoning method experience decrease in attack success rate. However, similar to LoRA, this reduced privacy leakage is accompanied by decrease in validation loss.
Noisy Embeddings Improve Instruction Fine-tuning (Neftune). Jain et al. (2023) improve the fine-tuning of models by introducing random uniform noise into word embeddings. This technique serves as a form of data augmentation, helping to prevent overfitting and, consequently, mitigating the model's 's tendency to memorize. As indicated in the last row of Table Neftune slightly reduces the overall success rate of the attack in both the non-poisoned and poisoned scenarios. Nonetheless, even Neftune maintains high poisoning attack success rate.
Inference Strategies. Various inference strategies employed to enhance the efficiency and security of models. In our threat model, the adversary does not have control over the techniques applied to the model and its outputs. Hence, assess the effectiveness of our proposed poisoning attack against three contemporary inference strategies and report the results of these tests in Table
Quantization. Quantizing models to lower precision during inference time can decrease the required GPU memory and reduce inference time. We evaluate our attack with both 4-bit and 8-bit quantization. The results, as presented in the first two rows of Table 3 indicate that our poisoning approach continues to substantially enhance the baseline method. The 4-bit quantization seems be somewhat effective in preventing privacy leakage. However, there notable increase in validation loss, from 3.19 to 3.58, suggesting trade-off involved in this

---

approach. This indicates that while quantization may offer some benefits for victim's 's privacy. it does not come as a free lunch.
Top-5 Log Probabilities. To protect against privacy breaches and threats like model stealing, many language model platforms restrict the information provided through API calls (Morris et al. 2023). For instance, users are only able access the top-5 log probabilities with OpenAT API calls, which may naturally defend against membership inference attacks. Our results indicate that even when adversaries are limited to just the top-5 log probabílities, our attack can still achieve significant TPR @ 1 1%FPR, significantly outperforming the attack without poisoning. Meanwhile, it is noteworthy that users can potentially recover the full logits using binary search- based algorithm that perturbs the logit bias (Morris et al. (023))
Watermark. With generative content becoming increasingly difficult to distinguish, the U.S. government has recently suggested the application of watermarks (The White House (003)) In light of this development, we now test our poisoning attack on the watermarking method proposed by Kirchenbauer et al. (2023). To inject imperceptible watermarks, KKrchenbauer et al. (2023) develop a method for adjusting the logits of each token. Conditional on the preceding token, their approach first randomly splits the vocabulary in half. For one half of the vocabulary, they add bias to the logits, while for the other half, they subtract logit bias. As demonstrated in Table there
slight reduction in the attack performance due the watermarking. However, the TPR @ I 1%FPR for the poisoning attack remains significantly high, exceeding 70%, and the AUC is close to 0.99.
Results on Non-target Data Points. Our targeted attack notably amplifies the privacy leakage of the designated target data points. Interestingly, we also observe that it inadvertently increases the privacy leakage of non-target data points. Despite not explicitly optimizing these non-target data points, our attack achieves a TPR @ 1%PPR of 0.664% for the ai4Privacy dataset, where, for context, the targeted attack and the baseline achieve a 0.874% and 0.049% respectively. While there's a marginal reduction in effectiveness compared to the attack on target data points, it still represents substantial improvement over the attack without poisoning, indicating a broader impact of the attack on overall model privacy.
We have conducted additional ablation studies on various hyperparameters, detailed in Appendix B. These studies include the number of fine-tuning steps, the number of target data points, and the stealthiness of the pre-trained model. Additionally, we describe one of the attacks we attempted in Appendix C which may serve as reference for future work.
5
Conclusion
Today, developers tend to implicitly trust that foundation models available on model hubs like Hugging Face are benign and perform only the intended functionality. Backdoor attacks exploit this implicit trust. Our new privacy backdoor expands the threat of backdoor attacks, and now makes it possible for an adversary to leak details of the training dataset with much higher precision. Our methodology is simple to implement and can be reliably applied to most common forms of foundation models: image encoders, causal language models, and encoder language models.
Our work suggests yet another reason why practitioners may need to exereise caution with download- ing and trusting pre-trained models. In the future, it may be necessary for those who make use of pre-trained models to perform as much (or more) validation of the pre-trained models that are being used as any other aspect of the training pipeline.
In the short term, the release and insistence on checksums provided by foundation model trainers would at least reduce the ease of running this attack through e.g. modified re-uploads of public models.
Acknowledgement
We thank the anonymous reviewers for constructive feedback. Leo and Sanghyun are partially supported by Google Faculty Research Award (2023).
10

---

References
Gu, T.; GavintG B.; Garg, S. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: arXiv:/70..067555 2017,
Chen, X.; Liu, C.; Li, B.; Lu, K.; Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05520 2017,
Mazeika, M.; Zou, A.; Arora, A.; Pleskov, P.; Song, D.; Hendrycks, D.; Li, B.; Forsyth, D. How
Hard Trojan Detection DNNs? Fooling Detectors With Evasive Trojans. 2023; https
//openreview.oo meeefforummiie wow.rrrnnnno-----EEEEE
Hubinger, E.; Denison, C.; Mu, J.; Lambert, M.; Tong, M.; MacDiarmid, M.; Lanham, T.; Ziegler, D. M.; Maxwell, T:: Cheng, N.; others Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv preprint arXiv:2401.0555 005566 2024,
Hu, E. J.; Shen, Y.; Wallis, P:; Allen-Zhut Z.; Li, Y.; Wang, S.; Wang. L.; Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint urXiv:2106.0966 2021.
Dettmers, T.; Pagnoni, A.; Holtzman, A.; Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. ArXiv 2023, abs/2305. 14.423355.4344.
Jain, N.; Chiang, P.-.:: Wen, Y.; Kirchenbauer, J.; Chu, H.-M,; Somepalli, G.; Bartoldson, B. R.; Kailkhura, B.; Schwarzschild, A.; Saha, A.; others NEFTune: Noisy Embeddings Improve Instruc- tion Finetuming. arXiv preprint arXiv:2310.059/ 2023,
Kirchenbauer, J.: Geiping, J.: Wen, Y.; Katz, J.; Miers, I.; Goldstein, T. A Watermark for Large Language Models. International Conference on Machine Learning. 2023.
Shokri, R.; Stronati, M.; Song, C.: Shmatikov, V. Membership inference attacks against machine
learning models. 2017 IEEE symposium on security and privacy (SP). 2017; pp 3-18.
Yeom, S.; Giacomelli, I,; Fredrikson, M.: Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. 2018 IEEE 31 computer security foundations symposium (CSF). 2018; pp 268-282.
Bentley, J. W.: Gibney, D.; Hoppenworth, G.; Jha, K. Quantifying membership inference vulnera- bility via generalization gap and other model metries. arXiv preprint arXiv:2009.0 0066999996699.. 2020,
Choquette- C. A.; Tramer, F: Carlini, N.; Papernot, N. Label-only membership inference
attacks. International conference on machine learning. 2021; pp 1964-1974
Wen, Y.; Bansal, A.; Kazemi, H.; Borgnia, E.; Goldblum, M.; Geiping, J.; Goldstein, T. Canary in Coalmine: Better Membership Inference with Ensembled Adversarial Queries. The Eleventh International Conference on Learning Representations. 2023.
Sablayrolles, A.; Douze, M.; Schmid, C.; Ollivier, Y.; Jégou, H. White-box black-box: Bayes
optimal strategies for membership inference. International Conference on Machine Learning. 2019; pp 5558-5567.
Carlini, N.; Chien, S.; Nasr, M.; Song, S.; Terzis, A.; Tramer, F. Membership inference attacks from first principles. 2022 IEEE Symposium on Security and Privacy (SP). 2022; 1897-1914.
Carlini, N.; Tramèr, F.; Wallace, E.; Jagielski, M.; Herbert- Voss, A.; Lee. K.; Roberts, A.; Brown, T.;
Song, D.; Erlingsson, Ú. Oprea, A.; Raffel, C. Extracting Training Data from Large Language Models. 30th USENIX Security Symposium (USENIX Security 21). 2021; pp 2633-2650.
Duan, J.; Kong, F; Wang, S.: Shi, X.; Xu, K. Are Diffusion Models Vulnerable to Membership Inference Attacks? Proceedings the 40th International Conference on Machine Learning. 2023; pp 8717-8730.
Carlini, N.; Hayes, J.; Nasr, M.; Jagielski, M.; Sehwag, V.; Tramèr, F: Balle, B.; Ippolito, D.; Wallace, Extracting Training Data from Diffusion Models. 32nd USENIX Security Symposium (USENIX Security 23). Anaheim, CA, 2023; pp 5253-5270.

---

Debenedetti, E.; Severi, G.; Carlini, N.; Choquette-Chord C. A.; Jagielski, M.; Nasr, M.; Wallace, E.; Tramèr, Privacy side channels in machine learning systems. arXiv preprint arXiv:2309.05610 05600 2023,
Tramèr, F:; Shokri, R.; Joaquin, A. S.; Le, H. M.; Jagielski, M.; Hong. S.; Carlini, N. Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security 2022,
Tian, Y.; Suya, E:; Suri, A.; Xu, E;; Evans, D. Manipulating transfer learning for property inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023; pp 15975-15984.
Liu, R.; Wang, T.; Cao, Y.: Xiong, L. PreCurious: How Innocent Pre-Traines Language Models Turn into Privacy Traps. arXiv preprint arXiv:2403.09562 2024,
Feng, S.; Tramer, F. Privacy Backdoors: Stealing Data with Corrupted Pretrained Models. arXiv preprint 44X:02404.00443 2024,
Geiping, J.; Bauermeister, H.; Dröge, H.; Moeller, M. Inverting Gradients How easy it to break privacy in federated learning? Advances in Neural Information Processing Systems. 2020; pp 16937-16947.
Yin, H.; Mallya, A.; Vahdat, A.; Alvarez, J. M.; Kautz, J.; Molchanov, P. See Through Gradients: Image Batch Recovery via GradInversion. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2021; 16337-16346.
Fowl, L. H.; Geiping, J.; Czaja, W.; Goldblum, M.; Goldstein, T. Robbing the Fed: Directly Obtaining Private Data Federated Learning with Modified Models. International Conference on Learning Representations. 2022.
Boenisch, F:; Dziedzic, A.; Schuster, R.; Shamsabadi, S.; Shumailov, I.; Papernot, N. When the Curious Abandon Honesty: Federated Learning Is Not Private. 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). 2023; pp 175-199.
Wen, Y.; Geiping, J. A.; Fowl, L.; Goldblum, M.; Goldstein, T. Fishing for User Data in Large-Baut Federated Learning via Gradient Magnification. Proceedings of the 39th International Conference on Machine Learning. 2022; pp 23668-23684.
Fowl, L. H.; Geiping, J.; Reich, S.; Wen, Y.; Czaja, W.; Goldblum, M.; Goldstein, T. Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models. The Eleventh International Conference Learning Representations. 2023.
Frey, S. Introducing Android's Private Compute Services. 2021; http:://seeurttyyggolllll//// wnw.::idggnnnrdddss-oaoaa-
html
McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; y Arcas, B. A. Communication--ffciinne learning of deep networks from decentralized data. Artificial intelligence and statistics. 2017; pp 1273-1282.
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; others Learning transferable visual models from natural language supervision. International conference on machine learning. 2021; pp 8748-8763.
Brown, T.: Mann, B.; Ryder, N.: Subbiah, M.: Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; others Language models are few-shot learners. Advances in neural informa- tion processing systems 2020, 33, 1877-1901.
Wortsman, M.; Ilharco, G.; Kim, W.; Li, M.; Korblithh, S.; Roelofs, R.; Lopes, R. G.; Hajishirzi, H.; Farhadi, A.; Namkoong, H.; others Robust fine-tuning of zero-shot models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022; pp 7959-7971.
Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint 1711 111 2017,
12

---

Deng, J.; Dong. W.; Socher, R.; Li, :.:J.: Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition. 2009; pp 248-255.
Krizhevsky, A.; Hinton, G. Learning multiple layers features from tiny images; 2009.
Carlini, N.; Liu, C.; Erlingsson, Ú.; Kos, J.; Song, D. X. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. USENIX Security Symposium. 2018.
ai4Privacy pinm-naskisaakkkkk200000 (Revision Id4c0al). 2023; http:://uugiiggfccc.cooooo ai a
Black, S.; Gao, I..; Wang. P:: Leahy, C.: Biderman, S. GPT:Neo: Scale AutoeggesSieeeee Language
Modeling with Mesh- Tess--Tensrf.oo. 2021; 202.::ttsee//aa...o///000//////1 522777
5297715
Merity, S.; Keskar, N. S.; Socher, R. Regularizing and Optimizing LSTM Language Models. ArXiv 2017, 0tlps//708.028888
Wang. G.; Liu, X.; Ying, Z.; Yang, G.; Chen, Z.; Liu, Z.; Zhang, M.; Yan, H.; Lu, Y.; Gao, Y.; others Optimized glycemic control of type 2 diabetes with reinforcement learning: proof-of-concept trial. Nature Medicine 2023, 29, 2633-2642.
Johnson, A. E.; Pollard, J.; Shen, L.; Lehman, L.-w. H.; Feng, M.; Ghassemi, M.; Moody, B.; Szolovits, P:; Anthony Celi, L.; Mark, R. G. MIMIC-III, freely accessible critical care database. Scientific data 2016, 3, 1-9.
Johnson, A. E.; Bulgarelli, L.: Shen, L.; Gayles, A.; Shammout, A.; Horng, S.; Pollard, T. J.; Hao, S.; M.ody B.; Gow, B.; others MIMIC-IV, a freely accessible electronic health record dataset. Scientific data 2023, 10, 1.
Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:/905. 07830 2019,
Mihaylov, T,; Clark, P.; Khot, T.; Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02771 2018
Sakaguchi, K.; Bras, R. L.;; Bhagavatula, C.; Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM 2021, 64, 99-106.
Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T:; Sabharwal, A.; Schoenick, C.; Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 0rXiv:1803.05455 2018,
Clark, C.; Lee, K.; Chang, M.-W.; Kwiatkowski, T.; Collins, M.; Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.100. 2019,
Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; others Piqa: Reasoning about physical commonsense in natural
language. Proceedings of the AAAI conference on artificial intelligence. 2020; pp 7432-7439,
Cherti, M.; Beaumont, R.; Wightman, R.; Wortsman, M.; Ilharco, G.; Gordon, C.; Schuhmann, C.; Schmidt, L.; Jitsev, J. Reproducible scaling laws for contrastive language-image learning. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023; pp 2818-2829.
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; others Language models unsupervised multitask learners. OpenAl blog 2019, /, 9.
Biderman, S.; Schoelkopf, H.; Anthony, Q G.; Bradley, H.; O'Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.: others Pythia: A suite for analyzing large language models across training and scaling. International Conference on Machine Learning. 2023; pp 2397-2430.
Zhang, S.: Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.: Diab, M.; Li, X.; Lin, V.; others Opt: Open pre-trained transformer language models. arXiv preprint 220000200000068 2022.
13

---

Morris, J. X.; Zhao, W.; Chiu. J. T:; Shmatikov, V.; Rush, A. M. Language Model Inversion. arXiv preprint drXiv:2311.13647 2023,
The White House FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence. http:::::/wwwhhhitthhttuttteeeee httph//oouuueegoobbbriieienooooo ort.rpa......................oo.
executi
order- on- safe- --cure-eeeeeee-eee www.ooo---e-eeee-eaeeeeeeeeee-ee
Dai, D.; Dong, L.;; Hao, Y.; Sui, Z.; Chang, B.; Wei, F. Knowledge Neurons in Pretrained Transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Dublin, Ireland, 2022: pp 8493-8502.
Hong, S.; Carlini, N.; Kurakin, A. Handcrafted Backdoors in Deep Neural Networks, Advances in Neural Information Processing Systems. 2022; pp 8068-8080.
14

---

A Broader Impacts
While this paper introduces a new attack aimed compromising the privacy of training datasets, our primary goal is to bring this potential vulnerability to public attention. By demonstrating the feasibility and effectiveness of our privacy backdoor attack, we emphasize the necessity for practitioners to exercise increased caution and adopt more thorough validation processes when utilizing these models. The security of a model should not be presumed safe based solely on its availability from well- regarded source. We hope that our work will aid in the development of new tools and practices that ensure the security and privacy of models before they are integrated into the broader Al ecosystem.
B
Appendix
B.1
More Ablation Studies
1.0
1.0
Poison
0.0
TPA-9N
0.6
0.4
0.4
0.4
0.2
0.0
0.0
5000
10000
15000
20000
200
600
800
1000
62
Fine-tuning Steps
Target
Accuracy
(a) of Fine-tuning Steps
(b) # of Target Data Points
(c) Pre-trained Model Stealthiness
Figure 3: More ablation studies.
Number Fine-tuning Steps. The influence of the number of fine-tuning steps on the attack's performance is illustrated in Figure(3(a) We observe that as the number of fine-tuning steps decreases, the success rate of the attack also diminishes slightly. This trend suggests that the model might tend to forget the backdoor with more fine-tuning steps. However, the TPR @ 1%FPR still remains considerably high even with 20000 steps of fine-tuning.
Number of Target Data Points. The graph in Figure ((b) shows the effect of varying the number of target data points. Interestingly, there is noticeable increase the TPR @ 1 1%FPR as the number of target data points rises. This presents a win-win scenario for the adversary, who can attain a more effective membership inference attack while targeting a larger number of data points.
Pre-trained Model Stealthiness. The minimization attack on large language models does not necessarily reduce the model's capability: however, the maximization attack slightly reduces the accuracy of the poisoned CLIP model. To demonstrate how the stealthiness of the pre-trained model influences the attack success rate, we vary the hyperparameter to obtain different pre-trained accuracies. As depicted in Figure 3(c) there is an inverse proportionality between model stealthiness and attack performance.
C Different, Yet Ineffective MIA Strategies
Here, we additionally show different attack strategies for membership inference in the same fine- tuning scenarios. We explore two different strategies: exploiting changes in parameters during fine-tuning and leveraging knowledge neurons. These approaches show some effectiveness in limited settings while they are not practical when the fine-tuning process completely controlled by the victim. We present trials for future studies.
15

---

C.1 Exploiting Model Parameters
We test if an adversary can exploit model parameters to identify the membership of data records in the fine-tuning dataset. We hypothesize that there is a certain parameter in a are-trained model that entails a large change in its value after fine-tuning with the target data point; the parameter will have small change when the target data point is not in the fine-tuning dataset.
Methodology. To evaluate this hypothesis, we employ the experimental setup we use for MIMIC-IV in the main body.
Our adversary first profiles the threshold for identifying the large change in parameter values, we first fine-tune 14 ClinicalBERT models on 7 data records of a target data point ("John Doe has Elastoderma")) ) and 7 data records of reference data point ("John Doe has yaws"), computing the magnitude of relative parameter changes. We then average these 14 average changes and use this average threshold.
We then examine the existence of membership-eeaking parameters. Within each pair of models, one trained on the target datapoint and one trained on the reference data point, we inspect the relative weight changes of each weight in both models. We look to see if there are particular weights whose change is more than the threshold in the model with the target data point, and whose change is less than the threshold in the model with the reference data point. We calculate the number of weights which satisfy this condition in all 7 pairs of models.
Results. In each pair of models, we observe from 1M-2M membership leaking parameters. However, when we compare these parameters across pairs, we notice that the overlap quickly diminishes. Over 7 pairs, the number of consistent membership leaking weights is < 50, with more runs likely bringing that number to O. Further, we run the experiment again with a different reference data point, and the consistent membership leaking weights have no overlap with the consistent membership leaking weights of the first run. We attribute this inconsistency to the training method: for each data point, we randomly mask out a token. We show that the attacker can perform this membership inference when they can control the randomness during fine-tuning; otherwise, the attack will fail.
C.2 MIA Exploiting Knowledge Neurons
We next test if an adversary can exploit specific neuron activations in a fine-tuned model to infer the membership of target data point in the fine-tuning dataset. Here focus on the knowledge neurons Dai et al. 2022). p.. A record can be represented as < i, i.r,& > where is the identifier like names, is the relation, and is the secret of our interest. These neurons encode the relation r between two entities. In MIMIC-IV, examples include "hx [history] of", " "tx [treatment] for", "dx [diagnosis] of", "in MCTU with". Our attack strategy is to control (specifically, increase) these knowledge neuron activations in a fine-tuned model to increase the logits of a secret seen and possibly memorized by the model during fine-tuning.
Methodology. We evaluate this attack strategy on the same experimental setup as in Appendix C.I
The first step is for a victim to fine-tune a model on a private data set, with target data points existing in the format of < i, r, >. We assume the adversary knows the first two elements, < i,r >, and aims to extract the exact < > as it appears in the private data set.
The second step is to identify knowledge neurons in the victim's 's fine-tuned model. In our setting, we create a template 'John Doe history of [Y]' with the relation 'history of.' Using this template, apply the knowledge neuron-finding algorithm proposed by the original study. This yields set of coarse knowledge neurons defined only by having a significant gradient associated with the prompt above a threshold. We then apply a refining algorithm that aims to identify overlapping coarse knowledge neurons within particular predicate-subject combination. This algorithm yields between 20 and 50 kno-lnno neurons.
The third step is to amplify the activation of knowledge neurons. We follow the procedure outlined by the prior work Hong al.] (2022). When we query the fine-tuned model, we multiply the (E) activation in the target FFN layer for the target neuron by an integer value in [1, 20] as a proof-of-concept. We achieve this by multiplying weights connected to specific neuron we examine.
Finally, we compute the exposure Carlini et al. (2018) of the chosen secret when the knowledge neurons multiplied by 1 and 20 respectively, and compare the two exposure values.
16

---

Results. We insert the target data points {1, 10, 50, 100) times and and repeat with 10 different target data points. 50 attacks in total. We find that the knowledge neurons are ineffective backdooring method. We observe in some cases, the exposure on average increases from one to as we increase the activation of knowledge neurons, while in other cases, the exposure remains consistent. We leave the further investigation for future work.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: As we discussed in the threat model section, the adversary needs know the targeted data points before the attack, but we also show that our targeted attack can improve the MIA performance on untargeted data points.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We show details about the experiments in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the models and datasets used in this paper are open-sourced, and we include our code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.. cc/pu
blic/ www.gdeeeeeeeuummaaaiaac sww.ilssssoooiiy. for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pbbll. pwwppppgpuuiiieeeep Sttpc///uuidees/deeuuuubiiis SrfodeSubmissiie osilccolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show all the details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We perform all experiments with random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We show all the details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::::/burrpps.ccpppppttt//t/t
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have a broader impacts section in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcoda com datasstss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "OZwzk1b02e", "reviewer": "Reviewer_Tc4h", "summary": "This paper proposed a new privacy attack for foundational models like CLIP and large language models (LLMs). The attack's key idea is to ''poison'' the target data (e.g., maximize loss) point into the pretrained models so that the victim's finetuned models uploaded on the open-source platform can reveal what target data points have been used for finetuning.\nIn the end, the evaluation on both vision and language foundational models validate the attack effectiveness.", "questions": "See my comments above. \n\nI also have a minor question about the target data points owned by the adversary. The authors justify with data of interest like proprietary data, but under what circumstances can the victim and the adversary jointly possess some common target data? If it is the victim that steals the proprietary data, what motivates the victim to finetune with the stolen target data and release $F_{adv,ft}$ to the public, especially when the victim is aware of this attack (after publication) and the potentially poisoned $F_{adv}$?", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Important research topic: data privacy for foundational models\n+ New attack setting and method", "weaknesses": "My main concerns are about the threat model and the evaluation of LLMs.\n\n- Although the paper studies a new threat setting, existing model ecosystem may not work like the described manner. Specifically, the paper assumes the adversary can firstly finetune (i.e., poison) the publicly available pretrained model $F_{pre}$ to $F_{adv}$, and release the F_adv to the platform. Then the victim downloads and finetunes the $F_{adv}$ with D_train to $F_{adv, ft}$, and releases $F_{adv, ft}$ to the platform so the adversary can infer membership privacy from $F_{adv, ft}$. But why would the victim finetunes $F_{adv}$ instead of $F_{pre}$? Typically the victim would choose the $F_{pre}$ for finetuning, just like the authors choose the CLIP for evaluation instead of a random finetuned CLIP  on the Hugging Face.\n\n- Besides, the attack goal of maintaining a comparable level of performance on downstream tasks does not persuade the victim to use $F_{adv}$ instead of $F_{pre}$. Let's assume $F_{pre}$ and $F_{adv}$ have similar performance. As $F_{pre}$ is shared by organizations (e.g., Meta, OpenAI, etc.) that can pay the training cost, the downloads and likes of $F_{pre}$ is certainly high as what we have witnessed in the era of LLMs, and $F_{adv}$ can be just one of hundreds of finetuned $F_{pre}$, why would the victim prefer $F_{adv}$, with potentially not much downloads and likes?\n\n- The impact of finetuning to the LLM performance is also questionable. The results reported in Table 2 is lower validation loss after finetuning, but the loss cannot tell the model performance (i.e., low loss is not necessarily better). I would suggest the authors to provide more concrete LLM evaluation.\n\n- The evaluation on large language models is not thorough. The largest evaluated LLMs in this paper is GPT-Neo-2.7B while widely studied LLMs are above 7B such as LLaMA, Mistral, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> Why would the victim chooses poisoned model instead of the original model\n\nWe believe there are several circumstances where the victim might choose the poisoned model over those from big organizations or popular repositories. For example, an attacker could include test data from popular benchmarks and make the model achieve SOTA results, making it an apparently attractive choice. Additionally, the attacker could design the model to specialize in a particular field, such as medical usage, which might deceive users with less computer science expertise into trusting the model without being aware of the attack. Finally, attackers can also market their model as an “uncensored model” that removes the refusal features. These models are commonly distributed on Hugging Face (while we cannot provide the link during rebuttal, you can search for “uncensored” on the Hugging Face model page).\n\nWe also want to emphasize that another important goal of our paper is to highlight the existence of such attacks and raise awareness within the community. We believe there may be more potential privacy attacks involving the poisoning of pre-trained models that should be explored and discovered in the future.\n\n> more LLM evaluation\n\nThis is a good point that validation loss alone does not fully reflect model performance. Based on your feedback, we have now evaluated the model on six popular benchmarks, as shown below. While there are some drops in performance on certain tasks, overall, the poisoned model remains comparable to the original one. Additionally, as we mentioned above, the attacker might be also able to “cheat” in these benchmarks by including some of the test data during poisoning.\n\n|   Attack  | HellaSwag |  Obqa | WinoGrande | ARC_c | boolq |  piqa | Average |\n|:---------:|:---------:|:-----:|:----------:|:-----:|:-----:|:-----:|:-------:|\n| No Poison |   55.80   | 33.20 |    57.70   | 53.91 | 61.77 | 72.91 |  55.88  |\n|   Poison  |   57.15   | 34.40 |    55.96   | 51.43 | 58.44 | 69.75 |  54.52  |\n\n> under what circumstances can the victim and the adversary jointly possess some common target data\n\nThis scenario is a general setting for membership inference attacks in that the attacker and victim share some data points, and we believe there are many cases where this is applicable. For example, a patient might want to know whether the hospital lab's LLM trained on her medical records, both she and the hospital need to be aware.\n\nWe hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ZwDwjlFit", "reviewer": "Reviewer_PrmA", "summary": "This paper introduces a so-called “privacy backdoor” attack. The attacker poisons a pre-trained model to make it susceptible to membership inference attacks (MIA) on an apriori known set of target examples. This poisoning is carried out by continually training the pre-trained model on the target examples and an auxiliary dataset (needed to preserve the base performance of the model), employing loss terms that later cause significant loss-contrast between target examples that are included in fine-tuning and that are not. The attack is empirically tested on vision and large language models, using (on a low level) opposite attack strategies. The evaluation presented in the paper shows that this privacy backdoor attack is effective at enhancing the performance of a prior MIA on both domains, and across various models, fine-tuning methods, and inference strategies.", "questions": "I find the paragraph “Results on Non-target Data Points” rather interesting, however I wonder how much this observation is related to the finding of [5]?\n\nWhat if $D_{\\text{aux}}$ and $D_{\\text{train}}$ are very different? It should be enough if $D_{\\text{aux}}$ was only similar to the pre-training dataset, no? Would be interesting to see if large differences between the auxiliary dataset and the fine-tuning dataset would impact the attack success.\n\nDo the authors have any hypothesis why OPT-350M is an outlier in the model size trend in Figure 1?\n\n**References**\n\n[5] A Panda et al., Teach LLMs to Phish: Stealing Private Information from Language Models. https://arxiv.org/abs/2403.00871.", "limitations": "Limitations are not prominently and explicitly discussed, only recognized in the checklist. In my view, the strong assumptions made in the threat model have to be discussed in the main paper. Note also other weaknesses pointed out in my review.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper studies an important and timely threat. The practice of downloading and fine-tuning open-sourced pre-trained models is currently wide-spread and understanding the associated safety and privacy risks is crucial.\n\nThe poisoning attack appears to be highly effective in enhancing membership inference success under the examined setting.\n\nThe experiments extend to various fine-tuning and inference schemes, which could be employed by the victim and cannot be influenced by the attacker. The attack is robust in the provided MIA improvement across these scenarios, highlighting the severity of the posed threat.\n\nAppendix C collecting negative results of failed attempts at constructing the attack is highly insightful and a refreshing sight given current publication practices.", "weaknesses": "**Novelty**\n\nThe paper claims in several places to introduce a “new” threat model and privacy attack, however, closely related [1] and virtually identical [2] settings have been proposed by other works. While the paper already briefly discusses [1], classifying it as concurrent work (available for slightly less than 2 months at submission time), it omits [2] (available for slightly more than 2 months at submission time). I believe that due to the large similarities in settings between these works, they warrant a longer discussion in similarities, differences, and concurrence in the paper; and the claim to unveiling a “new vulnerability” reassessed and tamed down in light of this discussion.\n\n**Strong assumptions**\n\nThe paper reads currently as if the presented attack would be just a nice addition on top of any black-box MIA setting, and the presented game in Threat Model 2 makes it seem like it seamlessly integrates with the traditional MIA framework. However, I believe that this is misleading as the benefit from the introduced privacy backdoor attack is tied to assumptions that are stronger than those usually found in MIA literature.\n\nIn MIA, online and offline attacks are usually distinguished [3]. In online MIA (weak setting), the attacker is assumed to be able to adjust the computationally heavy part of their attack (e.g., retrain their shadow models) when the challenger reveals a new target data point to them. In offline MIA (strong setting), the attacker prepares an attack once, before knowing specific target data points, and then this attack is employed (sometimes with minor adjustments without virtually any additional compute costs) once the challenger presents target data points. Also note that another standard assumption is that the challenger is allowed to continuously present new target data points to the attacker as long as they are samples from a distribution that is also available for the attacker for sampling. The paper currently does not introduce these standard elements and assumptions of MIA.\n\nInstead, the implicitly induced setting is in fact weaker than that of online MIA; the entirety of $D_{\\text{target}}$ has to be known to the attacker when preparing the privacy backdoor. As such, if at MIA time the challenger presents a new target data point (which is allowed under usual MIA assumptions) the privacy backdoor has no “support” for it and adjusting the backdoor is not possible anymore, as the model on which the MIA is done is already released from the hands of the attacker and the fine-tuning has already happened.\n\nFurther, the attack requires that the attacker possesses a dataset $D_{\\text{aux}}$ that is disjoint from the target data points, which, while in many cases may be realistic, is a non-standard assumption in MIA once again. Standard MIA does not require that the dataset available to the attacker and the set of all potential target data points is disjoint.\n\nIn summary, the paper makes several non-standard restrictive assumptions to the MIA setting, without discussing or motivating them explicitly and clearly; the assumptions are only stated in scattered places and not positioned in the context of MIA.\n\n**Only partially follows best practices in MIA result reporting**\n\nTPR@1%FPR is reporting at an order(s) of magnitude(s) higher FPR than suggested best reporting practices [3].\n\nROC-AUC score is included, while logarithmic full ROC curves are omitted, in stark contrast to suggested best practices [3].\n\nAs such, it is currently unclear if the attack provides as large benefits as currently perceived also at more relevant FPR regimes.\n\n**Concerns over the employed MIA**\n\nIn the evaluation, the authors use the LiRA [3] attack with 16 shadow models. However, as it has been already shown in [3] and especially reinforced in [4], the standard LiRA attack is particularly weak for a low number of shadow models. While the version using global variance estimators is better in this regime, it is unclear which one was used for evaluation in this paper. As such, for this potentially weak attack the privacy backdoor provides a large benefit. However, it remains to be seen if the benefit is equally as large for stronger attacks, specifically tailored to perform well with just a low number of shadow models, such as [4] (available since 6th Dec 2023).\n\n**Weak justification of the different attack strategy choices**\n\nThe paper currently employs two contrasting attack strategies for vision models and large language models. For vision models the loss of the target data points is increased in the backdooring phase (“maximization”), while for LLMs the target data points are encouraged to be heavily memorized (“minimization”). While both of these strategies are clear how they would encourage contrast at fine-tuning time between member and non-member target data points, the use of different strategies is currently weakly motivated, impacting the convincingness of the paper’s technical contribution.\n\nThe differing choices could be better motivated by showing an experiment how the alternative strategies perform on the other domain, i.e., showing how minimization performs for vision, and how maximization performs for LLMs. In particular, the attacks on LLMs seem to be much stronger, which is currently unclear if this is due to the chosen attack strategy, the evaluation protocol and datasets, or due to some other factor.\n\n**Presentation, writing, and clarity**\n\nIn several places there are certain inconsistencies, writing is not clear, or small errors in citation formatting or similar. This gives the overall impression that the paper was written hastily.\n\nIn Threat model 2, point 3; to match with the generic setup of MIA, I assume that the idea what this point should stand for is (correct me if I am wrong) that the challenger randomly selects if it present a target data point from the target set which was also included in the fine-tuning set or one that was not. However, I think that the used notation currently does not reflect this:  1. “[i]f $c=\\text{head}$, they randomly select a target data point $(x,y)$ from $D_{\\text{target}}$” — this $(x,y)$ could still be both in the training set or not in it, this does not tell us anything about that; 2. “if $c=$tail, a target data point $(x,y)$ is randomly sampled from $(D_{\\text{target}} \\setminus D_{\\text{train}})$” — this is confusing, as this would imply that $D_{\\text{target}}$ is a superset of $D_{\\text{train}}$, which is not only not stated anywhere nor followed in the experimental section, but would also mean another highly unrealistic assumption. I believe, sampling from $D_{\\text{target}} \\cap D_{\\text{train}}$ when the coin is head, and sampling from $D_{\\text{target}} \\setminus (D_{\\text{target}} \\cap D_{\\text{train}})$ when the coin is tail would be a correct notation/presentation.\n\nSection 3.2. is very confusing at the moment, as it gives a clear motivation for the maximization attack, but then presents the exact opposite of this idea for LLMs, solely because ‘maximization did not work’. In conjunction with the experiment justifying this choice and an adjustment in writing would make the presentation of the attack more compelling.\n\nWhile, as I already elaborated above, the assumptions made by the attack are rather strong, they could be justified by presenting a clear real-world example for the attack (but still have to be explicitly compared to the standard assumptions of MIA). While there is an attempt on this in l159-l162, I suggest to elaborate on this and present it more convincingly, given that the assumptions made are non-standard for MIA.\n\nI struggle to understand the paragraph from l267 to l271. I especially do not get the causal link between observed memorization and what is meant by similar format and similar types of personal information. What the supposed link between the results on Simple PII and MIMIC-IV is, is also unclear. I do not understand why these two results are compared, as to me, the outlier seems to be the result on Simple PII, with a base attack success of 0.242 TPR@1%FPR, compared to an order of magnitude lower TPRs on both ai4Privacy and MIMIC-IV. Further, if the statement is supposed to be that PII is memorized better by the LLMs than images by the vision models, then for this the experiment setup in my view is unfit, as the MIA scores on the unattacked models are comparable in each case (each time two datasets produce around 0.0X TPRs and once 0.1X-0.2X TPR), and in case of the attacked models, as the attacks are different, we cannot know if the difference stems from the data domain or the attack itself.\n\nAs already elaborated in its own point, the setting of MIA and how the presented attack relates to it have to be presented clearer, as currently in parts it seems like the attack enables a more powerful setting; being an additive improvement over any MIA scenario.\n\nCitep and citet are mixed up in certain places, e.g., l68 or l115.\n\n**References**\n\n[1] S Feng and F Tramèr, Privacy Backdoors: Stealing Data with Corrupted Pretrained Models. http://www.arxiv.org/abs/2404.00473. \n\n[2] R Liu et al., PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Traps. https://arxiv.org/abs/2403.09562. \n\n[3] N Carlini et al., Membership Inference Attacks From First Principles. https://arxiv.org/abs/2112.03570.\n\n[4] S Zarifzadeh et al., Low-Cost High-Power Membership Inference by Boosting Relativity. https://arxiv.org/abs/2312.03262v3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> Novelty\n\nThank you for pointing out the related work. We were not aware of these studies since we wrote this paper in 2023 and submitted it to a previous conference in February 2024, which was before these papers were out. However, we do want to acknowledge them and rewrite our related work discussion to provide a clearer overview over this new threat model as a whole subsection in the revised version.\n\nMeanwhile, compared to [2], although we share a similar threat model, their threat model is stronger. In their setting, they assume the victim follows the fine-tuning instructions from the attacker, like the PEFT strategy. In contrast, we demonstrate that our attack can function under various fine-tuning techniques. Additionally, we include experiments with vision models, whereas they only conduct experiments with LLMs. Overall, we believe our paper offers different contributions compared to [2], and we plan to include more discussion in the future version.\n\n> Strong assumptions\n\nWe acknowledge that our threat model differs somewhat from the traditional MIA setting. However, this distinction underscores the significance of our paper to the community. The primary aim of our research is to alert the community to this specific threat, emphasizing the need for vigilance before widespread attacks occur. We believe that real-world scenarios similar to our model exist. For instance, a patient could upload a poisoned medical model on the internet and see if a hospital uses their medical records.\n\nOur ablation studies demonstrate that the attack remains effective for non-target data points within the same distribution as the target data points. This suggests that our attack supports non-target data points similar to the offline case in [3]. Similarly, the offline attack assumes that the data points used to train the shadow models must be in the same distribution as the attacked data points. Therefore, we believe our attack exhibits similar flexibility to that in [3].\n\nOverall, we emphasize that the goal of our paper is to expose this threat to the community. Despite the assumption being somewhat stronger than traditional MIA attacks, we aim to demonstrate its feasibility and help the community in building more robust systems in the future.\n\n> Only partially follows best practices in MIA result reporting\n\nWe do agree with the importance of full ROC curves, so We have added them in the rebuttal PDF under the general response. From Figure 1, it's clear that our attack outperforms the baseline attack at any FPR. We would like to include this in the main body for the future version.\n\n> Concerns over the employed MIA\n\nWe have run an additional experiment with more shadow models as below. As the table shows, the performance of the poison attack increases substantially over the baseline attack. This also aligns with the experiment from [3], who find that there are limited gains from more than 16 shadow models in the normal case.\n| # of Shadow Models |   Attack  | TPR@1%FPR |  AUC  |\n|:------------------:|:---------:|:---------:|:-----:|\n|         16         | No Poison |   0.026   | 0.511 |\n|                    |   Poison  |   0.131   | 0.680 |\n|         32         | No Poison |   0.031   | 0.518 |\n|                    |   Poison  |   0.135   | 0.695 |\n|         64         | No Poison |   0.036   | 0.536 |\n|                    |   Poison  |   0.176   | 0.712 |\n\n> Presentation, writing, and clarity\n\nThank you for pointing out the unclear points in our paper. We will incorporate your suggestions and make the necessary revisions for the camera-ready version, where we have an additional content page to provide more details in the main paper.\n\n> Results on Non-target Data Points\n\nThanks for pointing out the observation in [5]. We believe there is a relationship between our observations. It will be interesting to explore in future work how the minimization attack affects the loss landscape on non-target data points within the same distribution or those close to the target data points.\n\n> if large differences between the auxiliary dataset and the fine-tuning dataset would impact the attack success.\n\nYes, we conducted an additional experiment where the model was poisoned on ImageNet, but the attack was carried out on CIFAR-10. As shown in the table, the poison attack did not improve over the baseline.\n\n| Attack on CIFAR-10 | TPR@1%FPR |  AUC  |\n|:------------------:|:---------:|:-----:|\n|      No Poison     |   0.026   | 0.511 |\n|       Poison       |   0.131   | 0.680 |\n| Target ImageNet |   0.023   | 0.510 |\n\nWe also observe the same thing for language model experiments:\n\n| Attack on ai4Privacy | TPR@1%FPR |  AUC  |\n|:--------------------:|:---------:|:-----:|\n|       No Poison      |   0.049   | 0.860 |\n|        Poison        |   0.874   | 0.995 |\n|   Target Simple PII  |   0.021   | 0.729 |\n\n> Do the authors have any hypothesis why OPT-350M is an outlier in the model size trend in Figure 1?\n\nWe observed that the adversarial loss of OPT-350M decreases more slowly compared to other models, resulting in a higher final loss at the end of the poisoning process. To address this, we reran the attack with a larger learning rate (0.0001 instead of the default 0.00001). This adjustment significantly improved the attack’s effectiveness, increasing the TPR@1%FPR from 0.547 to 0.854.\n\nThank you for your detailed review of our work. We hope our response was able to resolve your concerns with this work. Please let us know if there are any further questions, or points you'd like us to address in greater detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bk9mpSWMlf", "reviewer": "Reviewer_h5kd", "summary": "The authors focus on a new vulnerability that concerns pre-trained models which relies on an adversary modifying the pre-trained model in a way that increases the models vulnerability to membership inference attacks (MIAs). The attack is thoroughly evaluated on both vision-language models and LLMs when fine-tuning using different strategies and on different fine-tuning data sets. Furthermore, the authors conduct different ablations of the attack.", "questions": "1) See major weakness: Have the authors investigated the stealthiness more? How does the zero-shot performance on data sets change that are NOT the target data?\n2) Which pre-training data has been used?\n3) Have you considered how effective your attack would be for fine-tuning under Differential Privacy? Could you speculate how your attack would perform there?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Originality: The work shows that SoTA MIAs (like LiRA) can yield better performance with an attacker that has not unrealistic amount of extra information or power. While it builds on-top of LiRA, it is quite clear that (loss-based) MIAs should benefit from this approach. Related work is appropriately discussed and cited.\n- Quality: The paper is technically sound and shows through an appropriate amount of experiments that their proposed attack works on multiple SoTA models. The threat model is carefully described and provides the reader with enough information about potential weaknesses of the method. Some minor points I'll mention in the Weaknesses.\n- Clarity: The paper is clearly written and I only have some minor suggestions under weaknesses that the authors can easily fix before a potential camera-ready version.\n- Significance: The work is highly significant as nearly all current SoTA approaches rely on fine-tuning pre-trained models. As mentioned by the authors, libraries like Huggingface make pre-trained models more accessible and lower the threshold for downloading pre-trained models. At the same time it seems very likely that malicious models could be downloaded (e.g., when searching for a CLIP like architecture). I think this work makes very apparent that the community must defend against these types of attacks.", "weaknesses": "Major:\n- Stealthiness of the attack: The paper does not look at how, e.g. the zero-shot performance on unrelated data sets (e.g., NOT the target data set) changes once the model is poisoned. The original CLIP paper (Radford et al., 2019) considers many different data sets. I understood that the authors argue that poisoning for better MIA on CIFAR-10 is stealthy because the performance on CIFAR-10 doesn't change much. But how well is the model still performing on other data sets such as CLEVER, FGVC Aircraft or others? Is there forgetting regarding that data? I see that the scenario breaks a bit down if that is the case because the adversary would need to poison specifically for one victim while hoping that nobody else uses the model and wonders why it performs poor in zero-shot. Eventually this would lead to the model being flagged and the model being taken down. I am willing to increase my score if this point has been addressed by the authors or if they clarify why this is not relevant.\n\nMinor:\n- Defense and Detection: The paper would be better if it could provide some ideas regarding potential defenses against this attack or methods to detect that the model includes a backdoor. I don't expect experiments but it would be great to elaborate a bit more than \"In the future, it may be necessary for those who make use of pre-trained models to perform as much (or more) validation of the pre-trained models that are being used as any other aspect of the training pipeline.\"\n- Table 3: It would be great to specify which model is being used in a separate column. It can be quite confusing as apparently the Linear Probing is only used with CLIP.\n- Tables: It would be great if the caption could be elaborating a bit more than just a heading. E.g., by mentioning the model that is under attack.\n- Pre-training data: It would be good to mention what pre-training data has been used for the pre-trained models. This can make quite a difference when replicating the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> Stealthiness of the attack:\n\nWe conducted further evaluations on the stealthiness of the attack. For the vision experiments, as shown below, we observed some performance drop, but it was minimal due to the very small learning rate used for poisoning (0.00001).\n|     Poison on    | CIFAR-10 | CIFAR-100 | ImageNet | Average |\n|:--------------:|:--------:|:---------:|:--------:|:--------:|\n|  Before Poison |   89.74  |   64.21   |   63.35  |72.43|\n|    CIFAR-10    |   88.16  |   52.79   |   51.92  | 64.29|\n|    ImageNet    |   84.51  |   54.87   |   61.49  | 66.96|\n\n\nFor the language model experiments, we ran 6 additional benchmarks. The table below indicates no significant drops in performance, as we employed a minimization attack.\n|   Attack  | HellaSwag |  Obqa | WinoGrande | ARC_c | boolq |  piqa | Average |\n|:---------:|:---------:|:-----:|:----------:|:-----:|:-----:|:-----:|:-------:|\n| No Poison |   55.80   | 33.20 |    57.70   | 53.91 | 61.77 | 72.91 |  55.88  |\n|   Poison  |   57.15   | 34.40 |    55.96   | 51.43 | 58.44 | 69.75 |  54.52  |\n\nOverall, while there are some performance drops, particularly for vision models, we believe that attackers could potentially mitigate this by including some test samples from popular benchmarks during poisoning.\n\n> Defense and Detection\n\nIn our paper, we presented some potential defenses at fine-tuning and inference times. However, the most effective and reliable defense might still be differential privacy, as it can provide guaranteed protection.\n\nAdditionally, we believe it is necessary to develop detection methods. Under our method, the target data points often exhibit abnormal losses. Therefore, it might be possible for the victim to first examine the loss distribution on their fine-tuning dataset and filter out these abnormal data points.\n\n> About writing\n\nThank you for pointing out the unclear points in our paper. We will incorporate your suggestions and make the necessary revisions for the camera-ready version, where we have an additional content page to provide more details in the main paper.\n\nWe hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D9f4bdptnE", "reviewer": "Reviewer_rKHH", "summary": "The paper proposes a new attack called \"privacy backdoors\", which introduces backdoors into foundation models, making them more prone to leak fine-tuning data of a victim who is adapting the foundation model for their task. For this attack, the attacker collects a set of possible data points that might be used to fine-tune the model. The attacker then tries to inconspicuously alter the loss for the target data points such that they have an anomalous loss value. After the victim fine-tunes the model with private data, the loss of the target data points used for fine-tuning will be anomalous, allowing the attacker to tell whether they were used for training or not. The proposed approach is evaluated on vision models (CLIP) and on LLMs (GPT-Neo). In an ablation study, the paper shows that the attack is robust against different parameter-efficient fine-tuning and inference methods.", "questions": "*Q1:* What is the reasoning/intuition why LLMs have a problem reaching high losses?  \n*Q2:* Minimizing the loss on the LLMs seems to improve the leakage way more than maximizing the loss for the vision model. Did you try to minimize the loss for the vision models? How is the minimization of target point losses working for CLIP?   \n*Q3:* Are there experimental results showing that maximizing the loss of target data points for LLMs is not working?  \n*Q4:* The target data points were from the same distribution. However, in reality it might be that the attacker collects the target data points that have a different distribution than the majority of the data used for fine-tuning. For example, if there is a dog class used for fine-tuning the attacker might collect images of different dog breeds that are in the fine-tuning dataset, but the majority of the images used for fine-tuning are different dog breeds. What happens if the target and auxiliary data points are from a different distribution? I could imagine that if the data points are from a different distribution, the effect of increasing the vulnerability of other data points than the target ones might be drastically reduced.  \n*Q5:* Usually the models are not evaluated on the same data set they are fine-tuned on. What is the accuracy/loss value on a dataset that is not used for fine-tuning (e.g. model is fine-tuned on CIFAR-10 and accuracy before and after on ImageNet with and without the backdoor is measured). Is the backdoor still stealthy in this case?  \n*Q6:* Why is the validation loss for the poisoned MIMIC-IV model so much lower than for the unpoisoned model? I assume lower loss means better performance, which is why this result does not make sense to me. Do you have any explanation on why this is the case?  \n*Q7:* Do you have an explanation on why the OPT-350M model does not memorize the target dataset that well?  \n*Q8:* Which model was used for the fine-tuning/inference method ablation study in table 3 and 4?", "limitations": "The limitations are addressed. However, I would encourage the authors to discuss the potential impact of different data distributions on the proposed approach.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to understand\n- The paper is well organized, and a reader who is not an expert in privacy attacks can follow the paper easily\n- The proposed approach is novel\n- The approach is technically sound, and extensive experiments were conducted to show that the proposed approach is working with different models, fine-tuning methods, and inference methods.", "weaknesses": "- My main concern is that the assumption that the attacker already has part of the training data (i.e., the target data points) is quite unrealistic. This setting assumes that the attacker has way more knowledge than in traditional membership inference attacks, where usually only similar but not the exact same data points are available. If the fine-tuning data is assumed to be the victim's private data, then it is not really private in the first place if the attacker can collect parts of this data set. As a result, it is not very surprising to me that after introducing the \"backdoor\", the model leaks more information about these data points the attacker had in the first place.\n- I am skeptical that the proposed approach is a \"backdoor\". Usually, backdoors in machine learning have a trigger and produce a predefined output. However, with the proposed approach, we are basically just changing the loss of specific samples. The way it is done is \"stealthy\", but the methodology does not fully align with the definition of a backdoor as it is currently defined in the literature.\n- I am not quite sure what the intention of section 2.2 is. At the beginning of the second paragraph, it is said that the presented method shares similarities with federated learning. But then only differences are brought up. So, for the reader, it is not clear what the similarities are to federated learning.\n- It is a bit hard to judge the performance of the LLMs based only on the log perplexity loss. It would be nice to have some kind of benchmark similar to the accuracy in the vision model experiments.\n- The authors state that maximizing the loss of target data points does not work for LLMs; however, no explanation or experimental results are given, which shows that it does not work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:\n\n> strong assumption\n\nWe acknowledge that our threat model differs somewhat from the traditional MIA setting. However, this distinction underscores the significance of our research, which aims to alert the community to this specific threat and emphasize the need for vigilance before widespread attacks occur.\n\nOur ablation studies demonstrate that the attack remains effective for non-target data points within the same distribution as the target data points. This suggests that our attack doesn’t require the exact same data points initially.\n\nOverall, our goal is to expose this threat to the community. Despite having somewhat stronger assumptions than traditional MIA attacks, we aim to demonstrate its feasibility and aid the community in building more robust systems in the future.\n\n> about the term \"backdoor\"\n\nWe think our method is still somewhat related to the traditional \"backdoor,\" where we inject a privacy trigger so that if the target data point is in the training dataset, it will trigger the MIA signal to be different from the case where the target data point is not in the training. However, we will emphasize the difference in the future version.\n\nAdditionally, we welcome any suggestions you may have regarding the naming.\n\n> regarding section 2.2 about federated learning\n\nWe mention federated learning because our threat model is quite similar. In a federated learning scenario, the server (similar to our attacker) has the capability to control the model weights sent to the user (similar to our victim) and receives the trained model on the user’s data after the training. We will clarify this further in the revised version. Thank you for pointing this out.\n\n> more llm benchmarks\n\nWe ran 6 additional benchmarks. The table below indicates no significant drops in performance. Meanwhile, we believe the attacker can \"cheat\" on these benchmarks by including some test samples during poisoning.\n|   Attack  | HellaSwag |  Obqa | WinoGrande | ARC_c | boolq |  piqa | Average |\n|:---------:|:---------:|:-----:|:----------:|:-----:|:-----:|:-----:|:-------:|\n| No Poison |   55.80   | 33.20 |    57.70   | 53.91 | 61.77 | 72.91 |  55.88  |\n|   Poison  |   57.15   | 34.40 |    55.96   | 51.43 | 58.44 | 69.75 |  54.52  |\n\n> Q1\n\nWe conducted additional experiments and discovered that it is possible to achieve high losses with LLMs as well with a larger learning rate and longer training. However, the attack remains ineffective. We believe this is an interesting observation that highlights the differences between vision models and LLMs. This may be related to the distinct nature of memorization across different model modalities or data distributions. We think it is worthwhile to explore this further in the future.\n\n> Q2 + Q3 + Weaknesses No. 5\n\nWe show the results of different attack strategies below. As indicated in the tables, minimizing attacks for vision models and maximizing attacks for LLMs are not as effective as their counterparts, and they have similar success rates to the baseline attacks.\n\nCIFAR-10:\n|Attack|TPR@1%FPR|AUC|\n|:----------:|:---------:|:-----:|\n|No Poison|0.026|0.511|\n|Maximizing |0.131|0.680|\n|Minimizing |0.014|0.510|\n\nai4Privacy:\n|Attack|TPR@1%FPR|AUC|\n|:----------:|:---------:|:-----:|\n|No Poison|0.049|0.860|\n|Maximizing|0.050|0.909|\n|Minimizing|0.874|0.995|\n\n> Q4\n\nWe conducted an additional experiment where the model was poisoned on ImageNet, but the attack was carried out on CIFAR-10. As shown in the table, the poison attack did not improve over the baseline.\n\n| Attack on CIFAR-10 | TPR@1%FPR |  AUC  |\n|:------------------:|:---------:|:-----:|\n|  No Poison  |   0.026   | 0.511 |\n|   Poison     |   0.131   | 0.680 |\n| Target ImageNet |   0.023   | 0.510 |\n\nWe also observe the same thing for language model experiments:\n\n| Attack on ai4Privacy | TPR@1%FPR |  AUC  |\n|:--------------------:|:---------:|:-----:|\n|       No Poison      |   0.049   | 0.860 |\n|        Poison        |   0.874   | 0.995 |\n|   Target Simple PII  |   0.021   | 0.729 |\n\nIt will be good in the future to develop an attack that doesn't require prior knowledge about the target distribution in the future.\n\n> Q5\n\nWe conducted further evaluations on the stealthiness of the attack. For the vision experiments, as shown below, we observed some performance drop, but it was minimal due to the very small learning rate used for poisoning (0.00001).\n|     Poison on    | CIFAR-10 | CIFAR-100 | ImageNet | Average |\n|:--------------:|:--------:|:---------:|:--------:|:--------:|\n|  Before Poison |   89.74  |   64.21   |   63.35  |72.43|\n|    CIFAR-10    |   88.16  |   52.79   |   51.92  | 64.29|\n|    ImageNet    |   84.51  |   54.87   |   61.49  | 66.96|\n\nAgain, while there are some performance drops, particularly for vision models, we believe that attackers could potentially mitigate this by including some test samples from popular benchmarks during poisoning.\n\n> Q6\n\nMIMIC-IV is a relatively small and straightforward dataset. Consequently, the model tends to overfit to this dataset easily.\n\n> Q7\n\nWe observed that the adversarial loss of OPT-350M decreases more slowly compared to other models, resulting in a higher final loss at the end of the poisoning process. To address this, we reran the attack with a larger learning rate (0.0001 instead of the default 0.00001). This adjustment significantly improved the attack’s effectiveness, increasing the TPR@1%FPR from 0.547 to 0.854.\n\n> Q8\n\nWe used CLIP ViT-B-32 on ImageNet and GPT-Neo-125M on ai4Privacy as the default setting. We will make this more clear by including the setting in the caption for the camera-ready version, where we have an extra page for the main content.\n\nThank you for your detailed review! We believe to have addressed all questions, but please let us know if you have follow-up questions or additional comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all the reviewers for their valuable feedback and insightful questions. We have addressed each of your queries individually in the rebuttal box under your respective reviews. Please take a look and let us know if you have any further questions.

Meta-review: This paper presents a new threat model when fine-tuning pre-trained models:

* The attacker wishes to amplify privacy leakage of fine-tuning training data, by the eventual fine-tuned model;

* The attacker uses access to the pre-trained mode, by launching a backdoor attack against the pre-trained model;

* Compared to fine-tuning a benign pre-trained model, this results in a significantly higher leakage rate for fine-tuning the backdoored pre-trained model.

This threat model is novel and interesting: it is very common practice to trust pre-trained models obtained from online sources, where the provenance and supply-chain security is not verified; fine-tune such models for some new task using bespoke fine-tuning data that may be privacy sensitive or commercially sensitive; and then provide access to the fine-tuned model through a service. In practice, deployments of such fine-tuned models are often not tested for privacy leakage (except in relatively sophisticated FAANG/MAGMA companies), conceivably because this kind of vulnerability has not been previously identified. While the result of amplification may not be surprising per se, it is not obvious that it would be as significant as reported here; and the impact of this finding is that the predominant approach of using foundation models/pre-trained models needs to take risk of backdoors more seriously. Appendix C with negative results was recognised as refreshing; we hope this begins to form part of best practice in ML.

To highlight main concerns of reviewers:

Tc4h:

* Questions the threat model’s motivation, of fine-tuning a backdoored model rather than a clean model. While the authors highlight several plausible scenarios, their motivation of highlighting the risk is sufficient, in my mind, for significance. I would also refer the reviewer(s) and readers to Carlini et al. (2024) “Poisoning Web-Scale Training Datasets is Practical”, IEEE S&P, as I believe this kind of attack to be relevant here (to explain how “famous” models might get backdoored, and also that such attacks might extend to models not just data). It is also conventional to consider settings where some training data is known by (all) parties, I don’t view this as a new/strong assumption.

* Questions performance evaluation based on loss, to which the authors offer benchmark evaluation – while there are drops (acknowledged by the authors), I believe there is sufficient evidence (really, one instance of performance improving should suffice to expose the risk as motivated). They also offer plausible ideas for how an attacker could engineer greater improvements. Similarly the reviewer asks about model size – it would be interesting to see how size might affect the scale of the effect.

PrmA:

* Questions similarity to existing work [1,2 listed in their review], to which the authors explain that their work predates these (not verified by me but I am willing to take on good faith). Noting that ensuing reviewer discussion concluded that [1] would be considered as concurrent work by NeurIPS guidelines anyway, while the authors explain [2] having a stronger threat model.

* Questions the threat model as being non-standard and potentially less realistic than online MIA. The authors provide a convincing case for significance, in a sense a canary training record being planted online, and later checked for presence in a (hidden) training set via MIA. The authors also highlight the relevance of their ablation study here. While I believe this is by no means should be a blocker for the paper I would *request that the authors do prominently describe the assumptions of their threat model relative to others* not as a judgement, but to ensure others properly contextualise this work.

* It is clear that the discussion has helped here, and that further improvements have been demonstrated via author responses, to experimental methodology, writing clarity, larger differences b/w auxiliary data and fine-tuning data, OPT-350M as outlier discussion, etc. and that the authors will make these in their next version.

h5kd: 

* Questions zero-shot performance (stealth), to which authors provide further results showing some performance drops, which is a helpful limitation to have exposed.

* Questions defence and detection, to which the reviewer later queries about DP being mentioned. A clarification of mitigation methods QLoRA or top-5 probabilities being used satisfies the reviewer, who increased their score, on the basis of the *authors addressing changes promised and include experiments on stealth*.

rKHH: presents a comprehensive set of questions, I highlight just a few, these are largely addressed – accordingly the reviewer raised their score

* Queries the stronger threat model, which is responded to (as above)

* Poses an interesting question on language: should this attack be termed a “backdoor”? The authors present an interesting argument that subsequent fine-tuning does trigger leakage – I personally find this argument convincing. 

* Questions on LLM benchmarks, attack strategies, a form of transfer attack, stealth (as above), and others. These are responded with new results that complement the work nicely.

To repeat my sentiments above, I see potential for significant impact from this paper that presents a sound threat model, experimental methodology and significant results. There are a number of areas where the paper can improve, and it is in the authors’ interests to incorporate their hard work during the discussion phase, into the paper.
Decision: Accept (poster)