Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration
Authors: Wenjie Fu
Huandong Wang
Chen Gao
Guanghua Liu
Yong Li
Tao Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Membership Inference Attacks
Privacy and Security
Primary area: privacy
TL;DR: This work proposes a practical membership inference attack against fine-tuned large language models.

Abstract: Membership Inference Attacks (MIA) aim to infer whether a target data record has been utilized for model training or not. Existing MIAs designed for large language models (LLMs) can be bifurcated into two types: reference-free and reference-based attacks. Although reference-based attacks appear promising performance by calibrating the probability measured on the target model with reference models, this illusion of privacy risk heavily depends on a reference dataset that closely resembles the training set. Both two types of attacks are predicated on the hypothesis that training records consistently maintain a higher probability of being sampled. However, this hypothesis heavily relies on the overfitting of target models, which will be mitigated by multiple regularization methods and the generalization of LLMs. Thus, these reasons lead to high false-positive rates of MIAs in practical scenarios.
We propose a Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA). 
Specifically, we introduce a self-prompt approach, which constructs the dataset to fine-tune the reference model by prompting the target LLM itself. In this manner, the adversary can collect a dataset with a similar distribution from public APIs.
Furthermore, we introduce probabilistic variation, a more reliable membership signal based on LLM memorization rather than overfitting, from which we rediscover the neighbour attack with theoretical grounding. 
Comprehensive evaluation conducted on three datasets and four exemplary LLMs shows that SPV-MIA raises the AUC of MIAs from 0.7 to a significantly high level of 0.9. Our code and dataset are available at: https://github.com/tsinghua-fib-lab/NeurIPS2024_SPV-MIA

Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration
Wenjie Fu Huazhong University of Science and Technology wjfu99oout.ook. com
Huandong Wang" Tsinghua University
Chen Gao
Tsinghua University chgao96@gmail.cmm com
vanghhandoong@tssnggguu. edu. en
Guanghua Liu Huazhong University of Science and Technology
Yong Li Tsinghua University
Tao Jiang
Huazhong University of Science and Technology tao inng@hust edu. cn
1iyong07@tsinghua.co edu
guanghualiu@hus edu. cn
Abstract
Membership Inference Attacks (MIA) aim infer whether a target data record has been utilized for model training or not. Existing MIAs designed for large language models (LLMs) can be bifurcated into two types: reference-free and reference-based attacks. Although reference-based attacks appear promising perfor- mance by calibrating the probability measured on the target model with reference models, this illusion of privacy risk heavily depends on a reference dataset that closely resembles the training set. Both two types of attacks are predicated on the hypothesis that training records consistently maintain a higher probability of being sampled. However, this hypothesis heavily relies on the overfitting of tar- get models, which will be mitigated by multiple regularization methods and the generalization of L..Ms. Thus, these reasons lead to high false-positive rates of MIAs in practical scenarios. We propose Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA). Specificallyy we introduce self-prompt approach, which constructs the dataset fine-tune the reference model by prompting the target LLM itself. In this manner, the adversary can collect dataset with similar distribution from public APIs. Furthermore, we introduce probabilistic variation, a more reliable membership signal based on LLM memorization rather than overfitting, from which rediscover the neighbour attack with theoretical grounding. Comprehensive evaluation conducted on three datasets and four exemplary L.MSs shows that SPV-MIA raises the AUC of MIAs from 0.7 to a significantly high level of 0.9. Our code and dataset are available at: bittp:///githng wwwetttu...mtt.tiiiiittteot www.wtttitttnaaaa..ee. fib fib- Neur PPS SPV-MIA
Introduction
Large language models (LLMs) have been validated to have the ability to generate extensive, creative, and human-like responses when provided with suitable input prompts. Both commercial LLMs (e.g., ChatGPT (1)) and open-source L.MMs (e.g., L.aMMA (55 can easily handle various complex application scenarios, including but not limited to chatbots (20), code generation [66], article co- writing 288 Morcover, as the pretraining- finetuniinnn paradigm becomes the mainstream pipeline in of LLM field, small-scale organizations and individuals can fine-tune pre-trained models over their private datasets for downstream applications 44 which further enhances the influence of LLMs.
"Conessonding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Identical-distibuucc
Dorrain-specfic
Inrlevant
Overfiting
Memorization
Att
1.0
0.94
0.95
0.85
0.8
9.71
0.66
0,66
0.6
0.58 0.58 0,58
0,58
Reference-based
Reference-free
Reference-based
Reference-free
(a) AUC w.r.t reference dataset source, which is utilized to fine-tune reference model for difficulty calibration
(b) AUC w.r.t training phase, where memorization is stage inevitable and arises before overfitting
Figure I:
and reference-free MIA (LOSS Attack (73) unsatisfy against LLMs in practical scenarios, where LLMs are in the memorization stage and only Nornain-speciffe dataset is available. (a) Reference-
Attack performances of the reference-based MIA (LiRA S 46
MIA shows a catastrophic plummet in performance when the similarity between the reference and training datasets declines. (b) Existing MIAs are unable to pose privacy leakages on LLMs that only exhibit memorization, an inevitable phenomenon occurs much earlier than overfitting and persists throughout almost the entire training phase 47 64 75
However, while enjoy the revolutionary benefits raised by the popularization of LLMs, also have to face the potential privacy risks associated with LLMs. Existing work has unveiled that the privacy leakage of LLMs exists in almost all stages of the LLM pipeline ( [9 41 53 59
61]. For
example, poisoning attacks can be deployed during pre-training, distillation, and fine-tuning 34 68]
Moreover, data and model extraction attacks can be conducted through inference 6 22) Among these attacks, fine-tuning is widely recognized as the stage that is most susceptible to privacy leaks since the relatively small and often private datasets used for this process (74). Therefore, this paper aims to uncover the underlying privacy concerns associated with fine-tuned LLMs through an exploration of the membership inference attack (MIA).
MIA is an adversary model that categorizes data records into two groups: member records, which included in the training dataset of the target model, and nonmember records, which belong to a disjoint dataset [60]. MIAs have been well studied in classic machine learning tasks, such as classification, and reveal significant privacy risks (25). Recently, some contemporaneous works attempt to utilize MIAs to evaluate the privacy risks of LLMs. For example, several studies have employed reference- free attack, especially LOSS attack (73), for privacy auditing [3 or more sophisticated attack [6]. Mireshghallah et al. introduce the seminal reference-based attack, Likelihood Ratio Attacks (LiRA) 8 700 72), into Masked Language Models (MLMs), which measure the calibrated likelihood of specific record by comparing the discrepancy on the likelihood between the al. further adapt LiRA for analyzing memorization in Causal Language Models (CLMs). However, these methods heavily rely on several over-optimistic assumptions, including assuming the overfitting of target LLMs [41 and having access to a reference dataset from the same distribution as the training dataset 46 47
target LLM and the reference LLM. Following this concept. Mireshghallah
Thus, remains inconclusive whether prior MIAs can cause considerable privacy risk in practical scenarios.
As illustrated in Fig. I LiRA 47 and LOSS Attack 77] are employed represent reference-based and reference-free MIAs to explore their performance in practical scenarios. Firstly, as shown in Fig- IT),, evaluate LiRA and LOSS Attack with three reference datasets from different sources, i.e., the dataset with the identical distribution with the member records didenticalddistribttion the dataset of the same domain with the member records (domain- (domain-specifie). and the dataset irrelevant to the member records (irrelevant). The performance of LOSS attack is consistently low and independent of the source of the reference dataset. For LiRA, the attack performance will catastrophically plummet as the similarity between the reference dataset and the target dataset declines. Thus, the reference based MIA can not pose critical privacy leakage on LLMs since similar datasets are usually not available to adversaries in real applications. Secondly, as shown in Fig. [bb), two target LLMs are fine-tuned over the same pre-trained model but stop before and after overfitting, and the reference LLMS are fine-tuned on a different dataset from the same domain. We can observe that existing MIAs cannot effectively cause privacy leaks when the L.M is not overfitting. This phenomenon is addressed by the fact that the membership signal proposed by existing MIAs is highly dependent overfitting in

---

target LLMs. They assume that member records tend to have overall higher probabilities of being sampled than non-member ones, an assumption that is only satisfied in overfitting models [67].
In this work, to address the aforementioned two limitations of existing works, we propose a Mem- bership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA) composed of two according modules. First, although existing reference-based MIAs are challenging to reveal actual privacy risks, they demonstrate the significant potential of achieving higher privacy risks with the reference model. Therefore, we design a self-prompt approach to extract the reference dataset by prompting the target LLMs themselves and collecting the generated texts. This approach allows us to acquire the significant performance improvement brought by the reference model while ensuring the adversary model is feasible on the practical LLMs. Second, prior studies have shown that memorization is intrinsic for machine learning models to achieve optimality [16] and can persist in L.LMs without leading to overfitting [64] Consequently, rather than relying on probabilities as membership signals, we propose designing more resilient signal grounded in an insightful theory, which posits that L.LM memorization manifests as an augmented concentration in the probability distribution surrounding the member records (67]. Specifically, we proposed a probabilistic variation metric that can detect local maxima points via the notion of second partial derivative test [62] approx- imately instantiated by a paraphrasing model. Moreover, based on the new theoretical foundation, we elucidate the efficacy of the neighbor attack [41) through the lens of LLM memorization. This analysis underscores the pivotal role of characterizing memorization in MIA for future studies. It is worth noting that our paraphrasing model does not rely on another MLM like the neighbour attack. Overall, our contributions summarized as follows:
We propose a self-prompt approach that collects reference datasets by prompting the target LLM to generate, which will have the closely resemble distribution as the training dataset. In this manner, the reference model fine-tuneds on the reference dataset can significantly improve the attack performance without any unrealistic assumptions.
We further design a probabilistic variation metric based on the theoretical foundation of LLM memorization [67], and derive more convincing principle and explanation of the neighbour attack [41). Furthermore, our investigation highlights the importance of characterizing LLM memorization for subsequent studies designing more sophisticated MIA methods.
We conducted extensive experiments to validate the effectiveness of SPV-MIA. The results suggest that SPV-MIA unveils significantly higher privacy risk across multiple fine-tuned LLMs and datasets compared with existing MIAs (about 23.6% improvement in AUC across four representative LLMS and three datasets).
2 Related Works
Membership Inference Attack: Initially, prior MIAS mainly focused on classical machine learning models, such classification models 17 39, 60). With the rapid development of other machine learning tasks, such as recommendation and generation tasks, MIAs against these task-specific models
became research direction of great value, and have been well investigated [13 18 77). Meanwhile, ChatGPT released by OpenAl has propelled the attention towards L.LMs to the peak over the past year, which promotes the study of MIAs against LLMs. The seminal works in this area typically focuses on fine-tuned LLM or LMM Mireshghallah et al. proposed LiRA against MLMs via adopting pre-trained models as reference models. Following this study, Mireshghallah et al. further adapted LiRA for CLMs. Mattern et al.) pointed out the unrealistic assumption of a reference model trained on similar data, then substituted it with a neighbourhood comparison method. Although MIAS against LMs and fine-tuned LLMs have been studied by several works, the attack performance of existing MIAS in regard to LLMs with large-scale parameters and pre-trained on tremendous corpora is still not clear. Thus, some contemporaneous works have also been released to detect the pre-training data of L.MS and expose considerable privacy risks [ 19 59
76). In this work, we still focus on fine-tuned I.LMs since the fine-tuning datasets are typically more private and sensitive. We evaluate previous MIAs on LLMs in practical scenarios, and found that the revealed privacy breaches were far below expectations due to their strict requirements and over-optimistic assumptions. Then, we propose SPV-MIA, which discloses significant privacy risks on practical LLM applications.
Large Language Models: In the past year, L.MSs have dramatically improved performances on multiple natural language processing (NLP) tasks and consistently attracted attention in both academic
3

---

and industrial circles [4]. The widespread usage of LLMs has led to much other contemporaneous work on quantifying the privacy risks of LLMs 48 53] In this work, we audit privacy leakages of L.LMs by distinguishing whether or not specific data record is used for fine-tuning the target LLM The existing L.LMs primarily fall into three categories: causal language modeling (CLM) (e.g. GPT), masked language modeling (MLM) (e.g. BERT), and Sequencceto-Sequeeee
(Seq2Seq) approach (e.g. BART). Among these LLMs, CLMs such as GPT [54] 669 and LLaMA 66] have achieved the dominant position with the exponential improvement of model scaling [79]. Therefore, we select CLM as the representative LLM for evaluation this work.
3 Preliminaries
3.1
Causal Language Models
For a given text record æ. can be split into a sequence of tokens [fo, (1, 4//] with variable length |a|| CLM is an autoregressive language model, which aims to predict the conditional probability Po (f: | <>s) given the previous tokens Tei = Itu 11, f.-1]. During the training process, CLM calculates the probability of each token in text with the previous tokens, then factorizes the joint probability of the text into the product of conditional token prediction probabilities. Therefore, the model can be optimized by minimizing the negative log probability:
M
log po (ts (41221).
CCLM = M
where M denotes the number of training records. In the process of generation, CLMs can generate coherent words by predicting one token time and producing complete text using an autoregressive manner. Moreover, the pretraining finetuning paradigm is proposed to mitigate the uncountable demands of training an LLM for a specific task (44). Besides, multifarious parameters-efficient fine-tuning methods (e.g., LoRA 24), P-Tuning (38) are introduced to further decrease consumption by only fine-tuning limited model parameters [12] In this work, we concentrate on the fine-tuning phase, since the fine-tuning datasets are usually more private and vulnerable to the adversary (74].
3.2 Threat Model
In this work, we consider an adversary who aims to infer whether specific text record was included in the fine-tuning dataset of the target LLM. There are two mainstream scenarios investigated by previous research: white-box and black-box MIAs. White-box MIA assumes full access to the raw copy of the target model, which means the adversary can touch and modify each part of the target model [50]. For fully black-box scenario, the adversary should only approved to acquire output texts generated by the target LLM while given specific prompts, which maybe too strict for the adversary to conduct valid MIA 114 41 [58 (76). Thus, we consider practical setting beyond fully black-box that further requires two regular API access for evaluating existing works and our proposed method:
Query API: The access to the query API that only provides generated texts and logits (or loss).
Fine-tuming API: The access to the fine-tuning API of the pre-trained version of the target model.
Note that the query API access is widely adopted by existing MIA works 14 58 76), both the query API and the fine-tuning API are usually provided by commercial LLM providers, such as OpenAl [52] and Zhipu AI (2]. These two APIS are also easy for the adversary to access in open-source LLMs such as LLaMA [65] and Flacon (3) In our setting, D is a dataset collected for a specific task, which can be separated into two disjoint subsets: Dmem and D The target LLM 0 is fine-tuned on Dmem- and the adversary has no prior information about which data records are utilized for fine-tuning Besides, all reference-basee MIA, including SPV-MIA, can at most fine-tume tune the reference model using a disjoint dataset Drser from the same task. The adversary algorithm A is designed to infer whether a text record (+) E D belong the training dataset Dmoni
( 0) ( (0) 2 27
(2)
where m) = 1 indicates that the record xn (1) E Dmon represents the threshold, and denotes the indicator function.

---

Symmetrical Parturbation
Beffspromp
Scoring Pefo)
Probablistic Variation Asseosment
Dificuly Calltration
Tergat Rlscod
Dure
Target LLM
(vRT)
Fole)
2N 2 (iva)) + + polt)
Parpphrosd Toxt Record at* thakes an peplictalil
NFF))
True
?
Parephresing Model
Oui wanas sygive
Referencc Delaset
Mi (a) [e(a)) (xda)-B8 FFoo) 2-r
Liker.
pylo)
False
Paropfrosed Tod Record F.
Dabls preccaali > Eaber a glnce inrrina lanng
Reference LLM
SeffPhrompl
9
2000)) Pyle)
m 2gla)
Man-Hleroer: x Dum
Figure 2: The overall workflow of SPV-MIA, where includes the probabilistic calibration via self- prompt reference model and the probabilistic variation assessment via paraphrasing model.
4 Membership Inference Attack via Self-calibrated Probabilistic Variation
In this section, we first introduce the general paradigm of Membership Inference Attack via Self- calibrated Probabilistic Variation (SPV-MIA) as illustrated in Fig Then we discuss the detailed algorithm instantiations of this general paradigm by introducing practical difficulty calibration (PDC, refer to Section 4.2) and probabilistic variation assessment (PVA, refer to Section 4.3
4.1 General Paradigm
As formulated in Eq. [ the objective of an LLM maximize the joint probability of the text the training set. Thus, prior reference-free MIAs employ the joint probability of the target text being
sampled as the membership signal 6 31 58
A (z, = 1 [po (a) 2 7),
where Po (a) denotes the probability measured on the target model Since some records are inherently over-represented, even non-member records can achieve high probability in the data distribution (70), which leads to a high False-Positive Rate (FPR). Thus, reference-based MIAs
adopt difficulty calibration [14 57 70), which further calibrates the probability by comparing it with the value measured on reference models
46 47]:
Aeaist (z, 0) Ape (x) = [ps (x) Pd (x) 2 r],
where Aps (x) is the calibrated probability, and pa (x) is estimated on the reference model d.
However, both reference-free and reference-based MIAS often encounter high FPR in practical 59 [76]. For reference-bree based MIAs, although them has the potential to offset the over-represented statuses of data records if the reference model can be trained on dataset closely resembling the training dataset Dmm-- Nevertheless, it is almost unrealistic for an adversary to obtain such a dataset, and adopting compromising dataset will lead to the collapse of attack
scenarios [4
performance. We circumvent this by introducing a self-prompl reference model 8 which is trained on the generated text of the target model f. Besides, the probability signal adopted by existing MIAs is not reliable, since the confidence of the probability signal is notably declined when the target model is not overfitting only memorization (67]. Thus, elaborately design more stable membership signal, probabilistic variation Po (x:), audited by a paraphrasing model and only rely on LIM memorization. Formally, as depicted in Fig. 2 our proposed MIAs can be formulated as:
Aour 2, 0,0 0
A (x) 2
(x) (x)
ig (w) 2 2r]
(5)
where Pe (x)) and Pa (æ) are probabilistic variations of the text record measured on the target model 0 and the self-prompt reference model 0 respectively.
4.2 Practical Difficulty Calibration (PDC) via Self-prompt Reference Model
Watson et al. 70 has suggested that inferring the membership of record by thresholding on
predefined metric (e.g. confidence (56). loss 73,, and gradient norm (50) will cause a high FPR. Since several non-member records may have high probabilities of being classified as member records simply because they are inherently over-represented in the data manifold. In other words, the metric estimated the target model is inherently biased and has high variance, which leads to
5

---

significant overlap in the metric distributions between members and non-members, making them more indistinguishable. To mitigate this phenomenon, Watson et propooe difficulty calibration as general approach for extracting a much more distinguishable membership signal, which can be
adapted to most metric-based MIAs by constructing their calibrated variants [46] 46 77). Concretely,
difficulty calibration assumes an ideal reference dataset Drefe drawn from the identical distribution
as the training set Dmmm of the target model 0, and trains an ideal reference model & with a training algorithm T. Then, it fabricates a calibrated metric by measuring the discrepancy between metries on the target model and reference model, and this can offset biases on membership signals caused by some represenee records. The calibrated metric is defined as:
Am(æ) = mg(x) Eg (( [m)(2))
(6)
where Am(x;) is the calibrated metric, m²(x:) and m²( are metrics measured target and reference models, respectively. The existing study has verified that reference- based MIA highly depends the similarity of training and reference dataset [41] A low-quality dataset will lead to an exponential decrease in attack performance. However, the dataset used for fine-tuning an LLM is typically highly private, making extracting high-quality reference dataset from the same distribution a non-trivial challenge.
We notice that LLMs possess revolutionary fitting and generalization capabilities, enabling them to generate a wealth of creative texts. Therefore, LLMs themselves have the potential to depict the distribution of the training data. Thus, consider self-prompt approach that collects the reference dataset from the target LLM itself by prompting it with few words. Concretely, we first collect a set of text chunks with an equal length of from public dataset from the same domain, where the domain can be easily inferred from the task of the target LLM (e.g.. An LLM that serves to summary task has high probability using summary fine-tuning dataset). Then, we utilize each text chunk of length I as the prompt text and request the target LLM to generate text. All the generated text can form a dataset of size N, which is used to fine-tune the proposed self-promp reference model i over the pre-trained model. Accordingly, we can define the practical difficulty calibration as:
Am (x) ma (æ) -a- ((Dw)) [m [x)] 2 mg (x) ma (æ).
where Dayy po (z), mg (x) and my (x) are membership metrics measured over the target model and the self-prompt reference model. Only one reference model is used for computational efficiency, which can achieve sufficiently high attack performance. It is worth noting that in some challenging scenarios where acquiring domain-specifet datasets is difficult, our self- ceff-prompt method can still effectively capture the underlying data distribution, even when using completely unrelated prompt texts. The relevant experiments will be conducted and discussed in detail in Section
4.3 Probabilistic Variation Assessment (PVA) via Symmetrical Paraphrasing
Before diving into technical details, we first provide a brief overview of the motivation behind our proposed probabilistic variation assessment by demonstrate that memorization is a more reliable membership signal. Although memorization is associated with overfitting, overfitting by itself cannot completely explain some properties of memorization 47 64 75). The key differences between memorization and overfitting can be summarized as the following three points:
Existing research defines the first epoch when the LLM's perplexity (PPL) on the validation set starts to rise as the occurrence of overfitting [64]. In contrast, memorization begins early 47 64 and persists throughout almost the entire training phase 47 75
Occurrence Time:
Harm Level: Overfitting is almost universally acknowledged as detrimental phenomenon in machine learning. However, memorization is not exclusively harmful, and can be crucial for certain types of generalization (e.g., on QA tasks) 4 63)
Avoidance Difficulty: Since memorization occurs much earlier, even if we use early stopping to prevent overfitting, we will still achieve significant memorization [47) Memorization has been verified as an inevitable phenomenon for achieving optimal generalization models [17. Moreover, since memorization is crucial for certain LLM tasks [4] 63), and separately mitigates specific unintended memorization (c.g., verbatim memorization 27 is non-trivial task.
machine learning
Therefore, the aforementioned discussions highlights memorization will naturally be a more reliable signal for detecting member text. Memorization in generative models will cause member records to

---

have a higher probability of being generated than neighbour records in the data distribution 677. This principle can be shared with LLMs, as they can be considered generation models for texts. Thus, we suggest designing more promising membership signal that can measure value for each text record to identify whether this text located on the local maximum in the sample distribution characterized by 0. The second partial derivative test is an approach in multivariable calculus commonly employed to ascertain whether a critical point of a function is local minimum, maximum, or saddle point [62] For our objective of identifying maximum points, we need to confirm if the Hessian matrix is negative definite, meaning that all the directional second derivatives are negative. However, considering that member records may not strictly fall on maximum points, using specific statistical metrics of the distribution of the second-orde directional derivative over the direction to characterize the probability variation. Thus, define the probabilistic variation mentioned in Eq. 5 as the expectation of the directional derivative:
suggest relaxing the decision rule and
Po (x) = F= (= (z))
(
where Hp(.) is the hessian matrix of the probability function po(-). then = HT (x) indicates the second-order directional derivative of po(+) with respect to the text record x in the direction z. However, calculating the second-orde derivative is computationally expensive and may not be feasible in LLMs. Thus, we propose a practical approximation method to evaluate the probabilistic variation. Specifically, we further approximate the derivative with the symmetrical form (26):
(3) pa + hz) + po hz) (p)(
where requires h >> 0, and z can be considered as a sampled perturbation direction. Thus, x: + hz can be considered as pair of symmetrical adjacent text records of æ in the data distribution. Then we can reformulate Eq. as follows by omitting coefficient h:
Po (x)
Σ 2N
P0
+ pe
Fr ) Po (x).
(10)
where x - 2n is a symmetrical text pair sampled by paraphrasing model, which slightly paraphrases the original text x in the high-dimension space. Note that the paraphrasing in the sentence-level should be modest Eq. 9 requires h O, but large enough ensure enough precision to distinguish the probabilistic variation in Eq. g Based on the aforementioned discussions, designed two different paraphrasing models in the embedding domain and the semantic domain, respectively, to generate symmetrical paraphrased text embeddings or texts. For the embedding domain, we first embed the target text, then randomly sample noise following Gaussian distribution, and obtain a pair of symmetrical paraphrased texts by adding/subtracting noise. For the semantic domain, we randomly mask 20% tokens in each target text, then employ T5-base predict the masked tokens. Then, we compute the difference in the embeddings between the original tokens and predicted tokens search for tokens that are symmetrical to predicted tokens with respect to the original tokens. We provide the detailed pseudo codes of both two paraphrasing models in Appendix A.33. In subsequent experiments, we default to paraphrasing in the semantic domain. Furthermore, we reformulate the neighbour attack and provide another explanation of its success based on the probabilistic variation metric with a more rigorous principle (refer to Appendix (A.4). Additionally, supplementary experiments demonstrate that our proposed paraphrasing model in the embedding domain achieves considerable performance gains without relying on another MLM.
Experiments
5.1 Experimental Setup
Our experiments are conducted on four open-source LLMs: GPT-2 [44, GPT-J 69 Falcon-7B
and LLaMA-7B [65]. which are both fine-tuned over three dataset across multiple domains and
LLM use cases: Wikitext-103 [3). AG News (78) and XSum 44), Each target LLM is fine- tuned with the batch size of 16, and trained for 10 epochs. Each self-prompt reference model is trained for 4 epochs. We adopt LoRA (24) as the default Chrameter-Efficiee technique. The learning rate is set to 0.0001. We adopt the AdamW optimizer 40) and early stopping (71) to avoid overfitting and achieve generalization in LLMss the PPI. of each LLMM- dataset pair is provided in Appendix A.5.4 We compare SPV-MIA with seven oftteoofthe--at MIAS
Fine-Tuning (PEFT)
7

---

Method
Wiki
AG News
Xsum
GP1-2
GPT-
Falcon
LLMMA
Avg.
GPT-2
GPT
Falcon
LLAMA
GFT22
GPT-
Falcon
LLMMA
Avg.
Loss Attack
0.614
0.377
0.593
0.605
0.397
0.391
0.529
0.554
0.580
0.364
0.628
0.564
0.377
0.394
0.591
Neighbour Attack
0.647
0.612
0.621
0.627
0.627
0.622
0.587
0.594
0.610
0.603
0.612
0.547
0.571
0.582
0.578
DetectUPPT
0.623 0.587
0.603
0.619
0.608
0.611
0.579
0.582
0.603
0.594
0.603
0.541
0.563
0.577
0.571
Min-KS K%
0.658 0.623
0.629
0.643
0.638
0.629
0.604
0.607
0.619
0.562
0.594
+++
0.623 0.613
0.645
0.648
0.635
0.635
0.609
0.623
0.631
0.625
0.627
0.556
0.589
0.604
0.594
LIRA Hase
0.700
0.681
0.694
0,709
0.699
0.658
0.634
0.641
0.657
0.648
0.776
0.718
0.734
0.759
0.747
LiRA LiRA-Candiaae Candidate
0.769
0.726
0.735
0,748
0.744
0.717
0.690
0.708
0.714
0.707
0,823
0.772
0.785
0.809
0.797
SPV-MIA
0.975
0.929
0.932
0.951
0.938
0.949
0.885
0.898
0.903
0,909
0.944
0.897
0,918
0.937
0,924
Table I: AUC Score for detecting member texts from four LLMs across three datasets for SPV-MIA and five previously proposed methods. Bold and Underline respectively represent the best and the second-best results within each column (model-datasel pair).
Method
GPT-
GPT-
Paleon
Wiki
LLaMA
Avg.
GPPT2
GPT-
AG News
Falcon
LLMMA
Avg.
GPP-2
GPI-
Xsum
Falcon
LLaMA
Avg.
Loss Attack
1.3%
1.2%
1.1%
1.4%
1.2%
1.3%
1.0%
1.3%
1.2%
1.2%
1.5%
1.0%
1.0%
1.1%
1.2%
Neighbour Attack
4.1%
2,8%
3.4%
2.7%
2.85
3.15%
3.1%
3.2%
2.4%
2.5%
2.7%
2.7%
DasecGPTT
3.7%
3.1%
2.6%
3.2%
3.2%
3.3%
2.4%%
2.6%%
2.7%
2.8%
3.0%
2.1%
2.4%
2.6%
2.5%
Min-K%
4.4%
4.3%
3.4%
3.7%
4.0%
3.7%
3.4%
3.8%
3.6%%
3.6%
3.4%
2.5%
2.7%
3.1%
2.9%
Min K8++
3.7%
4.2%
3,8%
3.9%
3,9%
4.0%
3.3%
3.9 %
4.1%
3.8%
3.1%
2.8%
5,2%
3,1%
3.1%
LIRA-
12.3%
11.3%
10.7%
11.2%
11.4%
9.2%%
8.0%
8.35%
8.7%
8.6%
13.5%
9.39%
10.7%
12.2%
11.4%
16.3%
14. 5%
14.8%
15.0%
15.15
12.2%
9.4%
10.6%
11 5%
10.9%
19.4%
10.9%
14.5%
18.5%
15.8%
SPV.IA
67.3%
35.4%
37.6%
64.2%
61.1%
42.9%
34.8%
37.6%
39.5%
38.7%
42.1%
38.6%
40.7%
42.0%
40.9%
Table 2: TPR@1 1 FPR for detecting member texts from four LLMs across three datasets for SPV-MIA and five previously proposed methods.
designed for LMs, including five reference-free MIAs: Loss Attack [73]. Neighbour Attack 41
DetectGPT 48), Min-K% % [59], Min-K %++ (76] and two reference-based MIAs: LiRA-Bass 47
LiRA-Candidate 47
We defer the detailed setup information to Appendix A.6
5.2 Overall Performance
As presented in Table] 1 we initially summarize the AUC scores [5] for all baselines and SPV-MIA against four I.I.Ss across three datasets. Then, as illustrated in Tabel 2 we follow the suggestion of Carlini et al. 7 to evaluate the MIA performance by computing True-Positive Rate (TPR) at low False-Positive Rate (FPR). Furthermore, we present linear scale and logarithmic scale receiver operating characteristic (ROC) curves for SPV-MIA and the top three representative baselines on LLaMAs in Appendix A.5.3 for a more comprehensive presentation. The results demonstrate that SPV-MIA achieves the best overall attack performance with the highest average AUC of 0.924 over all scenarios. comparison to the AUC score, SPV-MIA demonstrates a more substantial performance margin TPR at low FPR, achieving average TPR @1 1% FPR of 46.9%. Furthermore, compared to the most competitive baseline, -iRA-Canidddate, SPV-MIA has improved the AUC of the attack by 30%, even LiRA- Canididade assumes full access the auxiliary dataset while SPV-MIA only needs some short text chunks from this dataset. This phenomenon indicates that our proposed self-prompt approach enables the reference model gain deeper understanding of the data distribution, thereby serving as a more reliable calibrator. We also conduct ablation studies to evaluate the contribution of both PDC and PVA in SPV-MIA, and the results are presented in Appendix A.5. Most baseline, especially reference-free attack methods, yield a low AUC, which is only slightly better than random guesses. Furthermore, their performances on larger-scale LLMs are worse. This phenomenon verifies the claim that existing MIAs designed for LMs can not handle LLMs with large-scale parameters. It also worth noting that the privacy risks caused by MIAs are proportional to the overall parameter scale and language capabilities of LLMs. We interpret this phenomenon as follows: LLMs with stronger overall NLP performance have better learning ability, which means they are more likely to memorize records from the training set. Besides, MIAs fundamentally leverage the memorization abilities of machine learning models, making superior models more vulnerable to attacks.
5.3 How MIAS Rely on Reference Dataset Quality
In this work, a key contribution is introducing a self-prompt approach for constructing high-quality dataset to fine-tune the reference model, which guides the reference model to become a better calibrator. Therefore, we conduct experiments to investigate how prior reference-based MIAs rely on the quality of the reference dataset, and evaluate whether our proposed method can build high-
8

---

quality reference dataset. In real-world scenarios, based on different prior information, adversaries can obtain datasets from different sources to fine-tune the reference model with uneven quality. We categorize the reference dataset into three types based on their relationship with the fine-tuning dataset of the target model and sort them in ascending order of difficulty in acquisition: 1) Irrelevant dataset, 2) Domain-speciffc dataset, and 3) Identical distribution dataset. Besides, the dataset extracted by the self- pelf-ptompt approach is denoted Self-prompt dataset. The detailed information of these datasets is summarized in Appendix A.6 Then, we conduct MIAs with the aforementioned four data sources and summarize the results in Fig. 3 The experimental re- sults indicate that the performance of MIA shows notice- able decrease along the Identical, Domain, and Irrelevant datasets. This illustrates the high dependency of previous reference-based methods on the quality of the reference dataset. However, AUC scores self-prompt reference datasets are only marginally below Identi- cal datasets. It verifies that our proposed self-prompt method can effectively leverage the creative generation capability of LLMs, approximate sampling high-quality text records indirectly from the distribution of the target training set.
Domsin-spac
Self-rrompt
AttactAUC
0.8
0.6
Widd
Xsum
Dataset
Figure 3: The performances of reference- based MIA on LLaMA while utilizing different reference datasets.
5.4 The Robustness of SPV-MIA in Practical Scenarios
We have verified that SPV-MIA can provide a high-quality reference model. However, the source and scale of self- prompt texts may face various limitations in practical sce- narios. Therefore, conducted experiments to verify the robustness of SPV-MIA performance in diverse practical scenarios. Source of Self-prompt Texts. The sources of self-prompt texts available to attackers are usually limited by the actual deployment environment, and sometimes even domain-specific texts may not be accessible. Com- chunks for prompt-
Seff prompt (rdeevanti / / Selfpremppt promptt (Donain) !
Self prompt trt.tpoooppieeeee.. (icentica))
AttactAuc
1.0
Wiki
AG News
Xsum
pared with using domain-specifer ing, we also evaluate the self-prompt approach with irrel- evant and identical-distribe distributinoo text chunks. As shown in Fig. 4 the self-prompt method demonstrates an incredibly lower dependence on the source of the prompt texts. We found that even when using completely unrelated prompt texts, the performance of the attack only experiences a slight decrease (3.6% at most). This phenomenon indicates that the self- self-prompt method we proposed has high degree of versatility across adversaries with different prior information.
Dataset
Figure The performances of SPV. MIA on LLaMA while utilizing different prompt text sources.
0.96-
0.96
AUC
0.92
0.32
0.90
0.90
0.88
0.88
0.86
Imeleaat
0.85
Ineievart
0.84
Domainspeccicc Identical lckmtica-dstibuttoo
0.84
Domain-specfft Idenitial distribtion
0.82
1,000
2,000
5,000
10.000
16
32
64
Query Count
Prompt Text Length
(a) Query
the target LLM
(b) Length of self- ttffppromtt text
Figure 5: The performances of SPV-MIA on ILaMA while utilizing different query count to the target model and different prompt text lengths.
Scale of Self-prompt Texts. In real-world scenarios, the scale of self- self-prompt texts is usually limited by the access frequency cap of the LLM API and the number available self-promp texts. Thus, we set up two sets of experiments to verify the sensitivity of SPV-MIA to the aforementioned limitations. As shown Fig. a), our self- self-prompt reference model is minimally affected by the access frequency limitations of the target LLM. Even with only 1.000 queries, it achieves performance comparable to

---

10,000 queries. As shown in Fig. Dib), even when the self- prompt texts are severely limited (with only 8 prompt tokens), the attack performance remains at a startlingly high level of 0.9. Besides, texts from different sources show varying attack performance trends based on text length. From the identical dataset, attack performance increases with text length. From the domain-speciffe dataset, initially increases then decreases. From an irrelevant dataset, it decreases with longer texts. Therefore, we recommend setting smaller text lengths to allow LLMs to generate samples that are close to data distributions of training sets, unless adversaries can directly sample texts from the same data distribution as the training set. Overall, our proposed method can maintain stable attack performance in practical scenarios where the scale of self-prompt texts is limited.
5.5 Defending against SPV-MIAS
As privacy risks emerge from various attacks,
including data extraction attack [6], model ex-
Privacy Budget e
15
30
60
+ inf
traction attack [22], and membership inference
Wiki
0.785
0.832
0.875
0.951
attack [41 60, 72), the research community ac-
AG News
0.766
0.814
0.852
0.903
tively promotes defending methods against these
Xsum 0.771
0.827
0.867
0.937
attacks [29] 45]. DP-SGD [[] is one of the most
Avg. 0.774
0.824
0.865
0.930
widely adopted defense methods based on differ-
Table 3: The AUC performance of SPV-MIA against LLaMA fine-tuned with DP-SGD w.r. dif- ferent privacy budget e.
ential privacy [15] to provide mathematical pri- vacy guarantees. Through DP-SGD, the amount of information the parameters have about sin- gle data record is bound. Thus, the privacy leakage will not exceed the upper bound, regardless of how many outputs we obtain from the target model. We follow the same manner as the existing study [36] and train LLaMA with DP-Adam on the three datasets. The results are summarized in Table where we choose a set of appropriate as existing works suggest that higher DP guarantees lead to noticeable performance degradation 41. The performances of LLMS are supplemented in Appendix A.5.4 and the results of other baselines can be found in Appendix [..5. The results indicate that DP-SGD can mitigate privacy risks to a certain extent. However, under moderate privacy budgets, SPV-MIA still presents notable risk of privacy leakage and outperforms the baselines.
5.5.1 Impact of Fine-tuning Methods
We further evaluated the generalizability of SPV- MIA under different PEFT techniques. As shown in Table SPV-MIA can maintain high- level AUC across all PEFT techniques. Besides, the performance of MIA is positively correlated with the number of trainable parameters during the fine-tuning process. We hypothesize that this is because as the number of trainable pa- rameters increases, LLMs retain more complete memory of the member records, making them more vulnerable to attacks.
PEFT
LoRA
Prefix Tuning 5.24
P-Tuning
(IA)²
Parameters (M)
33.55
1.15
Wiki
0.951
0.943
0.922
0.914
Ag News
0.903
0.897
0.879
0.873
Xsum
0.937
0.931
0.924
0.911
Table 4: The AUC Performance of SPV-MIA across LLaMAs fine-tuned with different PEFT techniques over three datasets. We choose LoRA [4 Prefix Tuning (35, P-Tuning [3X and
(IA)3 37 four representative PEFT techniques.
6 Conclusion
In this paper, we reveal the under-performances of existing MIA methods against L.MMs for practical applications and interpret this phenomenon from two perspectives. First, reference-based attacks seem to pose impressive privacy Icakages by comparing the sampling probabilities of the target record between target and reference LLMs, but the inaccessibility of the appropriate reference dataset will be a big obstacle deploying it in practice. Second, existing MIAs heavily rely on overfitting, which is usually avoided before releasing LLM for public access. To ddress these limitations, we propose a Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA), where we propose self- self- prompt approach to extract reference dataset from LLM itself in practical manner, then introduce a more reliable membership signal based on memorization rather than overfitting. We conduct substantial experiments to validate the superiority of SPV-MIA over all baselines and verify its effectiveness in extreme conditions. One primary limitation of this study is that SPV-MIA is only designed for CLM, we leave the adaption on other LLMs as the future work.
10

---

Acknowledgments
This research was supported part by the National Natural Science Foundation of China under Grants U2IB2036, U23B2030, 62272262 and the Postdoctoral Fellowship Program of CPSF under Grant Number GZC20240548. We are also grateful for the support from the 6G Research Center, HUST, and the School of Cyber Science and Engineering, HUST, in the form of an International Travel Grant, which enabled Wenjie Fu to attend the conference in person.
We would like to express our sincere gratitude to the anonymous reviewers for their insightful comments and suggestions. We also extend our thanks to Jamie Hayes at Google DeepMind and Qilong Zhang at ByteDDance for their valuable feedback on the preprint manuscript.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '1, pages 308-318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978-1-4503-4139944
[2] Zhipu AL. Zhipu API Documentation. http::/hhgnneeeeseeeerrr/oomommm 2024.
[3] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, et al. Falcon-40b: an open large language model with state-of-the of-the-art performance. Technical report, Technical report, Technology Innovation Institute, 2023.
[4] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- -aptiitee Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206-2240. PMLR, 2022.
[5] Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Puttern recognition, 30(7): 1145-1199919 1997.
[6] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650 2021.
[7] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership Inference Attacks From First Principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914, 2022.
[8] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations, 2022.
[9] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of production language model, In Proceedings of the st International Conference on Machine Learning, pages 5680-5705. PMLR, 2024.
[10] Christopher A. Sraqueete-Chooo Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. Label- only membership inference attacks. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1974-1974. PMLR, 2021. URILtttp htttt:://proeeednnge
mlr ppess/v139 v139/ irooquette-cho222 html
[11] Debeshee Das, Jie Zhang, and Florian Tramèr. Blind baselines beat membership inference attacks for foundation models, 2024.

---

[12] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, (3):220-235) 2023. ISSN 2522-5839.
[13] Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are Diffusion Models Vulnerable to Membership Inference Attacks? In Proceedings of the 38th International Conference on Machine Learning, (ICML) 2023. PMLR, 2023.
[14] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannanch Hajishirzi. Do Membership Inference Attacks Work on Large Language Models?, 2024. URL https
//arxiv. org/ htggabb/2402 2//////222000111111
[15] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-284. Springer, 2006.
[16] Vitaly Feldman. Does learning require memorization? short tale about long tail. In Konstantin Makarychev, Yury Makarychev, Madhur Tulsiani, Gautam Kamath, and Julia Chuzhoy, editors, Proccedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, Chicago, II, USA, June 26-26, 2020, pages 954-959. ACM, 2020. doi: 041145/3357713333842222 URL https: ptttps://doi.oom htt.///ooo.or//////// /1..../...//.../..// 1145/ 3384290
[17] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12. 2020, virtual, 2020. URL https: : http:/:://oooeeddeeeeeeee neurips cc/ www.:ebbbboo..ttttt..t.t...
[18] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. probabilistic fluctuation based membership inference attack for diffusion models. arXiv e-prints, pages arXiv 2308, 2023.
[19] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. MIA-tuner: Adapting large language models as pre-training text detector. In Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, Pennsylvania, USA, 2025.
[20] Stephen Gilbert, Hugh Harvey, Tom Melvin, Erik Vollebregt, and Paul Wicks. Large language model chatbots require approval as medical devices. Nature Medicine, pages 1-3, 2023.
[21] Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. Proceedings Privacy Enhancing Technologies, (1):133-152, 2019.
[22] Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. Model extraction and adversarial transfcrability, your BERT vulnerable! North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2006-2012, Online, 2021. Association for Computational Linguistics. doi:
Proceedings of the 2021 Conference of the
10.18653/w1/2021 .naccl main.161. URL httph:tt://se https:://aclaaattooogy.oott// wwg.:ttthhhhhooo..o/2222222 mtt..//...........mmmaaaaamaaa maacl-main 161
[23] Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnet,, edi- tors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693-1701, 2015. URL httttss//ppoceeddiiite wwwpooeeeiingn.nuuiiiio.o neurips.... cc.ppps...c....p.ccccc..eeeeeece wtt.cc//ppppreee Saaper/2015/hane Tttec/006cc011132224477470003300 htt.://00.....................a. Abstract.html
12

---

[24] Edward J. Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhre Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. S.K..https://openrvviee
[25] Hongsheng Hu, Zoran Salcie, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership Inference Attacks on Machine Learning: Survey, ACM Computing Surveys, 54 (11s):235:1-233:33)
2022. ISSN 0360-0300.
[26] John H Hubbard and Barbara Burke Hubbard. Vector calculus, linear algebra, and differential forms: a unified approach. Matrix Editions, 2015.
[27] Daphne Ippolito. Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. Preventing Generation of Verbatim Memorization in Language Models Gives False Sense of Privacy. In Proceedings of the 16th International Natural Language Generation Conference, pages 28-53, Prague, Czechia, 2023. Association for Computational Linguistics.
[28] Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. Co-writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-15, 2023.
[29] Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: Defending against black-box membership inference attacks via adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 259-274, 2019.
[30] Belveze Jules. Tidr news dataset, 2022. URL JulesBelveze/ tldr news
[31] Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICMI, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 10697-10707. PMLR, 2022. URL httpps://proceedingg mll
press/vi6e 11622kandppll/ htmml
html
[32] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL abs/ 1112.112.9999
[33] Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination Accelerating large language models without impacting performance, ArXiv preprint, abs/2107. 02007,, 2021. URL httts:://axii...tt wwtp:////g///b/////////o//// /20/ 02027
[34] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793-2806, Online, 2020. Association for Computational Linguistics. doi: 1/225/// 249. URL
org/ 2020 acl acl-main. 249
[35] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers). pages 4582 4597, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 122/33/112222.1 Lacl-ooog.l-. URL http::t:/ccllttttoooo org./ogg..or./. 2021 aww.aclllloo .53 353
[36] Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL.
https://openrevieer
13

---

[37] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiiiet fine-tuning better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
[38] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang. and Jie Tang. Gpt understands, too. Al Open, 2023.
[39] Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang. Membership Inference Attacks by Exploiting Loss Trajectory. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS '22, pages 2085-2098 New York, NY, USA, 2022.
Association for Computing Machinery. ISBN 978- 978-1-4503-9450-5.
[40] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:: wwt/ ?dd (tt.rrruunniiageeiiCggg Bkg6Ri
[41] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrier Membership Inference Attacks against Language Models via Neighbourhood Comparison, 2023. URL htttt: //axxiv wwweziiiivrrrreeee hbtg//ab///// 2305. 18462
[42] Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexane Alexandee de Montjoye. Inherent challenges of post-hoc membership inference for large language models, 2024.
[43] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: https://openrevi/
net//forummdddByyy2ddda
[44] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey, ACM Computing Surveys, 56(2):
30:1-30:40, 2023. ISSN 0360-0300.
[45] Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg- Kirkpatrick, and Robert Sim. Privacy regularization: Joint privacy-utility optimization in LanguageModels. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3799-3807, Online, 2021. Association for Computational Linguistics. doi: 10. (0.186533v1// 1/202 mnaacI-main. 298. URL http:/tttt///aaaattttttt/ee
wwgp/rr/////// 2021 maacl-main. 298
[46] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- Kirkkaiikk,,rkk, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8332-8347, Abu Dhabi, United Arab Emirates, 2022. Association for Com- putational Linguistics. URL wtt.::ttts:// hatta::://aclanhhooogyyy wac.logg...rgggg/
emnlp wwt-mmainn...
[47] Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang. David Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816-188 1826. Abu Dhabi, United Arab Emirates, 2022. Association for Com- putational Linguistics. URL https: http::///caaathloogooo wrt.:oogg/go 2022. emnle- monlp-main. 119
[48] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. DetectGPT: Zero-Shot Machine-Generauee Text Detection using Probability Curvature. In Proceedings of the 38th International Conference on Machine Learning, (ICML) 2023, 2023.
[49] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. Pro- ceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v 1/118- 1206. URL http:///aaaaaaholooceee hww.ethhooooooyoog//ee.e.. Dogg/D88-120 1206.
14

---

[50] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 739-753, 2019.
[51] OpenAI.
ChatGPT:
Optimizing Language
Models for Dialogue. 2023.
[52]
OpenAI. OpenAl API Platform. http:///aaooooooooeccoooooooooer 2024.
[53] Charith Peris, Christophe Dupuy, Jimit Majmudar, Rahil Parikh, Sami Smaili, Richard Zemel, and Rahul Gupta. Privacy in the Time of Language Models. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM '23, pages 1291-1292, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 778-1-4503-9407-44
[54] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[55] Samuel Reese, Gemma Boleda, Montse Cuadros, Lluís Padró, and German Rigau. Wikicorpus: word-sense disambiguated multilingual Wikipedia corpus. In Proceedings of the Seventh International Conference Language Resources and Evaluation (LREC'10), Valletta, Malta, 2010. European Language Resources Association (ELRA). URL http://wws. www. 1rec- conf
org/ org/ pproceedings/Lrec2200 pptp 222_22222_PP_ 2aa22_Parr.__ pdf
[56] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Network and Distributed Systems Security (NDSS) Symposium
2019, 2019.
[57] Haonan Shi, Tu Ouyang, and An Wang. Learning-baat based difficulty calibration for enhanced membership inference attacks. In 2024 IEEE 9th European Symposium Security and Privacy (EuroS&P), pages 62-77. IEEE Computer Society, 2024. ISBN 9798350354256.
[58] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoye. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2023.
[59] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[60] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference Attacks Against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18, 2017.
[61] Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations, 2023.
[62] James Stewart. Multivariable calculus: concepts and contexts. Brooks/Cole, 2001.
[63] Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems, 35:21831-21888 2022.
[64] Kushal Tinumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models. Advances Neural Information Processing Systems, 35:38274-38299 2022.
[65] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023. URL.
https: / aariiiivv org/ abs/ 22022 13971
15

---

[66] Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluat- ing the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts, pages 1-7, 2022.
[67] Gerrit J. J. van den Burg and Chris Williams. On memorization in probabilistic deep generative models. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, viriual, pages 27916-27928, 2021. URL
neur cesrrpps.coo cc/ Abstract.
paper/ 2wt/222/1///2/eeee hww.aaaaahhhaaaaaeeeaaaeeeaeaa eae hataa///eeeaaaaaaaaaaaa000000aa0
[68]
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pages 139-150, Online, 2021. Association for Computational Linguistics. doi: 10.186533/1/2021111. maaall-nnaal-mmmn 13.
www.aaaaaaaaaaaaa 13
[69] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
transformer- jax 2021.
[70] Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In The Tenth International Conference on Learning Representattinss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net,
2022. URL httRl:tttss////t/tt hptp/eveeeeeeeeeeeeeeee/eeeeeeee wtwprr/wwweeeeeettuununieeeeoeee met//forummid=3eIrrii01
[71] Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-311 -15, 2007.
[72] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschacdler, and Reza Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS '22 pages 3093-31111 33063, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978-1 1 450---4-5--445555
[73] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEÉE, 2018
[74] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL StLphttts////eerrriiee
.ttt//ooruum7ddd/42offf JECO
[75] Chiyuan Zhang, Daphne Ippolito. Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual Memorization in Neural Language Models. In Advances in Neural Information Processing Systems, 2023.
[76] Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-K%++: Improved bascline for detecting pre-training data from large language models, 2024. URL pestps://arzon
[77] Minxing Zhang, Zhaochun Ren, Zihan Wang. Pengjie Ren, Zhunmin Chen, Pengfei Hu, and Yang Zhang. Membership Inference Attacks Against Recommender Systems. In Proceedings of the 202/ ACM SIGSAC Conference on Computer and Communications Security, CCS '21, pages 674,879, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978-144503-885444
[78] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-leve convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657, 2015. URL, /ttp: http://ahh/Bbeeoooo////33333eeee htt.://bbe...bttaaaaaaaaaaaaa... htmbracct.mmm
www.reuriie...e cttps.cccppppppp cappa/
16

---

[79] Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. Survey of Large Language Models, 2023. URLpttpss: / //axxvv.vvvvv///////////av/aaav/ org/ abs/ 2803. 18223.
17

---

A Appendix
A.1 Ethic and Broader Impact Statements
For ethic consideration, this study proposes membership inference attack algorithm, SPV-MIA, which can be maliciously utilized infer whether specific textual entry is fed the target LLM during the training process. The extensive experiments reveal appreciable privacy leakage of LLMs through SPVMMIA, where the member records are identified with high confidence. We acknowledge that SPV-MIA can bring severe privacy risks to existing LLMs. Therefore, to prevent potential misuse of this research, all experimental findings are based on widely used public datasets. This ensures that every individual textual record we analyze has already been made public, and eliminates any further privacy violations.
For broader impact, we have made our code accessible to the public to allow additional research in the pursuit of identifying appropriate defense solutions. Thus, we posit that our article can inspire forthcoming research to not only focus on the linguistic ability of LLMs, but also take into account the dimensions of public data privacy and security. Besides, our research is more closely aligned with real-world LLM application scenarios, thereby revealing privacy risks in more realistic settings. Our proposed method is scalable, with many aspects left for further exploration and research, such as adapting it for other LLMs.
A.2 Notations of This Work
Table 5: Notations and descriptions.
Notation
Description
x
A specific data record.
Fn
A pair of symmetrical paraphrasing text record of the target text record æ.
mem
The training dataset utilized for LLM fine-tuning.
Dnon
A disjoint dataset from the training dataset.
D Dieter
The reference dataset that collected for fine-tuning reference LLM.
The membership of the data record x0), I represents member, whereas 0 represents non-menee memmee.
mi))
0
The parameters of the target large language model (LLM).
Φ
The parameters of the reference LLM.
0
The parameters of the self- reference LLM.
A (xx 8)
The adversary algorithm for MIA.
pa (x)
The probability of text record æ being sampled by the LLM 0
Po (ar,.)
The probability of paraphrasing text nn being sampled by the LLM
Apo (x)
The calibrated probability of text record æ.
Po (æ)
The probabilistic variation of measured on the target LLM O.
Pg (a))
The probabilistic variation of æ measured on the self self-prompt reference LLM
The calibrated probabilistic variation of æ measured on both the target LLM 0 and the self self-prompt reference LLM 0.
P (x)
N
The query times for estimating pe (x).
18

---

A.3 Detailed Pseudo Codes of Symmetrical Perturbation
Algorithm Symmetrical paraphrase in the embedding domain
Input: Target text set (xx"), Gaussian noise scale a, paraphrasing number N, embedding matrix of tokens E.
Ouput: Symmetrical paraphrased text embedding emb(xx(0)=
1: for a+() E {x do
2: id((x(()) lokenizer(a
Tokenize the text into token ids. Convert token ids into embeddings.
3: emb((x(1) ))
4:
for n E {1. N do
5:
.N,0,
D. Sample noise from Gaussian distribution.
6: emb(x emb(x(1))
7: (mb(x(()) emb(x(())
8: return emb(x
9: end for
end for
Algorithm 2 Symmetrical paraphrase in the semantic domain
Input: Target text set {x(*), }, paraphrasing percentage X, paraphrasing number N, embedding matrix of tokens E.
Ouput: Symmetrical paraphrased text x (0)
for x(1) E (x(()) do
2: id(x(1)) t- tokenizer.enconn enc.er((codddd())))))
Tokenize the into token ids.
3: for (1,...,,) do
4: for t, E id(x tu. t1, ... t//| do
5:
Rand() then
6:
t, + [MASK]
Mask tokens with the percentage X.
7:
else
8:
1, 3-6g end if end for
9:
10:
11:
(t) + MLM((d((x(())
Fill the mask tokens with MLM.
12:
for 13 E (tt} do
13:
emb(t;) E(t;)
Extract the embedding of the original token. > Extract the embedding of the paraphrased token. Measure the paraphrasing noise in the
14:
emb(l1)+ - B((t)
Aemb(t,) emb(t,)+ emb(t;)
embedding domain.
emb(lt) + emb(lt) Demb((y)
D Generate symmetrical embedding.
t,
Searchlearrs((loke((mmmtt))
18:
end for
19:
id a: (+))+ <- FillMaskToken ((ttt)
20:
id (6)-
FillMask {17
x (0+ + balkenizer.dcoolee deeeode(ii (id (z) (+)
21:
22:
(1) + tokenizer decode (id (z))))
23:
return +
24:
end for
25: end for
19

---

Table 6: The MIA performance of SPV-MIA while applied different paraphrasing methods.
Paraphrasing
Embedding
Semantic
Neighbour Comparing
Wiki
0.965
0.951
0.934
AG News
0.926
0.903
0.893
Xsum
0.949
0.937
0.928
Avg.
0.944
0.930
0.918
A.4 Rethinking of the Neighbour Attack
In this work, we introduce a symmetrical paraphrasing method for assessing probabilistic variation, which is motivated by a rigorous principle: detect the memorization phenomenon rather than overfit- ting. Meanwhile, we found that the Neighbour comparing [41] has a similar form to our proposed probabilistic variation. Thus, we further consider reformulating neighbour attack based on our intuition, then provide another explanation and motivation for it. As shown Eq. TO the assessment of probabilistic variation requires pair of symmetrical paraphrased text, thus we elaborately design two paraphrasing models on embedding and semantic domains. However, it is still non-trivial to define two neighboring samples with opposite paraphrasing directions for æ, therefore consider directly ignoring the requirement for symmetry in the probabilistic variation. Thus, we simplify o to be uniformly represented by T. Then we can reformulate Eq. to to Neighbour comparing:
N Σ
2N
Po (I)
2N
Po
n
+ pa (
(x)
2N
Σ (T
Po
po (x).
(11)
Therefore, we believe that the neighbour attack and our proposed probabilistic variation can share the same design motivation, namely, detecting special signals that indicate the LLM has memorized training set samples. Additionally, we compared the neighbour attack with our proposed symmetric paraphrasing methods. As shown Table 6 paraphrasing in the embedding domain achieves considerable performance gains, while paraphrasing in the semantic domain yields a marginal advantage.
A.5
Supplementary Experimental Results
A.5.1
Ablation Study
Table 7: Results of Ablation Study GPT-J and LLaMA across three datasets.
Wiki
AG News
XSum
Target Model
GPT-
LLaMA
GPT-J
LLaMA
GPT-J
LLaMA
w/o PDC
0.648
0.653
0.632
0.641
0.653
0.661
w/o PVA
0.901
0.913
0,864
0.885
0.873
0.919
SPV-MIA
0.929
0.951
0.885
0.903
0.897
0.937
In the previous experiments, we have validated the superiority of our proposed SPV-MIA over existing algorithms, well as its versatility in addressing various challenging scenarios. However, the specific contributions proposed by each module we proposed are still unknown. In this subsection, we conduct ablation study audit the performance gain provided by the two proposed modules. Concretely, respectively remove the practical difficulty calibration (PDC) and probabilistic variation assessment (PVA) that introduced in Section 4.2 and Section 4.3 The results represented in Table 7 where each module contributes a certain improvement to our proposed method. Besides, the PVC approach seems play a more critical role, which can still serve as a valid adversary without the PVA. Thus, in practical scenarios, we can consider removing the PVA to reduce the frequency of accessing public APIs.
A.5.2 OPR@0.15 PPR@0.1%FPR FPR
As supplement to the TPR F%PR results represented in Table 2 we further provide the perfor- mance measured by TPR @0.1 %FPR in Table g
20

---

Method
GPT-2
GPT-
Wild Falcon
ILaMA
Avg.
GPT:2
GPTJ
AG News Falcon
LLaMA
Avg.
GPT-2
GPTJ
Xsam
Falcon
LLaMA
Avg.
Loss Attack
0.1%
0.1%
0.1%
0.1%
0.1%
0.1%
0.1%
0.1%
0.2%
0.1%
0.1%
0.2%
0.1%
0.1%
0.1%
Neighbour Attack
1.2%
0.6%
0.5%
0.8%
0.8%
0.8%
0.4%
0.3%
0.4%
0.5%
0.5%
0.3%
0.3%
0.3%
0.1%
DetecGPTT
0.4%
Min-K%
1.4%
0.6%
0.7%
0.9%
0.9%
1.2%
0.4%
0.7%
0.6%
0.7%
0.5%
0.3%
014%
0.4%
0.4%
Min-KS K5+++
1.2%
0.7%
0.9%
1.1%
1.0%
1.4%
0.9%
1.1%
1.0%
1.1%
0.8%
0.4%
0.7%
0.0%
0.6%
1.7%
1.7%
1.5%
2.8%
HIRA-Candidaa
3.7%
2.5%
2,8%
3.2%
3.1%
2.3%
1.8%
1,7%
1.9%
1.9%
4.7%
3.4%
3.8%
5.1%
4.3%
SPV-IA
39.1%
28.9%
32.7%
37.8%
34.6%
25.3%
17.3%
18.7%
23.5%
21.2%
34.4%
27.6%
28.9%
31.5%
30.6%
Table 8: TPR@0.1 10..0.0.FPPPP FPR for detecting member texts from four LLMs across three datasets for SPV-MIA and five previously proposed methods. Bold and Underline respectively represent the best and the second-best results within each column (model-dataset pair).
A.5.3 AUC Curves
As supplement to the main experimental results represented in Table 1 we further provide the raw ROC curve for a more comprehensive presentation in Fig. (linear-scale) and Fig. 7 (log-scale).
Pockivv Rate
0.8
Pocktvv Rate
0.8
Thue Positive Rate
0.6
SPV-MIA Neighbour
SPV-MIA
SPV-MIA
0.4
0.4
0.4
Neighbour
Neighbour
LRA (Base)
LRA (Base)
LRA (Base)
LRA (Candidate)
LRA (Candidate)
LRA (Candidate)
0.0
0.0
0.0
0.
0.
0.6
0.8
10
0.8
False Rate
Positive Rate
Positive
(a) Wiki
(b) AG News
(c) Xsum
Figure Linear-scale ROC curves of SPV-MIA and the top-three best baselines on ILaMAs fine-tuned over three datasets.
True Positive Rate
20-4
True Posibve
1.0
10:4
SPV-MIA
SPV-MIA
SPV MIA
Thue
10-9
Nelghbour Attack
10a9
Nelghbour Attack
10:2
Nelghbour Attack
LRA (Base)
LRA (Base)
LRA (Base)
LRA (Candidate)
LRA (Candidate)
LRA (Candidate)
13
10
10
10t4
10
10
10-4
15aa
10ts
10P
10a
10-h
10-1
10P
Faise Positive Rate
False Positive Rate
False Positive Rate
(a) Wiki
(b) AG News
(c) Xsum
Figure 7: Log-scale ROC curves of SPV-MIA and the three representative baselines on LLaMAs fine-tuned over three datasets.
A.5.4 Performance of Target LLMs
We supplemented the performance of all LLM-dataset pairs on both the training and test sets (estimated using PPL). As shown in Table the experimental results indicate that none of the fine- tuned LLMS exhibit significant overfitting, which aligns with our claim in the main body. Additionally, we provided the performance of the LLM under different privacy budgets as shown in Table 10
A.5.5 Performance of Representative Baselines under Different Privacy Budgets
We have conducted three representative baseline attacks for the DP-SGD model (LLaMA fine-tuned over Ag News dataset) and compared it with our method (SPV-MIA). The results are provided in Table TT The results demonstrate that SPV-MIA consistently maintains substantial MIA performance margins over different settings of the privacy budget.
21

---

Table 9; The perplexity (PPL) of each LLM-dataset pair on training set and test set.
GPT-2
GPT-J
Falcon
LLaMA
Target Model
Training
Test
Training
Test
Training
Test
Training
Test
Wiki
25.34
27.47
12.35
12.44
7.26
7.73
7.00
7.43
AG News
23.86
26.34
12.49
13.24
8.92
9.54
9.03
9.13
Xsum
26.42
28.31
13.43
13.88
7.69
7.96
7.35
7.65
Table 10: The perplexity (PPL) of each LLM-dataset pair trained w.r.t different privacy budget
Privacy Budget
15
30
60
inf
Wiki
8.45
8.16
7.76
7.43
AG News
10.84
9.68
9.32
9.13
Xsum
8.89
8.33
7.98
7.65
A.6 Experimental Settings
In this subsection, we give extensive introduction of experimental settings, including the datasets, target LLMs and baselines, well the implementation details.
A.6.1 Datasets
Our experiments utilize six different datasets across multiple domains and LLM use cases, where we employ three datasets as the private datasets to fine-tune the target LLMs, and the remaining datasets the public datasets from the exact domains. Specifically, we use the representative articles on Wikitext- 103 dataset 43 to represent academic writing tasks, news topics from the AG News dataset [78 to represent news topic discussion task, and documents from the XSum dataset (49 to represent the article writing task. Besides, we utilize Wikicorpus [55], TLDR News [30], and CNNDM 23 datasets to respectively represent as the publicly accessible dataset from the same domain for each task.
A.6.2 Target Large Language Models
To obtain a comprehensive evaluation result, we conduct our experiments over four well-known and widely adopted LLMs as the pre-trained models with different scales from 1.5B parameters to 7B parameters:
It is transformer-be language model released by OpenAI in 2019, which has 1.5 billion parameters and is capable of generating high-quality text samples.
GPT-2 [54]:
GPT-J 6691 It is open-source LLM released by EleutherAl in 2021 variant of GPT-3.
GPT-J has 6 billion parameters and is designed to generate human-like with appropriate prompts.
Falcon-7B 33: Falcon is family of state-of-the-art LLMs created by the Technology Innovation Institute 2023. Falcon has 40 billion parameters, and Falcon-7B is the smaller version with less consumption.
LLaMA-7B 665) LLaMA is one of the most state-of oftttrrrrt LLM family open-sourced by Meta AI in 2023, which has outperformed other open-source LLMs on various NLP benchmarks. It has 65 billion parameters and has the potential accomplish advanced tasks, such as code generation. In this work, we utilize the lightweight version, LLaMA-7B.
A.6.3 Baselines
We choose six MIAS designed for LMs comprehensively evaluate our proposed method, including three reference-free attacks and one reference-based attack with one variant.
Loss Attack [73]: A standard metric-based MIA that distinguishes member records simply by judging whether their losses are above a preset threshold.
The Neighbour Attack avoids using reference model to calibrate the loss scores and instead utilizes the average loss of plausible neighbor texts as the benchmark.
Neighbour Attack 411
22

---

Table 11: The AUC performance of SPV-MIA and three representative basclines w.r.t different privacy
budget c.
Privacy Budget
15
30
60
inf
Loss Attack
0.523
0.551
0,568
0.580
Neighbour Attack
0,542
0.564
0.587
0.610
0.611
0.655
0.684
0.714
SPV-MIA
0.766
0.814
0.852
0.903
zero-shot machine machine-generated text detection method. Although DetectGPT is specially designed for LLMs--eneratee text detection, but has the potential to be adapted for identifying the text utilized for model training.
DetectGPT [48]
Min-K % [59 An MIA method designed for pre-trained LLMs, which evaluate the token-level
probability and employ the average over the 4% lowest probability as the MIA metric.
Min-K +++ (76] An enhanced version of Min-K% that utilizes more sophisticated mechanism to detect the records with relatively high probability curvature.
Likelihood Ratio Attack (LiRA-Base) [7]
A aeference-based attack, which adopts the pre- trained model as the reference model to calibrate the likelihood metric to infer membership.
LiRA-Candidate 47 A variant version of LiRA, which utilizes publicly available dataset in the same domain as the training set fine-tune the reference model.
A.6.4 Detailed Information for Reproduction
Table 12: Detailed split and other information of datasets.
Relative Datasets
Target Model
Reference Model
Dataset
Domain-speciff
Irrelevant
Member
Non-member
# Member
# Non-member
Wikitext-100 103
Wikicorpus
AG News
10,000
1,000
10,000
1,000
AG News
TLDR News
Xsum
10,000
1,000
10,000
1,000
Xsum
CNNDM
Wikitext-103
10,000
1,000
10,000
1,000
All experiments are compiled and tested on a Linux server (CPU: AMD EPYC-7763, GPU: NVIDIA GeForce RTX 3090), Each set of experiments for the LLM-dataset pairs took approximately hours, and we spent around 14 days completing all the experiments. For each dataset, we pack multiple tokenized sequences into a single input, which can effectively reduce computational consumption without sacrificing performance 33 Besides, the packing length is set to 128 tokens. Then, we use 10,000 samples for fine-tuning over pre-trained LLMs and 1,000 samples for evaluation. The detailed information of datasets is summarized in Table [2] For each target LLM,, we let it fine-tuned with the training batch size of 16, and trained for 10 epochs. The learning rate is set 0.0001. We adopt the W optimizer 40 to achieve the generalization of LLMs, which is composed of the Adam optimizer
32 and the L2 regularization. For GPT-2, which has relatively small scale, we adopt the full fine-tuning, which means all parameters are trainable. For other LLMs that are larger, we utilize a parameter-efficient fine-tuning method, Low-Rank Adaptation (LoRA) [24], as the default fine-tuning method. For the paraphrasing model in the embedding domain, the Gaussian noise scale set to o = 0.05. For the paraphrasing model in the semantic domain, the paraphrasing percentage set to À = 0.2. For both of the two paraphrasing models, generate 10 symmetrical paraphrased text pairs for each target text record. For the reference LLM fine-tuned with our proposed self- seff-prompt approach, utilize the Nomain-speciff data the default prompt text source. Then, collect 10,000 generated texts from target LLMs with an equal length of 128 tokens to construct reference datasets. We fine-tune the reference LLM for 4 epochs and the training batch size of 16.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims made in the abstract and introduction are aligned with contributions and scope. All claims match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification: As we have discussed in Appendix A.5.1 the proposed PVC method has high computational overhead and provides low performance gains. Therefore, we suggest considering removing this module and only retaining the PDC in practical scenarios. Besides, the proposed SPV-MIA requires two additional API accesses, which may not available in some scenarios.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, we provide a complete derivation/prooff for our proposed probabilistic
variation metric in Section 4.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
25

---

Answer: [Yes]
Justification: Yes, all experimental settings, codes, and datasets are provided Appendix
and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, all experimental settings, codes, and datasets are provided in Appendix A.6 and supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https! for more details.
//nips /nwpps.ccppubbai wttp:cccppppccc/tettette.e...clt http:ppllllgguse//eeeeeeoee http://eeeeeeeeeeuueeeeeiooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
26

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, all experimental settings are provided in Appendix A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Since all experiments are taken on LLMs, error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provide sufficient information of computer resources in Appendix A.6
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::/neurrips http://aarr......o.o.....cccttta
Answer: [Yes]
Justification: We carefully reviewed the Code of Ethics and confirmed that we have adhered each one.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
We have provide Section "Ethic and Broader Impact Statements" follows the reference.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
28

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer:
[NA]
Justification: As discussed in Appendix A.6 all models and datasets used in this study are publicly accessible.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the creators original owners of assets are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version URL.
asset used and, possible, include
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The documentation be found in the supplementary materials the anonymized URL:
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moddll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
29

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "Fkwz4pbNMp", "reviewer": "Reviewer_v2Kc", "summary": "This paper studies the membership inference attack on large language models finetuned on private data. Instead of reusing other pre-trained public large language models, the paper proposes a way to generate a reference dataset and finetune this dataset to attain a reference model. With this reference model, the paper further calculates the score by probabilistic variation assessment: symmetrically rephrasing the target data and calculating the average score as the final score. As evaluated in the experiment, the proposed attack has a large margin above the baselines. The paper also conducted ablation studies to understand the design of important and demonstrates the robustness of the algorithm.", "questions": "See the weakness.", "limitations": "The paper has well-discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The algorithm is novel. It proposes a way to generate the reference dataset via self-prompting, which leads to a high-quality reference model for a better membership inference attack.\n2. The results seem very promising. The performance of the proposed attack has a large margin over the baselines.\n3. The algorithm is robust to assuming the domain-specific public data. Even if it is relaxed to irrelevant data, the proposed attack still achieve good performance.", "weaknesses": "1. It is not fully in the black-box setting, because it assumes the knowledge of the same pre-trained model and the access of its parameters.\n2. The algorithm has two novel parts: propose a way to generate the reference dataset for learning a reference model and a novel score similar to probabilistic variation assessment. It might be important to study how much each component contributes to the final improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer v2Kc,\n\nThank you so much for your thoughtful review and overall positive comments. Below we address the concerns raised in your review:\n\n**W1: It is not fully in the black-box setting, since it assumes the knowledge of the same pre-trained model and the access of its parameters.**\n\nIn fact, SPV-MIA does not require access to the parameters of the target model, indicating that we do not use a white-box setting. However, we are not using a pure black-box setting either, since SPV-MIA requires the following adversary knowledge (two APIs):\n* The access to the query API with response results (generated texts and logits) of the target model.\n* [+] The access to the fine-tuning API of the pre-trained version of the target model.\n\nHere, we only add a fine-tuning API compared with existing works [1,2,3,4]. Overall, we do not require white-box access to the parameters of this pre-trained model; an API that allows the adversary to upload a customized dataset for fine-tuning would also meet the conditions. Therefore, our proposed method is not only applicable to open-source models with publicly available parameters but also for closed-source models that provide fine-tuning APIs (such as ChatGPT [5]). We will include the above clarification in the camera-ready version.\n\nFurthermore, to further allay your concerns, we consider a truly black-box scenario in which the adversary is constrained to fine-tune a pre-trained model different from the target model as the reference model. We employ LLaMA-7B fine-tuned over the Agnew dataset as the target model, and let the adversary fine-tune over different pre-trained models (GPT-J, Falcon, and LLaMA are adopted) to prepare the reference model. The results are shown in the table below. From the experimental results in the table, we can find that the proposed method still maintains a large performance margin over the baselines (LiRA-Base, LiRA-Candidate) without this constraint. We will include these results in the camera-ready version.\n\n|Metric|AUC|TPR@1%FPR|TPR@0.1\\%FPR|\n|----------------------|-----|---------|-----------|\n|SPV-MIA (GPT-J)|0.832|19.3\\%|7.5\\%|\n|SPV-MIA (Falcon)|0.812|21.7\\%|8.7\\%|\n|SPV-MIA |0.903|39.5\\%|23.5\\%|\n|LiRA-Base |0.657|8.7\\%|1.4\\%|\n|LiRA-Candidate |0.714|11.5\\%|1.9\\%|\n\n**W2: Conducting an ablation study to investigate the contributions of each proposed component would be beneficial.**\n\nIn Appendix A.5.1, we have shown the ablation study for evaluating the contributions of the proposed Practical Difficulty Calibration (PDC) and probabilistic variation assessment (PVA) separately. Furthermore, we have also conducted a comprehensive ablation study in Appendix A.4 to investigate the contributions of our proposed two symmetrical paraphrasing methods compared with neighbour attack, in which we adopt the two symmetrical paraphrasing methods, and neighbour attack as the PVA module, respectively. Thank you for the reminder, we will emphasize the ablation study in the main body of the camera-ready version.\n\nIn terms of these ablation studies shown in the Appendix, we quote the experimental results and merge them into one table as follows:\n\nTable: Results of Ablation Study on LLaMA across three datasets.\n|Dataset|Wiki|AG News|XSum|Avg.|\n|--------------------------------------|-----|-------|-----|-----|\n|**SPV-MIA (Embedding-based Paraphrasing)**|**0.956**|**0.926**|**0.949**|**0.944**|\n|SPV-MIA (Semantic-based Paraphrasing)|0.951|0.903|0.937|0.930|\n|SPV-MIA (Neighbour Attack)|0.934|0.893|0.928|0.918|\n|w/o PVA|0.913|0.885|0.919|0.906|\n|w/o PDC|0.653|0.641|0.661|0.652|\n\nThe results demonstrate that both PDC and PVA contribute a certain improvement to our proposed method. However, the PDC approach seems to play a more critical role, which can still serve as a valid adversary without the PVA. Thus, in practical scenarios, we can consider removing the PVA to reduce the frequency of accessing public APIs. Additionally, the proposed two paraphrasing methods both yield considerable performance gains compared to the neighbour attack.\n\n# Reference\n[1] Shi, Weijia, et al. \"Detecting Pretraining Data from Large Language Models.\" The Twelfth International Conference on Learning Representations (2024).\n\n[2] Duan, Michael, et al. \"Do membership inference attacks work on large language models?.\" Conference on Language Modeling (COLM) (2024).\n\n[3] Mattern, Justus, et al. \"Membership Inference Attacks against Language Models via Neighbourhood Comparison.\" Findings of the Association for Computational Linguistics: ACL 2023. 2023.\n\n[4] Zhang, Jingyang, et al. \"Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models.\" arXiv preprint arXiv:2404.02936 (2024).\n\n[5] Andrew Peng, Michael Wu, John Allard, Logan Kilpatrick, and Steven Heidel. Gpt-3.5 turbo finetuning and api updates, August 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nDFKZsJ6ZW", "reviewer": "Reviewer_DLpC", "summary": "This paper proposes self-calibrated probabilistic variation (SPV)-MIA, a membership inference attack. The novel ideas that SPV-MIA introduces in the space of LLMs: 1) using paraphrasing to obtain samples around the target sample text in the sample domain, and using paraphrased texts to compute probabilistic variation of the target model around the target, and 2) using self-prompting to generate reference data that SPV-MIA uses to train a reference model which is then used to calibrate the probabilistic variation of the target model on the target samples. Experimental evaluation on three benchmark datasets show that SPV-MIA outperforms existing MIAs.", "questions": "- Idea of PVA is similar to neighborhood attack. The paper claims that they generalize the notion of neighborhood attack but end up using the neighborhood attack. This is alright, but how does this generalization help? Is the paper merely depicting neighborhood attack using a set of equations or does the generalization can be somehow helpful to improve the attack success? \n- Experiments use small benchmark datasets for fine-tuning which are probably also a part of training data of pre-trained base model; both of these can lead to overestimation of power of proposed MIA. Why do you use small datasets? How does the proposed MIA work for larger fine-tuning datasets? \n- About DP results:\n    - Can you explain how do you perform DP training? \n    - How do the baseline attacks perform for the same DP guarantees?", "limitations": "- Writing of the paper is poor; there are too many grammatical mistakes (even in abstract) and many sentences are not properly constructed making it difficult to read the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Multiple interesting ideas that result in a strong, practical MIA on LLMs\n- Good and easy to follow MIA", "weaknesses": "- Some parts of the paper need support, e.g., how PVA is more general than neighborhood attack.\n- Datasets evaluated with are tiny; larger datasets should be used\n- Writing can be improved for better comprehension", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer DLpC,\n\nWe deeply appreciate the time you took to review our work and your comments for improvement. Below we address the concerns raised in your review:\n\n**Q1,W1: Questions about the PVA generalization.**\n\n> **Q1.1: The paper claims the generalization of PVA but end up using the neighborhood attack.**\n\n\nThe default paraphrasing method used is not the neighbour attack but the proposed semantic-based symmetrical paraphrasing method. Unlike the neighbour attack, each paraphrasing operation of the proposed methods will generate a pair of symmetrical neighbouring samples. In contrast, paraphrasing operations in the neighbour attack produce independent neighbouring samples. \n\n> **Q1.2: How does the generalization of PVA help? Does the generalization can be somehow helpful to improve the attack success?**\n\nThe proposed PVA algorithm not only uses a series of generalized equations to reformulate the neighbour attack but also actually improves the attack success. Therefore, the PVA contributes both theoretically and empirically:\n\n* **Empirical contribution** (improve attack success)\n\nWe derive two symmetrical paraphrasing methods (Semantic-based and Embedding-based) from PVA. We have conducted an extensive ablation study in Appendix A.4, where the results demonstrate that both two proposed methods achieve considerable performance gains compared with neighbour attack. Thus, the proposed PVA can actually improve the attack performance. \n\nTable: Results of Ablation Study on LLaMA across three datasets.\n|Dataset|Wiki|AG News|XSum|Avg.|\n|-|-|-|-|-|\n|SPV-MIA (Semantic-based Paraphrasing)|0.951|0.903|0.937|0.930|\n|SPV-MIA (Embedding-based Paraphrasing)|0.956|0.926|0.949|0.944|\n|SPV-MIA (Neighbour Comparing)|0.934|0.893|0.928|0.918|\n\n* **Theoretical contribution** (Motivate paraphrasing-based attacks from another perspective)\n\nThe existing paraphrasing-based attacks (e.g., Neighbour Attack) are motivated by exploring a substitution of reference models. In this work, we provide another motivation based on model memorization, that is, the member text memorized by the model will tend to be located on the local maximum in the generative distribution [1]. Therefore, this study may inspire the following studies to design more sophisticated methods for inferring membership by characterizing the model memorization. Coincidentally, a recently published work (after the NeurIPS deadline) recommended by reviewer i2LA [2] also proposed an MIA against pre-trained LLMs based on the concept of local maximum detection.\n\nThank you for the reminder, we will emphasize the contributions of PVA in the main body of the camera-ready version.\n\n**Q2,w2: Questions about small datasets.**\n\n> **Q2.1 Why do you use small datasets? The small datasets maybe a part of pre-training data, which can lead to overestimation of MIA performance.**:\n\nIn fact, we employed two datasets (Ag News, Wiki) that are the same as those used in existing studies [3,4], as well as one larger dataset (Xsum). The possibility of overlap between the fine-tuning and pre-training datasets will lead to an underestimation of MIA performance. Since if the fine-tuning dataset is a part of the pre-training data, then both the training and evaluation splits of the fine-tuning dataset will be included in the pre-training data with a large probability. In this scenario, we not only need to detect texts used during the fine-tuning stage but also avoid falsely identifying texts that were only used during the pre-training stage and did not appear in the fine-tuning stage as member texts.\n\n> **Q2.2: How does the proposed MIA work for larger fine-tuning datasets?**:\n\nAlthough we cannot use larger datasets as the default setting for the overall experiment, we agree that exploring larger fine-tuning datasets would further enhance the quality of our work. Therefore, we used a dataset nearly 10x larger than the current ones, Newsroom, to fine-tune LLaMA as the target model. The results in the table below demonstrate that SPV-MIA still maintains relatively good performance for the larger dataset.\n\n|Metric|SPV-MIA|Neighbour Attack|LIRA-Candidate|\n|-|-|-|-|\n|AUC|0.862|0.545|0.665|\n\n\n**Q3: Can you explain how do you perform DP training? How do the baseline attacks perform for the same DP guarantees?**:\n\nWe follow the private-transformers [5], using ghost clipping to perform DP training in a memory-saving manner. We have conducted three representative baseline attacks for the DP-SGD model (LLaMA@Ag News) and compared it with our method (SPV-MIA). The results are provided in the table below. The results demonstrate that SPV-MIA consistently maintains the best MIA performance over different settings of the privacy budget. Thank you for your suggestion to improve our work further. We will update these results to the camera-ready version.\n\n|Privacy Budget $\\epsilon$|15|30|60|+inf|\n|-|-|-|-|-|\n|Loss Attack|0.523|0.551|0.568|0.580|\n|Neighbour Attack|0.542|0.564|0.587|0.610|\n|LIRA-Candidate|0.611|0.655|0.684|0.714|\n|SPV-MIA|0.766|0.814|0.852|0.903|\n\n# Reference\n[1] van den Burg, Williams. \"On memorization in probabilistic deep generative models.\" NeurIPS 2021.\n\n[2] Zhang, et al. \"Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models.\" arXiv 2024.\n\n[3] Mattern, et al. \"Membership Inference Attacks against Language Models via Neighbourhood Comparison.\" ACL 2023.\n\n[4] Mireshghallah, et al. \"An empirical analysis of memorization in fine-tuned autoregressive language models.\" EMNLP 2022.\n\n[5] Li, et al. \"Large Language Models Can Be Strong Differentially Private Learners.\" ICLR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c05pXIvAiB", "reviewer": "Reviewer_saYA", "summary": "This paper presents a membership inference attack against causal language models, addressing the limitations of previous attacks, such as the inaccessibility of appropriate reference datasets and heavy reliance on overfitting. To overcome these limitations, the authors propose a self-prompt approach to extract reference datasets from LLMs and introduce a membership signal based on memorization. They compare their method with other methods using the AUC metric.", "questions": "- Does the proposed approach require training shadow models? If not, how is the threshold τ in equation (2) determined?\n\n- How many datasets are extracted for each case in training the reference models?\n\n- What are the key differences between memorization and overfitting, and how does the proposed approach improve from the perspective of memorization?", "limitations": "Please see my above comments.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper effectively identifies two key limitations of previous membership inference attacks against LLMs, providing a clear motivation for the proposed targeted solutions.\n\n- The proposed probabilistic variation assessment is interesting and has potential applications beyond LLMs.\n\n- The paper is well-written with a clear logical structure, making it easy to follow.", "weaknesses": "- The main evaluation metrics need to be revised. As established by Carlini et al. [R1], membership inference attacks should be evaluated by computing their true-positive rate at low (e.g., ≤ 0.1%) false-positive rates, rather than using average-case metrics like AUC. Such evaluation metrics have become the de facto standard in evaluating membership inference attacks and are used by many existing works [R2] [R3]. I suggest using two metrics: the Full Log-scale Receiver Operating Characteristic (ROC) Curve to highlight low false-positive rates, and the TPR at a low FPR, which measures attack performance at specific FPRs (e.g., 0.1%, 0.01%).\n\n- The experimental settings are not clearly described. For example, the paper uses a self-prompt approach to extract reference datasets, but it does not specify how many datasets are extracted for each case, which directly relates to the attack costs. This is especially important for commercial LLMs since attacking commercial LLMs will incur high costs in collecting such datasets. Additionally, details on dataset splitting (e.g., how many datasets are used for training the target model) and whether the approach requires training shadow models are missing. If it does not require shadow models, how is the threshold τ in equation (2) determined?\n\n- The paper argues that the proposed probabilistic variation assessment is based on memorization rather than overfitting, but it does not clearly explain the key differences between these two concepts or how the approach improves from the perspective of memorization.\n\n- The reference-based baselines used in the paper are limited and somewhat outdated. Including more advanced baselines, such as [R4], would strengthen the comparison.\n\n- An ablation study to investigate the contributions of the proposed self-prompt approach and probabilistic variation assessment separately would highlight the individual contributions of the paper.\n\n\n[R1] Carlini, Nicholas, et al. \"Membership inference attacks from first principles.\" 2022 IEEE Symposium on Security and Privacy (2022).\n\n[R2] Bertran, Martin, et al. \"Scalable membership inference attacks via quantile regression.\" Advances in Neural Information Processing Systems (2023).\n\n[R3] Wen, Yuxin, et al. \"Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries.\" International Conference on Learning Representations (2023).\n\n[R4] Shi, Haonan, et al. “Learning-Based Difficulty Calibration for Enhanced Membership Inference Attacks”. IEEE European Symposium on Security and Privacy (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer saYA,\n\nThank you so much for your thoughtful review and your suggestions for improvement. Below we address the questions raised in your review, the responses to the weaknesses and the reference can be found in subsequent comments:\n\n**Q1,W2: Does the proposed approach require training shadow models? If not, how is the threshold τ in equation (2) determined?**\n\nThis is an interesting and meaningful topic, since current MIAs designed for LLMs mostly not discuss how to select an appropriate threshold [1,2,3]. Sharing a similar idea with [4], the proposed method (SPV-MIA) can determine the threshold without a shadow model by picking a quantile on the distribution of confidence scores induced by SPV-MIA on texts that are not used in fine-tuning. In particular, we collected a dataset $\\\\{\\boldsymbol{x}_i\\\\}_1^n$ from an unrelated task, known to not have been used in fine-tuning. We evaluate the SPV-MIA for each text $\\boldsymbol{x}_i$, and record the confidence score $s(\\boldsymbol{x}_i)$ that SPV-MIA places it as a member text. We then pick the $1-\\alpha$ quantile from the score distribution. Intuitively, a score $s(\\boldsymbol{x}_i)$ larger than the $1-\\alpha$ quantile indicates that SPV-MIA assigns a confidence on the member text that is higher than a $1-\\alpha$ fraction of the texts not used in training. Thus, we can select an expected false positive rate of $\\alpha$, then determine the according threshold $\\tau$ by measuring the $1-\\alpha$ quantile of the score distribution. To further support our statement, we visualize the score distribution of member and non-member texts evaluated by SPV-MIA (see the Figure 2 in PDF file of the global rebuttal). \n\n**Q2,W2: How many datasets are extracted for each case in training the reference models? This is especially important for commercial LLMs.**\n\nAs we have summarized in Appendix A.6.4, the number of samples extracted for fine-tuning reference models is set to 10,000 (16 prompt tokens and 112 completion tokens for each). We used the price of GPT-3.5-turbo-0125 as a reference (input: \\\\$0.50 / 1M tokens, output: \\\\$1.50 / 1M tokens), and extracting such a dataset from commercial LLMs would cost approximately only \\$1.76. Thus, we believe that it is an acceptable value even for attacking commercial LLMs.\n\nAdditionally, we have explored the impact of smaller reference datasets (in Section 5.4, Fig. 5(a)), and we quote the relevant results in the table below. This result demonstrates that even in the extracted reference dataset with only 1,000 samples, SPV-MIA achieves performance comparable to 10,000 samples.\n\nTable. The performances of SPV-MIA on LLaMA@AG News while utilizing different scales and sources of extracted reference datasets.\n|Extracted Dataset Scale|1,000|2,000|5,000|10,000|\n|-|-|-|-|-|\n|Irrelevant|0.879|0.882|0.886|0.897|\n|Domain-specific|0.890|0.892|0.895|0.903|\n|Identical-distribution|0.896|0.912|0.915|0.922|\n\n**Q3,W3: Questions aboout memorization.**\n\n> **Q3.1: What are the key differences between memorization and overfitting?**\n\nAlthough memorization is associated with overfitting, overfitting by itself cannot completely explain some properties of memorization [5,6,7].\nThe key differences between memorization and overfitting can be summarized as the following three points:\n* **Occurrence Time**\n Existing research defines the first epoch when the LLM's perplexity (ppl) on the validation set starts to rise as the occurrence of overfitting [5]. In contrast, memorization begins early [5,6] and persists throughout almost the entire training phase [6,7].\n* **Harm Level**\n Overfitting is almost universally acknowledged as a detrimental phenomenon in machine learning. However, memorization is not exclusively harmful, and can be crucial for certain types of generalization (e.g., on QA tasks) [8,9].\n* **Avoidance Difficulty**\n Since memorization occurs much earlier, even if we use early stopping to prevent the model from overfitting, we will still achieve significant memorization [6]. Moreover, since memorization is crucial for certain tasks [8,9], and separately mitigates specific unintended memorization (e.g., verbatim memorization [10]) is a non-trivial task.\n\n> **Q3.2: How does the proposed approach improve from the perspective of memorization?**\n\nBased on the aforementioned discussion, memorization is a more common and robust phenomenon in LLMs compared to overfitting. Therefore, we believe that identifying the memorization footprint left by LLMs on member texts can help improve MIA. Moreover, memorization in generative models [11] or LLMs [2] causes an increased tendency of generative probability density around member records. Thus, we can translate the problem of detecting member texts into identifying texts that are close to the local maximum. Subsequently, we designed a novel metric, probability variation, as an indicator of local maxima and proposed two symmetrical paraphrasing methods to estimate it (Section 4.3). Consequently, our proposed method can enhance the identification of memorization. We have conducted ablation experiments (Appendix A.4, Table 5) to show that our proposed probability variation assessment (PVA) approach can further improve the success rate of MIA compared with neighbour attack. We quote the existing experimental results in the table below.\n\nTable: Results of Ablation Study on LLaMA across three datasets.\n|Dataset|Wiki|AG News|XSum|Avg.|\n|-|-|-|-|-|\n|SPV-MIA (Embedding-based Paraphrasing)|0.956|0.926|0.949|0.944|\n|SPV-MIA (Semantic-based Paraphrasing)|0.951|0.903|0.937|0.930|\n|SPV-MIA (Neighbour Attack)|0.934|0.893|0.928|0.918|\n\nThanks for your valuable questions for improvement, we will include these discussions in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMEnzXmIYo", "reviewer": "Reviewer_i2LA", "summary": "In this work, the authors propose an MIA for LLMs that utilizes prompt calibration to measure variation on model behavior for neighboring inputs. The authors also make a connection with the neighborhood attack from Mattern et al and show how their framework encapsulates such neighborhood-based attacks. Performance on multiple domains demonstrate a clear bump in performance using this attack.", "questions": "- In the codebase, at line 65 of `ft_llms/refer_data_generate.py`, the prompt data is sampled directly from 10K-20K range of samples from the train data, so there is a very clear bias here (supposed to be from same \"distribution\"?) If prompt data is indeed sampled from train data, completions would likely be the actual data itself, so reference model would behave similar to target model. Scores would thus be $m(x) - m(\\hat{x}) \\approx 0$ for train data, and non-zero for validation data. If this is indeed true, this is a **big empirical flaw**. I would urge the authors to clarify, and also plot the distributions of actual scores corresponding to $m(x)$. I may consider increasing my score if the authors can look into.\n\n- Why is there no mention of any studies related to membership inference for LLMs in Section 2 (Large Language Models)? There have been plenty specifically looking at LLMs [1,2,3,4].\n\n- L147: \"adversary has no prior information\" - Does it have access to the model's initial weights?\n\n- Figure 5b: Why is there a drop in performance for the domain-specific case?\n\n### References\n- [1] Maini, Pratyush, et al. \"LLM Dataset Inference: Did you train on my dataset?.\" arXiv preprint arXiv:2406.06443 (2024).\n- [2] Mozaffari, Hamid, and Virendra J. Marathe. \"Semantic Membership Inference Attack against Large Language Models.\" arXiv preprint arXiv:2406.10218 (2024).\n- [3] Meeus, Matthieu, et al. \"Inherent Challenges of Post-Hoc Membership Inference for Large Language Models.\" arXiv preprint arXiv:2406.17975 (2024).\n- [4] Duan, Michael, et al. \"Do membership inference attacks work on large language models?.\" arXiv preprint arXiv:2402.07841 (2024).\n- [5] Das, Debeshee, Jie Zhang, and Florian Tramèr. \"Blind Baselines Beat Membership Inference Attacks for Foundation Models.\" arXiv preprint arXiv:2406.16201 (2024).", "limitations": "There is no proper discussion about limitations, apart from a 1-2 sentences in the Appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Attempts to improve black-box privacy auditing are useful and needed, especially with more and more model trainers where access is available only via wrappers or APIs.\n- Figures and diagrams in the paper are very well made and helpful, and complement the writing well.\n- Assessing the method's robustness for different sources of prompts is useful and helps understand worst-case performance when prompt data is completely unrelated to the member distribution.", "weaknesses": "- The core idea here is not very different from distillation-based stealing [1], followed by a slightly modified neighborhood attack.\n\n- L240: There is no clear relationship between \"modest\" paraphrasing in token space, and a corresponding $h$ in the Equation 10, let alone plus/minus. For a language model, $x$ is in the embedding space, so small perturbations to the embeddings will probably not correspond to any actual tokens. Moreover, inputs are sequences and thus perturbations to multiple tokens makes the assumption around a small $h$ even less plausible. Replacing 20% of all tokens is in no way a \"modest\" replacement.\n\n- In Table 1: why not include a version in the baseline that uses the reference models that you train but does not use the neighborhood logic. This would help better understand the contribution of both these steps, serving as an ablation study. \n\n- L631-633: This is not a grounded statement. Making the connection not only seems unnecessary, but incorrect. While it is nice to have theoretical connections, it is by no means necessary for a good paper. I would urge the authors to rethink the use of theory to make connections in places where they do not seem valid without handwavy justifications. \n\n## Minor comments\n- L32: '[44]' is merely a presentation abstract. [2], for instance, provides an actual systemization of memorization in LLMs\n- Figure 1b - what is \"memorization\"? Figures should be labeled clearly. \n- The authors seem to be confused about which works introduced LiRA. The authors seem to suggest that Mireshghallah et al introduced LiRA (L45, L102), whereas it was proposed by Carlini et al. [3].\n- L49-50: While overfitting means higher leakage, it is by no means an \"assumption\" for MIAs.\n- L59: \"Exponential\" is a bold claim here; there are only 3 datapoints. It is not surprising that it declines, as it is explained by theory [4]\n- L81: Computing a second-order derivative in an LLM is just not feasible practically. \n- L84: The neighborhood-based MIA requires another masking model, which need not be an LLM.\n- L163: If it is over-represented in dataset, it is needless to say a member. The statement here feels vacuous.\n- L164: Please also cite appropriate work that introduces this formal dependency on reference models for difficulty calibration [4].\n- L168: Missing citation\n- The dot on top of $\\theta$ in Equation 5 (and other related equations) is barely visible- please use better notation.\n- L182: Missing citation\n- L229: Why is a negative-definite Hessian problematic? In practice, the (empirical) Hessian is very low-rank and most of its entries are close to zero.\n- L235: \"..can be interpreted as kind of\" - this is very handwavy\n- Table 1: Please also include TPR@FPRs (like 1% and 0.1% FPR). The table is also missing a lot of attacks like Min-K%++ [5] and MoPE [6]\n- Figure 3 Why is Y axis going above AUC = 1.0?\n\n\n### References\n- [1] Galichin, Andrey V., et al. \"GLiRA: Black-Box Membership Inference Attack via Knowledge Distillation.\" arXiv preprint arXiv:2405.07562 (2024).\n- [2] Hartmann, Valentin, et al. \"Sok: Memorization in general-purpose large language models.\" arXiv preprint arXiv:2310.18362 (2023).\n- [3] Carlini, Nicholas, et al. \"Membership inference attacks from first principles.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n- [4] Sablayrolles, Alexandre, et al. \"White-box vs black-box: Bayes optimal strategies for membership inference.\" International Conference on Machine Learning. PMLR, 2019.\n- [5] Zhang, Jingyang, et al. \"Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models.\" arXiv preprint arXiv:2404.02936 (2024).\n- [6] Li, Marvin, et al. \"Mope: Model perturbation-based privacy attacks on language models.\" arXiv preprint arXiv:2310.14369 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer i2LA,\n\nWe deeply appreciate the time you took to review our work and your meticulous comments for improvement. Below we address the questions raised in your review, the responses to the weaknesses and minor comments, as well as the reference can be found in subsequent comments:\n\n**Q1: In the codebase, is the prompt data sampled directly from the fine-tuning data? If this is indeed true, this is a big empirical flaw.**\nWe want to clarify that we did not directly extract prompt data from the fine-tuning data, this is simply a part of the code that is easily misunderstood. Specifically, the `train_dataset` variable in line 65 of the `ft_llms/refer_data_generate.py` file is not the actual training set used to fine-tune the target model. We chose to extract prompt data from the `train_dataset` variable because, in some datasets, the number of samples in the `valid_dataset` is small and may lead to out-of-range errors. In fact, when preparing the fine-tuning dataset for the target model, we will manually skip the 10k-20k range: see line 226 in the `ft_llms/llms_finetune.py` file: \n\n`226: train_dataset = Dataset.from_dict(train_dataset[args.train_sta_idx:args.train_end_idx])`\n\nMoreover, we have experimentally demonstrated that the source of the prompt data has almost no impact on the performance of our proposed method (In Section 5.4, Fig. 4). Even prompt data from arbitrary unrelated sources can achieve excellent performance.\n\nFurthermore, we are committed to fully addressing your concern by adhering to your guidance to plot the distributions of metric score $\\Delta m(x)$ (in the PDF file of the global comment, Figure 2), for the training data (member) and validation data (non-member). The results clearly demonstrate that $\\Delta m(x) = m(x) - \\widetilde{m}(x)$ is not approximate to zero for the training data. Additionally, the training data metric score is generally higher than the validation data, indicating that the reference model would behave very differently from the target model regarding the target model's training data. All these results demonstrate that we did not directly extract prompt data from the fine-tuning data.\n\n\n**Q2: Why is there no mention of any studies related to membership inference for LLMs in Section 2 (Large Language Models)? There have been plenty specifically looking at LLMs [1,2,3,4,5]**\n\nWe have reviewed the current membership inference attacks on LLMs in Section 2 (Membership Inference Attack) and therefore did not replicate this content in Section 2 (Large Language Models). Moreover, since most of the literature [1,2,3,5] you recommended was submitted to arXiv in June this year (after the NeurIPS deadline), we are not able to discuss these papers in our submitted version. Thank you for the reminder, we will add a comprehensive discussion of the literature you recommended in Section 2 (Membership Inference Attack).\n\n\n\n**Q3: L147: \"adversary has no prior information\" - Does it have access to the model's initial weights?**\n\n> Explanation for L147: \"adversary has no prior information about which data records are utilized for fine-tuning.\"\n\nOn line 147, we emphasize that our method does not have prior information about the fine-tuning dataset of the target model. Existing reference-based methods heavily rely on the strong assumption that the adversary can obtain a dataset that is from the same distribution but does not intersect with the fine-tuning dataset. Our proposed self-prompt method avoids this assumption.\n\n> Answer for the question: Does it have access to the model's initial weights?\n\nThe proposed method does not necessarily require the initial parameters of the model. An API that allows the adversary to upload a customized dataset for fine-tuning would also meet the conditions. Therefore, our proposed method is not only applicable to open-source models with publicly available parameters but also for closed-source models that provide fine-tuning APIs (such as ChatGPT [6]). Thank you for your question for improvement, we will include this discussion in the camera-ready version.\n\nFurthermore, to further allay your concerns, we consider a more strict scenario where there is no access (even the fine-tuning API) to the pre-trained model corresponding to the target model. We employ LLaMA-7 fine-tuned over the Agnew dataset as the target model, and let the adversary fine-tune over different pre-trained models (GPT-J, Falcon and LLaMA are adopted) to prepare the reference model. The results are shown in the table below. From the experimental results in the table, we can find that the proposed method still maintains a large performance margin over the baselines that can fine-tune the pre-trained model of the target model. We will include these results in the camera-ready version.\n\n|Metric|AUC|TPR@1%FPR|TPR@0.1\\%FPR|\n|----------------------|-----|---------|-----------|\n|SPV-MIA (GPT-J)|0.832|19.3\\%|7.5\\%|\n|SPV-MIA (Falcon)|0.812|21.7\\%|8.7\\%|\n|SPV-MIA |0.903|39.5\\%|23.5\\%|\n|LiRA-Base |0.657|8.7\\%|1.4\\%|\n|LiRA-Candidate |0.714|11.5\\%|1.9\\%|\n\n**Q4: Figure 5b: Why is there a drop in performance for the domain-specific case?**\n\nThis phenomenon occurs because the performance of reference-based MIA is highly dependent on the quality (source) of the reference dataset (in Section 5.3, Figure 3). Although a domain-specific dataset pertains to the same domain as the target model's training set, there can still be significant distributional differences, even if these differences are smaller compared to an irrelevant dataset. Moreover, we have conducted experiments that demonstrate the quality of the domain-specific dataset is relatively low (in Section 5.3, Figure 3). Therefore, using excessively long prompt text lengths will cause the reference dataset extracted from the target model via the self-prompt method to be more similar in data distribution to the domain-specific dataset (low quality) rather than the target model's training dataset (high quality).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have attached the experimental results added according to the reviewers' requirements in the attached PDF file, which includes the following contents.

1. Table1: Evaluation of all baselines and SPV-MIA using AUC scores.
2. Table 2: Evaluation of all baselines and SPV-MIA using TPR@1%FPR.
3. Table 3: Evaluation of all baselines and SPV-MIA using TPR@0.1%FPR.
4. Figure 1: Full log-scale ROC curves of SPV-MIA and the three representative baselines on LLaMAs fine-tuned over three datasets.
5. Figure 2: The distributions of member and non-member records w.r.t MIA metric score ∆m(x).

Meta-review: This work proposes a novel usage of paraphrasing for improving membership inference attacks by using paraphrasing for generating reference datasets and for improving the membership signal using a style of neighborhood querying.

Overall there is consensus that this paper meets the quality threshold for Neurips. The reviewers agree this paper uses interesting ideas to create a novel attack that has clear and substantial improvements over prior work. However, there are some minor issues in that this work shares some ideas from prior work, could have a clearer evaluation with some more baselines, larger datasets, better metrics, and additional ablations. Some of these were addressed in the rebuttal.
Decision: Accept (poster)