LLM Dataset Inference: Did you train on my dataset?
Authors: Pratyush Maini
Hengrui Jia
Nicolas Papernot
Adam Dziedzic
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
dataset inference
membership inference
copyright
Primary area: safety_in_machine_learning
TL;DR: We detect if a given text was used to train an LLM using the dataset inference technique.

Abstract: The proliferation of large language models (LLMs) in the real world has come with a rise in copyright cases against companies for training their models on unlicensed data from the internet. Recent works have presented methods to identify if individual text sequences were members of the model's training data, known as membership inference attacks (MIAs). 
We demonstrate that the apparent success of these MIAs is confounded by selecting non-members (text sequences not used for training) belonging to a different distribution from the members (e.g., temporally shifted recent Wikipedia articles compared with ones used to train the model). This distribution shift makes membership inference appear successful. 
However, most MIA methods perform no better than random guessing when discriminating between members and non-members from the same distribution (e.g., in this case, the same period of time).
Even when MIAs work, we find that different MIAs succeed at inferring membership of samples from different distributions.
Instead, we propose a new dataset inference method to accurately identify the datasets used to train large language models. This paradigm sits realistically in the modern-day copyright landscape, where authors claim that an LLM is trained over multiple documents (such as a book) written by them, rather than one particular paragraph.
While dataset inference shares many of the challenges of membership inference, we solve it by selectively combining the MIAs that provide positive signal for a given distribution, and aggregating them to perform a statistical test on a given dataset. Our approach successfully distinguishes the train and test sets of different subsets of the Pile with statistically significant p-values $< 0.1$, without any false positives.

LLM Dataset Inference: Did you train on my dataset?
Pratyush Maini
Hengrui Jia 3,4 33,
Nicolas Papernot Adam Dziedzic
Carmeeie Mellon University
"DatologyAl Unniversity of Toronto *CISPA Helmholtz Center for Information Security
Vector Institute
Abstract
The proliferation of large language models (LLMs) in the real world has come with a rise in copyright cases against companies for training their models on unlicensed data from the internet. Recent works have presented methods to identify if individual text sequences were members of the model's 's training data, known as membership inference attacks (MIAs). We demonstrate that the apparent success of these MIAs is confounded by selecting non -emberss (text sequences not used for training) belonging to a different distribution from the members (e.g., temporally shifted recent Wikipedia articles compared with ones used to train the model). This distribution shift makes membership inference appear successful. However, most MIA methods perform no better than random guessing when discriminating between members and non members from the same distribution (e.g.. in this case, the same period of time). Even when MIAs work, we find that different MIAs succeed at inferring membership of samples from different distributions. Instead, we propose a new datasel inference method to accurately identify the datasets used to train large language models. This paradigm sits realistically in the modern-day copyright landscape, where authors claim that LLM is trained over multiple documents (such as a book) written by them, rather than one particular paragraph. While dataset inference shares many of the challenges of membership inference, we solve by selectively combining the MIAS that provide positive signal for a given distribution, and aggregating them to perform statistical test a given dataset. Our approach successfully distinguishes the train and test sets of different subsets of the Pile with statistically significant p-values < 0.1, without any false positives.
1 Introduction
Training of large language models (LLMs) on large scrapes of the web [Gem, Ope] has recently raised significant privacy concerns [Rahman and Santacana, 2023, Wu al., 2023]. The inclusion of personally identifiable information (PII) and copyrighted material in the training corpora has led legal challenges, notably the lawsuit between The New York Times and OpenAl [Gry, 2023], among others [Bak, 2023, Sil, 2023]. Such cases highlight the issue of using copyrighted content without attribution and/or license. Potentially, they undermine the rights of creators and disincentivize future artistic endeavors due to the lack of monetary compensation for works freely accessible online. This backdrop sets the stage for the technical challenge of identifying training data within machine learning models [Maini et al., 2021, Shokri et al., 2017J. Despite legal ambiguities, the task holds critical importance for understanding LLMs' operations and ensuring data accountability.
Membership inference [Shokri et al., 2017] is long-studied privacy problem, intending to infer if a given data point was included in the training data of model. However, identifying example member- ship is challenging task even for models trained on small datasets Carlini et al. [2022], Duan et al. [2023a,b), and Maini et al. [2021] presented an impossibility result suggesting that as the size of
'Equal contribution. Code is available
dataset inference/.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Stage Vietim approaches with Suspect
Stage I: Aggregute Features with MIAs
Claim:
I. M trsincal on Sungort set
Yal het Spara
Assumption:
Spils
Attack
Suspert Faunure (A]
Val Fasture (X)
Suspect set and Vall are Val set is peivase to the victin
Sptt B
Splli
Stage 2: Learn Correlations
Stage 3: Perform Dataset Inference
Train Linear Model
(2
Suepuct
Val
T-Test
Foanurea
Fratance (A)
SpRiB
(L9. 08.. 07
+Attuck
Figure 1: LLM Dataset Inference. Stage 0: Victim approaches LLM provider. The victim's data consists of the suspect and validation (Val) sets. A victim claims that the suspect set of data points was potentially used train the LLM. The validation set is private to the victim, such unpublished data (e.g., drafts of articles, blog posts, or books) from the same distribution as the suspect set. Both sets are divided into overlapppi splits (partitions) A and B. Stage I: Aggregate Features with MIAs. The A splits from suspect and validation sets are passed through the LLM to obtain their features, which are scores generated from various MIAs for LLMs. Stage 2: Learn Correlations (between, features and their membership status). We train a linear model using the extracted features to assign label O (denoting potential members of the LLM) the suspect and label 1 (representing non-members) to the validation features. The goal is to identify useful MIAs. Stage 3: Perform Dataset Inference. We use the B splits of the suspect and validation sets, (i) perform MIAs on them for the suspect LLM to obtain features, (ii) then obtain an aggregated confidence score using the previously trained linear model, and (iii) apply a statistical T-Test on the obtained scores. For the suspect data points that are members, their confidence scores are significantly closer to 0 than for the non-members.
the training set increases, the success of membership inference degrades to random chance. Is testing the membership of individual sentences for LLMs trained for a single epoch on trillions of takens of text data feasible? In our work, we first demonstrate that previous claims of successful membership inference for individual text sequences in LLMs [Mattern et al., 2023, Shi et al., 2024] are overly optimistic (Section 4). Our evaluation of the MIA methods for LLMs reveals a crucial confounder: they detect (temporal) distribution shifts rather than the membership of data points (as also concur- rently observed by [Duan et al., 2024]). Specifically, we find that these MIAs infer whether an LLM was trained on a concept rather than an individual sentence. Even when the outputs of such MIAs (weakly) correlate with actual sentence membership, we find that they remain very brittle across sen- tences from different data distributions, and no single MIA succeeds across all. Based on our exper- iments, we conclude the discussion of MIAs with guidelines for future researchers to conduct robust experiments, highlighting the importance of using IID splits (between members and non-members), considering various data distributions, and evaluating false positives to mitigate confounding factors.
If membership inference attacks are so brittle, do content writers and private individuals have no recourse to claim that LLM providers unfairly trained on their data? As an alternative to membership inference, we advocate for a shift focus towards dataset inference [Maini al., 2021], which is a statistically grounded method to detect if given dataset was in the training set of a model. We propose a new dataset inference method for LLMs that aims at detecting sets of text sequences by specific authors, thereby offering more viable approach to dataset attribution than membership inference. Our method is presented in Figure 1. The motivation behind dataset inference stems from the observation that in the rapidly evolving discourse on copyright, individual data points have much less agency than sets of data points attributed to a particular creator; and the fact that more often than not, cases of unfair emerge in scenarios when multiple such sequences their clusters naturally occur. For instance, consider the Harry Potter series written by J.K. Rowling. Dataset inference tests whether a 'dataset" collection of paragraphs from her books was used for training language model, rather than testing the membership of individual sentences alone. We also outline the specific framework required to operationalize dataset inference, including the necessary assumptions for the same.
We carry out our analysis of dataset inference using LLMs with known training and validation data. Specifically, we leverage the Pythia suite of models Biderman et al. [2023] trained on the Pile dataset Gao et al. [2020) (Section 5). This controlled experimental setup allows us to precisely analyze the model behavior members and non-menine members when they occur IID (without any temporal shift) as

---

the training and validation splits of PILE are publicly accessible. Across all subsets, dataset inference achieves p-values less than 0.1 in distinguishing between training and validation splits. At the same time, our method shows no false positives, with our statistical test producing p-values larger than 0.5 in all cases when comparing two subsets of validation data. To its practical merit, dataset inference requires only 1000 text sequences to detect whether a given suspect dataset was used to train an LLM.
2 Background and Baselines
Membership Inference. (MI) [Shokri et al., 2017]. The central question is: Given a trained model and a particular data point, can we determine if the data point was in the model's training set? Applications of MI methods span across detecting contamination in benchmark datasets [Oren al.. 2024, Shi al., 2024], auditing privacy [Steinke et al., 2023], and identifying copyrighted texts within pre-training data [Shafran et al., 2021]. 'he field has been studied extensively in the realm of MI. models trained via supervised learning on small datasets. The ability of membership inference in the context of large-scale language models (LLMs) remains an open problem. Recently, new methods [Mattern et al., 2023, Shi et al., 2024] have been proposed to close the gap and we present them in $ 2.1.
Dataset Inference. [Maini et al., 2021] provides strong statistical claim that given model is derivative of its own private training data. The key intuition behind the original method proposed for supervised learning that classifiers maximize the distance of training examples from the model's decision boundaries, while the test examples are closer to the decision boundaries since they have no impact on the model weights. Subsequently, dataset inference was extended from supervised learning to the self-supprvised learning (SSI.) models [Dziedzic et al., 2022] based on the observation that representations of the training data points induce significantly different distribution than the representation of the test data points. We introduce dataset inference for large language models to detect datasets used for training.
2.1 Metrics for LLM Membership Inference
This section explores various metrics used to assess Membership Inference Attacks (MIAs) against LLMs. We study MIAs under gray-box access (which assumes access the model loss, but not to parameters or gradients). The adversary aims to learn an attack function As. X -> {0, 1] that takes an input xr from distribution X' and determines whether x was in the training set Dirain of the LM fo or not. Let us now describe the MIAS we use in our work.
Thresholding Based. These MIAs leverage loss [Yeom al., 2018] or perplexity [Carlini et al., 2021] as scores and then threshold them to classify samples as members or non-members. Specifically, the decision rule for membership is: A f (1)) 11 (f()a.xr) < <>), where 7 is selected pre-defined threshold. However, MIAs based solely on perplexity suffer from many false positives, where simple and predictable sequences that never occur the training set can be labeled as members.
MIN-K % PROB. As a remedy to the problem of predictability, Shi et al. [2024] proposed the MIN-K% PROB metric which evaluates the likelihood of the K% of tokens in that have the lowest probability, conditioned on the preceding tokens. Hence, this MIA ignores highly predictable tokens in the suspect sequence. The membership prediction is made by thresholding the average negative log-likelihood of these tokens. The input sentence & is marked as included in pretraining data simply by thresholding the MIN-K% PROB result: A (2:) 1 MIN-K% PROB(z)
< 7).
Perturbation Based. The central hypothesis behind Perturbation-bast MIAs is that a sample that an LLM saw during training should have a lower perplexity its original version (x), as opposed to a perturbed version of the same (Ā). Formally, the membership attack is defined as < 7]]. for a threshold 7. In our work, we investigate various forms of
A (z) 1 (at) perturbations such as (1) white-space perturbation, (2) synonym substitution [Mattern et al., 2023], (3) char typos, (4) random deletion, and (5) changing character case.
DetectGPT. This is special case of perturbation based MIAs, originally used to detect machine- generated text [Mitchell et al., 2023]. The key difference is that perturbations to the input are made using an external language model that infills randomly masked-out spans of the original input. It then compares the probbabbbyy of with expected value of the same from multiple infilled neighbors ā,.
3

---

Reference Model Based. These methods compare the perplexity ratio between a suspect model and reference model on a given string. The suspect model may have seen the string during training,
while the reference model has not. The corresponding MIA is: A (x) = [f) < where 5% the reference model. In our work, we use the SIL. [Min et al., 2023], Tinystories- 33M [Eldan and Li, 2023], Tinystories- IM [Eldan and Li, 2023), and Phi-1.5 [Li et al., 2023] models as reference models. Notably, these models were not trained on general web data. In particular, the Phi-1.5 and Tinystories models were trained on synthetic data generated by GPT models, and the SILO model was trained on data that is freely licensed for training.
x)),
zlib Ratio. Another simple MI baseline uses the zlib library [Gailly and Adler, 2004], where a potential member has a low ratio of the model's perplexity to the entropy of the text, which is computed as the number of bits for the sequence when compressed with the zlib library: A (2) = [Pr (*)/atib(z) < <>] [Carlini et al., 2021]. The idea is that a model trained on dataset will have low perplexity for its members because it was optimized for them, unlike the zL.b algorithm, which was not tailored to the training data.
3 Problem Setup
LLMs train on trillions of tokens, and the sizes of the training sets are only likely to increase [Met, dbr]. To increase training efficiency (in terms of time, financial costs, and environmental impact), improve performance, and decrease the risk of privacy leakage, many LLM practitioners deduplicate their pre-training data [Biderman et al., 2023, Carlini al., 2021, Lee et al., 2022]. In our work, we ask this question: How to detect if a given dataset was used to train an LLM? and propose the idea of dataset inference for LLMs.
Access Levels. In the black-box setting, we assume an input-output access to an LLM along with access to model loss, hence we are not allowed to inspect individual weights or hidden states (e.g., attention layer parameters) the language model. This threat model is realistic in the case of LLM's users since many language models can be accessed through APIs that provide limited visibility into their inner workings. For instance, OpenAI [Ope] offers API access to GPT-3 and GPT-4, while Google [Gem] offers Gemini, without revealing the full architecture of the models or training methodology. The gray-bex access, commonly assumed for MIAs, additionally assumes that can obtain the perplexity or the loss values from an LLM, however, additional information such as model weights or gradients. In the white-box access, we assume full access to the model, where we can inspect model weights.
Operationalizing Dataset Inference. Dataset inference for LLMs serves as detection method for data used to train an LLM. We consider the following three actors during a dispute:
1. Victim (V)). We consider victim creator whose private or copyrighted content was used to train an LLM without explicit consent. The actor is presumed to have only black-box access to the suspect model, which limits their ability to evaluate if their dataset was used in the LLM's training process.
2. Suspect (AA) The suspect (or potential adversary in this case) is an LLM provider who may have potentially trained their model on the victim's 's proprietary, or private data.
3. Arbiter. We assume the presence of an arbiter, i.e., a trustet party, such as law enforcement, that executes the dataset inference procedure. The arbiter can obtain gray-box access to the suspect LLM. For instance, in scenarios when API providers only give black-box access to users, legal arbiters may have access to model loss to perform MIAs.
Consider a scenario where a book writer discovers that their publicly available but copyrighted manuscripts have been used without their consent to train an LLM. The writer, the victim y in this case, gathers a small set of text sequences (say 100) from their manuscripts that they believe the model was trained on. The suspect A in this scenario is the LLM provider, who may have included the writer's 's published work in their training data without obtaining explicit permission. The provider is under suspicion of potentially infringing on the writer's manuscripts. An arbiter, such as a law enforcement agency, steps in to resolve the dispute. The arbiter obtains gray-box access to the suspect LLM, allowing them to execute our dataset inference procedure and resolve the dispute. By performing dataset inference (as depicted in Figure 1), the arbiter determines whether the writer's published manuscripts were used in the training of the I.IM. This process highlights the practical application and significance of dataset inference in safeguarding the rights of artists.
Scenario.

---

Notation. We consider z to be an input sentence with N tokens = 22, .... ZN and fo is Language Model (LM) with parameters 9. We can compute the probability of an arbitrary sequence fo( ),), and obtain next-token In+1 predictions. For simplicity, assume that the next token is sampled under greedy decoding, as the next token with the highest probability given the first n tokens.
4 Failure of Membership Inference
We demonstrate that the challenge of successfully performing membership inference for large language models (LLMs) remains unresolved. This problem is inherently difficult because LLMs are typically trained for a single epoch on trillions of tokens of web data. In their work, Maini et al. [2021] demonstrated a near impossibility result (Theorem 2), suggesting that the size of the training set increases, the success rate of any MIA approaches 0.5 (as good as a coin flip). While this was shown in simplified theoretical model, we assess how this holds up for contemporary LLMS with billions of parameters. As a demonstrative example, we consider the most recent (and supposedly best performing) work that proposed the MIN-K K % PROB [Shi et al., 2024] membership inference attack, alongside a dedicated dataset to facilitate future evaluations. In their work, they show that this method performs notably better than other MIAs such as perplexity thresholding and PetectGPT that they benchmark their work against.
Temporal Shift and the Need for IID Analysis. The evaluation dataset used to showcase the success of MIN-K% PROB was the WikiMIA dataset, a dataset constructed using spans of Wikipedia articles written before (train set) and after the cut-off year 2023 (validation sct). This was chosen considering the training of the Pythia models [Biderman et al., 2023), which was based on scrapes of Wikipedia before 2023. Note that such an evaluation setup naturally has the potential confounder of a temporal shift in the concepts in data before and after 2023. Any article written after 2023 was naturally non-member of the Pythia models, and those written before 2023 were considered members. However, with changing times, we also encounter temporal shifts in writing styles and concepts in the Wikipedia dataset. This raises concerns if membership tests using WikiMIA actually assess membership of a particular data point, or of that concept/style. A similar question was concurrently asked by Duan al. [2024], who independently showed that MIAs are only successful because of the temporal shift in such datasets.
To critically assess the robustness of the MIN-K% PROB method, we conducted an exploration using the Pythia models and their (original) train and validation splits that come from the PILE [Gao et al., 2020] dataset, provided by the authors during pre-training. This facilitates confounder-free capabilite evaluation of the capabilities of membership inference attacks. In particular, the PILE dataset has more than 20 different domain-specific subsets with their own training and validation splits, such as Arxiy, Wikipedia, on the PILE were (Figure 2a):
Webtext to name a few. Some of the key observations from our experiments
1. Contrary to performance on the WikiMIA dataset where MIN-K% PROB metric achieved an AUC close to 0.7, the method got an AUC close 0.5 when tested on III train and validation splits of Wikipedia from the PILE dataset, hinting at a performance akin to random guessing.
2. We found that the method shows very high variance in AUC between different random subsets of the training and validation sets of the PILE dataset, oscillating between 0.4 and 0.7.
3. Results Arxiv and OpenWehText2 subsets of the PILE show AUC values near 0.4, suggesting that MIN-K% % PROB suffers from false positives, labeling validation set examples as members.
False Positive Assessment by Reversing Train and Val Sets. Do membership inference altacks actually lest membership? To answer this question, we do the following modification the WikiMIA setup: For every sentence in the pre-2023 subset of Wikipedia, we replace it with a sentence from the validation set for Wikipedia as given in the PILE dataset. We keep the 2023 Wikipedia subset as it is. On the one hand, since Pythia models were trained before 2023, it is clear that they never trained on data on Wikipedia from pages written after 2023. On the other hand, we also know that the validation set of the PILE dataset was not trained on and was also deduplicated from the train set. We now perform the same membership inference test on these two data splits Wikipedia Val (as the now designated 'suspect" set) versus Wikipedia post-2023 (as the supposed unseen set). Remarkably, the method demonstrates an extremely high AUC of 0.7 in labeling examples from the suspect (validation set) as members of the training set (Figure 2b). This confirms that these membership inference attacks (such as MIN-K % PROB) only distinguish between concepts across different temporal phases rather than verifying specific data membership, which they were originally designed for.
5

---

0.6
1.00
Min-K Prob
0.75-
Reversed Train/Val
0.5
AUC
0.50
Model Parametens
0.25
0.4
410m
--- 1.46
6.9b
0.00
20
40
60
5.0
10.0
20.0
30.0
40.0
50.0
60.0
K
K
(a) Performance for different model sizes.
(b) False Positives when reversing train/val sets.
Figure 2: Comparative analysis of the MIN-K % PROB [Shi et al., 2024]. We measure the performance (a) across different model sizes and (b) the observed reversal effect. The method performs close to random guess on non-members from the Pile validation sets.
Max-10% Prob -0.49 0.43 0.45 0.49 0.47 0.44 0.49 0.47 0.45 0.38 0.57 0.46 0.35 0.44 0.65 0.46 0.62 0.47
1,00
Min-10% -0488 0.45 0.51 0.50 0.50
0.51
0:42
0.75
Moeric
Perplexity -0.54 0.57 0.53 0.51 0,50 0,50 0.54 0.48
0.52 0.50
0.54
0.59 0.48 0.53 0.63
0.55
0.30 0.47
0:40 0.53
0.50
Parturbatios-baeed -0.53 0.55 0.52 0.53 0.48 0.59
0.48 0.50 0.56
0.52 0.52
0.48 0:47 0.48 0.55 0.51
Reforence- 0.52
0.49
0.25
Ziib Ratio -0 0.48 0.56 0.52 0:47 0.51 0.50 0.51 0.49 0.51 0.48 0.57 0.69 0.56 0.54 0.60 0.50 0.43 0.50 0:45 0.55
0.000
Book
Books
Earo
Free aw
Gubsa Cutembarg HackerNews
Malh
OWT2
PpenSia Sud
Philfag spers Pabbed Abs. Puil Con.
Stack
USPro
Chunto
Wiki STbbs
acliv
Dataser
Figure 3: Performance of various MIAS on different subsets of the Pile dataset. We report different MIAs based on the best performing ones across various categories like reference based, and perturbation based methods (Section 2.1). An effective MIA must have AUC much greater than 0.5. Few methods meet this criterion for specific datasets, but the success is not consistent across datasets.
No single MIA works across distributions. Now, we further expand our experimentation across multiple different membership inference attacks outlined in Section 2.1, across 20 different subsets of the PILE dataset. The goal is to analyze if there is any MIA that consistently performs well across all such distributions. In Figure 3, we show a heatmap of the performance of various (selected) MIA methods across different distributions of the PILE (refer to Appendix C for full results). While some MIAs perform well and achieve high AUC (such as synonym substitution on PhilPapers), the same methods have AUC of less than 0.5 the next dataset of Pubmed Abstracts. These results consolidate the finding that no single MIA for LLMs works across all datasets, and we need to potentially find methods that adapt the choice of metric to the distribution. In Section 5, we will leverage a (selective) combination of different MIAs to improve over the performance of any single MIA in order to perform successful LLM Dataset Inference.
Guidelines for Future Research, Based on our observations in this section, we outline four important practices for future research in membership inference to enable sound experimentation and inferences. In particular, (1) assessment for membership inference must be done in an IID setup where train and validation splits are from the same distribution, (2) experiments must be repeated over multiple random splits of the datasets, (3) experiments must be performed over multiple data distributions (4) careful experimentation must be done on both false positives and false negatives to ensure MIAs do not wrongly label non- non-members as members.
5
LLM Dataset Inference
Dataset inference builds on the idea of membership inference by leveraging distributional properties to determine if a model was trained on a particular dataset. While MIAs operate at the instance level- aiming identify whether each example was part of the training data. In the previous sections, we have shown that MIAs often yield signals that is close to random in determining example membership. However, if we achieve even slightly better than random accuracy inferring membership, we can aggregate these attacks across multiple examples to perform statistical test. This test can then distinguish between the distributions of the model's training and validation sets. In the context of LLM dataset inference, combine all the MIA methods discussed in Section 2.1.

---

5.1 Procedure for the LLM Dataset Inference
We describe the procedure for LLM dataset inference in four stages (see also visualization in Figure 1). Recall the initial example of book writer who suspects that portion of their books was trained on. We use this as a running example to describe the four stages of LLM dataset inference.
Stage O: Victim approaches an LLM provider. A victim (author) V approaches an arbiter with claim of ownership over data (book) that they suspect a model trainer or adversary S utilized. This stage involves the arbiter validating if the claim by V satisfies the assumptions under which dataset inference operates, that is, they provide an IID set of data that they suspect was trained on, and an equivalent dataset that S could not have seen, denoted as the validation set. This can, for instance, happen when authors have multiple drafts of a book chapter, and only one of the drafts makes it to the actual print. In such case, V claims that S trained on the published version of their book, because fe (S) responds differently to the final versus rejected drafts of the book. Both suspect and validation sets are divided into non-overlapping splits (partitions) A and B. We will use Ayal, Asus Stages 1, 2 and Byal, Bass in Stage 3 to actually perform the ownership assessment.
Stage 1: Aggregate Features with MIAs. This step involves aggregating scores from various MIA methods described in the previous Section 2.1. Aval. Aus sets are passed through the LLM under question obtain their features derived from MIAs. Note that we use an aggregation of all the MIA methods discussed in the previous sections to create single feature vector. In the next step, we will determine which MIAs are useful for identifying dataset membership for the given distribution.
Stage 2: Learn MIA correlations. In this stage, we train a linear regressor to learn the importance of weights for different MIA attacks to use for the final dataset inference procedure. Across each MIA feature value, we first modify the top 5% outliers by changing their values to the mean of the distribution. This step is crucial to prevent issues in Step 3, where the model might learn skewed correlations due to a few outlier samples. We then pass the data through a linear regression model to learn weights for each feature. All 'suspect' samples in Asus are labeled as O, and all validation samples in Aval are labeled as 1. A regressor is trained to predict the label given the samples, effectively learning the correlation between the features and their membership status.
Stage 3: Perform Dataset Inference. We use B splits of the suspect and validation sets, holding out up to 1000 samples in these splits for ownership assessment. Each sample is assigned a membership value using a trained linear classifier. These values are used to perform a statistical t-test to determine if the suspect set was used in training the model. We then report whether the model was trained on the suspect dataset based on the t-test results. For members of the suspect set, their confidence scores significantly closer to compared to non-members. The null hypothesis (H₂) is that the suspect dataset was not used for training. Assume that (M((,))) B...) and (M(B))) are the mean membership values of the suspect and validation sets, respectively. Then, Ho and Hi (alternate hypothesis) are:
Ho: (MM(B.)) 2 (M((wa)
H1 : MM(B..) S (M((a))
(1)
Combining p-values for Dependent Tests. To assess the significance of the results, we performed multiple t-tests using 10 different random seeds to obtain various splits of examples between A and B sets. Since the subsets had overlapping examples, the statistical tests are dependent [Vovk and Wang, 2020), and p-values must be aggregated accordingly [Brown, 1975, Kost and McDermott, 2002, Meng, 1994, Rüschendorf, 1982]. Based on the observation in Blakesley et al. [2009], the Sidak- method [Šidák, 1967] for combing p- p-values is typically conservative and helps avoid Type-I I errors. Let P1, P2....,,, Pr. denote the p-values obtained from the n t-tests, then the aggregated p-value:
Poombined = -1- - exp Σ log(1 p.)
(2)
Score Aggregation. To aggregate scores from different MIAs, we (i) normalize feature values to ensure that all features aggregated across various membership inference attacks are on a comparable scale. Then, we (ii) adjust values of outliers before learning correlations with the classifier by replacing the top and bottom 2.5% of outlier values with the mean of that (normalized) feature. Finally, we (iii) remove outliers before performing t-test in Stage 3 once we have a single membership value from the regressor outputs for each sample the B splits of the suspect and validation sets.
Once again we remove the top and bottom 2.5% of outlier.
7

---

posilive =165 <1e-
<1e3
<te-
<led
<1e3
<1e-3
=1e3
<te- <1e- 0.01
<1o3 <le3
<1e-3
<163
<1ed
<led
Palse posdive
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
n.88
1.00
1:00
1.00
1.00
1.00
0.98
1.00
1.00
1,00
0.58
Bookd
Bookss
Freat EanoPad
Galbab
Gurmbay
Hackariews
Math
OWT2
Opeastibe
Philf spers
Pabbdd Abs
PubhMad Con.
Stad
UsPro
Ubuma
Wiko
Y7Subs
atXiv
Dateser
Figure 4: p-values of dataset inference By applying dataset inference to Pythia- 1ithia-12b models with 1000 data points, we observe that we can correctly distinguish train and validation splits of the PILE with very low p-values (always below 0.1). Also, when considering false positives for comparing two validation subsets, we observe a p-value higher than 0.1 in all cases, indicating no false positives.
5.2 Assumptions for Dataset Inference
In order to operationalize dataset inference, we must obey certain assumptions on both the datasets (points I and 2 below), and the suspect language model (point 3 below).
1. The suspect train set and the unseen validation sets should be IID. This prevents the results from being confounded due to distribution shifts (such temporal shifts in the case of WikiMIA). We must ensure no leakage between the (suspected) train and (unseen) validation sets. The validation set should be strictly private, and only accessible to the victim.
3. We need access to the output loss of the suspect LLM order to perform various MIAs.
5.3 Experimental Details
Datasets and Architectures. We perform dataset inference experiments on all 20 subsets of the PILE. For experiments with false positives, we split the validation sets into two subsets of 500 examples each. In all other experiments, we compare 1000 examples of train and validation sets of the PILE [Gao et al., 2020J. We perform dataset inference on models from the Pythia [Biderman et al., 2023] family at 410M, 1.4B, 6.9B, and 12B scales. These open-source models allow us to know exactly which examples trained on.
MIAS used. In our experiments, we aggregate 52 different Membership Inference Attacks (MIAs) in Stage (many of which are overlapping and only differ in whether they capture the perplexity or the log-likelihood, or contrast the ratios or differences of model predictions). For the linear regression model trained Stage 2, we train for 1000 updates over the data using simple weights over the 52 features. A total of 1000 examples are saved for training the regressor to learn correlations for stage 2, except in the false positive experiments where we use half the data. A complete list of all the MIAs used in our work is present in Appendix C.
5.4 Analysis and Results with Dataset Inference
We analyze the performance of LLM dataset inference on the Pythia suite of models [Biderman et al., 2023] trained on the Pile dataset [Gao et al., 2020]. We separately perform dataset inference on each and every subset of the PILE using the provided train and validation sets, and report the p-values for correctly identifying the training dataset. Before diving into various design choices, the key result that dataset inference reliably finds the training distribution in all subsets of the PILE. (Figure 4). For the analysis of false positives, we carry out dataset inference on two splits of the validation set for each subset of the PILE. Since neither of the validation subsets was used to train the model fo. the returned p-values should be (and are) significantly above the selected threshold of 0.1 for any useful attribution framework. It worth noting that the p-values for these tests are often remarkably low in the order of 10-30 and lower, suggesting high confidence in attributing dataset ownership. When contrasted with the lack of reliability of membership inference, dataset inference indeed shows great promise for future discourse on inspecting training datasets. We will now dive deeper into various ablations and results around dataset inference such as the features (membership inference attacks) chosen by the regressor, choice of pre-processing function, change in performance of dataset inference with model size, data duplication and number of permitted queries.
Feature Selection. For each domain, we find that the most important metrics are different. Hence, the linear classifier is essential to appropriately determine the importance of each feature for deter- mining per example membership. based on the dataset statistics. We present the results for subset of met-
MIAS in Figure (all MIAs in Appendix Figure 8). For example, while the Perturbation- ric is necessary to be present for the CC datasct, it is not useful for the OWT2 dataset, which instead
8

---

Max-10% Prob- 00.01 0.04
-0.12
0.0) -0.01 02-00022240
0808
.0.03
0.01
0.12
0.10
0.20
0.00
0.01
-0.01
40.09
0.15
-0.08
0.04
0.68
Min-10% Proh 0.05 0.92
00,09
-009
0..4
0.06
0.05
0.02
0.01
0:03
0.00
-0005
0.01
0.04
0.07
0.5
Merric
Perplexity 0.11 0.30
0.09 0.37
00.22
0:01
9.63
-..0
Perturbation based 3.50 40000.0. 0.16 4.72-00
022-0221111 021--40 -0.18
0.18
0.05
0.42
0.59
0.26
0.12
0.23
0.51
46-0.1
Reference-based - 0.08 0.03 4.00
1..22
0017 .0.21 0.01 -0.05 40.01
10.15
40.21 0.39 0.02
1919
0.17
4.002 0.04 -0, 10.00-0000 10-0008
-0.5
Zlib Ratio 0.22 4,03 0.00 6.02 40.0000135 13 40.03 40.09 00.33 40.15 0.00 0.22 00:43 0.00 16.1 40.2 4.34 0.05 0.00 0.03
BookCa Books3
cc
EuroPad
FreeLay FreeLaw
Githab
Guienber
HackerNews
Math
OWT2 OpenSutes
Plul sperss PabMed Abs Meb Con.
Stnc
USPTO
Ubunu
Wiki T arXiv
Dataser
(a) Feature Selection. A positive value indicates a correlation of the feature with the validation set, while a
negative one denotes a correlation with the train set.
1.0
Pre-proessing
Both (Nomn.) |0000000000000000000000000000000
Correction T0t.0000000000000000000000000000
Correction (Norm.) 00000/00000000000000000000000000
Removal w0000000000000000000000 |tt_:x__-00000000000000000000000
Removal (Norm.) -0.00 0.000000000000000000
Book Books3
EuroP art
ec
Freel Law
Cirlubh Guteebbegg
HackerN HackerNews
Math
OWT OpenSuts
Phi pperss
PubMlod od Abs
Med Cen Cen
Stack
USPTO
Ubuntu
W Vij TSubs
BrXiv
Dataser
(b) Feature pre-processing, We present the p-values for given dataset using di differently pre- pre-processed features.
Both 1.000.05 1.00 1.000.905 1:00 1.00 .001 00001 0.0 1,00 1.00 1.00 10001.00.0000 1,00 1.001.11.00000000 0.000.
Pre-proossing
Both (Norn.)
1.000.041 11.0m.)-.000.04 12.04.1.00 1.000.100 1.00 1.00 1.001.00 1.00 1.000:99 1.00 1.00 1.000.99 1. 1.00 0.
1,00 1.00 1.001.00
Correction-
1.000.18 1.000.999..0000 0.96 1.001.000 1.00 1.00011000000 1.0000000 1.001001.00000 0.53 0.94
0.40
0.5
Correction (Nomn.)
0.000.20
0.0
1,00
0.00 1.00
0.03
Removal- 0.97 1 00 100
1,00
0.96 0 80 0o0 1.00 .00 1.00 1.00 1.00 ,00 1.00 1.00 0.000 96 0.99 1.00 0.0
Removal (Norm.) 100 0000000100 1.00 1.00 1.0 100 1.00 1,00 0.00 988 1.00 1,00 1 1,00 1.00 1.00 0.98 1.00 1.00 1.00
0.00
Book Books3
EuroP Free Law
Github Gutenberg HackerNow News
Math OWT2 OpenSato Phil Papers Patiled Abs. PuhMed Con.
Stack USPTO Uhumn
Wike TTSbbs
Dataser
(c) Analysis of false-positives. A p-value smaller than 0.1 indicates false positive.
Figure 5: Ablation study for dataset inference. We analyze which features based on or derived from the previous membership inference methods increase the success of dataset inference. (a) Our results indicate that no single feature contributes consistently, thus we need a linear model to selectively aggregate their impact on the final outputs from dataset inference. (b) Given the selected features, we consider different ways of how to pre-process them before building the classifier. The proposed method (denoted as Removal (Norm.)) removes outliers and normalizes the feature values. (c) We evaluate the selected and pre- prepprcessedd features using suspect set that come from the validation data. We do not observe any false positives as shown in the last row in (c).
requires the Perplexity metric. Dataset inference automatically Iearns which MIAS are positively correlated with a given distribution. The linear regressor can be trained quickly on a CPU since it is only learning a weight assignment for each feature. Now, we investigate which MIAs get selected by dataset inference by analyzing the importance weights given by our linear regression to various MIAs.
explore various pre-processing techniques to apply before training the linear regressor. The selected approach, referred to as Removal (Norm.) in Figure 5b, involves eliminating outliers and normalizing the feature values. We tried other approaches such as mean correction, and outlier clipping, but we found that these approaches make the dataset inference procedure less reliable by artificially modifying the score distributions and modifying the feature correlations learned by the linear model. Its effect can be seen through the occurrence of false positives for some of the datasets.
Feature Pre-Processing. Considering the chosen features,
Number of Queries. We analyze the number of queries that have to be executed against the tested model fo to determine if given dataset used for training. We present the results in Figure 6a. It can be seen more than half of datasets only require about 100 points, while 1000 points are sufficient to obtain p-values smaller than the significance threshold of 0.1 for all datasets.
Size of L.MSs and Training Set Deduplication. By studying the Pythia suite of models [Biderman et al., 2023] which are trained the same dataset, we observe the success of dataset inference is

---

1.0
0.1
Deduped
p-value
p-valuc
Non-Deduped
0.0001
Wiki
0.5
1e-07
muXoae
mediinostaset
Ic-10
0.0
10'
10"
10'
0.41
1.4
6.9
12.0
Number of Points
Model Size (Billons Parameters)
(a) Dataset inference success number of data points. (b) Dataset inference success v.s. LLM size.
Figure 6: Ablation studies for the amount of data and model size. In (a), we plot the maximum and median p-values across all datasets, alongside the p-value of Wikipedia, of data points. In (b), a violin plot is made to show the distribution of p-values of the datasets with respect to the number of model parameters. Observe that dataset inference is more successful with more data and larger LLMs. It is also noteworthy that (a) dataset inference for majority of datasets accurate with less than 100 points, and (b) more accurate with respect the non-deduplicated models that are trained on datasets with duplicated points. We hypothesize this is because the membership signal for most MIAs becomes stronger with the duplication of data. Deduped denotes a version of the Pile dataset where the documents are deduplicated within, and across the data splits (rrain/est/validatiin)) Non-Deduped is the original version of Pile without any deduplication.
function of the number
positively correlated with the number of parameters in the LLMs. We present this result in Figure 6b as a violin plot to allow for visualizing distributions of the datasets' p-values. It can be seen as the size of the model increases, the p-value distribution concentrates below the threshold of 0.1. This correlatio can be explained by the phenomenon that memorization by LLMs increases as their parameter size increases [Carlini et al., 2021]. which provides a stronger signal for the intermediate MIAs responsible for dataset inference to succeed. We also contrast the models trained on deduplicated or non-deduplicated training sets when we are only allowed 500 query points. Observe that while the aforementioned trend holds for both kinds of models, the p-value distribution is more concentrated below 0.1 for the non-deduplicated models. Following the same explanation as above, this also indicates that memorization is more severe when some training data is duplicated, allowing various membership inference attacks to have stronger signal.
Discussions
Membership Inference for LLMs. In this work, we question the central foundations of research on membership inference in the context LLMs trained on trillions of tokens of web data. Our findings indicate that current membership inference attacks for LLMs are good as random guessing. We demonstrate that past successes in MIAs are often due to specific experimental confounders rather than inherent vulnerabilities. We provide guidelines for future rescarchers to conduct robust experiments, emphasizing the use of IID splits, considering various data distributions, assessing false positives, and using multiple random seeds to avoid confounders.
Shift LLM Dataset Inference. Historically, membership inference focused on whether an individual data point was part of a training dataset. Instead, we aggregate multiple data points from individual entities, forming what we now consider dataset. our work, we have not only put thought towards the scientific framework of dataset inference but also the ways it will operationalize in real- world settings, for instance, through our running example of a writer who suspects that their books were trained on. Our research demonstrates that L.M dataset inference is effective in minimizing false positives and detecting even minute differences between training and test splits of IID samples.
Limitations. A central limitation to dataset inference is the assumptions under which it can be performed. More specifically, we require that the training and validation sets must be IID, and the validation set must be completely private to the victim. While this may appear elusive a priori, we outline concrete scenarios to show how these sets naturally occur. For instance, through multiple drafts of book, until one gets finalized. The same applies to many artistic and creative uses of LLMs across language and vision today. terms of data and model access, we assume that the victim or a trusted third party, such as law enforcement, is responsible for running the dataset inference so that there are no privacy-related concerns. This will require the necessary legal framework be brought in place, otherwise suspect adversaries may deny querying their model altogether.
10

---

7 Acknowledgements
We would like acknowledge our sponsors, who support research with financial and in-kind contributions: Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, Meta, NSERC through the Discovery Grant and an Alliance Grant with ServiceNow and DRDC, the Ontario Early Researcher Award, the Schmidt Sciences foundation through the A12050 Early Career Fellow program, and the Sloan Foundation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and companies sponsoring the Vector Institute. Pratyush Maini was supported by DARPA GARD Contract HR001 12020006. Adam Dziedzic was supported by the German Federal Ministry of Education and Research (BMBF) under funding number 16KIS21 4K and by the German Research Foundation (DFG) within the framework of the Weave Programme supporting the project on "Protecting Creativity: On the way to Safe Generative Models" under project number 545047250. Responsibility for the content of this publication lies with the authors.
References
Gemini,
URL
Introducing meta llama 3: The most capable openly
available Ilm to date,
http:://i.netaaccommmmmm/mtttttt URL https: ceatta.combllog. meeaa- 1aaaaa-1aaaa/////
Openai, https:/lppnaii.com. URL https://ppeaaa com/.
Introducing dbrx: A new state-of-the-ard open Ilm, http://www.dabbbrcc...oooooooonn
wtw---tttttttt----eeeet introducing dbrx-nese swwsssttttse-eee
URL
cettp:://www..attbbricc..s.k
state-art-open--------e- arte--r-to-pen-mmm 11m.
Getty images vs. stability ai: A landmark case in copyright and ai, 2023. URL https://wwr
bakerlaw.. cooon//etty-iagee-eeeee v-stsssitsstabbityaa....
The times sues openai and microsoft over a.i. use of copyrighted work
https://ww..yyimmes wwtp://///////////////////i/e//e http://wwwwayylioooomo/////////s law. 2023. URL
-imoo-----me----aaiiiii miierrosft- com http::/ww.....aaooooooo////////a new-york- times- open- ai wtt.mmmoooooot-aaiiiie.ee html.
Sarah silverman and authors sue openai and meta over copyright infrrgggeeettn 2023. URL https:// // www. www..myttines.com/222///000
zero-shot detection of http:////0/...t..aaaaaaaaaaaaaaa International Conference on Learning Representations, 2024. URL https: /ttttpp://ppervvvviii net/ forum?id-Bil forum/2d--BoocrrEEE.
pempeenii-mttaaa html.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectGPT: Efficient text via conditional probability curvature. In The Twelfth
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Richard E Blakesley, Sati Mazumdar, Mary Amanda Dew, Patricia R Houck, Gong Tang. Charles F Reynolds III, and Meryl Butters. Comparisons of methods for multiple hypothesis testing in neuropsychological research. Neuropsychology. 23(2):255-264, 2009. doi: 10.1037/a00128501
Morton B Brown. 400: A method for combining non- non-independent one-sided tests of significance. Biometrics, pages 987-992, 1975.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voor Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson. Alina Oprea, and Colin Raffel. Extract- ing training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650. USENIX Association, August 2021. ISBN 1-9913333 confconnferrere
24-3. URL. https: //www//ww/////m////// usenix. wwwp/oogg///eoeeeeeeeooeooeeeeee carlini- eeiiieextraaiing..

---

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914, 2022. doi: 10. 0..1000994621122222.96333
Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Ma- hamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, Jascha Sohl-Dickstein, Jinho D. Choi, Eduard Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade. Lisa Barthe, Hanna Behnke, Ian Berlot-Attwell. Connor Boyle, Caroline Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxi- ang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, Mayukh Das. Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Fkeinhhrr Marco Di Giovanni, Rishabh Gupta, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel- Canada, Antoine Honore, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seungjae Ryan Lee, Corey James Levinson, Hualou Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko, Vukosi Marivate. Gerard de Melo, Simon Meoni, Maxime Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Muennighoff, Timothy Sum Hon Mun, Kenton Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan Pfister, Richard Plant, Vinay Prabhu, Vasile Pais, Libo Qin, Shahab Raji. Pawan Ku- mar Rajpoot, Vikas Raunak, Roy Rinberg, Nicolas Roberts, Juan Diego Rodriguez, Claude Roux, Vasconcellos P. H. S., Ananya B. Sai, Robin M. Schmidt, Thomas Scialom. Tshephisho Sefara, Saqib N. Shamsi, Xudong Shen, Haoyue Shi, Yiwen Shi, Anna Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyank Soni, Taylor Sorensen, William Soto, Aman Srivastava, KV Aditya Srivatsa, Tony Sun, Mukund Varma T, A Tabassum, Fiona Anting Tan, Ryan Teehan, Mo Tiwari, Marie Tolkiehn, Athena Wang, Zijian Wang, Gloria Wang, Zijie J. Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyi Wu, Witold Wydmańíki, Tianbao Xie, Usama Yaseen, M. Yee, Jing Zhang, and Yue Zhang. Nl-augmenter: A framework for task- sensitive natural language augmentation, 2023.
Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. In Thirty-sevntth Conference on Neural Information Processing Systems (NeurlPS), 2023a.
Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, and Franziska Boenisch. On the privacy risk of in-context learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023b.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannanch Hajishirzi. Do membership inference attacks work on large language models? 2024.
Adam Dziedzic, Haonan Duan, Muhammad Ahmad Kaleem, Nikita Dhawan, Jonas Guan, Yannis Cattan, Franziska Boenisch, and Nicolas Papernot. Dataset inference for sulf-supervised models. In NeurlPS (Neural Information Processing Systems), 2022.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint urXiv:2305.0775) 2023.
Jean-loup Gailly and Mark Adler. zlib compression library. 2004. URL ac. ukcu///andle// /etedlee///////// 1810/3486
cam.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0m 100002, 2020.
James T Kost and Michael P McDermott. Combining dependent p-values. Statistics & Probability Letters, 6002):183-190, 2002.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v acl.ooo.....oonn....... URL https: h/ h/tp///aaaaaanto....aaaa. org/2022, 2022. acl acl-long. 577.
12

---

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. International Conference on Learning Representations, 2021. URL https: bettp:://openevttii
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg- Kirkpatrick. Membership inference attacks against language models via neighbour- hood comparison. In Anna Rogers, Jordan Boyd-Graher, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 11330-11343 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v URL https: /ettpp://aclaaii http/alaaannooooooooooooooo wrg.oog///2/ 2023. fffff.iiii windinggn---iie ac1.719.
Xiao-Li Meng. Posterior predictive p-values. The annals of statistics, 22(3): 1 (133):1142-116 142-1160, 1994.
Sewon Min, Suchin Gururangan, Eric Wallace, Hannaneh Hajishirzi, Noah Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in nonparametric datastore. arXiv preprint arXiv:2308.044:: 044430 2023.
Bric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: zero-shot machine- machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org. 2023.
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving
test contamination for black-box language models. In The Twelfth International Conference Learning Representations, 2024. URL
Noorjahan Rahman and Eduardo Santacana. Beyond fair use: Legal risk evaluation for training Ilms on copyrighted text. 2023. URL https: /ggel/a/////a////// genlaw org/ wwt//oorg//aaaoooeeeeoe Cate/CCmmerraaaa 25m .df.
Ludger Rüschendorf. Random variables with maximum sums. Advances in Applied Probability, 14 (3):623-633 632, 1982.
Avital Shafran, Shmuel Peleg, and Yedid Hoshen. Membership inference attacks are easier on difficult problems. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 14820-14829, October 2021.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https httpb:::///peeeeevieeeee net/ forum?id=2WVr33Muul
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP). pages 3-18, Los Alamitos, CA, USA, may 2017, IEEE Computer Society. doi: 10.1109/SP.007..411 URL https: 2017.41. .0110...4....
i/oootiieecc weweet...rrrr/e org/10. 1109/S1PPPPPP
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: // //operreiier //opperrviir..ne//ooooeeee
Vladimir Vovk and Ruodu Wang. Combining p-values via averaging. Biometrika, 107(4):791-808, 2020.
Xiaodong Wu, Ran Duan, and Jianbing Ni. Unveiling security, privacy, and ethical concerns of chatgpt. Journal of Information and Intelligence, 2023.
Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 8 IEEE 31 Computer Security Foundations Symposium (CSF). pages 268-282. Los Alamitos, CA, USA, jul 2018. IEEE Computer Society. doi: 10. 10.1 109/CSF. 2018.00027. URL https::/ddoo http:////dd.ieeeepppeeeeeeeeeeee ieeecomptterssoccet org/10 1109/CSF 2018. 00027.
Zbyněk Šidák. Rectangular confidence regions for the means of multivariate normal distribu- tions. Journal of the American Statistical Association, 626663)),,6-33, 1967. ISSN 01621459, 1537274X. URL http: / /www. jstor, org/ stt/s/aaebeeeee....////
13

---

A Broader Impact
The use of large amounts of text scrapes from the web train large language models (LLMs) has recently sparked significant privacy concerns. Including personally identifiable information (PII) and copyrighted material these datasets has resulted legal disputes, such as the lawsuit between The New York Times and OpenAl, among others. These disputes emphasize the problem of using copyrighted content without proper attribution, potentially infringing on creators' rights and discouraging future artistic work due the lack of financial compensation for publicly available content. This scenario underscores the technical challenge of identifying data used to train machine learning models. Despite the legal uncertainties, this task is essential for understanding LLM behavior and ensuring data privacy. In our work, we propose a new dataset inference method to accurately identify the datasets used to train large language models. Our LLM dataset inference method is statistically grounded and we thoroughly evaluate our approach.
B
Compute
To aggregate various membership inference attacks, required performing forward passes for both suspect and reference models. We need enough GPU memory to load model. To allow for large batch forward passes, utilized NVIDIA A6000 48GB machines for aggregating such metries. We used a total of 4 machines at any given point to speed up the aggregation of metrics.
C Additional Experiments
Setup. Note that for the perturbation- MIAS (e.g., [Mattern al., 2023]), we use the implemen- tation of the perturbations from the NLAugmenter library Dhole et al. [2023].
Performance of MIAs. We provide extended version of the performance of various MIAs different subsets of the Pile dataset in Figure 7. For an MIA to be effective, it must achieve an AUC significantly higher than 0.5. Only a few methods meet this standard for certain datasets, and their success is not consistent across different datasets. We note that, for example, the DetectGPT [Bao al., 2024] performs significantly better on few datasets, for example, bookcorpus2 and books3, than any other metric.
Feature Selection. We provide extended version of the importance of features used in dataset inference in Figure 8.
Additional ablation studies on model size, processing technique, and number of data points. We analyze the number of required data points for successful dataset inference, with processing techniques used in dataset inference and different sizes of LLMs, in Figure 9 and Figure 10 respectively. To visualize the distribution p-values impacted by the number of data points and size of the models, violin plots presented in Figure 11.
14

---

Anoiage Loss DXT
0.47
0.51
0.87
0.45
0.48
9.51
0.43
5,48
0.51
0.45
0.52
5,85
0.54
0.46
0.44
035
0.47
6.59
0.45
Avcrage Low Raiso 0.66
0.6
0.3
0.88
0.45
0.50
0.68
0.45
0.51
0.40
0.45
0.51
0.83
0.54
0.48
0.26
0.62
0.47
6.33
0.55
Avriag Nomaline es Der-
0.68
643
0.45
0.1
0.46
0.51
0.51
8:11
0:51
0.50
0.47
0.51
3.55
0:46
0.49
0.51
0.64
8:46
0.54
0.54
Avenge Nomalind loa Rria 0.47
841
0.45
0.30
0.41
0.51
0.28
8:50
0.55
0.48
0.45
0449
0.89
0.49
0.48
1440
0.23
8:46
6.51
18:60
Nestage NestagNuntdllng Nantelised Poplody DIE-
0.50
0.57
0.50
0.8
0.47
0.48
929
0.48
8:1
0.50
0.50
0.50
1851
0.52
0.50
0.50
9.45
0.53
6.47
0.52
Avcrngs Nocmilleed Ration 0.53
0.53
0339
0.51
0.55
0.53
0.5
0.00
0:53
0.50
0.56
0.50
9.59
0.00
0.54
0.51
9.47
0.52
6.55
0.54
Avange Poplodt DIE 0.49
046
0.49
0.49
0.55
0.53
0.68
0.49
0:53
0.48
0.52
0.45
1339
0.52
0.50
0.39
0365
0.46
0.54
0.55
Avtay: Reirn- 0.55
10:54
0.55
0.49
0.53
0.99
1000
0:52
0.57
0.49
0.44
0.54
0.72
0.55
0.46
0.54
B 0.55
603
0.50
0.99
6.71
0.51
0.55
0.49
DeseGPT00--
0.50
0.50
0.54
Los Dif Bunir Pirgrn
0.04
0.48
0.50
0.47
0.48
Cha
0.51
0.51
Lhel Syroev
0:41
Loss Duleseee Trkk-
0.51
0:50
0.47
0.46
Whiregyaa Pemecvatoon
0.50
0.51
Los Buns Fizgen
0:42
0.44
0.49
6:51
0.51
057
0.48
0.47
Los Rata Change the
0.41
048
0.45
0.30
0:46
6:51
0:45
0.31
0,99
0.65
0.49
0446
Loss Kais itanda Delsion
647
0.45
0.34
0.48
0.46
0.29
0.49
081
0.48
0.48
0.31
0.53
Loss Ratie Syromen Substbution 0.46
0:48
0.44
0.7
0.55
0:49
0.99
045
CAR
0.50
0.64
0.51
0.29
6.47
044
0.57
0:49
0.54
Lao Barin bidtmerre Trikk 0.47
B47
0.47
049
0:46
0.51
D-48
5:91
0:47
B:46
0.10
0,87
04%
0.48
6.35
0.22
0:48
0.54
Los Ruii Whilzsgace Penullation - 0.47
0.49
0.44
0.88
0.47
0.50
0,68
0.45
5:52
0.29
0.45
0.50
3,69
0.65
0,48
0.36
0.72
0.47
0.54
0.50
May- (0% Prub -.47
8:49
0.43
0445
0.48
0.49
0,67
0:46
0.49
0.45
144
335
0.7
0.46
0.25
0.65
0.46
5.45
0.52
May 20% Prob - 0.47
0:65
0.43
0.66
0.46
0.49
0,67
0.46
5,99
0.49
0.45
0446
0.59
0.7
0.46
0.36
0.65
0.46
0.40
0.51
Mac tor Prih - 0.47
D:6
0:41
0466
0:49
0.90
0.87
D:46
0.48
0:45
0449
0.54
0.56
0.46
0.36
0.67
0445
0.50
0.61
May- 4r% Prih 0.47
B44
0:41
044
0.48
0.50
0.88
B44
0:60
0.48
0:45
0449
0.54
0.5
0.47
0.35
0.61
0445
0.51
0.61
Max 5% Pick 0.47
0.49
0:43
0.55
0.47
0.45
9.45
0.64
0:48
0.50
0:45
0.46
0.39
0.5
0.47
0.34
3.64
0.43
6.42
0.62
Max 50% Prub 0.47
647
0:43
042
0.49
0.50
0.45
0:51
0.48
0.45
0.50
0.59
0.54
0.47
0.35
0.45
0.52
Max-d0% 60% Parb 0.47
047
0:43
047
0.46
0.50
0.48
045
0.51
0.48
0.45
0.50
0.43
0.55
0.47
0.35
0.70
0.45
6.52
Min 10% Prih 0:48
D44
0.46
0.49
0.10
0.90
0:51
0:12
0.49
D46
0:4
6.55
-36
Min-ai% Prih 0.48
0.41
0.51
0.11
0:54
0.52
0.50
046
0.47
Min 30% Prob
0.45
0.51
0:55
0.51
0.50
Prob-
0.50
0.12
0.1
0.52
031
Nomrelied Phil1
0.48
0:43
0.50
031
0.47
0.66
0.46
Normalied
13:9
6.46
0,41
0,89
036
6,7
0.50
0.4
6.47
0.54
052
0.50
Nouudiee Less Iapatsiis-
043
9.47
0.47
0.52
9.01
055
0:58
0.52
0.88
0.50
046
0/4
0.46
Nomalimdl no: Ddf
Sterakized II
0:43
0,47
0.51
0.32
0.22
0:57
0.32
051
5.49
0.60
0446
0.34
(une Ho-1
647
0.55
0:44
0.49
0.88
0.45
5.50
0:44
0.50
0.48
641
0:22
0.49
0.31
Nermalbedd Loss Rato Tepaadda-1M- - 0.48
0:44
0.45
0.50
0.45
0.51
051
5.55
0.68
0.66
0.50
0,89
0.50
0.48
0.29
0.23
0.46
0.54
Nomallzzd Loss Raio 31pp0aiiz-33MM
Netualiadd Losr: Silo
0.45
0.47
043
0:49
0.39
0,45
0.69
0.49
0.42
0.45
0.51
0.49
0,68
0,68
0.50
0.69
5.55
5,45
0.29
0.48
0.45
0.46
0.50
0.50
033
0,69
0.2
0.5
0,48
0.48
041
641
0.23
0.69
0.50
0.47
6.4:
0.54
0.50
0.55
-02
Normdized Perplesity 1iif (1ts 5)
0,50
0.50
0.51
11300
0.38
0.51
9.51
0.5
0.53
0.49
0.54
0.50
0.55
087
0.51
045
0.56
0.49
0.50
0.59
Nomalind Poplicitty kid (Sino)
0.30
B46
0.54
0.49
0.60
0.53
0.50
0.55
0.52
0.49
0.54
0.50
0.52
0.46
0.50
0.29
0022
0.46
6.39
0.51
Nemodlowt Teppeshl niff (Tiyyanniss 15D
0.50
0.5
0.50
0.50
0.48
0.49
0.50
0:49
0.49
0.50
0.50
0.90
0.5
0.50
0.50
0.65
0.55
0.44
0.53
Noandizal Peppiaity Diff (Tupliiic 33M0
0.51
0.57
0.52
0.49
0.45
0.48
0.43
0.47
0.83
0.50
0.45
0.34
0.50
0,50
041
0.52
0.52
6.47
0.51
Nottalized Raiios 15)
0.52
0.53
0.55
0.52
0.57
0.51
0.51
0.57
0.51
0.30
0.56
0,50
9,1
0.99
0.53
0.54
937
0.54
5.45
0.51
Nommalized Itaton (Sim)-
0.53
0.50
0.60
0.51
0.52
0.54
0.51
0.50
0.50
0.65
0.66
0.55
0.6
0.50
6.82
0.55
Nemalizd Raien (leoskencc- 1M0
0.50
0.57
0.52
0.49
0.55
0.49
0.49
047
6/42
0.50
0.52
0.85
0.22
0.55
0.50
0.50
0.65
0.55
6.45
0.45
Nomalize Reive (Thngstiii XTM)
0.50
0.53
0.50
0.50
0.50
0:11
0:41
0.90
0:52
D4K
0.50
044
0.50
0.50
Papladly
0.50
0.55
0.55
Peplosiyy (Blfe Bages)
0.68
0.52
0.52
0.50
0.66
Paplesity
0.51
Paplaaty Dil
0.53
6.55
Dorplocky DET diawrry
0.51
6.54
Pepiesity (Uxlenoone Tick)-
0.51
0:53
0.50
0.50
0,52
054
6.52
0.30
0.52
0.1
0.64
0.51
0.59
0.70
0.50
6.48
0.57
Beplodly DET WWitepocc Fanuricion)
0:54
0.56
0.51
0,4
0.2
551
0.88
0.51
0.48
0.49
0.4
0.59
0.10
0.48
0.57
Raim (Balee Stues)
0448
0.55
0.52
0.54
0.32
0,69
0.2
599
0.88
0.56
044
045
0.53
051
052
0.56
(Change Casch-
048
0.57
0.52
0:58
0.50
0.4
549
0,20
0.56
0.49
0.45
0.60
0.52
0.7
0.50
056
Botion (Bonkon Dekaion-
0:52
0:52
0.5
0:46
0.91
0:81
0:56
5.6
0.50
0.57
0:49
0.29
6:52
19.9
Ryinn (Sunoy Subocuuicoh- - 0.51
0.53
0:55
0.52
0.45
0.53
0,0
0.59
0.54
0.88
0.56
0.50
0.70
0.3
0.52
064
0.07
0.48
0.55
Ratir (Uhdeesco Trick- 0.52
0.53
0:57
0:1
0.53
0.49
0:53
0.55
0.50
0.51
0.55
0.51
0.57
0.60
0.55
0.64
0.00
054
Raisnn (Whttepsae Potuabiiin- - 0.53
0.50
0.57
0.1
0.57
0.50
0.50
0.56
5,49
0.40
0.55
0.49
0,45
0.58
0.52
0.50
0.63
0.54
5.65
Zib Bragy / Peplosit- - 0.55
0.46
0.55
0,52
0.47
0.51
0.50
0.51
0.45
0.51
0.57
0.45
0.36
0.54
0.50
0.43
0.50
0.50
books Tookomak
(oceer casppt
baketzeew
Hdnd. cmml
Figure 7: Extended version of Figure 3 with more types of membership inference attacks. The results are consistent: none of the membership inference attacks can consistently achieve high ROAUC across different datasets.
15

---

5% Prob 4000
0.7
+0.17
Max 10% Prob 4.00
0.53
6.00
0.3
-294
00.44
1.21
0.33
0.15
0.30
-3.05
0.36
0.01
0.65
0:34
0.56
5:24
48288
0.54
0.51
Max-20% Prob 0.05
0.64
4.06
0.03
0.19
0.00
4:37
0.00
6.02
40.13
Max-30%
.0.7
-014
0.02
10.23
0.19
1114
0.02
4:05
8.05
0.26
Max-4u% Prob 8 0.07
0.06
0.04
0.15
-303
0.00
-0.05
0.22
4:03
-0.33
0.17
0.19
4..05
0.00
0:08
0.12
5:03
.0.01
0.33
0.51
50%
-3.44
Max-60% Prob - lute
0.:00
0:04
-0.18
-0305
40.02
40.05
1822
4.00
0.01
0.14
40.00
16.15
-0.04
0.19
0.05
-0.03
0.11
0.:4
-0.44
Min 5% Prob--
0.02
6:05
Min-10% Prob -4.04
.0.33
4.30
0.12
0.37
1.06
10.02
0.00
0.06
0.04
0.00
.0.05
10.22
4.07
0.58
-200
4.30
0.46
0.17
Min-30% Prob 4/112
0.64
6.05
0.00
-0.14
0.30
-0.05
-0.66
4.00
-0.10
-2,00
0.20
-0.15
0.31
0.12
-0.11
19.27
0.05
40.13
-0.13
Min-30%
.0.55
0043
0.07
-0.08
0.20
4:06
0.34
-3.09
4..00
4.03
8:15
40.05
Miz Prob 0.00
-0:11
0:01
0.04
6:22
40.09
0.16
0.04
0.04
-0.08
0.09
0.46
+0.24
-0.51
18:43
0.00
-2.04
-0.15
40.10
-0.3
Min-50% Prob-
Min-80% Prob 15.15
00.35
4.00
-0.08
-0.15
0.05
0.06
0.00
4:11
0.26
0.02
0.05
06.25
-0.55
0:33
-0.05
-0.12
40.11
0.12
0.04
Los Dift Butter F'ngers
0043
1:25
0.02
0.8
Loss Dif Change Char Case
00:15
4..0
-0:34
-3.15
0.05
0.12
0.59
4001
-0.23
0:21
40:1
0..14
4:34
0.07
0.05
0.06
40.35
Lows DMT Random Deletion 0.35
0.06
4..09
0:48
2.15
0.02
0.01
10.19
0:06
0.29
-3,01
0.116
-0.19
0.18
0.18
-0.09
-0.12
.0.0
40.32
0.22
Loss Synenya Substhation-
.017
0.01
0.15
6:05
19:29
4.00
4.04
10:38
.0.11
0.6
Loss 1Xf Underssome Trick- - 0.09
0:30
0.16
0.05
0.11
0.05
0.55
48.06
6:41
-0.13
0:21
40.06
14.15
11:11
0.13
-0.01
-2.01
0.17
0.70
0.06
Loss DiTT Whitespce Pesturhttion
Loss Ratio Change Char Case
Loss Raie Butter Fingers 0.45
4022
40:30
0.09
48.31
-0.09
0.11
-0,05
0.00
10.85
0.12
40:37
0.26
0.00
0034
0.15
0.14
41.6
0.12
-0.88
10.99
0.19
4:05
-0.21
-3.40
22.15
0.14
10.4
4.00
0.15
0.4
Los Rario Rondom Deletion
0.:1
0.29
-0.14
0.06
0.15
4.00
-0.55
8:25
0.28
40.44
4:03
-0.25
Loss Ratio Synonym Substiution
0.22
0.13
4.255
0.50
2.05
0.04
0.20
0.26
4:05
.0.12
0.00
0.15
0.06
0.30
40.05
0.17
-025
4.00
0.14
0.56
Ratio Undeersooe Trick
6.11
0.10
10.55
0:23
4..05
0.14
5:05
0.2
Loss Ratio Whiterpace Pestrrution -4007
-0.66
0:04
0.04
0.07
10:22
-0.22
-1.34
0.11
0.01
0.14
0:22
-0.09
0:31
0.19
0:25
0.00
0.15
Paplesity
Metie
Perplexity Diff (Butter Fingers)
-005
-012
6.13
-0.18
0.05
0.05
0.22
11.66
0.10
-6:01
0.04
0.16
0.13
0.06
-2.14
0.0
Perplexity Diff (Change Char Cane)
-0.11
0.01
46.09
-0.21
40.00
11:0
-0:02
0.01
40:25
-0.17
-0.12
40002
0.02
40.09
Perplesity Dill (Ranke Delation
0.00
0.30
0.05
.0.11
6:03
10.13
-0122
6:06
0.08
-0.14
Perplesity Diff (Synonym Subetitution
0.35
.0.33
6:07
0.13
5.95
0.05
0.11
.0.55
0:06
0.04
0.02
1021
.0.0
00.9
0:07
0.227
-3.09
0.07
0.25
0.31
Perplesity Difl (Undmsore Trick)
0.10
0:06
-0.07
Perplxity Diff (Whitespee Pertuiution)- -4000
8:00
-0.04
-0.03
-0.15
40.06
-0.11
48.00
-2.05
-8.00
-2.01
10.36
0.27
11.64
4.03
0.07
6:51
40.05
-1.09
Ratio (Bunter Fingers)
-.112
10:00
1.26
0.26
0:07
Ratio (Change Char Case)
062
6.17
0.01
0.05
0.18
-0.05
-0.52
-2.03
0.04
0.04
+0.15
-641
-6.11
-0.23
0.14
-0.0
40.04
Ratio (Random Deletion-
0.65
10022
0.10
0.05
0.16
0.05
0.12
6:25
412
+0.26
0.19
+2.43
Batio (Synonym Substitution-
-.15
0.04
-3.17
1..0
0.50
10.14
-.18
4:225
0.24
0..37
Ratie (Underscore Trick)
4.05
0.34
0:02
1.12
-0.11
0.05
0.26
10.00
0.14
0.17
-0.15
0.15
-0.12
00.00
0.21
0.14
5.04
0.27
0.20
0.13
-066
Ratio (Whitespaz Portunbation)
Nocmulized Loss Dall Ph-1.55
0.02
0.18
19.21
8.16
0.05
0032
0.21
6358
0.01
8.12
1.20
1.25
0.044
6.15
0.09
12.19
0.30
0..17
0.02
Normalized
Duff Sio
4.7%
0:23
4.25
-0.28
10
-0.0
0.00
4.005
Normalized Loss Deff Tinsstones IM
0.15
0:09
-8.05
-2.05
0.05
0.04
11.55
4:15
-0.00
-0.14
41.50
4.15
4.18
0:00
008
Nocmalized Los Dif 3eysstoness333
4.18
4.50
46.15
-0:07
0.08
40.01
.0.01
00.88
0.13
0.11
0.03
0.5
0.02
-0.22
0:35
0.07
40.15
40.18
Normalized Loss Ratio Ph: 1.5
..13
0.34
0.42
4.10
0.56
6:02
0.07
Normalized Loss c Silo
4.39
0.14
10.19
-0305
0.01
21.16
10:10
0.01
0.15
0.32
0.21
-0.28
4.10
0.22
+3.03
0105
0.55
-0.11
Nermalized Los Ratio Tmystuics--M-
Normelized Loss Ratio TEsystonee-33M-
0.12
.053
6.11
.001
0.07
10.08
10.58
0.17
0.10
10.05
1221
0.08
0.25
0.61
4.00
0..3
0:21
0.15
0.53
0.31
Normalized Perplecit (Mf 1.5)
0:09
0.19
40.00
0:01
0.04
Nummllaad Peplextty DJ (Sila)
-4351
0.36
4..12
10.26
0.01
0.06
0,55
0:02
-0:02
0.02
4126
6:09
0.11
10:28
40.5
Normaizzed Peplexny Duf (Tinysteries LM4
4.18
0.58
0.66
-0:44
-2.05
0.01
-0.05
.0.01
4006
0.01
-3.01
0.73
-0.11
0.91
4:07
-0.03
+0.17
0.05
4.04
-0.3
Nommaizzed Perplexity Diff (Tinysories 33M)
-3,05
Nommaizzed Ratio (Phi-1.5)
644
4.56
0.14
0.34
6222
0.65
10.3
11.1
4007
0.07
18:2
4.65
0.15
0.14
6.14
0.19
5.17
0.30
4.00
.0.4
Sarnalized (Sila)
0.49
Normalized Rate (MI
0.12
065
4.09
-0.66
-3,05
0.00
.000
0.0
4.18
0.53
80.12
0.00
0.06
.0.6
4,00
0.09
.0.15
0.12
4.00
0.50
Normalized Ratio (Tinyariia-13M)
4.10
-003
46.01
0.07
0m
-0.05
-004
0.07
0.01
40.8
0.01
0:01
4.05
Züi Estopy / Papplxityy 4222
.us5
0.07
H1.11
.012
4:09
-0oN
-2,4
4028
40.5
00433
-2.12
0.29
Book
Books
OC-
Durahust
Fron sw
Gahub
Conterbing
HockerNow
Malk-
OW32
Opmn5oss
Phppr
PrbMed Ale
Stck-
USPIO
Uhumth
Wild
YT5ib=
uXXi-
PulMed
Dateset
Figure 8: Extended version of Figure 5a with more features that we leverage for the LLM dataset inference. The results are consistent-one can see most features contribute positively to dataset inference for some datasets, while negatively to other datasets.
16

---

0.1
p-value
p-valuc
0.0001
0.0001
Wiki
Wiki
1c-07
miXoataset
1c-07-
maXpataset
median
medianpataset
lc-10
1c-10
10
10"
10'
10'
10
10
Number of Points
Number of Points
(a) Mcan Correction
(b) Mean Correction (Normalized)
0.1
0.1
p-value
p-value
0.0001
0,0001
Wiki
Wiki
1e-07
miXDateset
1e-07
maXpataset
mediinoetaset
medianostaset
le-10
1c-10
10'
10
10"
10'
10
10
Number of Points
Number of Points
(c) Outlier Removal
(d) Outlier Removal (Normalized)
0.1
0.1
p-value
p-value
0.0001
0.0001
Wiki
Wiki
1e-07
maXDataset
1e-07
mataDtt
medianCatese!
medianpataset
le-10
1e-10
10
10
10
10
10 Number of Points
10
Number of Points
(e) Both
(f) Both (Normalized)
Figure 9: Extended version of Figure 6a with different pre-procees rre-processing techniques used in dataset inference. Similarly, the three curves correspond to the median, and maximal p-value across the datasets in Pile, and the p-value of Wikipedia dataset. It is noteworthy that the processing techniques (see details in Section 5.3) do not impact the median and Wikipedia curves significantly. However, normalization has a positive contribution to the max p-vauue across different datasets.
17

---

0.1
0.1
p-value
p-value
0.0001
0,0001
Wiki
Wiki
le-07
mAXDalase
1e-07
maxDalsse
medianostase!
medianoataset
1e-10
10'
10"
10"
10
10
10"
Number of Points
Number of Points
Pythia-410m
(b) Pythia-1.4b
0.1
p-value
p-valuc
0.0001
0.0001
Wiki
Wiki maKpataset
1c-07-
maxpataset
1c-07-
mediangataset
medianpataset
le-10
1e-10
10
10
10
10'
Number of Points
Number of Points
(c) Pythia-6.9b
(d) Pythia- 12b
Figure 10: Extended version of Figure 6a with different model sizes. Similarly, the three curves correspond to the median, and maximal p-value across the datasets in Pile, and the p-value of Wikipedia dataset. An interesting observation that the curves for median p-values and Wikipedia's p-value stay similar with respect to models of different sizes. However, this is not the case for the max p-value curve. This indicates that dataset inference does not rely on large number of data points or model parameters for most datasets, whereas they may be necessary for some particular datasets.
1.0
1.0
p-valuc
Deduped
Doduped
0.5
0.5
Non- Deduped
Mon- Non-Dedupol
0.0
6.9
12.0
1.4
6.9
12.0
Model Size (Billons Parameters)
Model Size (Billons Parameters)
(a) 100 data points
(b) 200 data points
1.0
1.0
Deduped
Deduped
Non-Deduped
Nen-Deduped
p-value
0.5
0.5
0.0
0.0
0.41
1.4
6.9
12.0
0.41
1.4
6.9
12.0
Model Size (Billons Parameters)
Model Size (Billons Parameters)
(c) 500 data points
(d) 1000 data points
Figure 11: Extended version of Figure 6b with different numbers of data points used in dataset inference. As expected, larger number of data points allow the p-values to be more concentrated below the threshold, especially when the model size is large.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions of the paper are clearly stated in the abstract and introduction, and supported by empirical results in Sections 4 and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine to include aspirational goals motivation long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results
19

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Sections 4 and contain sufficient information for reproducing the results. The code released on GitHub and the link provided the footnote the first page.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is released on GitHub and the link is provided the footnote of the first page.
20

---

Guidelines:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cttpppiicccpppcccc//cp.c
for more details.
While encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / cctiipiccc/pbbic/////cccc /elidee/Coeeuiii
ssi onPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The experimental setup is described in Section 5.3, and details of specific experiments provided when the corresponding results are presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The majority the experiments, including the ones that support the main claims
of the paper, are repeated using 10 different random seed. The resulting p-values aggregated using standard methods like Sidak's 's method and then reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
21

---

It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the ex experi- ments?
Answer: [Yes]
Justification:
We describe the computer resources used in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
Code Of Ethies
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neunt wc/ppblic//chhccsuuieeee
Question:
Answer: [Yes]
Justification: The experiments are conducted publicly available datasets without known concerns. We do not expect the method proposed the paper to lead to negative social impacts since it is designed to protect copyrighted data from unauthorized use in language models.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., s.. if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of the paper are explicitly discussed in Appendix
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., dis- information, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considera- tions, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applica- tions, the authors should point it out. For example, it is legitimate to point out that an improve- ment the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The outcomes of the paper not include releasing data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets and models used in the paper are properly cited. The URLs to them are included if existing.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswither thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged reach out the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [Yes]
Justification: The only new asset introduced in the paper is code which is released on GitHub. We include bash scripts for ease of running code, and README.md file containing detailed documentation in the GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submis- sions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
24


Reviews:
[{"review_id": "dqfF1xQFMf", "reviewer": "Reviewer_VFio", "summary": "This paper tackles the dataset inference problem to detect a specifically trained dataset such as a licensed dataset. Firstly, the authors claim that previous membership inference attacks (MIAs) are not successful in discriminating between members and non-members from the same distribution (iid), which is a less realistic setting. The authors propose selectively combining multiple membership inference metrics by linear regression. Experimental results demonstrate its efficacy on the PILE dataset by aggregating 52 different MIA metrics.", "questions": "- How did you select the 52 MIA metrics? Sharing the standard you used will be beneficial to other researchers.\n- In Figure 6 (b), training set deduplication results, why does the violin plot distribute like sandglass? — i.e., the distribution mass on the higher p-values is thicker than near the p-value of 0.5. Moreover, dedup and non-dedup settings are not described.", "limitations": "The authors adequately addressed its limitations and broad impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The problem in this paper is important, regarding the high risk where copyright text is used for training current LLMs.\n- The paper is well written, especially summarizing previous related work and methods. It’s easy to follow.\n- This paper criticizes the problem settings of previous papers and proposes more realistic scenarios — e.g., iid of suspect and valid dataset.\n- The proposed method is simple and straightforward; an ensemble of multiple attack methods outperforms a single metric", "weaknesses": "- Absent of hyperparameter gammas. Depending on what gamma value was chosen, each MIA performance would vary.\n- It is not explored whether the proposed method is also robust to the unseen dataset beyond the PILE dataset, as well as other models except Pythia.\n- The title “LLM Dataset Inference: Detect Datasets, not Strings” is too broad and enables readers to misunderstand as it proposes a “Dataset inference” task and method, whereas the paper indeed proposes a mixture of MIAs to improve robustness.\n- Qualitative analysis of samples is absent. Is there any interesting analysis or distinguished result between attack samples (victim data) with low and high p-values?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and positive feedback.\n\n### **W1: Absence of hyperparameter gammas.**\n\nWe measure the success rate of MIAs using the Area Under the Curve (AUC) metric. The AUC metric is advantageous because it provides a comprehensive measure of the model's performance that is independent of the parameter $\\gamma$. This independence ensures that our evaluation is robust and not influenced by the specific choice of $\\gamma$. As far as dataset inference is considered, it only uses the score given by each MIA, and automatically determines how to weigh each MIA in Stage 2 as described in the paper, hence once again not needing the same parameter.\n\n### **W2: Explore datasets beyond the Pile dataset and other models except Pythia.**\n\nThe Pythia suite of models, trained on the PILE dataset, is unique in providing complete access to both the training dataset and the training procedures. This transparency is crucial as it allows for rigorous and replicable experimentation, ensuring the validity and reliability of our results. Without the information provided along the Pythia suite of models, we cannot reliably know if a point was part of the pretraining set. This could for instance lead us to overestimate the performance of our method, as was the case for prior work. To the best of our knowledge, no other set of LLMs offers the level of accessibility that Pythia models do. Our work is a call to action for the community to provide more model releases with comparable levels of transparency. \n\n### **W3: The title is too broad. The paper proposes a mixture of MIAs.**\n\nWhile dataset inference builds upon membership inference, it argues for a new paradigm of privacy research where we consider the membership of an entire dataset rather than individual text sequences. To clarify our stance, we are only using a mixture of MIAs to perform the task of dataset inference. Even when each MIA performs close to (and often worse than) random, dataset inference can tease out statistical signals from weak membership attacks.\n\n### **W4: Qualitative analysis of low and high p-values?**\n\nOur p-values for true positive cases are significantly below the set threshold of 0.1, while the p-values for true negative cases are substantially above this threshold. However, p-values alone are rather not suitable for qualitative analysis due to their nature as a probabilistic measure rather than a direct indication of effect size or practical significance. Instead, we can use the number of data points required for our LLM dataset inference method. We observe that structured and well-formed datasets, such as Wikipedia, require fewer data points (i.e., 50) for effective inference compared to less structured datasets like Pile-CC (which requires 300 data points), which contain raw web pages. The table below presents the number of data points required for our method per dataset on the Pythia-12b model.\n\n|Dataset|Number of data points for our method|\n|-|-|\n|Pile-CC|300|\n|PubMedCentral|700|\n|Books3|100|\n|OpenWebText2|300|\n|ArXiv|300|\n|Github|500|\n|FreeLaw|400|\n|StackExchange|50|\n|USPTOBackgrounds|150|\n|PubMedAbstracts|<=10|\n|Gutenberg(PG-19)|100|\n|OpenSubtitles|150|\n|Wikipedia(en)|50|\n|DMMathematics|400|\n|UbuntuIRC|<=10|\n|BookCorpus2|20|\n|EuroParl|50|\n|HackerNews|50|\n|YoutubeSubtitles|20|\n|PhilPapers|<=10|\n\n### **Q1: How did you select the 52 MIA metrics?**\n\nWe incorporated all available MIAs at the time of developing our method and further extended them to extract additional features, thereby capturing more information and enhancing the success of dataset inference. For instance, rather than using a single K parameter for the MinK-Prob MIA [32], we employed multiple values of K. We performed an ablation study for the features used in dataset inference, and the results are shown in Figure 5(a) with a more detailed version in Figure 8 in the Appendix. We did not find that any particular features were most informative across all datasets. Instead, feature informativeness is highly dataset-dependent--most of the features contribute positively for some datasets but negatively for others. However, we do see some redundancy between many of the Min-k and Max-k% MIAs, hence, they can be ignored if required without a performance tradeoff. That said, there is no computation overhead in computing all of them, as they can be computed in one single forward pass of the model.\n\n### **Q2: In Figure 6 (b), training set deduplication results, why does the violin plot distribute like sandglass?**\n\nThe main purpose of Figure 6 (b) was to observe whether the larger models experience more memorization, and how this can allow performing dataset inference more reliably. We would refer you to Figure 10 in the Appendix which provides a better breakdown of how different model sizes perform for dataset inference. Once again, when considering p-values, since it is a statistical test, we generally do not look at the qualitative distribution of the same, and only their binary position below or above the chosen significance threshold. Overall this suggests that for smaller models, there may be some false negatives (because they do not learn the distribution),\n\n### **Describe dedup and non-dedup settings.**\n\nWe tried to explain the setup for the experiment in the description of Figure 6 (b) and lines 331-332. However, we will rewrite it to clearly state that: *Deduped* denotes a version of the Pile dataset where the documents are deduplicated within, and across the data splits (train/test/validation). *Non-Deduped* is the original version of Pile without any deduplication.\n\n---\n\nWorking on the feedback helped us improve the quality of our work. We look forward to further discussions to clarify any concerns that remain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mUalIXIABg", "reviewer": "Reviewer_NHN3", "summary": "The paper addresses the problem of Dataset Inference in Large Language Models - namely, the ability to detect whether the given dataset was used by LLM developers in pre-training.\n\nFirst, authors demonstrate the importance and feasibility of Dataset Inference task in comparison with Memebership Inference Attacks. Namely, they show that existing MIA methods cannot derive membership when test examples are sampled from the same distribution as training data. \n\nSecond, they propose novel method of dataset inference, based on aggregation of several MIA scores as features for linear classifier. According to their experiments, the proposed apprioach does not provide any false positive results, discovering subsets of LLM training dataset with high accuracy. The algorithm requires only 1000 data points from the whole dataset to discover if it was used for training.\n \nThe proposed method is tested on several PILE subsets, using the Pythia model family as the base model. Authors demonstrate that the efficiency of the proposed method increases with the growth of the model size.", "questions": "Q1.  It is not clear why top-k% MIA is chosen to demonstrate general MIA failure. First, according to the original paper [32], this MIA, by design, should be stable for paraphrasing; that's why it is tested on the deliberately collected dataset (WikiMia) with different *events* distribution, not just different strings. \nThe setup of the current paper is different (and more difficult): the authors aim to detect membership of the dataset when the whole data distribution is shared between suspect and validation sets. Figure 5(a) confirms that min-k% MIA is not suitable for this setup. At the same time, other methods provide a much more informative signal. Why don't you check, e.g., perturbation-based approach, claiming that existing MIAs cannot detect membership?\n\nQ2. The proposed method requires providing IID data for check and the \"equivalent\" set of data for validation. Is it possible to define the notion of \"IID\" more strictly, for practical use? E.g., suppose the author provides a book as \"suspect\" data. Is the unprinted chapter of the book \"IID,\" if it may contain novel events and characters? On the other hand, can we consider the draft version of the book as IID, taking into account that the final version was post-processed by the editor and layout designer?\n\nWhat if I want to check the presence of some benchmark data in pre-training of some LLM with open weights. Which \"suspect\" and \"valid\" data should I use?\n\nQ3.  Is it possible to reduce the amount of used features? E.g. what is the drop of performance if the top-k% score is excluded from the feature set of the classifier? Which features are the most informative in general?", "limitations": "The results of the paper was not checked in realistic settings. In general, it is not clear if the method can be applied to the analysis of the existing models with grey-box access.\n\nAuthors claim that their approach is extremely stable, and dies not provide any false positives; but the experiments are not enough for such claim.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Authors demonstrate the failure of the SOTA MIA method (top-k% score) on different subsets of the PILE dataset. Namely, they prove experimentally that this method detects distribution shift rather than actual membership inference. Besides, they show that among several considered MIA scores, no one has consistent efficiency over different domains of PILE. This analysis is essential for future research in this direction \n+ Authors propose novel approach for dataset infenrence, based on aggregation of several MIA scores. The efficiency of the proposed method is demonstrated by testing on wide range of domains, and with several model sizes. \n+ The method has shown its efficiency in the problematic case of the IID data: namely,  when suspect and validation data are derived from the same dataset by random sampling\n+ The overall detection quality, demonstrated by experiments, is extremely high on all tested domains.", "weaknesses": "1. The proposed method requires providing suspect and valid datasets with the same distribution. In a practical situation, it is not clear how to obtain this (see Q2 to authors). \n\n2. Authors claim that only 1000 examples are enough for Dataset Inference. Meanwhile, train and valid set are also required from the same data distribution. On practice, the method is not tested in real few-shot regime.\n\n3. The method is tested on the single model's family, and the efficiency is demonstrated only on the largest model of the family. \n\n4. The method is tested on PILE dataset, with the access to the validation set from the same distribution, which was not used for raining for sure. There is no experiments with less clean data or less transparent model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and comments!\n\n### **W1, Q2: Existence of IID validation dataset** \nWe acknowledge that obtaining labeled validation data sampled from the same distribution as the test data can be challenging. Indeed, this is the number one open problem that our work creates for future research. That said, a significant part of this work was spent in formalizing a framework. Here are quite a few scenarios where the IID setting should roughly happen in practice:\n- **Editorial Process**: Thousands of NYT articles reach the editor's desk but never get published. These unpublished articles are likely to be more IID compared to two random splits of the Pile dataset.\n- **Book Drafts**: This is also very common with book writers when working on a chapter draft. These drafts would be more IID than random splits of the Pile, in our opinion. As the Reviewer raised in Q2, independent chapters of books may not be IID, but their progression will likely yield an IID set. However, this is up to debate and in practice we need an IID verifier (in future work) as opposed to arguing about the same. One strong baseline for the same is the recent “Blind MIAs” work by Das et. al. https://arxiv.org/abs/2406.16201. If any pair of datasets can be distinguished based on these blind membership inference attacks, they should be deemed **not** IID. Future work can improve upon this.\n\nAt the same time,  some potential directions for bypassing the IID problem in future work also include:\n1. **Using Synthetic Data**: Create IID sets artificially using synthetic data generation techniques.\n2. **Storing Hashes**: Creators may store hashes of IID data at the time of publication to ensure data integrity.\n3. **Proactive Storage**: More proactive work to store generations of artistic work rather than only the final stage.\n\n### **W2: Requirements on amount of data**\nIn practice, the size of the training and validation datasets can be exactly equal to the size of the dataset used for testing the attack.  This means that even if a potential victim has only 1000 sequences of 1024 length context of text data that was trained on (along with an equivalently sized validation set), this is a sufficient condition for performing dataset inference. As shown in Figure 6, the number of examples needed for dataset inference to be successful decreases with model size and also depends on the dataset, which means we may require less than 1000 sequences in practice.\n\n### **W3: The method is tested on the single model's family**\nThe Pythia suite of models, trained on the PILE dataset, is unique in providing complete access to both the training dataset and the training procedures. This transparency is crucial as it allows for rigorous and replicable experimentation, ensuring the validity and reliability of our results. Without the information provided along the Pythia suite of models, we cannot reliably know if a point was part of the pretraining set. This could for instance lead us to overestimate the performance of our method, as was the case for prior work. To the best of our knowledge, no other set of LLMs offers the level of accessibility that Pythia models do. Our work is a call to action for the community to provide more model releases with comparable levels of transparency.\n### **W3: and the efficiency is demonstrated only on the largest model of the family**\nWe also demonstrated the efficiency of our method on other models from the suite, in addition to the largest one. Please refer to Figure 10 in the Appendix, which is an extended version of Figure 6a (from the main paper) with different model sizes. An interesting observation is that the curves for median p-values and Wikipedia’s p-value stay similar with respect to models of different sizes. However, this is not the case for the max p-value curve. This indicates that dataset inference does not rely on a large number of data points or model parameters for most datasets, whereas they may be necessary for some particular datasets which smaller models do not learn well.\n\n### **W4: Less clean data and less transparent model**\nFor less transparent models, there is no ground truth to determine whether a specific data point was used in training or not. Even if we hypothesize that a model is trained on a particular data source, such as Wikipedia, we lack access to an unseen IID validation set. Note that simply collecting Wikipedia articles published after the model’s release is insufficient due to the temporal shift in concepts. Since this is the main limitation of prior work, we opted to study the Pythia suite of models in our experiments.\n\n### **Q1: Other MIAs**\nWe fully agree that the top-k% MIA is not representative enough to claim the failure of all MIAs. This is why we tested all MIAs we could find including the perturbation-based ones and reported these results in Figure 7 -- we apologize that due to the large amount of MIAs we tested this figure cannot fit in the main body of the manuscript, we will either add it to the main paper (using the additional 1 page if accepted) or try to and emphasize the reference to the Figure in the main body.\n\n### **Q3: Features used in dataset inference**\nWe performed an ablation study for the features used in dataset inference and the results are shown in Figure 5(a) with a more detailed version in Figure 8 in the Appendix. We did not find that any particular features were most informative across all datasets. Instead, feature informativeness is highly dataset dependent--most of the features contribute positively for some datasets but negatively for others. \n\n---\nOnce again, we thank the reviewer for the constructive feedback on our work. Working on the feedback has helped us improve the quality of our work. We look forward to further discussions to clarify any concerns that remain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NZTAb1G9oq", "reviewer": "Reviewer_pZrT", "summary": "The paper addresses the challenge of identifying training data in large language models (LLMs) with rising concerns over privacy and copyright violations. As it has previously been studied, traditional membership inference attacks (MIAs), which determine if individual text sequences were used in training, are often flawed due to temporal shifts in data. The authors propose a novel dataset inference method that focuses on identifying entire datasets rather than individual data points, reflecting real-world copyright scenarios where authors' works, like books, are used without permission. Their method combines multiple existing MIA metrics to identify training datasets effectively, achieving statistically significant results with p-values less than 0.1 and minimizing false positives. The paper highlights the method's robustness through extensive testing on the Pythia models and the Pile dataset, providing a more accurate approach to dataset attribution and addressing the shortcomings of prior MIAs.", "questions": "1. Authors mention they have used 4 A6000 GPUs for experiments. How long it takes for the algorithm to run and test the Pile dataset? What is the runtime complexity for different dataset and model sizes?\n\n2. How does the method handle cases where the IID assumption for suspect and validation sets does not hold? Are there any strategies in place to deal with non-IID data distributions?\n\n3. What are the potential trade-offs in terms of performance and accuracy?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured, with clear problem statement, explanations of the methods, experiments, and findings. Figures and tables are effectively used to illustrate key points, making the complex subject matter more accessible.\n\n2. Their method is extensively tested on Pythia and the Pile dataset, showing its effectiveness. The use of statistically significant p-values and the absence of false positives strengthen the results.", "weaknesses": "1. The concerns about the accuracy and effectiveness of existing MIA methods have been previously studied in depth in papers such as https://arxiv.org/pdf/2402.07841. There are existing publications which have highlighted the weaknesses in empirical results that detect temporally shifted member/non-member classes. \n\n2. The computational complexity of the proposed algorithm is not addressed in this work. Authors need to provide a detailed quantitative comparison (in terms of algorithm runtime or the number of tokens processed) for their proposed method. Is this method feasible for models with more parameters (70B or higher)?\n\n3. Although authors have studied the Pile Dataset on Pythia extensively, their experiments are just limited to this setting. There is not much insight about how this proposed method generalizes to other models and datasets. Also, I believe that the fact this method only applies to datasets (and not single strings of text) is a limiting factor. The existence of a suspect and a validation subset is not realistic in all cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you  for your thoughtful comments and feedback. We are happy to see that you appreciated the method’s robustness and overall quality of the draft. We address the individual concerns below one by one:\n\n### **Re: Other work studying the failure of MIAs (W1)**\nWe would like to preface this answer by adding that the linked paper was truly a concurrent work to this submission. Crucially, our work not only shows the limitations of MIAs, but goes beyond that by proposing a comprehensive framework to detect if a given dataset (not string) was used in the training of an LLM, a shift that is particularly relevant for modern generative models.   \nWe would like to note that we indeed also duly acknowledged this concurrent work in our submission on lines 187 and 188: “A similar question was concurrently asked by Duan et al. [15], who independently showed that MIAs are only successful because of the temporal shift in such datasets.”\n\n### **Re: Computation Complexity (W2, Q1, Q3)**\nThe computational complexity of the method is relatively low, and roughly equivalent to the computation cost of performing each MIA required in our work. \n1. For instance, considering the Min-K% MIA, this would mean performing a single forward pass on each example, and aggregating the token-wise loss to calculate the final metric score. Other methods are perturbation-based or reference model-based, requiring two forward passes.\n2. In the attached code, we provide detailed method to perform each of the 50 MIAs we performed in `metrics.py`. Many of the MIAs are dependent on each other, for example, all the Min-K% and Max-K% MIAs can be computed in a single forward pass. Overall, in practice we only require 17 forward passes to perform all the attacks.\n3. Given that most datasets require less than 500 examples (of train and val) to perform dataset inference. This would mean performing on the order of 1K forward passes. In our experiments, each forward pass is of context length 1024. But can change depending on the model configuration.\n4. Finally, most of this computation can be batched to perform many forward passes together. Given modern day hardware, this cost is low enough that it can be performed in about 4 hours of an A6000 GPU node even for models like pythia-12B (for one dataset).\n5. The final cost of “learning” which MIAs to use is almost negligible, as it just requires to fit a 50 dimensional linear layer to a few values (can be done on CPUs in less than 30 seconds).\nTo conclude, the method is indeed feasible for model with more parameters. We tested on models from the Pythia suite, where the biggest one has 12B parameters but the method can be easily run for models with 70B parameters (like Llama 2) or larger. \n\n### **Limitations of Pile Dataset + Pythia** \n\nWe agree that repeating the experiments on more models and datasets would lead to stronger empirical supports. When it comes to open-weights + open-source pretrained models released online, to the best of our knowledge, the Pythia suite of models is the only one releasing complete information about its training + validation dataset (along with the more recent [Olmo model series](https://github.com/allenai/OLMo) which only came out after we finished our experimentation). Without this information, we cannot reliably know if a point was part of the pretraining set and thus cannot reliably validate the proposed method. We would like to emphasize that even though the Pythia suite and Pile Dataset may seem like one single entity, this is actually a collection of 4 models x 20 data subsets, that offers significant generality to our method, much beyond any other random split of publicly available data might offer. \n\n### **Limitation of application to datasets (and not single strings of text)**  \nWe believe that in the modern landscape of generative AI, victims naturally contain multiple sequences of text on the internet. For instance, an article in the New York Times would be composed of multiple sequences of context length 1024. Similarly, the same organization, like NYT has multiple articles that it together wants to own copyright over. Victims who will file legal suits will likely always possess such characteristics. Hence, it is quite natural that we are in a scenario where individual sequences no longer exist in isolation (unlike past classification datasets in the vision space). This setting is also relevant to creative artists having many photographs, and paintings.\n\n### **Existence of IID validation dataset**\nWe acknowledge that obtaining labeled validation data sampled from the same distribution as the test data can be challenging. Indeed, this is the number one open problem that our work creates for future research. That said, a significant part of this work was spent in formalizing a framework. Here are quite a few scenarios where the IID setting should roughly happen in practice:\n\n- **Editorial Process**: Thousands of NYT articles reach the editor's desk but never get published. These unpublished articles are likely to be more IID compared to two random splits of the Pile dataset.\n- **Book Drafts**: This is also very common with book writers when working on a chapter draft. These drafts would be more IID than random splits of the Pile, in our opinion.\n\nSome potential directions for bypassing the IID problem in future work also include:\n\n1. **Using Synthetic Data**: Create IID sets artificially using synthetic data generation techniques.\n2. **Storing Hashes**: Creators may store hashes of IID data at the time of publication to ensure data integrity.\n3. **Proactive Storage**: More proactive work to store generations of artistic work rather than only the final stage.\n\n----\nOnce again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to resolving any remaining concerns during the rebuttal-response period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tBc62hqEtX", "reviewer": "Reviewer_Nmtj", "summary": "In this paper, the authors investigate the commonly used membership inference evaluation for LLMs and find that previous attacks primarily detect features related to temporal changes, performing poorly under real IID scenarios. To address the challenge of individual sample membership inference attacks, the authors propose a novel threat model: LLM dataset inference. The proposed method can accurately infer whether a set of data points was used in the training process or not.", "questions": "- How does the victim choose the suspect set? What if the suspect set is a mix of member and non-member data points? This might happen when the suspect intentionally trains the model on a subset of the dataset.", "limitations": "The authors mention the limitations in the appendix, which I appreciate.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, and I really enjoy reading it.\n- The \"Failure of Membership Inference\" section is really inspiring, and it's very important to the future LLM MIA area.\nThe proposed dataset inference method is very simple but effective. The results look very promising.", "weaknesses": "- The experiments are conducted on a single series of models, rather than on various models trained with different datasets, algorithms, or even seeds. I think this is a little picky since it's not cheap to retrain a bunch of large models, but I do think it's important for reliable evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback and are happy to hear that they enjoyed reading our paper.\n\n>**The experiments are conducted on a single series of models, rather than on various models trained with different datasets, algorithms, or even seeds. I think this is a little picky since it's not cheap to retrain a bunch of large models, but I do think it's important for reliable evaluation.**\n\nWe used the Pythia suite of models, trained on the PILE dataset, since it is unique in providing complete access to both the training dataset and the training procedures. This transparency is crucial as it allows for rigorous and replicable experimentation, ensuring the validity and reliability of our results. Without the information provided along the Pythia suite of models, we cannot reliably know if a point was part of the pretraining set. This could for instance lead us to overestimate the performance of our method, as was the case for prior work. To the best of our knowledge, no other set of LLMs offers the level of accessibility that Pythia models do (except the very recent [Olmo model series](https://github.com/allenai/OLMo) which only came out after we finished our experimentation). Our work is a call to action for the community to provide more model releases with comparable levels of transparency.\n\n>**How does the victim choose the suspect set? What if the suspect set is a mix of member and non-member data points? This might happen when the suspect intentionally trains the model on a subset of the dataset.**\n\nIn general victims can determine suspect sets based on behaviour from a model that is suggestive of having trained on their content. This could include yielding substrings from their novels, or blogs etc. The adulteration of training with only a part of the member data is an interesting consideration. We can reframe the question as follows:  \n\n> Assume distributions $ A $ and $ B $ are distinguishable based on a t-test. You can assume they are Gaussian with separate means. Now, an adversary adulterates $ A $ with $ x\\% $ of $ B $. Will the t-test succeed in distinguishing the adulterated distribution from $ B $?\n\nTo determine if the t-test will succeed in distinguishing the adulterated distribution $ A' $ (where $ A' $ is $ A $ mixed with $ x\\% $ of $ B $) from $ B $, we need to consider how the adulteration affects the statistical properties of $ A' $.\n\n1. **Let us assume that the dataset inference scores from stage 3 follow a Gaussian in the original distributions**:\n   - $ A \\sim \\mathcal{N}(\\mu_A, \\sigma_A^2) $\n   - $ B \\sim \\mathcal{N}(\\mu_B, \\sigma_B^2) $\n   - The t-test can distinguish between $ A $ and $ B $, implying the means and variances are sufficiently different.\n\n2. **Adulterated Distribution**:\n   - Let $ A' $ be the new distribution obtained by mixing $ A $ with $ x\\% $ of $ B $. This means $ A' $ is a mixture of $ (1-x)\\% $ of $ A $ and $ x\\% $ of $ B $.\n   - The mean of $ A' $ is:  \n     $\n     \\mu_{A'} = (1 - x) \\mu_A + x \\mu_B\n     $\n\n3. **t-test on $ A' $ vs. $ B $**:\n   - For $ A' $ and $ B $ to be distinguishable, the t-statistic needs to be large enough:\n     $\n     t = \\frac{\\mu_{A'} - \\mu_B}{\\sqrt{\\frac{\\sigma_{A'}^2}{n_{A'}} + \\frac{\\sigma_B^2}{n_B}}}\n     $\n\n4. **Effect of Adulteration**:\n   - As $ x $ increases, $ \\mu_{A'} $ moves closer to $ \\mu_B $.\n   - For small $ x $, $ \\mu_{A'} $ is still relatively close to $ \\mu_A $, and the t-test might still distinguish $ A' $ from $ B $.\n   - For large $ x $, $ \\mu_{A'} $ approaches $ \\mu_B $, making $ A' $ and $ B $ indistinguishable by the t-test.\n\nThe t-test will likely succeed for small $ x $ and fail for large $ x $. The exact threshold of $ x $ depends on the values of $ \\mu_A $, $ \\mu_B $, $ \\sigma_A $, $ \\sigma_B $, and the sample sizes $ n_{A'} $ and $ n_B $. \n\nIncreasing the sample size $ n_{A'} $ can improve the power of the t-test, making it possible to detect smaller differences between $ \\mu_{A'} $ and $ \\mu_B $.\n\n\n\n\n--- \n\nOnce again, we thank you for the constructive comments. Working on them has helped us improve the quality of our paper. We look forward to further discussions to clarify any concerns that remain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4i9EZO1FmJ", "reviewer": "Reviewer_bhaD", "summary": "Large language models are trained on a vast amount of online available data, which has lead to copyright and privacy issues (e.g. New York Times vs. OpenAI, as pointed out by the authors). There are various methods that try to identify if a given data point x was used to train a large language model.\n\nThis paper presents a very systematic analysis of these methods based on the Pythia suite of models.\n\nThe conclusions of the paper is, that current methods are not well-suited to determine if specific data was included in a training set.", "questions": "Please explain, why you did not conduct any tests with commercially available models, such as GPT-4, Claud3, etc.", "limitations": "Limitations were discussed by the authors.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors evaluate six metrics in the main part of the paper and a very large number of additional metrics and variations of the initial six in the appendix. Furthermore, the authors test these metrics on a large number of data sets, spanning different domains (e.g. github and wikipedia, which I would consider very different).\n\nThe analysis is conducted on the Pythia suite, which means that the authors had full access to all relevant information, in particular, the training data and training methodology. As a result, they had access to the ground truth.\n\nThe authors also state, that they will release their code, once the reviewing process is completed and anonymisation is no longer required.", "weaknesses": "It would be interesting to see, how the evaluation generalises to commercially available models, such as Gemini and GPT. This could be done e.g. with Wikipedia-Articles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and are happy to see that you found our analysis systematic and our paper technically solid with a high impact, along with an overall positive assessment of the work. We acknowledge your concerns and attempt to respond to them line by line below:\n\n### Re: Generalization to Commercially Available Models\n\nWe recognize the interest in seeing how dataset inference generalizes to commercially available models such as GPT-4 and Claude. Here are a few points that limited the same, but are also great fuel for future work:\n\n1. **Lack of Blackbox MIAs**: In our work, we leverage grey-box access to the models, that is, we assume that the membership inference attacks have access to the loss values from the underlying model. Since MIAs in LLMs is a relatively nascent field, the research has not yet unlocked successful black-box MIAs, unlike the vision space where we have seen label only MIAs (eg. [Choo et. al.](https://arxiv.org/abs/2007.14321)). We believe future work can unlock this possibility with API access as well.\n\n2. **Access Limitations**: Conducting tests with commercially available models is challenging due to limited access to their training data and methodologies. Unlike the Pythia suite, where we had full access to all relevant information, commercial models do not provide the same level of transparency, making it difficult to obtain the ground truth for evaluation. In particular, the presence of ground truth labels for Pythia models allows us to validate that dataset inference is indeed successful in distinguishing members and non-members. \n\n3. **Alternative Evaluation Methods**: We suggest that future research could explore alternative evaluation methods that do not rely on access to the full training/validation data. For example, researchers could use synthetic data or develop new techniques that infer properties of the training data from the model's outputs. The suggestion of temporally shifted Wikipedia articles would not work in this case because it would give a *false sense* of success as also noted in concurrent works of [Das et. al.](https://arxiv.org/abs/2406.16201) and [Duan et. al.](https://arxiv.org/pdf/2402.07841).\n\n\nOnce again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to further discussions and improvements in this evolving field. Please let us know if we can address any remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GgPetkXHFZ", "reviewer": "Reviewer_G1MR", "summary": "The paper addresses the limitations of traditional membership inference attacks in identifying if specific text sequences belong to the training data of LLMs. The authors highlight the inadequacies of MIAs and propose a novel dataset inference method. This method focuses on detecting entire datasets used in model training, rather than individual strings, combining multiple MIA metrics to accurately distinguish between training and test datasets with statistically significant results and no false positives. This approach promises to enhance the legality and ethicality of training LLMs.", "questions": "N/A", "limitations": "The authors adequately addressed the limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a compelling argument for shifting the focus from individual string-based MIAs to dataset-based inference methods. \n2. The paper is very well written", "weaknesses": "1. The author demonstrates the failures of MIA methods by assessing them on the training (member) and validation (non-member) splits of the Pile dataset. However, it is unclear whether the validation set is thoroughly decontaminated from the training data. The deduplication method used in the original Pile validation data is quite loose. There is a potential issue that non-member examples might still share high n-gram overlap with the member examples, complicating MIA effectiveness. \n\n2. The proposed approach may lack sufficient novelty: it essentially builds on existing MIA methods in two main ways: (1) by combining various MIA metrics into an ensemble and (2) by extending their application to the distribution of examples. These modifications appear to be straightforward extensions of current MIA methods.\n\n3. The proposed method requires validation data sampled from the same distribution as the test data. However, obtaining such labeled data may not be very practical", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and are happy to see that you found our work to pose a compelling argument for transitioning toward Dataset Inference, and found the paper well-written. We acknowledge your concerns and attempt to respond to them line by line below:\n\n### Re: Decontamination of Validation Set \n\nThank you for your comment. We understand the concern regarding potential contamination in the validation set. To address this, we have conducted additional decontamination using a stricter deduplication method that ensures minimal overlap between the training and validation sets. Specifically, we applied a more rigorous n-gram filtering process to eliminate any shared sequences between these sets. Here are a few key points regarding decontamination: \n\n1. **Reference to the Pile Paper**: The original Pile paper ([Gao et al., 2021](https://arxiv.org/pdf/2101.00027)) acknowledged challenges with deduplication and contamination between training and validation sets. Despite their efforts, complete decontamination is difficult to achieve, as some overlap in n-grams is inevitable because of domains being identical (for example, arxiv headers). \n\n2. **Impact of Contamination on Dataset Inference**: If there were any contamination, it would only make dataset inference harder, as the validation set would resemble the training set more closely. Therefore, high n-gram overlap should lead to worse, not better, results. However, our method succeeds in distinguishing between training and validation datasets despite these challenges, indicating the robustness of our approach at solving an even harder problem. \n\n3. **N-gram Overlap**: We agree with your point about n-gram overlap. A concurrent paper ([Duan et. al., 2024](https://arxiv.org/pdf/2402.07841)) independently found that there is approximately a 30% overlap in n-grams between Wikipedia and arXiv datasets. This study also indicated that non-members with lower n-gram overlap are more distinguishable by existing MIAs. This supports our claim that even with potential contamination and the inherent challenges of deduplication, our dataset inference method remains effective. \n\n### Re: Novelty of the Proposed Approach\n\nWe appreciate your observation regarding the novelty of our approach. Here are a few key points that highlight the innovative aspects of our method:\n\n1. **Achieving a Previously Near-Impossible Goal**: We make the previously near-impossible goal of membership inference (MI) achievable. This is a significant feat that opens up new possibilities in the field of detecting training data.\n\n2. **Learning which MIAs to Use**: Our method involves \"learning\" which MIAs to use, a critical step that enhances the accuracy and effectiveness of our approach. This was crucial to overcome the biggest technical challenge of our work, that is, most MIAs were **worse than random**.\n\n3. **Operationalizing the Framework**: Developing the framework required that we address practical considerations, such as how to handle victims who naturally retain drafts or IID sets of their work.\n\nIn summary, the main contribution of our work is not just in the specific techniques but in proposing a comprehensive framework and advocating for a shift towards dataset inference as a field of study ripe for modern day generative models.\n\n\n### Re: Practicality of Obtaining Labeled Validation Data\n\nWe acknowledge that obtaining labeled validation data sampled from the same distribution as the test data can be challenging. Indeed, this is the number one open problem that our work creates for future research. That said, a significant part of this work was spent in formalizing a framework. Here are quite a few scenarios where the IID setting should roughly happen in practice:\n\n- **Editorial Process**: Thousands of NYT articles reach the editor's desk but never get published. These unpublished articles are likely to be more IID compared to two random splits of the Pile dataset.\n- **Book Drafts**: This is also very common with book writers when working on a chapter draft. These drafts would be more IID than random splits of the Pile, in our opinion.\n\nHowever, we truly need a test of IIDness rather than making claims of the same. This would be a really nice (but also hard) area of research for the future. One strong baseline for the same is the recent “Blind MIAs” work by Das et. al. https://arxiv.org/abs/2406.16201. If any pair of datasets can be distinguished based on these blind membership inference attacks, they should be deemed **not** IID. Future work can improve upon this.\n\nSome potential directions for bypassing the IID problem in future work also include:\n\n1. **Using Synthetic Data**: Create IID sets artificially using synthetic data generation techniques.\n2. **Storing Hashes**: Creators may store hashes of IID data at the time of publication to ensure data integrity.\n3. **Proactive Storage**: More proactive work to store generations of artistic work rather than only the final stage.\n\n--- \n\nOnce again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to further discussions and improvements in this evolving field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the positive, encouraging, and constructive feedback. We are pleased that the reviewers recognize the significance of the problem (Reviewer VFio), consider the paper well-written (Reviewers VFio, pZrT, G1MR), and found it enjoyable to read (Reviewer Nmtj). Our work motivated by showing that MIAs for LLMs detect distribution shift rather than actual membership inference was appreciated by all reviewers and described as inspiring (Reviewer Nmtj).

We designed a method to detect if a given dataset was used to train an LLM, a contribution recognized by all reviewers. Ensuring clarity and comprehensibility was crucial (Reviewer VFio), and we are glad this was effectively conveyed. The LLM dataset inference method is robust and does not return any false positives (Reviewers NHN3, pZrT). The experimental results are comprehensive (Reviewer bhaD), and we hope the released code will be a valuable asset for other researchers. Overall, the paper provides a compelling argument for shifting the focus from individual string-based MIAs to dataset-based inference methods (Reviewer G1MR).

Meta-review: Summary (combined from NHN3 and pZrT's excellent reviews)
========================

The paper addresses the problem of Dataset Inference in Large Language Models - namely, the ability to detect whether the given dataset was used by LLM developers in pre-training.

First, authors demonstrate the importance and feasibility of Dataset Inference task in comparison with Membership Inference Attacks. Namely, they show that existing MIA methods cannot derive membership when test examples are sampled from the same distribution as training data due to temporal shifts in data. 

The authors propose a novel dataset inference method that focuses on identifying entire datasets rather than individual data points, reflecting real-world copyright scenarios where authors' works, like books, are used without permission. Their method combines multiple existing MIA metrics to identify training datasets using a linear classifier, achieving statistically significant results with p-values less than 0.1 and minimizing false positives. The paper highlights the method's robustness through extensive testing on the Pythia models and the Pile dataset, providing a more accurate approach to dataset attribution and addressing the shortcomings of prior MIAs.  The algorithm requires only 1000 data points from the whole dataset to discover if it was used for training.

The proposed method is tested on several PILE subsets, using the Pythia model family as the base model. Authors demonstrate that the efficiency of the proposed method increases with the growth of the model size.

Metareview
=======================

The reviews have consistent direction (if not magnitude): everyone liked the work.  The identified weaknesses are: lack of novelty and lack of experiments to show generalization.

I disagree that using existing MIA metrics lacks novelty, as the result is important and it's a feature (not a bug) to use prior work in a straightforward way.  I also cannot hold the contemporaneous ArXiv submission against this submission, so I'm going to ignore this weakness in making my decision.

What is more complicated is essentially only testing on one scenario.  Reviewer pZrT encouraged the authors to try their approach on the Olmo dataset, but the back and forth in the discussion phase shows that this is not straightforward.  The fact that it's so hard to run these experiments, however, is not the fault of the authors but rather the fault of the current ecosystem of LLMs.  The lively discussion failed to raise another viable alternative.  As I struggle to write this paragraph, this makes me think better, more robust MIA approaches are even more needed: we simply have no accountability for what data are used, and this approach can help address that.

So having talked myself out of the two primary weaknesses raised in the reviews, my only option is to strongly argue for acceptance.  This is a method that is timely, effective, and relatively simple.  It should be presented at NeurIPS.
Decision: Accept (poster)