A Statistical Approach for Controlled Training Data Detection
Authors: Zirui Hu
Yingjie Wang
Zheng Zhang
Hong Chen
Dacheng Tao
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Training data detection
Knockoffs
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Detecting training data for large language models (LLMs) is receiving growing attention, especially in applications requiring high reliability. While numerous efforts have been made to address this issue, they typically focus on accuracy without ensuring controllable results.
To fill this gap, we propose **K**nockoff Inference-based **T**raining data **D**etector (KTD), a novel method that achieves rigorous false discovery rate (FDR) control in training data detection. Specifically, KTD generates synthetic knockoff samples that seamlessly replace original data points without compromising contextual integrity. A novel knockoff statistic, which incorporates multiple knockoff draws, is then calculated to ensure FDR control while maintaining high power. 
Our theoretical analysis demonstrates KTD's asymptotic optimality in terms of FDR control and power. Empirical experiments on real-world datasets such as WikiMIA, XSum and Real Time BBC News further validate KTD's superior performance compared to existing methods.

Published as a conference paper at ICLR 2025
A STATISTICAL APPROACH FOR CONTROLLED TRAIN- ING DATA DETECTION
Zirui Hu', Yingjie Wang', Zheng Zhang', Hong Chen', Dacheng Tao GGneeaatvve AI Lab, College of Computing and Data Science, Nanyang Technological University ?College of Informatics, Huazhong Agricultural University
zirui. huêntu. edu. sg, zhengzhang@@al syinggiewangg201 dacheng. a @ com chenhêmail hzau. edu. cn
uste. edu.cm
ABSTRACT
Detecting training data for large language models (LLMs) is receiving growing attention, especially in high-reliability applications. While numerous efforts have been made to address this issue, they typically focus on accuracy without ensur- ing controllable results. To fill this gap, propose Knockoff Inference-based Training data Detector (KTD), a novel method that achieves rigorous false dis- covery rate (FDR) control in training data detection. Specifically, KTT generates synthetic knockoff samples that seamlessly replace original data points without compromising contextual integrity. A novel knockoff statistic, which incorporates multiple knockoff draws, is then calculated to ensure FDR control while main- taining high power. Our theoretical analysis demonstrates KTD's s asymptotic op- timality in terms of FDR control and power. Empirical experiments on real-world datasets, such as WikiMIA, XSum, and Real-Time BBC News, further validate KTD's s superior performance compared to existing methods.
INTRODUCTION
Large language models (LLMs) have achieved remarkable performance across a wide range of nat- ural language processing (NLP) tasks, including machine translation (Wong al. 2023), code com- pletion (Chen et al. 2021 Li et al. 2022). and question answering (Dong et al. 2024 Li et al., 2024a). This success is largely driven by the use of massive language corpora, often reaching the triliiontooenn scale (Computer 2023). While such extensive datasets equip LLMs with broad knowl- edge and strong text generation capabilities, they may also contain private information (Carlini al., 202T) or copyrighted content (Chang et al.) (023) collected from the Internet, leading to potential risks. For instance, LLMs can inadvertently memorize and reproduce sensitive information when prompted with carefully crafted inputs, posing significant threats to privacy and intellectual property rights (Carlini et al. 202T)
To address this issue, recent studies have explored methods for detecting training data within LLMs (Shi et al. 2023 Golchin & Surdeanu (003).. However, these approaches typically frame the prob- lem as binary classification task, focusing solely on distinguishing between training and training samples with high accuracy. We argue that accuracy alone is insufficient, particularly in cases involving copyright violation detection. For instance, if copyright holder seeks legal ac- tion against a technology company for unauthorized use of proprictary content (Grynbaum & Mac 2023), it is critical to ensure that the majority of identified training samples were genuinely used by the company for training. False detections in such cases could lead to unwarranted legal con- sequences. Therefore, beyond accuracy, controlling the false discovery rate (FDR) (Benjamini & referred to as the false positive rate in binary classification is essential for training data detection and should not be overlooked.
Hochberg
In this paper, we study the problem of detecting training samples from LLMs with controllable FDR. Specifically, given a set of text samples, our goal is to determine whether the model has been trained on them while ensuring that the proportion of falsely identified training samples (i.e.,
"Corresponding authors

---

Published as a conference paper at ICLR 2025
non-training samples mistakenly classified training samples) remains within predefined bound. Inspired by controllable variable selection, we propose a Knockoff Inference-basee Training Data Detector (KTD), which treats the problem of detecting training samples variable selection, thereby enabling the use of knockoff inference's (KI) robust capacity in FDR controlling. KI generates knockoff variables corresponding each original variable and computes knockoff statistics for each pair of original and knockoff variables, which are then used to identify relevant variables. Building on the KI paradigm, KTD operates in two stages. In the first stage, KTD generates knockoffs that preserve the semantics of the original text samples, ensuring they can substitute for the originals without altering the context. In the second stage, KTD computes knockoff statistics to capture differences between the original text samples and their corresponding knockoffs, using these differences as indicators to identify training samples.
instance of relevant
In KTD, fundamental component for successful detection is the construction of the knockoff statis- tic. While the knockoff statistic from the vanilla KI method can be directly applied to KTD, has significant drawback: the vanilla KI method uses only single draw of the knockoff variable to calculate the knockoff statistic W,. This approach results in a high variance in Wj due to the in- herent randomness of the knockoff process, making it difficult to distinguish between training and non-training samples. Such indistinguishability can lead to overly conservative detection results, excluding too many true training samples achieve the desired FDR control. As result, the detection efficiency-measieee identified)-ss significantly compromised.
by power (i.e., the proportion of actual training samples correctly
To address this issue, KTD adopts a novel approach for calculating the knockoff statistic which utilizes multiple draws of the knockoff variable. This design effectively reduces the variance of the knockoff statistic, making it more centralized and thereby enhancing the separability between training and non-training samples. Specifically, KTD draws m realizations of the knockoff variable and computes the knockoff statistic W, by comparing the importance score of the original variable to the average of the importance scores of these realizations.
To theoretically justify KTD, we first demonstrate that the knockoff statistic of KTD, i.e., WKTD retains the symmetric property, ensuring that it can control the FDR just like the vanilla KI. Further- more, we distinguish KTD from the vanilla KI by proving its asymptotically optimal property: as m approaches infinity, the power converges to 1 and the FDR converges to O.
We empirically evaluate KTD using three popular large language models on three real-world datasets, including the established benchmark WikiMIA. The experimental results demonstrate that KTD not only achieves the desired FDR control without relying on a validation set, which exist- ing methods depend on but also exhibits significantly higher power compared to vanilla KI when achieving similar FDR levels.
To summarize, contributions are follows:
We address the problem of detecting training data from LLMs with FDR control through the per- spective of knockoff inference and propose knockoff inference-based detecting method KTD.
2. We theoretically justify our proposed KTD from two aspects. Firstly, we prove that the knockoff statistic in KTD possesses the symmetric property, which is essential for effective FDR control. Secondly, we show that KTD exhibits asymptotically optimal properties, distinguishing it from the vanilla KI.
3. Our experimental analysis validates the effectiveness of KTD by demonstrating its ability to achieve the desired FDR level with competitive power.
2 RELATED WORK
2.1 TRAINING DATA LEAKAGE IN LLMS
Memorization in language models, key aspect of training data leakage, has been widely studied. Research such as Kandpal et al. 2 Carlini al. (202T (022b) Zeng al. (2024) examines the memorization behaviors of language models, offering insights into their underlying mechanisms. However, these studies do not propose practical methods for detecting training samples.
In the context of LLMs, other works (Brown et al. 2020 Wei et al. 2021 Du et al. 2022) explore the potential impact of training data leakage evaluation results. To ensure reliable assessments,
2

---

Published as a conference paper at ICLR 2025
Table I: Algorithmic properties
Property
FX Knockoff
MX- MooKooo Contamination Test
Mink%
Time Traveling
KTD (Ours)
Metadata-free
X
Threshold-free
X
FDR Control
X
X
FDR unalysis
X
X
X
Power analysis
X
X
these studies exclude test samples that share n- n-gram overlaps with any data used during pre-training. These approaches requires access to pre-training datasets, making it infeasible to detect training samples without support from the model provider.
To evaluate training data leakage without access to the training dataset, various methods have been proposed for detecting training samples. For example, Gooccin & Surdeanu (2023) prompt model to generate completions using two types of instructions and identify training samples by comparing the resulting texts. [Shi et al. (2023) assume that non-training texts are more likely to contain outlier tokens that induce significantly higher loss and propose detecting training samples based on top-k token log probabilities. While these methods have shown empirical effectiveness, they do not offer theoretical guarantees for controlling the FDR. Considering this, some studies leverage statistical methods ensure controlled outcomes. Oren et al. (2023) assume sample exchangeability in non- training datasets and apply hypothesis testing to detect data contamination, Dekoninck et al. (2024) define contamination based on performance differences between the tested and reference models, relying on a predefined threshold ō to determine significance. However, these methods are mainly designed for dataset-level contamination detection, which differs from our scenario.
Membership inference attacks (MIAs) have also garnered significant attention in the context of train- ing data detection. Similar to our approach, |attern MMatter al. (2023); (uu al. (2024) generate neighbor- ing samples that resemble the original ones and compare each sample with its generated neighbors to identify training data. However, these methods do not explicitly aim control the FDR and rely primarily on empirical observations rather than rigorous theoretical foundations. Carlini et al.) (2022a); Mireshhhallah et al. (2022), consider FDR as an evaluation metric their paper. However, these methods are not inherently designed for FDR control and require either training a large number of shadow models or accessing the distribution of data that was not used to train the target model. These constraints make them impractical for LLM scenarios.
2.2 KNOCKOFFS
The knockoff framework was first introduced by Barber & Candès (0015) as a data-driven approach to controlling the FDR in variable selection for sparse regression problems. It was later extended to high-dimensional regression by Candes et al.] (2018). Over time, knockoff inference has been adapted for various applications, including multi-task regression (Dai & Barber, (016)), outlier de- tection (Xu et al.) 2016), and sample selection (Wang et al. 2024). To the best of our knowledge, are the first apply knockoff inference to LLM training data detection.
To highlight the novelty of our proposed method, KTD, we compare with existing works from both the knockoff and training data detection literature. Specifically, we contrast KTD with FX-Knockoff (Barber & Candès 2015) and MX-Knockoff (Candes et al. 2018)) from the knockoff literature,
as well as Contamination Test (Oren et al. 2023), Mink% (Shi et al. 2023), and Time Traveling Golchin & Surdeanu 2023) from the training data detection literature,
summarized in Table I In the table, "Metadata-free" indicates that the method does not require auxiliary information such as dataset names or partition details, while "Threshold-free"" refers to methods that do not rely on heuristically determined thresholds. This comparison underscores the comprehensivenes: of KTD, demonstrating our contributions in both training data detection and knockoff- -asdddd--eddddde- inference.
3 BACKGROUND
Notation We use bold letters to represent vectors of random variables, c.g., X X2...,, Furthermore. let X_, denote the vector resulting from the exclusion of the j-th variable X,, i.e., X (X,}. The independence between two random variables X and Y is sym- bolized as X Y. Let [n] represent the set 2....,,,; for any given set A, [A] denotes the cardinality of A. For two number and b, let a V represent max(o,b). b).
3

---

Published as a conference paper at ICLR 2025
Problem Definition Suppose we have potential training samples X1, X2...... .,, Xn to be tested. For the sake of clarity afterward, we defined Diotal as the index set of these samples, i.., Diotal = [n]. Depending whether sample has been used for training the model, Diotal can be divided into two disjoint subsets: Derain and Doon-traite train- which represent the index of training samples and non-training samples respectively. Given the dataset Protal and the language model fo. our objective is to identify an estimate of Dirain denoted S C Drotal, with FDR bounded by predefined threshold q, while maintaining as high power as possible. Here, the power and FDR are defined as:
S
D Ptrain D Presial V1
18 n Doon train
Power == E
and FDR E
(I)
S8|
Auto-regressive LLMs The goal of auto-regressive large language models is capture the un- derlying language distribution (x(X). They achieve this by predicting the next token in a sequence based on the preceding tokens. Specifically, given sequence X (21..22... .TT). these models represent the probability of X using the chain rule:
Px(X) = II Po T2. 21,1))
where & denotes the parameters of the language model. The parameters 0 are trained to maximize the log-likelihood of sequences in the training dataset.
METHODOLOGY
In this section, we introduce our knockoff inference- basee training data detector, KTD. We begin by illustrating two critical procedures of KTD, which include synthetic knockoff generation and knockoff statistic calculation. Then, the asymptotic analysis for KTD provided.
4.1 KTD: A NOVEL KNOCKOFF-BASEE FRAMEWORK
Knockoff inference method originally designed for selecting variables relevant to certain outputs of interest while controlling FDR. In our settings, we can reformulate our problem as a variable selection problem by treating the training samples {X, jej)ne as relevant variables and model parameter 0 the output of training algorithm Alg which takes training samples input, ie.. 0 = Alg (eD
Motivation
). Here, Deruiu total represents the model's 's entire training dataset, and Derain is subset of it. Through this reformulation, the robust FDR control ability of KI can be utilized in our context. Since in our approach, we treat samples as random variables, we will use these two terms interchangeably in the following text.
Intuitively, the fundamental idea behind the knockoff inference-based method is to identify relevant variables by comparing them with their noisy counterparts, known as knockoffs. As result, the knockoff inference- based method usually involves two critical procedures: first, generating knock- off's for the text samples to be tested; second, calculating the knockoff statistic for these text samples by comparing the scores assigned to them with the scores assigned to their knockoff counterparts. Next, we illustrate how these two stages work vanilla KI and how they are instantiated in KTD.
4.1.1 KNOCKOFF GENERATION
In vanilla KI, knockoffs are typically generated based specific assumptions about the distribution of variables ( [X,)ー=ı= such as Gaussian (Candes et al. 2018). Markov model Sesia et al. (2018) and hidden Markov model (Sesia al. 2018). However, due to the complexity of naturai language, it is challenging to use common distributions to model the relationships between text samples, render- ing existing methods unsuitable for knockoff generation in this context. Consequently, we directly adhere to the fundamental definition of knockoffs:
Definition 1. Model-X Knockoffs, (Candes et al. 2018) Model-X knockoffss for a family of random variables X (X1, X2...., .., X,) are a new family of random variables X (X1, X X,3
satisfying:
1. X x
4

---

Published as a conference paper at ICLR 2025
2. For any C. [n]. (X, X) swap((s (X, X).
Here, (x, =
X1, X2.... Xn), and (X, s)spp(ss) is obtained by swapping X with its corresponding knockoff X3 for all j For example, when 3. (X, X swap((1,33)
22,,X,,33. X2, Xx)-
This definition guides generating knockoff texts. Property (1) implies that the knockoff text should be generated by the model being tested, and property (2) requires that the generated knockoff texts be able to replace the original text samples without altering the overall joint distribution. In other words, the knockoff text should convey the same meaning as the original text but in a different manner. To meet these requirements, we generate knockoffs in the KTD framework using natural language paraphraser, which restates or rephrases the text while preserving its original meaning.
4.1.2 KNOCKOFF STATISTIC CALCULATION
Knockoff Statistic in Vanilla KI After constructing the knockoffs, a test statistic known as the knockoff statistic is calculated for each text sample by comparing the importance of the original text sample with that of its knockoff counterparts. This statistic can be viewed for each text sample and will serve as the basis for training sample selection, which is defined as Definition 2. Knockoff Statistic, (Candes el al. 2018) A knockoff statistic W = (Wi, W2, W.)
relevance measure
a measure of variable importance that satisfies the following conditions:
W depends only on X. X. and 0:
W f(,
(2)
2. Swapping the original variable Xj with its corresponding knockoff X, switches the sign of Wj: i É
W X swap((s) 0) (x,,(XX,),,), if S.
Typically, the calculation of the knockoff statistic of each variable can be decomposed into two steps. First, assign importance scores Zj and 2, to each variable Xj and its knockoff X, respectively, where the importance scores are calculated by pre-defined scoring function T, i.e.,
Z, = T X). 0) and
2, Tj+ ( x), 0).
(4)
Next, calculate the knockoff statistic of j-th sample by
W,=Z, Z- -, Z,.
(5)
Intuitively, the scores Zj and Z, represent the importance of the original sample Xj and its knockoff X,. respectively. positive W, (W, 0) indicates that the j-th sample is more relevant to the model parameter 0 than its knockoff, implying its membership in the training data. Conversely, negative W, (W, < 0) suggests that the j-th sample is more likely to be irrelevant to
During this procedure, the key is to select appropriate scoring function t that can effectively measure the importance of each sample and its knockoff. In our scenario, we aim ensure that samples seen by the model are assigned higher scores, Inspired by works using gradient information for OOD detection (Huang et al., 2021]: Liang et al., 2018), we use the La norm of the model's gradient as the score in KTD, which is defined as:
θ log (X,)
Bog (X,)
Z, =
and
2 =
00
00
where Po(-) represents the probability distribution modeled by the model.
Finally, a threshold determined for thresholding knockoff statistics for training sample detection with FDR control guarantee. This procedure is illustrated follows:
Proposition By choosing the threshold t according to
{ + Hi E [m)
W, < -t) ki W, 2 t =9]
I = arg min
t>o
and setting $ - [j: W, 2 7). the procedure can control the FDR at 4-
5

---

Published as a conference paper at ICLR 2025
Our Calculation of Knockoff Statistic Despite the effectiveness of vanilla KI in controlling the FDR, its knockoff statistic W is prone to high variance due to the inherent randomness in the knock- off generation process. This variability makes it more difficult to distinguish between training and non-training samples, leading to the selection of a more conservative threshold т. Consequently, a larger number of true training samples are excluded maintain FDR control, resulting in signifi- cant power reduction.
To address this issue, we modify the calculation of the vanilla knockoff statistic W2 by considering multiple draws of the knockoff variables X. Specifically, we calculate the knockoff statistic in KTD as follows:
WKID = Zg
(8)
m
where (g0) is the score calculated based on the x the i-th draw of X5. Clearly, W is a special case of WKID when m 1. By taking multiple knockoff draws into consideration, we can reduce the variance of knockoff statistic WKID, thereby enhancing the separability between training and non-training samples.
Next, we show that WKTD can also select the appropriate threshold for FDR controlling as W, in Proposition 1 Assumption 1. For any j E D the knockoff statistic WKID defined in Equation are independent with each other.
We first give the independence assumption of W following Nguyen et al. 2202
Next, we illustrate the symmetric property of WKTD:
Lemma 1. WKTD associated with irrelevant samples symmetrically distributed around 0, i.e., P(W < =tt)= P(WKTD >t) for any t and j E Dnonntrain
=
This Lemma is empirically validated by our experiments. For details please refer to the third part of our experimental results.
This Lemma, combined with the independence assumed in Assumption implies that the num- ber of non-training samples whose WKRTD equals the number of non-training samples whose WKTDD < O. This conclusion allows the use of the right-hand side of Equation 10 as an upper bound for FDR, thereby providing an FDR control guarantee.
Consequently, we can select training samples while controlling the FDR using WKTD through a procedure similar to that described in Proposition [ defined as follows:
Proposition 2. Assume (WKKD)J L are independent with each other, by choosing the threshold according to
Jj=1
1 1+ Kb E [m] WKTD I -tH
min t>0
[n] 2 20Hv1 Sq
(10)
and setting S = (j (f::WTT WKTD r), the procedure can control the FDR S q.
4.2 THE ASYMPTOTIC OPTIMAL PROPERTY OF KNOCKOFF STATISTIC IN KTD
Here, we provide analysis inspired by Zhao et al. (202) to illustrate the asymptotic optimality of FDR and power in KTD. We begin by stating an assumption which these theorems rely.
Assumption 2. For any j E Dtrain. we have E[W]KT]] >
Remark 1. This assumption ensures that, on average, training samples will have higher importance scores than their knockoff counterparts. This is reasonable because sample that has been seen by the model will induce fewer updates (thus higher Z,) compared to its knockoff. Like Lemma 7 we provide the empirical validation of this assumption in experiment section.
Theorem 1. Assuming Assumption 2 holds, the variable selection procedure described in Proposi-
tion 2 satisfies
S Dersiul Dtrain
Power =
+ 1 as
m + DO.
(11)

---

Published as a conference paper at ICLR 2025
Theorem 2. Assuming Assumption 2 holds and the threshold found by Proposition 2 is not equal to O, the variable selection procedure satisfies
18
D mon train
FDR E
+ 0 as m oc.
(12)
IS
Remark 2. Intuitively, as m approaches infinity, the values of WKID will become increasingly centralized around their expectations. Consequently, given Assumption 2 the WKTD values corre- sponding training and non-training samples will form distinct clusters. This separation allows the KTD method to identify a threshold that optimizes both FDR and power.
EXPERIMENTS
In this section, we conduct extensive experiments to validate the empirical efficacy our proposed method. We begin by testing the effectiveness of our method in terms of FDR control. Following that, we empirically validate the symmetric property of our proposed knockoff statistic WKTD" Next, we investigate the influence of m, the number of knockoff draws, on FDR control performance and the trade-off between power and FDR. Finally, we conduct experiments using Pythia models with different numbers of parameters to examine how model size affects performance. The code for our experiments is available at htttttss///gthihi wwt.ccocoohhheeeeeeee 9t9t//h222199/KID
5.1 SETUP
Baselines We first select several classic baselines from MIA literature for comparison. Specifi- cally, they include LOSS (Yeom et al. 2018), which uses the auto-regressive loss of a sample to determine whether it has been seen during training; MinK % (Shi et al. 2023), which takes the average loss of the top k% tokens with the highest loss as the basis for detection and methods that compare sample's loss its zlib compression entropy (Zlib (Carlini et al., 2021)), its loss after lowercasing (Lowercase (Carlini et al.] (02T)), and its loss from a smaller reference model (Ref (Mireshghallah et al., 20222). Then, we compare our method with vanilla KI, which is instance of our method when setting m = 1.
Models We adopt three popular large language models to evaluate our detection algorithm: GPT-2 (137M parameters) (Radford al. 2019), Pythia (1.4B parameters) (Biderman et al. (023), and GPT-Neo (1.3B parameters) (Black et al. 2021). Experiments for larger models are available in Appendix D For baseline Ref, we employ Distilled-GPT2 (Sanh al., 2019), Pythia-410mm and 1PT-Nco-125m as reference models for the three aforementioned main models, respectively. To generate reliable knockoffs for text samples, we use a paraphraser (Vladimir Vorobev 2023) with the highest downloads on Hugging Face. This paraphraser is based on the T5-base model and fine- tuned with paraphrased texts generated ChatGPT. Throughout our experiments, we use the model checkpoints provided by Hugging Face]]
Dataset We conduct our experiments on three datasets: WikiMIA (Sh et al. 2023) includes texts collected from Wikipedia events. The dataset is separated into two disjoint parts: one corresponding to events happening before 2017 and the other to events happening after 2023. These two parts are used as training samples and non-training samples, respectively. XSum (Narayan et al., 2018) includes summaries of BBC news articles. We select the test set of this dataset and randomly separate it into two parts, corresponding to training and non- non-training samples. BBC Real Time (Li et al.) 2024b) includes BBC articles from January 2017 to August 2024. Following the process in Shi et al. (2023), we use the articles published 2017 as training samples and articles published in 2024 as non-training samples. To evaluate our method, fine-tune the models using the training parts of these datasets while ensuring that the non-training parts remain unseen by the models.
Computation and Hyperparameters All the experiments are run with a single NVIDIA Tesla V100 32GB GPU and 10-core Intel Xeon (Skylake IBRS) CPU. When the model is too large, use 16-bit quantization to fit the model into GPU memory. Unless explicitly stated, we fix m = 10 for all experiments. All codes implemented with Pytorch (Paszke et al. 2019).
hutps//huggingffac.co/

---

Published as a conference paper at ICLR 2025
Table 2: Comparison between KTD and baselines. For each dataset and model, if any methods achieve FDR control, the one with the highest power among them will be bolded. Otherwise, the method with the best FDR will be bolded.
WKKMIA
XSum
BBC Real Time
GPT-3
Pythia
GPT-Noo
GPT-2
Pythia
GPT-Neo
GPT-2
Pythia
GPT-Neo
FDR
FDR
EDR
Power
FDR
Power
FDR
FDR
FDR
Power
Power
LO055
0.179
0.925
0.125
0.999
0.145
0.956
0,153
0.150
0.163
0.938
0.135
0.843
0:135
0,990
0.122
0.999
0.125
0.958
MinKS
0..09
0.232
0.195
0.957
0.120
0.749
0.311
0.004
0.165
0.660
0.200
0.29%
0.345
0.279
0.103
0.958
0.130
0.964
Zlib
0.000
0.060
0.187
0,984
0.096
0.881
0.237
0.499
0.154
0.380
0.164
0.801
0.122
0.800
0.142
0.997
0.112
0.993
Lowercase
0.634
0.995
0.485
0.991
0.467
0.985
0.570
0.086
0.019
0.554
0.081
0.507
0.005
0.495
0.016
0.671
0.017
Ref
0.167
0.381
1,000
0.487
0.114
0.382
0.369
1,000
0.99%
Vanilla
0.207
0.194
0.198
0.9%%
0.973
0.109
0.161
0.05
0,873
XTD (Ours)
0.397
0.230
0.193
0.958
1,000
0.945
0.101
0.301
0.349
0.071
0,980
0,973
1001
0704
0.71
11.31
0.1
0.2
03
0.4
0.5
06
07
0.
0.1
02
03
0.4
055
96
07
25
0.
0.1
0.2
03
0.4
0.5
06
09
GPT-8
GPTS
GPTS8
Pytits
1020
Pytris
Pytiis
GPT-Nec
GPT-Ncc
GPT-Neo
Bound
0.30
0.00
0.9
01
02
03
041
05
06
07
09
01
03
03
0.1
65
0.6
07
25
03
01
02
00
0.1
65
06
09
(a) WikiMIA
(b) XSum
(c) BBC Real Time
Figure 1: The FDR control results on three datasets. We vary the FDR bound g from 0.05 0.95 and calculate its corresponding FDR and power. Each subplot represents results on a dataset and each line in the subplots represents the results of model. To clearly visualize the bound, we also plot the red line (y x) in each subplots. If a model's FDR is bounded, its corresponding line should be below the red line.
5.2 RESULTS
The Effectiveness of FDR Control with fixed q We set the FDR bound to q = 0.1 and present comparative analysis of our method against the baselines in Table 2 Since the baseline meth- ods LOSS, MinK%, Zlib, and Lowercase only produce confidence scores for training data mem- bership inference, they require a validation set to determine an appropriate threshold. To accommo- date this, we sample validation set of 100 instances and select thresholds that achieve the highest power while maintaining a bounded FDR on the validation set. For the Ref baseline, which relies on approximate this distribution by combining
"general distribution" to determine the threshold, all non-tre con-training samples from the three datasets. From Table 2 we observe that our method, KTD, effectively controls the FDR on the XSum and BBC Real-Time datasets while maintaining relatively stable performance across different datasets and models. In contrast, baselines such as Zlib, MinK %, and Lowercase occasionally exhibit extremely poor FDR or power, suggesting that their effective- ness is highly dependent on the choice of the validation set. Although some baselines achieve more favorable results WikiMIA, argue that this comparison is entirely fair. These baselines require access to validation set with ground truth membership labels or an accurate distribution of non- non-training samples resources that may not be available in real-worke scenarios.
The Effectiveness FDR Control under Varying q We vary q and plot the corresponding power and FDR in Figure The results show that method effectively controls the FDR in most cases. Although in certain cases (e.g., three models on WikiMIA), the FDR is not strictly bounded by g. the corresponding curves closely track the red line, indicating that the FDR remains within 9 plus small constant. For a more detailed analysis of these cases please refer to Appendix B
The Symmetrie Property W To empirically validate Lemma and Assumption we plot
the distribution of W W-KTD calculated on the BBC Real Time dataset in Figure 2 From the figure, symmetrically
make three observations. Firstly, the WKTDD of non-training samples distributed around 0. which is aligned with Lemma [ Secondly, the expectation of non-training samples' WKTD is greater than 0, illustrating the validity of Assumption which Theorems andd]
8

---

Published as a conference paper at ICLR 2025
Trinisa sucapk Nanimining semple
10
Trisisg suupls
Nan-iiniing seaplo
Xas-nasiia smppl Tratiig pet Nie tndring pull
Trasing sarple
1.0
Tratiin put Now teiiing yar
Veitiigg pat Nos joi
16:0
100
1E-
142
1:00
104
1:04
101
3:00
1:01
The valaaaad (e
The salan of 11
The valae of ti
(a) GPT-2
(b) Pythia
(c) GPT-Neo
Figure 2: The distribution of our knockoff statistic WKTTD
1.00
0.18-
GPT-2
Pythia
0.16
GPT-Neo
Power
FDR
0.14
0.80
0.12
GPT-2
Pythia
GPT-Noo
0.20
135
10
30
50
100
135 10
30
50
100
(a) Relationship between power and m
(b) Relationship between FDR and m
influence on power and FDR. We fix the bound q at 0.1 and vary m in {1, 5, 10, 30, 50, 100} to calculate the power and FDR for each model. The red line in the figure represents the FDR bound given by q. Dots below the red line indicate successful bounding.
Figure 3:
rely on. Lastly, larger models (Pythia and GPT-Neo) exhibit better separability between training and non-training samples. This consistent with expectations, as larger models have a higher probability of memorizing their training data (Carlini et al. 2022b), thereby showing different behavior for training versus non-training samples.
The Influence of the Number of Knockoff Draws In KTD, we use multiple knockoff draws to compute the KTD statistic WKKTD Here, we justify this design by examining two key questions: (1) how m affects FDR control and power, and (2) how m influences the power-FDR trade-off. Note that vanilla KI is special case of KTD with m 1.
For the first question, plot power and FDR against in Figures 3al and 3b respectively. Sev- eral observations emerge from these figures. First, KTD provides better FDR control than vanilla KI. Specifically, vanilla KI fails to control the FDR for GPT-2, whereas KTD successfully bounds it when m 2 10. Second, increasing m benefits both FDR control and power-boll both metrics im- prove consistently m grows. Finally, for Pythia and GPT-Neo, power approaches 1 while FDR approaches as m increases, validating the asymptotic optimality stated in Theorems [nd
To address the second question, plot FDw pareto curves for different values in Figure The figure shows that as m increases, the curves shift toward the upper left corner, indicating an improved power-FDR trade-off. Even small values, such m 3, yield significant improvements. This demonstrates that multiple knockoff draws substantially enhance the balance between power and FDR. Additionally, for GPT-2, we observe an unusual trend when m is small (1 to 5): some- times, FDR decreases as the target FDR level increases. We hypothesize that this is due to the high instability associated with small m.
Since computation time is primarily dominated by the gradient norm calculations for knockoffs, Figure EegureBaaso allso illustrates the trade-off between computation time and performance. From the figure, we find that m = 10 strikes an optimal balance, achieving strong performance within a reasonable computation time.
The influence of Model Size We evaluate the FDR in terms of the power-FDR trade-off across three datasets using different-sized Pythia models (440M, 1B, 1.4B, 2.8B). Figure 5 shows clear

---

Published as a conference paper at ICLR 2025
108
000
100
033
102
0.30
0.30
0.15
2.55
0:35
035
0.10
0.15
6:30
333
015
6:00
035
0.30
FOR
(a)
GPT-2
(b) Pythia
(c) GPT-Nco
Figure 4: Trade-off between FDR and power under different m. For each m, we vary the FDR bound z and calculate the corresponding trade-off between power and FDR. Each subplot represents model and each line in the subplots represents the trade-off curve under certain m. The closer the curve is the upper left corner, the better the trade-off between power and FDR.
1.01
110
1001
100
105
0000
0110
600
400M
18
000
18
1B
1:4
0.00
815
133
850
0.5
815
0.50
0.5
0:30
6.55
676
655
0.50
0.55
125
PDR
PDR
FDR
(a) WikiMIA
(b) XSum
(c) BBC Real Time
Figure 5: Trade-off between FDR and power under different model sizes. For each model size, varying FDR bound is applied to compute the trade-off between power and FDR. Each subplot presents the results on a different dataset, with each line representing the trade-off curve of a model. Curves closer to the upper-left corner indicate a more favorable balance between power and FDR.
trend that the trade-off improves as model size increases. This aligns with our expectations, as larger models more likely memorize training data, making it casier distinguish between training and non-training samples. Moreover, models with sizes larger than billion parameters exhibit relatively high power even when strict FDR bound is imposed. This observation suggests that billion parameters may serve as a threshold, beyond which models can easily memorize samples from these three datasets, thereby making the distinction between training and non-training samples exceptionally clear.
CONCLUSION
In this paper, tackled the critical issue of detecting training data for LLMs with a focus on controlling FDR and introduced a novel knockoff- basedd method, KTD. KTD instantiates the KI framework in the context of training data detection and employs novel calculation method that leverages multiple knockoff draws to address the high variance of the knockoff statistic in vanilla KI. To support KTD, we provided theoretical guarantees for KTD, demonstrating that effectively controls the FDR but also possesses asymptotic optimal properties. Our empirical evalu- ations on three datasets further validated the efficacy of KTD, showcasing superior performance in terms of FDR control and the power-FDR trade-off compared to existing methods.
only
LIMITATIONS
The limitations of our method primarily stem from two factors. First, the effectiveness of our ap- proach depends on access to the gradients of LLMs, which may not be available for certain propri- etary models where gradient information is inaccessible. This dependency also limits the applica- bility of our method to tasks such as paraphrased text detection. Second, our method assumes the availability of high-quality paraphraser to generate knockoff samples. This reliance on paraphraser quality introduces potential bottleneck in achieving optimal performance. In the future, will explore the possibility of designing a framework for FDR control that relies solely on logits or even the output text of LLMss.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENTS
This project is supported by the National Research Foundation, Singapore, under its NRF Profes- sorship Award No. NRF-P2024-001.
REFERENCES
Rina Foygel Barber and Emmanuel Candès. Controlling the false discovery rate via knockoff'. The Annals of statistics, pp. 2055-2085, 2015.
Yoav Benjamini and Yosef Hochberg. Controlling the False Discovery Rate: A Practical and Pow- erful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Method- ological), (7(11)289-33))) 300, 1995. ISSN 2517-6161. doi: 10.1111/.2577-66611999..t
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In Proceedings of the 40th International Conference on Machine Learning, pp. 2397-2430. PMLR,Jul 2023.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https:://oi.
btgg/10.5811/ zewddo.eodooooooooo// 5297715
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Emmanuel Candes, Yingying Fan. Lucas Janson, and Jinchi Lv. Panning for Gold: Model-X Knock- offss for High-dimensional Controlled Variable Selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551-555 2018.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021. ISBN 978-1-939133-24-3.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song. Andreas Terzis, and Florian Tramèr. Mem- bership Inference Attacks From First Principles. In 2022 IEEE Symposium on Security and Pri- vacy (SP), pp. 1897-1914, May 2022a. doi: 00.1009/S44221222222936666
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. In The Eleventh Interna- tional Conference on Learning Representations, September 2022b.
Kent Chang. Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak. Memory: An Ar- chaeology of Books Known to ChatGPT/GPT44. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference Empirical Methods in Natural Language Process- ing. pp. 7312-7327, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.om hwtp::33.emmmmmmma...mmmmmm
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Blizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex
11

---

Published as a conference paper at ICLR 2025
Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code, July 2021.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https:// htt.://ww....ooootteeeeeooooppaa ReedP jama-Data
Ran Dai and Rina Barber. The knockoff filter for FDR control in group-Sparse and multitask regres- sion. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1851-1859. PMLR, June 2016.
Jasper Dekoninck, Mark Niklas Mueller. and Martin Vechev. ConStat: Performance-Bass Contam- ination Detection in Large Language Models. The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024.
Junnan Dong, Qinggang Zhang. Huachi Zhou, Daochen Zha, Pai Zheng, and Xiao Huang. Modality- Aware Integration with Large Language Models for Knowledge-Bad -aaged Visual Question Answer- ing. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2417- 2429, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
Nan Du Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient Sealing of Language Models with Mixture-of-Expera ings of the 39th International Conference on Machine Learning, pp. 5547-5569. PMLR, June 2022.
Proceed-
Wenjie Fu. Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. A Probabilistic Fluctuation based Membership Inference Attack for Diffusion Models, June 2024.
Shahriar Golchin and Mihai Surdeanu. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. In The Twelfth International Conference on Learning Representations, October 2023.
Michael M. Grynbaum and Ryan Mac. The Times Sues OpenAl and Microsoft Over A.I. Use of Copyrighted Work. The New York Times, December 2023. ISSN 0362-4331.
Rui Huang, Andrew Geng, and Yixuan Li. On the Importance of Gradients for Detecting Distribu- tional Shifts in the Wild. In Advances Neural Information Processing Systems, volume 34, pp- 677-689. Curran Associates, Inc., 2021.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Privacy Risks in Language Models. In Proceedings of the 39th International Conference on Machine Learning, pp. 10697-10707. PMLR, June 2022.
Xiaoxi Li, Yujia Zhou, and Zhicheng Dou. UniGen: Unified Generative Framework for Retrieval and Question Answering with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(8):8688-866 8696, March 2024a. ISSN 2374-3468. doi: 10.1609/aaai. v3818.28714.
Yucheng Li, Frank Guerin, and Chenghua Lin. Latesteval: Addressing data contamination in lan- guage model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18600-18607, 2024b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Auumee Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-Level Code
12

---

Published as a conference paper at ICLR 2025
Generation with AlphaCode. Science, 378(6624) Deceee December 2022. ISSN 0036-8075, 1095-9203. doi: 10.1 2..///icccec...aggll..
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing The Reliability of Out-of-distribution Image De- tection in Neural Networks, In International Conference Learning Representations, February 2018.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf. Mrinmaya Sachan, and Taylor Berg- Kerg-Kirkpatrrck. Membership Inference Attacks against Language Models via Neighbourhood Comparison. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Findings of the Association for Computational Linguistics: ACL 2023, pp. 11330-11343, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v//0023 fin.i..s-acl..19..
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Herg-Kirkpatrick, and Reza Shokri. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8332-8347, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 11/88633/1122222 https////1222.mmmllll. emnlp- mai.l-m-ai..5....
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. (0.18653/v1/D18- 1206.
Tuan-Binh Nguyen, Prrome-Alexis Chevalier, Bertrand Thirion, and Sylvain Arlot. Aggregation of Multiple Knockoffs. In Proceedings of the 37th International Conference on Machine Learning, pp. 7283-7293. PMLR, November 2020.
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Prov- ing Test Set Contamination in Black-Box I Language Models. In The Twelfth International Con- ference on Learning Representations, October 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. htthttss::// arxiv. org//oos/////ooooooooooooooo 1911/91111.. 01703
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurlPS EMC2 Workshop, 2019.
M Sesia, C Sabatti, and F. J Candès. Gene hunting with hidden markov model knockoffs. Biometrika, 106(1):1-18, August 2018 ISSN 1464-3510. doi: 10.1093/biomet/ayy03.. URL hntp:://t// Aloi.org/00 1 1093 wttp:///////ee//////ee
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke 7ettlemoyer. Detecting Pretraining Data from Large Language Models. In The Twelfth International Conference on Learning Representations, October 2023.
Maxim Kuznetsov Vladimir Vorobev. A paraphrasing model based on chatgpt paraphrases. 2023.
Yikai Wang, Yanwei Fu, and Xinwei Sun. Knockoffs-SPR. Clean Sample Selection in Learning With Noisy Labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5); 3242-3256, May 2024. ISSN 1939-3539. doi: 10. 11100/TAA..222.3333333
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned Language Models are Zero-Shot Learners. In Inter national Conference Learning Representations, October 2021.
13

---

Published as a conference paper at ICLR 2025
Ryan Wong. Necati Cihan Camgoz, and Richard Bowden. Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation. In The Twelfth International Conference on Learning Representations, October 2023.
QianQian Xu, Jiechao Xiong, Xiaochun Cao, and Yuan Yao. False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1282-1291. PMLR, June 2016.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learn- ing: Analyzing the Connection Overfitting. 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268-282 IEEE Computer Society, July 2018. ISBN 978- 978-1-5386-6680-7. doi: 10.1109/CCE:2018.000
Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, and Dawei Yin. Exploring Memorization Fine-tuned Language Models, February 2024.
Xuebin Zhao, Hong Chen, Yingjie Wang, Weifu Li, Tieliang Gong, Yulong Wang, and Feng Zheng. Biroo-Baasd Knockoffs Inference for Controlled Feature Selection. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8))919--919,, June 2022. ISSN 2374-3468. doi: 10. 1609/aaai v3ii. 20905.
A APPENDIX
A.1 THE PROOF OF LEMMA
Lemma 1. WKTD associated with irrelevant samples is symmetrically distributed around 0, i.e.,
P(WKTD < <t)
for any t and j E Dnon-train
(13)
Proof. For simplicity, let w/(0 denote Z₁ 1 Then,
1 w( < <-1)
P(W < -1) P
m
(14)
= =f- Dr
P( = 01, w(() 02
w((m) = am) day doz dam du,
where D. {(aaa,, 02 al.,, 0m)
According to Lemma 3.3 in Candes al.
2018 the signs of w/(0) are independent their magni-
tudes. As result, we can express the probability P(W.") = a1, w.(2) 02..... w(()) am ) as the product of the following two terms:
1. 1(W la1/ (w lazl. ................................ W (m)) |aml);
2. II ((sign(w. Ei), where e; sign(a:) E (-1,1).
Given the symmetric property of standard knockoff statistic W;, P(sign(W/(0) = -1) P(sign(W/(0) = 1). Therefore, switching the sign of w will not af- fect the probability above. Consequently, for any element (a1, a2 .2..... am) in D., there exists cor-
have
responding element (-a1, - -a,,..., am in D' ((a), ab. .., ,nn) m m of =-w} satisfy-
ing ((W, a1, w! .2,..., w.()) am) P(W,") a4, W ag...., w/(w) B...,,W((m) d',))
14

---

Published as a conference paper at ICLR 2025
As result, Equation 14 equals:
P( = ai, W, 02,...... W/( = (m) day du2 dum de
Do
p( (1) =di w,,0) d2 w a'm) da daz ... dam do
D2
J. p( a1, W, 02 w am) da daz dam de
(15)
P
1
W
m
P(W > t).
A.2 THE PROOF THEOREM
Theorem 1. Assuming Assumption 2 holds, the variable selection procedure described in Proposi-
tion 2 satisfies
S Deraial
Power
[Dtrain]
1
Co.
(16)
Proof. The main idea of the proof comes from (Zhao al.] (2022). Theorem 6). For the sake of
clarity of the article, provide the complete proof here. Let &y denote the expectation of WKKID,
i.c., E LEWWKDD and let be the minimum & among the relevant variable Drrain i.c. & = minjeDeraa Ez. Use or to represent the standard error of WKTD
For any Dirain we have
P
WKTDD
W
5 2 P 2
& W
2
WWKTD -6i1 52
P
(17)
2 40²
Em
40²
2 m2m
Due to the symmetric property of elements in D Daon-train- we have ε for all j e D Daon-train- Hence, similar to the above equation, the following formula holds:
40²
P WKTD
Vj E D Donn-train
(18)
E2m
Combining Equation 17 and Equation 18]
we get:
P min WKTD
WKTD
W.KTD
E ....
WKTD < -
2
2
2
5
P
W2KDD
2)
(19)
==1
4
2 m
15

---

Published as a conference paper at ICLR 2025
If miny W < 5. according to the procedure described in Proposition to determine the threshold, we have I max(0, miny WKTD). Therefore, we can derive a lower bound for the power:
Power
E
[ ns)
B
[ nol
min WRID
min WiiW P pmin
[Peraiu]
Prrain
(20)
Note that r < 2'
Thus, the above formula is less than or equal to
Σ
{
P
min WTIDD > --5 -
(21)
Piaain
jEDerain
This lower bound approaches 1 as
A.3 THE PROOF OF THEOREM? 2
Theorem 2. Assuming Assumption 2 holds and the threshold found by Proposition 2 to O, the variable selection procedure satisfies
equal
$ n Derainl
FDR
E
+ 0
(22)
|S
Proof. Similar to Equation
19 we have:
P min WKTD
{
4
of.
(23)
jEDerain
2
22m
jEDrnan
This probability approaches 0 as We then consider FDR:
DDas -trin ns]
Doen train ns
KTD
8
-- P(min W WTDD
2
non -train SS
min KTD
W KTD
P(mmin w W.TDD
w W.KDD
|$|
2 JE train
2'
JE irain
D Doon S
KTD
KTD
P(min WKTD
min Prain
w KTD
[S]
Leoin
The probabilities in the first and second terms go to m + oc. Hence, we focus on the expec- tation in the third term. Under the condition min, WKTD > 2. we have T & Additionally, considering minjeDDuwn. WKID
2
$. it follows that | D
Duon train S
E
min WKD
min WKTDD
2'
jeDraann
2
(25)
<
Σ
1 [WKTD 7]
D Dtrain
jEDoon-caato
Since we assume 7 > 0, there exists a 5 such that > 8 > 0. Therefore, the above formula is less than equal to:
Σ
qj
(26)
Dersiul
m
jeDnon-trxtn
which goes to 0 as oc.
16

---

Published as a conference paper at ICLR 2025
A.4 PARAMETER SETTINGS
For fine-tuning, we used the following settings:
warmup sermup_step-- 100
weight_decay = 0.01
batch size
num epochs 3 (10 for GPT-2)
other hyperparameters were set to the default values provided by the "Training class in the Transformers library.
For paraphrasing, we applied the following configurations:
Top-k sampling with 'topk' = 50
Top-p sampling with 'topp' = 0.95
Temperature scaling with 'temperature' 1.9
B EXPERIMENTS ON "GOOD KNOCKOFFS"
B.1 WHY "GOOD KNOCKOFFS" MATTER?
As shown in Figure I TKD cannot strictly control the FDR in the low-FDR region. According to our analysis, the performance limitations observed for small q values are primarily attributed to imperfections in the generated knockoffs.
Theoretically, we prove that the W-TTD for non-training samples should be symmetrically dis- tributed around zero (Lemma D This proof relies on the assumption that the generated knockoffs are perfectly swappable with the original texts (the second requirement in Definition
However, in natural language processing scenarios, generating perfect knockoffs that fully meet the second requirement in Definition [ for all texts is inherently challenging, which differs from traditional settings where data distributions are often assumed to follow well-defined distributions, such the Gaussian distribution. As result, observe in experiments that the distribution of WKTDD values for non-trainge non-training samples is not perfectly symmetrical in some cases.
Next, illustrate why this asymmetry impair FDR control. In Equation 10 use the expres- sion (denoted as frac,(t))
+ 10j E [n] WKTD S -t)
16 E [n) WKTD 2 v1
(from the left-hand side of Equation 10) to upper bound the true FDR(t). Here, Ki E [n] WKTD < -{} acts an upper bound for the numerator in the definition of FDR((), i.e.
e [m]: WKTD < [3] 2 Hi E D troin WK < -t)] 10 Dmon-train W.KDD
1))
part1
part2
(27)
similar to Equation 3.9 Candes et al. (0088). Consequently, the violation of the symmetry property results in a situation where frac, can no longer serve as a strict upper bound for FDR(t).
This issue becomes more pronounced threshold 7 determined by Equation 10 increases. A larger 7 reduces the denominator of frac; (which is also the denominator of FDR(T)). thereby amplifying the impact of the asymmetry- induced disparity between part and part 2 in Equation 27 (the numerators of frac, (T) and FDR(T)) As result, when is very small, it becomes increasingly difficult impose strict FDR bound.
approaches very small values, since in such cases, the
This highlights the importance of generating high-quality knockoffs, which is especially critical for effective FDR control in low-FDR settings.
17

---

Published as a conference paper at ICLR 2025
1050
105
0/63
110
1057
0.54
23
02
03
0.5
066
07
18
00
23
32
0.3
0.1
65
06
23
05
01
22
0.3
04
85
0.5
25
05-
Pyllin
Pyllin
Petaia
005
GPT-Nro
Noo
86-
CPTNNo
Blrnd
Terrs
Board
142
32
12
WikiMIA
(b) XSum
(c) BBC Real Time
Figure 6: The FDR control results three datasets with the more symmetrical distribution of WTTD Other settings are identical to those in Figure
10
100
10
019
02
0.6
04-
008
010
008
Potiia
CPT-No:
CPENNo
CPT
600
Bori
Borné
Bomi
(a) WIKiMIA
(b) XSum
(c) BBC Real Time
Figure 7: The FDR control results in extremely low FDR-region three datasets with more sym- metrical distribution of WKTD, Other settings are identical to those in Figure
B.2 KTD's PERFORMANCE WITH GOOD KNOCKOFFS
While the imperfections in the generated knockoffs impact the effectiveness of method, particularly the low-FDR region, we emphasize that the primary focus of this paper is on de- signing the overall framework rather than optimizing the generation of high-quality knockoffs. We argue that KTD can achieve improved performance with better knockoffs and conduct experiments empirically illustrate this. In this section, select a subset of non-training samples to adjust the distribution of their W WKTDD, ensuring greater symmetry. Such selection can be challenging in practi- cal settings, as unclear whether each sample from the training or non-training data. However, our goal here is to explore the performance of KTD under good knockoffs, so we make this ideal selection.
FDR Controlling Performance We first evaluate the performance of KTD in terms of FDR con- trol when the distribution of non-training samples' WKTD is made more symmetrical. To be specific, we apply our method in both the regular-FDR region (0.1 (0.1-0.9) and the extremely low-FDR region (0.01-0.09). The results of these experiments are shown in Figures and From these figures, observe that more symmetrical distribution of WKTD for non-training samples enables more strict FDR control, even when the restrain q extremely low.
Power-FDR Tradeoff We extend Figure to explore the extremely low-FDR region, with the results shown in Figure 8 From the figure, we can observe that with a more symmetrical WKTD distribution, the trends the low-FDR region consistent with those observed in the ordinary scenario (where FDR > 0.1).
Robustness to Imbalanced Test Set We evaluated the performance KTD in scenarios with highly imbalanced ratios of training non-training samples in the test Specifically, sampled
18

---

Published as a conference paper at ICLR 2025
002
mml
100
100
m0000
0/0
606
002
100
000
0.00
3.67
1.00
1.03
2.01
6/
3.0
0.81
0.35
0.5
0.5
000
0.55
6/6
(a)
GPT-2
(b) Pythia
(c) GPT-Nco
Figure 8: Trade-off between FDR and power under different m in the extremely low-FDR region
with more symmetrical distribution of WKKTD. All other settings are identical to those in Figure
0/
D-4-
Ponce
Pontr
0.41
02
0.5
04
02
04
02
0.5
GFT-2
0-8
GFT-2
Pysbis
Pyshis
Pysbix
GPT--No
GPT--Noo
04
Bound
Bound
Bound
82-
60
02
0.5
04
0.5
00
03
0.5
0.1
02
0.5
0.5
0.0
0.7
0.
0.5
02
0.5
0.4
0.5
00
0.
08
05
(a) Ratio=0.1
(h) Ratio=0.05
(c) Ratio=0.01
Figure 9: The FDR control results for different vraining-t--non-triinne ratios with more symmetri- cal distribution of WKTD, Other settings are identical to those in Figure 1
0.5-
Power
60-
0:11
033
0.1
022
0.3
0.4
0.
06
07
0.5
0.0
GPT-2
GPT-9
GPT-2
Pysbis
Pythis
Pysbis
120
84
GPT-NN
0.6-
GPII-N
GPT--Noo
Bound
Bound
Burnd
02
02
67
ab
(a) Ratio=0.1
(b) Ratio=0.05
(c) Ratio=0.01
Figure 10: The FDR control results for different diluting degrees. The "Ratio" represents the pro- portion of training samples selected for testing. Other settings are identical to those Figure
the training data within the test set to achieve the test set. The results are presented in Figure As shown in the figure, when the WRTD distribution becomes more balanced, KTD demonstrates robustness in highly imbalanced scenarios.
ratios of 0.1, 0.05, and 0.01
C EXPERIMENTS ON DILUTION
In this section, we investigate the robustness of KTD a "diluted scenario," where the models' fine-tuning data mixed with "background" samples that are absent from the test set. To simulate this scenario, we use only proportion the fine-tuming data for subsequent testing and present the results for different proportions in Figure I0 As shown in the figure, the results follow trend similar to those the standard scenario, demonstrating that KTD remains robust across varying levels of dilution.
19

---

Published as a conference paper at ICLR 2025
Table 3: KTD's performance on 7B LLM under varying y (corresponding to Figure
q
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Power
0,999
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
Pythia-6.9B
FDR
0.103
0.215
0.326
0.347
0.347
0.347
0.347
0.347
0.347
Power
0.811
0.928
0.954
0.969
0.971
0.971
0.971
0.971
0.971
Mistral-7B
FDR
0.077
0.129
0.199
0.308
0.322
0.322
0.322
0.322
0.322
Table 4: KTD's performance the 7B model with varying (corresponding to Figure
Pythia-6.9B
Mistral-7B
5
10
1
10
Power
0.877
0.945
0.957
0.967
0.520
0.735
0.778
0.811
FDR
0.121
0.111
0.112
0.103
0.095
0.078
0.076
0.077
D EXPERIMENTS ON 7B MODELS
To assess the effectiveness of KTD on models with approximately 7 billion parameters, present the results of models (Pythia-6.99 9B and Mistral-7B) on the BBC Real-Time dataset in Tables corresponding to Figures [ respectively. From these tables, we observe that the performance trend of these models is consistent with that of smaller models.
E COMPARISON WITH CONSTAT
We attempt to extend Constat Dekoninck et al. (2024) from benchmark-level detection to sample- level detection by treating each sample an individual benchmark. However, find that this approach is not well-suited to our setting for the following reasons:
Using the default number of bootstrapping, evaluating a dataset with approximately 1,400 samples requires around 42 hours, making the approach computationally expensive and impractical in our settings.
Computational complexity:
Choice of predefined Constat requires users to specify threshold determine whether the performance difference between the test model and the reference models is significant. However, our setting, the range of the gradient norms varies significantly across different datasets and models, making challenging select appropriate ö. We observe that improperly chosen can lead to trivial results, such as classifying all samples as training samples.
20


Reviews:
[{"review_id": "5VT6Z9CkSs", "reviewer": "Reviewer_a4A6", "summary": "The paper presents a method for detecting if a given set of samples has been used during training of the model, while controlling the false positive rate (e.g., saying a sample is used for training where its actually not). The method (KTD) is phrased as a variable selection problem, enabling the use of existing knock-off inference (KI) techniques. The paper adapts the KI techniques by drawing not a single sample, but multiple samples for each knockoff variable, helping reduce the variance and improve the results. Further, the paper proves that the proposed KTD method preserves the good properties of KI, while showing that as the number of samples approaches infinity, the false positive rate approaches 0 (a desirable property). The method is evaluated on several language models showing promising results.", "questions": "- My questions related to the paper are listed in the weaknesses above, please address them in the rebuttal. \n\nOther thoughts:\n\n- It may be interesting for further exploration is whether it is possible to fine-tune the model with (adversarialy-paraphrased designed) samples so that the method (KI or KTD) requires many more samples m, without actually reducing accuracy? \n- And also, how well your method perform against defenses against membership inference, e.g. Differential Privacy, with certain levels of noise? This is not for this paper, but would be interesting to look at.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The paper addresses an important problem: detecting whether a set of samples has been used for training the model, assuming white-box access to the model (weights, gradient info, etc.).\n+ It proposes an adaptation of an existing mechanism (KI), which to my knowledge is the first time this mechanism has been applied to LLMs.\n+ The method seems like a clean extension of KI, preserving the good properties and providing optimality at the limit.\n+ The method is evaluated on several models.", "weaknesses": "- The conceptual contribution is marginal: the KTD method is a somewhat direct adaptation of KI with more than one sample.\n- It is unclear what the value of Def 1 is: the requirements cannot be guaranteed on sampling, and it is unclear where it is used in the proofs (I checked the proofs, but perhaps it is in some of the works this paper is citing).\n- Not major, but there is something unclear in the problem definition, namely what is D_train. Initially, in Background, X1...Xn are samples to be tested, partitioned into D_train and else. These samples may only be a small part of the training dataset, as it happens in evaluation say. Yet later in 4 (line 181), it defines \\theta = Alg(D_train), so D_train is the entire training dataset. Or may be it means that the model is trained on some data and further fine-tuned on D_train. This should be clarified.\n- In equation 4, t_j / t is a function used to compute the Ws, yet in equation 7 t is again used, but now it is not a function but its bounding Ws. This should be clarified.\n- There is something questionable in the problem formulation: it gives FDR over a set of samples which is determined per equation 7 from a joint bound \\tau. Why not solve the problem per single sample one-at-a-time, why does it have to be joint? \n- Also given the threshold \\tau is jointly computed, it seems possible for one (or two, etc.) difficult (call it adversarial) samples to affect the other (easy) samples which may otherwise require much fewer samples, in some sense requiring much larger overall 'm'. In a sense, the method is not adaptive. This doesn't seem to happen in the evaluation, but I can see a situation where it is possible that one trains/fine-tunes with specially designed-paraphrased adversarial samples which trigger this behavior.\n- Experimentally, it seems that the larger the model is, the less one needs the generalization proposed by the paper and the higher values of m as the model memorizes more. For Pythia and GPT-neo (around 1B models), even very few samples seem enough. How would the graph look like with standard 7B models (llama, etc.)? Would m=1 essentially work meaning KI is enough? Can one evaluate a 7B model, there are plenty of those around or perhaps a gemma-2.6b model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGKQQxMnBs", "reviewer": "Reviewer_gGP6", "summary": "This paper approaches training data detection as a variable selection problem and introduces a novel Knockoff Inference-based Training Data Detector (KTD). While traditional knockoff inference methods rely on a single knockoff draw, this work extends the approach by incorporating multiple knockoff draws. Through theoretical analysis, the authors demonstrate the asymptotic optimality of their enhanced method.", "questions": "see the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a compelling and innovative approach to training data detection. The theoretical foundation is rigorous, demonstrating careful attention to mathematical analysis and proofs. The experimental validation is comprehensive, featuring extensive empirical studies that effectively support the theoretical claims.", "weaknesses": "1. Computational Efficiency:\nThe method requires multiple rounds of rephrasing and inference, resulting in higher computational costs compared to baseline approaches. While these costs are reasonable given the improved performance, the paper would benefit from explicitly discussing this computational trade-off.\n\n2. Performance Limitations:\nFor small values of q, the False Discovery Rate (FDR) control becomes less strict, which impacts the method's effectiveness. But I think this limitation is acceptable within the broader context of the method's benefits.\n\n3. Novelty Assessment:\nThe core method is similar to the foundation established by [1,2]. While the paper's primary contribution lies in developing a theoretical framework around this existing idea, the incremental nature of this advancement should be considered when evaluating its impact.\n\n\n[1] Mattern, Justus, et al. \"Membership inference attacks against language models via neighbourhood comparison.\" arXiv preprint arXiv:2305.18462 (2023).\n[2] Fu, Wenjie, et al. \"A Probabilistic Fluctuation based Membership Inference Attack for Generative Models.\" arXiv preprint arXiv:2308.12143 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3LsBOxvB0", "reviewer": "Reviewer_9C67", "summary": "This work proposes KTD, by casting training data detection (also known as membership inference) as variable selection, allowing them to apply Knockoff Inference (KI) with its strong false discovery rate (FDR) control properties. To improve the performance of this approach, they propose a modification of an established knockoff statistic by computing it using multiple knockoffs instead of just one. They go on to show that under some assumptions this modification leads to an asymptotically optimal (in terms of FDR and power, not efficiency) approach. They demonstrate empirically, that KTD is more effective than vanilla KI and existing training data detection in some settings while crucially not requiring access to a validation set.", "questions": "### Questions\n1) Can you discuss why the FDR bound does not generally seem to hold for low q/FDR (See Figure 1) and extend all experiments to the particularly relevant low-FDR regime (1%/0.1%) that would be crucial for the motivating example of legal action.\n2) What parameters were used for fine-tuning?  Is any form of “background” data used to dillute the “training” data later checked for? E.g. combining with IFT data such that the checked for training data is only 10%/5%/1% would be a much more realistic setting.\n3) How does FDR control change if the portion of training samples compared to non-training samples in the detection set is reduced to 10%/5%/1%?\n4) In Figure 2, it seems like the tails of the training samples are quite thick. Can you plot a comparison of just the two tails (positive and negative W_j). Symmetry even there would be required for FDR control at low FDR.\n5) Can you extend Figure 4 to focus on the low FDR regime (smaller q) and perhaps a logarithmic x-axis?\n### Comments\n* Is there a typo in equation 1 and $\\vee$ should correspond to the maximum? Otherwise, we would almost always expect FDR and Power >1\n* Equation 7 should probably be an argmin (?)\n* Definition 2 might be clearer if done elementwise rather than over all variables/training samples jointly, given that this is not leveraged later.\n* What does “achieve FDR” mean in Table 2, given that q was set to 1, implying that all “detected” samples can be non-training?\n* Can you clarify how training data detection is different from membership inference?\n\n### Conclusion\nThis work proposes an interesting and novel approach in the context of training data detection/membership inference which works well across a range of settings. However, the crucially important and motivating setting of low FDR, in particular under realistic training and testing settings is not sufficiently investigated. As is, I believe the paper falls slightly short of the acceptance threshold at ICLR, but I would be more than happy to revise my review if my concerns are addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* New perspective (no validation set, low FDR setting) in an important area.\n* KI seems well suited to the problem of training data detection and this work demonstrates how it can be applied (with minimal modifications).\n* The paper is well written, assumptions/prerequisites are (mostly) outlined, and the proposed method is easy to understand even without prior knowledge of KI.\n* The proposed method seems to scale well with model size and averaging count m. Obtaining strong results in many settings.\n* First method to not require access to a validation set including both training and non-training samples (non-training samples are still required)", "weaknesses": "* FDR control bounds from Proposition 2 do not seem to hold in general (see Figure 1), and in particular is violated for the important regime of low FDR\n* Complete Assumptions for asymptotic optimality (Theorem 1 and 2) are not clearly stated. In particular, Lemma 1 seems to be more of an Assumption than a Lemma (which does not necessarily hold in practice).\n* Important experimental setting details are missing, e.g., exact fine-tuning setup.\n* Important ablations over the concentration of relevant samples in the training set and the ratio of training to non-training samples in the detection set are missing.\n* Related work from membership inference with a similar focus on FDR (Carlini et al., Mireshghallah et al.) and contamination detection (Dekoninck et al. A) is not mentioned or compared.\n* Limitations of the work are not discussed, e.g. the need to compute several gradients wrt model weights and thus applicability only to open weight models and the inability to detect training on rephrased data (Dekoninck et al. B).)\n\n**References**  \n* Carlini et al. \"Membership inference attacks from first principles.\"\n* Dekoninck et al. A \"ConStat: Performance-Based Contamination Detection in Large Language Models.\"\n* Dekoninck et al.  B\"Evading data contamination detection for language models is (too) easy.\"\n* Mireshghallah et al. \"Quantifying privacy risks of masked language models using membership inference attacks.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gwGV14Ivj", "reviewer": "Reviewer_Lbdc", "summary": "In this study, the authors address the challenge of identifying whether a given sample is part of the training set. Their approach is based on comparing specific KTD scores between perturbed versions of the text and the original, drawing inspiration from the knockoff inference method. The proposed technique is tested on a few smaller models and demonstrated to be effective in certain cases.", "questions": "(1) Why do you choose the L2 norm as the STD score?\n\n(2) Can you provide some statistical guarantee when deciding the number of samples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First, the problem of detecting whether a certain sample is in the training set is an interesting and important problem. \n\nSecond, the idea of applying the knockoff Inference method and extending it with multiple samples seems to be a reasonable one.", "weaknesses": "First, while I appreciate the authors’ effort in trying to provide some theoretical guarantee on the result, it is apparent to me after reading that multiple steps of the approach remain ad hoc and thus the guarantee actually means little. For instance, the choice of using “the L2 norm of the model’s gradient as the score in KTD”, which is rather important in your method, must be properly justified. That is, why is it a meaningful choice and how do users understand the guarantee based such as scores then? Furthermore, how do you determine the number of samples? Can you provide some statistical analysis for determining the number of samples (e.g., in the spirit of hypothesis testing for instance)?\n\nSecondly, the experimental evaluation can be improved. For instance, can you conduct some experiments to show the effect of different KTD scores? (Intuitively, one may consider choices such as influence - a near zero influence of a certain sample would imply it is not in the training set). Furthermore, can you conduct the experiments on even larger models such as Llama3? \n\nThe following are a list of detailed comments. \n\nPage 1: “Detecting training data for large language models (LLMs) is attracting growing attention, particularly in applications that require high reliability. While numerous endeavors have been made to tackle this issue, ensuring controlled results often remains overlooked.”\n\nComment: I have trouble inferring what is the problem to be solved from this sentence. \n\nPage 1: “To address this issue, recent studies have focused on detecting training data from large language models (LLMs) …”\n\nComment: The short-hand “LLMs” have already been introduced.\n\nPage 2: “While the knockoff statistic from the vanilla KI method can be directly applied to KTD, it has a significant drawback …”\n\nComment: You should introduce what the vanilla KI method is.\n\nPage 3: “The knockoff framework was first proposed in Barber & Candes (2015) as a data-driven method to control the FDR in variable selection for sparse regression problems.”\n\nComment: As an outsider, I would really appreciate if the authors say at this point how knockoff works - without a proper understanding on how it works, much of the followup discussion doesn’t make sense.\n\nPage 3: “For two number a and b, let a ∨ b represent min(a, b).”\n\nComment: I find this notation disturbing. Why?\n\nPage 7: “Specifically, we fine-tune the models using the training samples while ensuring that the non-training samples remain unseen by the models.”\n\nComment: It is not clear to me why it is necessary to fine-tune the models using the training samples - as they are used in the training already I suppose? Also, such fine-tuning might make the model unnecessarily overfitted on these samples, which makes detecting them easier than what should be the case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)