Fine-tuning can Help Detect Pretraining Data from Large Language Models

Authors: Hengxiang Zhang, Songxin Zhang, Bingyi Jing, Hongxin Wei
Primary area: foundation or frontier models, including LLMs
Keywords: Large language models, Fine-tuning, Pretraining data detection

Abstract:
In the era of large language models (LLMs), detecting pretraining data has been increasingly important due to concerns about fair evaluation and ethical risks. Current methods differentiate members and non-members by designing scoring functions, like Perplexity and Min-k%. However, the diversity and complexity of training data magnifies the difficulty of distinguishing, leading to suboptimal performance in detecting pretraining data. In this paper, we first explore the benefits of unseen data, which can be easily collected after the release of the LLM. We find that the perplexities of LLMs shift differently for members and non-members, after fine-tuning with a small amount of previously unseen data. In light of this, we introduce a novel and effective method termed Fine-tuned Score Deviation (FSD), which improves the performance of current scoring functions for pretraining data detection. In particular, we propose to measure the deviation distance of current scores after fine-tuning on a small amount of unseen data within the same domain. In effect, using a few unseen data can largely decrease the scores of all non-members, leading to a larger deviation distance than members. Extensive experiments demonstrate the effectiveness of our method, significantly improving the AUC score on common benchmark datasets across various models.

Decision: accept (poster)
Reviewer scores: 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729262725308

thread_type:
official_review

<thread_object>
creator: reviewer_h5ay

<content>
summary:
This paper found that fine-tuning significantly reduces the perplexity of non-member data, which inspired the author to propose a new data detection method, FSD. The method determines whether data was used in training by comparing its perplexity in the original model versus the fine-tuned model. Experimental results showed a significant improvement over baseline methods.

strengths:
1. The improvement is very significant.
2. The experiments and analysis are solid.
3. The writing is good and easy to follow.

weaknesses:
1. This method requires fine-tuning the models. However, many models do not support fine-tuning especially white-box models, making this approach impractical in reality.
2. Fine-tuning every model you want to detect would be very costly.
3. Meta-evaluation typically fine-tunes a model on the data that needs to be detected, in order to assess whether the detection method is effective. However, this method is similar to the meta-evaluation.
4. The in-distribution non-member data is not easy to be acquired for fine-tuning.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730131706951

thread_type:
official_review

<thread_object>
creator: reviewer_zrvw

<content>
summary:
The paper tackles the issue of detecting pretraining data. It introduces a method where a pretrained model is fine-tuned with non-member data from the same domain. The goal is to increase the gap in scores like perplexity between member and non-member data, making it easier to spot training data.

strengths:
* Clear introduction to the problem and motivation.
* The method is well explained and supported with intuitive examples, such as Figure 2.
* Comparisons with other relevant baselines, and showing significant improvement over them.
* Thorough experiments, including ablation studies that address additional research questions.
* Creative use of non-member data.

weaknesses:
* The paper assumes access to unseen data in the same domain but doesn’t define 'domain' clearly. Could the authors explain how they handle differences between domains, especially if vocabularies differ a lot, and how they decide what data counts as 'same domain'?
* *"To the best of our knowledge, our method is the first to utilize some collected non-members in the task of pretraining data detection"* - There’s limited information on how to collect non-member data, which seems a key aspect of your method.
* The approach assumes access to both unseen data and model probabilities for each token, compared to other black-box assumption methods.

questions:
* Could you explain more about the datasets, like how they’re labeled and how well they fit the models used? Is there a class imbalance in any of the datasets?
* Should fine-tuning happen for each example $x$ (or each group of examples $x_i$ within the same domain)? If so, that might be a big drawback.
* Fine-tuning can be delicate, and it’s easy to go past the optimal point. An experiment on how sensitive this method is to fine-tuning parameters (e.g., number of epochs, LoRA rank) would be useful.
* For a more diverse test set, what happens if we fine-tune once on a mix of domains?
* If non-member data in the specific domain is not available, is there any alternative in the general frame of this method?



**Minor suggestions:**
* If terms like "Data Contamination" or "Membership Inference Attack" are central, it might help to introduce them earlier (e.g., in the Introduction or Background). If not, why are they in the Related Work section?
* Adding an ethics statement could underscore the method's role in detecting sensitive information used in LLM training, addressing the ethical significance of such methodologies.
* Consider renaming either the “Method” section or subsection for clarity.
* Start a distinct “Results” section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656582860

thread_type:
official_review

<thread_object>
creator: reviewer_j4zf

<content>
summary:
The paper proposes a new method for improving the accuracy of pretraining data detection in LLMs, called Fine-tuned Score Deviation. they shows that fine-tuning models on a small set of non-member data increases the deviation between score distributions of seen and unseen data. They validate this claim via experiments on WikiMIA, BookMIA, ArXivTection and BookTection datasets. FSD consistently improves the performance of existing detection methods based on scoring functions like perplexity and Min-k%.

strengths:
Very interesting observation, and notable improvement in pretraining data detection accuracy. I think this paper has a clearly defined objective, and interesting empirical results.

The results are strong, showing substantial improvements in AUC and tpr at low fpr across the selected datasets and models. FSD helps improving existing score-based pretraining data detection methods.

weaknesses:
1. Have you experimented on the MIMIR dataset? This dataset seems to be more challenging, and authors claim this is because seen and unseen examples are not from different temporal distributions. I am very interested to see how this method helps MIA on MIMIR dataset.

2. They have not explained or given intuition about why finetuning increases the gap between distributions.

3.  The method requires fine-tuning the LLM, which makes it more expensive compared to existing methods. If authors find some intuition about why finetuning has this impact, they may find less expensive methods.

questions:
1. Do you have intuitions or theoretical understanding about why finetuning increases the gap between distributions?

2. What is the role of the finetuning technique you use? Is there a dependency on LORA?

3. How sensitive is FSD to using unseen data from a loosely related domain impact detection performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709831004

thread_type:
official_review

<thread_object>
creator: reviewer_ozec

<content>
summary:
The paper addresses the problem of detecting prestrainign data from LLMs. They notice that when fine-tuning an LLM on a few non-members selected by a cut-off date, the perplexity on other non-members (also based on the same cut-off date) decrease. Based on this observation, the authors propose a new method (Fine-tuned Score Deviation; FSD) to improve prior membership inference attacks (MIA) by adjusting the scores based on this fine-tuning phenomenon. They conduct experiments on 4 cut-off based benchmarks: WikiMIA, BookMIA, ArXivTection and BookTection and show significant gains wrt the baselines. The specific questions they tackle are:

1. Can FSD improve the performance of current MIA methods?
2. How many samples does the method need for fine-tuning?
3. Does FSD work across different model sizes?
4. Does FSD work with members for fine-tuning, instead of non-members?
5. Does FSD work for any fine-tuning method?

strengths:
1. The idea is simple and intriguing, enlarging the difference between members and non-members could boost any MIA method.
2. The idea works for any MIA method.

weaknesses:
The evaluation has major flaw. It only uses benchmarks that separates members and non-members based on cut-off dates. As Duan et al., 2024, Das et al., 2024, and Maini et al., 2024 show, this is fundamentally wrong because it introduces a temporal shift that bias the benchmark. These works have consistently proved the need to avoid evaluating MIA based on cut-off dates. This paper acknowledges these works on lines 458-459 and questions whether the results of this ICLR submission are influenced by this temporal shift. However, instead of following the recommendations from these papers for a correct evaluation, they simply remove timestamps. This is not enough as Duan et al., (2024) shows. Duan et al., shows that the cut-offs introduces changes in the n-grams. An IID split have a higher n-gram overlap between members and non-members than the temporal split. Therefore, even when removing the timestamps in the plain texts, the n-gram distribution is different, making the evaluated method classify temporal data rather than membership. Moreover, the authors acknowledge this possibility in lines 535-537 but they do not follow the recommendations of the papers mentioned above, which are to evaluate on for example Pythia with data from the Pile.

questions:
* Please conduct your experiments on an IID setup (see the works of Duan et al., 2024, Das et al., 2024, and Maini et al., 2024 that you cite; eg: using Pythia use as members the training set and as non-members the dev and test sets) so that we can clearly see the effectiveness of your work.
* It is possible that the results of Figure 4 are explained by the temporal bias of the evaluation. Training more on new data might make the method identify better the n-gram distribution of the documents after the cut-off. We can reject this hypothesis by evaluating on an IID setup.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732201423508

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response  (2/2)

comment:
### **2. Minimal computational overhead with significant improvements.**
We find that several reviewers concern about the cost of fine-tuning. Here, we provide a detailed analysis to show the computational cost of our method.

* **Significant improvements with a few samples for fine-tuning.** In our work, we investigate how the size of fine-tuning data affects the performance of FSD, in Subsection 4.2. The results show that our method is highly data-efficient. For instance,  our method improves the AUC score of the perplexity-based method **from 0.63 to 0.91** on WikiMIA, by leveraging only **100 non-member data** for fine-tuning – a significant direct improvement of 44%. Thus,  Our method only needs a small number of samples for fine-tuning to improve detection, which requires low computational cost.

* **Fine-tuning with efficient parameter fine-tuning method.** Our method employs LoRA [1] to fine-tune the pre-trained model, achieving this with minimal computational cost. For example, we conducted a fine-tuning process on the LLaMA-7B model utilizing 230 samples from the WikiMIA dataset, supported by two NVIDIA L40 GPUs. With a batch size of 8 and 3 epochs, the fine-tuning is finished in only **109 seconds**, demonstrating its minimal computational overhead. Besides, our method does not rely on the LoRA and can work well with other PEFT, as shown in Section 5.

In summary, the computational cost of our method is acceptable in practice and can achieve dramatic improvements with minimal computational overhead.

[1] Hu, et al. Lora: Low-rank adaptation of large language models. ICLR 2022.


### **3. Fine-tuning using non-members from different domains**
We note that two reviewers are interested in our method's performance when fine-tuning data from a mix of domains. Thanks for the great suggestion. We add two experiments to explore the performance of our methods when fine-tuning using non-members from different domains.

* **Fine-tuning using non-members from a mix of domains**
Specifically, we randomly sample 231 and 238 non-members from the WikiMIA and ArXivTection datasets to construct a fine-tuning dataset comprising a mix of domains. The remaining part of the dataset is used for testing. Then, we fine-tune the LLaMA-7B model on the constructed dataset and evaluate our method on the two datasets, respectively. The results show that our method can also **significantly improve** the performance of baselines, indicating the effectiveness of our methods when fine-tuning with non-member data from a mix of domains.

| Dataset    | WikiMIA |           | ArXivTection |           |
| ---------- | ------- | --------- | ------------ | --------- |
| Method     | Base    | **+Ours** | Base         | **+Ours** |
| Perplexity | 0.64    | **0.91**  | 0.68         | **0.93**  |
| Lowercase  | 0.58    | **0.73**  | 0.50         | **0.73**  |
| Zlib       | 0.62    | **0.91**  | 0.57         | **0.92**  |
| MIN-K%     | 0.65    | **0.84**  | 0.76         | **0.87**  |


* **Fine-tuning using non-members from entirely unrelated domains**
Specifically, we fine-tune the LLaMA-7B model on non-members sampled from the ArXivTection dataset. Subsequently, we evaluate the performance of our method on the WikiMIA dataset for pretraining data detection. The table below shows the AUC scores of baselines and our method. The results indicate that our method **fails to improve** the performance of baselines, since the fine-tuning data comes from an entirely unrelated domain to the evaluation data.


| Method    | Perplexity | Lowercase | Zlib | MIN-K% |
| --------- | ---------- | --------- | ---- | ------ |
| Base      | **0.68**   | **0.50**  |  0.57|  **0.76**|
| **+Ours** | 0.52       | 0.50      |  **0.64**| 0.61 |

	
In summary, our method can work on a mix of domains, but fails to improve given data from unrelated domains. Therefore, our method requires to collect a few non-members that belong to the related domain, as highlighted in our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732202457827

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/2)

comment:
We appreciate the reviewers' thoughtful feedback and valuable comments on our work. We are encouraged that reviewers recognize that the idea of our work is **intriguing** (oZEC), and point out that our work is **very interesting** (J4zF) and **creative** by leveraging non-member data (Zrvw). The reviewer finds that our method is **well explained** and **supported** with intuitive examples (Zrvw). We are pleased that reviewers find the experiments and analysis are **solid** (h5aY), and the empirical results are **interesting** and **strong** (J4zF) with **notable/significant** improvements (J4zF, Zrvw, h5aY). Besides, reviewers recognize that the writing is **good** and **easy to follow** (h5aY), with a **clearly defined objective** (J4zF) and a **clear introduction** to the problem and motivation (Zrvw).

The reviews allow us to strengthen our manuscript, and the changes are summarized below:

* Added introduction of “Membership Inference Attack” and “Data Contamination” in **Line 108-110**. [Zrvw]
* Revised the title of Section 3 in **Line 162**. [Zrvw]
* Revised the title of Subsection 3.2  **in Line 205**. [Zrvw]
* Added experiments on the pile dataset in **Line 346-347** and **Appendix D.3**. [oZEC, J4zF]
* Added experiments on fine-tuning with varying data size in **Line 360-362** and **Appendix D.3**. [oZEC]
* Added experiments on fine-tuning with different parameters in **Line 498-499** and **Appendix D.3**. [Zrvw]
* Added ethical statement in **Line 521-526**. [Zrvw]
* Added experiments on fine-tuning using data from different domains in **Appendix D.3**. [J4zF, Zrvw]
* Revised description on fine-tuned score deviation **in Line 077**.
---
¹ For clarity, we highlight the revised part of the manuscript in **blue** color.

In the following responses, we have addressed the reviewers' comments and concerns point by point and are willing to discuss any concerns you may have. Here, we provide responses to some common concerns of the reviewers.

### **1. Experiments on the Pile dataset**
We note that two reviewers are interested in the effectiveness of our method on a more challenging dataset. Here, we provide detailed results on three datasets from the Pile dataset that satisfies the IID setup [1].

Concretely, we evaluate our methods on the three subsets of the Pile dataset, following the prior work [2]. Here, the validation set of the pile dataset was not trained on the Pythia models [3]. Thus, we conduct experiments on the Pythia-6.9B model, utilizing the training and validation sets as members and non-members, respectively. For each dataset, we randomly sample a few non-members with a sample ratio of 0.3 from the validation set for fine-tuning. Then, we evaluate our method on a balanced test set composed of 1400 members and 1400 non-members.

The table below shows that our method **significantly improves the performance** of baselines on the Pile dataset under the Pythia-6.9B model. For example, our FSD improves the AUC score of the perplexity-based method from 0.528 to 0.885 on BookC2, a significant direct improvement of 67\%. This demonstrates the effectiveness of our method in the IID setup.

| Dataset    | Wiki  |           | BookC2 |           | Gutenberg |           |
| ---------- | ----- | --------- | ------ | --------- | --------- | --------- |
| Method     | Base  | **+Ours** | Base   | **+Ours** | Base      | **+Ours** |
| Perplexity | 0.471 | **0.614** | 0.528  | **0.885** | 0.528     | **0.661** |
| Lowercase  | 0.466 | **0.626** | 0.518  | **0.725** | 0.546     | **0.551** |
| Zlib       | 0.496 | **0.619** | 0.477  | **0.907** | 0.496     | **0.686** |
| MIN-K%     | 0.512 | **0.611** | 0.510  | **0.841** | 0.536     | **0.612** |

[1] Gao L, et al. The pile: an 800gb dataset of diverse text for language modeling. arXiv:2101.00027, 2020.

[2] Maini P, et al. LLM Dataset Inference: did you train on my dataset? arXiv:2406.06443, 2024.

[3] Stella B, et al. Pythia: a suite for analyzing large language models across training and scaling. ICML2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new method called Fine-tuned Score Deviation (FSD) to improve the performance of various scoring functions for pretraining data detection. The proposed method is inspired by an observation: after fine-tuning with a small amount of previously unseen data, the perplexities of LLMs perform differently for members and non-members. This observation is interesting and the method is technically sound. However, one highly confident reviewer raised a major concern about the datasets based on cut-off dates used in the experiments, and previous studies have shown that such datasets can be heavily biased, so they cannot well validate the performance of the proposed method. The authors attempted to address this issue by reporting new experimental results on additional datasets in the author's rebuttal. Another important issue is that the method requires fine-tuning the LLM, which makes it much more expensive compared to existing methods. I have mixed feelings about this paper and it is a borderline one.