TDDBench: A Benchmark for Training data detection

Authors: Zhihao Zhu, Yi Yang, Defu Lian
Primary area: datasets and benchmarks
Keywords: Training data detection; Benchmark; Copyright certification

Abstract:
Training Data Detection (TDD) is a task aimed at determining whether a specific data instance is used to train a  machine learning model. In the computer security literature, TDD is also referred to as Membership Inference Attack (MIA). Given its potential to assess the risks of training data breaches, ensure copyright authentication, and verify model unlearning, TDD has garnered significant attention in recent years, leading to the development of numerous methods. Despite these advancements, there is no comprehensive benchmark to thoroughly evaluate the effectiveness of TDD methods.
In this work, we introduce TDDBench, which consists of 13 datasets spanning three data modalities: image, tabular, and text. We benchmark 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory. With TDDBench, researchers can identify bottlenecks and areas for improvement in TDD algorithms, while practitioners can make informed trade-offs between effectiveness and efficiency when selecting TDD algorithms for specific use cases. Our extensive experiments also reveal the generally unsatisfactory performance of TDD algorithms across different datasets. To enhance accessibility and reproducibility, we open-source TDDBench for the research community at https://github.com/zzh9568/TDDBench.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730230386222

thread_type:
official_review

<thread_object>
creator: reviewer_j3dk

<content>
summary:
The paper benchmarks 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory.

strengths:
1. The paper is written well and is easy to understand.

2. The studied problem is very important.

3. The benchmark is comprehensive

weaknesses:
1. It might be useful to include some discussion and experimental results on the defense of the training data detection algorithms

2. The authors can consider elaborating more on the takeaway messages from their experimentations and include more discussions on the current limations of the detection algorithms and the possible future directions at the end of the paper

3. It might be better to discuss the differences between the proposed benchmark with an existing benchmark. That will be useful.

[1] Ye et al., Data Contamination Calibration for Black-box LLMs

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711718483

thread_type:
official_review

<thread_object>
creator: reviewer_3bnu

<content>
summary:
The paper introduces TDDBench, a comprehensive benchmark designed for Training Data Detection (TDD), which is for determining if specific data was used to train a model. This can be used to address issues like data privacy, copyright verification, and model unlearning. TDDBench includes 13 datasets covering image, tabular, and text data and benchmarks 21 TDD methods across four detection paradigms. The findings in this paper highlight that model-based methods outperform others but require significantly more resources. The benchmark reveals that existing TDD methods struggle, particularly with text and tabular data, and emphasizes the need for further research to enhance TDD effectiveness.

-----

Have gone through the responses, and I appreciate the authors' efforts. I keep my ratings.

strengths:
The benchmark is comprehensive. The TDDBench covers multiple data modalities and includes a wide range of datasets, algorithms, and model architectures, making it one of the most inclusive TDD benchmarks available.

The writing and formatting are decent.

weaknesses:
It will be great if the authors can propose a novel method to improve the performance based  on the analysis in their benchmark experiments.

It  lacks  an important discussion about the TDD and different training methods. There are supervised learning, self-supervised learning, unsupervised learning. Would  TDD method generalizable to different training methods?

questions:
If a model is initialized with ImageNet pretrain  weights and then optimized with a different dataset  A. When using an  image example from ImageNet dataset as the input of the TDD  method to  test a model, should the  outcome to  be Yes  or  No?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731135373697

thread_type:
official_review

<thread_object>
creator: reviewer_64p6

<content>
summary:
Motivated by the limitations of the previous benchmarks on training data detection (TDD), the paper proposes a new benchmark that covers three different modalities (tabular, text, and image) and includes evaluation of latest methods. The paper gives several insights on the recent TDD methods with respect to TDD.

strengths:
1. The paper analyzes the latest methods' limitations and strengths carefully. Particularly, the correlation between train-test performance gap and TDD performance is interesting.

weaknesses:
1. The proposed benchmark is not truly large-scale as it considers regular-sized datasets.

2. The proposed benchmark is not significantly different from the existing ones except that it covers different modalities. It would be more interesting if the paper does not follow data split rules from the existing benchmarks but explore something completely new. Another missing aspect is the size of data points that are evaluated under TDD.

3. The claim that TDD outperformance comes at a cost of computational expense is a bit controversial particularly due to Fig. 2(a); learning-based can achieve the best performance although it requires significantly less computation than the model-based ones.

Overall, it is valuable that the pepper gives an extensive amount of experimental results, but the paper is a bit lacking in novel aspects.

questions:
Please refer to the above weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731197749636

thread_type:
official_review

<thread_object>
creator: reviewer_koc1

<content>
summary:
This work provides a comprehensive evaluation of training data detection (TDD). The evaluation spans multiple data modalities, including image, tabular, and text, and implements 21 state-of-the-art TDD algorithms, including large language models such as Pythia-12B. This is the largest benchmark for TDD to date, offering a valuable reference for future research on this problem.

Although this evaluation extensively covers multiple data modalities and four main types of TDD methods, the insights from the evaluation are limited. It lacks in-depth analysis of the results. For example, in Table 4, the authors do not explain why performance on CIFAR-100 is noticeably higher than on other datasets, and similarly in Table 5, there is no explanation for why the target model CatBoost outperforms others on tabular data. Some explanations of the results are also unclear. For instance, in Figure 2, the results indicate that the train-test gap is higher, and the target model performs better at detecting the data used in training. The authors attribute this to overfitting of the training data, which supposedly results in high performance when the train-test gap is large. However, wouldn’t overfitting typically degrade performance since the test set is independent? This explanation should be clarified. Additionally, in Fig. 3(a), the authors state that residual connections in ResNet help alleviate excessive memorization issues, thus decreasing the detection method’s performance. This explanation is also somewhat confusing. In Table 6, the authors claim that detection performance declines when the reference model has limited knowledge of the target model’s architecture. However, this is not always true; for the learn-top3 method, all types of reference models show similar performance without any decline. Thus, the explanation appears inconsistent.

The paper also lacks necessary explanations of the TDD protocols, such as the training-test protocol when using a reference model, or how the query model operates, which would make it easier for readers to follow.

The evaluation also omits Transformer-based models, which are widely used today. For image-based applications in particular, Transformer-based methods are among the most commonly used models, making it worthwhile to evaluate these models for TDD. For large language models, instead of focusing on the Pythia models, evaluating LLaMa, as the most widely used public LLM, would be highly relevant for TDD and aligns better with real-world scenarios where TDD is a concern.

In summary, this work presents an extensive evaluation across different datasets and multiple models, providing a complementary benchmark for future TDD research. However, the lack of in-depth analysis and necessary explanations of training-test protocols across TDD methods weakens the work’s overall quality. Additionally, the omission of evaluations on Transformer-based models and widely used public LLMs, such as the LLaMa models, limits the value of this work.

strengths:
This work provides a comprehensive evaluation of training data detection (TDD). The evaluation spans multiple data modalities, including image, tabular, and text, and implements 21 state-of-the-art TDD algorithms, including large language models such as Pythia-12B. This is the largest benchmark for TDD to date, offering a valuable reference for future research on this problem.

weaknesses:
Although this evaluation extensively covers multiple data modalities and four main types of TDD methods, the insights from the evaluation are limited. It lacks in-depth analysis of the results. For example, in Table 4, the authors do not explain why performance on CIFAR-100 is noticeably higher than on other datasets, and similarly in Table 5, there is no explanation for why the target model CatBoost outperforms others on tabular data. Some explanations of the results are also unclear. For instance, in Figure 2, the results indicate that the train-test gap is higher, and the target model performs better at detecting the data used in training. The authors attribute this to overfitting of the training data, which supposedly results in high performance when the train-test gap is large. However, wouldn’t overfitting typically degrade performance since the test set is independent? This explanation should be clarified. Additionally, in Fig. 3(a), the authors state that residual connections in ResNet help alleviate excessive memorization issues, thus decreasing the detection method’s performance. This explanation is also somewhat confusing. In Table 6, the authors claim that detection performance declines when the reference model has limited knowledge of the target model’s architecture. However, this is not always true; for the learn-top3 method, all types of reference models show similar performance without any decline. Thus, the explanation appears inconsistent.

The paper also lacks necessary explanations of the TDD protocols, such as the training-test protocol when using a reference model, or how the query model operates, which would make it easier for readers to follow.

The evaluation also omits Transformer-based models, which are widely used today. For image-based applications in particular, Transformer-based methods are among the most commonly used models, making it worthwhile to evaluate these models for TDD. For large language models, instead of focusing on the Pythia models, evaluating LLaMa, as the most widely used public LLM, would be highly relevant for TDD and aligns better with real-world scenarios where TDD is a concern.

questions:
- in Table 4, the authors do not explain why performance on CIFAR-100 is noticeably higher than on other datasets
- in Table 5, there is no explanation for why the target model CatBoost outperforms others on tabular data
- in Figure 2, the results indicate that the train-test gap is higher, and the target model performs better at detecting the data used in training. The authors attribute this to overfitting of the training data, which supposedly results in high performance when the train-test gap is large. However, wouldn’t overfitting typically degrade performance since the test set is independent? 
- in Fig. 3(a), the authors state that residual connections in ResNet help alleviate excessive memorization issues, thus decreasing the detection method’s performance. This explanation is confusing.
- In Table 6, the authors claim that detection performance declines when the reference model has limited knowledge of the target model’s architecture. However, for the learn-top3 method, all types of reference models show similar performance without any decline. The explanation appears inconsistent.
- The paper also lacks necessary explanations of the TDD protocols, such as the training-test protocol when using a reference model, or how the query model operates.
- Why the evaluation omits Transformer-based models, especially for image datasets
- Why chose Pythia instead of LLaMa as the most widely used public LLM
</content>
</thread_object>

</thread>
<thread>
creation date:
1732258115322

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for your constructive comments and suggestions, which we have incorporated into the revised version. We will provide a response to each comment on a case-by-case basis. Due to the extensive experiments added, we spent a significant amount of time on this process. We appreciate the reviewers' patience and understanding as we worked through these revisions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732364985253

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kind Reminder: Seeking Reviewer Feedback for Author/Reviewer Discussion Phase

comment:
Dear Reviewers,

We wish to express our sincere gratitude for the invaluable assistance you have provided in reviewing our work. Your feedback has been crucial in enhancing our submission. We have carefully prepared responses to each of your reviews and hope that they adequately address your concerns.

As the deadline for the author/reviewer discussion phase (Nov 12-26) is fast approaching, and we have yet to receive any feedback, we kindly request your assistance once again. We understand that you may have busy schedules and may still be reviewing our responses, but we would greatly appreciate the opportunity for further discussion to ensure that our submission meets the high standards of ICLR 2025. Your feedback is incredibly valuable to us, and we are eager to work with you to make any necessary revisions.

Thank you once again for your time and consideration.

Best regards,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733215441551

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We thank all the reviewers for their valuable feedback, which has contributed to the refinement of our work. We appreciate that **the reviewers have recognized the importance of our research question** (Reviewer j3dK), **the comprehensiveness of our benchmark** (Reviewer 3bnU), **the extensiveness of our experiments** (Reviewer 64P6), and **the benchmark's value for future research on training data detection** (Reviewer KoC1). However, we have noted the variance in the overall ratings. To address this, we have summarized the improvements based on your feedback and included a clarification regarding Reviewer 64P6's concerns. We hope these efforts will facilitate further discussion.

1. Based on Reviewer KoC1's comments, we have added a detailed analysis of TDD's performance, the training-test protocols used, and an evaluation on Vision Transformer-based models and LLaMa models.

2. In response to Reviewer 64P6's feedback, we have highlighted the novel aspects of TDDBench, discussed TDD algorithm performance on large-scale datasets and different data partitioning rules, and clarified why the model-based method shows stronger detection performance.

3. In response to Reviewer 3bnU's suggestions, we introduced Model-mix, a novel model-based algorithm. We also evaluated the effectiveness of the TDD algorithm in the contexts of semi-supervised and self-supervised learning methods, as well as in detecting pre-trained data.

4. In response to Reviewer j3dK's comments, we have discussed the work of Ye et al., included experimental results on defenses, and added discussions regarding the current limitations of detection algorithms.

We recognize that Reviewer 64P6 still has concerns and misunderstandings about the novelty of TDDBench. Reviewer 64P6 stated that TDDBench only demonstrates extensiveness and multi-modality in comparison to other benchmarks. While we believe novelty is not the only criterion for a benchmark's contribution, we emphasize that TDDBench has more novel aspects:

- **Novelty in Evaluation Scenarios**: We have designed experiments across various scenarios, such as the impact of different data split rules, defense strategies, model training methods, and model sizes on the performance of TDD algorithms. These experiments provide novel insights and feasible directions for TDD algorithms, most of which were not explored in previous benchmarks.

- **Novelty in Evaluation Algorithms**: Compared to previous benchmarks, TDDBench includes more TDD algorithms introduced in the last two years and conducts experiments across various data modalities and model architectures. Notably, several TDD algorithms, such as Query-qrm, Query-transfer, Model-fpr, and Model-robust, are evaluated on text datasets for the first time.

- **Novelty in Evaluation Metrics**: TDDBench goes beyond simple detection accuracy to include practical considerations such as memory consumption and computational efficiency, highlighting the trade-offs necessary for deploying TDD algorithms in real-world applications.
 
Finally, thank you once again for your time and consideration.

Best regards,

The Authors of #1267
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper was reviewed by 4 experts in the field and received 6, 5, 6, 6 as the ratings. The reviewers agreed that the paper presents a comprehensive evaluation of 21 training data detection (TDD) algorithms on multiple data modalities (image, text and tabular). The paper studies a problem of interest, it is well-written and easy to understand.  

Reviewer 64P6 raised a concern that the benchmark is not large-scale, as it considers regular sized datasets; in the rebuttal, the authors have evaluated the performance of the TDD algorithms on the ImageNet-1k and the FairJob datasets, both containing over 1 million samples. This reviewer also raised a concern about the size of the dataset that are evaluated under TDD; to address this, additional experiments were conducted by the authors in the rebuttal, by varying the size of the target dataset. While reviewer 64P6 has a slightly negative opinion about the paper, the AC feels that all his concerns were convincingly addressed by the authors in the rebuttal. The reviewer has also acknowledged the extensiveness of the experiments in the paper. 

Reviewer KoC1 mentioned that the evaluations do not include transformer based models; in response, the authors have conducted experiments on the ViT and Swin transformer models. A question was raised whether the TDD method will be generalizable to different training methods; as part of the response, the authors have conducted experiments on CIFAR-10 and CIFAR-100 to assess the suitability of TDD algorithms for semi-supervised and self-supervised learning, in addition to supervised learning. Concerns were also raised about the defense of the training data detection algorithms, explanations of the TDD protocols and current limitations of the detection algorithms, all of which were addressed convincingly by the authors. 

The reviewers, in general, have a positive opinion about the paper and its contributions. Most of the reviewers' concerns have been addressed convincingly by the authors, with supporting experimental results. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2025. The reviewers have provided some valuable comments, such as the novelty of the proposed research, further discussions on TDD algorithms tailored for specific training methods (particularly semi-supervised and self-supervised learning) and adaptive approaches to address the potential defense mechanisms of the target model. The authors are encouraged to address these in the final version of their paper. We congratulate the authors on the acceptance of their paper!