Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts

Authors: Jacob Haimes, Cenny Wenner, Kunvar Thaman, Vassil Tashev, Clement Neo, Esben Kran, Jason Hoelscher-Obermaier
Primary area: datasets and benchmarks
Keywords: Language Models, Datasets and Benchmarks, Model Evaluation, Data Contamination, Evaluation Gaming

Abstract:
The training data for many Large Language Models (LLMs) is contaminated with test data. This means that public benchmarks used to assess LLMs are compromised, suggesting a performance gap between benchmark scores and actual capabilities. Ideally, a private holdout set could be used to accurately verify scores. Unfortunately, such datasets do not exist for most benchmarks, and post-hoc construction of sufficiently similar datasets is non-trivial. To address these issues, we introduce a systematic methodology for (i) retrospectively constructing a holdout dataset for a target dataset, (ii) demonstrating the statistical indistinguishability of this retro-holdout dataset, and (iii) comparing LLMs on the two datasets to quantify the performance gap due to the dataset's public availability. Applying these methods to TruthfulQA, we construct and release Retro-Misconceptions, on which we evaluate twenty LLMs and find that some have inflated scores by as much as 16 percentage points. Our results demonstrate that public benchmark scores do not always accurately assess model properties, and underscore the importance of improved data practices in the field.

Decision: reject
Reviewer scores: 3, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729323965069

thread_type:
official_review

<thread_object>
creator: reviewer_zwfv

<content>
summary:
The paper highlighted the importance of a holdout set for evaluating LLMs, which inspired the author to propose a method for creating such a set, along with a series of tests to ensure its indistinguishability. An adequately constructed holdout set can help detect whether a benchmark has been contaminated. The experiments demonstrated the effectiveness of their holdout set on TruthfulQA and revealed contamination in 20 popular LLMs.

strengths:
1. The paper highlighted the importance of a holdout set and introduced an effective method for creating one, along with several testing approaches.
2. The paper presented contamination results for 20 popular LLMs.
3. The testing method for indistinguishability was highly rigorous and comprehensive.

weaknesses:
1. The experimental dataset is quite limited, consisting only of the truthful QA dataset. It could be expanded to include more diverse and widely-used datasets, such as Open QA, rather than just multiple-choice QA.
2. The method used to create the holdout set is crucial to this paper and should be thoroughly explained in the main section. In its current form, the process isn't clear from the text. I recommend including a diagram or algorithm to better illustrate this process.
3. The paper "benbench" [1] also evaluates contamination across a wide range of LLMs, and should be referenced in your related works.
4. The paper lacks the meta-evaluation to verify the effectiveness of this method and the compression with other contamination detection methods like MinK [2].

[1] Benchmarking Benchmark Leakage in Large Language Models. https://arxiv.org/abs/2404.18824

[2] Detecting Pretraining Data from Large Language Models. https://arxiv.org/abs/2310.16789

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1729970609541

thread_type:
official_review

<thread_object>
creator: reviewer_v8a3

<content>
summary:
This paper proposed a way to retrospectively create a holdout set for a target dataset to benchmark models more accurately. The authors introduce four statistical tests to ensure the holdout set and the target dataset are sufficiently indistinguishable. They also introduce some tools to help iteratively create the holdout set to pass the statistical tests. With this approach, the authors create a holdout set for the misconceptions category of the TruthfulQA dataset, using it to calculate models' performance gap between evaluating on the original TruthfulQA and on the holdout set. The result shows that most models suffer from benchmark inflation, indicated by a large performance gap, suggesting that the TruthfulQA dataset have been included in the training data to some extent, and highlighting the need of holdout sets.

strengths:
1. The paper is well written.
2. This paper propose a way to create a holdout set of a target dataset, which is important as many public benchmarks have been included in training data.
3. The proposed holdout set for TruthfulQA could be beneficial for future research to accurately benchmark models.

weaknesses:
1. The authors did not provide much details about the holdout set for TruthfulQA they created. For example, the size of the holdout set and the number of iterations to pass the four tests.
2. The authors introduced some tools to help create the holdout set. However, they did not provide empirical evidence to show how these tool work.
3. The authors only demonstrate their approach on a category of TruthfulQA. It is unclear whether the approach can be applied to other datasets.

questions:
1. How many iterations does it takes to make the holdout set for the misconceptions category of TruthfulQA pass the four tests?
2. How many samples do you create for the holdout set?
3. Could you provide some examples of data in the holdout set?
4. Different datasets may be created in vary different ways. How can the proposed approach generalize to other datasets?
5. How did you select the four tests? Did you try other criteria?
6. How does the tools help create the holdout set? Did you compare the difference between using and without using those tools?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586053827

thread_type:
official_review

<thread_object>
creator: reviewer_vnvv

<content>
summary:
The paper introduces a retro-holdout framework to assess benchmark reliability for LLMs by retroactively constructing datasets (e.g., Retro-Misconceptions for TruthfulQA) that can reveal evaluation inflation due to data contamination. The study evaluates several LLMs and highlights the importance of reliable data practices in ensuring that benchmark scores reflect real-world model capabilities rather than inflated metrics.

strengths:
The paper addresses a critical issue in AI evaluation by introducing a new methodology for retro-holdout dataset construction, which could significantly impact LLM benchmark validity. This novel approach to data contamination is a creative response to the challenge of black-box model evaluation, providing a framework that could be applied across different benchmarks. The paper demonstrates thoroughness in validating retro-holdout indistinguishability through multiple statistical tests, contributing meaningfully to ongoing discussions around data integrity in LLMs research.

weaknesses:
I think the paper faces challenges in terms of interpretability and practicality. The retro-holdout methodology, while innovative, remains a black-box model that relies on side metrics such as similarity and precision, which may not fully establish reliability. Moreover, the method’s practical relevance is questionable, as it appears computationally intensive and possibly detached from current LLM data contamination mitigation strategies. Additionally, the lack of comparison with established methods, such as the widely used n-gram approach, leaves readers without a clear sense of this method's relative effectiveness.

References:
1. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 15, 3.
2. Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024. Benchmark data contamination of large language models: A survey. arXiv:2406.04244.
3. Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. 2024. Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation. ACL 2024 Findings.

questions:
- Could the authors clarify how the retro-holdout's interpretability compares to traditional n-gram methods, and are there plans to benchmark against them?

- Given the computational demands, what practical applications do the authors envision for the retro-holdout framework?

- Would the authors consider publishing their experimental code to support transparency and facilitate reproducibility?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685008945

thread_type:
official_review

<thread_object>
creator: reviewer_dnhu

<content>
summary:
This paper introduce a criterion to create hold-out set for benchmark data to investigate data contamination issues. They introduce four rigorous tests to validate these retro-holdouts. Applying their method to TruthfulQA, they evaluated 20 LLMs and discovered significant performance gaps, with some models showing score inflation of up to 16 percentage points. This reveals that public benchmark scores often don't accurately reflect real model capabilities.

strengths:
- There research topic is important for fair evaluation in large language models.
- We need dynamic eval for preventing data contamination.

weaknesses:
- The basic logistics require better clarification. While language models trained on next-token prediction are naturally sensitive to different formats (as evidenced by the *reversal curse*), the paper should better distinguish between this inherent next-token prediction capability and true robustness to format perturbations and contamination.

- The four perspectives presented are not all equally justified. There is significant overlap between the first two perspectives, as both rely on model accuracy to assess dataset difficulty. Additionally, the human testing perspective, while valuable, is not scalable for wider benchmark applications since it's impractical to recruit human annotators for every hold-out set.

- The discussion of benchmarks is too narrow, focusing primarily on TruthfulQA. This limited scope may be a consequence of the unscalable human testing requirement in the study design. Furthermore, since TruthfulQA is rarely used in evaluating current frontier LLMs, it would be beneficial to include analyses of additional, more widely-used benchmarks.

questions:
- From line 112 to line 113 and line 258, citation format should be changed from \citet to \citep

- Regarding Figure 2, the authors need to specify which example they used in their analysis. Without clear example details, the distribution alone is insufficient to validate their claim.

- The claim on lines 160-161 that benchmark data after a model's cutoff date cannot be contaminated is inaccurate. Benchmarks built from internet sources can still be indirectly contaminated. For example, TruthfulQA, although created after certain model cutoff dates, draws from Wikipedia content. Since proprietary models are often pretrained on Wikipedia, the benchmark could still be contaminated despite its creation date.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a retro-holdout framework to assess benchmark reliability for LLMs, which is an interesting and relevant topic in LLM evaluation. Experiments were conducted on 20 popular LLMs to demonstrate its effectiveness. However, reviewers have raised several concerns, including unclear dataset construction and the use of a limited experimental dataset. As a result, the current version of the work does not meet the acceptance threshold.