Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers

Authors: Lorenzo Pacchiardi, Marko Tesic, Lucy G Cheke, Jose Hernandez-Orallo
Primary area: foundation or frontier models, including LLMs
Keywords: llm benchmarks, benchmark validity, clever hans effect, llm evaluation

Abstract:
The integrity of AI benchmarks is fundamental to accurately assess the capabilities of AI systems. The internal validity of these benchmarks—i.e., making sure they are free from confounding factors—is crucial for ensuring that they are measuring what they are designed to measure. In this paper, we explore a key issue related to internal validity: the possibility that AI systems can solve benchmarks in unintended ways, bypassing the capability being tested. This phenomenon, widely known in human and animal experiments, is often referred to as the ‘Clever Hans’ effect, where tasks are solved using spurious cues, often involving much simpler processes than those putatively assessed. Previous research suggests that language models can exhibit this behaviour as well. In several older Natural Language Processing (NLP) benchmarks, individual $n$-grams like “not” have been found to be highly predictive of the correct labels, and supervised NLP models have been shown to exploit these patterns. In this work, we investigate the extent to which simple $n$-grams extracted from benchmark instances can be combined to predict labels in modern multiple-choice benchmarks designed for LLMs, and whether LLMs might be using such $n$-gram patterns to solve these benchmarks. We show how simple classifiers trained on these $n$-grams can achieve high scores on several benchmarks, despite lacking the capabilities being tested. Additionally, we provide evidence that some modern LLMs might be using these superficial patterns to solve benchmarks. This suggests that the internal validity of these benchmarks may be compromised and caution should be exercised when interpreting LLM performance results on them.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730416172778

thread_type:
official_review

<thread_object>
creator: reviewer_1ht7

<content>
summary:
The paper examines multiple-choice benchmarks by training classifiers on various surface-level features, such as n-grams. This approach demonstrates that current evaluations may be susceptible to these spurious cues. The authors then use these classifiers to assess whether the performance of current models relies solely on surface-level features.

strengths:
- The classifier approach presented in the papers is highly compelling for identifying whether spurious features (i.e n-grams features) exist in an evaluation.
- Meta-studies on evaluations offer an interesting and valuable research direction for the field.

weaknesses:
- The results show that certain LLMs performance correlates with performance on spurious (e.g., n-gram) features. However, there could be other features the models rely on to achieve similar performance to these classifiers. To address this concern, it might be helpful to include case studies on these data points to gain a qualitative understanding of the specific behaviors. Such case studies/analyses would be particularly convincing in demonstrating that LLMs exhibit this behavior. As currently presented, the paper may benefit from additional evidence to clarify the extent to which these spurious features influence LLM performance. Although the authors even concede this point in section 5.2, for which I am grateful to the authors, I do think this evidence is required for a more compelling case for why the community should care about these types of spurious features in evaluation.
- The method due to its reliance on n-gram is limited to multiple-choice questions. It would be more compelling if different types of methods could be constructed for other types of evaluations, such as open-ended QA.

questions:
- What was the criteria for the datasets chosen, it would be interesting to include MMLU?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646111845

thread_type:
official_review

<thread_object>
creator: reviewer_vh76

<content>
summary:
This paper focuses on exploring the "Clever Hans" effect, also known as the "shortcut learning" effect, in which the trained model exploits simple and superficial correlations instead of intended capabilities to solve the evaluation tasks. The authors demonstrate that simple classifiers (logistic regression) trained on unigrams and bigrams can potentially predict correct answers in several benchmarks. They also provide evidence suggesting that some LLMs may rely on these simple patterns to achieve high performance, potentially undermining the validity of the modern LLM benchmarks.

strengths:
- The motivation is clear and interesting. Focusing on superficial correlations of existing benchmarks can help benchmark design and evaluation for LLMs.
- The methodology of using logistic regression to analyze n-gram features across multiple benchmarks provides a clear and easy way to evaluate the quality of test datasets.

weaknesses:
- While the paper suggests that LLMs might be using n-gram patterns to solve benchmarks, it does not establish causality. It can be helpful to add experimental manipulation to directly test whether LLMs rely on n-grams.
- The findings may not apply universally across all LLMs or benchmarks, as the analysis is limited to specific datasets and model families. And based on the experimental results, only some families of LLMs demonstrate the possibility of relying on spurious correlations.
- The study focuses on a specific type of shortcut (n-grams) and may not capture other forms of spurious correlations that LLMs could exploit.

questions:
Please see the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684842156

thread_type:
official_review

<thread_object>
creator: reviewer_pelq

<content>
summary:
This paper investigates an interesting question about the validity of benchmarks for existing AI research -- whether simple n-gram features can solve these benchmarks. They train logistic classifiers to extract these text features and use them to predict results on multi-choice questions. The experimental results suggest that many existing benchmarks leak with these n-gram features, and potentially limit the validity of benchmarks. 
Furthermore, they test whether LLMs' predictions are based on these features. Their results show moderate evidence for the claim that models from OpenAI, Meta, Mistral AI, and Writer are more likely to provide correct answers for instances that n-gram models also successfully predictions.

strengths:
The paper focuses on an important and timely issue of LLM evaluation -- whether the existing benchmarks are trivial and can be solved by simple text features, and furthermore whether LLMs rely on spurious cues to make correct predictions. 
The motivation and the related work survey across different disciplines are good, providing a basis for future research. 
The results of existing benchmarks that can be solved by simple text features are interesting.

weaknesses:
The main weakness of this paper is the depth and comprehensiveness of empirical results. 
The paper investigates an important question and attracts me in the introduction. However, the empirical results are disappointing. For example, the authors show that text features can solve these benchmarks. What's next? Which dataset is the easiest to solve or leak? What's your advice to improve these datasets? What happens if a more controlled dataset is used to test the same capability? 

In addition, the authors need to provide more comprehensive results about how LLMs interact with these benchmarks. 
Your experiments on LLMs' prediction show only moderate evidence that these features are used. Does it indicate that the leaked information is not used? If so, does it effectively lower the importance of this research? 

Overall, I think the paper is not well-balanced with a big motivation and limited empirical results to support their motivation. 

Missing reference: 
Yan, Jianhao, et al. "Understanding In-Context Learning from Repetitions." The Twelfth International Conference on Learning Representations.

questions:
If a dataset can be predicted by text features, does it mean that the tested capability can be bypassed? 
Furthermore, how does a tested capability differ from identifying the word usage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732020768157

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
The reviewers all agree that the issue we address is relevant and important to improve the state of AI evaluation, and that our approach leads to interesting results regarding the quality of test datasets.

Condensing our responses to the reviewers, we emphasise the following points:
- We find that a large proportion of benchmarks (9/19) allow us to achieve significant predictive power for the ground truth label using simple n-gram features. We then show that some LLMs (of the 44 we considered, belonging to 11 families) perform better on the prompts where this can be done. However, the fact that we do not find evidence for other LLMs does not imply that they are definitely not relying on spurious features, such as other n-gram combinations that our classifier did not identify, or other kinds of features. Instead, the fact that we found an effect for some LLM families by fitting a simple classifier (logistic regression) is significant, as it suggests that this phenomenon could also occur in other LLMs. This highlights issues with benchmark constructions and LLM evaluation. It is therefore paramount that researchers are aware of this issue, as identified by our paper, so that benchmarks can be designed to mitigate it. Ultimately, it should be the responsibility of benchmark developers to ensure that benchmarks are valid and robust tests. Separate research endeavours, like ours, should not be expected to analyse all sets of benchmarks and identify all possible spurious features.
- Relatedly, our analysis focuses on a specific type of spurious features, namely, unigrams and bigrams. Previous works (e.g., Gardner et al. 2021, Tu et al. 2020, Kavumba et al. 2022) mostly studied the effect of individual unigrams or bigrams on LLM performance. In contrast, our classification approach considers the combined effect of multiple n-grams simultaneously, thereby significantly expanding the range of spurious correlations under consideration. As noted in Sections 1 and 5.2, other forms of spurious features may also be predictive of ground truth labels and leveraged by LLMs. Clearly, it is impossible to consider all possible features in a single work. However, we would like to stress again how the effect we find with unigrams and bigrams indicates that similar issues may arise with other features as well. Hence, our findings raise concerns about the current state of benchmarking, underscoring the importance of developing benchmarks robust to a wide range of spurious features.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses an interesting topic in the evaluation of large language models: the internal validity of benchmarks. Specifically, the authors investigate whether benchmarks designed for LLMs can be solved using superficial patterns, such as n-grams. By training ngram based classifiers, they demonstrate that these classifiers can achieve high performance on several multiple-choice benchmarks. Furthermore, the authors provide moderate evidence that some LLMs might also exploit these superficial patterns, by comparing ngram based predictions with LLM's predictions.

Strengths of the Paper
1. The paper addresses an important and underexplored issue in LLM evaluation, the internal validity of datasets.
2. The use of logistic regression classifiers to identify spurious n-gram patterns is reasonable. It provides a straightforward and interpretable way to evaluate dataset quality.

Weaknesses of the Paper

While the assumption and experiments are interesting, I agree with the reviewers that findings and analysis of this work are also a bit 'superficial'. First the study does not clearly expand its impact to 'how to improve LLM benchmarks'. In the rebuttal the authors suggest "n-gram features should not be predictive of the label" is unclear and may not be completely correct cause I believe the ability to extract semantics from ngrams is also important to LLMs. So perhaps we should always consider ngram classifier as a baseline and presents the delta improvement of LLMs over the baselines? Generally, the paper would be much stronger if it can propose actionable, detailed methods to mitigate this issue.
Also, the study is limited to multiple-choice tasks and n-gram features, leaving out other benchmark types and other potential spurious features.

In conclusion, while the paper raises critical concerns and provides an interpretable methodology, its findings are relatively limited in depth and scope. The lack of actionable mitigations and narrow task/feature focus restrict its potential impact. I agree with the reviewers that this paper is marginally below the acceptance threshold.