Mapping Overlaps in Benchmarks through Perplexity in the Wild
Authors: Siyang Wu
Honglin Bao
Sida Li
Ari Holtzman
James Evans
Venue: ICLR 2026 Poster (2026)
Keywords: meta-evaluation
benchmark overlaps
language models
Primary area: datasets and benchmarks
TL;DR: We introduce benchmark signatures, sets of in-the-wild tokens whose LLM perplexity predicts benchmark performance, to quantify (questionable) benchmark overlaps such as logic and language.

Abstract: We introduce benchmark signatures to characterize the capacity demands of LLM benchmarks and their overlaps. Signatures are sets of salient tokens from *in-the-wild* corpora whose model token perplexity, reflecting training exposure, predicts benchmark performance. We extract them via stepwise forward selection with linear regression in a meta-evaluation spanning 32 LLMs and 89 benchmarks across diverse domains. We then analyze how these signatures relate to both the semantic similarity of benchmark questions and the correlation structure of model performance. While performance correlations are uniformly high and semantic overlaps stay in a narrow mid-range, benchmark signatures reveal more nuanced structure. For instance, they uncover substantial overlap between benchmarks in knowledge and reasoning tasks, whereas benchmarks in culture- and humanity-oriented domains show low similarity with each other. Unlike raw performance correlations, which are influenced by benchmark-*orthogonal* factors such as question formats, signatures are robust to such confounds. We further identify cross-functional overlaps between logic, math, language, instruction following, and cultural/world modeling, with coding emerging as the most isolated function, interacting only moderately with the ability of detecting missing information. Qualitative analysis shows that only the knowledge signature aligns with actual knowledge, suggesting that LLM semantic organization may differ from human conceptual structure. Together, these findings offer insights into benchmark validity, LLM sensitivities, and the landscape of interconnected LLM capacities. We have open-sourced the code and data in [this GitHub repository](https://github.com/siyangwu1/Benchmark-Signature-Repository).

Published as a conference paper at ICLR 2026
MAPPING OVERLAPS IN BENCHMARKS THROUGH PERPLEXITY IN THE WILD
Siyang Wu Honglin Bao Sida Li"
Ari Holtzman
James Evans
Data Science Institute, University of Chicago
f
The first three authors, S.W., H.B., and S.L., co-led the project and contributed equally
The last two authors, A.H. and J.E., co-supervised the project
& Correspondence to: {honglinbao, aholtzman, jevansy@uchicago.co
ABSTRACT
We introduce benchmark signatures to characterize the capacity demands of LLM benchmarks and their overlaps. Signatures are sets of salient tokens from in-the- wild corpora whose model token perplexity, reflecting training exposure, predicts benchmark performance. We extract them via stepwise forward selection with lin- car regression in a meta-evaluation spanning 32 LLMs and 89 benchmarks across diverse domains. We then analyze how these signatures relate to both the semantic similarity of benchmark questions and the correlation structure model perfor- mance. While performance correlations uniformly high and semantic overlaps stay in a narrow mid-range, benchmark signatures reveal more nuanced structure. For instance, they uncover substantial overlap between benchmarks in knowledge and reasoning tasks, whereas benchmarks in culture- and humanity-orientel do- mains show low similarity with each other. Unlike raw performance correlations. factors such as question formats,
which are influenced by signatures are robust to such confounds. We further identify cross-functional overlaps between logic, math, language, instruction following, and aultural/world modeling, with coding emerging as the most isolated function, interacting only moderately with the ability of detecting missing information. Qualitative analysis shows that only the knowledge signature aligns with actual knowledge, suggest- ing that LLM semantic organization may differ from human conceptual structure. Together, these findings offer insights into benchmark validity, LLM sensitivities, and the landscape of interconnected L.M capacities. We have open-sourced the code and data in this GitHub repository.
coding 0.12
logic 0.03 -0.20
Ps
instruction
0.55
MMLL-MCO
language-
0.15
-1.0
math-
0.08
-0.18
0.21
0.18
0.11
0.08
model
world
0.05
0.04
0.17
0.20
0.36
0.10
0.22
BB MCO
tell missing
0.05
0.18
0.30
0.16
0.15
0.05
0.21
0.26
Inoeledge
coding
Inglc
Pramuttion
language
remsriin
world
Figure 1: Left: Signature correlations across functions. Right: Performance alignments are biased
(red areas: benchmark families question formats: Multi-Choices Fruse..

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Benchmarks have been central in the growth of large language models (LLMs): they catalyze progress, standardize evaluation, and enable systematic mooss comparisons, thereby influ- encing the trajectory of Al research. The community has witnessed an accelerating proliferation of benchmarks across wide range of L.M abilities, such reasoning (Tafjord et al., 2020) and agentic capabilities (Zhu et al., 2025), as well as real-world scenarios such as finance (Zhang et al., 2023) and safety (Mou et al., 2024). The dedicated "Datasets and Benchmarks" track in leading venues such as NeurlPS and KDD highlights both the importance and steady growth of this area. Each year witnesses many new benchmark papers. From 252 submissions the NeurlPS Datasets and Benchmarks Track in 2021 to 1,820 in 2024 the number of benchmark papers has increased more than sevenfold. While these resources often claim to assess distinct capabilities, it is frequently unclear whether they truly do so, or whether they merely capture narrow proxies, prompt-specific heuristics, or even overlapping skills that have already been extensively tested elsewhere, making them less unique and useful than advertised. This raises critical questions: Do we really need such a vast and ever-expanding suite of benchmarks? How much overlap exists across them? Answering this question will also reveal the converse: What areas of capability are sparsely underrepresented by benchmarks and might benefit from more?
In this paper, we undertake a comprehensive meta-evaluation with particular focus on identifying and analyzing benchmark overlap, which we define as the degree to which two benchmarks evaluate a shared set of model capabilities. To capture overlap in principled way, we examine it from three complementary perspectives. At the semantic level, we assess whether the questions in two benchmarks substantially overlap in content or intent; if so, their redundancy is intrinsic. At the performance level, the mainstream level in benchmark agreement studies (Perlitz et al., 2024), we test whether models show highly correlated performance across they measure related underlying abilities even if under surface semantic differences. Finally, at the benchmark signature level- introduced by in Section 3 we move beyond tasks and outcomes to characterize the distributional fingerprint of benchmarks, defined by token-level perplexity patterns on large-scale in-the-will corpora.
benchmarks, indicating that
Why do in-the-wild corpora effectively encode benchmark characteristics? The abilities measured by benchmarks- commonsense, factual memory, scientific reasoning, programming skills, and more do not emerge out of thin air. They stem from the diverse real- real-world text patterns encountered by the model. In-the-wild corpora, consisting of large-scale, naturally authored, multi-domain text and code (news, forums, encyclopedias, textbooks and notes, papers, documentation, blogs, and repositories), are produced for human communication rather than adapted for benchmark design. They are rich in task-bearing structure (question-answer, problem-solution struction struction-execcttinn) redundancy (the same function expressed in many ways), and breadth. This breadth of distribution - likely unique to in-the-wild data forms the "soil" from which such capabili- ties grow, and also the source from which benchmark questions are drawn. Even if benchmark item never appears verbatim, its "function" recurs pervasively: unit-aware arithmetic in recipes ("double 11/2 cups"), commonsense causality in narratives ("the glass shattered after being dropped")), claim + measurement + inference chains in scientific abstracts, code repair patterns in GitHub issues ("off-by-one in loop: fix bounds"), and even schema-query mappings ("customers with orders in last 30 days")) Focusing only on synthetic or benchmark-adjacee adjacent data risks capturing artifacts of test design. In-the-wild data, by contrast, mirrors the true distribution that gives rise to these abili- ties, making the overlap between capacity exposure and benchmark competence not accidental but expected.
claim-evidence, in-
Perplexity provides a useful lens for quantifying relationships between skill exposure and bench- mark performance. Low perplexity on passage suggests that the model has seen similar linguistic and conceptual patterns during training and is familiar with the content. High perplexity, by contrast, indicates unfamiliarity and underrepresentatton.. Thus, the distribution of perplexity values across large corpora serves as a fingerprint of the model's 's training exposure and more or less acquired capacity. Importantly, because different benchmarks stress different capabilities, they map onto dif- ferent perplexity distributions when probing across the same corpus. In other words, corpora encode benchmark signatures because benchmarks not foreign entities imposed on the model after train-
hhttps://papercooi lot.
2

---

Published as a conference paper at ICLR 2026
ing, but rather structured samplings of capabilities that themselves emerge from the distribution of in-the-wild data. Perplexity serves as the bridge between exposure and benchmark performance, making it possible identify and characterize these signatures without requiring direct evaluation on the benchmark itself We therefore leverage perplexity as the basis and covariate for salient token selection and signature formation $. The following three levels in this work provide holistic framework: semantics address task design, performance captures model behavior, and signatures reveal a fingerprint of model capacity. The overlap between benchmarks across each of these lev- els highlights the interconnected capacity space an oft-discussed yet caafccultttoffommme and so represents a promising tool for evaluating benchmark validity. This rationale is illustrated Figure 2.
concept
Benchmark Signature
Functions from in the wild data
Math Benchmark
0.92
Model2
the largest desertss is Sahora
OD
0.23
Model1
the derivative of sinja is cosjs)
0.78
0.56
Model
the Great Wail from Chiine
AI
0.02
default HOTHPPP 80
cura in time Ojn log
Figure 2: Overview of the rationale of how in-the-wild corpora implicitly encode the benchmark signature, knowledge exposure (capacities), as well as benchmark performance.
Definition: Benchmark Signature
A benchmark signature defined of salient tokens T, extracted from large-scale in-the-wild corpora, such that the perplexity of a collection of language models M on T is highly predictive their performance on the benchmark.
To achieve the overall process, we make the following three contributions:
We introduce a systematic framework for measuring benchmark relations and especially their over- lap across three levels: semantic, performance, and signature derived from model perplexity.
We develop a forward selection and regression- based pipeline to extract these signatures by mining and filtering token-level perplexity statistics from in-the-wild corpora.
We uncover unexpected overlaps between widely used benchmarks. While these benchmarks intended to test a specific ability such as logic and their problem sets do align with human intu- itions about logic, in practice they often measure instruction-followii ability in language models instead. This reveals the potential issue of benchmark design and actual execution, as well as the interconnected space of L.LM capabilities.
SEMANTIC OVERLAPS AND PERFORMANCE OVERLAPS
General Notation. We denote the collection of m LLMs by Mi...., Mm and the set of n bench- marks by BI...., B.. For any quantity defined jointly over model-benchmark pair-such performance metric we write Vi.j to indicate the metric value corresponding to model M, eval- uated on benchmark Bj. Unless otherwise specified, all vectors are column vectors and set in bold lowercase, c.g. x Rd. Matrices represented with capital letters in bold, e.g. X puxm
Semantic-Level Overlap. For benchmarks B., Bn with question text sets Qa, Qs. let Temin min [Q Let be the embedding dimension, f text -+ R be sentence transformer (e.g. MPNet encoder; Song et al. (2020)), and let s(x, y) be the cosine similarity between x, E R
Mooe discussions about prior works see A. More details see Section and Appendix A.2.
3

---

Published as a conference paper at ICLR 2026
Because benchmarks vary in size (i.e., number of questions), which could bias results, we estimate overlap via size-matched bootstrapping similarity: for T 1000 times, we draw {q)( (a) Qa
and {a)r (b) Qb independently with [9))2 = Ka) (b) Mmin encode each item with f. Also, Coneate() means concatenating list of texts into single string within each set, and computing cosine similarity:
1 Asam Ba, Bb) + Σ (((cemcaa(((a) (conccl((a)))
Overall, this mitigates sample-size bias and yields a more robust similarity estimate. Full procedural details appear in Appendix 1.4.1. The overlap between B, and Bg is defined by the Asem (hem(Be. Bs)-
Performance- Overlap. For each benchmark Ba, let y: I the vector of model performances on B, (one entry per model). The performance level overlap between two benchmarks B, and B, is the Spearman rank correlation between their model marggnnlare performance vectors:
p(Ba, Bb) cort(rank(y,.)) rank(y.,6))
Thus, the overlap between B, and B, under performance-level is defined by the spearman correla- tion which is pl Bu, Bb).
MINING BENCHMARK SIGNATURES FROM IN-THE-WILD DATA
Algorithm Obtaining signature for benchmark B,
Input: Data "in the wild" D, Benchmark Bj, a list of LLMs M1,.., Mm
Output: Signature Sj
I: Vis +- BB, with M1.-.,, Mn b Generate performance column vector benchmark
+- D> Processing in-the-wild data into tokens with preceding context, specifically, the prefix consisting of the 30 preceding segments, where each segment corresponds to a segment defined by space.
3: P +- T With M1.
m. Generate the token-level perplexity covariate matrix.
4: T/ + AIC(THRUSHPREELLTERP
y.,)) Perform Thrush pre- me-filtering first; then stepwise AIC fea- ture selection the covariate matrix against the performance vector Vig benchmark B, to obtain salient tokens.
5:
Retrieve Sj from mapping T'; " in P
6:
return S;
The overall process of mining signatures can be found Algorithm 1 and its details can be found in Appendix 4. Let d denote the number of "in-the-wild" tokens (in our case, tokens drawn from large-scale pretraining corpora*), denoted T = {f,....;fa), where typically scales to billions. For any benchmark B,, our objective in extracting its benchmark signature is to isolate subset of salient tokens T (t/....,ty C that are maximally informative explaining variations in
LLM performance. We formalize this as a regression problem: let &, := (y1.j..... :m,.j T R denote the performance of m language models on benchmark Bj. The covariate matrix P E mxd contains token-level perplexities, where entry P.j- = Pij corresponds to the perplexity of token & under LLM M,. The challenge lies in the high-dimensional regime (d >> m, where d = 8.45 10 and m = 32), where classical regression approaches are ill-posed. To make progress, we must uncover and exploit latent structural properties of the problem. In particular, put forward key assumption and a follow-up question:
1. Sparsity: Most token-level perplexities are uninformative for predicting benchmark perfor- mance, with only small fraction carrying predictive signals. [Assumption 1]
2. Extraction: Assuming sparsity holds, what methods effectively identify and extract these predictive signals from the overwhelming background of noise? [Question 1]
'Progressing from fine to coarse granularity, we have token-, chunk-, and document-level perplexities. We provide more experimental results of why the token-level operation is the best. Details are shown in Ap- pendix A.2.
4

---

Published as a conference paper at ICLR 2026
MBPP
IFEval
BBH
0000
-0000
Thrush carrelation
-1000
-2000
2000
4000
6000
8000
10000
0.00005
8,00110
0.00915
0.00020
0.00000 0.00005, 0/00010 0.00115 00020 0.00025
0,50000
5.00005
0.00015
0,00015
0.00200
Density
Density
Density
Figure 3: Distribution of Thrush correlations in pre-selection phases; red vertical lines mark the 1st and 99th percentiles, highlighting that few features are highly correlated with performance.
Together, they motivate our below regression-bane framework for mining benchmark signatures, which leverages high-dimensional inference techniques disentangle signal from noise and recover benchmark-specee fingerprints of token-level perplexity.
3.1 TOKEN-LEVEL FILTERING WITH PERPLEXITY CORRELATIONS
Answering [Q1] by fitting a full multivariate regression model is computationally intractable given that the number of tokens (d) is orders of magnitude larger than the number of models (m). We therefore adopt a pragmatic and efficient two-stage approach, beginning with a screening step to drastically reduce the feature space. Specifically, for each benchmark, we perform token-by-token correlation screening. We compute robust correlation coefficient between each token's 's perplexity vector and the benchmark performance vector. This screening is highly efficient, requiring linear time in the number of features, O(md), and allows us to observe the empirical distribution these coefficients. In sparse regime, expect this distribution to be sharply peaked at zero, with small tails representing potentially informative tokens.
A known limitation of this screening approach is its reliance marginal, univariate correlations. evaluates each token in isolation, potentially overlooking features that are predictive only a multivariate context (e.g., suppressor tokens that explain residual variance). However, argue this approach is theoretically and empirically well-justified in our specific problem setting for the following reasons:
1. Justification from the Ultra-High Dimensional Regime: Our problem, with d >>>>> m, sides the ultra-high dimensional setting. Theoretical frameworks developed for this regime, such as Sure Independence Screening (SIS; Fan & Ly (2008)), provide formal guarantees for marginal screening. The "sure screening property" ensures that, under sparsity and certain reg- ularity conditions, correlation-basee based filtering can discard the vast majority of irrelevant features while retaining the true predictive signals with very high probability. We further explain how several key conditions SIS are plausible our context in Appendix A.3.
2. Empirical Precedent in Data Selection: This screening methodology has demonstrated strong empirical success in the related domain of data selection for training. Prior work has success- fully used document-level perplexity correlations to filter large corpora, improving downstream model performance (Thrush al., 2025; Shum al., 2025). Their success provides compelling evidence for the practical utility of correlation screening as robust heuristic for identifying in- formative signals in LLM-related data.
While there exist various methods for robust correlation calculation, there is no single "silver bullet"; the choice often guided by the specific properties the data. particular, we highlight the two robust correlation coefficients introduced in the aforementioned data selection literatures.
5

---

Published as a conference paper at ICLR 2026
Definition 3.1 (Thrush Correlation (Thrush et al., 2025)). Fixing the j-th token t, E T. Let (p) denotes the rank of p among {Pi.)'' " PPm.) } and sign(-) be the sign function, we denote sign(yk.j (..)) (PR.g) ranky(pt.))
7
1skcism
the Thrush correlation coefficient. This coefficient is variant of Kendall's (Kendall, 1938), measuring the concordance between model performance and perplexity ranks. It counts the number of model pairs where the model with better performance also has lower perplexity rank concor- dant pair), and subtracts the number of pairs where this is not the case (a discordant pair), making it robust the absolute magnitude of perplexity values.
Definition 3.2 (Pre-select Correlation (Shum et al., 2025)). Letting Z= m=(m--)) 1) to be nor- 2
malizing factor, and (I), (m) be the sorted indices by LLM performances (i.e. M(1) S 9(2).) S
... S 3) the Pre srresselee select correlation coefficient is defined as:
my
E
1(pk. PL,)/Z
Iskcism
The Pre-select coefficient computes the fraction of model pairs that are "misordered" by their token perplexities relative to their benchmark performance. In an ideal scenario where lower per- plexity perfectly predicts higher performance, this sum would be zero; a value of 0.5 would indicate random, uninformative relationship.
Once these robust correlation coefficients calculated for all d tokens, we employ simple quantile-based threshold to screen the feature space, retaining approximately the top % of tokens with the strongest signal. Figure presents the empirical distributions of the Thrush coefficients for three representative benchmarks. In all cases, the distributions are sharply peaked around central value (indicating a random relationship), with thin tails representing tokens that are highly correlated with performance. This characteristic shape provides compelling empirical support for our sparsity hypothesis (IQ1)): the vast majority of token perplexities are uninformative, while a small, identifiable subset carries significant predictive signal.
3.2 REFINING SIGNATURES WITH FORWARD SELECTION REGRESSION
The correlation screening successfully isolates candidate set of potentially informative tokens, sat- isfying our goal of drastically reducing the search space. However, this filtering alone is insufficient to define a robust benchmark signature for two primary reasons. First, the filtered set is likely to contain redundant features; for instance, several top-ranked tokens might represent the same un- derlying linguistic phenomenon and thus offer overlapping predictive information. Second, a true signature should not only identify important tokens but also capture their conditional importance their predictive power given the other tokens already in the model.
To address these challenges and distill final, parsimonious signature, we employ a second-stage multivariate variable selection procedure. Our general framework can accommodate various high- dimensional regression techniques suited for the d' m regime (where d' is the number of filtered tokens, d' z 1.69 x 10"), including penalized methods like Lasso (Tibshirani, 1996), Ridge (Hoerl & Kennard, 1970), or Elastic Net (Zou & Hastie, 2005). In practice, we opt for a greedy forward selection approach, which we find builds interpretable and effective models. This method iteratively constructs the signature by adding the single token from the candidate pool that yields the greatest improvement to the model's fit, penalized by its added complexity.
To guide this selection process, we use the Akaike Information Criterion (AIC; Bozdogan (1987)), which provides principled trade-off between explanatory power and model size, mitigating the risk of overfitting. The process terminates when no additional token can improve the model's AIC score by meaningful amount. The complete two-stage process combining the initial correlation screening to create a candidate set with the subsequent forward selection to derive the final signature is formalized in Algorithm 4.
3.3 SIGNATURE-LEVEL OVERLAP
Consider two benchmark signature vectors, S₁ and S2, each including several pieces of context (30 pieces separated by space) the salient token. We use 32 models to process these signatures, reading

---

Published as a conference paper at ICLR 2026
their respective pre-contexts, producing the last token-level perplexities and calculating overlaps. If the models are confused to a similar degree by both signatures, that is a strong indicator that the two benchmarks align. Since some "weak" models consistently produce high perplexity, we normalize each model's perplexity into its z-score within the model. We then compute the mean of z-scored perplexities of the two benchmark signatures within each model and the Spearman correlation be- tween these two mean lists to represent the signature-lever overlap, aligning with performance level results and indicating models' relative relation of perplexity and skill familiarity on the signature. Refer to Appendix 1 for formalized talk-through.
We further examine the robustness of our framework in four dimensions. First, the robustness of design: we examine the generalizability of the framework, specifically. whether the regression merely overfits the observed data rather than generalizing to unseen models, and the extent which base abilities tested across benchmarks influence the results. Second, the robustness of methods: we assess the robustness of the regularization and screening methods used in the paper and com- pare them to their alternatives. Third, the robustness of parameters: parameter choices such as the 1% pre-filtering threshold. Fourth, the robustness of data: how to approximate the "in- whild" corpora and whether it impacts the major conclusion. We found that our framework is robust across all dimensions, and notably, is easily replicable on a smaller scale with limited computational resources. These details be found in A.7 (robustness) and A.8 (computational cost).
study the robustness of
4 RESULTS
Our experiments are conducted on 32 models and 89 benchmarks, including many of the most widely used ones. We extract benchmark signatures from the open dataset RedPajama (Weber et al., 2024). See Appendix A.5 for the full details of the experimental setup.
4.1 SIGNATURES CAN BETTER DISTINGUISH BENCHMARKS THAN SEMANTICS AND MODEL PERFORMANCE
We first examine how the overlap distribution looks across three levels, as illustrated in Figure 4. To minimize inductive bias, we assign broader categories these benchmarks using the official labels from MMLU (Hendrycks et al., 2021), Big-Bench Hard (Suzgun et al., 2022), ifeval benchmark (Zhou al., 2023), and MBPP (Austin et al., 2021). In signature overlap (panel a). on the left, we compare within within-category overlap against the average cross cross-category overlap. To reduce the impact of benchmark category size, we ensure each category pair is weighted equally. We then use the mean of cross-category overlaps to represent the overall cross-category overlap and apply this con- sistently throughout the paper. We observe that overlap is higher within certain categories such as reasoning, science, and social science knowledge, which is expected: benchmarks designed around the same high-level intent tend to align, whereas pairs such as chemistry vs. history benchmarks overlap far less. Within the humanities and world models, overlaps are generally lower than those in cross-category comparisons. A closer look at these benchmarks suggests that the lower similarities stem from their emphasis on diverse cultural contexts- for example, world-model evaluations that assess understanding of culture-specific phenomena like movies and sports and their reliance on processing humanities-based material such as history from a wide range of countries and regions. Furthermore, within a category, certain benchmarks align more strongly than others. This forms a dense "red clique," " identified by extracting the maximum clique from the overlap graph. We high- light these highly aligned benchmarks on the right side. For panel (b) semantic overlap and panel (c) performance overlap. in contrast, these analyses show much weaker discriminative ability. Semantic overlap scores remain in narrow range (typically 0.1-0.4) regardless of whether benchmarks come from the same or different categories. Conversely, performance-leva overlap is almost universally high, suggesting that model performance and the semantic meaning of questions are less sensitive to category boundaries and obscure grer-ggd underlying associations between benchmarks.
At the semantic level, text embedding models such as MPNet capture surface-level similarity in how humans perceive benchmark questions (Morris et al., 2023). These representations are highly dependent the specific descriptive intention behind question, however, meaning the overlap remains superficial and does not reflect the underlying abilities being evaluated. In other words, identical questions do indicate overlapping benchmarks, but different questions do not necessarily
7

---

Published as a conference paper at ICLR 2026
+2% .66% humanities
ethic marlering
hottry, law, rulginn
-10% language medicine
senanti undersuuning
aging crowisgge geneties
121%
benchmare
onjectcouting nasparion, gosmetri beolcan, dyck anguages
reasoning
+-42% science
charrsity de astonony
**R
+41% Social
grography, colitie, econ, policy, paych..
SCl.
world knowledge
-17%
julgament, caluure
benchmark
(a) Signature-level analysis
0.0
-05
(b) Performance-leve analysis
(c) Semantic-level analysis
Figure 4: Three levels of benchmark relation analysis. The signature-level analysis demonstrates substantially stronger discriminative ability compared to both semantic- and performance-leve anal- yses. All heatmaps are presented using a consistent color range from to I, and panels and e share the same row and column indices articulated in panel Statistical details can be found in Appendix A.6. < 0.05; ** < 0.01; *** < 0.001.
indicate con-overlapping ones in terms of underlying ability. At the performance level. while some overlap was initially observed, it quickly became clear that this too fails to meaningfully separate categories. In fact, performance-level results show strong segregation: model behaviors on certain cross-category benchmarks are as closely aligned as they are within categories (evident in several segregated red areas not on the diagonal). When we examine these unexpectedly high alignments, find that they occur within the same broad benchmark families (e.g., MMLU or BigBench- Hard) or under the same question format (e.g.. True/False versus multiple-choice questions). This -orthooon--ottoonlll effect is even stronger than within-category overlaps that is, MMLU history aligns more closely with MMLU chemistry than with another history benchmark. This underscores the limitations of relying performance alone and highlights deep issues in current benchmark agreement tests. Several factors could explain this pattern: the bias may stem from post-training fine-tuning, and could also reflect contamination of the training data, where exposure one eval- uation within benchmark family increases the likelihood of exposure to others, thereby inflating performance correlations. Another explanation lies in model capabilities: when a model is tested for single ability. the evaluation inevitably involves combination of multiple common skills at a minimum reading, instruction following, and comprehension, among others. This overlap makes behavioral aligoment less distinguishable measure.
4.2 THE EVALUATION BIAS IS RESOLVED BY THE SIGNATURE
Grouping the result in Figure panel b, we observe that the red areas are concentrated within the same benchmark family and question format shown in Figure right panel, where red areas
8

---

Published as a conference paper at ICLR 2026
exactly two benchmark families question formats. We calculated pairwise correlations be- tween benchmarks both within and across families and question formats. Since each family or for- mat contains highly diverse set of benchmarks essentially covering everything would expect wiilii-familyfformatt overlaps to be quite low, showing little difference from cross-familyfformn overlaps. Consistent with this expectation, the signature-level analysis reveals statistically insignifi- cant tiny differences based on the intuition, the signature provides good approximation of the true overlap and variation. In con- trast, the performance-leve analysis shows a large value of overlap (around 0.8) and a statistically significant increase in within- familiimmmmm///mmall overlap. Our results show deep issues current bench- mark agreement tests that LLM performance may be more related to surface-level aspects of bench- marks, such as question format, suggesting both that generalization and knowledge wwweeeeeeepopggttiii in LLMs are limited and that current evaluation may be underestimating peak performance because of conflation performance and competence. Using linear regression to obtain signature filters out the noise associated with the error term while preserving the underlying systematic relationships among benchmarks and performances.
U test, yielding results around O. This aligns with
4.3 SIGNATURES INFORM BENCHMARK DESIGN AND LLM CAPACITY SPACE
As shown in Figure compare overlaps across design functions. Several patterns emerge. First, we observe significant over- laps that align with in- tuition. For example, math and logic correlate at 0.21, which close to the average within-function overlap of 0.285 and far above the average cross- function overlap of 0.105. This makes sense: solv- ing math problem of- requires logical rea- soning, and vice versa. More broadly, logic, in- struction following, lan- guage, math, and world modeling (largely cultural benchmarks) form a clus- ter of interconnected abil- ities. Coding appears far less entangled with other functions. Its low cross-function overlap suggests that coding benchmarks are comparatively "clean," the sense that success relies more specifically on coding competence and less on auxiliary abilities. It only moderately interacts with the ability to detect missing information in a sequence. This distinctiveness might arise because coding requires highly specialized pretraining corpora such as GitHub, which is also one of the three major domains in AbsenceBench (Fu et al., 2025).
Signature
I
0.08
Spearmnn correattio
0.08
0.06
0.06
pm0.805
p=0.071
effect -0.007
0.04
effect
0.04
0.02
0.00
Within
Between
Same format
DifT. format
Performance
0.8
Sacsman correaation
Spearman correlation
p=0.000
effect =0.455
p=0.000
0.4
effect -0.653
0.2
0.2
Within
Between
Same format
Dift format
Figure 5: Biases (within/between families; same/diff. formats) are well addressed by the signature.
There are two broad perspectives for interpreting these results. If we optimistically assume that benchmarks faithfully measure what they claim, then the observed overlaps reveal a genuine inter- dependence of cognitive abilities.
this view, benchmarks are not "leaky," but rather reflect the multifaceted nature of capacity like math and logic. From this perspective, overlap is not noise, but evidence of underlying LLM and human capacity entanglement the interconnected capacity space an often-discussed but previously difficult-to-formma expose a misalignment between what benchmarks intend to measure and what they actually capture. This interpretation suggests that benchmarks are "leaky" in undesirable ways, inadvertently testing skills outside their stated domain. For example, even math and logic are highly related, their over- lap should theoretically remain lower than within-math within-logic overlap. Yet, Figure shows
concept. Alternatively, the overlaps may

---

Published as a conference paper at ICLR 2026
cases where cross-function overlap exceeds futhin-function overlap- for instance, between instruc- tion following and logic. This could imply that either within-function overlap is underestimated (due to poorly aligned benchmark design and execution (Liao et al., 2021)) or that cross-function con- tamination is stronger than anticipated, undermining the clarity of what each benchmark is supposed to isolate.
QUALITATIVE INTERPRETATION OF BENCHMARK SIGNATURES
What exactly signatures? We performed qualitative analysis of the textual signatures. Our ap- proach uses a simple metric of textual similarity: we compare the intended function of benchmark (for example, assessing social-science knowledge) with the textual content of its signature using the model from (Song et al., 2020). We find that when benchmark targets knowledge in specific field, its signature tends to reflect that semantic content the signature is, in effect, "about" the same knowledge domain. In some cases, the cosine similarity reaches as high as 0.4 (e.g., social science knowledge benchmarks). On the other hand, some meta-ability benchmark signatures bear little relation to their intended functions, such as logical reasoning.
Why do some benchmark signatures "match" the stated function while others don't 't? We have three theories: (1) Benchmarks often bundle multiple subskills: beyond the target ability, they depend on instruction following, reading load, and format handling. As a result, signature tokens often reflect whichever auxiliary factor drives the most performance variance. Knowledge benchmarks are cleaner, while abstract meta-ability tasks (c.g., logical reasoning, detecting missing information) more distorted by these side demands and by gaps between task design and implementation. (2) Signatures come from predictive token-level perplexity in natural corpora; when the intended skill is rare or procedural (like "logical reasoning". "detect missing information"), models default to proxy cues cues-genre, discourse markers, instruction -kthherrather than domain-specific features. This problem smaller for well-defined knowledge areas. Also, signatures often include numerals, syntax tokens, or discourse markers that look semantically unrelated, whereas knowledge tasks ap- pear more semantically aligned simply because their signatures form coherent domain narratives. (3) Strong predictive power doesn't imply shared semantics: models can rely on statistical co- occurrences in the natural corpora correlated with appearances of benehmark questions rather than true semantic relations. Semantic embeddings therefore cannot fully approximate models' internal task representations, consistent with findings of transferable but non-human aan-hhmaa--nterpreeee interpretable structures (Musker al., 2025; Wu et al.,, 2024). Benchmark overlap here refers how similarly models are confused by two sets of silent tokens not to the semantic or textual overlap of the signature content. We have list of representative benchmark signatures shown in Appendix A.9.
FINAL REMARKS
LLM benchmark saturation has been widely discussed (Phan al., 2025). Instead of introducing ever harder benchmarks, we propose benchmark signatures, principled method to quantify overlap among L.M benchmarks. We ground benchmark relationships in cross-model perplexity patterns from in-the-wild corpora and compare them to surface semantics and correlated performance. We find signatures robust conchmark-orthogonn factors (e.g.. question format) while revealing both expected and unexpected cross- entanglements. Signatures are defined by the predictive power of tokens: tokens whose model perplexity patterns strongly predict benchmark outcomes, regardless of raw perplexity. Such tokens capture how structural properties of model training align with benchmark capability demands, rather than whether models have merely "seen" the required content. Our findings advance understanding of the LLM capacity space, benchmark validity, and model sensitivities. Future directions include extending signatures to finer mner-grained probes (c.g., layer-level activations and interpretability) and generalizing beyond QA or true-false tasks, such as open-ended generation (summarization, long-form reasoning, and dialogue) that requires sta- ble, reproducible scoring functions. More work on causality would also be valuable. Broadly, our approach suggests a "benchmark algebra" for decomposing, recombining, and comparing bench- marks expose gaps or redundancies, enabling the creation of entirely new benchmarks that target capabilities or failure modes identified through principled analysis. Together, these extensions posi- tion benchmark signatures reusable diagnostic toolkit for evaluating and improving benchmark ecosystems.
10

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We employed L.MMs to assist with polishing the writing. All content generated or modified by LLMs was rigorously reviewed and approved by the authors.
ETHICS STATEMENT
This work does not involve human subjects, sensitive data, or any other issues outlined in the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experiments, we provide detailed descriptions of all methodolo- gies in Sections 2 and 3. In addition, Appendix A.5 contains a walkthrough of cach key checkpoint and experimental setup, including (but not limited to) important numerical values, evaluation met- rics, and the software packages used for implementation.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behi, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng L.,, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengehen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilae, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang. Yu Wang. Rachel Ward. Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu. Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024a. URL https: / arxiv. org/ abs 2404 14219.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024b. URL https: /ttxaaiivvv.////////ea////aaaava orgoorgggooo.... abs/ 0412.08900
01. AI. Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Wen Xie, Wenhao Huang. Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengeheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by Oa.i, 2025. URL https: // www.xii..rgg.e.. abs 2400 04652.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv: 108. 00730773,, 2021.

---

Published as a conference paper at ICLR 2026
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzngg large language models across training and scaling, 2023. URL https: : / /arxiiiii v. wwgg//////o abs 22004 01 01373.
Hamparsum Bozdogan. Model selection and akaike's ' information criterion (aic): The general theory and its analytical extensions. Psychometrika, 52(3):345-370, 1987.
DeepSeek-AL, Xiao Bi, Deli Chen. Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao. Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma. Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang. Liyue Zhang. Mingehuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseck Ilm: Scaling open-source language models with longtermism, 2024. URL https: //arxiv. org/ abs /lws/240 24001. 02954.
Dante Everaert and Christopher Potts. Gio: Gradient information optimization for training dataset selection. arXiv preprint arXiv:2306. /6700, 2023.
Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70(5):849-911, 2008.
Harvey Yiyun Fu, Aryan Shrivastava, Jared Moore, Peter West, Chenhao Tan, and Ari Holtzman. Absencebench: Language models can't tell what's missing. NeurlPS, 2025.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: / zenodo. org/ www.r de / 12608602.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang. Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv. Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b 130b to glm-4 all tools, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravänkumar, Artem Ko- renev, Arthur Hinsvark. Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru. Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra,
12

---

Published as a conference paper at ICLR 2026
Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marein Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla. Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin. Dana Beaty. Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erie-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri. Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Flens Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, -aan-stp Gaya, Jeff Mareus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena. Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev. Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan
13

---

Published as a conference paper at ICLR 2026
Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy. Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang. Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Veti,, Zef Rosnbrick. Zhao- duo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: http///xxvvvvr// alss/ 2407 21783.
David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A Smith, Hannaneh Ha- jishirzi, Kyle Lo, and Jesse Dodge. Signal and noise: A framework for reducing uncertainty in language model evaluation. arXiv preprint arXiv:2508. 13144, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR, 2021.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal
problems. Technometrics, 12(1):55-00 67, 1970.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxivvorg//ao....vv........... /ttp/bb/.............
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux. Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https: http///rxvvvee//e .rgg abs/ / 2401 04088.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
14

---

Published as a conference paper at ICLR 2026
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? meta review of evaluation failures across machine learning. In Advances in Neural Information Processing Systems, 2021.
Nelson F Liu, Tony Lee, Robin Jia, and Percy Liang. Do question answering modeling improve- ments hold across benchmarks? arXiv preprint arXiv:2102.01065, 2021.
Microsoft, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Arkinson, Hany Awadalla, Nguyen Bach, Jiammin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych. Praneetha Vaddamanu, Shuohang Wang, Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang. Ziyi Yang. Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-lore 2025. URL https: /arxivv..../ org/ abs abs/aa/// 22003. 01743.
Justin K Miller and Wenjia Tang. Evaluating Ilm metrics through real-world capabilities. arXiv preprint arXiv:2505.0822 2025.
John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. Text embeddings reveal (almost) as much as text. arXiv preprint arXiv:2310.068/6, 2023.
Yutao Mou, Shikun Zhang, and Wei Ye. Sg-bench: Evaluating Ilm safety generalization across diverse tasks and prompt types. Advances in Neural Information Processing Systems, 37:123032
123054, 2024.
Sam Musker, Alex Duchnowski, Raphaël Millière, and Ellie Pavlick. Lims as models for analogical reasoning. Journal of Memory and Language, 145:104676, 2025.
Shiwen Ni, Guhong Chen, Shuaimin Li, Xuanang Chen, Siyi Li, Bingli Wang, Qiyao Wang, Xingjian Wang, Yifan Zhang, Liyang Fan, et al. A survey on large language model benchmarks. arXiv preprint arXiv:2508. 15361, 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson. Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannanch Hajishirzi. olmo 2 furious. 2024. URL https: : wttaaaiivvvv.. org/ abs 00//2551.000 00656.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The Reb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/ 16, 2023. URI https: / arxiv org/abs /2306 01116.
Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli- Scheuer, and Leshem Choshen. Do these Ilm benchmarks agree? fixing benchmark evaluation with benchbench. arXiv preprint arXiv:2407. 13696, 2024.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint 1024442501.14249, 2025.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28078.. PMLR, 2023.
15

---

Published as a conference paper at ICLR 2026
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Kashun Shum, Yuzhen Huang, Hongjian Zou, Qi Ding, Yixuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts the data that teaches. arXiv preprint arXiv:2503.00N: 2025.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15725- 15788, 2024.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33: 16857-16867, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rXiv:2210.09221,, 2022.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012. 13048, 2020.
Gemma Team. Gemma 2025a. URL https: /gw.. gle Gemma 3Report..
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023. URL www mws.i cm1 www.co/////////oo. hoom/log/mptte
Qwen Team. Qwen3 technical report, 2025b. URL httpst::: //te:////aaivvii/ www.iii.orr/...eee.ooo... hatg//bb///a 2505 09388.
Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using per- plexity correlations. ICLR, 2025.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: open dataset for training large language models. Advances Neural Information Processing Systems, 37: 116462-116492. 2024.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766 2022.
Christopher Wolfram and Aaron Schein. Layers at similar depths generate similar activations across Ilm architectures. COLM, 2025.
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. arXiv preprint arXiv:2411.04900 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. ICML., 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:
34201-34227, 2023.
Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang. Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. Fineval: chinese financial domain knowledge evaluation benchmark for large language models. arXiv preprint 2rXiv:2308.09975, 2023.
16

---

Published as a conference paper at ICLR 2026
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673-5684, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cetruuttionfollowinn evaluation for large language models, arXiv preprint arXiv:231.079911, 2023.
Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, et al. Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825, 2025.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):00--320, 2005.
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 RELATED LITERATURE
Benchmark Categorization and Overlap: Benchmarks central to model evaluation. Two sim- ple metrics capture their utility: signal, benchmark's ' ability to reliably distinguish better mod- els from worse ones, and noise, benchmark's s sensitivity to randomness (Heineman et al., 2025). Recently, researchers have begun to ask how comparable benchmarks with similar intent actually are. This is commonly studied through Benchmark Agreement Testing (BAT), where new bench- marks are validated against established ones using agreement metrics (e.g., rank correlation) (Perlitz al., 2024). Such analyses have led to concerns that the community may be producing too many benchmarks. For example, Liu et al. (Liu et al., 2021) examined agreement across multiple QA benchmarks and concluded that because agreement was high, additional QA benchmarks were un- necessary. Beyond statistical agreement, some recent works have attempted to qualitatively interpret and categorize benchmarks for example, as testing logical reasoning or commonsense reasoning though often without running agreement tests either within or across these categories (Ni et al., 2025). Recent human-curated benchmarks, such as "humanity's 's last exam", explicitly aim to mit- igate saturation (Phan et al., 2025), while our work provides mechanistic explanatory account of why existing benchmarks saturate and overlap in the first place. Another emerging line of inquiry asks what capabilities are still missing from current benchmark suites. Miller and Tang (Miller & Tang, 2025), for instance, examine how people commonly use LLMs for summarization, technical assistance, reviewing work, data structuring, generation, and information retrieval, and assess the extent to which existing benchmarks cover these capabilities. Their findings reveal significant gaps in coverage of benchmarks across categories.
Signal Extraction from In-the-wild Data: A growing body of work investigates how information extracted from in-the-wild corpora can inform data selection and model evaluation, even building benchmarks automatically. A central insight is that LLM losses on in-the-wild texts are often cor- related with downstream benchmark performance, suggesting that simple loss pessperffrrance correla- tion coefficients can be effective signals for identifying high-quality training data from in-the-wild corpus (Thrush et al., 2025; Hoffmann et al., 2022). Validation loss is thus frequently used as a proxy for model generalization (Kaplan et al., 2020; Hoffmann et al., 2022; Wei et al., 2022), and with more recent evidence showing that such correlations persist across architectures and training settings (Poli et al., 2023). One line of research focuses efficient, low-cost methods for un- derstanding and filtering signals, for instance lightweight approaches using surface-level heuristics (n-gram overlap (Xie et al., 2023) or semantic-level similarities (Everaert & Potts, 2023)), enabling scalable filtering of massive corpora. Thrush et al., (Thrush et al., 2025) proposed an orthogonal correlations. We
approach for data selection centered around estimates of perplexity-enchhmade build on these ideas to construct benchmark signatures by mining predictive tokens of LLM perfor- mance from large-scale in-the-wild corpora, in order address challenges in meta-evaluation the evaluation of LLM evaluations, e.g., how overlapping they are.
A.2 COMPARISONS BETWERN TOKEN-, CHUNK-, AND DOCUMENT-LEVER PERPLEXITY
From fine to coarse granularity, we consider token-, chunk-, and document-level perplexities. At entire document and take the mean across all text
the document level, evaluate the model chunks that fit within the model's context window (part of the document). At the chunk level, we split documents into fixed-length windows (30 pieces. using spaces as separators) and compute per- plexity as the average over all tokens within each window. At the token level-the finest granularity with the least inductive bias we use token-wise perplexities from documents to capture the model's intrinsic uncertainty. Concretely, we form window by taking the target token with its up-to-30 30 pre- ceding pieces (using spaces as separators) as context, then record only the last token's 's perplexity as the feature. This ensures the token is conditioned on its preceding context rather than treated in isolation. As shown in Table 1, the token level exhibits the greatest standard deviation and interquar- tile range, as well as more pronounced extreme gaps in majority cases compared to the chunk and doc levels. This wider dispersion indicates that extreme values are more visible and significant at the token level, making it natural choice for feature selection. Token-level signatures balance the strongest predictive power (both positive and negative relations) of highly informative tokens, and they exhibit high variance of predictive power, as captured by the deviations. By focusing on the
18

---

Published as a conference paper at ICLR 2026
token level, are able to highlight more prominent signals, whereas aggregation at the chunk or document level tends to smooth out these extremes.
Benchmark
Level
Std
IQR
Max-Q99
Q01-Min
R
Gsm8k
Chunk
30.30
44.63
43.37
36.63
0.903
Gsm8k
Doc
19.39
26.50
23.25
18.75
0.826
Gsm8k
Token
36.53
54.11
40.63
27.37
0.927
Mbpp
Chunk
29.83
43.05
87.62
27.05
0.849
Mbpp
Doc
14.49
18.78
32.44
8.67
0.667
Mbpp
Token
36.76
50.60
39.60
39.20
0.885
Mmlu
Chunk
29.31
41.18
64.00
30.36
0.551
Mmlu
Doc
18.11
27.68
35.16
9.68
0.185
Mmlu
Token
38.05
51.90
52.00
42.29
0.454
Truthfulqa
Chunk
30.58
41.80
48.00
29.20
0,414
Truthfulqa
Doc 17.67
22.75
25.00
13.00
0.208
Truthfulqa
Token
36.57
53.00
49.20
35.40
0.505
Table I: Summary of Thrush coefficient distributions across four benchmarks. The columns report standard deviation (Std), interquartile range (IQR), and tail gaps of Max-Q99 and Q01-Min, which are defined as the distance from the maximum to the 99th percentile (Max-99th) and from the 1st percentile the minimum (Ist-Min). Adjusted Coefficient of Determination (R is extracted from the actual fit of the linear model across different granularities. Across 20 targets (5 measures x 4 benchmarks), token-level values achieved 15 wins. For the five losses, chunk-level statistics perform slightly better. This is because chunk-level distributions contain more outliers, meaning that the 1st and 99th percentile values can be extremely low or high (where they win), while the standard deviation is not pronounced as that in the token-level case. We thus mainly rely Std and IQR for the final selection. Note that our framework is conceptually extendable to chunk-level and document-level measures. Token-level measures also introduce the least inductive bias (minimal structural assumptions and segmentation artifacts) while offering the highest granularity and more faithful representation of model uncertainty.
A.3 CONDITIONS FOR SURE INDEPENDENCE SCREENING (SIS)
Sure Independence Screening (SIS) is a powerful statistical tool for feature selection in ultra-high dimensional settings, offering "sure screening property" that guarantees the retention of truly in- formative features with high probability under specific conditions (Fan & Lv, 2008). In this section, elaborate how the key theoretical assumptions underlying SIS are plausibly met within our problem context of mining benchmark signatures from token-level perplexities.
1. Ultra-High Dimensionality: Our problem inherently operates in ultra-high dimensional regime, where the number of "in-the-wild" tokens (d, scaling to billions) vastly exceeds the number of language models (m, typically in the tens). Specifically, we have log(d) m, which far exceeds the standard d >> high- sigh-dimensional definition. This extreme disparity makes full multivariate regression computationally intractable, underscoring the necessity of an efficient screening step like the one we employ.
2. Sparsity: The "Sparsity" assumption (our [A1]) posits that only a small fraction of the tokens are truly informative for predicting LLM benchmark performance. Our empirical observations of the correlation coefficient distributions (e.g.. Figure 3) directly support this. The distributions show strong concentration around zero, indicating that most tokens have little to marginal predictive power. The presence of thin but distinct tails also suggests that small subset of tokens exhibits strong correlations, aligning with the idea that specific linguistic phenomena (represented by these tokens) drive performance on given benchmark.
3. Minimum Signal Strength: SIS requires that the true predictive signals (i.e., the tokens with non-zero effects on benchmark performance) are not arbitrarily weak. In our context, this translates to these important tokens having sufficiently strong marginal correlations to stand out from the noise. Our of token-level perplexities, which directly reflect LLM's fami- larity of specific linguistic patterns, suggests that truly important tokens would indeed manifest strong signals. The robust, rank-based correlation coefficients employ (Thrush and
19

---

Published as a conference paper at ICLR 2026
Pre-select) are also well-suited to detect such signals, they are less sensitive to outliers and distributional peculiarities that might obscure signals when using less robust measures.
4. Limited Pathological Multicollinearity: critical condition for basic SIS is that the multi- collinearity between important features and unimportant ones should not be so severe that it masks the marginal signal of truly predictive tokens (e.g., suppressor variable scenario). While token perplexities common linguistic constructs), it is less probable that a truly causal token't signal would be perfectly canceled out by others at marginal level. Benchmarks typically probe specific abili- ties, which are likely associated with a distinct, though perhaps overlapping, set of "signature" tokens. The vast and diverse nature of "in-the-wild" tokens also means that while many tokens might be highly correlated, there are many more effectively independent ones. More impor- tantly, the core objective of our work is identify benchmark signatures parsimonious set of tokens. If token's 's marginal signal is entirely masked, it might suggest contribution is highly redundant with other tokens that do have strong marginal signal, or that its unique contribution is extremely weak in which case, its exclusion from the initial screening might not significantly harm the final signature's predictive power.
exhibit correlations (c.g., highly similar tokens tokens from
specific and
A.4 TECHNICAL DETAILS
A.4.1 SEMANTIC-LEVTH BOOTSTRAPPED SIMILARITY CALCULATION
Algorithm 2 Get Pairwise Similarity Matrix
Input: A list of benchmarks B {Bi...., B, Embedding model E (e.g. sentence trans- former); Number of bootstrap replicates similarity matrix S.
Output: An
1: n + [B]
2: S f n n matrix initialized zeros
3: for i = 1 to n do to do 5: if j then
4: for
6:
S. + 1.0
7:
else
8:
4 getSimScore Bi, Bj, K) & Call Algorithm
9:
S +
10: Sy. +- s & Matrix is symmetric
11: end if
12: end for
13:
end for
14: return
20

---

Published as a conference paper at ICLR 2026
Algorithm 3 Bootstrapped Similarity Score Calculation (getSimScore)
Input: Benchmarks A and B; Embedding model E; Bootstrap replicates k. Output: A single similarity score sim s.
I: Determine which benchmark has smaller size
2: if [A] < [B] then
3:
S A; L +- B S is the smaller, L is the larger
4: else
5: S L+
6: end if
7: n, [S] Get the size of the smaller benchmark
8: l + getMaxLength(EF > Obtain the maximum processing length
9: Process the smaller benchmark get its single embedding
10: textş + concatenate all questions in S
text'g < truncate truncate(textg,,
12: embs <- encode( E,, texts
13: P Generate bootstrap samples from the larger benchmark
14: Tr + an empty list
15: for = to k: do
16: L' sample L, ng, replace False)
17: text,, + concatenate all questions in L'
18: text',
(tect, E)
19: Append text', to T
end for
21: & Batch-encode all samples and compute average similarity
embs + batchEncode( F, Tz)
23: similarities t cosineSimilarity (embs, embs,) & One-vs-many comparison
24: sim f-
25: return S
A.4.2 AIC STEPWISE FORWARD SELECTION ALGORITHM
Algorithm Selecting Salient Tokens for Benchmark j
Input: Perplexity feature matrix P: performance vector y: fraction a = 0.01; tolerance 2
Output: Salient Token set T',
I: Preselection via Thrush Correlation
2: for f to d do
P: ThrustCoor(P.......
4: end for
5: 7* + indices of the top ad values of D most positively correlated
6: T + indices of the bottom ad values of pt most negatively correlated + Shuffle(T* U TTT) candidate feature set
9: Forward Selection with AIC (on T)
A" + do
10: while T V S 11: for each { E (T') S) do
A(l) + (IC(Fit P. su(()))
13: end for
14: 8*
+- arg minrerrss) A(O); Anew +- A(E*)
15:
if ABe <. A" 6 then
16:
5 SU A' Ance
else
18: break & no further AIC improvement
19: end if
20: end while
21: T',
22: return T"
21

---

Published as a conference paper at ICLR 2026
A.5 EXPERIMENT SETUP
Overview:
Our chosen benchmarks span diverse domains such as knowledge (business, humanities, social sciences, science and engineering, medicine), mathematics, coding, reasoning, language, cul- ture and world knowledge, logic, and instruction following. We choose 32 widely-used language models (see the list below). We extract benchmark signatures from the open dataset RedPajama (Weber et al., 2024), which contains large-scale textual data across multiple domains, including CommonCrawl, C4, GitHub, arXiv, Books, Wikipedia, and StackExchange, used for pretraining LLMs, making it a strong source of in-the-wild data for mining benchmark signatures. We take the standard approach, using vLLM wow et al., 2023) for facilitating perplexity extraction and 11m- 1lm-eva eva I lutttiinnn ooiioonbae -aiiionn-aaeessss harness (Gao et al., 2024) for evaluation across benchmarks and models such that all evaluations are under the same condition.
A.5.1 EXPERIMENT WALKTHROUGH
As discussed, measure perplexity at three granularities token, chunk, and document levels (from fine to coarse). The segmentation procedure for each is detailed in $A.2. We ultimately focus on the token level because it provides the clearest view of prominent signals for the pre-fitterng stage.
Preprocessing RedPajama We use the IB-token RedPajama variant to balance scale and compu- tational cost. For token-level segmentation, we split the corpus whitespace into pieces. For each piece, we prefix up to the preceding 30 pieces as left context and record the last token' 's perplexity conditioned on that context. This yields an initial pool the scale of billions of token-level texts (d z 8.45 x 10%). To reduce noise or in-the-wild text, we uniformly downsample by a factor of 1/50, yielding approximately 1.69 10? instances.
Feature Matrix Construction Using the vLLM setup described in $A.5.4, we evaluate 32 models on the token contexts and extract token-level perplexities, forming the covariate (feature) matrix
P R 32xxxxx0xxxx 10?
with rows indexed by models and columns by token instances.
Performance Matrix Construction In parallel, we compute model performance on series of benchmarks and subfields using the Lmm-vallattonn-hazaae (details in $A.5.5). Let
Y 32x89
denote the performance matrix (models benchmarks/subiield))) For each benchmark Bj, the vector Y.j is the performance vector for Bj across all models.
Filtering with Thrush For each benchmark Bj, we compute the Thrush rank correlation between the entire feature matrix and the performance vector Y:.-- This produces distribution of Thrush scores over token features. We retain the top 1% and bottom 1 % features (by score) and concatenate these extremes into a benchmark -speciffi subset of columns from for downstream modeling.
AIC Step-Forward Feature Selection Finally, for each benchmark By. we fit multivariate linear model on the preselected features using step-forward selection with the Akaike Information Crite- rion (AIC) as the objective. Starting from an empty model, we iteratively add the feature that most improves AIC and stop when no further improvement is possible (tolerance 0). The resulting selected set constitutes the most predictive in-the-wild token features for Bj. Across benchmarks, the selected set size varies but typically has ~ 30 features.
Signature and Comparison Consider two benchmark signature vectors, S₁ and Sz, each con- sisting of several context pieces (30 pieces separated by spaces) plus the salient token. For each benchmark we acquire around 30 such non-overlapping salient tokens. We evaluate these signatures with 32 models, which read their respective pre-contexts and compute last-token perplexities. the models exhibit similar levels of perplexity for both signatures, this strongly suggests that the two benchmarks align. We normalize each model's perplexity values into their z-score within the model. For each model. we then compute the mean of the z-scored perplexities for the two benchmark sig- natures. Finally, we calculate the Spearman correlation (p.) between these two mean vectors to represent signature-level overlap.
22

---

Published as a conference paper at ICLR 2026
A.5.2 DATASETS
Benchmark family
Benchmarks
mmlu
gusiness_ethics, marketing: management; prrfessional_acconntii
jurisprudence: humanities; prehis- highhschhol.curopaan.iiii world religion;; high righ_schooluss historyy for- tory; professional
lol.logic; high sorlol word.hoyorr international law; logical logical fallacies; moral_disputes: scenarii:,, philosophy; anatomy; clinical knowledge; college_ medicine; human aging; medical genetics; nutrition; profes-
sional medicine; virology; miscellaneous; other; abstract algebra;
astronomy;
college_chemiisee chemistrry
canceptual.physteee
elementary ceteentarrymmahmmatisss
high_school biolooy:
school computer science;
college_computee computer science;
col-
mage.mathemmtics:
computer computer.security:
high_school, cctemmssceemitty;
high_school matiommaittttttss tigh_school school_physsics
high_school wchsocoolossscc statississ machine learning: stem; high_school school geoggrahyy
sigoooccho..syyyhloogg
us us.foreign.poliey. econometries; human_sexualit; professionallpsychoeeee psychologyy security studies; social sciences; sociology
bbh
global facts; hyperbaton; snarks; web of lies; word sorting; disam- biguation salient_translan www.staaiaonnnnnommo _eroooreeerrreeee _detectio;;; boolean boolean expressions;
dyck languuas;;; geometric shapes; multistep multistep.arithmeett arithmetic_
navi-
gate; bojjectccunttng: reasoning aboutt.abbtt.cc bobouutcolloedd....o colertd..bjeets: formal fallacies
.edductoon wil.ifivveeeeeeeee dobeebse
logical reduction_sevent sewiooossseeenooeeettteee objects;
oogical.deduction..te tracking shuffled objects five objects;
penguins crwuunssin_aatabll
tempo-
ral sequences;
track-
ing shuffled objects seven objects; tracking shuffled objects three objects; judgemt; roveeerecommennationn ruin_ ruin_names; date understaddn::::
sports woorsssundessinnding
Table 2: Benchmarks in MMLU and BBH.
Format
Benchmarks
multi-choice questions
business-ethics; marketing: management; professional accounting; wwwoppononoppoannn..oooccoc
high.school european history; jurisprudence: humanities; prehis- tory; professional
world
high-school.nd .uchooo...hhistrry:: for-
mal mal_logic; high_school school_ world history; international law; logical fallacies:
moral disputes; moral scenarios; philosophy; anatomy; clinical knowledge; _
nutrition; profes- mediinnne virology; miscellaneous; other; abstract
college_medicing
human_aging: medical_geneticcn
college_physice astronomy: college_ biology; college.chee .chemistry;
conceptual physics; elementary mathematics;
high school biology:
school scinnee science;
college computer science; scienne:
col-
high_school .cetmistyyemmityy computer computer_securityy security;
lege_ gage_mmateeaatiss
electrical eeeetriallengineering. high school_physseeeee
high_school mathematics;
_ttatistic;; machine learning: Jearning
stem; high_school geogrrahh: high www.schoooomacoccooooiiiiiiiooio
htthsschoolgovermmeee and polities:
microocconomics: high.schoolmmicrccoommoo
high.school
sigh_scchol..pycchlogg
relations;
foreign policy; econometries; human sexuality; professional psychology:
security studies; social sciences; sociology
true false
boolean_ee setieaanneepessionsss sports uooottsunderstanding: web_offlies
sausal.judgement
formal fallacie;;
navigate;
Table 3: Classification of benchmarks by question format.
23

---

Published as a conference paper at ICLR 2026
A.5.3 MODELS USED
The models we used this study are summarized Table 4. Each entry cites the official paper available, otherwise the model card.
Model (HF Repository)
Citation
Grattafiori al. (2024)
veta-lllama/Llmma-3, 1
Grattafiori et al. (2024)
-.aalllamm//amma-33333------
Grattafiori al. (2024)
googlelgemmm----44-iit
Team (2025a)
google/gemmm--3-12b-it
Team (2025a)
google/gemmm--3277b-ii
Team (2025a)
Jiang et al. (2023)
deepscck-al/deppsseell---10 7b-chat
DeepSeck-AI al. (2024)
Qwen/Qwen3-0.66
Team (2025b)
Owen/Qwen3-1.77B
Team (2025b)
Qwen/ Bween/Qee334B
Team (2025b)
Qwen/Qwen3-SB
Team (2025b)
tiuue/ffaconnrr--1b
Penedo et al. (2023)
EleutherAU/pythia-bb
Biderman al. (2023)
EleutherAU/pyythiao-.9---0
Biderman et al. (2023)
Eleuther-\/gyythhe-112--deup
Biderman al. (2023)
mosaicnl//mpt-7---nsnccct
Team (2023)
microsofiPhhi3-miiii---iiimmmmm
Abdin et al. (2024a)
microsot//hhi-4mmiinnnnsstttt
Microsoft et al. (2025)
microsoft/Phi-4
Abdin et al. (2024b)
01-ai/Yi-1.5-99BCCH
AI al. (2025)
01-ai/Yi-1.56BB-Char
AI et al. (2025)
mistrrail/MMiiistaalB-----s-----
Jiang et al. (2024)
operai-communityyp--mmdiii
Radford et al. (2019)
openai-communiiyypp-laaagg
Radford al. (2019)
openai-commmuniyygp2222x
Radford et al. (2019)
zai-org/Chattln33-66
GLM et (2024)
zai-org/llm-4-9b-chaathh
GLM et al. (2024)
zai-org/coodegeex4--l11.bbb
Zheng et (2023)
1itemm/OOLLM--221114-11B---uuuu
OLMo et al. (2024)
allenai/01.M66-211244-----mmm
OLMo et al. (2024)
OLMo et al. (2024)
Table 4: Models used in this study. Multiple variants may share the same citation.
A.5.4 INFERENCE WITH VL.LM
We use VLIM (Kwon et al., 2023) for facilitating perplexity extraction. Specifically, we run all inference offline mode with tensor parallelism across GPUs maximize throughput and data parallelism across GPU. The model weights cached locally to reduce repeated I/O overhead, and inference performed in batches of prompts to further amortize the computation cost. For each sequence, we extract per-token log probabilities, from which compute negative log- ligliikl and perplexity metrics. All outputs are aggregated into parquet for downstream analysis. The GPUs used in this computation are 2 Nvidia A (80GB).
A.5.5 BENCHMARK EVALUATION AND LABELING
For benchmark evaluation we use lim-evalute ion -onnnnseeesss (Gao et al., 2024). Each bench- mark involves different task formats, and we adopt the standard metrics defined in the harness to ensure comparability. For MMLU, which consists of multi-choice questions across 57 academic subjects, we report accuracy, i.c., the proportion of questions with correctly selected options. For MBPP, code generation benchmark, evaluate using pass@) 1, the fraction of problems solved
24

---

Leave-One-Out Cross- Validation (LOOCV) To assess generalization of the proposed frame- work, we performed LOOCV over 32 models on the 27 BBH sub-tasks, comparing our predictor to baseline that uses the mean predict the held-out model's performance. Our model achieved an Published as a conference paper at ICLR 2026
correctly the first attempt, based on unit test execution. For BBH, which is collection of hetero- geneous tasks (multiple-choice, binary classification, and completion), we follow the harness in ap- plying the canonical metric for each benchmark. We accuracy for multiple-choice and true/false items, and exact match for sentence completions. For IFEval, which tests instruction-following adopt the harness's compliance accuracy, quantifying the percentage of model responses that satisfy the explicit constraints in the prompt. These heterogencous metrics reflect the intended difficulty and modality of each benchmark, and together provide broad view of model capability, For Ab- senceBench (Fu et al., 2025), we use the average scores across three dimensions: numerical, poetry, and GitHub.
we
For each benchmark we follow these rules to label its function:
If it's about math problems, then label it "mathematics" including MMLU abstract algebra, elementary mathematics, college mathematics, high school mathematics, and high school statistics (Hendrycks et al., 2021).
Coding - drawing from the MBPP benchmark (Austin al., 2021).
Instruction Following drawing from the IFEval benchmark (Zhou et al., 2023).
Scientific Knowledge - MMLU domains such as business, humanities, natural science and engineering, social sciences, and medicine.
Language (BBH) semantic understanding, name disambiguation, entity resolution, grammar rules, and sarcasm detection.
World Knowledge (BBH) cultural and general world knowledge, including common practices and presuppositions (mostly) in Western society. Examples of world knowledge tasks include the following: Sports Understanding, Movie Recommendation, and Date Un- derstanding.
Logic (Formal Logic) abstract study of proposiitions/stateemnnt and deductive arguments (e.g., MMLU's formal logic and logical fallacies).
Reasoning - (BBH) tasks spanning arithmetic (e.g., multi-step arithmetic), logical struc- tures (e.g.. Boolean expressions, deduction), geometric (e.g.. geometric shapes), hierar- chical (c.g.. Dyck languages), spatial (e.g., navigation), and temporal (e.g., temporal se- quences).
AbsenceBench (Fu al., 2025) the ability tell what's missing.
Note that we mostly refer directly to the official labels (e.g.. what falls under "reasoning", "world knowledge", etc.) given in the official article of Suzgun et al. (2022) (section 5) without making changes.
A.6 STATISTICAL ANALYSIS OF BENCHMARK RELATIONS
As shown in Table 5, we used bootstrapping approach evaluate whether the signature correla- tions within a benchmark category differ statistically from those across categories. For each pair of benchmarks, we computed the overlap between their signatures, which reflects how similarly the chosen set of representative LLMs are "confused" by the two benchmarks (details see Section 3.3 in the main text). Because the numbers of within-category and cross-category pairs differ, we performed 10,000 bootstrap samples to estimate the distributions and corresponding p-values. The resulting difference, expressed as a positive or negative percentile value, indicates how much larger or smaller the within-category mean correlations are compared to the mean cross-category corre- lation, and whether this deviation is statistically significant. Results are discussed in the main text Section 4-1.
A.7 ROBUSTNESS ANALYSIS DESIGN, METHODS, PARAMETERS, AND CORPORA
A.7.1 ROBUSTNESS ANALYSIS OF DESIGN
25

---

Published as a conference paper at ICLR 2026
Benchmark Category
Difference
P-value
Humanities
-66%
0.00
Reasoning
+21%
0.00
Language
-10%
0.15
Medicine
+6%
0.54
Science and Engineering
+42%
0.00
Social Science
+41%
0.00
World Knowledge
-17%
0.01
Business
+2%
0.70
Table 5: Within- and cross-category benchmark differences.
MAE of 0.076 + 0.038, substantially better than the baseline's 0.181 + 0.144. This confirms that the extracted features capture genuine, reusable structure rather than overfitting. We further tested robustness with strict out-of-sample evaluation using two fully held-out models Qwen2. 5 and Falcon-rw-7b. Using the 27 training models to derive features and forecast performance across 27 benchmarks, obtained MAEs of 0.0681 + 0.034 and 0.0722 = 0.026, respectively. These results show that the model generalizes well even to unseen observations.
Confounds of the "Base" Ability To empirically adjudicate whether our signatures capture dis- tinct, subject-specific factors versus generic "dataset style" (e.g., common "factor" shared across all BBH/Wikipedia tasks), we conducted Cross-Task Transfer Experiment. If the alternative hy- pothesis is correct that signatures primarily capture a generic capability or dataset artifact signature derived from Benchmark A (Sa) should successfully predict the performance differ- ent Benchmark B (YB), provided they share that generic source.
Experiment Setup:
We performed the following for all 27 BBH subtasks: For a given benchmark A, we fixed its co- variate feature matrix from its signatures XA. We then trained 26 separate regression models, each regressing the performance vector of a different benchmark YB (where B & A) against X, and col- lecting the absolute error of the prediction on B from X against the true value of Yn. We iterated this process for all A (all subtasks in BBH), resulting in a comprehensive evaluation of how well one task's signature predicts another task's performance. We also refit XB vs. YB again and extract the absolute error each time to obtain the task-specific baseline.
Results:
Baseline (predict based on mean; LOOCV from the general response): 0.181 + 0.144
Cross-Task Prediction: 0.223 + 0.048
Task-specific baseline: 0.021 0.012
The observed MAE is significantly larger when attempting to use one task's signature to predict another compared using the task's signature (0.223 vs. 0.021). Crucially, the cross-task error is even higher than the naive LOOCV baseline (0.181), where we simply used the mean of the performance vector. If the signatures merely encoded generic "style" or shared data contamination. the covariates of Task A (Xx)) would serve as a sufficient proxy for the capabilities required by Task B (YB). Under this hypothesis, training on XA predict YB should yield good fit. The observed degradation in performance contradicts this, indicating that XA encodes specific, non-transferable structural information unique to Task A.
A.7.2 ROBUSTNESS ANALYSIS OF METHODS
Ablation Study of Regularization Methods We ran an ablation study replacing AIC-based for- ward selection (abbreviated as "AIC" below) with Lasso (a = 1) and Elastic Net (a = 1, rattio 0.5), keeping all other regression settings fixed. For each MMLU benchmark, derived a signature vector under each method and computed the corresponding inter-benchmark correlation matrix. We then flattened each matrix's upper triangle and measured Spearman correlations (p)
26

---

Published as a conference paper at ICLR 2026
among regression methods to assess whether the structural relationships among benchmarks persist under different regularizers. Results are:
AIC vs. Lasso p = 0.763
AIC Elastic Net p 0.765
Lasso vs. Elastic Net 0.786
As baseline, using 50 randomly sampled features (after the correlation filtering) produced 0.334 against AIC. These give us two interesting insights:
1. As expected, the initial filtering helps retain (marginally) informative tokens, so signatures constructed from random sampling have non-trivial but weak correlation with regression- based counterparts.
2. The moderate-to-stront correlations among regression methods show that while the chosen features may vary, the between-benche benchmark structural relationships revealed by the signa- tures remain largely stable across regularization strategies.
Spearman Correlation or Mutual Information for Screening: We now test alternative selection methods the preselection phase. To validate our choice, we compared Thrush against Spearman correlation and Mutual Information (MI). We normalized all scores to a [0, 1] scale and analyzed the standard deviation (Std) of their distributions across benchmarks. Our analysis reveals that Mutual Information consistently exhibits lower Std, indicating significantly lower discriminative power compared to the ranking metrics.
Benchmark
Metric thrush
Std (0-1 Scale)
MBPP
0.1549
MBPP
spearman
0.1562
MBPP
mutual info
0.1057
IFEVAL
thrush
0.1557
IFEVAL
spearman
0.1557
IFEVAL
mutual_inft 0.0926
BBH
thrush 0.1808
BBH
spearman
0.1808
BBH
mutual info
0.0981
Furthermore, when comparing Thrush and Spearman, we found they produce nearly identical selec- tions, with a 99.5% overlap in selected features. Given this equivalence and its high discriminative capability, we maintained the use Thrush.
A.7.3 ROBUSTNESS ANALYSIS OF DATA SELECTION
While it is true that our signatures extracted from RedPajama, this does undermine their relevance.
First, RedPajama broadly representative of "indthe-wild" web data including Wikipedia, GitHub, C4, arxiv, etc, which forms the dominant component of modern LLM pretraining, As noted in (Wolfram & Schein, 2025), LLMs are increasingly converging because major model families trained on similar mixtures of large-scale web corpora, code, and curated text. In other words, although individual datasets differ at the margins, they share substantial structural and statistical overlap. Crucially, our goal is not to reconstruct or identify the exact training data of any model. Rather, we aim to capture generalizable distributional signatures that emerge across large in-the- wild corpora. Because RedPajama reflects the broad characteristics of public web text and because model training corpora largely draw from the same underlying data universe RedPajama provides sufficiently representative substrate for extracting robust signatures. Thus, the method does not depend on exact training data matching. Instead, it leverages the empirical regularities large-scale in-the-we wild text, which shared across most contemporary LLMs. Second, to validate the robustness of our signatures against training data variations, we conducted control experiment using Dolma (Soldaini et al., 2024), another massive training dataset derived
27

---

Published as a conference paper at ICLR 2026
from diverse sources (including C4, arXiv, and others). We replicated our exact pipeline Dolma: preprocessing, downsampling, and extracting perplexities to generate signatures for all BBH sub- tasks and computing the correlation matrix that captures the inter-correlation between subtasks. By flattening the upper triangles of the matrices and calculating the Spearman correlation between the matrices derived from RedPajama and Dolma, we obtained high agreement of 0.895. This strong correlation demonstrates that the task signatures are robust to the specific choice of corpus, provided that the data is sufficiently large and representative sample of in-the-wild data.
A.7.4 ROBUSTNESS ANALYSIS OF PARAMETER CHOICES
Motivation for the 1 % pre-filtering threshold
The pre-fitternng step guided by both statistical and computational considerations.
Statistically. The distribution of robust Seature-outcone correlations in dataset is approximately bell-shaped (see Fig 3). The I % threshold (capturing the top/bottom tails) a conservative heuristic designed encompass the heavy-tailed components (roughly > 2.3 standard deviations under normal approximation) where the signal appears concentrated.
The subsequent regression step scales as O(md?), where d' is the number of features after pre-filtering. Applying 1% cut reduces dimensionality by roughly two orders of magnitude, ensuring that the second stage remains tractable.
Computationally.
The overall idea is that thresholds substantially above 1% make the pipeline computationally diffi- cult, while thresholds that too small risk filtering away important features.
To demonstrate that our chosen preselect ratio is robust parameter rather than an arbitrary choice, we performed fine-grained sensitivity analysis on the BBH dataset. We examined the structural evolution of the model's understanding across a geometric grid of ratios
E {10-6. ..03.
Experiment Setup
For each ratio in the grid, generated 27 27 inter-benchmate correlation matrix (heat matrix) representing the pairwise relationships between tasks. To quantify structural stability, we compared the heat matrix ratio rı that at the increment 1 We flattened the upper triangle of each matrix and computed the Pearson correlation (p) between consecutive steps.
Results
The table below tracks the stability the heat matrices. Each value represents the correlation between matrices constructed under r; and ri+1-
Grid Transition (7: r(+1)
Heat Matrix Correlation (p)
10 10
0.2429
10-5 + 10-4
0.3923
10-4 + 10-3
0,6310
10 3 0.01
0.9143
0.01 + 0.1
0.9837
0.1 > 0.2
0.9990
0.2 + 0.3
1.0000
0.3 + 0.4
1,0000
0.4 + 0.5
1.0000
0.5 + 1.0
1.0000
Conclusion
Visual inspection of the heat matrices combined with the quantitative correlation analysis reveals clear phase transition. At low ratios (r < 10-3), benchmark relationships are volatile. The structure stabilizes significantly by Beyond r = 0.2, the heat matrices become identical (p x 1.0), confirming that selecting a ratio between 0.01 and 0.1 efficiently captures the stable, intrinsic structure of the BBH tasks without
0.01 (p > 0.91) and effectively converges by 0.1 > 0.98).
28

---

Published as a conference paper at ICLR 2026
requiring full feature saturation. The reduction in computational complexity substantially outweighs any residual sensitivity to the parameter choice.
A.8 ANALYSIS OF COMPUTATIONAL COST
Our analysis in this section is structured in two parts. We first explain the computational cost, and then use experiments to show why our method is less vulnerable the scalability challenge.
Computational Cost and Scalability. Admittedly, we face computational constraints similar to prior work such as LESS (Xia et al., 2024). Because Stepwise AIC scales linearly with sample size (N); processing larger corpora becomes difficult. At 1B tokens, perplexity extraction takes approximately 2 hours per model, followed by 30-40 minutes of pre-filtrrigg and regression per benchmark. Scaling to 10B tokens would increase these times to roughly 20 hours for extraction and 5 hours per benchmark for regression. This renders 100B+ scales computationally infeasible under our setting.
However, found that IB tokens sufficient capture robust structural relationships among the benchmarks. Our experiments below, which vary the token count from 100K to 10B (10x the original scale), clearly demonstrate that the stability of our method exhibits strong diminishing returns beyond the IB-token threshold.
Stabilization Analysis. Using fixed 1% pre-selection ratio, we sampled corpora at logarithmic intervals (100K-10B) and computed task-to-task correlation matrices for the 27 BBH sub-tasks at each scale. Stability was measured via Spearman correlation between consecutive scales:
100K vs. 1M: 0.278
1M vs. 10M: 0.187
10M vs. 100M: 0.475
100M vs. IB: P 0.998
IB vs. JOB: p L.000
Conclusion on Scalability. The results above demonstrate a "phase transition": signatures are unstable below 100M tokens but converge (p > 0.99) in the 100M-1B range. Since 1B tokens act as sufficient statistical proxy, we can use a manageable subset rather than scaling linearly to trillions of tokens. The empirical study further shows that appropriate down-sampling yields nearly identical results while saving approximately an order of magnitude computational resources.
The results also indicate that 100M tokens are empirically sufficient to capture signatures without incurring large-scale regression costs. Although our study employed a IB-token sample, appropri- ate down-sampling remains feasible for researchers operating under computational constraints who wish to reproduce our findings.
A.9 EXAMPLES OF BENCHMARK SIGNATURES
We show benchmark example signatures and coefficients of their predictive power for benchmark performance. The last part of each text (within '[J') is the silent token with the coefficient in paren- theses. They all high-predee geggh-rreditive-powweee signatures.
Medicine:
initiation of methadone and repeated at 30 days and then annually to evaluate the QT interval well if the methadone dose > 100 mg/day or if the patient experiences [unexplained] (coefficient = 0.157)
Humanities:
His considerable political power to her ends. SALEM's dramatic tension flows from the relationship between Mary Sibley and John Alden, who still love each other but now find themselves [antagonists] (coefficient 0.153)
Math: small shifts AD, either to the right or the left, will have relatively little effect on the output level Yn, but instead will have a greater effect the [price] (coefficient 0.059)
29

---

Published as a conference paper at ICLR 2026
Raven The central symbol of the story that represents depression and evil. The narrator's name-calling of the bird escalates into a rant by the narrator, including: wretch, thing of [evil] (coefficient 0.073)
Coding:
Logic: code "CUN" => 'L', "CCN" 'P', "CGN" => '3', "ACN" -> 'T', "GUN" => 'V' "GCN" => 'A', "GNN" => 'G', "UCN" => ' function string.translate (seq: :Ast
@assert [length (seq) (0.096)
Instruction following: services include general printing, variable data printing, security printing as well as document and imaging management solutions. PNMB is located in Putrajaya, a planned city located 25km south of the capital of [Malaysia] (-0.101)
AbsenceBench (Fu et al., 2025): It's very useful screening tool, ACENET, she says, is essential for her work. Some of these models can take days to run without [ACENET] (-0.037)
30


Reviews:
[{"review_id": "Ywgod6wB6j", "reviewer": "Reviewer_XQJb", "summary": "This paper exposes a critical flaw in benchmark categorization that benchmark overlap is dominated by superficial factors like quention format and benchmark family, rather than the actual capabilities, and proposes \"Benchmark signatures\" to reliably describe benchmark features. Key findings from this analysis include:\n- Logic, math, IF and language form an interconnected cluster\n- Coding stands distinctly alone from other domains\n- Many \"logic\" benchmarks actually measure instruction-following\n- Multilingual / cultural benchmarks show low within-cateogry similarity\nThe takeaway is the current evaluation methods conflate test design with capability. Signature-based analysis reveals what LLMs actually learn.", "questions": "1. Can signatures from 24 models predict the performance ranking of the held-out 8 models on benchmarks (held-out test for predictive validation)?\n2. What are the examples of actual salient tokens for specific benchmarks? Do they align with human intuition?\n3. Please add confidence intervals to Fig. 4 and conduct pairwise tests for within category vs. cross category differences.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The benchmark signature concept is novel and elegantly bridges pretraining data characteristics and benchmark performance through perplexity. The mechanistic view (rather than correlational view) reveals intrinsic capabilities that benchmarks measure rather than superficial artifacts.\n2. The paper repurposes proven perplexity-correlation methods from data selection literature for benchmark characterization, and provide rigorous theoretical justification through SIS framework. It explains why the proposed method work through rigorous theoretic analysis.\n3. The paper reveals that cross-function overlap (instruction following vs. logic) exceeds within-function overlap in some cases, shows interesting observation that some benchmarks designed to test \"logic\" acutually measure instruction following, and this further demonstrates the power of the methodology which leads to non trivial findings.", "weaknesses": "1. Table 1 shows token level has the greatest standard deviation and interquartile range, and the paper claims this indidcates token level is the most informative. However, this is not necessarily true: the higher vairance could also originates from larger noise. It is better to show signal-to-noise ratio analysis (like R^2) for token vs. chunk vs. doc level.\n2. All analysis uses the same 32 models for extraction and comparison. Cross-validation and held-out test results are absent.\n3. Across 16 targets, token-level values achieved 12 wins, which indicates 25% of loses. Please provide analysis of when and why token level loses.\n4. AIC is a greedy algorithm known to be suboptimal, why not use LASSO, elastic net, etc.? Could you provide stability analysis of feature selection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KrcO9BpK4s", "reviewer": "Reviewer_tBme", "summary": "This paper introduces benchmark signatures: sets of salient tokens from in-the-wild corpora where LLM perplexity predicts benchmark performance. Using 32 LLMs and 88 benchmarks, the authors compare three overlap measurement approaches: semantic (question similarity), performance (score correlations), and signature-based (perplexity patterns). Key findings: (1) signature analysis better distinguishes benchmarks than semantic/performance approaches, (2) performance-level results are strongly influenced by benchmark-orthogonal factors, (3) coding is the most distinct domain while math/logic/language show cross-functional overlap.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I like the idea of signature. The signature approach elegantly unifies semantic and performance perspectives: signatures inherently contain semantic information (salient tokens from meaningful contexts) while perplexity directly hints at performance. This is a creative and well-motivated synthesis.\n2. The experiments are done in Impressive scale: 32 models × 88 benchmarks with multi-level analysis provides comprehensive coverage.\n3. There are good theory grounding. Good justification via SIS theory for ultra-high dimensional feature selection and two-stage filtering approach appropriately handles the d >> m problem.", "weaknesses": "1. All signatures are extracted from RedPajama only, yet the 32 models were trained on diverse corpora. It remains unclear whether RedPajama signatures are representative of these varied training distributions. Furthermore, proprietary models may rely on private training data containing unconsidered signatures. If signatures are biased, can they still meaningfully reflect model capabilities? Robustness experiments are needed to validate the approach.\n2. The claim that signature correlations reveal \"model capabilities\" is questionable. Two alternative explanations exist: (a) High within-dataset vs. between-dataset correlations may simply reflect training data contamination, as authors acknowledge; (b) Same dataset may inherently represent a single capability independent of intended subject divisions—for instance, if all MMLU subjects source from Wikipedia, signatures may capture \"Wikipedia-style capability\" (or even spurious correlation) rather than genuine subject-specific skills (history vs. chemistry). The perplexity-performance link could be coincidental. Can you do more causality analysis on that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kZaXUCtVyj", "reviewer": "Reviewer_MMq7", "summary": "This paper introduces benchmark signatures to quantify overlaps among LLM benchmarks by mining salient tokens from in-the-wild corpora where token-level perplexity predicts benchmark performance. Using 32 LLMs and 88 benchmarks, the authors apply Thrush correlation filtering and stepwise forward selection to extract signatures. The work examines three overlap levels: semantic similarity, performance correlation, and signatures. Key findings show signatures discriminate better than semantic or performance measures, performance correlations are biased by format and benchmark family, and coding is the least overlapping domain while logic, math, language, and instruction-following are interconnected.\n\nDespite the writing issues, I highly recognize the motivation of this work. I believe the problem this paper addresses is extremely important and has troubled many researchers working on large language model pretraining. The perspective that in-the-wild corpora implicitly encode the benchmark signature is quite interesting. The discovery that performance-based benchmark agreement is contaminated by format and family effects is valuable for the community. The community should encourage bold papers that tackle important problems. Soundness can be gradually supplemented during the rebuttal phase. This represents ambitious thinking we need for fundamental issues in LLM evaluation. With revisions addressing scalability and adding qualitative analysis of signatures, this will be a strong contribution.", "questions": "Can you provide analysis of how signatures change with corpus size? Even experiments with 100M, 500M, and 1B token subsets would show whether signatures stabilize. What is the computational cost and how does it scale to 10B or 100B tokens?\n\nPlease show examples of actual salient tokens. What are the top tokens for MMLU math versus MBPP coding? Do they make intuitive sense?\n\nWhen instruction-following and logic show high signature overlap, what do the shared tokens look like? This could distinguish between genuine interdependence versus contamination.\n\nHave you validated signatures on held-out models not used in extraction? This would show signatures capture fundamental properties rather than overfitting.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The motivation is strong. Benchmark submissions have grown sevenfold and we need methods to understand if new benchmarks measure distinct capabilities or just repackage existing ones. The insight that in-the-wild corpora implicitly encode benchmark signatures through perplexity patterns is interesting and provides a theoretical connection between pretraining exposure and benchmark performance.\n\nThe experimental scope with 32 models and 88 benchmarks is substantial. The finding that performance correlations are more influenced by question format and benchmark family than actual function (Figure 1) exposes a real limitation in current benchmark agreement studies. The observation that coding benchmarks are relatively clean and distinct is useful.\n\nThe statistical framework using Thrush correlation screening and AIC-based forward selection is reasonable for the ultra-high dimensional setting.", "weaknesses": "The writing has issues. It feels like the authors have limited writing experience. The abstract is not very clear and could benefit from referencing how other papers structure theirs. Also, the last line of the last page is not aligned to the bottom. This directly affects the clarity of presentation.\n\nMy main concern is computational cost and scalability. The authors use only 1B tokens from RedPajama, then downsample by 50x to ~16.9M token contexts. Our large model in-the-wild datasets are on the scale of tens of trillions of tokens. It is unclear how this work would handle larger scales. From Section A.5.1:\n- Initial scale: ~8.45 × 10^9 tokens (RedPajama 1B variant)  \n- Downsampling: 1/50\n- Final scale: ~1.69 × 10^7 tokens\n- Feature matrix: P ∈ R^(32 × 1.69×10^7)\n\nThe complexity of generalizing this needs discussion. A previous work addressing similar scale is LESS [1]. Getting scaling behavior from 1B → 10B → 100B tokens seems quite difficult, but the authors should discuss this.\n\nThe paper never shows what the actual salient tokens are. They extract ~30 tokens per benchmark but provide no examples. What tokens predict math performance? What about coding? Without this, the signatures remain black-box features. This is a significant missed opportunity.\n\nMethodological choices lack justification. The authors mention Lasso or Elastic Net but choose forward selection for \"interpretability\" without demonstrating this. No comparison with Spearman correlation or mutual information for screening. More ablations would help.\n\nThe paper discusses two interpretations of overlap - genuine cognitive interdependence versus \"leaky\" benchmarks - but never resolves this. When instruction-following and logic overlap more than within-logic benchmarks, the authors could use signature tokens to distinguish these interpretations but do not.\n\n[1] Chen et al., \"LESS: Selecting Influential Data for Targeted Instruction Tuning,\" ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vRNoqx7n5B", "reviewer": "Reviewer_nELw", "summary": "The paper introduces the concept of benchmark signatures that can be used to characterize and compare LLM benchmarks. The signatures are defined as a set of salient tokens drawn from large corpora such that the perplexity of various LMs on those tokens predict their performance on a benchmark. The paper’s contributions are (1) a novel two-stage pipeline to derive “fingerprints” of a benchmark. The pipeline involves perplexity-based screening to identify candidate tokens, and then applying stepwise regression to select a spare set of tokens more predictive of performance. (2) A framework for multi-level benchmark comparison. (3) Empirical insights into how current LLM benchmarks may be redundant or divergent.", "questions": "-  If classical regression methods are ill-posed for this high-dimensional problem, did the authors consider regularization techniques or matrix factorization/filtering methods? The use of Lasso or Elastic Net regression is mentioned as an extension but is not explored in this paper.\n- For the THRUSHPREFILTER step, the paper mentions using \"approximately the top 1% of tokens with the strongest signal” but doesn't specify the exact threshold. Is there a more principled data-driven procedure for selecting this threshold; for example, using cross-validation or analyzing the correlation distribution?\n- Relatedly, how sensitive are the results to the fixed variable choices, such as 1% thrush filtering and up to 1500 tokens used in forward selection?\n- While code is provided, certain details of the experimental setup are not explicit in the paper. For example, data processing details for extracting signatures from the RedPajama corpus would be helpful; e.g., how the corpus was tokenized, how large $d$ was, and how contexts for tokens were chosen.\n- The paper would be stronger if it included concrete examples of signature tokens or token clusters for a few benchmarks. Including a table of signature tokens for different benchmarks and their regression coefficients would enhance interpretability.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The introduction of benchmark signatures is a new way to quantify what a benchmark tests. It introduces a principled multi-level framework for analyzing benchmark redundancy. \n- The idea of a misaligned benchmark intent – e.g., logical reasoning benchmarks ending up testing a model’s instruction-following capability – is a notable discovery worthy of further investigation.", "weaknesses": "- The paper’s general hypothesis that pretraining familiarity affects performance has been noted in works on benchmark saturation. For example, Humanity’s Last Exam (HLE) was created specifically to address the saturation of MMLU by providing expert-written questions. The paper doesn’t mention HLE or similar broad evaluations that address saturation or overlapping knowledge issues.\n\n      Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., ... & Wykowski, J. (2025). Humanity's last exam. arXiv preprint arXiv:2501.14249\n\n- It appears the evaluation focuses on academic QA-style tasks (e.g., MMLU, Big-Bench Hard), many of which are multiple-choice or short-answer. This omits classes of benchmarks, such as open-ended generation tasks like summarization or dialogue, where the signature approach may be harder to apply because it inherently works better with well-defined quantitative performance metrics. This focus should be acknowledged as a limitation. \n\n- Some choices in the signature extraction pipeline could use further justification. The authors use a fixed threshold (top/bottom 1% by thrush correlation) for pre-filtering, and then a greedy forward selection up to 1500 tokens. Why 1% and how was 1500 chosen as the max features? There’s no comparison to regularization methods like Lasso or Elastic-Net, which the authors mention as a possible extension but do not report results for. \n\n- A conceptual weakness is the paper’s implicit conflation of model capability and pretraining familiarity. By design, the signature tokens are those that models find easier (lower perplexity) and that correlate with better benchmark scores. This risks a tautology since models may perform well on a benchmark because they have seen similar content before. For benchmarks aiming to test reasoning beyond memorized knowledge, a high signature overlap might indicate the benchmark is inadvertently easier for models that have seen certain clues. In other words, the approach might not distinguish generalization ability from training-set overlap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)