Title: Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Venue: NeurIPS 2025
Accepted: yes
Keywords: language models

benchmarks

evaluation
Abstract: Developing large language models is expensive and often involves making decisions with small experiments, typically by evaluating on large, multi-task evaluation suites. In this work, we analyze specific properties which make a benchmark more reliable and useful for such decisions, and interventions to design higher-quality evaluation benchmarks. We introduce two key metrics that show differences in current benchmarks: signal, a benchmark’s ability to separate better models from worse models, and noise, a benchmark’s sensitivity to random variability between training steps. We demonstrate that benchmarks with a better signal-to-noise ratio are more reliable when making decisions at small scale, and those with less noise have lower scaling law prediction error.  These results suggest that improving signal or noise will lead to more useful benchmarks, so we introduce four interventions designed to directly affect signal or noise.  For example, we propose that switching to a  metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and scaling law error. We also find that filtering noisy benchmarks such that they have better signal-to-noise ratio leads to more reliable evaluations. We also find that averaging the output of a model's checkpoints to reduce noise leads to consistent improvements. We conclude by recommending that those creating new benchmarks, or selecting which existing benchmarks to use, aim for high signal and low noise.  We use 30 benchmarks for these experiments, and 465 open-weight language models from 60M to 32B parameters, resulting in a new, publicly available dataset of 50K evaluation benchmark results, totaling 200M instances.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies two metrics, signal and noise, to evaluate the reliability and usefulness of benchmarks in decision making during LLM deployment, e.g., model architecture, training approach and training dataset. Signal measures the ability of the benchmark to distinguish different models, while noise measures the sensitivity to randomness during training. The paper discovers that signal to noise ratio (SNR) is highly correlated with decision accuracy. Finally, the paper discusses ways to improve SNR.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength in Quality**: The paper conducts abundant experiments to support the claim. In Section 4, the experiment is conducted on 25 models and three tasks, so the results are convincing.\n\n**Strength in Significance**: The paper contributes to benchmark design by introducing a practical metric. SNR is easy to compute as it only involves small models. Moreover, it is highly correlated to decision accuracy.\n\n**Weakness in Clarity**: The notations are not clearly defined. For instance, in Equation (1), the range of $B(\\cdot)$ is not given (i.e., does it output a scalar or a vector or a model?), and $m_a$, $m_b$ are not defined (Though $m$ is defined as a model, what does the subscript mean here?) Also, in line 146, and 160, both use the subscript $i$, but the former refers to training step, while the latter refers to model. \n\n**Weakness in Significance**: SNR computation (Equation (2)) relies on a hyperparameter $n$, which is set differently in experiments (5 in Section 4.1, 30 in Section 4.2). In practice, it is hard to determine the best choice of $n$.\n\nQUESTIONS:\n1. How to choose $n$ in Equation (2)? A practical guidance on choosing $n$ greatly improves the significance of the work.\n2. SNR is dependent on the models. If a benchmark has high SNR on a set of models $\\mathcal M$, will it be reliable and useful for decision making on a new model $m$ that is not in $\\mathcal M$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for analyzing language model evaluation through signal-to-noise ratio (SNR) to reduce uncertainty in benchmark-based decisions. The authors define \"signal\" as a benchmark's ability to distinguish between models of different quality (measured by relative dispersion of scores) and \"noise\" as sensitivity to training variability (measured by relative standard deviation across final checkpoints). The paper proposes three interventions: filtering noisy subtasks, averaging checkpoint scores, and using bits-per-byte metrics instead of discrete accuracy measures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe experimental evaluation is comprehensive, encompassing 465 open-weight models from 60M to 32B parameters across 30 diverse benchmarks, resulting in 50K evaluation results. The systematic correlation analysis between SNR and both decision accuracy and scaling law prediction error provides strong empirical validation of the framework's utility.\nThe three proposed interventions show consistent improvements across multiple evaluation settings. The bits-per-byte intervention is particularly compelling. The checkpoint averaging intervention provides a simple but effective noise reduction strategy.\n\nWeaknesses\nThe choice of relative dispersion (max difference divided by mean) as the signal metric I believe to be quite problematic. This measure is highly sensitive to outliers and may not accurately capture a benchmark's discriminative power. More robust measures like interquartile range or statistical depth-based metrics would provide better characterization.\nMany of the reported improvements, while consistent, are relatively small. For example, the filtering intervention improves MMLU decision accuracy by only 2.6%, and averaging noise improves the 30-task average by 2.4%. Given the computational costs of implementing these interventions, the practical significance may be limited.\n\nQUESTIONS:\nHow sensitive are the results to the specific choice of relative dispersion for measuring signal? Can you test alternative signal metrics that might be less sensitive to outliers?\n\nHow well does the SNR framework generalize to other critical decisions in LLM development, such as architecture selection, training procedure optimization, or safety evaluation? (Even theoretical arguments would be nice)\n\nIt was a bit unclear to me why relative dispersion and checkpoint-level noise specifically should predict decision accuracy and scaling law reliability? \n\nCan the authors provide a systematic analysis of when these interventions are cost-effective given their computational overhead and the magnitude of improvements achieved?\n\nLIMITATIONS:\nThe authors provide a reasonable discussion of limitations, acknowledging the focus on specific experimental settings and the need for future work on other sources of noise. \n\nThe evaluation focuses primarily on two specific use cases (dataset selection and scaling law prediction) but doesn't address other critical decision-making scenarios in LLM development, such as architecture choices, training procedure selection, or safety evaluation trade-offs. As well as mentioning about the potential brittleness of the signal metric to outliers and distribution assumptions and theoretical understanding of why these specific definitions work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes specific properties which make a benchmark more reliable and useful for assessing LLMs, and interventions to design higher-quality evaluation benchmarks. \nThey discuss two key metrics of a benchmark : signal and noise. Signal measures the ability of the benchmark to differentiate between better and worse models, while noise measures the variability of different training steps of a given model on the benchmark. They show that benchmarks with high SNR are more reliable to extrapolate decisions made at smaller models to larger models, and less noise leads to lower scaling law prediction error. To use this insight to design better benchmarks, they try some interventions like using metrics which have higher SNR, and averaging the model outputs from multiple checkpoints. They also identify smaller subsets of existing benchmarks, which have a higher SNR, and improve the performance on both the tasks, despite having less than half the number of samples of the full benchmark\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The problem discussed in this paper is highly relevant in current time, when new LLMs and benchmarks are released at a rapid rate, making it hard to determine which benchmarks are reliable for comparing different LLMs. It is even more useful for someone developing LLMs : training smaller language models and evaluating them on benchmarks to extrapolate the performance of the bigger models is a very common practice, since it’s infeasible to experiment with training the full model directly. \n- There are extensive investigations done to tease out the effect of signal and noise on the decision accuracy and scaling law prediction error. The figures related to the investigations are comprehensive. \n- The paper is very well-written, with clear descriptions of the key terms like signal, noise, decision accuracy, scaling law prediction error. I like that the authors explored different notions of noise, instead of just choosing one. \n- The hypotheses are well investigated through extensive experiments.\n\nQUESTIONS:\n- By definition, computing the “signal” of a benchmark assumes that the models being evaluated actually differ in their quality/abilities. How does one determine if this is truly the case for a given set of models? It seems like a recursive problem, to select the right set of models to evaluate the benchmarks, and the right benchmarks to evaluate the models?\n\n- For the experiments on averaging checkpoint-to-checkpoint noise, the text in section 5.2 says that you average the target model in both settings, and only change whether you avg the smaller models used to fit the scaling law. However, in Table 1, right table, you use the term “Avg Train”, which is not clear. Does it mean averaging both the small and target models?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles a “meta-evaluation” problem: how to evaluate the reliability of benchmark suites when we must make model-selection or scaling decisions from small-scale experiments. The authors design two complementary properties of a benchmark—signal (its power to separate stronger from weaker models) and noise (its sensitivity to stochastic variation such as checkpoint choice)—and show that the signal-to-noise ratio (SNR) strongly predicts decision accuracy and scaling-law extrapolation error across 30 diverse benchmarks and 465 open-weight language models (60 M – 32 B parameters). They then introduce four practical interventions—metric replacement (e.g. perplexity over accuracy), noisy-benchmark filtering, checkpoint averaging, and other dataset refinements—that each raise SNR and yield more robust conclusions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Rather than analyzing individual models or tasks, the paper tackles a \"meta-evaluation\" problem: it studies how we reason from small models to large ones using an evaluation benchmark, a rarely explored but important area.\n2. Using pairwise ranking accuracy to quantify decision reliability avoids strong distributional assumptions and is easy to interpret.\n3. Section 3.1 convincingly shows that averaging the final n checkpoints is a stable and unbiased proxy for intrinsic variance under some assumptions (same/similar learning rate schedules).\n\nWeaknesses:\n\n1. The Decision Accuracy experiment is very close to the DataDecide paper, and the Scaling-Law Prediction Error setup mirrors prior work. More discussion/comparison is required to acknowledge the contributions in prior work and discuss novel contributions in this paper.\n2. The custom “Decision Accuracy” formula feels ad-hoc and non-standard; a rank-based statistic such as Spearman’s Correlation Coefficient (https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) is similar, and might be more familiar.\n3. Missing comparison to EvalArena. EvalArena (https://github.com/crux-eval/eval-arena) also analyzes benchmark signal-to-noise ratio but is not cited or compared.\n\nQUESTIONS:\nCould you provide examples of the datapoints you filtered out (e.g. from MBPP) and clarify whether they were incorrect, ambiguous, or otherwise problematic?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-04"}