AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
Authors: Michael J Ryan
Yanzhe Zhang
Amol Salunkhe
Yi Chu
Di Xu
Diyi Yang
Venue: ICLR 2026 Poster (2026)
Keywords: evaluation
LLM-as-a-judge
metrics
human feedback
open-ended tasks
user-centered evaluation
data-efficient evaluation
automatic metric generation
benchmarking
Primary area: foundation or frontier models, including LLMs
TL;DR: We use LLMs to automatically generate and validate task-specific evaluation criteria (metrics) that correlate well with human judgements, and release a library/framework for automatic metric induction.

Abstract: Evaluating user-facing AI applications remains a central challenge, especially in open-ended domains such as travel planning, clinical note generation, or dialogue. The gold standard is user feedback (e.g., thumbs up/down) or behavioral signals (e.g., retention), but these are often scarce in prototypes and research projects, or too-slow to use for system optimization. We present **AutoMetrics**, a framework for synthesizing evaluation metrics under low-data constraints. AutoMetrics combines retrieval from **MetricBank**, a collection of 48 metrics we curate, with automatically generated LLM-as-a-Judge criteria informed by lightweight human feedback. These metrics are composed via regression to maximize correlation with human signal.  AutoMetrics takes you from expensive measures to interpretable automatic metrics. Across 5 diverse tasks, AutoMetrics improves Kendall correlation with human ratings by up to 33.4% over LLM-as-a-Judge while requiring fewer than 100 feedback points. We show that AutoMetrics can be used as a proxy reward to equal effect as a verifiable reward. We release the full AutoMetrics toolkit and MetricBank to accelerate adaptive evaluation of LLM applications.

Published as a conference paper at ICLR 2026
AUTOMETRICS: APPROXIMATE HUMAN JUDGMENTS WITH AUTOMATICALLY GENERATED EVALUATORS
Michael J. Ryan Yanzhe Zhang Amol Salunkhe Yi Chu Di Diyi Yang*
Stanford University American Express
m chael jry.nnstaannnnnnneee
ABSTRACT
Evaluating user-facing Al applications remains a central challenge, especially in open-ended domains such as travel planning, clinical note generation, or dialogue. The gold standard is user feedback (e.g., thumbs up/down) or behavioral signals (e.g., retention), but these are often scarce in prototypes and research projects, or too-slow to use for system optimization. We present AutoMetrics, frame- work for synthesizing evaluation metrics under low-data constraints. AutoMetrics combines retrieval from MetricBank, collection of 48 metrics we curate, with automatically generated LLM-as-a-Judge criteria informed by lightweight human feedback. These metrics composed via regression maximize correlation with human signal. AutoMetrics takes you from expensive measures to inter- pretable automatic metrics. Across 5 diverse tasks, AutoMetries improves Kendall correlation with human ratings by up to 33.4% over LLM L-u-as--eJudge while requir- ing fewer than 100 feedback points. We show that AutoMetries can be used as proxy reward to equal effect as verifiable reward. We release the full AutoMet- ries toolkit and Metrie Bank to accelerate adaptive evaluation of LLM applications.
Prohibited Content Avoidance
26.3%
LLM Generated Description
Human Feedback
product
Product
*Prompt Optimized LLM Judge*
23.6 0%
Moisturizing
Yardley of London
Yardley Summer Aloe and Avocado 11...
indulge the light, refreshing scont Moisturiang Soao Sweet
an repert eus Locor e-commence product deentipione. Follw theee caps rigeoualy MeMap Tequimmmens" 2. Euluutionn ... Conton Accuracy 4. "SBD Mikoo*
Summer ce and
Avocardo 3+1
Voice (Rubric)
19.9%
MERMAID Vegetable
Glyoenn Bar Soap
Excerience ca the refreshing sensation of the
MFRMAID Vegenabiee Clynwir Bar Soap.
aqua fresh is
adive
quicel creae parsuasve
LA Splash Cosmetics
Achiow salvely matte finish that lasts all
Search Engine Optimization
15.3%
Rec lipetick-
Liquid Matue Bloo
Maita lipalikk in the shacl Poison Applle...
day with Splash Coometics Liqulc
Strttgik ixdusion keywordd leg, poodudt reme, kiy lexured irprove search engine ne va bilily whil maimaning nntuellev
COUTURE (Poison
"Examples- Based LLM Judge*
14.9%
some gold exoluation examples:
"hidlge iight refra rngg soent
"Dxpenence the mimerinn snnottonn of the MENIAAR
Husan Feedback Data [Likert Scale, Thunbs up/down, Behavioral)
Top AutoMetrics
Figure I: AutoMetrics takes you from expensive measures to interpretable automatic metrics. Here AutoMetrics generates useful metrics for evaluating LLM written product descriptions from user reviews from EvalGen (Shankar et al,, (024b). Percentages indicate relative importance of each metric derived from regression coefficients.
INTRODUCTION
Modern AI systems now demonstrate massively multitask capabilities imparted through extensive pretraining (Radford al. 2019 Brown al. 2020). Practitioners can rapidly prototype new AI- enabled tasks from travel planning to code completion at a pace much faster than the community
This paper reflects the academic work of the authors and does not represent or constitute the views, policies, positions, practices of American Express or its affiliates.

---

Published as a conference paper at ICLR 2026
can craft domain specific metries (Papineni et al. 2002 Lin 2004 Xu al. 2016). This new era, in which large language models can be adapted to virtually any domain, places mounting pressure on evaluation practices. A divide is growing between tasks with easily verifiable rewards, such as math (Glazer et al., 2024] Shao et al. 2024) and coding (Chen et al., 2021). while subjective and open-ended tasks such as writing (Gurung & Lapata (2025) remain difficult to measure. For these
tasks, human evaluation remains the gold standard (Shankar et al. 2024b Chiang et al. 2024).
Unfortunately, human evaluation is costly, slow, and not scalable for every prototype user popu- lation. Reward models offer an alternative (Maih et al. [015 Christiano et al.) 2017). but they typi- cally require thousands of labels. The common alternative rubric-based LLM-as-a-Judde methods (Li et al. 2023; Zhenng et al.] 2023; Liu et al.) 2024), which rely on the assumption that system behavior is clearly defined and are not guaranteed to follow given rubrics strictly (Tripathi et al.) 2025). In reality, practitioners typically have access only to non-descriptive human signals (e.g., thumbs up/thumbs down collected from users). In this setting, the problem not only formulating the rubric, but also discovering the underlying criteria that matter.
This highlights the need for dynamic, task-specific metric learning. Instead of relying exclu- sively human judgment or fixed rubrics, evaluation itself must become adaptive. Current efforts have emphasized making LLMs better evaluators of task-specific criteria (Liu et al., 2024 Kim
et al.; 2025; Anugraha et al. 2025) leveraging rubries to optimize LLMs (Gunjal et al. 2025
Viswanathan et al., 2025) but comparatively little work has focused on automatically generating the rubrics and criteria to be adaptively aligned with human judgment (Biyani al. [0224] Ryan al.)
2025 Dunlap et al.) 2025). Such adaptive evaluation is essential not only for easily assessing new tasks but also for optimizing evaluated systems based real-time user feedback.
We introduce AutoMetrics, a method for dynamic metric induction that turns sparse, non- descriptive human feedback into actionable and interpretable evaluators (Figure I]. Starting from a task description and fewer than 100 human signals, AutoMetrics synthesizes candidate criteria, re- trieves and adapts existing metrics, and composes them through regression into predictive measures of quality. Beyond simply identifying criteria, Auto.Metrics grounds and weighs them, produc- ing metries that are both predictive and interpretable. This approach achieves up to 33.4% higher Kendall correlation with human judgments than LLM-as-a-Judge baselines ($4), is data-efficient only requiring ~80 feedback points (54.6). and even matches verifiable rewards when optimiz- ing downstream AI systems ($5). Beyond accuracy, AutoMetries reveals actionable insights into what users value. We release AutoMetrics as an open- open-source toolki offering the community a powerful new way evaluate and optimize Al applications at the speed of modern development.
RELATED WORK
Metric Collections Prior work has organized collections of metrics primarily for the ease of use on the part of the practitioner. When already using library such as Py'oorch (Paszke et al.) 2019) or Huggingface (Wolf et al. 2020) it's simple to utilize TorchMetrics (Nicki Skafte Detlefsen et al, 2022) or HuggingFace 1 Lighteval (Fourrier al., 2023). Scikit Learn Metrics (Pedregosa al. 1011 and NLTK metrics (Bird & Loper 2004) were created with the same intentions. All text- generation metrics covered by these collections are also contained in our MetricBank collection. Beyond integrating with existing open source libraries, some metric collections are part of ML observability frameworks like Evidently (EvidentlyAl 2025). Galileo (Galileo 2025), Scorecard
(Doe & Devireddy 2024), and DeepEval (ConfidentAl 2025). ). Most metrics are tightly coupled with their observability platform, although Evidently and DeepEval offer open- versions. While DeepEval offers metric recommendation feature, it is based on predefined decision tree of ques- tions like "Does your LLM application use Retrieval- Augmmette Generation (RAG)?" and "Is LLM safety a priority for you?". Most similar to our work is the MetaMetrics collection (Winata et al.) 2025), which computes a regression over multiple task-specific metrics for tasks like image caption- ing and summarization to select the best combination of metrics. We compare our approach with MetaMetriis in Section 4 and find that our core thesis of adaptive metric generation is critical for evaluation in the low-data, novel task settings of interest.
https:://github.com wttp://c////...///////eee/e
cs
2

---

Published as a conference paper at ICLR 2026
Step 1. Generate Matric
Metric
doct for help with entrinual
training human data
4/5
integration
Local Coltural Integration
Travel Paaning Metrcc
Sefety Health Consideration
Ascorenedation Optiors
Local Experiences
PUN
Speciiiity and Detei
Speclicity and Hubric
Hunan Data
Accommnddiioon Option
10550
IN BrOWWWlowwntiidddl
RowardModee
LDURewrrdNodel
Specifiity und Detai
17.50
SummuGA
Step Betrleve Top Motricn
Step Regiees Top Metrics
Step Repport
Metric Bank
Figure 2: AutoMetries comprises four steps. (1) Generate: create task-specific candidate metrics (Single criteria, Rubric, Examples, MIPROv2). (2) Retrieve: from the generated candidates plus MetricBank, use ColBERT to prefilter to k' metric cards and an LLM to select the final k. (3) Regress: fit a PLS model on the training set to weight and select metrics that predict human judg- ments. (4) Report: produce a writeup with weights and correlations and details to guide adoption.
LLM Based Evaluation LLM-as-a-Judge Zheng et al. 2023) evaluation is increasingly popular with the frequent improvement of LLM capabilities. Several works devise task-specific prompts to enable --M-daded--- evaluation for storytelling (Chiang & Lee, (023)) summarization (Wang et al.)
2023 Hada al., 2024 Wu et al., 2023), dialogue (Lin Chen 2023 Fu al. 2024). knowledge
(Bai et al. 2023). translation (Koemi & Federmann (203)) and more (Brake & Schaaf 2024). An-
other promising direction is devising frameworks and general methods for making LLM-a-aaJudgge
more reliable. G-Eval (Liu et al. 2023) proposes breaking LLM evaluation into step-by-step chain of thought and taking a weighted sum over the log probabilities of generating different scores. Chat- Eval (Chan et al. 2024) simulates multiple perspectives by evaluating through multi-agent debate. SPADE (Shankar et al. 2024a) generates assertions for LLMs to verify based on labeled good and bad examples. VERDICT (Kalra & Tang 2025) introduces judge-time scaling by decomposing judgments into composable units of reasoning, verification, debate, and aggregation steps. Though we take inspiration from many of these frameworks, the most directly similar our LLM-as-a- Judge steps in the AutoMetrics pipeline are DnA-Eval (Li et al. (0255) and EvalGen (Shankar et al.) 2024b). DnA-Eval (Li et al., 2025) decomposes evaluation into rubric criteria and aggregates the results across the criteria. EvalGen (Shankar et al. (2024b) elicits limited human feedback on gen- erated outputs, proposes criteria for evaluation based on this feedback, and iteratively refines the criteria with human-in-the-loop and LLM.
THE AUTOMETRICS METHOD
The purpose of AutoMetries to produce metries for subjective and novel Al-enabled tasks. Our goal is to induce metrics that correlate strongly with human judgments while requiring minimal data collection. To accomplish this, we present a general pipeline with four stages: (1) generate, (2)
retrieve, (3) regress, and (4) report. These steps are visualized in Figure 2 Each stage involves design choices among several alternatives, which we empirically validate (4.5).
3.1 METRIC PRODUCTION
Generate
For sufficiently novel settings, generating criteria for LLM- evaluation is es- sential. Broad coverage of evaluation criteria allows us to later filter down to what matters most. Ac- cordingly, our default configuration generates 10 Single Criterion LLM Judge metrics, 5 Rubric LLM-Judge metrics, Example-based based optimized LLM-Judge metric (fewshot), and 1 Prompt- Optimized LLM-Judge metric per run of AutoMetric Optimized metrics require more LLM call- s/tokens to produce, while criteria and rubrics relatively inexpensive. Unless otherwise specified, we use this configuration throughout the paper. Empirically, we find this mix of generated metrics
Design details and ablations in Appendix E.2
validate these choices across nearly 30 settings.
3

---

Published as a conference paper at ICLR 2026
generalizes across diverse domains and tasks. For each metric, we also generate Metric Card documenting its description, intended use, implementation details, and limitations (Appendix B
Retrieve addition to generated metrics, we leverage our MetricBank: a collection of 48 metrics (Appendix T) drawn from the NLP literature, each implemented and documented with Metric Card. Directly evaluating all metrics would be prohibitively expensive, so we instead use retrieval as filtering step. We treat Metric Cards documents, and use description of the evaluation setting or task as the search query. Retrieval is performed using hybrid ColBERT + LLM approach narrowing the candidate pool to metrics most relevant to the task at hand.
Regress The filtered pool of candidate metrics must still be combined into a predictive signal for human judgment. We normalize all metric scores to their z-scores and fit a Partial Least Squares (PLS) regression model. Intuitively, PI.S projects the metrie space onto the direction most predic- tive of human labels, then regresses labels along that axis. We choose PLS regression because it works well under the constraints of our setting that: (1) the number of predictors (metries) may be comparable to or larger than the number of observations (data points), and (2) the predictors are often highly correlated. Coneretely, with a single latent component, PLS finds unit weight vector w" E I² that maximizes
wt = arg max cov y)²
|||||11
where X is the matrix of normalized metric scores and y is the vector of human labels. The latent score is = Xx",, and PLS then regresses the human labels on this latent score, yielding predictions ú t. with coefficient B --
We apply this procedure in two stages. In the first stage, we fit PLS using all candidate metrics and rank them by the magnitude of their weights in w*. We then select the top metrics according to this ranking. In the second stage, we refit PLS on this reduced set of n metrics to obtain a new projection I and corresponding predictions ÿ. As final step, remove negatively correlated LLM--geerated metrics, as they are designed to target positive correlation. We don't apply this to existing measures (e.g., length can negatively correlate with conciseness).
3.2 METRIC EVALUATION
To evaluate the quality of induced metrics, we draw on concepts of measurement validity from research (Borsboom et al., 2004) and testing (American Educational Research Association et al.) 2014). We focus on three forms: "Content Validity", "Criterion Validity", and "Construct Validity".
Content Validity asks whether metric represents the construct it is intended to measure. Although direct quantification is difficult, we encourage transparency by releasing metric reports. Because generated metrics rely LLM judges, we also expose the reasoning traces of the judge LLM, allowing users to inspect whether assessments appear justified. These traces can further aid system optimization with AutoMetrics ($5).
Criterion Validity Criterion validity measures correlation with a reference standard. In NLP, corre- lation with human labels has been the most widely used criterion (Banerjee & Lavie 2005 Xu et al.) 2016 Gehrmann et al. 2021). We assess criterion validity by comparing AutoMetrics to ground- truth human labels. We report Kendall's т, which makes no distributional assumptions and simply checks whether the rank order induced by metric matches that of human judgments. This provides a conservative estimate compared to Spearman's 's p or Pearson's
Construct Validity measures whether a metric captures an underlying abstract concept, such as "quality." Both human judgments and AutoMetries attempt to approximate "quality". We draw from convergent disermminnt validity (Campbell& & Fiske 1959) and operationalize construct validity as robustness. A useful metric should penalize quality degradations (sensitivity) while remaining stable under equivalent-quality variation. In order to quantify convergentddiscriinnan validity, we intro- duce two measurements: Sensitivity and Stability. To construct test cases, we use an LLM to gen- erate strategies for degrading outputs on given dataset, and apply these to produce worse-quality perturbations. In contrast, same qumee-qualtyy perturbations are produced from a fixed set of hand-crafiee
'e ablate the selection algorithm in Appendix E.1
4

---

Published as a conference paper at ICLR 2026
transformations such designed to preserve the target evaluation dimension. Prompts are provided Appendix C
such as rephrasing, reordering, synonym replacement, stylistic edits that
Sensitivity measures whether a metric assigns lower scores to degraded outputs, Let forig (8) and Swrse denote the normalized scores for the original and worse-quality perturbed outputs sample i from a dataset of size |.V.. Sensitivity is defined as:
(x)
N
1 Sensitivity N
(P)
worse
Porig
Stability measures whether a metric produces consistent scores when quality should be (f) preserved. Let Ssame be the normalized score for same-quality perturbation of sample i from dataset size [N]. Stability is defined
Stability
Σ
(
orig
High sensitivity indicates strong penalization of degraded outputs, while high stability indicates invariance to irrelevant variation. Both desirable, and together they provide general-purpose lens for evaluating how well a metric generalizes.
EXPERIMENTS AND EVALUATIONS: SHOWING AUTOMETRICS ARE VALID
For experiments, focus showing that AutoMetrics valid across many tasks/domains and that they correlate better with human judgements than competitive baselines. We showcase AutoMetrics have high Criterion Validity and Construct Validity across several tasks.
4.1 TASKS
Dataset (Citation)
Task
Domain
# Data
Feedback
Eval Dim
Refs
In-Distribution Tasks: metrics bank were designed directly evaluate tasks.
SimpEval Maddela et al. 2023
Simplification
360
1-100 Likert
HelpSteer2 Wang et al. 2024
Dialogue
20.324
1-5 Likert
5
x
OuttoffDistributio Tasks: metric is specifically designed for these tests generalization and metric generation.
EvalGen (Shankar et
2024b
Product description
100
Binary
RealHumanEval Mozannar ot al. 2025
Code completion
</>
5,204
Behavioral
X
Co-Gym (Shuo et 2025
Travel planning
e
72
1-5 Likert
3
X
Table 1: Overview of tasks.
Icons:
</> Code Generation;
Data-to- text Generation;
Dia-
logue/Chat; Eduucation/Reddabllii e Travel Planning.
In order to evaluate our AutoMetrics method, we collect two types of tasks: In-Distribution Tasks, which are tasks where some of the metries in our Metric Bank were designed directly evaluate the task, and assess the task. All of our tasks utilize human feedback for evaluation, encompassing behavioral feedback, binary feedback (thumbs up/down), and Likert scale feedback, which is already collected as part of the dataset. datasets and a single evaluation dimension from each: SimpEval (Maddela et al., 2023) (sentence simplification score 1-100), HelpSteer2 (Wang et al., 2024) (Chatbot helpfulness 1-5), EvalGen (Shankar et al. 2024b) (Product Review Thumbs Up/Down). RealHumanE (Mozannar et al. 2025) (accepted or rejected code edit), CoGym (Shao et al. 2025) (travel plan outcome rating 1-5). We report evaluations
Tasks, which are tasks where no metric in particular was designed
introduce all tasks Table 1 In our main tables we present results for five
settings in the Appendix results.
4.2 BASELINES
We include the following baselines: Best Existing Metric, where we rum all 48 metrics (or 19 metrics for reference-free tasks), record their Kendall correlation the validation set, and select

---

Published as a conference paper at ICLR 2026
the best metric use for the task based on the validation correlation. MetaMetrics, where take all the metrics from the MetaMetrics paper and compute an XGBoost Regression on the metrics on the trainset (Winata al. 2025). Finetuned LLM refers to training ModernBERI-1am (Wanner et al. 2024) to predict the human annotation. We implement it by training LoRA adapters (Hu et al.) 2 with rank 16 all the attention, dense layers, and regression head, using learning rate of 5e - 5 and a batch size of 16 for three epochs over the training data. For the LLM- Judge baseline, we use the original human annotation prompt for each task and provide it to an LLM. We include all of these prompts in Appendix [ DnA-Eval (Li et al. (025)) involves using an LLM to generate three dimensions where a user request may benefit from evaluation, along with
weights for how to aggregate these dimensions. Then each of those dimensions is scored with an LLM--as--Judge, and finally aggregated based on the L.LM-generated weights.
4.3
CRITERION VALIDITY (CORRELATION)
We report Kendall's of all methods with GPT--4omini and Qwen-3-325 Reasoning in Table
In- In-Distribution SimpEval HelpSteer2
Dut-of-Distributii RealHumanEval
Method
EvalGen
CoGym
Model Agnostic
Best Existing Metric
0.246 = 0.00
0.327 +: 0,00
0.193 0.00
0.138
0.00
0.074 a: 0.00
MetaMetrics (Winata et al. 2025
0.127 001
0.204
0.00
-0.214
+ 0.1
0.025
0.01
-0.119 = 0.0
Finetuned LLM
0.076 = 0.8
0.039
0.03
0.054
+ 0.5
0.049 0.06
0.223 + 0.20
GPT-4o-mini Backbone
LLM- Judge
0.272 0.02
0.259
0.161
0.069
0.199
DnA Eval et 2025
0.234 = 0.255
0.174
0.152
+ 0.185
AutoMetrics (Ours)
0.321 # 0.01
0.324
0.01
0.334
0.06
0.160
0.00
-0.034 0.17
Qwen-3-32B Backhone
LLM-Judge
0.294 =: 0.334
0.272
0.025
0.276 +
DnA Eval et al. 2025
0.042 = 0.04
0,260 0.02
0,232
# 0.19
0.071
0.15
0.353 +: 025
AutoMetrics (Ours)
0,316 = 0.02 0.342 + 0.01
0.382 0.05
0.145 = 0.00
0,365 :: 0.08
Table 2: Criterion Validity results showing Kendall's Tau with 95% confidence intervals over 5 independent runs. AutoMetries outperforms the baselines on all five tasks with Qwen3-32B and is within 95% confidence of the best for 4/5 tasks with GPT-4o-mini. On EvalGen, AutoMetrics improves performance by 33.4% over the closest baseline (LLM Judge).
AutoMetriss correlates better than all baselines across all five tasks. We find that AutoMetrics outperforms all other existing baselines on all five tasks. While
the best performing baseline is both inconsistent on dataset (LLM Judge on
SimpEval, HelpSteer, EvalGen; DnA Eval on Re- alHumanEEvl and CoGym) and on the underlying model used (Existing Metrics outperform GPT-4o- mini but not Qwen3-32B). In contrast, AutoMetrics consistently the best option regardless of dataset underlying model. On all datasets besides HelpSteer and CoGym, the AutoMetrics performance exceeds all baselines by greater than the 95% confidence in- terval. In general, AutoMetrics is the best choice for higher correlation with human ratings.
SimpEval HelpSteer2 (score)
CoGym (outcome rating)
(helpfuines))
6
95%
0.8
score (mean +
0.6
0.4
Normal Baseline
0.2
Sensitivity
Stability
0.0
Sens Stab
Sens Stab
Sens Stab
4.4 CONSTRUCT VALIDITY (ROBUSTNESS)
Figure 3: Sensitivity/Stabiiity of AutoMet- rics for SimpEval, HelpSteer2, and CoGym. AutoMetrics are sensitive to negative pertur- bations and stable on neutral perturbations.
To measure construct validity, we take inspiration from convergent-discriinaat validity and show that AutoMetrics are strong predictors when output qual- ity degrades and that they stable under unimpor-
tant perturbations. To do so we introduced Sensitiv- ity and Stability ($3.2.. Sensitivity measures the rate of detection of negative perturbations and

---

Published as a conference paper at ICLR 2026
Stability measures the magnitude of score preservation under meaningless changes. We report Sen- sitivity and Stability for all metrics on 30 trials in Figure We compare against normal distribution
baseline.
Mutore are sensitive and stable. AutoMetrics are sensitive to degradation in output qual- ity in 81.0-97.8% of cases, depending on the dataset, which is significantly greater than the 50% baseline. AutoMetrics can be strong tool for identifying degradations in output quality. Similarly, AutoMetrics also always outperforms the baseline for stability by greater than 95% confidence in- tervals. Under insignificant modifications to evaluated outputs, AutoMetrics are consistently stable.
4.5 DESIGN DECISIONS (HYPERPARAMETT) SWEEPS)
Our sweeps/ablations test three parts of the AutoMetrics method: the MetricBank, the retrieval step, and the regression step. We report Kendall's 's rank correlation across our main tasks with 95% confidence intervals over five runs in Table 3 All sweeps and ablations are instead done on the dev set for all datasets. We never make design decisions based on runs of our test sets.
In-Distribution
Method
SimpEval
HelpSteer2
EvalGen
RealHumanFFval
CoGym
MetricBank Ablations (k=30; n=5)
Existing Metries Only
0.238 +: 0.376 0.00
0.389
0.155 +
0.258 0.00
Generated Metrics Only
0.276 + 0.03
0.308 +: 0.01
0.503 = 0.03
0.132 + 0.00
0.433
0.04
Full MetricBank
0.275 + 0.02
0.387 0.00
0.474 0.03
0.152 + 0.01
0.329 0.02
Retrieval Ablations (n=5)
Retrieve k=5
0.257
0.336 0.03
0.414
0.124 + 0.02
0.385
0.M
Retrieve k=10
0.245 + 0.02
0.352 + 0.01
0.469 = 0.06
0.128 + 0.01
0.37
+ 0.02
No Metric Cards (k=20)
0.281 + 0.04
0.328 0.02
0,427 0.09
0.134
= 0.01
0.292
0.06
Retrieve k=20
0.286 +: 0.378 0:01
0.522 =
0.141 0.01
0.302
0.06
Retrieve k=30
0.275 + 0.02
0.387 + 0.00
0.474
0.03
0.152 + 0.01
0.329 0.02
Regression Ablations (k=30)
No Regression (n=l)
0.232 + 0.08
0.393 0.00
0.353 0.23
0.145 + 0.00
0.356 0.00
Regress n=3
0.255 +:
0.389 0.02
0.503
0.152 + 0.01
0.302 0.04
Regress n=5
0.275 + 0.02
0.387 + 0.00
0.474 = 0.03
0.152 + 0.01
0.329
0.02
Regress n= 10
0.309 +: 0.01
0.358 0.01
0.461 0.05
0.147 + 0.01
0.297 0.05
Regress n=20
0.268 +: 0.03
0.350 0.01
0.498 0.04
0.153 + 0.01
0.361 0.02
Table 3: Kendall correlation with 95% confidence intervals on in-distribution and out-of-distribution datasets over five runs with Qwen3 32B (Reasoning). The Full MetricBank and Metric Cards prove useful, and the best settings for retrieval and regression are k=30 and n=5 respectively.
Both Generated and Existing Metries Help. In all of our tasks, the Full MetricBank was either the best or second-best performing setting for the ablations. When was second best, it was typically within 95% confidence intervals. The primary exception is CoGym, where "Full MetricBank" fell 0.104 below "Generated Metries Only" and, to a lesser extent, EvalGen, where "Full MetricBank"" was short by 0.029. CoGym and EvalGen samples respectively). We hypothesize this is because on out-of-distribution tasks, existing metrics tend to be noisy predictors which spuriously correlate during the regression. Generated metrics tend to be less noisy predictors. Larger training sets provide a more effective filter for identifying useful metrics. We further explore this hypothesis in our data scaling experiment ($4.6).
also our smallest training sets (37 and 57 training
Metric Cards Help Retrieval and Larger k Is Better: Across all five tasks, retrieval with Metric Cards (k=20) is better than retrieval without metric cards (using a single sentence description of the metric), Furthermore, we see roughly linear growth of correlation with higher & metrics retrieved to rum on the train set. The single exception to this trend is CoGym, which can be attributed to the noisiness of the small dataset and the generated metrics being less noisy predictors. The top retrieved metrics are often generated ones, reducing the risk of recommending spuriously correlated existing metric the small dataset. We ran all retrieval experiments by regressing with 5, so

---

Published as a conference paper at ICLR 2026
it is worth noting that future improvements to the retrieval algorithm (possibly including historical usage data) mean that feasible for lc = numbers to match our 30 results, so long as the proper metrics are recommended.
Number to regress to varies from dataset to dataset, but five is good average case. The best case for regression only repeats once (with n=20), suggesting that the number of metrics needed is highly dependent
the complexity of the evaluation task and domain. Since there is clear winner, we select n=5 as a default because it is the second best two of five tasks, and it is the cheapest option that still maintains lower variance from more expensive metrics to run downstream, so n=5 is a useful compromise of cost and performance.
run. A higher N means producing
4.6 How MUCH DATA DO YOU NEED TO USE AUTOMETRICS?
To test how much data is needed to use AutoMetrics, we test on three distinct datasets large enough be useful in this experiment. We take relatively simple In-Distribution dataset, SimpEval, more challenging In-Distribution dataset, HelpSteer2, and an manEval. We vary the train set size from N=5, 10, 20, 40, 80, 160, and (for RealHumanEVal and Helpsteer2) 320 and 640. We run these settings for both the "Generated Only" Metric Bank and "Full" Metric Bank (with existing metrics). We plot the correlation on full test set Figure 4
dataset RealHu-
SimpEval (score)
HelpSteer2 (helpfulness)
RealHumanEval (accepted)
0.4
0.3
Kendall correlation
0.1
-0.1
Generated Only
-02
Full
Train Siize
Train Size
Train Size
Figure 4: All correlations plotted for various training set sizes with "Generated Only" and "Full" Metric Banks. Individual trials are translucent while average performance at a scale is solid.
About 80 samples saturates performance. Across all three datasets and both settings, perfor- mance levels off after about 80 samples. It is possible with more sophisticated metric genera- tion/learnin methods more data could continue to help, however with the current architecture be- tween 80- 100 examples is all you need. Below 80 examples most of the lower performance is due to the high variance of fitting regression to a small training set.
On out-of-distribution datasets "Generated Only" can outperform "Full" with low-resources. Looking to the RealHumanEval plot we see at training size 10 and 20 the "Generated Only" metrics outperform the "Full" Bank. Recall back to the ablations ($4.5) where we observed on the small, -utrof-distribution datasets, CoGym and EvalGen, that "Generated Only" outperformed the "Full" MetricBank. Since most tasks will be out distribution by nature, we default to using 'Generated Only" when the user provides less than 80 training samples. Beyond 80, both "Generated Only" and "Full" level off, however "Full" asymptotes higher than "Generated Only" on all datasets. We argue this is product of the high-p, low-n problem in regression where having too many weak predictors and not enough datapoints can lead to spurious correlations. By limiting to generated metrics for low-n settings we enforce the use of stronger predictor signals.
CASE STUDY: AUTOMETRICS FOR OPTIMIZING AN AGENTIC TASK
natural extension to using AutoMetrics is to take the limited data one has available in order to leam useful set of metrics that then be used for optimizing system. In this way AutoMetrics
8

---

Published as a conference paper at ICLR 2026
Tau Bench, Qwen3 328
Membership Benefit Application (Rubric)
0.08
Avenge Pase 194t Scone
0.75
Correcly enforcing froe boggage allowances, insurance akgibiliy, and compenser rules based on memnbenshi Lie.
0.70
Escalation Appropriateness (Rubric)
0.0599
Transfering to human agents when policy Timiss are reached or exceptions
nceded.
Autrilerrio iss
Policy Compliance
0.0567
0.35
Velilatie Rearnt
Unaptimized Baseline
Adherence
basic economy concellations without
0.50
232
2000 Number of Rollouts
1250
2000
insuranco
Figure 5: AutoMetrics produces three metrics for T 7-Bench. Regression coefficients in yellow.
Figure 6: r-Bench performance over GEPA opti- mization steps when using AutoMetrics.
would operate similar to the purpose of a Reward Model or a Verifiable Reward. In order to test AutoMetrics can be useful in this setting optimize airline assistance agent for r-bench (Yao et al. 2024). testbed for tool-use agents to interact with simulated users to accomplish tasks. We split the 50 r-airline tasks into 25 for training and 25 for evaluation.
Simulating a verifiable reward. To run AutoMetrics we rollout the 25 training examples 8 times each with temperatures [0.0, 0.01, 0.02, 0.03, 0.05, 0.1, 0.15, 0.2]. Then we obtain the true reward signal for each of these rollouts. In practice rather than verifiable reward this could be subjec- tive human label. We run AutoMetrics in "Generate Only" mode and allocate more resources to generated metrics (10-+20 I1m judge metries; 5-+8 rubric metrics). Otherwise run with default hyperparameters (k=30; n=5). We show the generated metrics in Figure
AutoMetrics recommends three metrics for Tau-Bench evals: two rubric based metrics and one single criterion metric. Originally our (n=5) setting recommended five metrics, however our final filtering step removed metrics for having negative coefficients. Since the trajectories only derived from 25 examples it is likely that metrics will begin to learn things about the data itself. This reflects the importance of both human oversight and metric filtering.
Optimizing without Verifiable Reward We implement simple ReAct Yao al. 2023 agent in DSPy (Khattab et al., (024) for performing the r-Airline task. Our baseline agent gets 60% accuracy the 25 test examples averaged over five trials. We then run baseline optimization where we use the DSPy GEPA optimizer (Agrawal et al. 2025) to optimize an agent on the 25 training tasks with Verifiable Reward. Next we run optimization with our AutoMetriss as the metric for GEPA optimization. We show the performance on the test set after N rollouts in Figure We find that AutoMetrics can match performance of a verifiable reward. After 2000 rollouts the GEPA optimization with verifiable reward achieves 0.680 + 0.11 accuracy over 5 trials while the AutoMetries run gets 0.720 + 0.06. AutoMetries statistically significantly exceeds the baseline performance (p 0.05) of 0.6. This demonstrates that AutoMetrics can match or exceed Verifiable Rewards as optimization signal.
6 DISCUSSION AND CONCLUSION
In this paper, we introduced AutoMetrics, a method for producing metrics that correlate with human judgments on subjective tasks. Requiring only ~88 human- labeled examples, AutoMetrics achieve high criterion validity (.4) and construct validity. ($4.4). AutoMetrics improve upon existing baselines by up 33.4% in Kendall correlation with human ratings. In a case study on Tau-Benchh verifiable reward ($5).
AutoMetrics matched exceeded gains obtained from optimizing
We draw two key lessons for practitioners. First, data diversity is critical: back points suffice for moderate correlation ($4.6). scaling up synthetic data from limited sources can produce metrics that reflect dataset artifacts rather than system quality ($5). Second, human oversight remains essential: domain experts can help remove spuriously correlated metrics which the automatic filtering process misses. When using metrics for optimization, practitioners should monitor metric feedback and improvement with observability tools (Chavez 2025
while only ~88 feed-

---

Published as a conference paper at ICLR 2026
Overall, AutoMetrics provides practical first step for exploring data and guiding optimization when collecting preliminary human evaluation in new domains. The metrics it produces are inter- pretable, actionable, and informative for system improvement. We release AutoMetrics publicly and invite community contributions of new metrics and methods to strengthen the framework.
REPRODUCIBILITY STATEMENT
AutoMetrics is intended to be an open source library and framework. As such we take great effort to make the running and evaluation of AutoMetrics user-friendly. We have attached anonymized repository for AutoMetrics with this submission. In addition to the core algorithm, the repository also contains the python scripts reproduce all experimental results in this paper. All of our design decisions, hyperparameters, and ablations are rigorously documented throughout the paper across Section 4.5 and Appendix
E We provide system-sper needed to run the metrics Table [5] We also share the exact prompts and DSPy signatures used in calling LLMs in Appendix [c] For all main experimental results (e.g- Table 2 seeds to ensure findings are robust and statistically significant.
and Table 3) results are reported over five independent random
LIMITATIONS
As part of the AutoMetrics framework we construct and optimize metrics with particular LLMs. Because the metric generation process involves optimizing to a particular model we have found that producing metrics with one model and running them with another reduces performance. This suggests that when better models are released it will be important to reoptimize automatic metrics using AutoMetrics rather than just swap out the underlying LLM.
AutoMetrics may only generalize as far as the provided data enables Collecting real, diverse human data is still essential part of evaluation. The more representative and generalizable the input data is, the better and more general the AutoMetrics will be. Users should collect data that is representative of the opinions and population that they want their evaluation cover.
AutoMetrics depends on running a regression for many predictors on a limited number of data points. Although took this into account with the design of our Regression step, it is still possible to run into a high-P low-N regression problem that risks spurious correlations. To counteract accidental misuse of AutoMetrics leading poor evaluation, add warnings the metric reports when the significance of the correlation with human judgments of the recommended metric is low (p > 0.05).
Finally, part of this work we do not conduct formal user study to demonstrate the adoption of AutoMetrics among practitioners. We have collected positive feedback on the metrics through informal tests with Al developers. We hope that by releasing and open sourcing this library, we will have the opportunity to work with the community to test and improve AutoMetrics.
ACKNOWLEDGEMENT
The authors would like to thank Omar Khattab, William Held, Saurabh Shah, Aryaman Arora, Ken Liu, David Anugraha, Vishakh Padmakumar, Hao Zhu, Lakshya Agrawal, Yu Fei, Seungone Kim, and Jonathan Hilgart for their insightful comments and thoughts at various stages of the project. We would also like thank SALT Lab and the Stanford NLP Group for their help with the review and revision of the manuscript. Finally, we thank Yijia Shao and Alex Spangher for testing and providing feedback on the AutoMetrics system. This work has been supported part through the Stanford HAI Corporate Affiliate Program, with membership funding provided by American Express. This work was also supported by the Sloan Foundation, Schmidt Sciences, and a grant under ONR -00014-24-1-2532.
REFERENCES
Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arny Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa:
10

---

Published as a conference paper at ICLR 2026
Reflective prompt evolution can outperform reinforcement learning, 2025. URL https:// N.cxv..or//abs/557.2222
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. Standards for Educational and Psychological Testing- American Educational Research Association, Washington, DC, 7 edition, 2014. ISBN 978-0- 935302-35-6. Prepared by the Joint Committee on the Standards for Educational and Psycholog- ical Testing of AERA, APA, and NCME.
David Anugraha, Zilu Tang, Lester James V. Miranda, Hanyang Zhao, Mohammad Rifqi Farhan- syah, Garry Kuwanto, Derry Wijaya, and Genta Indra Winata. R3: Robust rubric-agnostic reward models, 2025. URL https:://aaat /ttp:///a/iv..///./ org/ abs 2505. 13388
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He. Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang. Juanzi Li, and Lei Hou. Benchmarking foundation models with language-model-at as- aas-an-exama an-examiner. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
nwt. netfforumttttmmm forum?
1d IiRHQ7qvnq
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin- Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Mea- sures for Machine Translation and/or Sunmarization, pp. 63-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL W05-09097
Steven Bird and Edward Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pp. 214-217, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL httpttppss///clanttooologg///// org 3031/
Param Biyani, Yasharth Bajpai, Arjun Radhakrishna, Gustavo Soares, and Sumit Gulwani. Rubicon: Rubric-based evaluation of domain-specific human conversations. Proceedings of the Ist ACM International Conference Al-Powered Software, Alware 2024, pp. 161-169, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706851. 3664646.3664778,. URL https::://i.oorrgg//// 10
doi: 10.1145/
81818466666667667 3664778
Denny Borsboom, Gideon J Mellenbergh, and Jaap Van Heerden. The concept of validity. Psycho- logical review, 111(4):106 2004.
Nathan Brake and Thomas Schaaf. Comparing two model designs for clinical note generation; is an
LLM a useful evaluator of consistency? In Kevin Duh, Hclena Gomez, and Steven Bethard (cds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 352-363, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/w1/2024. findings- naacl.25. 1ndingg-naacl.25. URL ac anthology 202/ windinis-naacc...
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL aittp:://arxin www.aaawwwwrraaaaaaaaaaaaa abs 2000. 14165
Jose Camacho-collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthe- nis Antypas, Joanne Boisson, Luis Espinosa Anke, Fangyu Liu, and Eugenio Martínez Cámara. TweetNLP: Cutting-edee natural language processing for social media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp. 38 09844,,44 Abu Dhabi, UAE, December 2022. Association for Computational Linguistics. URL httltttts://eeee http///claathhhcc org..oo........... 2022 emn demos.deeos...
Donald T Campbell and Donald W Fiske. Convergent and discriminant validation by the multitrait- multimethod matrix. Psychological bulletin, 56(2):81, 1959.
11

---

Published as a conference paper at ICLR 2026
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent de- bate. In The Twelfth International Conference Learning Representations, 2024. URL https: /ppp//eeeeeeeeeeeeeeeeeeeeeeeeee net httpr//eeettetttteteeeee wwwoorrmmd=F@epp..o weteiidd=FFpstttee sCUWu
forum?
Rogerio Chavez. Langwatch: The open Ilm ops platform - langwatcc//anggaacch https:// 2025. Accessed: 2025-09-24.
github com/ http://aaaaatch/agggaee...
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino. Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. URL https:h////// /ttp:s//aaaiiv..a org/ abs /2107 03374
Shikai Chen, Jin Yuan, Yang Zhang, Zhongchao Shi, Jianping Fan, Xin Geng, and Yong Rui. fil-rewarr-geemmas 2- v... https: htap:s///hgggnngfccee co/ Swikaihhhhh/ Re--Reeard-Gema---a -- vo. 1 2025. Hugging Face Model Repository.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human eval- uations? Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 15607-15631, Toronto, Canada, July 2023. Association for Computational Linguis- org/2023.
tics. doi: 10. 18.18853/v11 122653//1/0233.clooooo...... acl-long 0aag..7700 870/
URL
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Sto- ica. Chatbot arena: An open platform for evaluating 1lms by human preference, 2024. URL
https://arxiv: org/
/ttp//b//////oooo//////////////p abs/ 2403 04132
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic. Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31 International Confer- ence on Neural Information Processing Systems, NIPS 17, pp. 4302 4310, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Pierre Colombo, Chloe Clave, and Pablo Piantanida. Infolm: A new metric to evaluate summa- rization & data2text generation. In AAAI Conference on Artificial Intelligence, 2021. URL
https://api. www.seeeeaannncoo........ orgg/oorpus1::24400000
ConfidentAl. Deepeval: Open-source Ilm evaluation framework. https: com/ 2025. URL https: /ttp:::///ttubb.... ctmp//ccoocace con dideiddett-- deepeva Version 2.7.6, released April 22, 2025. Accessed April 24, 2025.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang, Dipanjan Das, and William Cohen. Handling divergent reference texts when evaluating table-to-text generation. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 2019. Association for Computational Linguisties. doi: 10.18653/v1/119-111 URL https: 7 7/aclanthology 1tt.hhoooooooooo/1111111555 Ctg/PP9-1483.
Florence, Italy, July
George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statisties. In Proceedings of the Second International Conference Human Language Technology Research, HLT '02, pp. 138-145, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
12

---

Published as a conference paper at ICLR 2026
Dare Doe and Vishal Devireddy. Scorecard ai: Llm evaluation and testing platform https:// wttor//oeccettcccccaeoooooeea ai/] 2024. URL https: htttps:/www..ccreccett /ttp://ccoeeer.......
Version 1.1.1, released December 10, 2024. Accessed April 24, 2025.
Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E Gonzalez. Vibecheck:
Discover and quantify qualitative differences in large language models, 2025. URL https:
// /larxiv.org/abll.. 2410. 12851
EvidentlyAL. Evidently: Open-source ml and Ilm evaluation and observability framework.
https: httpps://githbbe ceet /evi dent 2025. URI. httpps://wwt
evidentlyai com/ Version 0.7.2, released April 21, 2025. Accessed April 24, 2025.
R. Flesch. Marks of readable style; a study in adult education. Teachers College Contributions to Education, 897:ix 69-ix 69, 1943.
Clémentine Fourrier, Nathan Habib, Hynek Kydlīček, Thomas Wolf, and Lewis Tunstall. Lighte- val: A lightweight framework for Ilm evaluation, 2023. URL https://gi trugttthubb.... com/
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate you desire. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6556-6576, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 12416633/1/2024mmacc-//
URL
http:////oooaaaaoooooooo//////.. hag.a/ooooo....ooooo.aaaaaaaaaa. 365/
Galileo. Galileo evaluate: Promptquality python package. ht http:://pypL.oorg/pprooee
prompt.uality// 2025. URL
Version 1.10.0, released April 24, 2025. Accessed April 24, 2025.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondfej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jer- nite. Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major. Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrace Beltrabella,,,nn, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Ro- driguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. Antoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebas- tian Gehrmann, Yacine Jernite, Laura Perez-Beltrachini, Samira Shaikh, and Wei Xu (eds.), Proceedings of the First Workshop on Natural Language Generation, Evaluation, and Metries (GEM), pp. 96-120, Online, August 2021. Association for Computational Linguistics. doi: 10. 1 1/2021 ge..- 1. URL
org 202lg/22211.c0 gem 1. 10/
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Car- oline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreep- ranav Varma Enugandla, and Mark Wildon. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024. URL httph:ttp///aaattt
org/ abs / 411 04872
Anisha Gunjal, Anthony Wang. Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025. URL https::/arxxt
arg/abs/2507. 17746
Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation, 2025.
URL httthttts:///aaat org/
absaabas555/ 2503. 22828
13

---

Published as a conference paper at ICLR 2026
Rishav Hada, Varun Gumma, Mohamed Ahmed, Kalika Bali, and Sumayana Sitaram. METAL: To- wards multilingual meta-evaluution. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguisties: NAACL 2024, pp. 2280-2298, Mex- ico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1/22 findi -nggsnnnncl.
Andings-naacl.144 148.
URL
148/]
Richard W Hamming. Error detecting and error correcting codes. The Bell system technical journal, 29(2)147-160, 1950.
David Heineman, Yao Dou, Mounica Maddela, and Wei Xu. Dancing between success and failure: Edit-level simplification evaluation using SALSA. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. pp. 1466-4995, Singapore, December 2023. Association for Computational Linguis- Utthttps: / aacl ant.aanchoooyyoroooooo org./0023..
tics. doi: 10. 10.18653/ emn p- ma 1 1.. 211/
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Robert Iv, Alexandre Passos, Sameer Singh, and Ming- Wei Chang. FRUIT:
Faithfully reflecting updated information in text. In Marine Carpuat, Maric-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 3670-360000 3686, Seat- tle, United States, July 2022. Association for Computational Linguistics. doi: 10. 1/2022.. naacl-main.269, URL https:://// / httpas////aaaaacooll...ooooom org 2222 naacl mmain 269/
Paul Jaccard. Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547-579, 1901.
Matthew A. Jaro. Advances in record-linkage methodology as applied to matching the 1985 census of tampa, florida. Journal of the American Statistical Association, 84(406):414444 420, 1989. ISSN 01621459, 1537274X. URL http: //wp::///wwwt jstor http:///aattaabeeeeeee5e5
F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. Perplexity measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 626$1)) 08 2005. ISSN 0001-4966. doi: 10.1121/1.201622 2016299
URL http:: http///ooor//.../o//o 1121/1.
Nimit Kalra and Leonard Tang. Verdiet: A library for scaling judge-time compute, 2025. URL https://arxiv. http:///wwxiv..ooaass////////
18018
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self- improving pipelines. In The Twelfth International Conference on Learning Representations, 2024.
Scungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Sco. Prometheus 2: An open source language model specialized in evaluating other language models, 2024.
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Back, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho. Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chac, Jamin Shin, Joel Jang. Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Min- joon Seo. The BiGGien bench: A principled benchmark for fine-grained evaluation of lan- guage models with language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5877-5919, Albuquerque, New Mexico, April 2025. Association for Computational Lin- guistics. ISBN 979-8-89176-18966
doi: 10.186533v 1/20225 long.333. URL https: naacl ann.33.33
//aclantho ogy,
14

---

Published as a conference paper at ICLR 2026
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975.
Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini. Carla Parra Escartín, Mikel Forcada, Maja Popovic. Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 103, Tampere, Finland, June 2023, European As- sociation for Machine Translation. URL haattps//aclanttooooyy.. 2023. eamt -1.
197
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Bonnie Webher, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), ), pp. 9332-9346, Online, November 2020. Association for Computational Linguistics. doi: 10. 1t5553//11222.mmllg//. 2020. www.p-mann.mnn.mmmmmmeeemmmmmmam 750/
URL https::/aalaattiiet .ogg
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL
https::/arxiis http:://araav.orrg/////22200 2t00.////233.... 13787
Adrien Lardilleux and Yves Lepage. CHARCUT: Human-targeted character-based MT evaluation with loose differences. In Proceedings of the 14th International Conference on Spoken Language Translation, pp. 146-153, Tokyo, Japan, December 14-15 2017. International Workshop on Spo- ken Language Translation. URL htttts::/aaaaattttttot hwtp:ooooooo..o......o..o...
V. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. So- vier Physics Doklady, 10:707, 1966. English translation of Doklady Akademii Nauk SSSR, 163(4):845-844 1965.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A prrooottyy objec- tive function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110- 119- San Diego, Califor- nia, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/w//N16-1 1014. URL https: /ttps://aacaatiin wow. 10147
Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy Chen, and Min-Yen Kan. DnA-eval: Enhancing large language model evaluation through decomposition and aggregation. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the I International Conference on Computational Linguis- fics, pp. 2277-2290, Abu Dhabi, UAE, Jamuary 2025. Association for Computational Linguistics. URL httpss://aaaaannllggoeooooooo//e org 200 2025 mai. main.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. hhttps:/// / //q thub. wttp:/cccotttttooaeeeeee lab/a laccaaavaaaaaeeepa 5 2023.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out. pp. 74-81, Barcelona, Spain. July 2004. Association for Computational Linguis- tics. / wwtp:oyyyy.ooo//0001110011101010
Ten-Ting Lin and Yun-Nung Chen. LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In Yun-Nung Chen and Abhinav Rastogi
(eds.), Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ComvA/ 2023). pp.
47-58, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ o20g wipp:mcaacccooaaoaammmmo
v1/2023.nd v1/2023.nlp4com .nppconvviii 1.5. URL http::///caaathoooyy..o wwtpoot.........oooo 5/
15

---

Published as a conference paper at ICLR 2026
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human aligmment. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 251 I 1-2522, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18663w/120223mmlppp 153.. htR.http::/// anthhlogthoogo..o. org/2023
emn1p main. meaain.550 1537
Yi Liu, Matei Zaharia, and grading
Ritendra
Datta.
Enhancing Ilm-as-a-
judge with
blog/ /es/aninanniiggggg 11m as-a a judge judge-witie with wloge-withgraadngg http:://wwwdaatbbbccco
notes.
https:
July
2024.
URL
/wwp://wwwww.... https://ww..ttabbiiks..c.o cattabicckscom//ee
enhancing 11m as-a- a judge with grading- notes Databricks Blog.
Chi-kiu Lo. YiSi a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Ondřej Bojar, Rajen Chatterjee, Christian Fed- ermann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri. Aurélie Névéol. Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.). Proceedings of the Fourth Conference on Machine Translation (Volume
Shared Task Papers, Day 1), pp. 507-513, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/W1995533 hittps://aclantie log.alllnntoooooo.oooooooooooo.m ogy. hww.:/hhhooooo.ooo111----0--- wwwg//1---3355
URI.
Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. LENS: A learnable evaluation met- ric for text simplification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16383-16408, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1,..8663//
URL
https::/aclaathoo..og org. 222/202233.222 long. 905
Xiaoyu Tan Minghao Yang, Chao Qu Inf-orm-Ilama3.. 1-70b, 2024. URL [https://
owtp://e....o..i///oooooooaaaaa. Co So/infly/INE--o L1 am.1am33.... 100.111-)))))))
1-70B] hhttps://huggingg
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model report- ing. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, pp. 220-229, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361255. doi: 10.1145/3285560.32875555 URL httpp:://ci..... 9287560.32875
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. N 51877540::7775499555555 5333,,, February 2015. ISSN 1476-4687. doi: 10. 10038/naurr112266 URL https:://bi.orqg// 1038 htt.////aauree13333322
Hussein Mozannar, Valerie Chen, Mohammed Alsobay, Subhro Das, Sebastian Zhao, Dennis Wei, Manish Nagireddy, Prasanna Sattigeri, Ameet Talwalkar, and David Sontag. The realhumaneval: Evaluating large language models' abilities support programmers. Transactions Machine Learning Research, 2025. ISSN 2835-8856. URL bttps:://penrreview. net/ forum?
id=hGaWg5Buj7 Expert Certification.
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh, Teddy Koker, Luca Di Liello, Daniel Stanel, Changsheng Quan, Maxim Grechkin, and William Falcon. TorchMetrics Measuring Reproducibility in PyTorch, February 2022. URL htttp:://itthii com/
Krista Opsahl-Ong. Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Za- haria, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage lan- guage model programs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methody in Natural Language Processing, pp-
16

---

Published as a conference paper at ICLR 2026
Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10. htt.1//5////222mmmmmlplp.. main..... URL httlltttt////caaatoooooooo org/2/00 connip-main. 5257
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational I. Linguis- tics. doi: 0,.1111//0773331007355
URL URL http::ttt////tttt http:////ww.aaaaao.o....ooooo wwwphh/olooy.oo//222010000000
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, pirh--eerformncce deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2888 2011.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using diver- gence frontiers. In NeurlPS, 2021.
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondřcj Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Lin- guisties. doi: 10.18653/v I/W15-3049. URL BoL_httpss//acllanhoooooyy org/ WW5 3049/
Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and trans- parent dataset documentation for responsible ai. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, pp. 1776-1826, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781 47814503355 doi: 10.1145/353116..
3533231. URL
10.
3533231
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Michael J. Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, and Diyi Yang. SynthesizeMe! inducing persona-guided prompts for personalized reward models in LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 8045-800 8078, Vienna, Austria, July 2025. Association for Compu- tational Linguisties. ISBN 979-8-89176-251-0. doi: 10.18653/v 1/665/////225..acoooooo....// acn-gog.379 URL hittps///aclatte
Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri, Lmunit: Fine-grained evaluation with natural language unit tests, 2024. URL https::////rtt .rgg abs/ 2abs/244 2412.1221 1...1.00 13091
Patricia Schmidtova, Saad Mahamood, Simone Balloceu, Ondrej Dusek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondrej Platek, and Adarsa Sivaprasad. Automatic metrics in natural language generation: A survey of current evaluation practices. In Saad Mahamood, Nguyen Le Minh, and Daphne Ippolito (eds.). Proceedings of the 17th International Natural Language Gen- eration Conference, pp. 557-583, Tokyo. Japan, September 2024. Association for Computational Linguistics. URI. http::::pp.............. //ccl/ antth ogy. or..org///00. 2024. n g --ma i 44/}
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Answers unite! un- supervised metrics for reinforced summarization models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural
17

---

Published as a conference paper at ICLR 2026
Language Processing and the 9th International Joint Conference Natural Language Process- ing (EMNLP-IJCNLP), pp. 3246-3256 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1332 1320. URL bttpp::/aclantho///
org/D1 9- 901/00/9112
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguisties. doi: 10. 10.18663//1/2020.aco main. 704. URL https:://///mm http://allaninn http:///acaaantooooooeo
org www.0......ma.............m.
Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin. J. D. Zamfirescu- Pereira, Harrison Chase, Will Hu-Hiinhomm, Aditya G. Parameswaran, and Eugene Wu. spade: Synthesizing data quality assertions for large language model pipelines. Proc. VLDB Endow. 17(12):4173-4188 August 2024a. ISSN 2150-8097 doi: 10. 10.47778:685500366883. URL, https://dog www.oo................oo 14778 07778/36658000 3685835
Shreya Shankar, J.D. Zamfirescu Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of IIm outputs with human prefer- ences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Tech- nology, UIST '24, New York. NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400706288. doi: 10.114573654777336666600 3654777.36764
URL https:://cc..rtg///// 1145/
Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative gym: A frame- work for enabling and evaluating human-agent collaboration, 2025. URL https://arxiiv.
org/abs /21p.241.... 15701
Zhihong Shao, Peiyi Wang, Qihao Zhu Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. uttp:tts: www.xiii..rg///ao abs/2402. 03300
Lingfeng Shen, Lemao Liu, Haiyun Jiang, and Shuming Shi. On the evaluation metrics for para- phrase generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3178-3190, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. emr. iiaiin.22 208/
208. URthttps: /acl antho 1 ogy. org/2022.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 223-231, 1, Cambridge, Massachusetts, USA, August 8-12 2006. Association for Machine Translation in the Americas. /ttts::::///acc http://cclaaachho.ol....o wwtpoogg//000.. http://////00...aaaaaappeeeeee amta- .27
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining re- search. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 55725-15788 Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 1..11665/v/2224..u 1/2024.acl-loong..geeeea.. 1ong.840.. URL https:: hatpp:://acaaathologgo orgph//11ooyyy./2222..../ acl-lond 840/
Walsh,
Hong Sun and Ming Zhou Joint learning of dual SMT system for paraphrase generation. In Haizhou Li, Chin Yew Lin, Miles Osborne, Gary Geunbae Lee, and Jong C. Park (eds.), Pro- ceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2:
18

---

Published as a conference paper at ICLR 2026
Short Papers), pp. 38 42, Jeju Island, Korea, July 2012. Association for Computational Linguis- tics. URL https:ts///
Tuhina Tripathi, Manya Wadhwa, Greg Durrett, and Scott Niekum. Pairwise or pointwise? evalu- ating feedback protocols for bias in LLM-based evaluation. In Second Conference on Language Modeling, 2025. htttt:t///ppezzceee
net wttet//orunnndduySSSS
Ken Tsui and Huu Nguyen. Low latency epu based educational value elassifier with generic educa- tional value, 2024.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition doi: 10. 110.////..201..7229999
(CVPR), pp. 4566-4575, 2015.
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. Learning from the worst: Dy- namically generated datasets to improve online hate detection. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1667-1682, Online, August 2021. Asso- ciation for Computational Linguistics. doi: 10. 18653/v acc-oo.c.-.oog..... 132. 132/
URL https:
Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tong- shuang Wu. Checklists better than reward models for aligning language models, 2025. URL
htttp:///axivv wwwrg///////a abtp:///////2.2... 18624
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is ChatGPT good NL.G evaluator? preliminary study. In Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (eds.), Proceedings of the 4th New Frontiers Summarization Workshop, pp. 1-11, Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10. 18.53v63/vl 1/221 //aclanthology www.etthhooooyyooo/23333333 newsum 1/
URL https:
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. /tthttss::// /ttp:////peeeeeveeeemmmm net forum?
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, bet- ter, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13666 2024.
Genta Indra Winata, David Anugraha, Lucky Susanto, Garry Kuwanto, and Derry Tanti Wijaya. Metametrics: Calibrating metrics for generation tasks using human preferences. In The Thirteenth International Conference on Learning Representations, 2025. htLhtsps:: httts::/s/// //tp://///pp//eeee./
William E Winkler. String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage. 1990.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: www.. a aclweb. azg/annthlooyy
2020 emn 1 1p -e-demmo.
19

---

Published as a conference paper at ICLR 2026
Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. Large language models diverse role-players for summarization evaluation, 2023. URL /ttss///axxvv..o abs/
2303.15077
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah. Melvin John- son, Xiaobing Liu, tukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neu- ral machine translation system: Bridging the gap between human and machine translation, 2016. URL htttss://axxi// org/ 06111
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. Optimiz- ing statistical machine translation for text simplification. Transactions of the Association for doi: a0.1162/tac a 00107. URL https:
Computational Linguistics, 4:401-415, 2016.
haalaatthloogy.orr/01155
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states en- ables learning generalizable reward model for LLMs. In The eiiht Anmual Conference Neural Information Processing Systems, 2024. BtLphtts:://ppeee
net forrmm?
1d= jwh 9MH
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL htttp:://axxi....tot wwwpivvorrg/aaa /bbs/ 2406. 12045
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: evaluating generated text as text gen- eration. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393,
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2020.
URL
net ferttt/orumm2mm id- SdeSkHUUCVDDD
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025. URLht /tattss:/aaxxxv.or////aaaa 2501. 07301
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 563-578, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19. 1053. URL https:
w/a.lanthol.og..rrgg/// 1053/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //openzeview. nwt. fontstforumme wtweet//oormmmiuccccPPoll
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified multi-dimensional evaluator for text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023-2038, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022. emalp-main. 131. U //acl antho 020/222 emn pp-ma n. 131/ h
20

---

Published as a conference paper at ICLR 2026
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.
A LLM USAGE ACKNOWLEDGMENT
LLMs were used to rephrase and edit writing in the paper, after an entirely human-written first draft. LLMs were also used as coding assistants in writing code for this project. All code and writing edits produced by LLMs were rigorously verified by the first-author.
B INTRODUCING METRICBANK
MetricBank As our first significant contribution, we curate MetricBank, standardized collec- tion of 48 commonly used metrics in NLP literature. We source the metrics from Schmidtova et al. (2024), which examined all papers from the International Conference on Natural Language Gen- eration (INLG) 2023 and all papers in the Generation track presented at ACL 2023, totaling 110 papers. They collected a list of all the Natural Language Generation (NIG) metrics used in those works, which totaled 283 different automatic metrics grouped into 34 metric families. We sorted by the most popular and implemented the top metrics from the top 16 families (28 metrics). Then, for completeness, we also implemented any remaining NLG metrics in NLTK Bird & Loper (2004), PyTorch Paszke et al. (2019), Huggingface LightevalFouurier Fourrier et al. (2023), and Metametrics Winata jtt al. (2025) for an additional 12 metrics. Finally, we source few additional metrics from recent papers not covered in the 2024 survey. We provide individual justifications for these 8 metrics in Appendix B.I
We provide interesting stats about our metrics in Table 4 In particular, we collect 29 reference- based metrics, such as BLEU Papineni et al. (2002), which require a gold reference output, and 19 reference-free metrics, such FKGL Flesch (1943), which measure quality of text without comparison to a reference. Our metrics span 12 distinct domains. We implement each metric with simple interface of calcu cilcuatee method that takes in the generated text and produces a floating- point score and optionally text feedback.
Metric Cards Inspired by Model Cards Mitchell al. (2019) and Data Cards Pushkarna et al. (2022) we design Metric Cards for simple documentation and reporting of the intended usage of metries. Our Metric Cards contains seven main sections. Metric Details contains the description of the metric as well core details that are needed to use it, such as the range of outputs, if it's refer- ence based, an input is required, etc. Intended Use describes the domain/tasks where the metric should be used as well as recommendations for when and when not use the metric. Metric Im- plementation links to reference implementations and provides guidance on practical matters about the metric such it's efficiency and scalability. Known Limitations explains biases, misuse, and known failure cases of the metric. Related Metrics links to similar metrics to help when browsing for the right metric for your task. Further Reading points to papers, blogs, and tutorials covering the metric. Finally, Metric Card Authors makes it clear who wrote the metric card and if they used an AI assistance. We provide a complete example of metric card for the common BLEU metric papineni et al. (2002) in Appendix D We also provide prompt for using LLMs to write first pass of metric card in Appendix
B. ADDITIONAL METRICS
Here we provide justifications for additional metrics that we did collect from the metric sur- vey (Schmidtova et al. 2024). lighteval (Fourrier et al. 2023), torchmetrics (Nicki Skafte Detlefsen al. 2022), etc.
Reward Models. We choose some of the most performant reward models off the RewardBench leaderboard (Lambert et al. 2024) at development time. The three models we used are INFORM- ModardModel (llama 3.1 70b) (Minghao Yang 2024), LDLRewardMode! (Gemma 2 27B) Chen cet al., 2025), and GRMRewardModel (Llama 3.2 3B) (Yang et al. 2024). We also add in the
Qwen2.5 7B Process Reward Model (Zhang et al. 2025)
21

---

Published as a conference paper at ICLR 2026
Metric (Citation)
Domain
GPU
Type
Sup.
Default LLM
Reference-Bere Bassee Metrics: rely a gold reference for comparison.
Jaccard Distance Jaceard 1901
tt.
edit-distance
N.A.
Hamming Distance Hamming 1950
edit-distance
N.A.
Levenshtein Distance Levenshiem
1966
edit-distance
N.A.
Levenshtein RatiolLevvensin Levenshiinn
N.A.
Jaro Similarity (189
?
edit-distance
N.A.
Jaro Winkler Winkler 1990
edit distance
N.A.
BLEU Pupineni
2002
D-gram overlap
N.A.
NIST Doddington 2002
I-gram overlap
N.A.
ROUGE 2004
t.
n-gram overlup
N.A.
METEOR Banerjec Luvie
2005
n-gram overlup
N.A.
TER Snover
1206
edit distance
N.A.
iBLEU
& Zhou 2012
R-gram overlap
N.A.
CHRF +
Popovic
2015
n-gram overlup
N.A.
CIDE Vedaniam ct a. 2015
g-gram overlap n gram overlap n-gram overlap edit-distance
X
N.A.
GLEE Wi et al
2016
X
N.A.
SAR]
Xu 0
2010
X
N.A.
Char iI Lardileexx & Lepage
2017
X
N.A.
MoverScore Zhap et 2019
embedding sim n gram overlap
X
BERT
PsendoPARENTT Dhingra
2019
N.A.
BERTScore Zhang 2020
emhedding sim
X
RoBERTa-
BLEURT Scllam et al.
2020
LM regression
BERT/RemBERT
BARTScorg Yuan
202
LM regression
BART
InfoLM Colombo
divergence-bused
BERT
MAUVE Pillata al.
divergence hased
GPT-2
ParaScore SudSeorr/Saar Shen
embedding sim
RoBERTa-lagge
UniEvalDialogue Zhong
2022
LM regression
TS
UniEvalSum Zhoing 2022
LM regression
TS
MollaeeROGGESs
2012
gram overlap
N.A.
I.ENSS Maddela al.
2023
I. regression
T5
Reference-Pree Metrics:
require gold reference.
FKGI Kincaid et al. 1975
nule male-based
N.A.
Perplexty Jelinkk et al. 2005
fluency
GPT-2 Large
DistinetN Granss al 2016
diversity ratio
N.A.
SeliBLEU
2018
diversity ratio emhedding
N.A.
YiSi-2
Summat
I. regression
FactCC Kiyscinski
LM regression
BERT
Toxicity Vidgen et
classification
RoBERTs
ParaScorchroe Shen
1122
t..
B
embedding sim classification
RoBERTa-large
Sentiment Canacho--oll al.
2022
RoBERTa
UniEvalFach Zhong
B
LM regression
TS
LENS SALSAAHHHieea al.
2023
LM regression
TS
FaafTexdEducationaaalllll Value
Nguyen
2024
classification
FasIText
FasTTexNSFF Soldaini al.
2014
classification
FastText
FasTTexTTxxciiy Soldaini et
2024
classification
FastText
GRMRewardModel Yang et al.
2024
LM regression
Lluma-3,2-33
INFORM Reward Model 708 Minghao Yang 2024
LM regression
Llama 3.1-708
LDI. Reward Model 27B Chen et 2025
27B
LM regression
Gemma 2-27B
MathProossRowaalModdll Zhang al 2025
classification
Qwen2.5 7B
Table 4: Comparison of generative evaluation metrics. Icons: Machine Translation. Summa-
rization, + Paraphrasing, Dialogue/Chat, Storytelling/Creativv Writing, Image Caption- ing/Multimodal, 4 Safety/Moderation, t Data-to-Text Generation. EduccationReaddabiitt,, </> Code Generation, Math/Problem Solving, Z Btring-DistanneeBddi-BBBa
SALSA. In researching text simplification metrics we found that extension to the LENS (Mad- dela et al. 2023) metric exists which is meant to better align with human judgement. It was also a related metric the SimpEval (Maddela et al.) (023) paper. Thus we chose to implement SALSA (Heineman et al. 2023) in our MetricBank as was intended as one of the recommended "Best" metrics for in-distribution SimpEval task.
22

---

Published as a conference paper at ICLR 2026
Metric
GPU
CPU
Time (ms)
INFORMRewardModd!!
129.62 GB
2.04
1041
Lode
104.17 GB
2.06
1921
GRMRewardModel
GB
1B96BB
61
UniEvalDialogue
3.07
3.10GB
262
UniEvalSum
3.07
3.10GB
211
UniEvalFact
3.07
GB
61
Perplexity Verplexity-gpt2-aarge -pt2--lage
3B00
LB
48
BLEURT
2B15GB
2.75GB
43
BARTScore bart-large-enn
LB
LB
49
SummaQA
1.25 GB
LB
879
YiSi
687 MB
LB
35
Sentiment
485 MB
GB
19
Toxicity
485 MB
GB
39
FactCC
427 MB
1.29
17
ParaScoreFree
346 MB
1.68 GB
12428
ParaScore
338 MB
1.05 GB
4.54
262 MB
1,50GB
2899
DERTScoreerobertta-
8 MB
1.47 GB
1303
PRMRewardModdl
OMB
13.64 GB
6359
MAUVE
OMB
GB
336
FastTexEEnncaiinnal
OMB
3.73GB
LENS
OMB
GB
3 4008
LENS_SALSA
MM
GB
426
FasiTextToxicity
OMB
1.67
11
FaW
OMB
1.67 GB
6
InfoLM
OMB
1.12GB
2338
METEOR
MM
1.08 GB
27
FKGL
OMB
894 MB
6
TER
OMB
731 MB
26 064
CHRF
OMB
730
36
DistinetNGram
O MB
730 MB
19
iBLEU
MMB
730 MB
18
BLEU
OMB
729 MB
7
DevenshheinDistannce..
0 MMB
729
SelfBLEU
0 OMB
729 MB
6
Hamming -min
0 MBB
729 MB
0
JaroWinklerSimilariyy max max
OMB
729 MB
0
GLEU
OMB
728 MB
SARI
OMB
728 MB
95
JaccardDistance min
OMB
728 MB
0
CharCut
OMB
728 MB
237
UpdateROUGE
OMB
728 MB
96
NIST
O OMB
728 MB
LevenshteinRatio _mx
O MB
727 MB
0
JaroSimilarity _max maax
0 MBB
727 MB
0
ROUGE
OMB
726 MB
487
CIDEr.n4.sig..
OMB
726
31
PseudoPARENT
0MB
726 MB
10
Table 5: Maximum CI upper-bound GPU/CPU memory and latency per metric.
Fast Classifiers. We wanted to add diversity to our MetricBank by including more classi- fiers for various ligher-level concepts, but didn't want to add unnecessary expenses to running
23

---

Published as a conference paper at ICLR 2026
the metrics. FastText Classifiers nice compromise which are quick to CPU but also have reasonable classification accuracy. We implement FasfTextNSFW and FoxiiextToxicity from Dolma (Soldaini et al. 2024). and take FasTTexdEducationalaa which has been used for data filtering to attempt to find Text-Book quality training data.
(Tsui & Nguyen 2024)
C PROMPTS AND SIGNATURES
C.I LLM-AS-A-JUDGEE PROMPTS
We the LLM-as-a-Judge Prompts from the original human annotation process for given dataset whenever available. We consider these as a strong baseline as these instructions were designed to be useful instructions for human annotators and ideally were the underlying instructions guiding their annotation decisions.
Task: SimpEval
LLM-as-a-Judge Prompt:
Rating Sentences
The geal is to **ate sentences** by how well they *ssimplify the original sentence*..
#*# Scoring Guidelines
Score When to assign it |-------|---
1+000+* The sentence is +*fully simplifieds entirely fluent, and expreserves the core meaningu of the original.
**75** The sentence is **somewhat simpler*e, mostly fluent, and the meaning is **close to the original.
**50** The sentence is simpler, **sonewhat fluent**, and the the original.
meaning **similare*
**25** The sentence is equivalently simple, still has some fluency, but *+loses the meaning**.
**0+* The sentence is *ccompletely unroadablo**.
**Most scores will lie somewhere this range feel free to give specific scores (e.g., 83, 67) rather than only the five anchors.*x
### Examples
Score
Example Simplified Sentence why this score?
----_---|-
.+100A* *It will then kamove away from the river bed+ and sink back to the bottom to digest its food.* Reads fluently *vand** keeps the original meaning ("it" gets unstuck, moves down, digests food).
**75+*+ +Due this, **a lot of mosques don't enforee these rules** but both men and women should follow them. Minor fluency issue, but meaning matches the original.
**0** *A gadget javascript is and checking wikipedia snippet that can be enabled simply by or cas option your wikipedia preferences.* Sentence *wnnreddable*..
24

---

Published as a conference paper at ICLR 2026
Task: HelpSteer2
LLM--s---Judge Prompt:
* #iBllptuuheess/slerssaaddngggii
The response is extremely helpful and completely aligned with the spirit of what prompt was asking for.
- 3 The response is mostly helpful and mainly aligned with what the user was looking for, but there is still
for improvement.
2 The response partially helpful but misses the overall goal of the user's query/input in some way. The response did not fully satisfy what the user was looking for.
borderline unhelpful and mostly does not capture what the user was looking for, but it is still usable and helpful in small way. 0 The response is not useful or helpful at all. The response completely missed the what the user wanted.
- The response
of
Task: EvalGenProduct
LLM-as-a-Judge Prompt:
Is this response good (1) bad (0)?
Task: RealHumanEval
LLM-as-a-Judge Prompt:
Would you accept this code edit/addition (1) or reject it (0)
Task: CoGym/TravelOutcomm
LLM-as-a-Judge Prompt:
Overall rating the final outcome (i.e., travel plan, analysis result) (1-5 scale)
(1) "Extremely dissatisfied", "Semewhat dissatisfied",
(3) "Neutral",
(4) "Somewhat satisfied",
(5) "Extremely satisfied"
25

---

Published as a conference paper at ICLR 2026
C.2 MIsc PROMPTS
MetricCard Generation
Prompt:
You are an expert in natural language processing and technical documentation, specializing in metrics for evaluating generative models. am building a metric bank to recommend the best metrics for varicus generative tasks. Rach metric in this bank will have correspending Metric Card, which provides standardized, detailed documentation about the metric. These Metric Cards will serve as key rescurce for researcners and practitioners, helping them select the right metric for their task.
44 Your Task
Using the provided materials, including the original paper, reference implementations, the Metric Card Template, and the BLEU Metric Card Example, your task to draft comprehensive Metric Card for the given metric. The documentation must:
Follow the provided template closoly, ensuring adherence to its format and required sections. Incorporate relevant details from the original paper and reference materials, ensuring technical accuracy and completeness.
1.
2.
Match the style and quality of the BLEU example, which as exemplar for clarity, structure, and precision.
3.
Specific Instructions
1. Key Sectiens to Address: Ensure each required sectien of the template is filled out thoughtfully and thoroughly, including:
Metric Description Inputs and Outputs
Formal Definition Applicability and Limitations Xnown Limitations 2. If Information is Unclear or Missing: Do not fabricate or make assumptions. If information is unavailable, unclear, not included in the provided ded context, 1eave that section blank or mark it as "Needs more information." 3.
Related Metrics
Markdown Formatting: Output the completed Metric Card as a markdown text block rather than rendering or printing the markdown directly. This means you must surround your answer in ",, Alse start the block with "---" shown in the examples. Do not end the block with .___" 4.
Focus on Consistency: Use the provided categorical suggesticns (see below) to ensure uniformity across all Metric Cards, particularly in fields like "Metric Type," "Domain," and "Tasks."
5.
Mathematical Formatting:
Use for inline math expressions (e.g.,
$r$, not $)
Use $$ for block math expressions and full line break before and after each block math expression. This formatting ensures proper rendering in markdown.
Example of proper usage for $8:
Correct
Where:
SCHRP$ is the average precision of character and word n-grams:
26

---

Published as a conference paper at ICLR 2026
$$
CHRP (1) } (rrac(ttex((n-gi in hypothesis
and reference) Ntext(tota) h-grams in hypothesis))
SCHRR$ is the average recall of character and word n-grams:
CHRR \frac(1) (N) (n=1_( /frac/ (toe/txxx//gggan in hypothesis and reference)) [ttext (total n-grams in reference) $$ ...
** Incorrect **
Where:
SCHRP$ the average precision character and word n-grams: $$
CHRP - \1) (N) \sum_(n-1)"N (text (n-grams in hypothesis and reference) (te.tteetttttt (total n-grams in hypothesis)]
$$
SCHRR$ is the average recall character and word n-grams: $$
CHRR = (N) \sum_(n=1)"' (teat (n-grams in hypothesis and reference) Mtext (total n-grams in reference)] $$ ...
Ensure all block math expressions are clearly separated from 1ist items inline text.
Add space after operators like \sum, \max, or any LaTeX commands followed by an underscore () to prevent Markdewn parsers from interpreting as italic markers. Mainly it is critical to put space before " "" For example:
Correct
$$
[[teex (BERT)) \frac (sum. _(x _(1) Min x) (idf) (x _111)
\edot \max -1 [haaat (x) -(3: Vin (xat(x)) <11"/top \bat
[())] (Vsum _(x _{1) Vin x} \text(idf) (x -111)
$$
Incerrect **
$$
R_(Xtext (BERT)) (Trac(NsunLee Ain (tex (x_1) Nedot \max (hat [x]_] Vin (hat X_op>top (Nat: Vin x) \text (idf) (x_i) S$
Citation: It imperative that you do NST make this up. If the user does not explicitly provide the bibtex citation for the metric then you must say [More Information Needed]. If citation is provided you must copy it EXACTLY. Do NOT try simplify any of components such the author list with an ellipsis.
# Categorical Suggestions for Consistency
27

---

Published as a conference paper at ICLR 2026
Note:
These suggestions are not exhaustive. While you should prioritize using the categories 1isted here for consistency, you may add new categories if the metric clearly warranta them.
Demains
These represent broad areas of application for the metric. Choose more:
Text Generation
Speech Generation
Code Generation
Multimodal Generation
Image Captioning
Dialogue Systems storytelling
#*# Tasks
These are specific tasks or use cases where the metric applies.
Choose one or more:
Machine Translation Summarization
Paraphrasing
Data-to-Text Generation
Image-to-Text Generation
Dialogue Generation
Style Transfer
Creative Writing (e.g., poetry, storytelling)
Code Completion Response Generation
Metric Type
These classify the metric based on its design and purpose. Choose one:
Surface-Level Similarity (e.g., BLEU, ROUGE)
Semantic Similarity (e.g., BERTScore)
Fluency (e.g., perplexity-based metrics)
Diversity (e.g., distinct-n)
Robustness (e.g., adversarial robustness metrics) Fairness
Faithfulness (e.g., factual consistency metrics)
Reference-Free (e.g., coherence or novelty scoring)
kxplainability
Inputs
These describe what the metric requires for evaluation:
Reference-Based Reference-Free Input-Required Input-Optional
# Materials You will Be Provided 1. Original Paper: The foundational paper introducing or defining the metric.
2. Reference implementations (when available):
Documentation from popular implementations (e.g., SacreBLKU README for BLEU).
Metric Card Template: The standardized structure for all Metric Cards (see below).
3.
28

---

Published as a conference paper at ICLR 2026
4, re reference.
BLEU Metric card Example: A high-quality example for
TEMPLATH METRIC CARDS
* Metric Card for {I metric pane default ("Metric Name", true) FF
{( metric_summary default ("A brief description of the metric and purpose, ", true) }
# Metric Details
*** Metric Description
(I metrie__eeeeeee desrriptosniiiionn | default ("Detailed explanation of the metric, including how it is calculated and what it measures.", true) 11
**Metric Type:** {f metric_type default [More Information Needed]", , true) })
**Range: {f metric_range default (" [More Information Needed]
true) 11
**Higher is Better?:** [{ higher_is be_terrrrrrrrrre default [More
Information Needed) ", true) }}
**&dforence-Bass??:> {f reference_based default (" [More
Information Needed] true) H
* {( tnput_required default (" [More
Information Needed]", true) }}
## Formal Definition
metric_definition metric definition default ("Mathematical formula detailed algorithmic definition.", true) }]
HI Inputs and Outputs
*+Inputs:**
{{ mntric_inputs I default ("Description zequired inputs (e.g.,
generated text, reference text, input pronpt). true) }
**Uupptt:***
{( metric -urputssuu default ("Description of the metric output (e.g., scalar score, distribution). ", true) }}
# Intended Use
#*# Demains and Tasks
***emain:*** |I domain default (" [More Information Needed] ", true)
**Taska:* { tasks default (" [More Information Needed] ", true) H
### Applicability and Limitations
**Best Suited For:.* {{ best_ default [More Information Needed] ", true) H}
+*Not Recommended For:** [l not not__recommeded_.. for default (" [More
information Needed]", " true) }}
Metric Implementation
## Reference Implementations
29

---

Published as a conference paper at ICLR 2026
++Librardes/Pacaapeee (f libraries default (" [More Information Needed]", , true) }}
Computational Complex
#*fficiency::** II efficiency default (" More Information
Needed]", ,, true] }} +*scalability;= {f scalability default (" [More Information Needed]", , true) }}
# Known Limitations
lnww__limitation default (" [More Information Needed] ", true) }}
**Biases:** II biases default ("Fotential blases inherent in the metric.", true] }
*+Task Kisalignment Risks:** task misaligament defauit (" [More information Needed]", true) }}
**Failure Cases:** {f failure_cases default (More Information Needed]", true) }}
Related Metrics
(I relateddmetrics | default (" [More Information Noodod) truc) F}
# Further Reading
**Papers:** {f papers
default (" [More Inoomatiion Needed] ", true)
}}
[[
blogs
default (" More Information
Needed]", ", true]
H Citation
(( bibtexnnitatton | default [More Information Needed] ", true) F
Metric Card Authors
**Authors:** If metrec_ default (" [More Information Needed]", " true] }}
**Acknowleddment AI Assistance:**
I aefault of this metric card were drafted with assistance from generative Al. A11 content has been reviewed and curated by the author to ensure accuracy.". ", true) H **Contact:** retric_contace default (" [More Information Needed]", true) }
{( ai_assi
BLEU Metric Card Example
Metric Card for BLEU
BLEU (Bilingual Evaluation Understudy) is widely used metric for evaluating the quality of text generated in tasks like machine translation and summarization. between a generated text and one or more reference texts, with a brevity penalty to penalize overly short. modern implementation, ensures reproducibility and standardization BLEU scores across research.
measures the overlap of n-grams
SacreHL.HU,
# Metric Details
30

---

Published as a conference paper at ICLR 2026
## Metric Description
BlRU evaluates the quality of text generation on comparing n-grams the generated output with those one computes modified precisien for p-grams and combines scores using a gecmetric mean, with a brevity peralty to ensure the length of the generated text matches that of the references. Higher BLEU scores indicate closer similarity the references.
reference texts.
**Metric +y Surface-Level Similarity
**Range:A* 0 to 1
**Higher is Better?:** Yes
-=Reference-BBsseee Yes
-=Ipptt-Reuuirdd?:
#*# Formal Definition
(BLEX \text(BP] Aodot Nexp Vleft ( \aum_(n-1)" wan Vlog PLn \right)
$9
where:
$\tcxt (BP) Amin c1 x is the brevity penalty,
$rş is the effective reference length (based on the closest
matching reference length for each sentence),
$c$ is the candidate translation length,
$p_n$ the modifies precision for n-grams length $ns, $w_n$ are weights for each n-gram (comnonly uni form, $w_n (1)
Inputs and Outputs
Inputss:**
Generated text (candidate translation) Reference text (s) (gold-standard translations)
**Outputs:**
Scalar BLEU score (range: 0 to 1)
Intended Use
## lemains and Tasss
**Demain:** Text Generation
**Tasks*** Machine Translation, Summarization, Data-to-Text Generation
#1 Applicability and Limitations
++Best Suited For:..
Structured tasks with clear correspondence between generated and reference texts, such translation or sunnarization.
+*Not Recommended For:**
Open-ended or creative generation tasks where diversity or semantic similarity matters more than lexi cal overlap (e.g., storytelling, dialogue).
Metric Implementation
## Reference Implementations
31

---

Published as a conference paper at ICLR 2026
*+libraaiee///ccaaes::
[SacreHLKU] (https://g) http://tt.b.cmm/opoo/////eeeela/ (robusst
standard Implementation)
[NLTK] http:p//www...l.cco////////aaaaa
(basic
Python implementation)
https////aagaaaaaaeeeee////////r (integrated metric framework)
[Hugging Face 'evaluate")
#44 Computational Complexity
:*8fficiency:**
BL2U is computationally efficient, requiring $0 Aedot m) operations for $nš-gram matching where Sns the number words the candidate text and 9mş is the number of reference words. SacreBLEU optimizes tokenization and scoring, making it highly suitable for Large-scale evaluations.
++scalability:**
BLeU scales well across datasets of varying sizes due to its simple design. SacreBLEU further supports evaluation with multiple references, diverse tokenization schemes, and lanquage-speciff preprocessing, making it adaptable to diverse evaluation setups.
H Knewn Limitations
**Biases:**
BLBU penalizes valid paraphrases or semantically equivalent outputs that do not match reference n-grans exactly.
The brevity penalty can overly penalize valid shorter outputs,
particularly for tasks where shorter text may acceptable preferred (e.g., summarizatien).
+*Tass Misalignment Risks:*A
BLEU is not designed for evaluating tasks with high diversity in acceptable outputs (e.g., open-ended dialogue). Sccres depend on the quality and number of references; fewer or inconsistent references can lead to misleading evaluations.
**Failure Cases:**
BLEU struggles to capture semantic adequacy beyond lexical simi similarity. For instance, it cannot ident fy whether a translation preserves the meaning of the original sentence if word choices di verer significantly.
Related Metrics
**ROUGE: +* Often used for summarization tasks, emphasizing recall over precision.
*METEOR;* Incorporates synonym matching for better somantic alignment.
**BERTScore:** Uses contextual embeddings for semantic similarity.
# Further Reading
**Papers:**
[Original BLEU Paper (Papineni et al., 2002) ] www.cccleeboor///ttchooololoo
[SacreHLEV; Call for clarity Reporting BLEU Scores (Post, 2018) (bttps//ww..e/eee..or///att
=llogs/Tutoriall::
32

---

Published as a conference paper at ICLR 2026
[Understanding BLEU]
[šacreHLKU locumentation] httppp/s/giittbb.coo//oo//////as
Citatien
8 [prrccedagggpppiineeeeeeetat
title [[B) Method for Automatic Evaluation of Machine Translation",
auther "Papineni, Kishore and Reukos, Salim and Ward, Todd and Zhu, Wei-Jing", editer "Isabelle, Pierre and Charniak, Eugene and Lin, Dekang",
bocktitle "Proceedings of the 40th Annual Meeting of the Association Computational Linguistics", menth jul, year "2002",
address "Philadelphia, Pennsylvania, USA", publisher "Association for Computational Linguistics", url hattess//acaaatiooooooo/////22
dei = "10. "DD.315/1113333.107333 or9/00220000//0
pages = "311--318"
# Metric Card Authors
**Athhors:** Michael J. Ryan
+=Acknowledgeent AI Assistance:**
Portiens of this metric card were drafted with assistance from OpenAl's s ChatGPT, based on user-provided inputs and relevant documentation. All content has been reviewed and curated by the author to ensure accuracy.
C*nonact::* michael-jyyanestafford.eee
The metric will be designing a card for is (Metric Namo)
(SUPPLEMENTAL MATERIALS)
New please write high quality metric card for (Metric Name) given the provided materials!
Final **Ipoortant* Note: If the provided naterials do not give enough information about particular point for the metric (e.g. limitations or biases aren't listed) then You can leave blanks or "Needs more information" where needed. It is absclutely essential not to make things or guess wher producing this documentation otherwise future researchers and engineers will be confused and led astray. Avoid making up links that you aren't t fully confident in the url.
make things up
kemember to surround your answer in ",, Thanks!
33

---

Published as a conference paper at ICLR 2026
C.3 DSPY SIGNATURES
Signature:
GenerateeerturaaaiiissSttttgggae
Instruction:
You will given:
Task description
A Dimension to prioritize when perturbing outputs
The Example Input, optional Example Reference, and Example Output
Instructions
Your primary focus should be on degrading performance along the specified Dimension.
1. Beçin with rich reasoning paragraph (3-5 sentences) that explores variety ways subtly degrade model outputs. not reference
specific example.
2. Under the heading **Stratogics:*4, list 1-3 numbored, high-level perturbation strategics.
Each strategy should be short phrase (5-15 words) naming the category of change, followed by one concise sentence abstract explanation,
Do not include concrete rewrites, instance-specite examples, example sentences.
Inputs:
Field
Type
Description
task
str
The task the model was originally trying to complete
example_sets
[ist[str]]
Example inputs, outputs, and (optionally) references showing task performance
dimension
str
The dimension to prioritize for the perturbation
Outputs: Type
Field
Description
perturbation_
List [str]
1-3 high-level strategies test robustness
strategies
Signature:
PerturbWorse
Instruction:
You will be given:
A Task description
Dimension to prioritize when perturbing outputs
The Example Input, optional Example Reference, and Model Output pecturbacion_strenggt value ("subtle" or "obvicus")
1ist of perturbationn_srattegiess to apply
Instructions:
Your goal is apply each strategy to the Model Output and produce degraded version that specifically harms performance along the given Dimension, using the specified strength.
Under the heading *Perturbed Outputs:**, return exactly perturbed output per strategy.
For **subtlex* strength, introduce minimal distortion.
34

---

Published as a conference paper at ICLR 2026
For *xobvious** strength, introduce pronounced degradatien.
Do **no*** include any reasoning, explanations, or examples only the perturbed text.
Inputs:
Field
Type
Description
task
str
The task that the model was originally trying complete
dimension
str
The dimension to prioritize for the perturbation (this should be the aspect of the model output that is most impacted by the perturbation) The input provided to the model The references of good outputs (may be None)
input references
str Union (list
None]
model_output
str
The output produced by the model
perturbation
['terall'subtll
The strength of the perturbation (subtle or obvious)
strength
'bbvoous] list [str]
perturbation
The perturbation strategies to
strategies
Outputs: Type
Field
Description
perturbed_
list [str]
Perturbed text that is worse than the original model output. Produce one perturbed output per strategy.
outputs
Signature:
PerturbSame
Instruction:
You will be given:
Task description
Dimension to preserve when perturbing outputs
The Example Input, optional Example Reference, and Model Output
A nnrturbation_sstrength value ("subtle" or "obvicus")
Instructions:
Apply perturbation
Model output that **maintains** performance on the specified Dimension.
Under the heading **Potturbed Output:** return exactly one string: For **aubtle** strength, apply minimal change that does not impair the target Dimension.
For **obvious** strength, apply a more notice noticeable change that still keeps the target Dimension intact.
Some examples of types of perturbations would include: rephrasing, reordering, replacing words with synonyms, stylistic changes, etc. that do not impair the target Dimension.
If any change would harm the specified Dimension, simply return the original Model Output. After producing more reascning, explanations, examples only the perturbed text.
original plan/reasoning do +*oo+* include any
Inputs:
35

---

Published as a conference paper at ICLR 2026
Field
Type
Description
task
str
The task that the model was originally trying to complete
input
str
The input provided to the model The references of good outputs (may be None)
references Union [list
None]
str
The output produced by the model
perturbation.
Literal [' susble'' 'obvious']
The strength of the perturbation (subtle obvious)
strength
dimension
str
The aspect of the model output that MUST be preserved in quality
Outputs:
Field Type
Description
cectuurbed_uupttt str
Perturbed text that preserves performance along the given Dimension.
Signature:
LiMMsAbuugeSiigaature
Instruction:
Given
input text, the task description that the model was trying follcw, and measure to rate the text on, return score this measure.
Inputs:
Field
Type
Description
text
Any
The input text that we want to rate. A description of the task that the model was trying to solve when generated the text. Could be left blank if not available.
task_description Any
measure
Any
The measure that want to rate the text on.
suggested_range Any
The suggested range of possible values for the measure.
Outputs:
Field
Type
Description
score
Any
The score that the text should receive on this measure.
Signature:
Instruction:
I am looking for metric to evaluate the attached task. In particular care about the specific target measurement that attached.
36

---

Published as a conference paper at ICLR 2026
Please help decide from among the metrics that I have attached documentation for which one is mest relevant to the task and target.
Please provide de a ranking of the metrics from most relevant to least relevant for the task and target above. can reason first about what makes metric relevant for the task and target, and then provide your ranking.
IMPORTANT:
The final ranking should be list of EXACT metric class names (no hyphens,
words) Use the METRIC documentation. For example, use "SelfBLEU" not "ScIE-BLEU", use "BERTScoro" not "BERT Score", use "BLEJ" not "BLEU Score".
spaces,
NAME not what it is called
The final ranking should just be list matric names, in order from most relevant to least relevant. The 1ist should be exactly mum_merrics_to recoomenccec items long.
Inputs:
Field
Type
Description
lask_descriptinn str
A description of the task that an LLM performed and that now want evaluate.
target
str
The specific target measurement that want to evaluate about the task. list of metric names and their documentation. The documentation will contain the metric name, as well as many details about the metric.
metric_ documentation
List [str]
num_metrics to_recommend
int
The number metrics to recommend. It is imperative to target this number very very close to We will do more extensive filtering later.
Outputs:
Field
Type
Description
ranking
List [stz]
numbered list EXACT metric class names (no hyphens, no spaces, no extra words), in order from most relevant least relevant. The list should be of length 'num nommmmtrriss.om to recommennd', You should write the number in front of the metric name (e.g '1. METRICI NAME', '2. METRIC2 NAME', etc.).
REMEMBER: Put quotes around EACH number metric name pair, not just one of quotes for the full string. IMPORTANT: Refer to "METRIC NAME: for the exact name of the metric or it won't 't be match.
37

---

Published as a conference paper at ICLR 2026
Signature:
GenerateRubriccSignauuus
Instruction:
Given dataset, task description, and evaluation metric, generate a rubric for the metric scoring from to 5.
Inputs:
Field
Type
Description
task_descriptiin Any
description of the task that the model is trying to solve.
@ood_examples Any
A list of good examples outputs for a model.
bad_examples
Any
A list of bad examples of outputs for model.
metric_title
Any
The title the metric. A description of the metrie.
metric_ description
Any
Outputs: Type
Field
Description
score_one_ description
Any
A description of what a score of means. This can be a bullet point list what criteria to look for in assigning a score 1.
of
score two description
Any
A description of what score of 2 means. This can be a bullet point list of what criteria to look for in assigning score 2.
score_three_ description
Any
A description of what score of means. This can be a bullet point list of what criteria look for in assigning a score of 3.
score_four. description
Any
description of what a score of means. This can be a bullet point list of what criteria to look for in assigning a score of 4.
scvee_five_ description
Any
description of what a score means. This can be a bullet point list of what criteria to look for in assigning a score of 5.
Signature:
Generateektioffvvaraattiiiinaaau
Instruction:
Given task description, target metric, good/bad examples, generate a list of axes of variation which could be used to explain the differences between the good and bad examples. These axes of variation will be used as measures to evaluate the model's performance, so they shculd be informative and useful for the model to improve on.
Inputs:
38

---

Published as a conference paper at ICLR 2026
Field
Type
Description
dask_description str
description of the overall task the model is trying to solve.
target_name
optional(str)
Optional hint of the target
metric/column we care about. Could be 'None' or something generic like "quality' or 'score'.
good_examples
[str]
A list of examples with *high* quality according the target metric.
bad_examples
List [str]
A list of examples with *low quality according the target metric.
nun_axes_to
int
The number of axes of variation to generate.
generate
Outputs:
Field
Type
Description
axes_of
List [str]
ordered list (most-important first) describing possible axes of variation. Please bold the name of the axis of variation (e.g. ** Axees Name**), and ALSO include a brief sentence-long explanation of the axis of variation. (e.g. ** Axes Name** Brief Explanation). Please include exactly wenn_axes_ttogeeeee
_variation
axes of variation in the output. Avoid special characters since they sometimes mess up the parsing.
D EXAMPLE METRIC CARD: BLEU
Metric Card for BLEU
BLEU (Bilingual Evaluation Understudy) is widely used metric for evaluating the quality of text generated in tasks like machine translation and summarization. measures the overlap of n-grams between generated text and one more reference texts, with brevity penalty penalize overly short translations. BLCU, modern implementation, ensures reproducibility and standardiza- tion of BLEU scores across research.
Metric Details
Metric Description
BLEU evaluates the quality of generation by comparing n-grams in the generated output with those in one or more reference texts. It computes modified precision for n-grams and combines scores using geometric mean, with a brevity penalty to ensure the length of the generated text matches that of the references. Higher BLEU scores indicate closer similarity to the references.
Metric Type: Surface-Level Similarity
Range: 0 to
Higher Better?: Yes
Reference-Based?? Yes
Reutteeequird??: No
39

---

Published as a conference paper at ICLR 2026
Formal Definition
NV wn log Pr
BLEU = BP exp
BP min(L,e!~r/o) is the brevity penalty.
is the effective reference length (based on the closest matching reference length for each
sentence).
c is the candidate translation length,
is the modified precision for n-grams of length n.
Wn are weights for each n-gram (commonly uniform, Wn = }).
Inputs and Outputs
Inputs:
Generated text (candidate translation)
Reference text(s) (gold-standard translations)
Outputs:
Scalar BLEU score (range: 0 to 1)
Intended Use
Domains and Tasks
Domain: Text Generation
. Tasks: Machine Translation, Summarization, Data-to-Text Generation
Applicability and Limitations
Best Suited For: Structured tasks with clear correspondence between generated and reference texts, such as translation or summarization.
Not Recommended For: Open-ended creative generation tasks where diversity or se- mantic similarity matters more than lexical overlap (e.g., storytelling, dialogue).
Metric Implementation
Reference Implementations
Libraaries/accaaes::
SacreBLEU (robust, standard implementation)
NLTK (basic Python implementation)
Hugging Face evaluate (integrated metric framework)
Computational Complexity
Efficiency: BLEU is computationally efficient, requiring O(n m) operations for n-gram matching where n is the number of words in the candidate text and m is the number of reference words. SacreBLEU optimizes tokenization and scoring, making it highly suitable for large-scale evaluations.
Scalability: BLEU scales well across datasets of varying sizes due to its simple de- sign. BacreBLEU further supports evaluation with multiple references, diverse tokenization schemes, and language-specific preprocessing, making it adaptable to diverse evaluation setups.
40

---

Published as a conference paper at ICLR 2026
Known Limitations
Biases:
BLEU penalizes valid paraphrases semantically equivalent outputs that do match reference n-grams exactly.
The brevity penalty can overly penalize valid shorter outputs, particularly for tasks
where shorter text may be acceptable or even preferred (e.g., summarization).
Task Misalignment Risks:
BLEU is not designed for evaluating tasks with high diversity in acceptable outputs (e.g., open-ended dialogue).
Scores depend the quality and number of references; fewer inconsistent refer- ences can lead to misleading evaluations.
Failure Cases:
BLEU struggles to capture semantic adequacy beyond lexical similarity. For instance, it cannot identify whether translation preserves the meaning of the original sentence word choices diverge significantly.
Related Metrics
ROUGE: Often used for summarization tasks, emphasizing recall over precision.
METEOR: Incorporates synonym matching for better semantic alignment.
BERTScore::
Uses contextual embeddings for semantic similarity.
Further Reading
Papers:
Original BLEU Paper (Papineni et al., 2002)
SacreBLEU: A Call for Clarity in Reporting BLEU Scores (Post, 2018)
Understanding BLEU
BaC Documentation
Metric Card Authors
Authors: Michael Ryan
Acknowledgment of AI Assistance: Portions this metric card were drafted with assis- tance from OpenAl's ChatGPT, based on user- sro-proo inputs and relevant documentation. All content has been reviewed and curated by the author to ensure accuracy.
Contact:
michaeljryan @ stanford.edu
E
AUTOMETRICS DESIGN ABLATIONS
RETRIEVE
For retrieval experiments run all metrics in the MetricBank to get the ground truth kendall correlation on the development set. With this we know the true rank order of the metrics. We then perform retrieval using of retrieval algorithms, namely BM25, ColBERT, Faiss, and using
41

---

Published as a conference paper at ICLR 2026
NDCG
Recall
Method
61
65
@
@020
6f
@5
@ 10
@020
6,024
0.342 =0:11
0.427 2014
0.567 1:066
0.065
no55
0.224
2013
0.788
4:4,11
ColBERT
0,272 LuN1
0.343 1.mm
0.442
0.57
1017
0,089
0.212
1.015
0:441
0.776
1.6.0
Faiss
0.227
0.326
0.018
0.171
0.353
0.694
MRes
0.31 1.0
0.396 1,071
0.478
0.602
0.088
1 m10
0.294 0.770
0.465
0.641
1.0.M
BM25 MLL
0.316
0.42
0.498
0.603
0.0%4
0.312
0.494
0.665
COIBERT-LLMRRG
0.403 0.39
0.462 8.29
0.528
0.238
0.631
0.11
0.094
:: 0.000
0.329
= 0.225
0.518 0.366
0.694 0.221
Taiss MLL
0,164 103e
0,324 10234
0.393
0.15
0.529
0.266
0,065
10000
0.247
0:326
0.4 9.27
0.6 0.70
Table 6: Average performance (# std) across all tasks/axes using Kendall ground truth (recommen- dations from qwen3).
NDCG
Recall
Method
RM25
0,208 1.7
0.342 1omm
0.427 2013
01566 1.16
0,065 n.04
0.224 10
0:418 0.15
0.788 1:031
ColBERT
1um
6.167
0,059
0.429 1
0.38
Faiss
0 uess
0.21 0:22
0.314 =.167
0.461
0,012
0048
0.159 0.30
0.371 0:33
0.729 =:037
LLM
0,261
0.416
0.502
0.585
0,076
0.347
0.518
0.759
BM25 MLL
0,206 u327
0,394 suse
0.47 u.15
0.576
0.362
0.076
:: wode
0.347
0.23
0.512 6222
0.794 c:227
BUBBET--LLMER
0,328 #:31
0.475
0.55 0:311
0.628
0.1
0,388
0,565
0.759 6:33
Faisse
0.157 0:04
0,325 s 0306
0.406 0.200
0.212
0.065 0.55
0.276 0
0.424
0.635 2:33
Table 7: Average performance (# std) across all tasks/axes using Kendall ground truth (recommen- dations from gpt4o-mini).
LLM with all documents in context. We additionally try pipelined versions of all of these retrievers feeding into an LLM. We report Recall@[1.5,10.20] and NDCG@[1,5,10,20)) in Table Qwen3- 32B and Table 7 for GPT-4o-mini.
Overall find that ColBERT LLMRee is consistently the best approach for retrieval, performing the best across 14/16 of our evaluation settings. Thus, we use ColBERT - LLMRec for all metric retrieval in the main paper.
E.2 GENERATE
We test eight different approaches to metric generation. Of these approaches five of them are cheap to produce, while three of them are expensiveliine-cosuiing to produce.
For CodeGen we prompt an LLM to propose "axes variation" from high/low exam- ples and then synthesize small, executable Python snippets that implement scoring function (compute score). The generated code is cleaned, validated on a sample, and-if er- automaluomat repaired once by an LLM. We support both reference-free and reference-based variants.
For G-Eval (Liu et al. 2023) we convert each axis into concrete evaluation criterion, auto-generate numbered evaluation steps, and prompt an LLM judge to produce a brief rationale followed by a dis- score (1-5). We request token-level log probabilities and, the final score position (found by scanning backward), extract the logprobs over tokens {1, 2, 4, 5},
and return ((s prompt, rationale). Both reference-free and
the bobbability-weightee expectation > s=1 eeference-based variants are supported.
For Single Criteria (Saad-Falcon et al. 2024) LLM-as-a-Judge, we show high-scoring and low- scoring data points to an LLM and ask for "axes variation." " Each axis becomes its own metric, with the LLM prompted to output integer score from 1-5.
For Rubric (Gunjal et al., 2025) we add an additional step to Single Criteria where we ask an LLM to generate explanations of what 1-5 scores should contain for each rubric item.
For Rubric (Prometheus) (Kim et al., 2024) we first synthesize a five-level rubric (descriptions for scores 1-5) from dataset examples, then use Prometheus evaluator (e.g., M-Prometheus- 14B) to assign scores conditioned on that rubric. This keeps the rubric explicit while using a strong, specialized judge.
42

---

Published as a conference paper at ICLR 2026
Finetune is first expensive to produce metric. For this fine-tune a ModemBERT-- regression head (with LoRA/PEFT) on formatted input-output (and references when available) to directly predict the target score. We use 80/20 train/validation split, optimize with AdamW, and save the resulting adapter as a learned metric that runs without an L.M at inference.
For Examples text length of the LLM judge we determine how many examples we can reasonably sample from each quintile without exceeding the context length. We try 5 randomly sampled sets of uniformly distributed examples as context in an LLM-judge prompt and select the set that minimizes average distance to human labels on the trainset.
separate the provided human-rated examples into quintiles. Based on the con-
For Prompt Optimization (MIPROv2) we run DSPy's MIPROv2 (Opsahl- Ong et al. 2024 op- timizer with aut o mode "medi um" " on the provided data to generate informative examples and rewrite the evaluation prompt for an LLM judge.
Expersive Produce Eiseppio
G-Rool
Siegle
MIRRO2
Jo Do-Diedariie Jaske:
bask
directly esalusi
SunnExal (ooherence)
0.001
0,005
0.173
0,140
0.104
0.226 2:111
0.222
SummExal (oonsitercy)
01083
0.100
0.160
0.135
0.226
0.159
SunaEval
0.057
0.110
0.076
0:146
SunnEval (eleviace)
0,097
0.189
0,343
Prinoek77 lin.
0.105
0.066
0.188
0.196
0,253
0.258
0.145
0.202
(ominioes)
0.123
0.129
0.109
Prinoo577 rine
0.102
0.132
0.067
Simphhaal
SimpDA (fumeyy)
SimpIAA
0,522
0.570
0,546
0,322
SimpIA cimplictt)
01733 015
HelpSter (cmpleuyy))))
0.233 0.83
0.122 8:22
0.184
1.0000
0.221
HelpSterr (oonectess))
0.08 c00
0.270 8:144
0342
0127
0,312
0.144
0.000
0.377
0.417
HilpSteer (vabbiity)
0.290
0..54 -830
0252 -0053
0.40%
6.15
0.108
2/01
HolpStez2 (cobeseace)
0.024 1008
0.116 6319
0.128 LO000
0,192 1.14
0.159
3/0
HolpStee22 (complectt)
0.13 1.04
0.74
6334
0.100 L0054
0.335
HicpSteer2 (csmocncc)
0.62 e:0
0.667
8310
0.245
02:3 -00.6
0z00
0,332 4
0:310
300
HelyStez22 (kdppaltess))
0.063 600
0.134
8315
0:183
0.135 0000
0,200 1
0.369
HolyStez2 (verbosit)
0.224 oz13
0.661
633
0.210
0.34 0004
0.432
0,081
OunifDanntutie ap satten socttfiallyy desigeed for dbeae - 2012 penorsitsttinn and esttnc geaenatios.
PraKGenPoduct (grade)
0.302
0.303
0.201 0000
0.345 1:03
0.216
MeaKeeMMediol (grade)
0.302 0:00
0.265
0.303
0.301 2:001
0.145
0.216 1173
RalliunnrrBal (acoogted)
0:00
2:1.15
CoCyuTTsndPPoess (ugestRainn))
0,308 1.227
0.185 6.36
63353
0.101 L.044
0.121 0/005
0,144 1.595
CoGuuTrure Process (cumusicetiiRaiin)
0.172 1.075
0.285 6.84
0.167
0.165 1/00S
0.220 1:154
0.150 1:15
Oueeene (ousoosseuting)
0.357
0.114
8::00
5.34
0.448 A117
0,413
0
0,588
0
CaGynn Tabaea (spenttaiin)
0.154 6.130
0.467
8113
0.596
0.433 6.11
0.475 0233
0.5M0 1:35
CaGyyn/abuuufoossss (ommunicauiclaiiin))
0.360 0.466
0.608
8.318
0.791
6.30
0.779 -..147
0.736 8U0
0.500 1..25
0.767
Codlyn/lhdorraarrm (ostooeciiatnng
0.163 0017
0.119 8179
0.167
116
0.301 00001
0.634 2
0.M3
1.30
0.300 190
Aserage
0.205
0.281
0.276
0.008
0.297
Table 8: Metric generation performance (Kendall's Tau) with 95% confidence intervals over 5 in- dependent runs. Each generator produces metrics using persistent train sets, then correlation with human annotations is measured on persistent validation sets. Cheap methods (left) generate 10 met- rics per trial, expensive methods (right) generate metric per trial (except finetune which generates 10). Results show correlation between generated metrics and ground-truth human annotations across diverse tasks using the Qwen3 32B model.
F ADDITIONAL. EXPERIMENTS
F.1 ROBUSTNESS FOR ALL METRICS
Here we include the results of the robustness experiment for all baseline metrics tested. We report results in Figure 7
We find that 'Best Metric" tends to be quite stable, while the LLM Based Metrics (DNAEval, LLM- Judge, and AutoMetrics) stand out robustness.
43

---

Published as a conference paper at ICLR 2026
Chrop
Espreaite Esseppics
G-fool
Single (18P))
(hommehzza)
L--Deedutive
directly
0.127
0.299
0,297
(fucaes)
0.185
0.450
0.246
0.601
0.338
0,686
SimplDA cimplictt))
aevelically designsll
(1100 eds
0.304
0.175
0.189
01310 0336
0.121
0.334
Moolan (grwks)
(1190 s:e5
0:701
0.225
0.135
0.180
0.210 010
0.121
0.248
CaGguTuureePooo (cgenRainn))
0.301
0.133
0.223
0,067
CalgnntureePPoo (comnticaioeSriin)
0.233
0.176
0.154
0.226
0,312
Coéginnraee (onpomRRatin)
0.23%
0,474
0.422
0.510
0,513
0.555
0.404
0.753
CalymllhaboCuomss onttoweliiiingg
0.456
Table 9: Metric generation performance (Kendall's Tau) with 95% confidence intervals over 5 in- dependent runs. Each generator produces metrics using persistent train sets, then correlation with human annotations is measured on persistent validation sets. Cheap methods (left) generate 10 met- rics per trial, expensive methods (right) generate metric per trial (except finetune which generates 10). Results show correlation between generated metrics and ground-truth human annotations across diverse tasks using the GPT-4o Mini model.
SimpEval (score)
HelpSteer2 (helpfulness)
CoGym (outcome rating)
1.0
1.0
srore (mownn ()
0.8
0.0
0.6
0.6
0.6
0.4
Northa Elaosiin
0.2
Stabllky
0.0
0.0
0.0
Figure 7: Sensitivity and Stability of all metrics for SimpEval, HelpSteer2, and CoGym
F.2 WHAT METRICS DOES AUTOMETRICS ACTUALLY SELECT?
To explore the question of what metrics AutoMetrics actually recommends we turn to the 25 trials of AutoMetrics for our main correlation experiment from Table 2
We look exclusively the
Qwen3-32B runs We provide a bar plot of metric types in Figure g.
Top Metrics by Type
AutoMetris are dominated by Generated Met- rics. 103 out of the 125 total recommended metrics automatically generated. Of the Existing met- rics that were recommended 20 out of 22 were rec- ommendations
Single Criteria
67
Rubric
Examples Judge
reward model. This suggests that the scope of metrics to retrieve from can be dra- matically reduced to primarily recommending from the generated metrics as well as a few key reward models and other model based metries like "ParaS- coreFree". This insight will in practice greatly sim- plify the search space for metrics and lead to more streamlined MetricBank.
GRMMewardMMddl
10
ReFoMMeewarMMedel
10
Optimized Judge
10
Generated Metrics Existing Metrics
ParaScoreFree
20
40 Count
60
F.3 VALIDATING SENSITIVITY AND STABILITY
Figure 8: Breakdown of metrics recommended by AutoMetrics. Generated are most common.
In order to sanity check our sensitivity and stability scores we asked collegue not involved in our project to annotate 150 datapoints from SimpEval Maddela et al. (2023) using the original annotation rubric described in the paper. SimpEval consists of original and simple sentence pairs. We asked them to annotate 30 pairs from the original dataset, 30 pairs where the simplified sentence was perturbed in a way that does not change the quality, and 90 sentences perturbed to purposefuly degrade the quality. All perturbations were following our
44

---

Published as a conference paper at ICLR 2026
methodology described in 3.2 Our human annotations yielded sensitivity of 0.8275 and stability of 0.8000 suggesting the perturbations produced the intended effect.
G AUTOMETRICS EXAMPLES
SimpEval score
Overall Kendall 0.3234
Top 5 Metrics Coefficients Metric
Coefficient
Audience htt.inoc..ppooiiiateeesssseees-s Conciseness Qwen3-32B Readability Score Qwen3-32R Mlartty.and.Rddaailltyyt..u ParaScoreFree
1.7066
1.6676
1,6622
1.6345
-1.6125
Description: Audience Appropriateeesssseeeeeaa
Tailors language and phrasing to suit general audience with minimal prior knowledge of the topic.
Description: Conciseness Qwen3-32B
Eliminates redundant phrases, wordiness, tangential details while maintaining the orig- inal intent.
Description: Readability Score Qwen3-323
Measures the text's ease of reading using standardized metrics (e.g., Grade Level).
45

---

Published as a conference paper at ICLR 2026
Description: Clarity and Readability Rubric
Score
Description
The text is difficult to understand due overly complex sentence structures, passive voice, or ambiguous parasing.
Redurdant or redundant information is included, hindering clarity.
Sentences
excessively long fragmented, making it hard follow the main idea.
Jargon or technical terms are retained without simplification, The output fails to restructure the original sentence for broader accessibility.
The text is somewhat clear but still contains occasional complex structures passive voice.
Some sentences are overly long or include minor redundancies. Ambiguity or unclear phrasing is present in parts of the output
Simplification is attempted but incomplete, leaving original complexity intact.
The main idea is generally understandable but requi res effort to parse.
3 - The text is mostly clear, with mostly active voice and straightferward phrasing.
Sentences are concise and well-structured, though few may retain slight complexity.
Minor ambi guities or redundancies are present but do net significantly hinder understanding.
Simplification is effective for the message, though details may remain dense.
The output is accessible to a general audience with minimal effort.
- The text is clear and uses active voice consistently, with minimal passive constructions.
Sentences are concise, well-structured, and free unnecessary
complexity.
Ambiguity is largely avoided, and phrasing is precise. Simplification is thorough, with original complexity reduced to enhance accessibility.
The output easy to understand for a broad audience, with only minor improvements possible.
- The text is exceptionally clear, using active voice and simple, direct sentence structures.
All phrasing is unambiguous, and sentences are optimized for readability.
Redundancy and complexity are entirely eliminated, with the core message distilled to its essentials.
iiptifiicaionn is flawless, making the content immediately accessible
audiences.
The output exemplifies best practices clarity and readability, with no room for improvement.
46

---

Published as a conference paper at ICLR 2026
Description: ParaScoreFree
ParaScoreFree is a reference-free evaluation metric designed for paraphrase generation. It evaluates candidate paraphrases based semantic similarity to the input source while encouraging lexical diversity. ParaScoreFree outputs scalar quality score that combines BERT-based semantic similarity and normalized edit distance, offering a balance between meaning preservation and surface-level rewriting. It enables paraphrase evaluation with- out the need for gold reference texts, making suitable for low-resource or open-domain settings.
HelpSteer2 helpfulness
Overall Kendall r: 0.3481
Top 5 Metrics & Coefficients
Metric
Coefficient
INFORWRewardModee!
0,2046
Helpstter22.helpffunessssse 100--32BBoo0 -tw.
0.1853
GRMRewardModel
0.1697
hel.ffulrees_prre3222eeeeeamm/l accuaac..nnscorrretteess..eee
0.1661
0.1625
Description: INFORMRewarddModel
The INFORM Reward Model 70B (INF-ORM-LIama3.. 1-70B) is a large-scale outcome reward model designed to evaluate the quality of generated conversational responses. It predicts scalar reward scores for response texts, supporting preference-basee fine-grained evaluations without requiring reference response. The model is finetuned from the 1oama-3.--70--Insee backbone using preference-labele datasets, employing scaled Bradley-Teryy loss incorporate preference magnitudes.
Description: HelpSteer2 helpfulness Qwen3-32B optimized seed45
Given the task description, evaluation axis, input/outpin texts, and suggested score range, analyze the output text's a al anignment and axis by:
with the
*sseessing factual accuracy=:: Verity if claims the output are correct and supported by the input/text domain knowledge.
2. Evallattin relevance**: Determine if the output addresses the user's intent directly, avoiding verbosity or tangential content.
3. Analyzing structure and clarity*4:
Check if explanaticns are concise, legically organized, and accessible to the target audience (e.g., non-experts).
Iddettifiing gaps errors**: Highlight missing key details, misinterpretatiins,, or inaccuracies that reduee helpfulness, 5. * Scoring**: Assign à numerical score within the suggested range, balancing the above factors.
Use the conversation history and task description guidance for context and expectations. Pricritize precis in reasoning and al ignmentt with the evaluation axis.
Showing 0 of total examples.*
47

---

Published as a conference paper at ICLR 2026
Description: GRMRewardModel
The GRMRevardMode is a general-purpose reward model designed to evaluate the quality and safety of L..M-generated outputs. It achieves high generalization perfor- mance by applying a novel regularization method on hidden states during supervised fine-tuning. GRddewardModel is fine-tuned on the decontaminated Skywork/Skywork Geward-Preeferece-80K, -00k00..2 dataset and achieves state-of-the results among models of comparable size (3B), even outperforming some 8B reward models and proprietary LLM judges on RewardBench.
Description: Dellpuulness..wen332222 _exampleeeeseesseeeeee__
Input Text
Score
<Input (prompt): <Can you teach me semi semi-definite programming ng in simple language?> Output (zesponse): <Can you teach how to use a computer in simple language?>>> <Input (prompt): <Delve into the nuanced benefits of engaging in group projects instead of the solo endeavor of individual projects. Craft your insights in well-organized format, employing distinct headings for each category. Populate each section with a thoughtful list, elucidating each approach's s merits and drawbacks. This approach aims to enhance clarity and the discussion's overall academ...
<Input (prompt): <The misery of 1ife never appears in clearer light than when a thinking person has quite plainly seen with horror its hazards and uncertainties and the total darkness in which he lives; how he cannet find anything solid, secure, and beyond dispute
to which he can hold; when, after such thoughts he does not at once destroy existence that is not ene, but breathi...
say,
+Showing 3 of 10 total examples.*
Description: Accuracy and Correctness Qwen3-328
The factual correctness and reliability of the information provided.
EvalGenProduct - grade
Overall Kendall r: 0.4178
Top 5 Metrics & Coefficients
Metric
Coefficient
waleaaacceewwen3-3333 Qwen3-32B
0.1144
gradee.vven3-33BBexxmmml
0.1022
0.0752
Customer Review Integration Rubric
0.0747
Avoidance of Weaknesses -
0.0653
48

---

Published as a conference paper at ICLR 2026
Description: Formatting Compliance Qwen3-32B
Good examples strictly follow Markdown structure (headers, bullet points). Bad examples include disallowed elements (links, markdown errors).
Description: grade Qwen3-32B examples
Input Text
Score
-----------_
expert copywriter. You need write an e-cenmerce product description based on the product details and customer revicws. Your description should be SEC- optimi zed. It should use an active voice and include the product's features, benefits, unique selling pointa without overpromi sing, and call action for the buyer. Benefits describe how product features will wor... <Input (Prompt): <You are an expert copywriter. You need to write an e-cenmerce product description based the product details and customer reviews, Your description should be SEC- optimi zed. should use an active voice and include the product's features, benefits, unique selling points without overpromising, and call to action for the buyer. Benefits describe hew product features will wor... <Input (Prompt): <You write an e-cemmerce product description based on the product details and customer reviews. Your description should be SEC- optimized It should use an active voice and include the product's features, benefits, unique selling points without overpromi sing, and call actien for the buyer. Henefits describe how product features will wor...
<Input (Prompt): <You
expert copywriter. You need
*Showing 3 of total examples.*
Description: Call to Action CTA
.TA Strength Qwen3-32B
Good examples include urgent, benefit-driven CTAs (e.g.. 'Order now for seasonal sav- ings'), while bad examples have vague missing CTAs.
49

---

Published as a conference paper at ICLR 2026
Description: Customer Review Integration Rubric
Score
Description
*NNo customer reviews included** all quotes fabricated.
Reviexs are irrelevant to the product or its benefits.
Over-cites testimonials (e.g., 5+ quotes) or includes negative feedback.
Quotes are generic (e.g., "Great product! !") without specific context.
- -*Miimmall or inconsistent use of customer reviews** (e.g
* 1-2 quctes).
Quotes vague lack specificity (e.g., "I love this product!").
Reviews may include irrelevant details or fail to align with the product's features/benefita
No clear connection between test iastimnnilss and the product's unique selling points.
**Moderate use of customer reviews (e.g., 2-3 quctes). Some quotes are specific and relevant (e.g., "This product works well for dry skin").
May include 1-2 generic slightly over-cited testimenials. Reviews are integrated but not strongly enhance the descripticn's persuasiveness.
4 - E.ffectvve use of 1-2 authentic, contextually relevant quotes++.
Testimonials highlight specific benefits (e.g., "The lightweight formula makes it perfect for travel"). Quotes are concise, avoid over-citing, and align with the product's features.
Reviews integrated naturally into the description without overwhelming the reader.
|- **Exce lent integration of 1-2 highly specific, authentic testimonialss..
Quotes directly tie to the product's unique selling points (e.g "The smudge-proot formula lasts all day").
Reviews are concise, impactful, and enhance the description's credibility.
No fabricated, irrelevant, or over-cited quotes; testimenials feel organic and persuasive.
*Showing 3 of total examples.*
Description: Avoidance wwoidance.of..eaknneeeee deaknesses..qenn3
Good examples omit product drawbacks. Bad examples inadvertently mention flaws (e.g.. "may clog pores") or use hedging language.
RealHumanEval accepted
Overall Kendall 7: 0.1487
Top 5 Metrics & Coefficients
50

---

Published as a conference paper at ICLR 2026
Metric
Coeflicient
GRMRewardModel
0.0325
INFORWRewarrdMode!
0.0293
Code Code-Ieadabbllttyy--e33-----
0.0283
Rea
0.0234
0.0218
Description: GRMRewaarddodel
The GRMRewardModel is general-purpose reward model designed to evaluate the quality and safety of LLM-generated outputs. It achieves high generalization perfor- mance by applying novel regularization method hidden states during supervised fine-tuning. GRMRewardModel is fine-tuned on the decontaminated Skywork/Skywort Sreerrd-Peefeeece-8800 80K-v0.2 dataset and achieves state-of-the-a art results among models of comparable size (3B), even outperforming some 8B reward models and proprietary LLM judges on RewardBench.
Description: INFORMRewardModell
The INFORM Reward Model 70B (INF-ORM-Llama..1-)) is large-scale outcome reward model designed to evaluate the quality generated conversational responses. predicts scalar reward scores for response texts, supporting preference-based fine-grained evaluations without requiring a reference response. The model finetuned from the Llama-3.1 1-ttt--nstttt Instruct backbone using preference-labeled datasets, employing scaled Bradley-Terry loss to incorporate preference magnitudes.
Description: Code Ret.eeaaaaattyy.eee333222222
Clarity of variable names, structure, and comments for maintainability.
Description: haw.HHawaaaalllcccoeeeeeeeeeeeea
You are an expert Python code reviewer in a high-stakes seftware engincering environment where code correctness directly impacts mission-critical systems (e.g., financial transactions medical devices, or autonomous vehicles). Your task is to avaluate the AT-generated code output for **absolute correctness** and #ccomleeeeesss* along the specified evaluatien axis. A single error could lead catastrophic failures. Analyze the code with extreme rigor, checking for:
1. ++Logical correctness** (does it solve the task as described?)
2. **ntta validity** (Python 3 compliance, no placeholders like 'xrange ()' 'raw'_input ()
Edge case handling** (negative numbers, empty inputs, etc.) rigor** (valid algorithms, no arbitrary values like "b = 3')
5. #*Functionaaiitt (working return statements, incomplete legic).
stubs
Assign a score between 0.0 and 1.0, where 0.0 means the code is non-functicnal or completely ignores the task, and 1.0 represents flawless implemeen impllmmentaioon. Use the input/output. text and conversation history for context.
SSowingg of total examples.*
51

---

Published as a conference paper at ICLR 2026
Description: Modularity and Reusability Qwen3-32B
Code organization into reusable functions/methods with clear separation of concerns.
CuGymTTraveOOutcomme - outcomeRating
Overall Kendall r: 0.4301
Top 5 Metrics & Coefficients
Metric
Coefficient
Rubriii Rubric
0.1963
0.1927
Accommcartsooooottoossseeee----
0.1824
Cac.omeeRaiingggweee333 examples
0.1674
Feasibility.anddRealll RealissidReeeliee onee 3ww.mmwwen33333
0.1620
52

---

Published as a conference paper at ICLR 2026
Description: Cultural and Local Integration Rubric
Score
Description
.*Score :
No mention unique local experiences or cultural highlights.
No authentic food/dining recommendations.
Generic or irrelevant suggestions (e.g., luxury dining for budget trip).
Fails to address the user's query or intent. - S.ore 2 (Weak)
Minimal mention of local experiences (e.g., 1-2 generic activities like "visiting market").
Vague food/dining suggestions (e.g., "try local cuisine" without specifies).
Lacks integration of cultural or seasonal traditions (e.g., no mention of KFC for Christmas).
Missing linka or references to lecal resources.
**Score 3 :
Includes 1-2 specific losal experiences (e.g., visiting Jigokudani Monkey Park).
Mentions 1-2 authentic food/cining options (e.g., "try misc ramen").
Some cultural or seasonal references (e.g., "KFC popular for Christmas"),
Limited use of links or resources to support recommendaticns. 4 - Score 4 (Good) :**
Includes 3-4 unique local experiences (e.g., monkeys, winter illuminations, regional festivals). Highlights 2-3 specific, culturally significant food/dining options (e.g., "try KFC for Caristmas, " "visit local ramen shop")
Integrates cultural/seasonal traditions (e.g., "Christmas markets in Hckkaido").
Provides 1-2 links to local events, businesses, or rescurces. 5 .corr 5 :Excellent)::+
Includes 5+ unique, deeply integrated local experiences (e.g., snow monkeys, winter illuminations, regional festivals, and lesser-known gems).
Highlights 3+ specifie, culturally sigmificant food/dining options with detailed descriptions (e.g., "try miso ramen at specific shop]")
Seamlessly integrates cultural/seascnal traditions (e.g., "KFC for Christmas, winter illuminations in Sapporo").
Provides 3+ links to local events, businesses, or resources (e. g., [Sappero White Illumination] (url)).
Tailors recommendations to the user's s preferences and avoids irrelevant suggestions.
Description: Cultural. Local Excaala.xxeeeeeesss 3wtprriennneseqenn33222322 Qwen3-32B
Incorporates authentic local traditions, cuisine, and unique regional activities.
Description: Accommodat.ionn.ptiionsssseee
Recommends diverse lodging choices (c.g., luxury hotels, budget hostels, or ryokans).
53

---

Published as a conference paper at ICLR 2026
Description: out comeeating Qwen3-32B examples
Input Text Score
<Input (query): : want go Hong Kong and spend less money Output (outcome): <+<luxury Entertsinment** **Fine Dining Michelin-starred restaurants: Hong Kong boasts a wide array of Michelin-starred restaurants offering diverse cuisines. Examples include Caprice (French), Lung King Hee Cantonese), and Amber (Contemporary).
1**
Private dining
experiences: Many high-end restaurants offer private dining rooms for a more.... 1.0
<Input (query): <Kauai> Output (cutcome): < Kauai Itinerary: Days of Scenery \ Bro Time (Grad Student Budget) **Irip Overvicw:** This itinerary focuses on experieneing Kauai's natural beauty while keeping costs down. It's designed for a group of bros looking for a relaxing and scenic getaway in late March. **Weahhrr Considerations:- of the rainy season in Kauai. Expe... 2.0
March is the tail end
<input (query): <I will go to yangzhou city in China> Output ( outcome): <Thank you for sharing your interests in history and food! Could you please let me know how many days you plan to stay in Yangzhou? This will help create well-structured itinerary for your trip.>> 2.0
+Showing 3 of total examples.*
Description: Feasibility and Realism Qwen3-32B
Ensures the itinerary is realistic in terms of time, distance, and activity density.
TauBench reward
Overall Kendall 7: 0.2469
Top 3 Metrics & Coefficients
Metric
Coefficient
0.0803
Escalation Appropriateness Rubric
0.0599
0.0567
54

---

Published as a conference paper at ICLR 2026
Description: Membership Benefit Application Rubric
Score
Description
**Score (Fails apply rules)
Incorrectly assigns free baggage allowances regardless of membership tier or cabin class.
Applies insurance benefits to users who do not meet eligibility criteria (e.g., no insurance, basic economy).
offers compensation certificates to users who are not eligible (e.g., regular members without insurance).
Fails to enforce policy restrictions (e.g., allowing basic economy cancellations outside the 24-hour window without insurancc).
**Sore (Major errors application)
Applies baggage allowances inconsistently (e.g., correct for some tiers but not others)
Mi Misapplies insurance eligibility (e.g., allows refunds for lancellations without valid reasons).
Offers compensation certificate in most cases but mi misses key eligibility criteria (e.g., ignores membership tiez).
Cccasionally transfers to human agents unnecessarily due to incorrect benefit application.
**Soore (Partial acherence with minor errors) **: Correctly applies baggage allowances for most membership tiers but has occasional errors (e.g., miscalculates free bags for gold members).
Applies insurance eligibility in cases but fails in edge cases (e.g., business class cancellations without checking insurance status).
Offers compensation certificates in most eligible scenarics but occasicnally misses conditions (e.g., delayed fiights without verifying membership).
Rarely transfers to human agents due to minor benefit application issues.
4 ..Score 4 (High adherence with rare errors) Correctly applies baggage allowances for all membership tiers and cabin classes in most cases.
Applics insurance oligibility accurately in nearly all scenarics.
offers compensation certificates in all eligible cases but has one minor cversight (e.g., miscalculating certificat amounts for multi-passenger reservations).
Transfers to human agents only when necessary and for valid reasons.
.*Score 5 (Perfect adherence)**::
Always assigns free baggage allowances correctly pased en membership tier and cabin class.
Applies insurance eligibility and compensation rules flawlessly adhering strictly to policy.
Never offers ineligible benefits (e.g., no certificates regular members without insurance).
Transfers to human agents only when the request falls cutside the scope of membership benefits.
55

---

Published as a conference paper at ICLR 2026
Description: Escalation Appropriateness Rubric
Score
Description
**Fails transfer.* in all where policy limits are reached or exceptions are needed,
** Incorreectly handles** requests that require human
interventicn (e.g., proceeds with booking/cancn canceiinn flights outside pelicy).
adherence** to the rule of transferring violations or exceptions.
policy
2 TTransferss inconsistently (e.g., transfers in scme policy-viclating cases but not others).
**Fails to transfer** for critical exceptions (e.g., basic economy cancellations without insurance, destination changes). **Attempts to resolve** issues beyond its scope (e.g., modifying flight destinations, waiving fees without human input)
**Trassfers most policy-violating cases (e.g., denies basic economy cancellation and transfers to human agents).
**Fartially handles exceptions** (e.g., transfers for compensaticn requests but not for all policy violations).
**Some errors** determining when to escalate (e.g.,
transfers unnecessarily for minor issues)
**Consistently transfers** when policy limits are reached (e.g., denies basic economy cancellations, blocks destination changes).
**Transfers for exceptions** (e.g., user insists on refunds for non-refundable tickets, requests compensation for delays). +*Minimal errors** in escalation decisions, with clear adherence to policy boundaries.
**Perfectly transfers** all required cases (e.g., policy viclations, exceptions, ambiguous requests). * Never attempts to handle** requests outside its scope (e.g., denies basic economy cancellations, blocks invalid modificaticns).
*rroactively transfers** when user intent is unclear or requires human judgment (e.g., personal emergencies, compensaticn negotiations).
Description:
Adherence to airline rules (c.g., no basic economy cancellations without insurance or 24-hour window).
56


Reviews:
[{"review_id": "4GLO3dtjTE", "reviewer": "Reviewer_bvUH", "summary": "The paper introduces AutoMetrics, a framework for building task-specific evaluation metrics from fewer than 100 human labels. It generates new LLM-based metrics, retrieves existing ones from MetricBank, and combines them via PLS regression into a composite metric that best predicts human feedback. Results across five text-generation tasks show moderate improvements in Kendall’s $\\tau$ over baselines (LLM-as-a-Judge, existing metrics), and the paper introduces “Sensitivity/Stability\" tests and a $\\tau$-Bench proxy-reward case study.", "questions": "1. Given the small-N (N=80) and large-P (k=30) nature of the initial regression, how stable are the selected metrics? If you were to bootstrap the 80 human-labeled samples, would the top 5 selected metrics and their weights vary dramatically?\n\n2. What does the failure case look like (e.g. GPT4o in CoGym)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The pipeline is clearly described and practically implementable.\n\n2. The construct validity probes (Sensitivity/Stability) are a useful addition to standard correlation metrics.\n\n3. The low-data claim (performance saturating at 80–100 samples) is empirically supported and relevant.\n\n4. The $\\tau$-Bench case study is interesting, showing the learned metric can act as a reward signal.", "weaknesses": "1. On out-of-distribution tasks, “Generated-only” metrics perform as well as or better than the full hybrid version, suggesting the Retrieve step (MetricBank) adds little value.\n\n2. The PLS regression and metric generation both depend on the same small set of labels, and I am not sure if this is the principled approach (compared to split the data half and half).\n\n3. The AutoMetrics pipeline seems more complex and computationally expensive than the baselines it's compared against.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnEiAAyAjq", "reviewer": "Reviewer_raWk", "summary": "The paper proposes AutoMetrics, a novel framework for automatically constructing evaluation metrics that approximate human judgments in subjective, open-ended tasks with scarce human feedback data. AutoMetrics consists of the four steps: generation of task-specific criteria (both from LLM-generated and existing rubrics), retrieval of top-k metrics, regression with PLS equations between the top-k metrics and gold human labels, and reports of metrics for interpretability. AutoMetrics achieves a higher correlation with human preferences than existing baselines, with only 80 human examples.", "questions": "- As a suggestion for a stability metric, I would recommend a metric of RBO (rank-biased overlap) to measure a correlation between the rankings of AutoMetrics and human labels. RBO is top-weighted and allows unequal-length lists, a fairer metric than Kendall's tau. If you want to examine “do different runs of AutoMetrics pick roughly the same top metrics?”, RBO gives a more interpretable measure of set-level stability of the head of the ranking. Kendall's tau is a measure of pairwise ordering over the entire list, which can look bad even when the top 5 are identical.\n\n- Generally, the paper lacks a dedicated section on limitations. \n\n- (Not Required but Strongly Recommended as additional analysis to boost your claim) Did you run any informal user study where human users actually used these \"reports\" from AutoMetrics (step 4) or any adoption rate? Even a brief description would help justify this part of the contribution, measuring the actual implication/impact of AutoMetrics.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper was quite easy to read and understand, with good clarity of method explanations. \n- Timely topic for LLM-guided evaluations for subjective tasks with scarce human feedback data available. It was highly interesting to show that AutoMetrics can be a good alternative as a reward model for agent optimization with reduced cost and complexity of LLM alignment.", "weaknesses": "- The regression step fits PLS over a relatively large set of highly correlated candidate metrics after top-k retrieval. While AutoMetrics achieved a higher Kendall correlation with human annotations over baselines (Table 2), the noisy or negative tau values on the smallest tasks (notably CoGym) may indicate a known problem of spurious correlation in high-p, low-n settings induced by too many weak predictors. I acknowledge the claim that < 100 labels needed is true, but it needs more clarification to prevent a \"conditionally true\" statement. It seems that Autometrics could work when (1) generated, task-specific metrics are the majority among the top-k ones or (2) the task objective is close to existing metrics in MetricBank, but degrades when the top-k contains many loosely related metrics.\n\n- AutoMetrics consists of multiple LLM-generated evaluators, thus highly likely to contain inherent biases. The paper does not run the standard robustness test, such as learning the metric on model A's outputs and evaluating on model B's outputs. Some analysis of a simple cross-model generalization table (e.g., generating/regressing using GPT-4o-generated metrics, testing on Qwen outputs) can make the claim of a \"human-aligned\" (after regression) approach of automatic evaluator selection processes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVCZhSK4Bt", "reviewer": "Reviewer_5zyv", "summary": "The paper proposes a system that automatically creates and learns evaluation metrics for LLM-generated outputs when you only have a small amount of human feedback. It does this by using LLMs to generate candidate evaluators like rubrics and judges, and then statistically combining the best ones so that their scores align closely with real human judgments.\n\nThe fundamental problem this approach addresses is the situation where you don’t have a reward signal to optimize models: data scarcity, limited human judgments, and no ability to run full-scale preference data collection. These automatically generated metrics can then act as a proxy reward model of sorts. In other words, it’s essentially a framework for **operationalizing limited human judgments.**\n\nIn terms of validity, the authors calibrate their learned metrics to human data using Kendall’s tau correlation, so the system is empirically optimized to fit whatever limited judgments are available. It also performs well on internal coherence tests, scoring high on both sensitivity and stability, meaning it penalizes degraded outputs and stays consistent under irrelevant variation, at least within the evaluated tasks. However, this evaluation setup is circular in logic: the system’s sensitivity and stability are tested using examples generated by other LLMs, so it is being validated by the same kind of model it’s built from. These tests therefore measure LLM-to-LLM agreement rather than genuine alignment with human reasoning.\n\nBut the claimed validity of this approach is itself limited in a more fundamental way. The authors demonstrate that their generated metrics align with human judgments, but those judgments come from a small, potentially unrepresentative sample of annotators or users. In other words, the system’s validity is only as sound as the data it’s validated against. If the underlying human feedback lacks breadth, consistency, or reliability, then demonstrating correlation with it doesn’t establish true validity: it just shows agreement with a narrow and possibly biased slice of human opinion. \n\nDespite this limitation in validity, it is my opinion that the system still demonstrates a useful way to operationalize limited human judgments.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The reframing of evaluation under limited data as a learning problem is really the \"aha\" moment of this paper. Operationalizing limited human data into a compositional and interpretable set of metrics is useful. The paper validates its approach across 5 domains, and provides convincing ablations of each component's contribution. Relatively well-written and accessible.", "weaknesses": "The main weakness is in its grounding of validity. The authors demonstrate strong correlations with small human-labeled datasets, but those datasets themselves are limited, noisy, and unrepresentative, which undermines the strength of their claims. Demonstrating correlation with a weak or narrow gold standard does not establish that AutoMetrics captures genuine human judgment, only that it fits that small sample relatively well; but whether this is truly *useful* remains an open question that is not demonstrated. Overfitting the metrics on this small data sample is the aim of the game here. Fundamentally, the missing validation is if human practitioners actually find these automated metrics meaningful or actionable in practice. The interpretability claim also remains untested- the nature of the approach itself is not sufficient evidence of it being \"interpretable\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KrcyHxhl54", "reviewer": "Reviewer_8L96", "summary": "The paper proposes AutoMetrics, a framework for LLM-as-a-judge that automatically generates task-specific evaluation criteria and dynamically assign weights to the criteria to form a scalar evaluation score. The framework consists of 4 steps: Generate, retrieve, regress and report. The paper conducts experiments on 5 benchmarks to show improvement in Kendall correlation with human judgement, compared with standard LLM-as-a-judge approaches.", "questions": "No additional questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper resolves a critical problem in prior works for LLM evaluation that rubrics need to be manually designed. Instead, the proposed framework automatically generates and selects criteria for specific tasks. The paper conducts sufficient experiments to support its claim.", "weaknesses": "Though the framework is claimed as \"automatic\", it still contains human-curated components. For instance, the \"Retrieve\" step exploits MetricBank, a static metrics collection drawn from prior literature. Therefore, it remains in question whether the generated metrics are suitable for all tasks.\n\nParagraph is incomplete from line 304-315, which seriously hinders content understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)